Madonna - None
Lady Gaga - The Fame (the very first edition of it)
Janet Jackson - All Of Them
Kylie Minogue - X
Beyonce - All Of Them
Rihanna - From MOTS to Rated R
Britney Spears - From Britney to Femme Fatale
Avril Lavigne - None
Kelly Clarkson - None
Björk - None
Katy Perry - None
Nicki Minaj - Pink Friday (and her Mixtapes)
Mariah Carey - All Of Them
Whitney Houston - All Of Them (My mom is a huge whitney stan)

Madonna- Erotica, Bedtime Stories and Hard Candy
Lady Gaga- All
Janet Jackson- None
Kylie Minogue- None
Beyonce- All except B'Day
Rihanna- Loud
Britney Spears- All
Avril Lavigne- None
Kelly Clarkson- None
Björk- None
Katy Perry- None
Nicki Minaj- None
Mariah Carey- Daydream and Music Box
Whitney Houston- None

EDIT: The only physical copies from the list I have are all of Britney's except FF, Daydream and Musicbox, 4, Loud and TFM.

Madonna - Erotica. I'm collecting them slowly.
Lady Gaga - The Fame, The Fame Monster, The Remix, and Born This Way
Janet Jackson - None
Kylie Minogue - None
Beyonce - 4
Rihanna - Talk That Talk
Britney Spears - None
Avril Lavigne - None
Kelly Clarkson - None
Björk - None. I'll collect all of them one day.
Katy Perry - None
Nicki Minaj - None
Mariah Carey - None
Whitney Houston - Greatest Hits

Britney - All
Rihanna - MOTS

All of Katy's

Madonna - MDNA, Hard Candy, Celebration, True Blue, Like A Virgin, Erotica
Lady Gaga - All
Janet Jackson - none
Kylie Minogue - X, Fever
Beyonce - All
Rihanna - All except Music of the Sun
Britney Spears - All studio albums
Avril Lavigne - Best Damn Thing and Goodbye Lullabye
Kelly Clarkson - Stronger, Breakaway
Björk - none
Katy Perry - All including Katy Hudson and the MTV Unplugged
Nicki Minaj - Both
Mariah Carey - Emancipation of MiMi, E=MC2
Whitney Houston = none

Lady Gaga - The Fame
Lady Gaga - The Fame Monster (Deluxe)
Lady Gaga - Born This Way (Standard)
Lady Gaga - The Remix
That's all

I only buy my fave's albums

Madonna - Ray Of Light, Music, American Life, MDNA
Lady Gaga - The Fame, The Fame Monster
Janet Jackson - Discipline
Kylie Minogue - X
Beyonce - B'Day Deluxe, I Am Sasha Fierce Platinum
Rihanna - Music Of The Sun, A Girl Like Me, GGGB, GGGB Reloaded, Rated R, Loud, TTT Deluxe
Avril Lavigne - The Best Damn Thing Deluxe, Goodbye Lullaby
Kelly Clarkson - Breakaway, My December, All I Ever Wanted, Stronger
Björk - Greatest Hits
Katy Perry - Teenage Dream
Nicki Minaj - PF: RR

Madonna - MDNA (deluxe, physical), Celebration (deluxe, digital)
Lady Gaga - All (Includes albums, remix albums (not for singles), singles (except 2) and BTW Vinyl)
Janet Jackson - none
Kylie Minogue - Light Years, Fever, Body Language, X + Aphrodite (Kylie - The Album (2000-2010)
Beyonce - 4
Rihanna - Talk That Talk, Rated R, LOUD
Britney Spears - Femme Fatale, Blackout, Circus, The Singles Collection
Avril Lavigne - Goodbye Lullaby
Kelly Clarkson - none
Björk - none
Katy Perry - Teenage Dream: The Complete Confection
Nicki Minaj - Pink Friday, Pink Friday:Roman Reloaded
Mariah Carey - none
Whitney Houston - none