Aside from the obvious argument that is "music not the bling" and the fact that she released "Born This Way" when she had the whole world waiting is extremely admirable and sent a far more important message than getting a smash hit (which it was anyway so slé).
That being said, her career obviously went on a downward slope throughout that album era. As some people have already pointed, it's to do with "Judas" - now I love that song and it's still on all of my dance/EDM playlists and I bop to it so whatever. However, unsurprisingly primitive people overly concerned with religion and unable to digest a simple metaphor and had to take it all literally made it into a controversy.
TBH I think it does have a lot to do with "Born This Way" (the song) and people previously being able to accept Gaga's gay stance, but once it spilled literally into her music it turned a lot of people off (charming I know). IDK, I guess Gaga then officially became a "gay" singer and lost a chunk of her general audience.
But it was just finally witnessed throughout the "Judas" era.