Of course Gaga alone is not the reason, but if u think, she don't bring ratings, you are stupid.
Gaga, Adele, Justin Timbarlake are the main reason (If we talk about artists missing).
There is probably another explanation for the lower ratings as well ..
Gaga could've at the very least, make some headlines on the Red Carpet which is enough to raise ratings a bit .. And if they said "Adele's comeback at the Grammys" probably 30M Viewers