I never really thought about it - but then again, even though I'm pansexual, I've always considered myself "gay" as an umbrella term. I mean, my same-sex attraction does affect my life and I claim it wholeheartedly - plus I identified as gay first before I began exploring my attraction to other genders. I even call myself bisexual too, just because more people understand that concept more.

But I guess for some people, it would be irritating. For instance a transgender male who is attracted to females is not gonna identify as a gay woman - he's not a woman, and he's not gay - his attraction is not really different to that of a straight cisgender male.

I think LGBT+ would be a more appropriate umbrella term, so it doesn't really bother me but I can see how others might be annoyed.