I agree with the OP. This has nothing to do with "SJW" or "PC". The problem with the word "Homophobia" is that the definition of Phobia is "fear" and they're not afraid of us. They just hate us. And they don't care about the true definition of "Homophobia". Their usual answer is: "I'm not afraid of gays, I just don't agree with their lifestyle".
Words have the meanings we give to them. In this case, the term "homophobia" is widely known to encompass all antigay hatred, bias and prejudice. Trying to change it now would only add to the confusion, and I don't see significant benefits in doing so.
We just talked about this last week in my human sexuality course, my professor said "antigay prejudice" is preferred by psychologists but that's not nearly as catchy
-phobic doesn't just mean fear, it indicates disgust or aversion as well. When you describe something as hydrophobic, you're not saying an inanimate object is afraid of water