Honestly, it's knowing that some of the people you're attracted to wouldn't even consider you.
I feel like str8 people don't deal with it as much because, in most parts of the world, heterosexuality is the default orientation, so they just assume that the object of their desire
might actually want them back (regardless of other factors, like being out of their league or whatever). For us, sometimes you really like someone but you KNOW it ain't gonna happen in a million years. And when he's hot and sweet and smart and compatible to you, it kinda hurts a little