I think it comes from the fact that they've more obsessed with the way he acts / his personality.
For instance, I always thought, and still partially do, that Benedict Cumberbatch looks like something not born on Earth, but after watching Sherlock, there were times, when in character, he seemed attractive. I guess it's true that not everything is in physical looks, a lot comes from behavior, posture and such.