You only know people are staring at you if you're staring at them though, which probably means you aren't really that attractive if you stare at people staring at you all the time. I am better looking than 90% of the population so I never feel like looking at basic looking people at all.
Or maybe you aren't looking at anyone and then you notice someone is staring at you?