It truly depends on my view of a person. I can hate you, but if I respect something about you like your intelligence or your dress, your words mean something to me. People I view as inferior to myself, on the other hand, can't hurt me at all.
I've always cared what others thought of me. No offense to anyone, but I think the cliché "Idgaf what nobody say" statement is entirely fallacious. We all react to
someone saying positive or negative things about us. We all care.