Well, I don't think saying "You have character" equates to attraction, first and foremost.
But, I'd say it depends on how its said. There is a difference between "You've got character" and "You're a character" as the former is something nice, while the second comes off as judgmental.