There would have to be physical attraction, but yes, it wouldn't bother me. I had a crush on one of my ex-boyfriend's roommates who was missing both his legs (he used prosthetics).
If someone was disfigured, it'd be a bit tougher because I'd have to be physically attracted to them. But sometimes that happens after knowing their personality for a bit.
I am friends with someone who has down syndrome and people with missing arms/hands/legs. But personally I would rather just stay friends, it's a personal preference.