Not physically but he had a mental and emotional affair with Neda.
Studies show that women take more offense to their significant other having an emotional/mental affair than a physical one. It's the reverse for guys.
Well, I wouldn't count that as having an affair since you can't help what feelings you develop for other people. The important thing is that he didn't act on them while he was in a relationship.