I always find this kind of scrutiny towards mistresses rather dumb. Not to say they should go unscathed, especially if they know the main they're creeping with is married (and sometimes they don't always know) but for the most part, the man is usually never attacked as harshly as the female when he's the one with the ring and should be loyal to his wife.
It's like people naturally expect all men to cheat so it's no surprise when he does, but lets hang the mistress out to dry.