Nicolle should stay, she gives a pretty unbiased Republican and Democratic view of issues while alligning herself sometimes on either side of issues. Also, she can hang with the funnier stories. Rosie should stay because she's good for the sports aspect and Broadway aspect. Whoopi needs to stay. Raven is a ok addition thus far. They need to just add one more woman to the table honestly that could be the "controversial" one from a completely different background than the rest and they'll have ratings gold.
The main reason I hate the show sometimes is the interviews with no names from Broadway too often (rather than booking actual celebrities), the lack of good content outside interviews and Hot Topics, and their choice sometimes to do a boring story for a week straight or longer when there isn't any new tea on it (like the Prison Break in NY). They need to offer new unique stories (trust me there's a lot of stuff going un-heard) and use that as their edge up. If they refocus on fluff, the Talk will destroy them. The Talk is the pro of boring stories and fluff yet making it SEEM actually important

. True levels of dumbing it down all around the table.