The difference though is that white people won't care about a BLM sign because it's just a movement that doesn't apply to them. For black people, All Lives Matter was created to directly obfuscate and smear the movement that was started with BLM.
So here, you have a person going into a white community supporting a movement that means nothing to them in comparison to that person going into a black community supporting a movement meant to divert one meant to support a specifically black problem. Of course black people are going to be more offended than the white people, especially when you consider the problem at hand doesn't affect white people at all.