This is an absurdly black-and-white way of looking at morality. Saying that pro-slavery people shouldn’t be assaulted is not by any stretch the same as being pro-slavery oneself.

I think I have a good way to illustrate this, actually. What do you think about, say, torturing pro-slavery people? Or perhaps their families?

It’s possible you think torturing them and their families is going too far. This “too far” feeling would be you defending them just as much as I am; you don’t support slavery, but that doesn’t mean you think any conceivable misfortune should be inflicted on them. Likewise, contrary to what you say, I don’t support slavery but still don’t think certain actions against them are justified.