They are monuments to racism, not reminders of history. They were mostly erected during Jim Crow and the Civil Rights eras. The impacts of the history of slavery and white nationalism belong in contextualized museums, not outdoor public monuments. That is the line between sympathizer and historian.
Welcome to the human race, our whole history is racism, violence, conquest and so on. Nobody is innocent. Today we live in uniquely peaceful times by historical standards, but you can't wipe everything that came before. Trying to do that is stupid and dangerous, because lack of understand of what happened makes it more likely to repeat itself. The best way to revive Jim Crow is to pretend it never happened.