They are monuments to racism, not reminders of history. They were mostly erected during Jim Crow and the Civil Rights eras. The impacts of the history of slavery and white nationalism belong in contextualized museums, not outdoor public monuments. That is the line between sympathizer and historian.
What about Grant... he owned slaves are we to take down statues of him and erase him from the books of being President?