For clarity and perspective, lets imagine the shoe being on the other foot for a minute.

Calling Sam Harris an alt-right figure, or cynical panderer to such is the equivalent of calling Thomas Piketty a SJW Marxist or panderer to such because his body of work regarding systemic inequality can be shoehorned to fit into Marx's interpretation of Hegel's dialectic, and also tends to validate the assertion that a person's starting position in life heavily influences their likely ending position. He also asserts that 'elites' have captured political process across all major parties, making him somewhat popular with some radical leftists. (just YouTube marx piketty if you want examples of this)

However, trying to label him a SJW or Marxist is absurd, as he is neither of those things. Anyone who tried to dismiss his positions not by addressing his logic or facts, but by trying to poison the well and associate him with people who cherry-pick portions of his work to support their Marxist positions, would also be acting in an absurd manner.

For who could be free when every other man's humour might domineer over him? - John Locke (2nd Treatise, sect 57)