Bernard Baruch in response to a question by Igor Cassini as to how he handled the seating arrangements at his dinner parties, as quoted in Shake Well Before Using: A New Collection of Impressions and Anecdotes Mostly Humorous (1948) by Bennett Cerf, p. 249; the full response was "I never bother about that. Those who matter don't mind, and those who mind don't matter." This anecdote is also quoted online at Chiasmus.com http://www.chiasmus.com/archive/msg00241.html. It has also become part of a larger expression, which has been commonly attributed to Dr. Seuss, even in print, but without citation of a specific work: "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
Misattributed
Variant: Be who you are and say what you feel because those who mind don't matter and those who matter don't mind.