“Any man who hates dogs and babies can't be all bad.”
Although a very common misconception is to attribute the final part of this quote to W.C. Fields himself, it was actually first said about him by Rosten during a "roast" of Fields at the Masquer's Club in Hollywood in 1939, as Rosten explains in his book, The Power of Positive Nonsense (1977).
Context: The only thing I can say about W. C. Fields … is this: Any man who hates dogs and babies can't be all bad.