“Satan has his companions, fellow-devils, to admire and encourage him; but I am solitary and detested.”

Einstein's tribute to Pablo Casals (30 March 1953), in Conversations with Casals (1957), page 11, by Josep Maria Corredor, translated from Conversations avec Pablo Casals : souvenirs et opinions d'un musicien (1955)
Variant translations or paraphrasing:
The world is a dangerous place to live, not because of the people who are evil, but because of the people who don't do anything about it.
As quoted in The Harper Book of Quotations by Robert I. Fitzhenry (1993), p. 356 http://books.google.com/books?id=THl7kUfSqCUC&lpg=PP1&pg=PA356#v=onepage&q&f=false
The world is a dangerous place, not because of those who do evil, but because of those who look on and do nothing.
As quoted in Conscious Courage : Turning Everyday Challenges Into Opportunities (2004) by Maureen Stearns, p. 99
The world will not be destroyed by those who do evil, but by those who watch them without doing anything.
1950s

Source: The Life of St. Teresa of Jesus (c.1565), Ch. XXV. "Divine Locutions. Discussions on That Subject" ¶ 26 & 27
Variant translation: I do not fear Satan half so much as I fear those who fear him.
Source: The Life of Saint Teresa of Ávila by Herself
Context: May it please His Majesty that we fear Him whom we ought to fear, and understand that one venial sin can do us more harm than all hell together; for that is the truth. The evil spirits keep us in terror, because we expose ourselves to the assaults of terror by our attachments to honours, possessions, and pleasures. For then the evil spirits, uniting themselves with us, — we become our own enemies when we love and seek what we ought to hate, — do us great harm. We ourselves put weapons into their hands, that they may assail us; those very weapons with which we should defend ourselves. It is a great pity. But if, for the love of God, we hated all this, and embraced the cross, and set about His service in earnest, Satan would fly away before such realities, as from the plague. He is the friend of lies, and a lie himself. He will have nothing to do with those who walk in the truth. When he sees the understanding of any one obscured, he simply helps to pluck out his eyes; if he sees any one already blind, seeking peace in vanities, — for all the things of this world are so utterly vanity, that they seem to be but the playthings of a child, — he sees at once that such a one is a child; he treats him as a child, and ventures to wrestle with him — not once, but often.
May it please our Lord that I be not one of these; and may His Majesty give me grace to take that for peace which is really peace, that for honour which is really honour, and that for delight which is really a delight. Let me never mistake one thing for another — and then I snap my fingers at all the devils, for they shall be afraid of me. I do not understand those terrors which make us cry out, Satan, Satan! when we may say, God, God! and make Satan tremble. Do we not know that he cannot stir without the permission of God? What does it mean? I am really much more afraid of those people who have so great a fear of the devil, than I am of the devil himself. Satan can do me no harm whatever, but they can trouble me very much, particularly if they be confessors. I have spent some years of such great anxiety, that even now I am amazed that I was able to bear it. Blessed be our Lord, who has so effectually helped me!

The Brothers Karamazov (1879–1880)

Majlisi, Bihārul Anwār, vol. 77, p. 149

1880s, Garfield's Words (1882)

Jessica Simpson
Adato, Allison; Ingrassia, Lisa; Baker, K C. (2006-06-12), "The Mane Man". People. 65 (23):107-108
About

Source: Trent's Own Case (1936), Chapter XVII: "Fine Body of Men"