“Crimes, like Virtues, are their own Rewards.”

Interview by Mark Bauerlein, " A Solitary Thinker https://www.chronicle.com/article/A-Solitary-Thinker/127464," The Chronicle of Higher Education, May 15, 2001

The Philosophy of Atheism

Part I, ch. 1. Compare: "Virtue is her own reward", John Dryden, Tyrannic Love, act iii, scene 1; "Virtue is to herself the best reward", Henry More, Cupid's Conflict; "Virtue is its own reward", Matthew Prior, Imitations of Horace, book iii. ode 2; John Gay, Epistle to Methuen; Home, Douglas, act iii, scene 1. "Virtue was sufficient of herself for happiness", Diogenes Laertius, Plato, xlii; "Ipsa quidem virtus sibimet pulcherrima merces" ("Virtue herself is her own fairest reward"), Silius Italicus (25?–99): Punica, lib. xiii. line 663.
The Compleat Angler (1653-1655)

Heretics and Heresies (1874)
Context: Nature never prompted a loving mother to throw her child into the Ganges. Nature never prompted men to exterminate each other for a difference of opinion concerning the baptism of infants. These crimes have been produced by religions filled with all that is illogical, cruel and hideous. These religions were produced for the most part by ignorance, tyranny and hypocrisy. Under the impression that the infinite ruler and creator of the universe had commanded the destruction of heretics and infidels, the church perpetrated all these crimes:
Men and women have been burned for thinking there is but one God; that there was none; that the Holy Ghost is younger than God; that God was somewhat older than his son; for insisting that good works will save a man without faith; that faith will do without good works; for declaring that a sweet babe will not be burned eternally, because its parents failed to have its head wet by a priest; for speaking of God as though he had a nose; for denying that Christ was his own father; for contending that three persons, rightly added together, make more than one; for believing in purgatory; for denying the reality of hell; for pretending that priests can forgive sins; for preaching that God is an essence; for denying that witches rode through the air on sticks; for doubting the total depravity of the human heart; for laughing at irresistible grace, predestination and particular redemption; for denying that good bread could be made of the body of a dead man; for pretending that the pope was not managing this world for God, and in the place of God; for disputing the efficacy of a vicarious atonement; for thinking the Virgin Mary was born like other people; for thinking that a man's rib was hardly sufficient to make a good-sized woman; for denying that God used his finger for a pen; for asserting that prayers are not answered, that diseases are not sent to punish unbelief; for denying the authority of the Bible; for having a Bible in their possession; for attending mass, and for refusing to attend; for wearing a surplice; for carrying a cross, and for refusing; for being a Catholic, and for being a Protestant; for being an Episcopalian, a Presbyterian, a Baptist, and for being a Quaker. In short, every virtue has been a crime, and every crime a virtue. The church has burned honesty and rewarded hypocrisy. And all this, because it was commanded by a book — a book that men had been taught implicitly to believe, long before they knew one word that was in it. They had been taught that to doubt the truth of this book — to examine it, even — was a crime of such enormity that it could not be forgiven, either in this world or in the next.

1840s, Essays: First Series (1841), Friendship
Variant: The only way to have a friend is to be one.

Ainsi que la vertu, le crime a ses dégrés;
Et jamais on n'a vu la timide innocence
Passer subitement à l'extrême licence.
Hippolyte, act IV, scene II.
Phèdre (1677)

East and West Poems, Part II, The Legends of the Rhine.

The Six Pillars of Self-Esteem (1994)

Book XIII, lines 663–665
Punica