“On the Disc the gods dealt severely with atheists.”

Mon Dernier soupir (My Last Sigh, 1983)

Where All Roads Lead (1922); this is often misquoted as "If there were no God, there would be no atheists."
Context: Atheism is, I suppose, the supreme example of a simple faith. The man says there is no God; if he really says it in his heart, he is a certain sort of man so designated in Scripture [i. e. a fool, Ps 53:2]. But, anyhow, when he has said it, he has said it; and there seems to be no more to be said. The conversation seems likely to languish. The truth is that the atmosphere of excitement, by which the atheist lived, was an atmosphere of thrilled and shuddering theism, and not of atheism at all; it was an atmosphere of defiance and not of denial. Irreverence is a very servile parasite of reverence; and has starved with its starving lord. After this first fuss about the merely aesthetic effect of blasphemy, the whole thing vanishes into its own void. If there were not God, there would be no atheists.

Source: Night-Thoughts (1742–1745), Night V, Line 177.

"The Erring" (1844).
Context: Speak kindly to the erring;
Thou yet may'st lead them back,
With holy words and tones of love,
From misery's thorny track.
Forget not thou hast often sinned.
And sinful yet must be;
Deal gently with the erring one,
As God hath dealt with thee.

Stardust Memories (1980).

ibid., chap. 30

Meaning of Life interview (2008)
Context: Well, God answers of course come in every flavor imaginable these days so God can be process-God can be mind-God … so there are all of these ways that God is now configured as well as the ones that come to us from traditional religions where God has much more power — then there's the whole personal God part which I do talk about in there at some point. So I don't think that even that there is a God framework out there at this point that I am either accepting or rejecting. My response is that I call myself a non-theist as opposed to an atheist because as I see an atheist as having a belief about God, i. e. that there isn't one. And my I've never been actually very interested in the question I guess is one way to put it. I see it as a question That can be summarized in the aphorism "Why is there anything at all rather than nothing." And science doesn't have any answer to That so what I articulated in the book and continued to do is what I call a covenant with mystery where mystery is itself a … noun but I am using it as literally in absence of category. It's not like I have a mystery then I put attributions onto it it just … I don't know the answers.