“In our civilization, and under our republican form of government, intelligence is so highly honored that it is rewarded by exemption from the cares of office.”

1870s, Message to the Senate and House of Representatives (1870)
Context: The framers of our Constitution firmly believed that a republican government could not endure without intelligence and education generally diffused among the people. The Father of his Country, in his Farewell Address, uses this language: Promote, then, as an object of primary importance, institutions for the general diffusion of knowledge. In proportion as the structure of a government gives force to public opinion, it is essential that public opinion should be enlightened.

33
1940s–present, Minority Report : H.L. Mencken's Notebooks (1956)

26 June 1787 per page 105 of "The Debates, Resolutions, and Other Proceedings, in Convention, on the Adoption of the Federal Constitution: Supplementary to the state Conventions" by Johnathan Elliot, published 1830 https://books.google.ca/books?id=-gtAAAAAYAAJ&pg=RA1-PA105
Debates of the Federal Convention (1787)

Statement to a reporter in the Boston Record, 14 April 1903. (quoted in Alpheus Thomas Mason, Brandeis: A Free Man's Life (1946), p. 122.)
Commonly paraphrased as "The most important office is that of the private citizen" or "The most important political office is that of the private citizen", and sometimes misattributed to his dissenting opinion in Olmstead v. United States.
Extra-judicial writings

2016, Interview with Bill Kristol (2016)

Protectionism: the -ism which teaches that waste makes wealth, 1888, paragraph 155 http://oll.libertyfund.org/titles/sumner-protectionism-the-ism-which-teaches-that-waste-makes-wealth.

1790s, Farewell Address (1796)
Source: George Washington's Farewell Address
Context: Hence, likewise, they will avoid the necessity of those overgrown military establishments, which, under any form of government, are inauspicious to liberty, and which are to be regarded as particularly hostile to Republican Liberty.
Context: While, then, every part of our country thus feels an immediate and particular interest in Union, all the parts combined cannot fail to find in the united mass of means and efforts greater strength, greater resource, proportionably greater security from external danger, a less frequent interruption of their peace by foreign nations; and, what is of inestimable value, they must derive from Union an exemption from those broils and wars between themselves, which so frequently afflict neighbouring countries not tied together by the same governments, which their own rivalships alone would be sufficient to produce, but which opposite foreign alliances, attachments, and intrigues would stimulate and embitter. Hence, likewise, they will avoid the necessity of those overgrown military establishments, which, under any form of government, are inauspicious to liberty, and which are to be regarded as particularly hostile to Republican Liberty. In this sense it is, that your Union ought to be considered as a main prop of your liberty, and that the love of the one ought to endear to you the preservation of the other.

Letter to Elbridge Gerry (13 May 1797)
1790s