“The liberty of the press ought to be left where the Constitution has placed it, without any power in Congress to abridge it; that if they abridge it, they will destroy it; and that whenever that falls, all our liberties must fall with it.”

Resolution offered in the Philadelphia Convention, May 29, 1787. The United States Constitution was enacted without any protection for religion or the press, but with the understanding that a Bill of Rights would shortly be enacted to address these concerns.

Source: Life and Adventures of Peter Porcupine (1796), P. 59.

Source: Atlas Shrugged

Though Lafayette, himself a Roman Catholic, might conceivably have said something translatable as this, the earliest source yet found for this is an anti-catholic pamphlet The Future Conflict : An Address, (1878), by Order of the American Union, p. 20, without any citation of original sources. It has also been quoted as: "If the liberties of the American people are ever destroyed, they will fall by the hands of the clergy."
Disputed

The Rights of the Colonists (1772)
Context: The natural liberty of man, by entering into society, is abridged or restrained, so far only as is necessary for the great end of society, the best good of the whole. In the state of nature every man is, under God, judge and sole judge of his own rights and of the injuries done him. By entering into society he agrees to an arbiter or indifferent judge between him and his neighbors; but he no more renounces his original right than by taking a cause out of the ordinary course of law, and leaving the decision to referees or indifferent arbitrators. In the last case, he must pay the referees for time and trouble. He should also be willing to pay his just quota for the support of government, the law, and the constitution; the end of which is to furnish indifferent and impartial judges in all cases that may happen, whether civil, ecclesiastical, marine, or military.

House of Representatives, Amendments to the Constitution, 17, 20 Aug. 1789 http://press-pubs.uchicago.edu/founders/documents/amendIIs6.html

His judgement in another case on the issue of Fundamental Rights.
Full Court Reference in Memory of The Late Justice M. Hidayatullah

Case of John Lambert and others (1793), 22 How. St. Tr. 1016.

Journal of John Quincy Adams (11 December 1838),
Context: The conflict between the principle of liberty and the fact of slavery is coming gradually to an issue. Slavery has now the power, and falls into convulsions at the approach of freedom. That the fall of slavery is predetermined in the counsels of Omnipotence I cannot doubt; it is a part of the great moral improvement in the condition of man, attested by all the records of history. But the conflict will be terrible, and the progress of improvement perhaps retrograde before its final progress to consummation.