“For trust and mistrust, alike ruin men.”

As paraphrased in The Great Quotations‎ (1960) by George Seldes, p. 460; this paraphrase has for some time become the most widely quoted form of Madison's statement.
1780s, The Debates in the Federal Convention (1787)

The Fourteenth Revelation, Chapter 42
Context: This is our Lord’s will, that our prayer and our trust be both alike large. For if we trust not as much as we pray, we do not full worship to our Lord in our prayer, and also we tarry and pain our self. The cause is, as I believe, that we know not truly that our Lord is Ground on whom our prayer springeth; and also that we know not that it is given us by the grace of His love. For if we knew this, it would make us to trust to have, of our Lord’s gift, all that we desire. For I am sure that no man asketh mercy and grace with true meaning, but if mercy and grace be first given to him.

No. 1, volume v, p. 286
Letters On a Regicide Peace (1796)

Man and Wife - Vol. II [Bernhard Tauchnitz, 1870] ( p. 235 https://books.google.com/books?id=Dp-ZFYLTW6QC&pg=PA235)
Also in Wilkie Collins: Man of Mystery and Imagination by Alexander Grinstein [International Universities Press, 2003, 0-823-66681-6] (p. 155)

(zh-TW) 性相近也、習相遠也。子曰、唯上知與下愚不移。 note: The Analects, Chapter I, Other chapters

Source: Ref: en.wikiquote.org - Confucius / Quotes / The Analects / Chapter I / Other chapters

Lincoln Hall Speech (1879)
Context: I know that my race must change. We cannot hold our own with the white men as we are. We only ask an even chance to live as other men live. We ask to be recognized as men. We ask that the same law shall work alike on all men. If an Indian breaks the law, punish him by the law. If a white man breaks the law, punish him also.

Source: He's Just Not That Into You: The No-Excuses Truth to Understanding Guys

On the racial divides in her household and community in “Oral history interview with Carmen Lomas Garza, 1997 Apr. 10-May 27” https://www.aaa.si.edu/collections/interviews/oral-history-interview-carmen-lomas-garza-13540#transcript (Smithsonian Archives of American Art)

Source: Father and Child Reunion (2001), p. 139.