“England is the most class-ridden country under the sun. It is a land of snobbery and privilege, ruled largely by the old and silly.”

Letter to Cobden (30 December 1853), quoted in G. M. Trevelyan, The Life of John Bright (London: Constable, 1913), p. 230.
1850s

Political Science for Civil Services Main Examination (2010)

The Debs Decision https://ia902703.us.archive.org/33/items/TheDebsDecision/Debs.htm (1919)
Context: Compared with the necessity of protecting ruling class privileges and prerogatives, the right of a man to express his mind goes for nothing. That is the lesson of history and that is what we are witnessing today. Men who have stirred up the people; men who have raised their voices in protest; men who thought straight; men who have loved their fellow men too much; men who have had conviction and courage and purpose; men who were willing to stick by their ideals—such men have suffered in every age.

Source: Letter to a working men's club (1867), quoted in William Flavelle Monypenny and George Earle Buckle, The Life of Benjamin Disraeli, Earl of Beaconsfield. Volume II. 1860–1881 (London: John Murray, 1929), p. 297.

Similar terminology is used about males as a defamation of character, or more broadly only about pressed males males: stud, wold, cat, stag, jack - and then it is used much more rarely, and often with a specifically sexual connotation.

Chapter Four
The Dialectic of Sex (1970)

Act IV.
Dantons Tod (Danton's Death) (1835)

A Tree Song,
Puck of Pook's Hill 1906

Source: The Natural and Artificial Right of Property Contrasted (1832), p. 51

Reported in Hoyt's New Cyclopedia Of Practical Quotations (1922), p. 86. The quotation has been attributed to many others, such as Lord Chief Justice Campbell, Lord Chesterfield, Sir William Harcourt, Lord Pembroke, Lord Westbury, and to an anonymous judge, and said to have been spoken in court to Garter King at Arms, Rouge Dragon Pursuivant, or some other high-ranking herald, who had confused a "bend" with a "bar" or had demanded fees to which he was not entitled. George Bernard Shaw quotes it in Pygmalion (1912) in the form, "The silly people dont [sic] know their own silly business."

Maule cannot be the original source of the quotation, as it is quoted nearly twenty years before his birth in Charles Jenner's The Placid Man: Or, The Memoirs of Sir Charles Beville (1770): "Sir Harry Clayton ... was perhaps far better qualified to have written a Peerage of England than Garter King at Arms, or Rouge Dragon, or any of those parti-coloured officers of the court of honor, who, as a great man complained on a late solemnity, are but too often so silly as not to know their own silly business." "Old Lord Pembroke" (Henry Herbert, 9th Earl of Pembroke) is said by Horace Walpole (in a letter of May 28, 1774 to the Rev. William Cole) to have directed the quip, "Thou silly fellow! Thou dost not know thy own silly business," at John Anstis, Garter King at Arms (though in his 1833 edition of Walpole's letters to Sir Horace Mann, George Agar-Ellis, 1st Baron Dover, attributes the saying to Lord Chesterfield in a footnote, in the form "You foolish man, you do not understand your own foolish business"). Edmund Burke also quotes it ("'Silly man, that dost not know thy own silly trade!' was once well said: but the trade here is not silly.") in a "Speech in the Impeachment of Warren Hastings, Esq." on May 7, 1789 (when Maule was just over a year old). Chesterfield or Pembroke fit best in point of time.
Attributed