In the 1740s, a house at 64 St. James was converted to a chocolate house and became, by the Regency era, a noted Tory Chocolate House.  Accounts of high and even foul play by the 1780s battled with reputation for hosting fashionable gentlemen (Timbs); the Cocoa Tree “had a double-edged reputation as a fashionable hell” (Thevoz, 2018).  Lord Byron, for example, was a member of the Cocoa Tree (Timbs, 1906).

Journal of the Society Arts, 1890

And another description:
Memorials of St. James Street, 1922

More about Chocolate Houses: https://www.atlasobscura.com/articles/history-of-gentlemens-clubs

The Surprisingly Manly History of Hot Cocoa

Thevoz, S. (2018).  Club Government: How the Early Victorian World Was Ruled from London Clubs.  I.B. Tauris.

Timbs, J. (1906).  Clubs and Club Life in London.