The private art collection is a distinguishing feature of the wealthy (for a modern update to some of the Duke of Bridgewater’s collection, see this fascinating article from 2008: http://www.independent.co.uk/arts-entertainment/art/news/sometimes-one-can-have-just-too-many-titians-913933.html)
At the beginning of the 19th century, the practice of opening collections to public inspection was becoming increasingly common (http://www.tate.org.uk/research/publications/tate-papers/24/william-hazlitts-account-of-mr-angersteins-collection-of-pictures). These collections and viewings would, literally, become the foundation of places like the British National Gallery and a way for the burgeoning middle class to contribute to the cultural conversation in a significant way(Angerstein’s collection was, after his death in 1823, purchased for the National Gallery) (Civilizing Rituals: Inside Public Art Museums, 2005).
This phenomena of art museums evolving from private and royal collections happened in the 18th and 19th centuries across Europe as part of a general expression of Enlightenment values and the increasing access of various classes to art and culture (The First Modern Museums of Art). The benefit to public museums was that visitor did not have to apply, or present letters of introduction, to the private collector to view the artwork which naturally would’ve diminished access (The First Modern Museums of Art, 2012).
Overall, it seems likely that a fashionable house might wish to be request permission to view a collection, if for nothing more than to give the appearance of culture and refinement. I am frankly amazed we don’t see more Reg Rom characters visiting these types of houses…there is something terribly romantic, even a bit thrilling, at peeping at the art of another.
Here is a catalogue of Marquis of Lansdowne’s collection: http://lcweb2.loc.gov/service/gdc/scd0001/2007/20078204002ca/20078204002ca.pdf and another 1897 catalogue
A national gallery look at the Angerstein collection: https://www.nationalgallery.org.uk/paintings/history/collectors-and-benefactors/john-julius-angerstein