Grose’s Classical Dictionary of the Vulgar Tongue, 1823
The term quiz didn’t start appearing in cant dictionaries until the 19th century. I found the usage of quiz to mean an odd fellow in a Maria Edgeworth book from 1806 (Moral tales for Young People).
Chambers’s Etymological Dictionary of the English Language, 1904
Most etymological references cite this incident in Dublin as the origin: “According to B.H.Smart, in his 1836 Walker Remodelled: quiz, something to puzzle; one whom an observer cannot make out, an odd fellow; also to examine narrowly with an air of mockery. All of these words, which occur only in vulgar or colloquial use, and which Webster traces to learned roots, originated in a joke: Daly, the manager of a Dublin play-house, wagered that a word of no meaning should be the common talk and puzzle of the city in twenty-four hours; in the course of that time the letters Q,u,i,zwere chalked or pasted on all the walls of Dublin with an effect that won the wager.
Of obscure origin. Perhaps from in-quis-ition. Smart’s anecdote is omitted in his 1840 edition.” (http://www.wischik.com/damon/Texts/popetym.html)
Basically, a quiz meant a weirdo.