On 19 May 1864 he married (Anne) Emma Haldane, youngest daughter of Alfred Hardcastle of Hatcham House, Surrey. She died at the birth of their second son, having borne him 4 children:
Margaret Eliza Emma Herschel (1865–1880) She had a brain tumor early on.
Emma Dorothea Herschel (1867–1954)
Reverend Sir John Charles William Herschel, 3rd Baronet (1869–1950)
Arthur Edward Hardcastle Herschel (1873–1924)
He lived at Warfield in Berkshire and at Littlemore in Oxforshire.  Upon his death the baronetcy passed to his son.
Fingerprints taken by Herschel 1859/60
Herschel is credited with being the first European to note the value of fingerprints for identification. He recognized that fingerprints were unique and permanent. Herschel documented his own fingerprints over his lifetime to prove permanence. He was also credited with being the first person to use fingerprints in a practical manner. As early as 1858, working as a British officer for the Indian Civil Service at Jangipur in the Bengal region of India, he started putting fingerprints on contracts.
He worked from his late teens all the way through until 2 years before death.