Francis Saxham Elwes Drury was the son of the Rev. Henry Drury, Archdeacon of Wiltshire, and the great-grandson of the Rev. Joseph Drury, D.D. (1750-1834), headmaster of Harrow School. Lord Byron wrote of him in his works relating to his school days. Drury's grandfather, Henry Thomas Joseph Drury (1778-1841), was assistant headmaster at the school for 41 years. Francis Drury commissioned a detailed history of the family in 1937, (A. Campling, A History of the Drury Family, London, 1937).