Joseph, 1st Baron Milton (1718-1798) of Milton Abbey, Dorset, and Shronell, co. Tipperary, was the first son of Joseph Damer, M.P. of Winterbourne Came, Dorset, and his wife Mary, daughter of John Churchill of Henbury, Dorset. He was educated at Trinity, Dublin and married in 1742 Lady Caroline, only surviving daughter of Lionel, 1st Duke of Dorset. They had three sons and one daughter; evidently his wife's death in 1775 caused him to retreat to 'a gloomy and sequestered splendour' at his estates at Milton Abbey where he could enjoy an income of between 15,000 and 30,000 a year, which he appears to have spent mainly on land acquisition. Although M.P. for Weymouth and Melcombe Regis, Bramber, and Dorchester between 1741 and 1762 he is not on record as taking much interest until late in life when, as Walpole comments: 'Lord Milton... (who) had appeared nowhere since his wife's death near two years before.... had been in opposition... till the breaking out of the American war, when he had taken up violent animosity to the Americans...'.