Peter Delmé of Erle Stoke, Wiltshire, was the second son of Peter Delmé (d. 1770), M.P., and his second wife Christian Pain, and a grandson of Sir Peter Delmé, Knt., director of the Bank of England, and Lord Mayor of London in 1723. He married, 16 February 1769, Lady Elizabeth Howard, 5th daughter of Henry Howard, Earl of Carlisle, the beauty of the Court of Queen Charlotte, and the subject of one of Sir Joshua Reynolds' finest portraits (National Gallery of Art, Washington).