Lady Caroline Lennox was born Caroline Paget, the eldest daughter of Henry William Paget, the 1st Marquis of Anglesey (1768-1854) and Lady Caroline Elizabeth Villiers (1774-1835). Paget, who was made Field Marshal in 1846 in honour of his military achievement, served two sessions as Lord Lieutenant of Ireland, for the duration of a year in 1828, and from 1830-33. Caroline married Charles Gordon-Lennox, 5th Duke of Richmond (1791-1860) on 10 April 1817. They had ten children, their eldest being Charles Henry Gordon-Lennox, who became the 6th Duke. Their second daughter, Lady Augusta Caroline Gordon Lennox, married his Serene Highness Prince William Augustus Edward of Saxe-Weimar.