Rufford Abbey was rebuilt by Bess of Hardwick and subsequently passed by marriage to the Savile family in the early 17th Century. The last of the direct line was Sir George Savile, Bt. (1726-1784): his sister Barbara was the wife of Richard Lumley, 4th Earl of Scarbrough, and their third son, John, subsequently the 7th Earl, inherited Rufford. This later passed to John Savile, his successor's natural son, who was created Baron Savile in 1888.

The traditional identification as Charles, 8th Earl of Lennox is untenable: his wife Elizabeth Cavendish, daughter of Bess of Hardwick, the builder of Rufford, was the half-sister of Lady Mary Talbot, wife of Sir George Savile, 1st Bt.