James Drummond Buller-Fullerton-Elphinstone (1788-1857), fourth son of the Hon. William Fullerton-Elphinstone, married Diana (died 1821), only daughter of Charles John Clavering; and secondly Anna Maria, only child of Sir Admiral Buller, Bt. M.P. of Trenant Park, Cornwall when he assumed the additional surname of Buller, by whom he had issue. His eldst son William succeeded as 15th Baron Elphinstone.

Buller-Fullerton-Elphinstone entered the Army as Cornet in the 7th Hussars 1810; was promoted to Lieutenant, 1811 and Captain, 1814. He was wounded at the Battle of Waterloo and taken prisoner at Quatre Bras where he met Napoleon. He was placed on half-pay as Lieutenant-Colonel and joined the 4th Battalion of the West India Regiment in 1818