James Douglas was born in 1703, the eldest son of George, 13th Earl of Morton, and his second wife Frances, daughter of William Addesley of Halsgow, Kent. He was a Representative Peer for Scotland between 1739 and 1768. He married firstly Agatha (d. 12 December 1748), daughter and heiress of James Halyburton of Pitcur; they had two surviving children. He married secondly on 31 July 1755 Bridget (d. 2 March 1805), daughter of Sir John Heathcote, Bt., of Normanton; they had several children. He died on 12 October, 1768, and was succeeded by his eldest son, Sholto, as 15th Earl.

The Earl was 14th in the English Peerage, and 13th in the Scottish.

The original painting, commissioned by Lord Morton in 1740 for his new house at Dalmaboy, is now in the Scottish National Portrait Gallery, Edinburgh (see J. Holloway, Patrons and Painters: Art in Scotland (1650-1760)