James Daniell married Lucy (d.1846), the second daughter of Peregrine Butler of Dungarvan, Ireland. They had issue. Their daughter, Frances married Lindsey Merrick Peter Burrell. Their son, Peter Robert Burrell succeeded as 5th Bt. and 4th Lord Gwydyr in 1870.
Eliza Martha Pasley (1776-1854), daughter of James Hodges, Esq. married the son of Lucy and James Daniell also called James in 1795.