Sir Richard Cholmley acquired the Whitby estate, with Abbey House by the ruins of the medieval Abbey. His son, Sir Hugh Cholmley, Ist Bt. (1600 - 1657), married Elizabeth Twisden, of the notable Kent royalist family, and defended Scarborough for King Charles I. His younger son and eventual heir, Sir Hugh Cholmley, 4th Bt. (1632 - 1688), inherited in 1665 and married Lady Anne Compton, daughter of Spencer Compton, 2nd Earl of Northampton. Like his father, Cholmley served his monarch. Tangier had passed to King Charles II as part of the dowry of Queen Catherine of Braganza and its development became a significant priority of the crown. Cholmley, who had previously undertaken the building of a mole at Whitby, was responsible for the construction of the mole in the harbour at Tangier between March 1663 and August 1676. The new mole was crucial to the trading prospects of the colony but was to be comprehensively demolished when Tangier was abandoned in 1683: a picture of the demolition, painted for the Ist Baron Legge, is on loan to the National Maritime Museum.
The first specific reference to the pictures in the Howsham papers is that of the inventory of 1852, prepared by Henrietta Strickland, sister of Sir George Strickland. In 1875 a fuller account was prepared:
'These Spanish pictures have been in the possession of the Cholmley family for several generations. As far as is known, there are no documents relating to them but they are supposed to have been brought to Whitby Abbey by Sir Hugh Cholmley on his return from his Government of Tangier in the reign of Charles II together with a collection of pictures purchased by him - that they had been painted in Mexico for the King of Spain and were on their passage to Europe when the ship they were on board of was captured.'
While there seems to be no reason to doubt this traditional account, Oswald pointed out in 1935 that the pictures might alternatively have been acquired in 1669 when, on his third journey to Tangier with the rank of Surveyor-General, Cholmley travelled overland through France and Spain. The pictures are not mentioned in Cholmley's journal, but his ownership of them is given tangential support by a reference in Cholmley's journal:
Prologue to the Conquest of Mexico, acted at Whitby, on Shrove Monday and Tuesday, the 10th and 11th of February, 1683 (o.s., i.e. 1684)
The 'play' may have been inspired by literary sources - de Castillo's account of the Conquest was published in 1632 while that of Antonio de Solis appeared in 1684 - but it may also have been suggested by this series of pictures. In view of Cholmley's visits to Tangier, it is not improbable that he had some grasp of Spanish and an interest in Spain may have been fortified by childhood recollections of an incident in June 1637 when a 'pickroon', or vessel, in the service of the King of Spain was driven into Whitby harbour by two Dutch men-of-war and his father intervened on its behalf.
The first English account of the Conquest of Mexico was that of the great Scottish historian William Robertson, whose History of America was issued in 1777. There is no evidence that Robertson knew of the pictures but in the nineteenth century these were brought to the attention of the great American historian, W.H. Prescott, whose History of the Conquest of Mexico of 1843 promptly achieved the status of a classic: his visit to Howsham to see the pictures is recorded in a note of 1875.