He married in 1870 Maria Catharine, daughter of the 5th earl de la Warr, and widow of the 2nd marquess of Salisbury.
Still his position was most delicate, especially when the betrothal between the grand-duke Peter and Sophia of AnhaltZerbst (afterwards Catharine II.) was carried through against his will, and Elizabeth of Holstein, the mother of the bride, arrived in the Prussian interests to spy upon him.
Nevertheless he was deprived of the chancellorship and banished to his estate at Goretovo (April 1759), where he remained till the accession of Catharine II., who recalled him to court and created him a field marshal.
In 1768 he accepted the invitation of the empress Catharine II.
In April 1547 he took chambers in the Inner Temple, and began to study law; but finding divinity more congenial, he removed, in the following year, to St Catharine's Hall, Cambridge, where he studied with such assiduity that in little more than a year he was admitted by special grace to the degree of master of arts, and was soon after made fellow of Pembroke Hall, the fellowship being "worth seven pound a year."