While a delegate to the Continental Congress in New York, James Monroe met Elizabeth Kortright in 1785. They were married the following year and eventually had three children—Eliza Kortright Monroe, James Spence Monroe (who died in infancy), and Maria Hester Monroe. Despite Monroe’s many trips abroad, he spent precious little time away from his family, since they usually accompanied him on his travels.

The Monroes were devoted parents and gave much attention to their daughters. James believed education was important for girls as well as boys, and his daughters were well-educated for the era. Even after the marriages of their daughters, James and Elizabeth remained in close contact with them and were fond of both their sons-in-law. Indeed, for a time, Eliza and her husband lived in the White House with her parents, and she served as White House hostess when her mother was unwell. After Elizabeth's death in 1830, James and Eliza moved to New York City to live with Maria and her family.