Millard Fillmore did not smoke, drink, or gamble in an era when almost every American man partook in at least one of these vices. With the recent death of President Taylor and a wife who was often unwell, social occasions at the Fillmore White House were few and far between. The family preferred to take quiet refuge upstairs, where their daughter, Mary, would play the piano while others read or worked.

Washington in the mid-nineteenth century was a far from hospitable place. Nearby swamps teemed with mosquitoes bearing malaria and other diseases. Sanitation was poor, and in the summer, the city became nearly unbearable with humidity. The White House itself was in poor condition and uncomfortable to live in. The mansion had fallen into grave disrepair during the 1840s—there were annoyed accounts of springs in battered furniture stabbing guests who tried to sit down in the White House. Not surprisingly, the Fillmores treasured escaping to the countryside, and they retreated there as often as possible.