Millard Fillmore and his family welcomed the escape from Washington after Pierce's election; they had never liked the city. Abigail Fillmore had been so unwell during most of the administration that her daughter, Mary, had been pressed into hostess duty for White House functions. The city, however, took a final, terrible swipe at the family. Abigail Fillmore, compelled to sit outside for hours on Pierce's cold, wet inauguration day, caught pneumonia and died less than a month later. Not long afterward, Fillmore's daughter, Mary, only twenty-two-years-old, died of cholera. The former President was devastated by the twin calamities, and he searched for something to take his mind off them.

Unsurprisingly, Fillmore found solace in politics. Countless former Whigs were now without a party, and some had organized a new one. It had the strange name of the Know-Nothing Party, and its aims were less than savory. Targeting native-born Americans made uneasy by the hordes of immigrants now flocking to the United States, the Know-Nothings advocated immigration restrictions and a waiting period for new citizens to vote. Irish Catholic immigrants, perceived as a threat to the existing labor force, were a prime target. Fillmore refused to be part of the anti-immigrant message dispensed by the party, but the Know-Nothings made him their presidential candidate in 1856, as did the remnants of the Whig Party. The former President received only 21 percent of the vote, but it prevented another candidate he deeply opposed—John C. Frémont of the new Republican Party—from winning against Democrat James Buchanan.

Fillmore would make no more attempts at political office. He retired in Buffalo and married Caroline McIntosh, a wealthy Albany widow, in early 1858. Fillmore was thereafter active in many causes and charities. When the Civil War erupted three years later, Fillmore became a staunch Unionist, helping to organize enlistment and war-financing drives. The last Whig President died of a stroke in March 1874, firmly established as Buffalo's leading citizen. Some, however, never forgot—or forgave—Millard Fillmore's support of the Fugitive Slave Law. After the assassination of Abraham Lincoln in 1865, a mob descended on Fillmore's opulent home and smudged black paint on the building.