Although stiff and formal with acquaintances, Benjamin Harrison opened up with his family. During his one term as President, he spent as little time as possible in the office, usually working only until noon. He loved to play with his grandchildren, many of whom had moved into the White House with their parents—Russell Benjamin Harrison, age thirty-six in 1890, and Mary Scott McKee, age thirty-two. The children were allowed to keep as many pets on the grounds as they wanted, including a goat whom they named Old Whiskers. One memorable story told of Harrison chasing the goat down Pennsylvania Avenue with his three grandchildren in tow and top hat in hand while waving his cane.

Harrison also tried to escape Washington as often as possible, frequently going on hunting trips in secret. One trip made the national press when he shot a farmer's pig by mistake. When Harrison lost the election to Cleveland in 1892, he told his family that he felt like he had been freed from prison.