As President, William Howard Taft left most family and domestic matters to Nellie Taft. Two of his three children, Robert Alphonso and Helen Herron, were college students during the White House years. Robert graduated from Yale in 1910, and then went on to Harvard Law School, where he graduated first in his class in 1913. He would go on to become one of the most distinguished and powerful senators of the twentieth century, earning the nickname "Mr. Republican." Helen earned her doctorate in history from Yale in 1917. The youngest sibling, Charles Phelps, aged eleven in 1908, spent his adolescence in the White House.

Taft loved to play sports. He was the first President to take up golf. Some western voters—those who equated Taft's Unitarianism with atheism—thought his golf playing indecent if not immoral. His love for the sport caused a golf boom in the nation, doubling the number of players on public courses. Taft's affection for golf also caused political problems during his presidency, when critics thought he would do well to spend less time on the links and more time at work in the White House.

What most people associated with Taft, however, was his enormous size, and the image of his 300 plus pounds of presidential flesh offended some people and amused many others. His size made him the subject of countless jokes such as “Taft was the most polite man in Washington. One day he gave up his seat on a streetcar to three women.” Within the capital's social circle, Taft frequently embarrassed his family and associates by falling asleep at concerts, during presidential briefing sessions, and while presiding over his cabinet. At ease with his uncontrolled appetite and his need for sleep after eating or after exerting himself, Taft simply refused to be embarrassed by his weight or his behavior. He accepted his size and so did most of the American public in time.