Because Garfield never knew his father, he always held a special place in his heart for his mother, to whom he credited his success. Eliza Ballou Garfield, the first mother of a President to attend her son's inauguration, survived her son's death by seven years. She lived at the White House with her son's family during Garfield's brief term of office. She was a frail woman who dressed only in black and wore a lace handkerchief on her head to hide her thinning white hair. Garfield, a strong man, standing over six feet in height, personally carried his mother up and down the White House stairs.

When Garfield was assassinated in 1881, he had four sons and one daughter who ranged in age from nine (Abram) to nineteen (Harry). Two other children had died in infancy. His daughter Mary "Mollie", age fourteen in 1881, met her husband, Joseph Stanley-Brown, who was Garfield's presidential secretary, while living in the White House. The other two boys, James, age sixteen, and Irvin, age eleven, kept the President and his wife busy and happy with their youthful escapades.

The children all grew up to be successful and productive citizens. In 1908, Harry, a professor of politics at Princeton, became the president of Williams College. During World War I, he also served as Woodrow Wilson's fuel administrator. In 1907, James became secretary of the Interior under President Theodore Roosevelt. Irvin became a successful corporate lawyer in Boston, and Abram, a graduate of MIT, worked as an architect in Cleveland. Mary, whose husband became a prominent investment banker, was active in civic affairs in New York and Pasadena, California.