Even though a comfortable private life awaited James Buchanan at his Pennsylvania home, his last years were difficult. Rightly or wrongly, considerable blame for the Civil War fell upon him. His portrait had to be removed from the Capitol to keep vandals from damaging it, and posters captioned "Judas" depicted him with his neck in a hangman's noose.

A wave of second-guessing condemned Buchanan's actions with regard to Fort Sumter. The Republican press attacked him while absolving the Republican Party and Lincoln from all responsibility for the conflict. Although Buchanan vocally supported the Union cause, many branded him an appeaser of the South and a lover of slavery. Finally, the former President decided to write a book telling his side of the story. It appeared in 1866, one year after the war ended, but the public largely ignored the book, which blames the Civil War on the Republican Party and the abolitionists. After publication, Buchanan all but vanished from public life. He retreated inside the walls of his home and saw only close friends. He died there in June 1868.