How one of the worst US presidents in history alienated Congress to the point that he was impeached

Then, the vice president-elect headed off to his inauguration.

The weather outside was terrible, so the ceremony took place in the crammed Senate chamber. Things went downhill after Johnson was sworn in.

"The inauguration went off very well except that the Vice President Elect was too drunk to perform his duties & disgraced himself & the Senate by making a drunken, foolish speech," Michigan Republican Senator Zachariah Chandler recalled in a letter to his wife, according to the official US Senate website. "I was never so mortified in my life, had I been able to find a hole I would have dropped through it out of sight."

About a month later, Abraham Lincoln was cut down by an assassin's bullet and the "drunken, foolish" Johnson became president.

While Lincoln is remembered as one of history's greatest chief executives, Johnson consistently ranks in the bottom five in most contemporary polls. So, what went wrong with the so-called "Tennessee Tailor"?

To find out, Business Insider spoke with Annette Gordon-Reed, a professor of history and law at Harvard University and the author of "Andrew Johnson."

According to Gordon-Reed, Johnson's tarnished reputation is well-deserved, thanks to his deeply-held prejudices and general failings as a leader.

"I think he's one of the worst," she told Business Insider. "James Buchanan is usually the person who's ranked as the worst, but Buchanan was dealing with a lot of really tough problems. Johnson had a way. There was a path forward."

Humble beginnings

Johnson was born in a log cabin to a poor family in North Carolina. His humble beginnings imbued him with a lifelong sympathy for poor white Southerners, as well as bitterness toward wealthy Southern plantation owners, according to Gordon-Reed.

After working for a time as a tailor's apprentice at his mother's behest, he ran away and eventually settled in Tennessee. He didn't learn to write until the age of 18, when his new, 16-year-old wife Eliza McCardle taught him.

"He decides to remain in the Union, which is a very courageous thing to do," Gordon-Reed says. "He faced death threats and attacks and so forth. He wanted to stay in the Union because he believed in the Union."

His decision attracted the attention of President Lincoln, who appointed him military governor of the largely-recaptured Tennessee in 1862. Johnson's new position, along with his bold rhetoric against the rebel states, led him to become a reviled figure in the Confederacy.

It also proved a boon to his political career. When it came time for Lincoln's re-election in 1864, he cast aside his original vice president, Hannibal Hamlin, in favor of Johnson. The move was a bid to show his commitment to reconciliation between the North and South in the waning days of the war.

After the war

After Lincoln's assassination, Gordon-Reed says that many Southerners were initially frightened of Johnson, especially after he began calling prominent Confederates before him.

"At first, it made people think that he was going to be really hard on them," Gordon-Reed says. "They were frightened of him. He had been using all these harsh words. It must have been a great feeling for him to have them have to come up before him and ask for a pardon."

The harsh measures that Southerners feared never came to be. Johnson would eventually issue general clemency for all former Confederates in 1868.

Gordon-Reed says that Johnson's allegiances shifted after the war. His resentment toward the planter aristocrats dissipated as he began to realize that the Republican-controlled Congress hoped to reshape the South and give newly freed slaves political rights. Johnson's negative feelings towards African Americans went far beyond widespread 19th century prejudices, according to Gordon-Reed.

As a result, Johnson's opposition to Congress' Reconstruction agenda didn't just impact federal law.

She says the rift between the White House and Congress was also worsened by Johnson's abrasive leadership style.

"The stubbornness that helped him in some areas made it difficult for him to be an effective leader," she says. "People don't respond to threats or recalcitrance. He couldn't work with people. He thought he was right all the time and just wouldn't budge. His success would also probably predispose him to thinking that he's right. Think of it, you look back and you say, 'Look, I started out life as a tailor's apprentice who ran away. There were runaway ads for me. I didn't learn how to read until I was in my late teens. My wife taught me how to write. I am where I am because of my skills, because I know how to maneuver.'"

Them and us

As the war over Reconstruction raged between Johnson and Congress, the chief executive decided to embark on a speaking campaign known as the "Swing Around the Circle" in 1866, during which he would argue in favor of his lenient policies toward the South. The tour ended up being an excuse for the president to take his paranoid rhetoric on the road.

In "The Swing Around the Circle: Andrew Johnson and the Train Ride that Destroyed a Presidency," author Garry Boulard says that contemporary reporters recorded Johnson as arguing with speech attendees, suggesting that leading Republicans ought to be hanged as traitors, and declaring, "I don't care about my dignity" when questioned about his un-presidential conduct.

The "Swing Around the Circle" turned out to be a disaster, but Johnson's combative mentality did not change.

In her biography of Johnson, Gordon-Reed also records an instance where the president declared to a delegation of visiting black leaders, including the abolitionist and author Frederick Douglass, that he was "... certain that they shared the values of their masters when it came to poor whites, and that alleged universally shared disdain bore the evidence of cooperative effort to keep his 'people' down."

"His leadership style was 'You're for me or you're against me,'" Gordon-Reed told Business Insider. "It's a very Manichean view of the world. Them and us."

His impeachment came about in 1868, when the House of Representatives accused Johnson of violating a Congressional act by replacing Secretary of War Edwin Stanton, a Republican and a Lincoln appointee.

During his trial, the president blamed his troubles on "a mendacious press" and a "subsidized gang of hirelings," according to "The Trial of Andrew Johnson."

The House voted for impeachment, but Johnson was later acquitted by a single vote during the Senate trial.

He finished the rest of his presidency, but declined to run for a second term. According to the Joint Congressional Committee on Inauguration Ceremonies, Johnson ignored custom and did not attend his successor Ulysses S. Grant's inauguration.

Unconquerable prejudice

Despite his impeachment and one-term presidency, Johnson concluded his tenure having gained popularity with white Southerners, a group that had once despised and feared him.

In fact, Johnson's legacy also wasn't consistently considered dismal until somewhat recently. US President Harry Truman even wrote a piece defending him in a 1959 edition of the The North Carolina Historical Review, titled "The Most Mistreated of Presidents." UVA's Miller Center says that such positive accounts tend to praise Johnson's rags-to-riches background and stubbornness, ignoring his flagrant racism and inability to work with Congress.

Gordon-Reed says it's too simple to simply look at Johnson as an inept leader.

She says that in an era where the South would have likely been forced to accept any terms for reunification, the president's actions had a devastating impact.

"This may be bringing present day sensibilities into it, to suggest that of course you want all the citizens of the country to participate," she says. "That wasn't his view. He thought he was doing good by the country by keeping blacks out of power."

That being said, Gordon-Reed says that it's important to distinguish between "bad" and "important" presidents. Johnson was both a bad and important president, she argues, because of the impact his actions had on the country. His obstruction and failures led to violence against freedman, and the continued oppression of African Americans.

Johnson's deeply held racism — "unconquerable prejudice," in the words of contemporary Illinois congressman Elihu Washburne— led to what Gordon-Reed calls a "tragic circumstance."

"His feelings about African Americans were not just the kind of everyday racism that would have existed in the nineteenth century," she says. "He had a real antipathy towards blacks. You can see the tragic irony of someone who is overseeing the United States after the end of the Civil War who has that kind of feeling. There could have been a time, if we had a different president with a different attitude, even just sort of normal prejudices for the time, the country could have gone in a different direction. His prejudices, his hatred of African Americans, colored so many of his policy decisions. That's what his presidency after the war was all about."