Nations, like individuals, tell stories in order to understand what they are, where they come from, and what they want to be. National narratives, like personal ones, are prone to sentimentality, grievance, pride, shame, self-blindness.

There is never just one – they compete and constantly change. The most durable narratives are not the ones that stand up best to fact-checking. They’re the ones that address our deepest needs and desires. Americans know by now that democracy depends on a baseline of shared reality – when facts become fungible, we’re lost.

But just as no one can live a happy and productive life in nonstop self-criticism, nations require more than facts – they need stories that convey a moral identity. The long gaze in the mirror has to end in self-respect or it will swallow us up.

Tracing the evolution of these narratives can tell you something about a nation’s possibilities for change. Through much of the 20th century, the two political parties had clear identities and told distinct stories. The Republicans spoke for those who wanted to get ahead, and the Democrats spoke for those who wanted a fair shake.

Republicans emphasised individual enterprise, and Democrats emphasized social solidarity, eventually including Black people and abandoning the party’s commitment to Jim Crow. But, unlike today, the two parties were arguing over the same recognizable country. This arrangement held until the late ’60s – still within living memory.

The two parties reflected a society that was less free than today, less tolerant, and far less diverse, with fewer choices, but with more economic equality, more shared prosperity, and more political cooperation. Liberal Republicans and conservative Democrats played important roles in their respective parties.

Americans then were more uniform than we are in what they ate (tuna noodle casserole) and what they watched (Bullitt). Even their bodies looked more alike. They were more restrained than we are, more repressed – though restraint and repression were coming undone by 1968.

Since then, the two parties have just about traded places. By the turn of the millennium, the Democrats were becoming the home of affluent professionals, while the Republicans were starting to sound like populist insurgents. We have to understand this exchange in order to grasp how we got to where we are.

The 1970s ended post-war, bipartisan, middle-class America, and with it the two relatively stable narratives of getting ahead and the fair shake. In their place, four rival narratives have emerged, four accounts of America’s moral identity. They have roots in history, but they are shaped by new ways of thinking and living.

They reflect schisms on both sides of the divide that has made us two countries, extending and deepening the lines of fracture. Over the past four decades, the four narratives have taken turns exercising influence. They overlap, morph into one another, attract and repel one another. None can be understood apart from the others, because all four emerge from the same whole.

Call the first narrative “Free America.” In the past half century it’s been the most politically powerful of the four. Free America draws on libertarian ideas, which it installs in the high-powered engine of consumer capitalism. The freedom it champions is very different from Alexis de Tocqueville’s art of self-government. It’s personal freedom, without other people – the negative liberty of “Don’t tread on me.”

The conservative movement began to dominate the Republican Party in the 1970s, and then much of the country after 1980 with the presidency of Ronald Reagan. As the historian George H. Nash observed, it uneasily wove together several strands of thought.

One was traditionalist, a reaction against the utopian plans and moral chaos of modern secular civilization. The traditionalists were sin-fearing Protestants, orthodox Catholics, southern agrarians, would-be aristocrats, alienated individualists – dissidents in post-war America.

They were appalled by the complacent vulgarity of the semi-educated masses. Their hero was Edmund Burke, the avatar of conservative restraint, and their enemy was John Dewey, the philosopher of American democracy.

The traditionalists’ elitism put them at odds with the main currents of American life – the one passage of American history that most appealed to them was the quasi-feudal Old South – m but their writings inspired the next generation of conservatives, including William F. Buckley Jr., who introduced the first issue of National Review, in 1955, with the famous command to “Stand athwart history, yelling Stop.”

Adjacent to the traditionalists were the anti-Communists. Many of them were former Marxists, such as Whittaker Chambers and James Burnham, who carried their apocalyptic baggage with them when they moved from left to right. Politics for them was nothing less than the titanic struggle between good and evil, God and man.

The main target of their energy was the ameliorative creed of Eleanor Roosevelt and Arthur Schlesinger Jr., good old liberalism, which they believed to be a paler communism – “the ideology of Western suicide,” Burnham called it.

The anti-Communists, like the traditionalists, were sceptics of democracy – its softness would doom it to destruction when World War III broke out. If these hectoring pessimists were the sum of modern conservatism, the movement would have died of joylessness by 1960.

The libertarians were different. They slipped more easily into the American stream. In their insistence on freedom, they could claim to be descendants of Locke, Jefferson and the classical liberal tradition. Some of them interpreted the Constitution as a libertarian document for individual and states’ rights under a limited federal government, not as a framework for the strengthened nation that the authors of The Federalist Papers thought they were creating.

Oddly, the most influential libertarians were Europeans, especially the Austrian economist Friedrich Hayek, whose polemic against collectivism, The Road to Serfdom, was a publishing sensation in America in 1944, during the most dramatic mobilisation of economic resources by state power in history.

What distinguished libertarians from conventional, pro-business Republicans was their pure and uncompromising idea. What was it? Hayek: “Planning leads to dictatorship.” The purpose of government is to secure individual rights, and little else. One sip of social welfare and free government dies.

A 1937 Supreme Court decision upholding parts of the New Deal was the beginning of America’s decline and fall. Libertarians were in rebellion against the mid-century mixed-economy consensus. In spirit they were more radical than conservative. No compromise with Social Security administrators and central bankers! Death to Keynesian fiscal policy!

Despite or because of the purity of their idea, libertarians made common cause with segregationists, and racism informed their political movement from the beginning. Their first hero, Senator Barry Goldwater, ran for president in 1964 as an insurgent against his own party’s establishment while opposing the civil-rights bill on states’ rights grounds.

• Tell report/ Adapted from George Packer’s new book, Last Best Hope: America in Crisis and Renewal