It is becoming increasingly commonplace to observe that we live in an era of disorder and instability. Whether it is the seemingly endless conflicts and political upheavals in the Middle East, the return of geopolitics in Europe, the growing strategic tensions in our own region or the inability of governments the world over to satisfy the heightened expectations of their own people, much of the world looks messy and badly governed.

Some parts of the world have been synonymous with poor governance, corruption and state failure for much of their independent existences. While this may have much to do with their unfortunate recent histories and the impact of colonialism, the net result has been the same: underachievement and developmental failure. The difficulty of dealing with challenges like Ebola is a reminder of how much state capacity matters in responding to crises.

But why is it some parts of the world have governments that are more or less effective and some don’t? This is, perhaps, still the key question of our time. The answers are complex and contested, but more than academic reputations are at stake in trying to address them. For better or worse the individual and collective welfare of humankind continues to be determined by the political structures they consciously or unconsciously create.

Francis Fukuyama’s new book, Political Order and Political Decay: From the Industrial Revolution to the Globalization of Democracy, is the second instalment of his truly magisterial, highly ambitious survey of the evolution of increasingly complex and rule-governed political societies from our tribal beginnings.

The new volume concentrates the relatively recent past and considers why some countries have effective governments and some don’t, and why some are becoming increasingly ineffective despite the best of intentions.

Rather alarmingly, Fukuyama considers the US state to have become a ‘bloated and inefficient monstrosity’, that is vulnerable to capture by special interests and paralysed by an excess of checks and balances that exacerbate the political polarisation that has become such a feature of American politics.

Many of America’s current problems, Fukuyama contends, stem from the inability to modernise its system of government and adjust to new realities, rather than the circumstances that confronted the Founding Fathers. This helps to explain why, for example, Americans find it so difficult to confront gun violence and availability despite the fact that it claims thousands of lives every year.

The good news about the US, though, is that many of its problems are remediable because they do have formidable advantages, especially as far as underutilised and needlessly ineffective state capacity is concerned.

The reason the rest of the world agonises and obsesses about the US is that it matters to us, too. For half a century or more the US has been the institutional centre-piece of an international world order which even its critics would have to concede, could have been a lot worse.

One of the sources of the current disorder is the seeming decline of American influence and power in the world, which is – in part, at least – a consequence of its own mounting domestic problems. The contrast with China is striking and one that Fukuyama highlights in the new book and its predecessor.

The big take-home message is that there is no inevitable correlation between effective governance and democracy. On the contrary, Fukuyama argues China’s contemporary political structures potentially mean that:

In the hands of good leaders, such a system can actually perform better than a democratic system that is subject to rule of law and formal democratic procedures like multiparty elections. It can make large, difficult decisions without being hampered by interest groups, lobbying, litigation, or the need to form cumbersome political coalitions or educate the public as to their own self-interest.

For followers of Fukuyama’s career, this is quite a change of direction from the End of History and the idea that the world’s polities would inevitability become liberal democracies. The argument back then was that democracy is the only thing that would be accepted by educated and affluent populations. But as Keynes famously observed, when the facts change, you have to change your mind.

Farrar, Straus and Giroux

The inescapable reality is that things haven’t turned out as expected. Not only has much of the world failed to replicate the western experience of rapid economic development, but the values associated with western modernisation have often been repudiated, too.

The recent history of East Asia serves as a reminder that even in those parts of the world that have prospered, they have often done so with the aid of powerful, invariably undemocratic states, not through the sorts of neoliberal reforms that have been encouraged by the US and its institutional allies at the World Bank and the IMF. My colleagues in the so-called “Murdoch School” of political economy have done more than most to explain why social conflict and authoritarian rule persist in much of Asia.

The pivotal historical role of the US in influencing the course of international economic and political development is one reason why America’s own problems of domestic governance and order are so crucial for a world that is afflicted with growing disorder. Although Barack Obama has inevitably been the principal target of domestic and international criticism of American actions, Fukuyama’s brilliant analysis of long-term political decline suggests that America’s leadership problems have much more deep-seated roots.

It is possible to imagine that American decline might have been even more rapid, traumatic and violent with someone else in the White House. Whether you agree with Fukuyama’s conclusions and their implications or not, these two volumes are a massive – in every sense of the word – contribution to the literature. His conclusion about what ails the US is especially sobering for lovers of democracy and unlikely to be easily addressed:

The problems of American government arise because there is an imbalance between the strength and competence of the state on the one hand, and the institutions that were originally designed to constrain the state on the other. There is, in short, too much law and too much “democracy” relative to the American state capacity.

Whatever one thinks of American foreign policy and its role in creating the post-war international order it is worth considering what possible alternatives might have looked like. A dysfunctional, introspective and self-obsessed America is not good for the US and not good for the world.

Mark Beeson does not work for, consult to, own shares in or receive funding from any company or organisation that would benefit from this article, and has no relevant affiliations.