If you have a taste for prophecy, try this prescient warning by Samuel Butler in his essay ‘Darwin among the machines’ (1863):

“There are few things of which the present generation is more justly proud than of the wonderful improvements which are daily taking place in all sorts of mechanical appliances…Day by day, however, the machines are gaining ground upon us; day by day we are becoming more subservient to them; more men are daily bound down to them as slaves to tend them, more men are daily devoting the energies of their whole lives to the development of mechanical life. The upshot is simply a question of time, but that the time will come when the machines will hold the real supremacy over the world and its inhabitants is what no person of a truly philosophic mind can for a moment question.”

More than 120 years before James Cameron imagined a world dominated by Skynet and its killing machines in The Terminator, Butler had foreseen the awesome potential of technology to make mankind redundant. Today, that fear has migrated from science-fiction movies to mainstream political discourse – and with good reason.

As much as I revel in the digital revolution, I have long believed that the greatest threat to social stability is not immigration but automation. Though the wave of populist nationalism and nativism sweeping the world is deeply alarming, it is by no means destined to triumph. The opposing forces – globalisation, well-established pluralism, commercial interdependence, unprecedented population mobility – are formidable and undiminished. The fact that they are now under political attack (in different ways, in different countries) is no reason to assume that they will be defeated.

In contrast, no clear answer has yet emerged to the challenge of automation. What roles will be left for the majority of human beings in a world of driverless cars, unstaffed supermarkets, robotic baristas, computerised medical diagnosis, legal software, accountancy apps, and artificial intelligence of a sophistication that could only have been dreamt of a few years ago?

In his classic guide to the twenty-first century world, A Whole New Mind (2005), the business guru Daniel Pink predicted that automation would release humankind from drudgery, heralding a new era in which the right hemisphere of the brain – responsible for creativity, design and empathy – would assume exponentially greater significance. Pink’s thesis is correct, in as far as it goes. The nature of labour will indeed change – is already changing – in ways that favour the imagination rather than manual and technocratic skills.

All the same, no society, however enlightened, can employ its entire workforce as writers, designers, actors, therapists, painters, poets and sculptors. The liberation of humanity from most forms of work presents opportunities, but also a crisis of social trajectory. What will become of us all when most of what we do now can be done at marginal cost by machines?

In The rise of the robots, Martin Ford describes a world in which education will remain a force for decency but one incapable of keeping up with the galloping pace of automation. The state, he concludes, will be compelled to pay all citizens a basic income to compensate for this transformation of working culture. Glasgow city council has announced a pilot scheme offering such a subsidy, and Finland has launched its own experiment, paying 2,000 unemployed people a monthly income of around £500 (conditional upon them finding a job – which rather defeats its relevance in the age of automation). So far, these proposals are designed to simplify existing benefit systems and incentivise work. But what if there is no work to be done?

Yuval Noah Harari offers a bleak answer in his recent book, Homo Deus, predicting the comprehensive replacement of professional as well as semi-skilled labour by “highly intelligent algorithms”. In this dystopian vision, he foresees humanity splitting into “an algorithmic upper class owning most of our planet” and “a new massive class: people devoid of any economic, political or even artistic value”.

As an optimist, I prefer the analysis of Thomas H. Davenport and Julia Kirby in Only humans need apply, which champions an “augmentation agenda”. The authors urge employers and governments to acknowledge the scale of the challenge and to focus as much upon preparing humans to work with machines as upon the mechanization of existing labour functions. Their argument – persuasive in its appeal to human nature – is that, in a great many settings, consumers will still want to interact with people as well as machines.

So, for example, financial advisers will spend more time engaging in behavioural analysis of their clients, working face-to-face, while algorithms look for the best deals available. Marketers will be less preoccupied by routine data-crunching and able to concentrate on “higher-level processes”. Software may be able to diagnose disease and propose treatment. But patients, especially those in need of acute care, would still want to deal with doctors and nurses. Human beings are genetically programmed to be social animals, acting tribally rather than in isolation. There are limits to the degree of automation we would accept – limits that have less to do with grumpy Luddism than our deepest, most primal instincts to collaborate and communicate with one another.

There is no glib answer to this extraordinary challenge, no ready solution to be grabbed off the shelf. But the first and most important step is for businesses, politicians and policymakers to acknowledge the sheer scale and urgency of the task that stands before them. To do otherwise would be the most dangerous sort of delusion.

Matthew d’Ancona is chair of Bright Blue and a columnist for the Guardian and London Evening Standard. This is an article from Bright Blue’s magazine The robotic revolution published before the general election.