Secrets and spies: can espionage ever be justified?

States need information on what their enemies are thinking. But — since “hostile” and “allied” are fluid categories — this applies to friends too. Hence spying has been inseparable from civilisations.

It even has a biblical pedigree. As Christopher Andrew writes, the “first major figure in world literature to emphasise the importance of good intelligence was God” — ordering Moses to send undercover agents ahead to “spy out the land of Canaan”, a place divinely promised, yet unknown. The spooks, it seems, have always been among us.

The ancient origins of intelligence networks are rarely discussed. Yet, while their modern manifestations share the same fundamental rationale for existence as those biblical envoys, they are — in the permanence of their establishments and the size of their budgets — far from the scrappy, if talented, groups who made up the “secret services” before professionalisation in the 20th century. The greater destructiveness of wars and the added Armageddon quality of nuclear weaponry; the increased, lethal power of terrorism, especially jihadism — whose leading figures, such as the late Osama bin Laden, sought weapons of mass destruction; and the appearance of new challenges, such as cyber terrorism — all these bolster the case for the intelligence services, and the resources allotted to them.

Along the way, the secret services have become more valued, more visible — and more controversial. Public concern over the ability of spooks to monitor communications was roused by the leaks from the US National Security Agency, and deepened by the revelations of torture practised on terrorist suspects by the CIA, an agency that had claimed a high ethical ground.

This is all comprehensively captured by Andrew. The Secret World is both brilliant in its sweep and near-miraculous in the detail and confident judgments provided on two and a half millennia of spying. It covers everything from the argument for a professional intelligence service in a second-century BC Indian treatise on government, the Arthashastra, to the growth of religiously based terrorism in the 1980s and 1990s. The book is a crowning triumph of one of the most adventurous scholars of the security world, who, besides pursuing path-breaking research, has collaborated with two KGB defectors, Vasili Mitrokhin and Oleg Gordievsky, on several books illuminating the workings of their former organisation.

The rise and fall of secret states speaks to Andrew’s contention that little has been learnt from history. The great cryptanalysts of the Islamic world at its time of enlightenment from the eighth century were not known to later generations and their techniques had to be reinvented. The school of cryptanalysts headed by Edward Willes, British royal decipherer for 57 years, declined after his death in 1773. He had made it a family business that his descendants practised with decreasing skill with the result that “British deciphering was less effective at the beginning of the 19th century than for more than a century”.

In the 20th century, and into the 21st, intelligence became organised, routinised and massively funded, especially under the Nazi and Soviet dictatorships. That did not always deliver the intended benefits. Hitler often disregarded intelligence, most ruinously for his regime at Stalingrad, which — Andrew believes — need not have been the crucial defeat it was had he heeded reports on Red Army troop movements and numbers. As the Soviet terror deepened in the 1930s, intelligence was less important to Stalin than the urgings of the dictator’s paranoia and the servile hyperactivity of the leaders of the secret police.

US leaders had for the first half of the 20th century been disdainful of intelligence, seeing it as a tainted trade: Henry Stimson, secretary of state in the 1930s, declared that “gentlemen do not read each others’ mail”. The second world war, the subsequent identification of the Soviet threat, and, more recently, the 9/11 attack, impelled the US towards the creation of huge intelligence services which focused — sometimes to absurd and grossly intrusive lengths — on subversion in America. A major US report billed the country’s secret services as “the most capable intelligence apparatus of any country of the world”. Andrew dismisses this, pointing to the “bungling” of several cases of Soviet penetration, on the part of both the CIA and the FBI. Today, the close alliance between the UK and US agencies, and those of their Australian, Canadian and New Zealand counterparts — the so-called “Five Eyes” — will be strained under the Trump presidency, since the non-American services will seek more intelligence on the dominant partner to inform their political masters of President Donald Trump’s intentions — a challenge as formidable as any they will have faced.

Andrew ends on the most alarming of notes. It is, he writes, a matter of time before nuclear, chemical or biological terrorism will be attempted in target countries — “the question now is not whether some future group of (probably Islamist) terrorists will use WMD, but when they will do so”. The apocalyptic warning inspires both horror and the fervent wish that the intelligence services are as efficient, and their ethical standards as high, as they claim.

The “ethics of spying” would seem oxymoronic. Yet David Omand, a grandmaster of British intelligence, and Mark Phythian, among the sharpest of security scholars, seek to show what hold ethics should have on intelligence officials. This is a work of the highest seriousness. It is a teasing out, in Platonic dialogue form, of what ethical spine a spy should — in a democracy, must — have. It is an exercise never attempted before at this length.

Both authors accept the contradictions inherent in applying ethical imperatives to a trade where lies, deceptions, impersonations and deliberate placing of sources in peril of their lives are necessary and praised by superiors, to whom security officers must display loyalty, honesty and trustworthiness. Such is the necessary split personality.

Several elements aid their search for an ethical base. First, up to 1990, western intelligence services could claim the virtue of protecting freedom, rule of law, democratic politics and robust civil societies against those which had only the simulacrum of these. Other challenges — be it the Provisional IRA in the UK, left- and rightwing terrorism in France, Germany and Italy in the 1970s and 1980s, and presently the growth of terrorist jihadism — allow the comparison between societies of laws and personal freedom and those of ideologies of religious or political tyranny.

How far was Israel justified in assassinating Iranian civilian nuclear scientists, and their spouses, working on weapons for destruction of the Jewish state?

They also draw support from “Just War Theory”, a strain of thought that can be traced back to the efforts of Thomas Aquinas, the 13th-century Italian Dominican monk, to distinguish between conflict begun on reasoned grounds of defence or justified suspicion of attack and that undertaken for conquest, revenge or national aggrandisement. Omand and Phythian see “a usable set of concepts” as including just cause, right intention, proportionality, appropriate and legal authority, reasonable prospect of success, discrimination (between necessary and unnecessary action) and acts of last resort when no other way exists to right a wrong, or protect a people.

Some considerations are utilitarian: an ethically dubious, even wicked, action abroad can be justified if it saves lives at home. The book is enriched by examples — such as how far Israel was justified in assassinating Iranian civilian nuclear scientists, and their spouses, working on weapons meant for the destruction of the Jewish state.

In a chapter on the new ethical conundrums of the internet and associated technologies — “a revolution in human affairs” — there is a detailed discussion on the effects of Edward Snowden’s leaks from the NSA. Both authors believe the leaks had the paradoxically benign effect of forcing western states to be explicit for the first time about the bulk collection of communications data. But Omand insists that Snowden’s collection of hundreds of thousands of files, most of which he had not read, has had the effect of “direct damage” to intelligence on the Taliban, and beyond.

Historically speaking, little has been more damaging than the vast amounts of intelligence shovelled over to the Soviet Union by the members of a unique band of Cambridge-educated traitors — Anthony Blunt, Guy Burgess, John Cairncross, Donald Maclean and Kim Philby — from the 1930s to 1950s. In his biography of Maclean, Roland Philipps illuminates, in both broad and subtle strokes, a life lived on the razor edge of discovery by his colleagues and recurring suspicion on the part of his Soviet masters.

Maclean’s ostensible ethic was patriotism, his true morality a belief in Soviet egalitarianism and revulsion against the US. That he should have been as successful a spy as he was is testimony to a now near-incomprehensible class bias in government, high officialdom and the intelligentsia, that refused to believe that upper-middle-class Cambridge chaps could act as Maclean did. Knowledge of his often ungovernable drunken temper and open allegiance to the left remained within the charmed circles of the fellow diplomats who protected him.

Maclean greatly respected his father, a devout Scots Liberal MP, whose example of patriotism, abstinence and frankness haunted him as, increasingly alcoholic, he drove deeper into the grip of skilled Soviet spy handlers. But Maclean also cleaved to a belief from which he never openly wavered — that communism would spur the servile masses to rise and win the prize of a truly equal, co-operative human race. In an era of swiftly growing fascism, the immiseration of working people and continuing colonial oppression, treachery could be made to seem more like love of humanity than hatred of country.

Maclean probably altered the course of history. Privy to the secret correspondence between Prime Minister Churchill and US President Roosevelt, he provided so much of it to the Soviets that “the British and Americans were unable to outflank Stalin in their preparations for Europe . . . because so much of their thinking was revealed to him”.

As evidence mounted after the second world war of large security breaches in the Washington embassy, Maclean sensed, by early 1951, that the game was up. With his fellow Cambridge graduate and spy Burgess, he was spirited into the Soviet Union, where he remained for the rest of his life, dying in 1983. In our time, Maclean’s ideological dogmatism and disregard for the lives of others has been mirrored in the actions of religious fanatics, including some men and women willing to die for their cause.

The guardianship of a nation’s secrets is now a vast endeavour, its edges increasingly blurred as the online revolution renders more of our world transparent and as those who wish us harm develop more subtle and cruel ways of doing so. Public opinion hovers between pressing agencies to deliver an impossible guarantee of no more attacks — and fearing their hugely expanded grasp and reach. That conflicted state can only become more intense as the world becomes less manageable.

The Secret World: A History of Intelligence, by Christopher Andrew, Allen Lane, RRP£35, 896 pages

Principled Spying: The Ethics of Secret Intelligence, by David Omand and Mark Phythian, Georgetown University Press, RRP$32.95/£20, 304 pages

A Spy Named Orphan: The Enigma of Donald Maclean, by Roland Philipps, Bodley Head, RRP£20, 448 pages

John Lloyd is an FT contributing editor

Join our online book group on Facebook at FTBooksCafe. Subscribe to FT Life on YouTube for the latest FT Weekend videos