On the referendum #21: Branching histories of the 2016 referendum and ‘the frogs before the storm’

‘Politics is gambling for high stakes with other people’s money… Politics is a job that can be compared with navigation in uncharted waters. One has no idea how the weather or the currents will be or what storms one is in for. In politics, there is the added fact that one is largely dependent on the decisions of others, decisions on which one was counting and which then do not materialise; one’s actions are never completely one’s own. And if the friends on whose support one is relying change their minds, which is something that one cannot vouch for, the whole plan miscarries… One’s enemies one can count on – but one’s friends!’ Bismarck.

‘The most important thing is not to fool yourself and you are the easiest person to fool.’ Feynman. 

‘He lies like an eyewitness.’ Russian proverb.

In January 2014 I left the Department for Education and spent the next 18 months away from politics. A few days after the 2015 election I wrote a blog about Michael Gove’s new job touching on the referendum. When I wrote it I assumed I would carry on studying and would not be involved in it. About ten days later I was asked by an assortment of MPs, rich businessmen, and campaigners including Matthew Elliott to help put together an organisation that could fight the referendum. I was very reluctant and prevaricated but ended up agreeing. I left my happy life away from SW1 and spent eight weeks biking around London persuading people to take what was likely to be a car crash career decision – to quit their jobs and join a low probability proposition: hacking the political system to win a referendum against almost every force with power and money in politics. In September we had an office, in October ‘Vote Leave’ went public, in April we were designated the official campaign, 10 weeks later we won.

Why and how? The first draft of history was written in the days and weeks after the 23 June and the second draft has appeared over the past few weeks in the form of a handful of books. There is no competition between them. Shipman’s is by far the best and he is the only one to have spoken to key people. I will review it soon. One of his few errors is to give me the credit for things that were done by others, often people in their twenties like Oliver Lewis, Jonny Suart, and Cleo Watson who, unknown outside the office, made extreme efforts and ran rings around supposed ‘experts’. His book has encouraged people to exaggerate greatly my importance.

I have been urged by some of those who worked on the campaign to write about it. I have avoided it, and interviews, for a few reasons (though I had to write one blog to explain that with the formal closing of VL we had made the first online canvassing software that really works in the UK freely available HERE). For months I couldn’t face it. The idea of writing about the referendum made me feel sick. It still does but a bit less.

For about a year I worked on this project every day often for 18 hours and sometimes awake almost constantly. Most of the ‘debate’ was moronic as political debate always is. Many hours of life I’m never getting back were spent dealing with abysmal infighting among dysfunctional egomaniacs while trying to build a ~£10 million startup in 10 months when very few powerful people thought the probability of victory was worth the risk of helping us. (Two rare heroes who put up a lot of their own money and supported the team were Peter Cruddas and Stuart Wheeler.) Many of those involved regarded their TV appearances as by far the most important aspect of the campaign. Many regarded Vote Leave as ‘the real enemy’.

It is hard to explain the depth of TV derangement that gobbles up SW1 souls. Much of politics involves very similar tragi-comic scenes re-created by some of the basic atoms of human nature – fear, self-interest and vanity. The years, characters, and contexts change, the atoms shuffle, but the stories are the same year after year, century after century. Delusions and vanity dominate ‘rationality’ and ‘public service’. Progress, when it comes, is driven by the error-correcting institutions of science and markets when political institutions limit the damage done by decision makers at the apex of centralised hierarchies. It rarely comes from those people, and, when it does, it is usually accidental or incidental to their motives.

Discussions about things like ‘why did X win/lose?’ are structured to be misleading and I could not face trying to untangle everything. There are strong psychological pressures that lead people to create post facto stories that seem to add up to ‘I always said X and X happened.’ Even if people do not think this at the start they rapidly construct psychologically appealing stories that overwrite memories. Many involved with this extraordinary episode feel the need to justify themselves and this means a lot of rewriting of history. I also kept no diary so I have no clear source for what I really thought other than some notes here and there. I already know from talking to people that my lousy memory has conflated episodes, tried to impose patterns that did not actually exist and so on – all the usual psychological issues. To counter all this in detail would require going through big databases of emails, printouts of appointment diaries, notebooks and so on, and even then I would rarely be able to reconstruct reliably what I thought. Life’s too short.

I’ve learned over the years that ‘rational discussion’ accomplishes almost nothing in politics, particularly with people better educated than average. Most educated people are not set up to listen or change their minds about politics, however sensible they are in other fields. But I have also learned that when you say or write something, although it has roughly zero effect on powerful/prestigious people or the immediate course of any ‘debate’, you are throwing seeds into a wind and are often happily surprised. A few years ago I wrote something that was almost entirely ignored in SW1 but someone at Harvard I’d never met read it. This ended up having a decisive effect on the referendum.

A warning. Politics is not a field which meets the two basic criteria for true expertise (see below). An effect of this is that arguments made by people who win are taken too seriously. People in my position often see victory as confirmation of ideas they had before victory but people often win for reasons they never understand or even despite their own efforts. Cameron’s win in 2015 was like this – he fooled himself about some of the reasons why he’d won and this error contributed to his errors on the referendum. Maybe Leave won regardless of or even despite my ideas. Maybe I’m fooling myself like  Cameron. Some of my arguments below have as good an empirical support as is possible in politics (i.e. not very good objectively) but most of them do not even have that. Also, it is clear that almost nobody agrees with me about some of my general ideas. It is more likely that I am wrong than 99% of people who work in this field professionally. Still, cognitive diversity is inherently good for political analysis so I’ll say what I think and others will judge if there’s anything to learn.

Apologies for the length but I didn’t have time to make it shorter. The next ones will be short.


Reality has branching histories, not ‘a big why’

Much political analysis revolves around competing simple stories based on one big factor such that, in retrospect, ‘it was always clear that immigration would trump economic interest / Cameron’s negotiation was never going to be enough / there is an unstoppable populist tide’, and so on. Alternatives are quickly thought to have been impossible (even if X argued the exact opposite repeatedly). The big event must have had an equally big single cause. Confirmation bias kicks in and evidence seeming to suggest that what actually happened would happen looms larger. People who are quite wrong quickly persuade themselves they were ‘mostly right’ and ‘had a strong feeling’ unlike, of course, the blind fools around them. Soon our actual history seems like the only way things could have played out. Brexit had to happen. Trump had to win.

You see these dynamics all the time in historical accounts. History tends to present the 1866 war between Prussia and Austria as almost inevitable but historians spend much less time on why Bismarck pulled back from war in 1865 and how he might have done the same in 1866 (actually he prepared the ground so he could do this and he kept the option open until the last minute). The same is true about 1870. When some generals tried to bounce him into a quick preventive war against Russia in the late 1880s he squashed them flat warning against tying the probability of a Great Power war to ‘the passions of sheep stealers’ in the Balkans (a lesson even more important today than then). If he had wanted a war, students would now be writing essays on why the Russo-German War of 1888 was ‘inevitable’. Many portray the war that broke out in August 1914 as ‘inevitable’ but many decisions in the preceding month could have derailed it, just as decisions derailed general war in previous Balkan crises. Few realise how lucky we were to avoid nuclear war during the Cuban Missile crisis (cf. Vasili Arkhipov) and other terrifying near-miss nuclear wars. The whole 20th Century history of two world wars and a nuclear Cold War might have been avoided if one of the assassination attempts on Bismarck had succeeded. If Cohen-Blind’s aim had been very slightly different in May 1866 when he fired five bullets at Bismarck, then the German states would certainly have evolved in a different way and it is quite plausible that there would have been no unified German army with its fearsome General Staff, no World War I, no Lenin and Hitler, and so on. The branching histories are forgotten and the actual branch taken, often because of some relatively trivial event casting a huge shadow (perhaps as small as a half-second delay by Cohen-Blind), seems overwhelmingly probable. This ought to, but does not, make us apply extreme intelligent focus to those areas that can go catastrophically wrong, like accidental nuclear war, to try to narrow the range of possible histories but instead most people in politics spend almost all their time on trivia.

We evolved to make sense of this nonlinear and unpredictable world with stories. These stories are often very powerful. On one hand the work of Kahneman et al on ‘irrationality’ has given an exaggerated impression. The fact that we did not evolve to think as natural Bayesians does not make us as ‘irrational’ as some argue. We evolved to avoid disasters where the probability of disaster X happening was unknowable but the outcome was fatal. Rationality is more than ‘Bayesian updating’. On the other hand our stories do often obscure the branching histories of reality and they remain the primary way in which history is told. The mathematical models that illuminate complex reality in the physical sciences do not help us much with history yet. Only recently has reliable data science begun to play an important role in politics.

Andrew Marr wrote recently about the referendum with a classic post facto ‘big event must be caused by one big factor’ story:

‘Connected to this is the big “why?”. I don’t think we voted to leave the EU because of clever tacticians or not-quite-clever-enough pollsters, or even because Johnson decided that one of his columns was better than another. I think we voted to leave because so many British people had been left behind economically and culturally for so long, and were furious about it; and because, from the 2008 financial crisis onwards, they had accumulated so much contempt for the political elites. In these circumstances any referendum narrows down to a single question: “Are you happy with the way things are?” The answer was “no”.’ Andrew Marr, October 2016.

‘The big why?’ is psychologically appealing but it is a mistake. In general terms it is the wrong way to look at history and it is specifically wrong about the referendum. If it were accurate we would have won by much more than we did given millions who were not ‘happy with the way things are’ and would like to be out of the EU reluctantly voted IN out of fear. Such stories oversimplify and limit thinking about the much richer reality of branching histories.


Branching histories in 2016: three powerful forces, many possible campaigns

Sometimes the outcome of a vote is clear before a campaign starts such that it is reasonable to say ‘the campaign didn’t matter’ other than in the negative sense that, provided it avoids huge disasters, the twists and turns, the exact messages and adverts, thousands of decisions taken and so on very likely had no impact on the binary outcome. For example, Reagan’s re-election campaign in 1984 or Blair’s re-election campaign in 2001 were campaigns like this. Both won by so much and were clearly predicted by very large and historically very unusual poll leads well in advance. It is not plausible to say that the weeks of campaigning affected who won. At most the campaigns affected the scale of victory.

The referendum was not like this. Throughout the second half of 2015 and the beginning of 2016 the averages of polls – the only sensible way to look at polls – showed clear IN leads. All polls showed significant shifts towards Leave in the last five weeks (then a shift towards Remain at the end but this was at least partly because London-based pollsters changed their methodology thinking that they were making them more accurate – they fooled themselves). Polls tracking deeper attitudes that had been consistent for years suddenly changed in the last few months in ways that were significant given the close outcome. Recent claims that the polls ‘really’ showed Leave ahead all the time should be taken with very large pinches of salt given their dodgy statistical claims, charlatan authors like Matt Goodwin (who treats data dishonestly), and the inherent impossibility of discovering the truth of such a question.

One example from our private ICM polls (I will post the data tables for all these): Vote Leave asked people to choose between these options regularly to probe attitudes to the EU that are more informative than just the referendum question. The 11 point gain for ‘strong out’ is much bigger than the margin of error, is supported by other data, and is clearly significant.


The cold reality of the referendum is no clear story, no ‘one big causal factor’, and no inevitability – it was ‘men going at it blind’. The result was an emergent property of many individual actions playing out amid a combination of three big forces (see below). Many of these actions were profoundly nonlinear and interdependent and the result that we actually witnessed was very close. If about 600,000 people – just over 1% of registered voters – had decided differently, IN would have won. This is a small enough margin that it could easily have happened if quite a few specific events and decisions had turned out differently. If just one person had behaved differently the dominant story now would be ‘the economy was always going to trump a revolt against the elites, the status quo and “the economy stupid” always win’ – which is what the overwhelming majority of pundits said before 23 June and in some cases had drafted for their columns after the vote.

For example, if Michael Gove had stayed out of the campaign then Vote Leave would almost certainly have either collapsed (which it nearly did anyway) or been forced into fighting the campaign on a losing message like ‘Go Global’, a firm favourite for many years among a subset of MPs and Farage’s inner circle (Leave.EU adopted this as its first slogan) and a total loser with the public. (Therefore another counterfactual: why did Cameron and Osborne not try very hard to get a clear commitment from Gove that all he would do is issue a statement but would carry on with his day job and would not campaign? I hope he would have refused but it was worth a shot and they didn’t try very hard.)

Without Boris, Farage would have been a much more prominent face on TV during the crucial final weeks, probably the most prominent face. (We had to use Boris as leverage with the BBC to keep Farage off and even then they nearly screwed us as ITV did.) It is extremely plausible that this would have lost us over 600,000 vital middle class votes.

Without Victoria Woodcock, an absolutely phenomenal manager and by far the single most important person in the management of Vote Leave (and who would have been running Downing Street now but for the Gove-Boris debacle – more branching histories), we would not have been able to build anything like the structure we did and this could easily have cost us the winning margin of votes.

Anybody who says ‘I always knew X would win’ is fooling themselves. What actually happened was one of many branching histories and in many other branches of this network – branches that almost happened and still seem almost real to me – we lost.

Problems with Vote Leave

This is not a claim that ‘we won because of the Vote Leave campaign’. Our campaign failed to do much that we should have done. There were powerful connections between:

  • infighting over who appeared on broadcast and strategy,
  • the lack of resources (many kept clear because of the infighting and many used infighting as an excuse to keep clear of something they thought was doomed),
  • the extreme difficulty of finding a governance system that could work,
  • four crucial posts held by the wrong people (including the disastrous John Mills as first Chairman),
  • the fundamental structure of how the media works (see below),
  • the extreme difficulty of getting prominent people to say on TV what research showed was necessary to win, and
  • the lack of anything resembling a well-organised mass movement.

Despite many years to prepare, the eurosceptic community had built remarkably little to prepare for the battle. On the ground were many small ineffective and often warring little groups and essentially no serious machinery (though Business for Britain had begun to build a business network). All this had to be built almost entirely from scratch in an environment in which many of those in charge of the small groups were sure we would lose, were less interested in winning than they were in ‘preserving our group’s identity Dominic’, and were keen to get their hands on cash being handed out by Leave.EU on condition that they not contribute to the campaign with Vote Leave. At various points UKIP HQ sent out emails to UKIP activists telling them not to work with Vote Leave and some senior activists were told by Farage’s gang that they would lose their UKIP jobs if they helped our ground campaign (luckily most of those out on the ground ignored these instructions but they were disruptive).

The office implemented the winning message in ~125 million leaflets and nearly a billion targeted digital adverts regardless of all complaints. We recruited more active volunteers (~12,000) in 10 months than UKIP in 25 years (~7,000 according to Farage). Our GOTV effort targeted crucial voters identified by traditional polling, a new type of experimental polling, the ground campaign, and the social media campaign, all overseen by the data science team. But until the last 4-5 weeks we had a big problem getting those going on TV to give the same message. The office could only do so much. If Boris, Gove, and Gisela had not supported us and picked up the baseball bat marked ‘Turkey/NHS/£350 million’ with five weeks to go, then 650,000 votes might have been lost. In the awful weekly campaign committee meetings, there were constant complaints and arguments for variations on ‘Go Global’ (until all the polls swung our way and people remembered ‘I’ve always said stick with 350 million’.) The Big Three knocked this back despite great pressure.

Some people had spent a quarter of a century talking about things that appealed to about 10% of the population and they would not pay attention to what millions of normal people actually knew and thought (‘I’ve spent years trying to ignore the NHS in elections Dominic and I’m not going to change now’ said many like Peter Bone). Media planning was extremely hard. Paul Stephenson’s media team of half a dozen, massively outnumbered by hundreds of officials, did a fantastic job but we could have done so much more if more MPs had been more determined and more supportive.

It should be remembered that the net effect of Conservative MPs was strongly supportive of IN. We won despite the net effort of Conservative Party MPs, not because of them, though the support from a small fraction was vital. Although Leave voters were more enthusiastic and determined than Remain voters, Cameron and Osborne were more focused on winning than most Leave MPs were. (Almost all Labour MPs seemed to be in a parallel universe until they got intelligence from their constituencies about postal votes after which they panicked ineffectually.)

Most of the MPs we dealt with were not highly motivated to win and lacked extreme focus, even those who had been boring everybody about this for decades. They sort of wanted to win but they had other priorities. They were very happy having dinner parties and gossiping. They were very happy coming to meetings with people they thought were important. This wasted enormous amounts of time as we had to create a string of Potemkin committees for people to attend while the core team actually did the campaign, then reinvent them as people became convinced that there were other secret meetings that they were being excluded from. They were very happy to be on the Today Programme. But they didn’t want to win that much. Not enough to work weekends. Not enough to stop having all their usual skiing holidays and winter beach holidays. Not enough to get out on the streets day after day.  Not enough to miss a great shooting weekend. Not enough, most of them, to risk annoying a Prime Minister who they thought would still control their next job after 23 June.

This lack of motivation is connected to another important psychology – the willingness to fail conventionally. Most people in politics are, whether they know it or not, much more comfortable with failing conventionally than risking the social stigma of behaving unconventionally. They did not mind losing so much as being embarrassed, as standing out from the crowd. (The same phenomenon explains why the vast majority of active fund management destroys wealth and nobody learns from this fact repeated every year.)

Our core campaign team were not like this. They sacrificed weekends, holidays, and family events. They worked like dogs week in week out for little money often treated with appalling rudeness by people calling from their beach loungers (Boris, Gisela and Gove were three notable exceptions and all three were liked by junior staff partly because of their good, therefore rare, manners). We were happy to risk looking stupid to win. We knew that almost nobody in SW1 understood or agreed with what we were doing. We also knew we had more chance of winning if we did not explain a lot of it – most importantly the entire digital and data science element which (combined with the ground campaign and GOTV) gave us a chance to exploit strong network effects  (and which we hid from the Board and MPs, see HERE).

Example… We were urged by everyone to hire a big advertising agency and do traditional posters. ‘When can we discuss our posters?’ I was asked constantly by people who would then try to explain to me their creative ideas (‘we need another Labour Isn’t Working, Dominic, I’ve got an idea for a picture of the globe and arrows…’). One of the few reliable things we know about advertising amid the all-pervasive charlatanry is that, unsurprisingly, adverts are more effective the closer to the decision moment they hit the brain. Instead of spending a fortune on an expensive agency (with 15% going to them out of ‘controlled expenditure’) and putting up posters to be ‘part of the national conversation’ weeks or months before the vote, we decided to 1) hire extremely smart physicists to consider everything from first principles, 2) put almost all our money into digital (~98%), 3) hold the vast majority of our budget back and drop it all right at the end with money spent on those adverts that experiments had shown were most effective (internal code name ‘Waterloo’). When things are digital you can be more empirical and control the timing. The world of advertising agencies and PR companies were sure we had screwed up because they did not see what we were doing. (Tim Bell told everybody we were doomed because we hadn’t hired one of his companies.) This points to another important issue – it is actually hard even for very competent and determined people to track digital communication accurately, and it is important that the political media is not set up to do this. There was not a single report anywhere (and very little curiosity) on how the official Leave campaign spent 98% of its marketing budget. There was a lot of coverage of a few tactical posters.

There were some MP heroes.

Example… Steve Baker often disagreed with me, sometimes very strongly, but he was a rare person in the campaign – an honest man. Not only did Steve win some important Parliamentary battles he also played a vital role during the attempted coup of 25 January. If he had thrown in his lot with the coup, it might have proved fatal. Instead he spoke honestly about the situation. We did not agree and we were both under pressure from a set of people who thought that ‘if they [HQ/MPs] control the campaign we will lose, we [HQ/MPs] must control it’. We came to an agreement that we both stuck to. With five weeks to go, there was an attempt to revive the coup by a couple of VL Board members working with players from the January coup like Malcolm Pearson. The demand was to replace the Big Three (Boris, Gisela, Gove) and the core campaign team with Farage, and replace £350 million / NHS with ‘go global’ trade babble. This didn’t get past the usual weekend boozy chats partly because of Steve Baker telling them he thought it a mad plan. This also shows how volatile the situation was right until the end and how few prominent eurosceptics even then understood that a) the £350 million / NHS argument was necessary to win and b) their ‘go global’ message was a total loser.

Other MPs also made significant personal sacrifices – backbenchers like Anne Marie Trevelyan and Graham Stringer, and ministers like George Eustice and Dominic Raab.

Rough balance of forces

The IN side started with huge structural advantages.

  1. IN started in 2015 well ahead in the polls and had the advantage of having the status quo on its side which is intrinsically easier to explain than change is, as lots of historical data around the world shows. Usually the ‘change’ campaign has to start considerably ahead in order to win as it loses support as the campaign goes on. This argument was even stronger with something so much bigger and more complex like the EU. VL had to persuade millions of people to risk a profound change. Those on the IN side made this point repeatedly for many months. They were right then. After 23 June many of them say the exact opposite – it’s so complex to explain all the wonders of the EU, they say, and so easy to argue for change. This is laughable.
  2. IN had the government at its heart including the Downing Street machine, the Cabinet Office, and Government departments and agencies all of which added up to thousands of people including hundreds of press officers. Cameron and Heywood also instructed Permanent Secretaries not to share EU material with Secretaries of State supporting Vote Leave in order that they did not have access to new information about all the ways in which EU law affected policy. (In general Whitehall has made great efforts to hide the scope of EU control. It also preserves, Potemkin-style, old processes like circulating Cabinet papers ‘for approval’ where the only acceptable response is ‘approve’ – it is not actually legally possible not to ‘approve’ but still the papers are sent round via the absurd red box system daily.) VL had a few dozen effective people and no access to the official machine other than some leaks. We had a research team of about five. MPs proved largely useless in helping this team.
  3. IN controlled one side of the renegotiation and its timing. VL was at the mercy of events and could not get any ministers supporting us until the process ended.
  4. IN controlled the timing of the referendum. VL had to plan resources on the basis of many scenarios.
  5. IN controlled the Cabinet and junior ministers – bribes for support and threats to deter. They had the chance to set the terms for how ministers engaged in the campaign (though they partly blew this). VL had to meet ministers in secret, could guarantee them no jobs, and (as was pointed out to me by many) could not dodge the basic truth that purely from a personal career perspective it was usually better to support the PM.
  6. IN controlled the governing party and the Parliamentary timetable and procedures. VL had to work with a small number of MPs many of whom had spent many years in constant opposition to their own leadership and were unused to any sort of discipline or collective action.
  7. IN set the legal rules. VL faced a huge imbalance in how these worked. For example, Cameron even during the official campaign could do huge events at places like the British Museum and the IN campaign did not have to account for such events as part of their £7 million. Meanwhile VL was told by the Electoral Commission that if people we did not even know put up huge signs that appeared on TV we might get billed for them. There were many other consequences of the imbalance. E.g. the Government’s legal timetable meant we had to commit before the official start of the campaign to a load of activity that would occur after the official start of the campaign without knowing if we would be the official campaign and therefore legally entitled to spend this money. We therefore had to choose between either a) not do various things, be sure we would not break the law, and lower the chances of winning or b) do the right thing for the campaign and riski being judged to have broken the law. Obviously we did (b) though we had to hide this choice from some of those on our Board as this was exactly the sort of thing some of them were very weak about.
  8. IN had access to huge resources – financial, personnel etc. IN had the support of almost every entity with power in Britain, Europe, and the world from the senior civil service to the CBI to the big investment banks, to Obama and the world bureaucracy (G20, UN, IMF etc).  Very few senior people were prepared to risk supporting us. Those who did mostly did so in a small way and on their own terms without getting involved in our campaign. While IN could send out name after name to deliver their message, we could depend on very few names who would deliver our message. The Government machine, the Commission, and the Cabinet Office were effective in scaring off prominent people from supporting us; many of them told us (some embarrassed) about the phone calls they’d had and their ‘duty to shareholders’ and so on. Advanced media planning was almost impossible and we had to shuffle things around at short notice constantly. IN had millions more than us before the campaign ever started and used this money for direct voter communication. We could not afford this. We sent out one 10 million voter mailing to people identified by the physicists just before the spending limits started and we could only do this by tricking some of those on our Board about the numbers. (I was also  helped by Peter Cruddas saying, ‘Don’t worry about the fundraising situation, don’t listen to everybody panicking, just do whatever it takes to do the campaign, if the money doesn’t come I guarantee I’ll put in whatever you need’. I knew I could trust him. This gave us vital flexibility and also meant we could ignore some of those on the Board who were more focused on whether they may be liable for a bill post-23/6 than they were on winning.)
  9. IN had the support of most journalists and senior management in the main broadcasters. The broadcasters let the Government set the agenda on TV for almost the entire campaign, apart from ten crucial days after the immigration numbers on 26 May. VL had the support of some powerful papers but we were overwhelmed on TV news. (Two broadcast journalists who were conspicuous by their unusual professionalism and determination to act fairly despite the behaviour of some of their management were Laura K and Allegra Stratton.)
  10. IN started with legal access to vast amounts of electoral data from at least three political parties, unofficial / illegal access to vast amounts of data from things like CCHQ data and the Crosby/Messina models built during the campaign, and vast amounts of commercial data. (CCHQ laughably claimed that there were ‘Chinese walls’ that prevented any abuse of Party data.) VL had none of these things. We could not even afford to buy standard commercial datasets (though the physicists found ingenious ways around this). We had no way even to acquire the electoral roll until the official process allowed us in early 2016, after which we had to wait a couple of months for LAs to fulfil their legal obligations to provide us with the data (which they did patchily and often late).
  11. IN had a great boost to its fortunes in the form of a network linking Nigel Farage, Aaron Banks, assorted peers (e.g. Malcolm Pearson), MPs (e.g. Bill Cash), businessmen (e.g. Richard Smith), and a handful of Vote Leave Board members (including the one-time Chairman John Mills) and some staff foisted on us (one of whom won the title of the most repellent person I’ve met in politics – Nigel Griffiths, an ex-MP who some female staff refused to be in the same room with). Farage put off millions of (middle class in particular) voters who wanted to leave the EU but who were very clear in market research that a major obstacle to voting Leave was ‘I don’t want to vote for Farage, I’m not like that’. He also put off many prominent business people from supporting us. Over and over they would say ‘I agree with you the EU is a disaster and we should get out but I just cannot be on the same side as a guy who makes comments about people with HIV’.

On 25 January 2016 a network of these characters launched a coup. But for the actions of Stephen Parkinson, Paul Stephenson,  and Victoria Woodcock (supported by most but not all of the office) it would have succeeded. This would have given control of the official campaign to the Farage crowd. They ran with vapid slogans like ‘Be in the know’. Ironically for a group of people who claim to be anti-SW1 they rehashed the classic losing SW1 eurosceptic trope for 25 years – ‘Go Global’ – showing how little they understood the electorate and mass communication. They rejected the connection between immigration, £350 million and the NHS, which was absolutely vital, as the IN side has said after 23 June (see below). They published dumb offensive videos. They talked about privatising the NHS. They built little grassroots organisation and their claims about social media were (and remain ) laughable. Farage himself admitted after 23 June that they did not have the organisation to run the campaign if they had won designation: ‘quite what we would have done if we had got it I’m not really sure!’, which sums them up (Shipman, Location 4,150). The media would have covered this gang’s official campaign as a version of their own book – a bunch of childish dodgy boozers on an ego trip.

Before the 2015 election Farage said to me at Stuart Wheeler’s that he knew he could not be the leading face of the campaign – ‘I’m one of the generals but I can’t lead the army’ he said, to my relief. When I next saw him in the summer, I was amazed at how his tune had changed, his obsession with the debates, and his pessimism. One can only understand some of the behaviour from those around Farage if you realise that much of their operation was about positioning Farage for what they assumed would be defeat.

One of the biggest problems during the campaign and biggest misconceptions after concerns this issue. Those who argued ‘we need one campaign’ were wrong. Those who argue now ‘we would have won by more if there’d been one campaign’ are wrong. One campaign would have meant total bedlam and 60-40 defeat.

If MPs had had extreme focus on winning then they would not have used Farage as leverage against us viz official designation and therefore much of the infighting could have been avoided as Farage would have done a sensible deal with us early, realising much earlier that we would not compromise over him running the campaign under any circumstances. By encouraging Farage to think that he could get a much more prominent position, people like Bill Cash nearly destroyed everything.

Given all these huge advantages, if their campaign had been of equal effectiveness to Vote Leave then, all else remaining equal, Cameron would almost certainly (>95% likely) have won.

Why did all these forces not add up to overwhelming and devastating firepower? If you want to understand the combination of things that gives us largely dysfunctional government and therefore undermined the IN campaign – a mix of selecting and promoting the wrong people, wrong education and training, bad incentives, anti-adaptive institutions and so on – then read this in which I explain in detail why Whitehall does not and cannot work properly.

The approximate truth

The closest approximation to the truth that we can get is that Leave won because of a combination of 1) three big, powerful forces with global impact: the immigration crisis, the financial crisis, and the euro crisis which created conditions in which the referendum could be competitive; 2) Vote Leave implemented some unrecognised simplicities in its operations that focused attention more effectively than the other side on a simple and psychologically compelling story, thus taking advantage of those three big forces; and 3) Cameron and Osborne operated with a flawed model of what constitutes effective political action and had bad judgement about key people (particularly his chief of staff and director of communications) therefore they made critical errors. Even if (1) and (2) had played out the same, I think that if that duo had made one of a few crucial decisions differently they would very likely have won.

When I started to research opinion in 2014-15 and compared it to my experience of the euro campaign (1999-2002), it was clear three forces had changed opinion on the EU.

1) The immigration crisis. 15 years of immigration and, recently, a few years of the migration crisis from the East and Africa, dramatically portrayed on TV and social media, had a big effect. In 2000, focus groups were already unhappy with immigration but did not regard it as a problem caused by the EU. By 2015, the EU was blamed substantially for the immigration/asylum crisis and this was entangled with years of news stories about ‘European courts’ limiting action against terrorists and criminals. Actually often these stories concerned the Strasbourg court of the ECHR (not the ECJ) though, ironically, the EU’s adoption of its Charter of Fundamental Rights meant that many issues concerning the ECHR became relevant to the EU debate, something that almost nobody in SW1 realised and we tried and largely failed to explain (one of the very few who did understand this was Boris’s wife, an accomplished lawyer, who I discussed this with in autumn 2015).

2) The 2008 financial crisis. This undermined confidence in Government, politicians, big business, banks, and almost any entity thought to be speaking for those with power and money. Contra many pundits, Miliband was right that the centre of gravity has swung against free markets. Even among the world of Thatcherite small businesses and entrepreneurs opinion is deeply hostile to the way in which banks and  public company executive pay work. Over and over again outside London people would rant about how they had not/barely recovered from this recession ‘while the politicians and bankers and businessmen in London all keep raking in the money and us mugs on PAYE are paying for the bailouts, now they’re saying we’ve just got to put up with the EU being crap or else we’ll be unemployed, I don’t buy it, they’ve been wrong about everything else…’ All those amazed at why so little attention was paid to ‘the experts’ did not, and still do not, appreciate that these ‘experts’ are seen by most people of all political views as having botched financial regulation, made a load of rubbish predictions, then forced everybody else outside London to pay for the mess while they got richer and dodged responsibility. They are right. This is exactly what happened.

Many Tory MPs and ‘free market’ pundits / think tankers are living in a fantasy world in which they want hostility to big business to end even though everybody can see that those who failed largely escaped responsibility and have even gone back to doing the same things. (I’ve argued since 2001 for big changes on executive pay to almost zero effect. SW1 is full of people who think they’re ‘defending markets’ but are actually defending the opposite – corporate looting. In the 1930s Britain put people in jail because of what happened in the 1920s. We should have done the same after 2008.)

3) The euro crisis. Britain joined the EEC because it was a basket case in the 1970s and ‘Europe’ was seen as a modernising force that could help us recover and improve the economy and living standards. As the euro crisis hit, millions saw Greece in chaos, even flames, for month after month. This undermined confidence in the EU as a modern successful force – ‘it’s so bad even Germany’s in trouble now because of the euro’, ‘not even Germany can afford to sort this out’, people would say.

Together these three big forces undermined confidence in the EU project as a modern force for progress that brings prosperity and solves problems and pushed it into about 30-35% of the population (younger, richer, better educated) which increasingly saw the EU in terms of ‘are you racist / supporter of Farage?’ This feeling was central in 1975. It diminished gradually but was still partly there 1999-2002 when I was doing focus groups on the euro. (It is why I had so many arguments at the time with eurosceptics explaining to them that if we accepted Blair’s framing of the euro debate as IN/OUT of the EU, we would lose. Our two slogans were therefore ‘Europe yes, euro no’ and ‘Keep the pound, keep control’.)

Second, they undermined confidence in those in charge. There had been strong anti-Westminster feelings growing for over a decade. In 2004 with James Frayne and my uncle I set up the campaign to fight the referendum on the North East Regional Assembly as a training exercise for an EU referendum (then envisaged after Blair’s 2005 victory). We came from behind and won 80-20 (not a misprint) despite having almost no money, no support, and the entire North East establishment against us because we exploited this feeling (‘politicians talk, we pay‘ was our slogan). SW1 ignored the result. It did not appreciate the scale of this growing force even after the financial crash and expenses scandal. Normal electoral politics and the structural grip of established political parties fooled insiders about the extent of support for people like Cameron. Cameron won negatively – because he was not Brown or Miliband. There was very little positive feeling for him. They fought the referendum with him and Osborne at the front as if they were fighting Brown or Miliband and asking people to make a choice: this is not how most people saw it.

These three big forces and the failure of the parties to cope, combined with the daily resentment of paying taxes for the bill of the 2008 Crash, meant that in a vote like 2016 where people did not have to vote to stop Brown or Miliband ‘stealing my money’, millions who were unpersuaded by Cameron/Osborne felt free to vote positively for something (‘take back control’) and against a duo they disliked, distrusted, and saw as representative of politicians’ failure over many years.

These three big forces had global impact and had much more effect on people who pay a normal amount of attention to politics than every speech, article, pamphlet and ‘campaign’ about the EU over 15 years, the sum total of which had almost no discernible effect.

Those who think I am exaggerating the relative lack of influence of conscious SW1 activity could consider another example – the Gove education reforms 2010-14 (which I was closely involved with). These reforms were one of the most prominent stories of the 2010-15 Government with thousands of stories and broadcast discussions. I researched public attitudes to these reforms after I resigned from government in January 2014 (contrary to widespread belief the Cameron operation spent very little time and resources before 2014 on researching public opinion, they were focused on the media rather than the public). Approximately nothing of our arguments  – including the years of speeches by Blair too – had got through to the public.The entire SW1 media debate had approximately no impact on public opinion. People had some idea of some changes if they had kids in school but knew almost nothing of the arguments. Consider how much more motivated people were to learn about this than they were about the EU. (Part of the reason is that the language that Cameron and SW1 generally used was about ‘choice, competition’ and so on. I was almost totally unsuccessful in persuading people to talk about the issue in a different way which is one of the reasons I spent so little time on communication and almost all my time on management in the DfE. Gove knew the problem but also knew that there was no chance of getting Cameron to do things differently.)

This is relevant to the immigration argument in particular. Many pundits who described themselves as ‘modernisers’ wrote columns over the years arguing that immigration was an issue because Cameron was making foolish promises about it and the media therefore paid more attention to it. This is wrong. Cameron’s foolish promises certainly made his situation worse but it is wrong to think that public interest in an issue is proportional to the attention paid by politicians and newspapers in SW1. The public only pays attention to a tiny subset of issues that politicians and the media bang on about. It is largely impossible to predict which things will catch fire and which will not, though process stories and ‘scandals’ almost always have zero effect and insiders repeatedly get this wrong. Long before there was any prominent media discussion of ‘the Australian points system’ you could hear it being discussed in focus group after focus group to an extent that was very surprising to me and was very surprising to every single person I discussed it with, including Farage (who adopted the policy because of focus groups, the causal chain was not – Farage talks >> focus groups respond).

Making these three forces even more powerful was the nature of the reaction from those in charge in the EU and Britain – a general failure not only to grip the problems but even to show that they understood what the problems were. There was clearly no sensible movement for reform of the EU. As it lurched from crisis to crisis, its only response was ‘the EU needs more power’ (this is, of course, the founding logic of the Monnet-Delors system). The British Government clearly had no sensible plan for dealing with the EU’s crises and dysfunction. Worse, their responses were often obviously rubbish, such as the ‘tens of thousands’ immigration promise that people could see had no chance of being met yet politicians just kept repeating it. People naturally concluded – these guys in London don’t grasp the seriousness of the problems, they haven’t a clue what to do, and are treating us like idiots. Cameron’s renegotiation did not change this view. The Government therefore entered the campaign in a very different state to Wilson in 1975.

These three forces meant that by summer 2015 only about a third of the electorate positively wanted to be inside the EU. Another third strongly wanted to leave and were not worried about the economy. Another fifth had roughly the view that – the EU is rubbish, I’d like to be outside, but I’m worried about the short-term effects on jobs and living standards so maybe I’ll vote IN (see the ICM table above). Further, our research showed that the strong Leave third was significantly more enthusiastic about the referendum than the strong Remain third and the swing fifth, and therefore more likely to vote.

Vote Leave exploited these forces

I will go into this in much more detail and I will ignore all management/operational issues here.

Our story rested on five simple foundations that came from listening very hard to what people really knew, thought, and said:

1. ‘Let’s take back control’. The overall theme. When I researched opinion on the euro the best slogan we could come up with was ‘keep control’. I therefore played with variations of this. A lot of people have given me a lot of credit for coming up with it but all I really did was listen. (NB. ‘back’ plays into a strong evolved instinct – we hate losing things, especially control.)

2. ‘The official bill of EU membership is £350 million per week – let’s spend our money on our priorities like the NHS instead.’ (Sometimes we said ‘we send the EU £350m’ to provoke people into argument. This worked much better than I thought it would. There is no single definitive figure because there are different sets of official figures but the Treasury gross figure is slightly more than £350m of which we get back roughly half, though some of this is spent in absurd ways like subsidies for very rich landowners to do stupid things.)

Pundits and MPs kept saying ‘why isn’t Leave arguing about the economy and living standards’. They did not realise that for millions of people, £350m/NHS was about the economy and living standards – that’s why it was so effective. It was clearly the most effective argument not only with the crucial swing fifth but with almost every demographic. Even with UKIP voters it was level-pegging with immigration. Would we have won without immigration? No. Would we have won without £350m/NHS? All our research and the close result strongly suggests No. Would we have won by spending our time talking about trade and the Single Market? No way (see below).

NB. Unlike most of those on our side the IN campaign realised the effectiveness of this, as Cooper, Coetze and others said after 23 June. E.g. ‘The power of their £350 million a week can’t be overstated.’ Andrew Cooper, director of strategy for the IN campaign.

Some people now claim this was cynical and we never intended to spend more on the NHS. Wrong. Boris and Gove were agreed and determined to do exactly this. On the morning of 24 June they both came into HQ. In the tiny ‘operations room’ amid beer cans, champagne bottles, and general bedlam I said to Boris – on day one of being PM you should immediately announce the extra £100 million per week for the NHS [the specific pledge we’d made] is starting today and more will be coming – you should start off by being unusual, a political who actually delivers what they promise. ‘Absolutely. ABSOLUTELY. We MUST do this, no question, we’ll park our tanks EVERYWHERE’ he said. Gove strongly agreed. If they had not blown up this would have happened. The opposite impression was created because many Tories who did not like us talking about the NHS reverted to type within seconds of victory and immediately distanced themselves from it and the winning campaign. Unlike Gove and Boris they did not learn from the campaign, they did not listen to the public. Until people trust that the NHS is a financial priority for Tories, they will have no moral authority to discuss management issues. This obvious fact is psychologically hard to absorb because of the strength of gang feelings in politics.

(There are already myths about some of these events. The press conference of 24 June is now written up as the two of them ‘terrified of what they had done’ but this is completely wrong. They were subdued partly because they were genuinely sad about Cameron and partly because they did not want to be seen as dancing on his grave. Some of the media created the psychologically compelling story that they were regretful / frightened about victory but this was not at all their mood in HQ on the morning of 24 June. Boris came in punching the air like Maradona after a great goal, hugging staff and clearly euphoric. It is completely wrong to portray him as regretful.)

3. ‘Vote Leave to take back control of immigration policy. If we stay there will be more new countries like Turkey joining and you won’t get a vote. Cameron says he wants to “pave the road” from Turkey to here. That’s dangerous. If we leave we can have democratic control and a system like Australia’s. It’s safer to take back control.’

I was surprised at what a shock it was to IN when we hit them with Turkey. By the time this happened they were in an almost impossible position. I wanted them to announce a veto. It would not have been believed and would have had the opposite effect – people would have taken the danger of Turkey joining more seriously. If your life depended on winning for IN, the answer is clear: they should have said long before the campaign started as part of the renegotiation process that they would veto any accession.

4. ‘The euro is a nightmare, the EU is failing, unemployment is a disaster, their debts and pensions are a disaster, if we stay YOU will be paying the bills. It’s safer to take back control and have a new relationship based on free trade and friendly cooperation instead of the European Court being in charge of everything…’ (This is not an official text, just a summary of the notion off the top of my head.)

5. Anti-Establishment. E.g. We aligned our campaign with those who were furious with executive pay / corporate looting (about 99% of the country). We aligned ourselves with the public who had been let down by the system.

Mandelson regarded this as ‘sheer nerve, sheer chutzpah’. It was obvious. The hard thing was sticking to it despite the sensibilities of many of our own supporters. One of the most effective TV performances of the campaign was the day Boris hit the theme of corporate looting in a market square. No10 were rightly panicked and in response pushed out Heseltine a few hours later to make a very personal attack on Boris. This made sense tactically but was a strategic error. All such personal attacks helped persuade Boris to up the ante. This was vital with a month to go when the immigration figures came out. Rudd and others argue that Cameron should have attacked Boris and others more. Wrong. They should have played it Zen publicly and had a much better black ops team.

Cameron/Osborne mistakes

I’ll go into this separately but just to give a few examples…

1. Cameron never had to offer the referendum in the first place. His sudden U-turn was a classic example of how his Downing Street operation lurched without serious thought in response to media pressure, not because of junior people but because of Cameron himself and his terrible choice of two main advisers (Llewellyn and Oliver). This happened many times and I wrote about all the damage it caused on other issues after I left government (HERE). This was the biggest example. It was a product of a deeper error – a combination of his failure of party management (misleading them about the best way to handle the party) and failure to understand how swing voters really think and therefore the dangers of a vote (see below).

2. If Cameron/Osborne had had a top notch person like David Plouffe running their campaign and they did as they were told then they would have won (>95% confidence), all else being equal. They were warned many times by their closest friends about Oliver and Llewellyn, including by Gove, but would not listen.

3. Their renegotiation was flawed from the start and badly undermined their central message. They compounded their errors in 2015 by accepting the pathetic deal in 2016.  If they had walked away in February then Vote Leave would quickly have imploded and the flying monkeys would have taken over the campaign.

4. They made themselves too prominent in the campaign and were too crude. Lacking a feel for psychology they gradually undermined their own message. Oliver thought Obama’s ‘back of the queue’ was brilliant. It was counterproductive. They thought ratcheting up the warnings to DEFCON 1 was effective. It was counterproductive.

5. They doubled down on ‘tens of thousands’. They thought they would lose credibility if they didn’t. The opposite was true. They should have dropped this in 2015 – for example, in an exclusive to the Independent on a Saturday in early August 2015 – and gone into the campaign without it. Every time they defended it they were helping us.

6. They suckered themselves into over-prioritising their coalition versus message. Blair’s campaign against us in the North East did the same. When you do this you lose focus and clarity which is usually fatal. The error was perhaps most visible the day Cameron unveiled an absurd poster that effectively listed all the ‘serious people’ on their side and – creative genius! – a blank page for us. A total waste of valuable time. The fact of being the Government meant the broadcasters let them lead the news almost all the time but they often wasted it like this. (I would bet that that ad was never put in focus groups or if it was the results were ignored.)

7. One of my basic criticisms of Cameron/Osborne from the start was the way they steered by pundit. During the 2015 election Crosby partly corrected this and they partly learned the lesson. But left to their own devices in the referendum when under pressure they defaulted to their instincts at a crucial moment. The reaction to the dreadful murder was an example of how the media and SW1 can live effectively in a parallel universe. Somehow they convinced themselves that this event might undo over a decade of growing hostility for those in power. They therefore tried to push the theme that actually MPs are great, ‘they are in it for good reasons’ and so on. The media led themselves into a dead end and No10, defaulting to their instincts of steering by pundit, followed. As soon as I saw Osborne and Matt Hancock wasting their time tweeting broken multicoloured hearts and encouraging #weloveourMP, I knew they had screwed their own OODA loop. We knew from focus groups (conducted by the brilliant Henry de Zoete who also played a crucial role in coordinating the digital and data science teams) that opinion outside London was extremely different to that of MPs and those in charge of most news. We went straight back to what we knew were the winning messages leaving Hancock and co to tweet broken hearts.

BUT BUT… Roland Rudd and others have attacked them for their basic strategy of focus on the economy and argue there should have been ‘a positive campaign for the EU’. WRONG. Cameron and Osborne were right about this big call. There was not enough time or money to change basic attitudes. As the campaign developed and there were signs of pressure from Rudd and others I crossed my fingers and hoped they would shift strategy. No10 were right to ignore him.

I suspect that in general big mistakes cause defeat much more often than excellent moves cause victory. There are some theoretical reasons to suspect this is true from recent statistical analysis of human and computer decisions in chess. Two results are particularly interesting. 1) The very best computers seem to make moves that preserve  the widest possible choices in the future, just as the most effective person in politics for whom we have good sources, Bismarck, operated always on the principle of ‘keep two irons in the fire’. (We tried to mimic this by adopting a message that we thought had the highest probability of  winning in the largest number of plausible branching futures, hence £350m/NHS.) 2) Even great humans are distinguishable from great computers by their propensity to make clear tactical errors occasionally amid the fog of war. This is significant enough that it wipes out the advantage of going first – i.e. it being ‘your move’ is seen as a plus but in fact it is a minus for humans because of the probability of a significant error, while for computers this effect is absent. (See Human and Computer Preferences at Chess, 2014. It would be very interesting to know if these results are supported by the recent success of Deep Mind with computer GO.)

Summary of the false dichotomy

False: ‘Leave won because of the campaign.’ E.g. Without 15 years of out of control immigration, our message of ‘take back control’ would not have had enough traction. Campaigns can ride big waves but they almost never make them.

False: ‘Leave won because of a big event [immigration, 2008 crash etc], the campaign was irrelevant.’ E.g. If the campaign had not deployed £350 million and the NHS (which almost nobody on our side liked), we would not have neutralised/overwhelmed Project Fear.

True: ‘Leave won because 1) three big forces created conditions in which the contest was competitive, AND 2) Vote Leave exploited the situation imperfectly but effectively, AND 3) Cameron/Osborne made big mistakes. If just one of these had been different, it is very likely IN would have won.’

Overall, the now-mocked conventional wisdom that ‘the status quo almost always wins in referendums like this’ obviously has a lot of truth to it and it only proved false this time because of a combination of events that was improbable.


A ‘miracle’ to get 48%? Beaten by lies? Corbyn the AWOL saviour?

Since losing many inside the IN campaign now talk dejectedly as if they could never have won and tell rationalising fairy tales. They are wrong. They almost did win. Some have latched onto the idea that they were overwhelmed by an epic, global force of ‘right-wing populism’. Mandelson defends himself by saying  48% looks ‘like a miracle’ given the populist tide. Most have latched onto the idea that their ‘complex truth’ was overwhelmed by ‘simple lies’ and they are happy with their comforting ‘post-truth’ sobriquet – a delusion that leaves them very vulnerable to being shocked again. Many have even argued that they lost because they could not persuade Corbyn to make more speeches.

These stories are psychologically preferable to the idea that their own errors caused defeat (just as it is for some of those in Hilary’s campaign) but should not be taken seriously.

The least plausible claim is that Corbyn sabotaged what was otherwise a winning campaign. This is argued mainly by the same people (including Mandelson) who in a party context also argue that Corbyn is a joke who nobody takes seriously. The idea that more speeches by Corbyn would have persuaded vital swing voters has no good evidence. These people wanted to ‘take back control’. Corbyn’s message was – there should be not just more immigration but no limits on it. There are not many branching histories in which this is a winner.

This ‘epic global force’ of ‘populism’ was thought by the same people before 23 June to be puny in comparison with the force of the combined Establishment hammering a message of economic fear in support of the status quo. Having underestimated certain trends in public opinion the same people are now exaggerating them (see below).

This is connected to ‘complexity’. Month after month they argued (including to us in private discussions) that they would win largely because they had the advantage of the status quo – an advantage proved in votes around the world over many years. They were right. That was a big advantage. It is much simpler to argue for the status quo than for a very complex change – that is exactly why most ‘change’ referendums lose, just as they briefed the media. Now they say ‘The EU is very complex, it requires a lot of information to explain it’ (Craig Oliver). Their claim that actually they had the ‘complex’ argument to make against our ‘simple lies’ is laughable for exactly the reasons they gave themselves before they came unstuck.

Connected to this idea is that the great rationalists Cameron and Osborne – they of Project Fear and their comic ’emergency budget’ and in 2015 the pictures of Salmond picking pockets designed successfully to persuade the English that the Scots would steal their money – were undone by a great surge of ’emotion’. Osborne is taking this delusion so far he is writing a book titled ludicrously ‘The age of unreason’. When you lose and you blame it on millions of people being overtaken by ‘unreason’ – after previously winning by exploiting nationalist hostility – it’s a sure sign that you are the one not reasoning straight and able to face your errors. For the likes of Osborne it is ‘irrational’ to reject the views of people like him. For most of us, people like Osborne are not experts to be trusted – they are charlatans not to be taken seriously.

Many of those who blame defeat on ‘lies’, including Cameron, Osborne, and Clegg themselves told flat-out lies. One example will do. Cameron and Osborne claimed repeatedly on TV, almost always unchallenged, that their new deal meant ‘after six months if you haven’t got a job you have to leave’. This is not an argument over the fairness of using a gross/net figure, like ‘£350 million’, or even a properly bogus figure like the Treasury’s £4,000 per household figure. It is a different category of claim – a flat out 100% lie. (For more details see HERE.) How much time did TodayNewsnight, and the Guardian spend explaining to people that the PM and Chancellor were lying through their teeth? Approximately none. Why? Because very few of those complaining about lies really are cross about ‘lies’ – they are cross they lost and they are not so interested in discussing a lie that undermines the pro-EU campaign’s attempt to neutralise fear of immigration.

Further, many of the same people spent the entire campaign saying ‘Vote Leave has admitted a Leave vote means leaving the Single Market, this is what will happen make no mistake…’ and now say ‘the Single Market was not an issue, Vote Leave never had a policy on it and there is no mandate for leaving it’. Cameron, Osborne, Mandelson, Campbell and Clegg spent much of the last 20 years lying through their teeth to further their own interests and prestige. Now they whine about ‘lies’. They deserved worse than they got – and reasonable Remain-ers deserved better leadership.

Fools and knaves

Many of those who worked on the IN side are now wrongly attacked as fools by pundits who would have praised them as geniuses had they won, while many on the OUT campaign are wrongly praised.

Example… ‘If Remain wins Cameron ought to be hailed as the genius strategist of western democratic politics’ (Rentoul). Pundits who wrongly hailed Cameron as a genius after the 2015 election now wrongly describe him as a bumbling oaf. He was neither – he was the best of a bad bunch picked pseudo-randomly in a broken system and out of his depth. 600,000 votes either way does not make one set of people geniuses and another set of people morons. Geniuses in politics are rarer than in maths and physics and nobody involved in the referendum on either side is remotely close to one. Some of those who worked on the IN side were much more able than many on the winning side. It does not make sense to label people on the IN side idiots because of errors made by Cameron, Osborne, Llewellyn, and Oliver.

Example: many have said to me ‘you were so clever to hold back on immigration until the start of purdah’. Wrong. It is true that we did not do much on immigration before the 10 week official campaign. That is because, as I wrote in 2014, we did not need to. It was far more important to plant other seeds and recruit support that would have been put off if we had focused early on immigration. Immigration was a baseball bat that just needed picking up at the right time and in the right way. The right time was before purdah and we set in motion during January-April a series of things like the free referendum address with the right message but we could not persuade many prominent people to do what was needed until after 26 May. The right way was via the NHS (unifying) – not ‘we want our country back’ of Farage (divisive). The timing was not ‘a brilliant move’ by me, it was a combination of good luck and seizing a tactical chance to persuade people of something I’d failed to do for weeks, but such things get rewritten as such if you win.

It is also foolish to see the conflict in terms of who is ‘nicer’ and ‘nastier’. I don’t think the people on our side are nicer. There are lovely and loathsome people, liars and charlatans on both sides.

Many OUT-ers talk as if we were destined to win. Wrong. The IRA used to say ‘you have to get lucky every time but we only have to get lucky once’. For Leave to win a string of events had to happen many of which were independently improbable or 50-50 and therefore the combination was very improbable. The result was certainly not an inevitable outcome of ‘the great British public simply voting for democracy’ as some romantics delude themselves.


Oblonsky and the frogs before the thunderstorm: fashion, delusions of the educated, and the Single Market

‘I feel that, in some ways, this was a conflict between good forces in society and bad forces. I feel that the bad forces on 23 June won a very significant victory.’ Matthew Parris.

Matt Ridley: Matthew, you’re not saying that 17 million people are, deep down, racists? 

Matthew Parris: Yes. (Spectator, December 2016)

Why is almost all political analysis and discussion so depressing and fruitless? I think much has to do with the delusions of better educated people. It is easier to spread memes in SW1, N1, and among Guardian readers than in Easington Colliery.

Generally the better educated are more prone to irrational political opinions and political hysteria than the worse educated far from power. Why? In the field of political opinion they are more driven by fashion, a gang mentality, and the desire to pose about moral and political questions all of which exacerbate cognitive biases, encourage groupthink, and reduce accuracy. Those on average incomes are less likely to express political views to send signals; political views are much less important for signalling to one’s immediate in-group when you are on 20k a year. The former tend to see such questions in more general and abstract terms, and are more insulated from immediate worries about money. The latter tend to see such questions in more concrete and specific terms and ask ‘how does this affect me?’. The former live amid the emotional waves that ripple around powerful and tightly linked self-reinforcing networks. These waves rarely permeate the barrier around insiders and touch others.

These factors are deepened by the fact that almost all of those whose job it is to explain politics and campaigns have never been responsible for a complex organisation in general or a campaign in particular, so they are unsuited to understand how politics ripples out from decisions at the centre through dysfunctional bureaucracies to the ground. They almost always exaggerate the extent to which important decisions have been considered carefully by people who know what they are talking about. (The worse educated are actually often helped by their lack of education towards the truth.) They constantly discuss complex systems as though errors can be eradicated instead of asking how quickly errors are adapted to and learned from. This perspective biases them in favour of existing centralised systems that fail continually and against innovations with decentralised systems. They understand little about the challenges faced by small businesses and the lower middle classes.

The more closely involved people are in the media and politics the more they are driven by fashion and the feeling, rarely acknowledged and almost always rationalised, that ‘this is my gang’. Look at all those in SW1 who tweet attacks on Dacre to each other then retweet the praise from their friends, then look at those who attack them. Look at Robert Peston tweeting pictures of the London Eye and Habermas quotes on election night and his opponents ranting about ‘elites’. Both sides are just like football team fans defending their in-group and attacking their out-group enemies. The more they think of themselves as original the more likely they are to be conformist – and conformist within very narrow parameters.  We all fool ourselves but the more educated are particularly overconfident that they are not fooling themselves. They back their gang then fool themselves that they have reached their views by sensible, intelligent, reasoning.

This makes them particularly vulnerable to ‘influence operations’. It also makes them vulnerable to repeated errors about what the sort of people who ignore politics other than for a few weeks before voting time are thinking. It creates something of a paradox: it is almost impossible to get a good feel of public opinion, or of ‘the winning strategy’, by listening to those whose job it is to speculate about it. However often this happens, the lesson is never learned. It is very hard to see how it could change as it is so entangled with our evolved nature.

There is a wonderful passage in Anna Karenina that sums this up, much better than any ‘political scientist’ has done:

Oblonsky never chose his tendencies and opinions any more than he chose the style of his hat or coat. He always wore those which happened to be in fashion. Moving in a certain circle where a desire for some form of mental activity was part of maturity, he was obliged to hold views in the same way he was obliged to wear a hat. If he had a reason for preferring Liberalism to the Conservatism of many in his set, it was not that he considered the liberal outlook more rational but because it corresponded better with his mode of life… The Liberal Party said that marriage was an obsolete tradition which ought to be reformed, and indeed family life gave Oblonsky very little pleasure, forcing him to tell lies and dissemble, which was quite contrary to his nature. The Liberal Party said, or rather assumed, that religion was only a curb on the illiterate, and indeed Oblonsky could not stand through even the shortest church service without aching feet, or understand the point of all that dreadful high-flown talk about the other world when life in this world was really rather pleasant… Liberalism had become a habit with Oblonsky and he enjoyed his newspaper, as he did his after-dinner cigar, for the slight haze it produced in his brain.’

Towards the end of the novel, there is a discussion about the then big issue of Turkish atrocities and the rise of pan-Slavism. The old prince replies to the intellectuals who are talking rubbish about ‘the national feeling’ that they think is ‘sweeping the country’:

‘Yes, all the papers say the same thing. That’s true. So much the same that they are just like frogs before a storm! You can’t hear anything for their croaking.’

Many will reply, ‘Oblonsky is a dilettante, not a serious character, you can’t compare him with people like Robert Peston’. Oblonsky isn’t a dummy, he’s brighter than many of the posh duffers in his club. And also consider Anna’s husband, Karenin – a terrifying reminder that the ‘serious characters’ in politics are really no better than Oblonsky regarding fashion. In politics, just about all of us are some combination of Oblonsky and Karenin. If you think you aren’t, you’re probably fooling yourself. If you’re on TV a lot, you’re almost definitely fooling yourself.

There are many examples of how real Oblonskys, who control practically all important cultural institutions, think. They believed things about Stalin’s regime so outlandish that it is hard to appreciate now. They were more in favour of Britain joining the euro, not because they understood ‘the complexities’ better but because they were suckered into thinking about it as a moral test – are you on the side of the ‘baddies’ or the goodies’? As the BBC Europe editor said to me back then, in similar terms to Matthew Parris about the 2016 referendum, ‘the thing is Dominic, we like foreigners and cappuccinos and we hate racists’. Polls show that better educated people are less likely to have accurate views about the science of evolution and genetics (their desire to send moral signals suckers them into believing fairy tales).

The conformity of the educated is in some ways a good thing – most obviously, a basic consensus about things like not killing one’s domestic opponents that is extremely unusual historically. But it has many bad effects too. There is a collective lack of imagination which makes the system very susceptible to disastrous shocks. They share a narrow set of ideas about how the world works which mistakes their own view as the only possible sensible approach. They are aways writing about how ‘shocking’ things are to them – things that never were as low probability events as they imagine.  They can’t imagine something like Stalin deliberately creating a famine or deliberately murdering millions. They tell themselves that Hitler will be ‘more sensible in power’ and ‘engagement’ is the right path. Western liberals (like Clinton and many pro-euro campaigners) and conservatives (like Bush) talked of relations with Putin as if he is a normal western politician rather than an ex-KGB mafia overlord with views very far from western liberals. They tell each other ‘I can’t imagine President Trump, it just can’t happen’. Many conservatives are now telling themselves that they should not take Trump too literally but that too is a failure of imagination – his character is clear to those unblinded by gang mentality and he will govern in character.

The referendum was a great example of this. Large numbers of people better educated than average – the sort of people who work as producers at the BBC – talked about their vote like this:

‘Farage is racist, he hates gay people and made that comment about foreigners with HIV, he wants to turn the clock back and pull the drawbridge up, I’m not like that, my friends aren’t like that, I am on the other side to people like that, I am tolerant and modern, I will vote IN.’

All over the country sentiments almost identical to this were expressed in large numbers. The idea that millions of graduates voted because they ‘studied the issues’ is laughable to anybody who spent time measuring opinion honestly. Almost none of these people know more about what a Customs Union is than a bricky in Darlington. They did not vote on the basis of thinking hard about the dynamics of EMU or about how Brussels will cope with issues like gene drives. Millions thought – there’s two gangs and I know which one I’m in. Another subset of the better educated feared the short-term economic disruption of a Leave vote would cost them money. They also did not vote on the basis of deep consideration of the issues.

The modern day Oblonsky reads an op-ed about how ‘the CBI warns of the dangers of leaving the Single Market’ and ‘the dangers of racist extremists’ and, having no idea of what ‘the Single Market’ is, jabbers away at their dinner party about how concerned they are about leaving ‘the Single Market’, and a warm haze of knowing one is on the ‘good’ side of the argument envelops the brain.

When it comes to the central issues of the nature of the EU’s trading relationships and what a UK-EU relationship might look like outside the EU, we are dealing with a particularly strong example of this phenomenon. Not only do the Oblonskys not know what they are talking about, neither do almost any of the supposed experts and specialists.

Lots of people said to me ‘when are you going to set out the details of the UK-EU trade relationship if you win?’ What would have been the point of that?! Approximately nobody knows anything about the important details of how the EU works including the MPs who have spent years talking about it and the journalists who cover it – indeed, often those who talk about it most are the most ignorant (and most overconfident). This is still true six months after the vote – imagine how much more true it was in the six months before the vote.

I am not aware of a single MP or political journalist who understands the Single Market – its history, its nature, its dynamics, its legal system, the complex interactions between law, economics, business, history and so on. Cameron, Osborne and Clegg certainly don’t. Neither does Bill Cash. Neither does any head of the CBI. Neither do Jon Snow, Robert Peston, Evan Davis or John Humphreys so they do a rubbish job of exposing politicians’ ignorance.

The number of people who do is tiny. In our campaign there were two – Oliver Lewis and Richard Howell – who understood a large fraction of it and the common misconceptions. They constantly had to explain to MPs, MEPs, and journalists why their ideas were misunderstandings. Maybe there is a business/economics journalist somewhere who really understands it. There are certainly some exceptional lawyers who understand narrow aspects extremely well, though few of these also understand the political and business dimensions. I have spoken to many very successful business people and never met one on either side who understands the Single Market in depth. In the entire campaign I am not aware of a single programme on TV that even tried to delve into these issues seriously (Newsnight was particularly bad, combining smugness and vapidity such as dropping Evan Davis by helicopter on an offshore platform to babble about ‘sovereignty’ trying to make the Leave side look like a bunch of weirdo cranks). British elites handed over power to the Monnet-Delors project with barely one-in-a-thousand understanding in detail why, what it involved, and its likely evolution (and that  one-in-a-thousand almost all concluded that the public could not be trusted to know the truth – I’ll explore another time the ideas of this tiny group).

Further, it was clear that Cameron/Osborne intended to run a campaign based on hysterical warnings and bogus arguments/figures while ignoring the big questions about how the EU works and its trajectory. No10 tried to turn the whole complex issue into a question about whether the economy would grow a little bit slower over the next few years – a trivial issue relative to the significance of the overall question. They are not a duo who have ever engaged the public on a serious matter in a serious way. Their brains don’t work like that. They formed early habits of looking at everything through a very narrow prism of SW1 conventional political wisdom. Given this, the way the media works, how outnumbered we were among the influential broadcast media, and the way in which the media (inevitably to some extent) takes its lead from No10, why would I have tried to run a campaign based on educating normal people to a far higher level than the professionals and ‘experts’ who were fighting and covering the campaign? It would have been impossible to get even two sensible MPs to explain the same complex argument about such things on TV without cocking it up – it was hard enough to get people to say ‘let’s spend our money on our priorities’ without days of arguing. (If the vote had happened in 2017 and we’d had all that time to build sensibly more could have been done.)

We did try to get the media to focus on deeper questions of how the EU is run, its problems, its evolution and so on. We knew from the research that the more coverage of the EU, its powers, its record, its plans and so on the better for us. We had little to fear from serious policy discussion and much to gain. But we largely failed. (A big speech from Gove was turned by the Financial Times – yes, the FT that bemoaned the ‘low quality debate’ – into a story about whether he had ‘gaffed’ by mentioning Albania, though in plastering ‘Albania’ all over the place the FT accidentally helped us.) No10 calls up the BBC and says, ‘we’ve got a business letter tomorrow with dozens of household companies warning of Armageddon.’ If we published something worthy on the Eurozone’s debt and demographic nightmare, the structural problems of the Eurozone and implications of the Five Presidents’ Report, how far did this get? ‘Sounds boring. Who’s fronting it? Got any new names? Any chance of Boris putting the boot into Dave and George?’, is the first question from the BBC TV producer who has no interest in ‘the arguments’.

It was not in our power to change basics of how the media works. We therefore  twisted them to our advantage to hack the system.

Hack the medium, hack the message: ‘the alternative government’ launches Sunday 29 May

The media is obsessed with process and the snakes and ladders of careers. Many hacks said to me words to the effect: ‘I don’t care about the issues, I care about whether Cameron will still be PM at the end of the year.’ We could not match No10 in the golden currency of ‘names’. But we could give the media an even more valuable currency – a leadership story. When Boris and Gove decided to go for it after 29 May  immigration numbers, we launched the story of ‘the alternative government’.

The media were understandably obsessed with this story so we served it up to them in such a way that they also had to cover our message. For 10 days, we dominated the news with a set of stories on the Australian points system, VAT on fuel, Turkey, the NHS and so on all based on ‘it’s safer to take back control’. Broadcasters lapped it up – even ITV News which barely pretended to be impartial was useful.

What did the public hear? They heard that prominent Conservatives, particularly Boris and Gove, did not trust Cameron’s promises or warnings and that there was an alternative path – we could ‘take back control’, have ‘an Australian style immigration system’, and ‘spend our money on our priorities like the NHS’. In an environment in which the central arguments concerning trade and the economy were incomprehensible to the ‘experts’ themselves and the history and dynamics of the EU either unknown to or suppressed by broadcasters, people chose between two simple stories. Vote Leave’s was more psychologically compelling, given the three powerful forces at work and No10’s errors.

(NB. Whoever leaked the Hilary email story was probably doing something similar. This played into the media obsession with scandal and process such that they spent a ridiculous amount of time on it despite probably 80% of them wanting Hilary to win. It shows how powerfully the media is in the grip of dynamics they rarely reflect on themselves. Putin’s communication maestro, Surkov, uses these sorts of tricks all the time. Cf. Peter Pomerantsev’s great book, a must read for any MP before they pontificate on Putin’s mafia government.)


The political media and how to improve it

High prestige pundits and editors yield great power over the stories told (and have far more power over politicians like Cameron, unfortunately, than they realise) but the field is not based on real expertise. Fields dominated by real expertise are distinguished by two features: 1) there is enough informational structure in the environment such that reliable predictions are possible despite complexity and 2) there is effective feedback so learning is possible.

Neither condition applies generally to politics or the political media. In the most rigorous studies done, it has been shown that in general political experts are little better than the proverbial dart throwing chimp and that those most confident in their big picture views and are most often on TV  – people like Robert Peston, Jon Snow, and Evan Davis – are the least accurate political ‘experts’ (cf. HERE).

We know that cognitive diversity is vital for political accuracy yet almost all political institutions and the media – including the dominant people at Newsnight, the Economist, the FT, and Parliament – are actually remarkably homogenous, as discussed above, and they herd around very similar ideas about how the world works. Scientists and entrepreneurs in particular are almost totally excluded from political influence.

There is no structure to hold them to account either internally or externally so, like anyone when not forced to be rigorous, they fool themselves. It is normal to write month after month that the IN campaign cannot lose because of XYZ then just as confidently and authoritatively explain why IN lost without any intermediate step of identifying and explaining errors.

Despite the rise of social media most people get most of their news from TV. TV coverage of politics rarely illuminates much because there is no clear way to decide who is right about anything. The format makes it almost impossible for any useful discussion to happen. Interviewers, politicians, and pundits talk past each other with no clarity about assumptions. Questions are vague, often meaningless, posed by interviewers who rarely have more than a thin bluffer’s understanding of any policy issue and the same is usually true of those answering; the more famous the interviewer, the less likely it is they know anything about, say, education policy and like David Cameron they are bluffing. (As soon as a story is deemed ‘political’ it is taken out of the hands of specialists (who are very rarely actually specialists anyway) and given to ‘political’ hacks who have no idea of the policy.) Most of those professionally involved are much more interested in the ‘horse race’ political dimension than the policy. They obsess on process and scandal but most people have no interest in the process or ‘scandals’  because they assume ‘they’re all dodgy in some way’. Nobody tries to make predictions that can be checked and the shows don’t take what is said seriously enough to catalogue it. Simplistic stories compete so political analysis is dominated by endless false dichotomies.

Those making the shows do not understand how people learn so the dead format recycles grim clichés like Evan Davis saying ‘… economy down the plug hole’, while filming an actual plug hole, or Nick Robinson saying ‘… will the economy take off’ standing in front of a plane actually taking off (both of these have happened). Every night the News contains reports that are a mix of incomprehensible, facile, and boring to millions while also usually at best simplistic and often just wrong when it comes to policy / issues. The possibilities of the medium are largely ignored.

Insiders think of the masses as being irrational in paying so little attention to political debate. I think they are rational. If you want to understand politics you should read serious things and invest time and effort in researching public opinion. You should particularly make an effort to invert your point of view and consider opinions very different to your own. Time spent watching/listening to shows like Newsnight and Today is not just wasted – it is actively distorting reality and making you less informed. I often meet people who are cleverer than those in politics and successful but they have deluded views about politics because they pay too much attention to political analysis. Overall, unless you are professionally involved in politics you will be better off if you stop >95% active reading of political analysis. You will miss occasional worthwhile things but the effort of sorting them is not worth it. If something is genuinely very good / unusual and you have avoided isolating yourself in an echo chamber that insulates you from opinions very different to your own then someone reliable will send it to you. Even if you are professionally involved in politics I would do roughly the same. Extreme focus on important things you can control will repay far far more than time spent reading speculation about things you can’t control.

I read very little punditry during the campaign – just enough to preserve a sense of the gaps between the ‘croaking frogs’ and the real world. If I’d had less infighting to deal with I’d have read even less as I could have been less concerned about tracking certain things. In my entire time in the DfE (three years) I never listened to Today once (I listened to a handful of interviews on the web). I focused on managing priorities and saying ‘No, stop, that’s a waste of time’ every day.

This situation is particularly ironic because the media industry is in a panic about the internet, falling ad revenues and profits, the collapse of print journalism and so on.

A better way…

There is a better way.

Example 1. Shows should require precise quantitative predictions about well-formed questions as Superforecasters do. Newspapers should do the same when interviewing people. The next step is using this process to push people towards admitting conditional errors like ‘if I am proved wrong about X by date Y then I will admit I was wrong to claim Z’. If political shows pushed their guests to do this and kept track of the predictions it could have a big positive effect. (Next time they come on you can flash up their record on a screen so the public can see how often they are right.) It is vital to change incentives so people are encouraged to admit errors and learn instead of fooling themselves constantly. For those who refuse it would be easy to develop a protocol that categorises their vague comments and puts numbers on them. This will push them to ‘correct the record’.

Example 2. Rip up the format for political shows and base broadcasts on a) an empirical assessment of what people actually know and b) the science of how people really learn and how best to communicate. Instead of the tedious low-information interviews, imagine what could be done if one had a mix of artists, scientists, and policy specialists trying really hard to use the possibilities of film to explain things, then used cutting edge data science to test how effective they were as part of a learning cycle driving higher quality. A news broadcast now contains much less  information content and much higher noise than reading. The only way to improve this is experimenting with formats in a scientific way. Doing this would force those making the news to think more about policy and the audience would be much more engaged. People are interested in policy and ‘how X will affect me, my family, and my community’. It would also obviously require a lot of changes in the media but this is coming anyway because existing business models are blowing up.

Example 3. Pay for this partly by firing most of your political commentators like Dan Hodges. Broadcasters, fire 90% of your political correspondents. They are a waste of money. Hire a much smaller number of much better people with radically different skills and backgrounds and a different focus. (By doing #1 you will soon see who is more/less accurate so you’ll have a good benchmark.)

Together these changes could improve the quality by a factor of x10 or more. The same principle of focusing on precise quantitative predictions about well-formed questions could also be used to improve policy making and management of bureaucracies by developing clusters of well-formed questions that ‘surround’ a vaguer big question that is not so susceptible to measurement.

For example, break down ‘will Britain leaving the EU be a success?‘ into dozens of simpler more precise questions that can be quantified and which together give a useful part of an overall answer. This process could be put on a prediction platform for little money and dramatically improve the quality of decisions. The Superforecasters new consultancy could do this pretty simply with little help and not much money.

As usual in systems that are failing, the youngest people understand the problems and possibilities best while the most senior / prestigious figures can’t think of anything to do other than get overpaid for what they’ve always done. If you run a big media company, you should replace the expensive old schoolers like Jon Snow with  younger, cheaper, and brighter new schoolers with an extreme focus on the public, not SW1.


An example of a simple, powerful media story that is wrong and contributed to forecasting errors on Brexit – ‘the centre ground’

One of the most misleading stories in politics is the story of ‘the centre ground’. In this story people’s views are distributed on an X-axis with ‘extreme left’ at one end, ‘extreme right’ at the other end, and ‘the centre ground’ in the middle. People in ‘the centre’ are ‘moderate’. ‘Extremists’ are always ‘lurching’ while ‘sensible moderates’ are urged to ‘occupy the centre’.

This story is one of the dominant features of political discussion and the basis for endless interviews, columns, and attempts at political ‘strategy’. The story is deeply flawed and where it is not trivially true it is deeply misleading.

Swing voters who decide elections – both those who swing between Conservative/Labour and those who swing between IN/OUT – do not think like this. They support much tougher policies on violent crime than most Tory MPs AND much higher taxes on the rich than Blair, Brown, and Miliband. They support much tougher anti-terrorism laws than most Tory MPs AND they support much tougher action on white collar criminals and executive pay than Blair, Brown, and Miliband.

One of the key delusions that ‘the centre ground’ caused in SW1 concerned immigration. Most people convinced themselves that ‘swing voters’ must have a ‘moderate’ and ‘centre ground’ view between Farage and Corbyn. Wrong. About 80% of the country including almost all swing voters agreed with UKIP that immigration was out of control and something like an Australian points system was a good idea. This was true across party lines.

This was brought home to me very starkly one day. I was conducting focus groups of Conservative voters. I talked with them about immigration for 20 minutes (all focus groups now start with immigration and tend to revert to it within two minutes unless you stop them). We then moved onto the economy. After two minutes of listening I was puzzled and said – who did you vote for? Labour they all said. An admin error by the company meant that I had been talking to core Labour voters, not core Tory voters.  On the subject of immigration, these working class / lower middle class people were practically indistinguishable from all the Tories and UKIP people I had been talking to.

The media tried to categorise Vote Leave as ‘right wing’ while Tory MPs and Farage’s gang were screaming at me about our championing the NHS and our attacks on the indefensible pay of FTSE CEOs. SW1 did not understand our appeal but the crucial voters did because they do not think as the ‘experts’ think they think. We tried to speak to a majority in the country. Cameron and Osborne have never won even 40%. They approached it as they did previous battles but this greatly limited their appeal. Most UKIP and Tory voters (rather than MPs/insiders) agreed with us on the NHS and executive pay while also agreeing with us on the need to take back control of immigration policy from a system that has obviously failed. Our campaign was neither Left nor Right in the eyes of the crucial audience.

The media made a similar mistake with Trump. Trump did lots of things wrong and the post facto re-branding of his campaign as ‘brilliant’ is very silly. BUT he had a national message the core of which appealed to a big majority and which defied categorisation as Left/Right. Again the media do not realise this – they label it, like Vote Leave, as ‘populist right’ (abetted by some charlatan academics). But the reason why it is successful is exactly because it is not a simple right-wing message.

It doesn’t occur to SW1 and the media that outside London their general outlook is seen as extreme. Have an immigration policy that guarantees free movement rights even for murderers, so we cannot deport them or keep them locked up after they are released? Extreme. Have open doors to the EU and don’t build the infrastructure needed? Extreme. Take violent thugs who kick women down stairs on CCTV, there is no doubt about their identity, and either don’t send them to jail or they’re out in a few months? Extreme. Have a set of policies that stops you dealing with the likes of ‘the guy with the hook’ for over a decade while still giving benefits to his family? Extreme. Ignore warnings about the dangers of financial derivatives, including from the most successful investor in the history of the world, and just keep pocketing the taxes from the banks and spending your time on trivia rather than possible disasters? Extreme. Make us – living on average wages without all your lucky advantages – pay for your bailouts while you keep getting raises and bonuses? Extreme and stupid – and contemptible.

These views are held across educational lines, across party lines, and across class lines. Cameron, Blair, and Evan Davis agree about lots of these things and tell people constantly why they are wrong to think differently but to millions they are the extremists.

(This is not a post facto rationalisation. I wrote about the centre ground and the EU in 2014 HERE.)


Why I got involved and my role

Winning the referendum against Cameron was not the way I wanted things to happen. I thought the chances of winning a referendum against a PM on the other side, with all the possibilities for him to mobilise the system behind IN, were low. Many prominent Eurosceptics (not all) lobbied for it out of a combination of self-promotion and not knowing how to solve the real problem – what should the UK-EU relationship be? The referendum was very useful for many Out-ers: it provided a much simpler political focus than figuring out a complex positive agenda, removed the need for difficult thinking and action, and gave people a chance to pose on the side of ‘democracy’. I thought it foolish to push for a referendum while simultaneously not building a serious movement to win it. (I had tried to start building such a movement in 2004 after the euro battle was clearly won but could not persuade crucial people so decided to drop the issue for a while.) Romantic long shots are rarely wise in politics particularly if there is a better path.

I thought it wiser and safer to wait for Cameron to go then try to capture the Tory leadership and change the UK-EU relationship from Downing Street with someone who actually wanted to solve the problems (Cameron’s best friends would not claim that he wanted to spend his time trying to solve these deep problems, he wanted not to think about the EU and got into an existential battle he never wanted). If you are going to have a referendum, then have it when controlling the institutions and when you can set the agenda. A British PM could invite the EU to evolve such as to include a) those in the euro, Single Market and ‘free movement’, and b) those outside some or all of those three but with free trade and friendly cooperation between all. The chances are low that there would have been support for fundamental change but then a divorce could have happened after a serious clarifying debate which would have occurred ~2018-25, including the Eurozone countries figuring out what they would do. This would have been a much better way to proceed than the charade of Cameron’s ‘renegotiation’. Either Europe would have embraced a new and more open architecture (unlikely) or the Government would have won a Leave referendum with 60-70% and prompted a lot of clarifying thought across Europe.

I also thought it foolish of Cameron to cave into the pressure and promise a referendum in 2013. So did Gove and Osborne both of whom told Cameron not to do it. He mistakenly thought it would take the wind out of UKIP’s sails and did not understand why it would actually boost UKIP and Farage. (This was not hard to foresee and I suspect part of the problem was that Cameron did not appreciate that him promising a referendum would be thought by most as just a typical pre-election lie.) The idea that there was an irresistible force for a referendum is pushed by Farage’s and Cameron’s supporters. They are both wrong. The country supported one but without any passion outside the small fraction who had long been passionate about it. Most Tory MPs did not want it. Most Tory donors thought the timing was wrong and wanted a focus on stopping Miliband who they feared. Those MPs who did want it could mostly have been bought off or distracted in other ways – a mix of some policy, gongs, bribes, and so on in the usual fashion. Putting a date on the vote was particularly mistaken – it would have been far better to leave it open-ended ‘in the next Parliament’.

Once the election happened there was a sudden panic among OUT-ers. UKIP was an organisational disaster. There was no national campaign prepared. There were many tiny groups who often hated each other more than they wanted to win and were conditioned to expect failure and defeat. There was an abundance of people who thought that the campaign was quite simple – put me on TV, they thought, and the nation will appreciate my natural leadership. There was practically nothing of what was actually needed. Many quickly flipped into panic mode assuming the vote was unwinnable.

Having opposed the push for a referendum, I was faced with an uncomfortable choice in May 2015. Either keep out of politics, refuse to help, and then feel miserable about the tragicomic campaign, or re-engage with people I did not want to work with, feel miserable about the tragicomic campaign, and in almost every way make my life worse. In many ways irrationally, I chose the latter. My thinking was something like this: the chance of changing the whole political system (more profoundly than in a normal election) comes along very rarely, the chaos of the eurosceptics and the complacency of Cameron creates a very slim bridge to seize control and do it, a small chance of very high impact is worth the gamble. About a month or so later my wife was pregnant. If the timing had been slightly different I might well have stayed retired.

Why do it?

I thought that Leaving would improve the probability of 1) Britain contributing positively to the world and 2) minimising dangers. I thought it would:

  1. minimise Britain’s exposure to the problems caused by the EU;
  2. improve the probability that others in Europe would change course before more big crises hit, e.g. by limiting free movement which is the biggest threat to continued free trade;
  3. require and therefore hopefully spark big changes in the fundamental wiring of UK government including an extremely strong intelligent focus on making Britain the best place in the world for science and education;
  4. improve the probability of building new institutions for international cooperation to minimise the probability of disasters.

The foundation problem with the EU was best summarised by the brilliant physicist David Deutsch, the man who extended Alan Turing’s 1936 paper on computation into the realm of quantum mechanics. Deutsch said:

‘The EU is incompatible with Britain’s more advanced political culture. I’m voting Leave… [E]rror correction is the basic issue, and I can’t foresee the EU improving much in this respect… [P]reserving the institutions of error correction is more important than any policy… Whether errors can be corrected without violence is not a “concern” but a condition for successfully addressing concerns.’

Healthy and effective systems like our immune system and the English common law allow constant and rapid error-correction. Unhealthy and ineffective systems like the EU and modern Whitehall departments block error-correction. They are extremely centralised and hierarchical therefore information processing is blocked and problems are not solved. In politics this often leads to disasters when more and more resources are devoted to reinforcing failure. NB. This most fundamental question played effectively no role in the debate.

This fundamental problem generates its other problems. It arises because of how Monnet and Delors created its institutions deliberately in opposition to the Anglo-American system they bitterly opposed. The Foreign Office romantic delusion of ‘influence’ was peddled by every PM since Thatcher. Every one left office having demonstrated how empty the hope is. True influence comes from demonstrating success – not sitting in meetings for forty years in an institution that is programmed on principles that guarantee worse error-correction than the evolved institutions  of the Anglo-American system.

I will go into the problems of the EU another time. I will just make one important point here.

I thought very strongly that 1) a return to 1930s protectionism would be disastrous, 2) the fastest route to this is continuing with no democratic control over immigration or  human rights policies for terrorists and other serious criminals, therefore 3) the best practical policy is to reduce (for a while) unskilled immigration and increase high skills immigration particularly those with very hard skills in maths, physics and computer science, 4) this requires getting out of the EU, 5) hopefully it will prod the rest of Europe to limit immigration and therefore limit the extremist forces that otherwise will try to rip down free trade.

One of our campaign’s biggest failures was to get even SW1 to think seriously about this, never mind millions of voters. Instead the false idea spread and is still dominant that if you are on the side of free trade, think controlled immigration generally a positive force, and want more international cooperation rather than a return to competing nation states then you must support the EU. I think this error is caused by the moral signalling and gang mentality described above.

What was my role?

My role mainly involved:

a) trying to suppress/divert/overcome internal coalition warfare to a level where about ten crucial people were protected enough to do their jobs,

b) building the team,

c) management,

d) taking a small number of important decisions about policy, message, money, and the machine,

e) providing clear focus and priorities, including the vital job that nobody likes of saying ‘no’ to hundreds of people (thus making (a) harder), and

f) dealing with big problems.

The media tends to suggest my role was mainly talking to them. This is wrong. The same happened with my role in the DfE. In both projects my main role was management. Serious management means extreme focus and this requires saying No an awful lot. Contrary to the media story, I dislike confrontation and rows like most people but I am very strongly motivated by doing things in a certain way and am not motivated by people in SW1 liking me. This is often confused with having a personality that likes fighting with people. One of the basic reasons so much in politics is mismanaged is that so often those responsible are more interested in social relations than in results and unlike in other more successful fields the incentives are not structured to control this instinct.

Many have written that I got involved with this because of ‘hate’ or ‘loathing’ for Cameron. Wrong. I do not hate Cameron. I do not respect him, which is different. I thought that he was in politics for bad reasons – essentially because he was someone who wanted ‘To Be’, not someone who wanted ‘To Do’ (see the Colonel Boyd speech) and his priority was himself and a small gang, not the public. I also thought Cameron was mostly (not all) bad at the job, despite having some of the  necessary temperamental characteristics, and was flattered by having Brown then Miliband as opponents. I didn’t object to him blocking me from Government in 2010. He was entitled not to hire someone who did not take him seriously and ignored the orders of his Chief of Staff.

I spent a few years of my life (1999-2002) trying to stop Blair on the euro before anyone had heard of Cameron. In 2004 I co-founded the campaign that won the referendum on the North East Regional Assembly 80-20 as a training exercise for a possible future EU referendum. My motivation was the issue itself – not personal antipathy for Cameron or anybody else. I’ve never been a party person. I’m not Tory, libertarian, ‘populist’ or anything else. I follow projects I think are worthwhile.

Farage’s motley crew claim that I did this campaign in order to lose it deliberately then get a job in No 10 with Cameron. It is pointless to discuss this theory though the fact that they understood so little about the political environment, and struggled to use Google, was an important fact.

I am not clever, I have a hopeless memory, and have almost no proper ‘circle of competence’. I made lots of mistakes in the campaign. I have had success in building and managing teams. This success has not relied on a single original insight of any kind. It comes from applying what Charlie Munger calls unrecognised simplicities of effective action that one can see implemented by successful people/organisations.

Effective because they work reliably, simple enough that even I could implement them, and ‘unrecognised’ because they are hiding in plain sight but are rarely stolen and used. I found 10-15 highly motivated people who knew what they were doing and largely left them to get on with it while stopping people who did not know what they were doing interfering with them, we worked out a psychologically compelling simple story, and we applied some simple management principles that I will write about another time. It is hard to overstate the relative importance in campaigns of message over resources. Our success is an extreme example given the huge imbalance in forces on either side. In many ways Trump’s victory has little resemblance to what we did but in this respect he is another example.

We also got lucky.


I will post a number of blogs of the referendum to try to answer some basic questions including:

What were the main political, operational, financial/budgetary, and data/digital lessons from the campaign?

What worked and did not work?

How confident can we be about these judgements?

There is a natural set of categories and I will post links to blogs below:

  • Some basic numbers that summarise important elements.
  • Strategy, message, polls.
  • Policy.
  • Data and digital.

On data science, digital marketing, canvassing software made available for download. (NB. There has been some confusion about this blog. The VICS system is a web-based canvassing tool, the first proper one that works in the UK – it was one component of our overall data science approach and should not be equated with it. It is not a data science tool – it provided data to the data science team.)

  • The ground campaign.
  • The media.
  • Internal politics and the infighting.
  • Dynamics that affect ‘what next’.
  • The rules: how could they be improved to make future votes serve the public better?

Please leave comments and corrections below. I am happy to approve hostile comments if they have substance and will moderate comments to avoid putting sensible people off reading them.

On the referendum #20: the campaign, physics and data science – Vote Leave’s ‘Voter Intention Collection System’ (VICS) now available for all

‘If you don’t get this elementary, but mildly unnatural, mathematics of elementary probability into your repertoire, then you go through a long life like a one-legged man in an ass-kicking contest. You’re giving a huge advantage to everybody else. One of the advantages of a fellow like Buffett … is that he automatically thinks in terms of decision trees and the elementary math of permutations and combinations… It’s not that hard to learn. What is hard is to get so you use it routinely almost everyday of your life. The Fermat/Pascal system is dramatically consonant with the way that the world works. And it’s fundamental truth. So you simply have to have the technique…

‘One of the things that influenced me greatly was studying physics… If I were running the world, people who are qualified to do physics would not be allowed to elect out of taking it. I think that even people who aren’t [expecting to] go near physics and engineering learn a thinking system in physics that is not learned so well anywhere else… The tradition of always looking for the answer in the most fundamental way available – that is a great tradition.’ Charlie Munger, Warren Buffet’s partner.

During the ten week official campaign the implied probability from Betfair odds of IN winning ranged between 60-83% (rarely below 66%) and the probability of OUT winning ranged between 17-40% (rarely above 33%). One of the reasons why so few in London saw the result coming was that the use by campaigns of data is hard to track even if you know what to look for and few in politics or the media know what to look for yet. Almost all of Vote Leave’s digital communication and data science was invisible even if you read every single news story or column ever produced in the campaign or any of the books so far published (written pre-Shipman’s book).

Today we have made a software product available for download – Vote Leave’s ‘Voter Intention Collection System’ (VICS) – click HERE. It was named after Victoria Woodcock, Operations Director, known as Vics, who was the most indispensable person in the campaign. If she’d gone under a bus, Remain would have won. When comparing many things in life the difference between average and best is say 30% but some people are 50 times more effective than others. She is one of them. She had ‘meetings in her head’ as people said of Steve Wozniak. If she had been Cameron’s chief of staff instead of Llewellyn and Paul Stephenson had been director of communications instead of Oliver and he’d listened to them, then other things being equal Cameron would still be on the No10 sofa with a glass of red and a James Bond flick. They were the operational/management and communications foundation of the campaign. Over and over again, those two – along with others, often very junior – saved us from the consequences of my mistakes and ignorance.

Among the many brilliant things Vics did was manage the creation of VICS. When we started the campaign I had many meetings on the subject of canvassing software. Amazingly there was essentially no web-based canvassing software system for the UK that allowed live use and live monitoring. There have been many attempts by political parties and others to build such systems. All failed, expensively and often disastrously.

Unfortunately, early on (summer 2015) Richard Murphy was hired to manage the ground campaign. He wanted to use an old rubbish system that assumed the internet did not exist. This was one of the factors behind his departure and he decided to throw in his lot with Farage et al. He then inflicted this rubbish system on Grassroots Out which is one of the reasons why it was an organisational/management disaster and let down its volunteers. After Vote Leave won the official designation, many GO activists defected, against official instructions from Farage, and plugged into VICS. Once Murphy was replaced by Stephen Parkinson (now in No10) and Nick Varley, the ground campaign took off.

We created new software. This was a gamble but the whole campaign was a huge gamble and we had to take many calculated risks. One of our central ideas was that the campaign had to do things in the field of data that have never been done before. This included a) integrating data from social media, online advertising, websites, apps, canvassing, direct mail, polls, online fundraising, activist feedback, and some new things we tried such as a new way to do polling (about which I will write another time) and b) having experts in physics and machine learning do proper data science in the way only they can – i.e. far beyond the normal skills applied in political campaigns. We were the first campaign in the UK to put almost all our money into digital communication then have it partly controlled by people whose normal work was subjects like quantum information (combined with political input from Paul Stephenson and Henry de Zoete, and digital specialists AIQ). We could only do this properly if we had proper canvassing software. We built it partly in-house and partly using an external engineer who we sat in our office for months.

Many bigshot traditional advertising characters told us we were making a huge error. They were wrong. It is one of the reasons we won. We outperformed the IN campaign on data despite them starting with vast mounts of data while we started with almost zero, they had support from political parties while we did not, they had early access to the electoral roll while we did not, and they had the Crosby/Messina data and models from the 2015 election while we had to build everything from scratch without even the money to buy standard commercial databases (we found ways to scrape equivalents off the web saving hundreds of thousands of pounds).

If you want to make big improvements in communication, my advice is – hire physicists, not communications people from normal companies and never believe what advertising companies tell you about ‘data’ unless you can independently verify it. Physics, mathematics, and computer science are domains in which there are real experts, unlike macro-economic forecasting which satisfies neither of the necessary conditions – 1) enough structure in the information to enable good predictions, 2) conditions for good fast feedback and learning. Physicists and mathematicians regularly invade other fields but other fields do not invade theirs so we can see which fields are hardest for very talented people. It is no surprise that they can successfully invade politics and devise things that rout those who wrongly think they know what they are doing. Vote Leave paid very close attention to real experts. (The theoretical physicist Steve Hsu has a great blog HERE which often has stuff on this theme, e.g. HERE.)

More important than technology is the mindset – the hard discipline of obeying Richard Feynman’s advice: ‘The most important thing is not to fool yourself and you are the easiest person to fool.’ They were a hard floor on ‘fooling yourself’ and I empowered them to challenge everybody including me. They saved me from many bad decisions even though they had zero experience in politics and they forced me to change how I made important decisions like what got what money. We either operated scientifically or knew we were not, which is itself very useful knowledge. (One of the things they did was review the entire literature to see what reliable studies have been done on ‘what works’ in politics and what numbers are reliable.) Charlie Munger is one half of the most successful investment partnership in world history. He advises people – hire physicists. It works and the real prize is not the technology but a culture of making decisions in a rational way and systematically avoiding normal ways of fooling yourself as much as possible. This is very far from normal politics.

(One of the many ways in which Whitehall and Downing Street should be revolutionised is to integrate physicist-dominated data science in decision-making. There are really vast improvements possible in Government that could save hundreds of billions and avoid many disasters. Leaving the EU also requires the destruction of the normal Whitehall/Downing Street system and the development of new methods. A dysfunctional broken system is hardly likely to achieve the most complex UK government project since beating Nazi Germany, and this realisation is spreading – a subject I will return to.)

In 2015 they said to me: ‘If the polls average 50-50 at the end you will win because of differential turnout and even if the average is slightly behind you could easily win because all the pollsters live in London and hang out with people who will vote IN and can’t imagine you winning so they might easily tweak their polls in a way they think is making them more accurate but is actually fooling themselves and everybody else.’ This is what happened. Almost all the pollsters tweaked their polls and according to Curtice all the tweaks made them less accurate. Good physicists are trained to look for such errors. (I do not mean to imply that on 23 June I was sure we would win. I was not. Nor was I as pessimistic as most on our side. I will write about this later.)

VICS allows data to be input centrally (the electoral roll, which in the UK is a nightmare to gather from all the LAs) and then managed at a local level, whether that be at street level, constituency or wider areas. Security levels can be set centrally to ensure that no-one can access the whole database. During the campaign we used VICS to upload data models which predicted where we thought Leave voters were likely to be so that we could focus our canvassing efforts, which was important given limited time and resources on the ground. The model produced star ratings so that local teams could target the streets more likely to contain Leave voters.

Data flowed in on the ground and was then analysed by the data science team and integrated with all the other data streaming in. Data models helped us target the ground campaign resources and in turn data from the ground campaign helped test and refine the models in a learning cycle – i.e. VICS was not only useful to the ground campaign but also helped improve the models used for other things. (This was the point of our £50 million prize for predicting the results of the European football championships, which gathered data from people who usually ignore politics – I’m still frustrated we couldn’t persuade someone to insure a £350 million prize which is what I wanted to do.) In the official 10 week campaign we served about one billion targeted digital adverts, mostly via Facebook and strongly weighted to the period around postal voting and the last 10 days of the campaign. We ran many different versions of ads, tested them, dropped the less effective and reinforced the most effective in a constant iterative process. We combined this feedback with polls (conventional and unconventional) and focus groups to get an overall sense of what was getting through. The models honed by VICS also were used to produce dozens of different versions of the referendum address (46 million leaflets) and we tweaked the language and look according to the most reliable experiments done in the world (e.g. hence our very plain unbranded ‘The Facts’ leaflet which the other side tested, found very effective, and tried to copy). I will blog more about this.

These canvassing events represented 80-90% of our ground effort in the last few months, hence some of the reports by political scientists derived from Events pages on the campaign websites, which did not include canvassing sessions, are completely misleading about what actually happened (this includes M Goodwin who is badly confused and confusing, and kept telling the media duff information after he was told it was duff). There was also a big disinformation campaign by Farage’s gang, including Bone and Pursglove, who told the media ‘Vote Leave has no interest in the ground campaign’. This was the opposite of the truth. By the last 10 weeks we had over 12,000 people doing things every week (we had many more volunteers than this but the 12,000 were regularly active). When Farage came to see me for the last time (as always fixated only on his role in the debates and not the actual campaign which he was sure was lost) he said that he had 7,000 activists who actually did anything. He was stunned when I said that we had over 12,000. I think Farage et al believe their own spin on this subject and were deluded not lying. (Obviously there was a lot of overlap between these two figures.) These volunteers delivered about 70 million leaflets out of a total ~125 million that were delivered one way or another.

While there were some fantastic MPs who made huge efforts on the ground – e.g. Anne Marie Trevelyan – it was interesting how many MPs, nominally very committed to Leave, did nothing useful in their areas nor had any interest in ground campaigning and data. Many were far more interested in trying to get on TV and yapping to hacks than in gathering useful data, including prominent MPs on our Board and Campaign Committee, some of whom contributed ZERO useful data in the entire campaign. Some spent much of the campaign having boozy lunches with Farage gossiping about what would happen after we lost. Because so many of them proved untrustworthy and leaked everything I kept the data science team far from prying eyes – when in the office, if asked what they did they replied ‘oh I’m just a junior web guy’. It would have been better if we could have shared more but this was impossible given some of the characters.

VICS is the first of its kind in the UK and provided new opportunities. It is, of course, far from ideal. It was developed very quickly, we had to cut many corners, and it could be improved on. But it worked. Many on the ground, victims of previous such attempts, assumed it would blow up under the pressure of GOTV. It did not. It worked smoothly right through peak demand. This was also because we solved the hardware problem by giving it to Rackspace which did a great job – they have a system that allows automatic scaling depending on the demand so you don’t have to worry about big surges overwhelming the system.

There were many things we could have done much better. Our biggest obstacle was not the IN campaign and its vast resources but the appalling infighting on our own side driven by all the normal human motivations described in Thucydides – fear, interest, the pursuit of glory and so on. Without this obstacle we would have done far more on digital/data. Having seen what is offered by London’s best communications companies, vast improvements in performance are clearly possible if you hire the right people. A basic problem for people in politics is that approximately none have the hard skills necessary to distinguish great people from charlatans. It was therefore great good fortune that I was friends with our team before the campaign started.

During the campaign many thousands of people donated to Vote Leave. They paid for VICS. Given we spent a lot of money developing it and there is nothing equivalent available on the market and Vote Leave is no more (barring a very improbable event), we thought that we would make VICS available for anybody to use and improve though strictly on the basis that nobody can claim any intellectual property rights over it. It is being made available in the spirit of the open source movement and use of it should be openly acknowledged. Thanks again to the thousands of people who made millions of sacrifices – because of you we won everywhere except London, Scotland and Northern Ireland against the whole Government machine supported by almost every organisation with power and money.

I will write more about the campaign once the first wave of books is published.

PS. Do not believe the rubbish peddled by Farage and the leave.EU team about social media. E.g. a) They boasted publicly that they paid hundreds of thousands of pounds for over half a million Facebook ‘Likes’ without realising that b) Facebook’s algorithms no longer optimised news feeds for Likes (it is optimised for paid advertising). Leave.EU wasted hundreds of thousands just as many big companies spent millions building armies of Likes that were rendered largely irrelevant by Facebook’s algorithmic changes. This is just one of their blunders. Vote Leave put our money into targeted paid adverts, not buying Likes to spin stories to gullible hacks, MPs, and donors. Media organisations should have someone on the political staff who is a specialist in data or have a route to talk to their organisation’s own data science teams to help spot snake oil merchants.

PPS. If you are young, smart, and interested in politics, think very hard before studying politics / ‘political science’ / PPE at university. You will be far better off if you study maths or physics. It will be easy to move into politics later if you want to and you will have more general skills with much wider application and greater market value. PPE does not give such useful skills – indeed, it actually causes huge problems as it encourages people like Cameron and Ed Balls to ‘fool themselves’ and spread bad ideas with lots of confidence and bluffing. You can always read history books later but you won’t always be able to learn maths. If you have these general skills, then you will be much more effective than the PPE-ers you will compete against. In a few years, this will be more obvious as data science will be much more visible. A new interdisciplinary degree is urgently needed to replace PPE for those who want to go into politics. It should include the basics of modelling and involve practical exposure to people who are brilliant at managing large complex organisations.

PPPS. One of the projects that the Gove team did in the DfE was funding the development of a ‘Maths for Presidents’ course, in the same spirit as the great Berkeley course ‘Physics for Presidents’, based on ideas of Fields Medallist Tim Gowers. The statistics of polling would be a good subject for this course. This course could have a big cultural effect over 20 years if it is supported wisely.

On the referendum #9: Cameron begins his renegotiation, the Commission sets out its timetable for new Treaty pre-2025, BJ & SJ make moves, a Greek ‘no’

A few thoughts on developments over the past week or so…

1. No Treaty change before the referendum. On Thursday 25th at the start of the EU Council, it emerged that Cameron officially dropped the idea of the EU treaties being changed before the referendum. His pledge that there will be ‘legally binding’ promises by the other 27 members to change the EU treaties in certain ways a few years in the future is a useful development for the NO campaign. No such promise will be believed regardless of the choreography. A future EU Treaty can be vetoed by any member and some members will also require a referendum. Nobody can guarantee in advance that a new Treaty will be agreed at all or on what terms, as the EU has found a few times already. A promise before the end of 2017 to change the treaties at some point in the future is the political equivalent of ‘the cheque’s in the post, and it will be paid in a few years time if 28 people still agree to pay it’. The NO campaign will be able to say simply, ‘If you trust all these politicians’ promises vote YES, if you suspect they may be lying as usual, vote NO to get a better deal.’ Polls will show strong distrust.

2. Trivial substantive demands from Cameron. The Guardian leak on Friday confirms how little Cameron is asking for. Do people in No10 really think that deleting phrases like ‘ever closer union’ and having the EU formally say ‘OK we won’t force you to join the euro’ would persuade people that the EU has fundamentally changed?! DC’s approach so far has been to send Llewellyn and Liddington around asking foreign governments ‘what should we ask for that you can give us?’ Unsurprisingly, this approach to negotiations is seen by other countries as consistent with Cameron’s lack of understanding of how EU business is done, as the Monnet-ist Foreign Office officials also ruefully acknowledge. There is no sign that the long-standing desire of Open Europe for a deal whereby Britain remains in the EU and Single Market but is outside all non-Single Market stuff is on the table or that No10 is pushing for it to be on the table.

3. The Commission plans its new Treaty to ‘complete’ Economic and Monetary Union before 2025. Meanwhile, as Cameron plays his role in the re-enactment of Wilson’s 1975 deceit, the Commission has its own timetable. It will be more influential than Britain’s. The Commission has, since Monnet, seen disasters as ‘beneficial crises’ – the answer to a crisis is always ‘more Europe’ (meaning ‘more centralised bureaucracy’). This was true after 9/11 and after the Madrid bombings. It was true after the 2008 financial crisis. It is true again now with the Greek crisis and the immigration crisis in the Mediterranean.

This paper, ‘Completing Europe’s Economic and Monetary Union’, was published on 22 June 2015. It got little coverage in the UK media. It was written by ‘the five presidents’: European Commission President Jean-Claude Juncker; the President of the Euro Summit, Donald Tusk; the President of the Eurogroup, Jeroen Dijsselbloem; the President of the European Central Bank, Mario Draghi; and the President of the European Parliament, Martin Schulz.

It sets out three stages for the ‘completion’ of Economic and Monetary Union by 2025, building on what it thinks is ‘a credible and stable currency’ to create a financial union, a fiscal union, and a political union, in three stages:

  • Stage 1) using existing treaties to push further including: single bank supervision, single bank resolution, single deposit insurance, a European Deposit Insurance Scheme (EDIS) at the European level, ‘further measures … to address the still significant margin for discretion at national level’ for bank regulation, a Capital Markets Union to ‘strengthen cross-border risk-sharing through deepening integration of bond and equity markets’, ‘harmonisation of accounting and auditing practices … addressing the most important bottlenecks preventing the integration of capital markets in areas like insolvency law, company law, property rights’, and a ‘system of Competitiveness Authorities … in charge of tracking performance and policies in the field of competitiveness’.
  • A White Paper in spring 2017 to set out the transition to Stage 2 outlining the legal measures needed to ‘complete’ EMU, following explicitly the model of the Delors White Paper of 1985 which paved the way to the Single European Act.
  • Stage 2) ‘a common macroeconomic stabilisation function’ to ‘improve the cushioning of large macroeconomic shocks’, maybe built on the European Fund for Strategic Investments, and a European Fiscal Board to ‘coordinate and complement the national fiscal councils that have been set up in the context of the EU Directive on budgetary frameworks’.
  • Stage 3) completion of EMU will involve further major steps towards a political union including ideas such as a ‘euro area treasury’ and unified external EU representation on international financial institutions such as the IMF.
  • This will require a new Treaty. It will also ‘require Member States to accept increasingly joint decision-making on elements of their respective national budgets and economic policies’.

Dominic Lawson’s column also shows the dismissive attitude of the German Europe Minister to Cameron’s renegotiation. Cameron has long had what is to me a baffling hope that Merkel both wants to and is able to solve all his European problems. This delusion is similar to many other unrealistic delusions about the EU held in the Foreign Office and Downing Street over the decades.

4. The EU Council dinner witnessed a huge row over the migrant crisis. Although the migrant crisis has been in the news, it is perhaps not appreciated just how much concern it is causing across Europe. (The terrorist attacks also diverted media attention.) The situation is dire, it will get worse as thousands of Africans head north for the coast, it is obvious that the EU’s institutions and laws cannot cope, signatories to Schengen are starting to introduce informal measures that are strictly illegal (e.g. French police checking papers of people coming from Italy contra the Schengen rules), and nobody can agree on what to do. Worried diplomats have described the shouting at the dinner as some of the worst scenes seen in decades. It seems likely that the Italian government will be forced to declare some sort of emergency state and start ignoring EU and ECHR law to a much greater extent than previously. This will create all sorts of dynamics that affect the UK referendum.

5. Boris backs a ‘NO’ vote and a second referendum to get a better deal. In a previous blog in this series, I discussed the issue of exit plans and a second referendum. According to a story by Shipman in the Sunday Times, Boris Johnson, after reading this blog, is considering that this may be the best path for him to take:

‘Boris Johnson is preparing to call for a “no” vote in Britain’s referendum on the European Union in an attempt to extract greater concessions from Brussels than David Cameron is demanding.

In a stance that puts him on a collision course with the prime minister, the mayor of London believes Britain should reject any deal Cameron puts forward because the EU will not give enough ground.

Johnson has told friends that a “no” vote is desirable because it would prompt Brussels to offer a much better deal, which the public could then support in a second referendum.

Johnson said: “We need to be bold. You have to show them that you are serious.”

The mayor’s views, shared with friends last week, will send shockwaves through Downing Street. Both the “yes” and “no” camps had assumed that he would support Cameron in arguing for Britain to vote yes.

Johnson made the comments after reading a blog by Dominic Cummings, the former Tory aide who is organising the “no” campaign, in which he argued that Eurosceptics should say: “If you want to say ‘stop’, vote no and you will get another chance to vote on the new deal.”

A friend of the mayor said: “I don’t think in his heart Boris wants us to walk away. But he’s interested in us saying no because it won’t be what we want. That would mean a second vote. He thinks the only way to deal with these people is to play hardball.”’

A Guardian story on Monday said that BJ sources confirmed the Shipman story and Forsyth’s Spectator story similarly confirmed it.

Since I blogged about this idea, many people have got in touch.

A. It seems likely to many people that a NO vote would have to be followed by a second referendum on a new deal because the scale of importance of the UK-EU agreement, dwarfing the issues in normal general elections, would require giving people a vote.

B. It is clear that escaping the supremacy of EU law enshrined in the 1972 European Communities Act will be a complicated process stretching over years – it will not be a simple event. A NO vote in the first referendum would not, as a matter of fact or law, mean we had left the EU or would immediately leave. It would in practice be a rejection of Cameron’s deal and a direction from the public for a new government team to negotiate a new deal.

C. This issue is entangled in the Conservative Party leadership campaign. Some leadership candidates will like the idea of a second referendum – it allows them to position themselves against Cameron’s deal without committing themselves to OUT.

They will be able to say, ‘David Cameron has got a bad deal that does not solve our problems on immigration or anything else, he’s wasted the historic opportunity handed him on a plate by the euro crisis and migrant crisis to negotiate a completely different European system, and if we vote NO we can get a better deal, we finally have a chance to do this properly’ etc.

D. If it becomes clearer that a NO vote will mean a second referendum on a new deal, then the probability of NO winning is likely to rise.

5. Sajid Javid tells the CBI that they are undermining efforts to reform the EU. SJ has previously said that leaving the EU is not something to be afraid of. SJ gave a speech to the CBI this week in which he said:

‘I heard that the CBI thinks the UK should remain in the European Union no matter what. That the people of Britain should vote to stay in regardless of whether or not the Prime Minister wins the concessions that British business so badly needs… [D]oes it really make sense to say, so early in the process, that ‘the rules of this club need to change, but don’t worry – we’ll always be members no matter what’?

‘You know how negotiation works. You wouldn’t sit down at the start of a merger or acquisition and, like a poker player showing his hand to the table, announce exactly what terms you were prepared to accept. It doesn’t work in the boardroom and it won’t work in Brussels.’

SJ is right. Polls have shown for over a decade that most businesses regard the costs of the EU and the Single Market as greater than the gains and want many more powers brought back than Cameron is now asking for (e.g. ICM, April 2004).

SJ could have added a historical lesson for the CBI about its long record of being wrong on big issues. Its forerunner advocated appeasement in the 1930s with the old ‘stability’ argument wheeled out. The CBI played an important role in pushing Britain into the disaster of the ERM. It tried to play an important tole in pushing Britain into the euro which would have been a disaster.

Fortunately, businesspeople like Stanley Kalms and Michael Edwardes formed Business for Sterling (which I worked for 1999-2002). We surveyed British businesses and proved that the CBI was lying about business opinion and was systematically cheating its own membership surveys to give the false impression to the FT and BBC that ‘British business overwhelmingly wants the euro’ – sound familiar? In fact big businesses were split and small businesses were hostile by about 2:1.

However, the power of the UK Government and the EU Commission makes it extremely hard for senior FTSE people to speak out against the EU while they get brownie points by backing the EU (cf. Branson who still speaks in support of Britain entering the euro). Many businesses were told in 1999 – if you support Business for Sterling, we will screw you. The same thing is happening now. Few journalists understand the politics of company boards whereby pro-EU people are licensed to speak out while anti-EU people are told to pipe down to avoid causing blowback.

Within a year of starting, by January 2000 we had forced the CBI to withdraw from the euro campaign.  Meanwhile the IOD and FSB were clearly hostile to the euro.

31 March 1999, Daily Telegraph: BfS/ICM poll showed business opposition to euro

2015-06-06 17.32.55

31 January 2000, FT: CBI withdraws from euro campaign

2015-06-06 17.35.25


The CBI is now arguing that Britain should stay in the EU on any terms. This view is out of whack with the general view of British businesses but the cabal that controls the CBI has never cared about this and the BBC has very rarely challenged them.

The CBI has also just announced that Cridland will be replaced by a former ‘head of strategy’ at the BBC and ITV. Mike Rake said, without apparent irony, that she has ‘an impressive background as an economist, journalist, management consultant and policy strategist’. The CBI represents hired managers, management consultants, lawyers etc – it has never represented successful entrepreneurs. It is always controlled by a small number of politically powerful multinational firms (generally run by non-entrepreneur hired managers) that can be crucified by the Commission. This is why they are not taken seriously as the ‘voice of British business’ other than, unfortunately, by the BBC.

The people who control the CBI should consider 1999. Unless the CBI changes its position, 1999 will be a picnic compared to 2016.

6. Business for Britain has serialised a big report on the economics of the EU which will be published in full shortly.

7. The Greeks have voted NO. Those in the Commission, Eurostat, and other EU institutions who colluded with Goldman Sachs and others to cheat the numbers to ease Greece into the euro have got away with it. The euro financial system was set up so that a lot of bankers made a lot of money out of artificially low Greek bond prices. What about when the music stops? IBGYBG (‘I’ll be gone, you’ll be gone’ as assholes in the City say to each other when ripping off their clients/customers.) Those like Mandelson who predicted the euro would be great for Greece still have their huge pension pots paid for by taxpayers and are invited on TV to pontificate about the EU, largely un-reminded by the BBC of their previous duff predictions.

There are deep problems with the global financial architecture, from China’s shadow banking system to the recurrent flash crashes driven by high frequency algorithmic trading. There are deep problems with the euro financial architecture. Since 2008 global debt has increased enormously. There has been a huge distortion of debt markets with investors holding massive quantities of government bonds that offer very little future reward and great future risk. CDO’s, CDS’s, all sorts of synthetic credit derivatives that contributed to the 2008 crisis are back and being sold to idiots who don’t understand them by some of the same people who used such complicated scams to cheat the figures for Greece’s euro entry. Bureaucrats keep bailing out financiers. The public quite rightly rages that ‘us idiots on PAYE are bailing all these crooks out’. Politicians largely ignore them. In Britain, Cameron even defended the indefensible non-dom rules and has done nothing about the grotesque abuse of executive pay by hired managers paying themselves as if they are successful entrepreneurs, with institutional shareholders happily pushing the merry-go-round and getting their kickbacks. Everywhere one looks one sees insiders ripping off the public and politicians either colluding or helpless spectators.

The EU system is, characteristically, not admitting its own terrible errors that have contributed to the destruction of the Greek economy. Even the IMF has told the EU that Greece’s debts are unsustainable and will need a haircut. But the EU leaders feel they cannot face this reality because it would lead to an explosion of demands from Spain, Portugal, and Italy. Instead of facing reality, Europe’s leaders have decided to turn the disaster into a morality play in which ‘the lazy Greeks’ are blamed for  everything. Meanwhile, Brussels’s real answer is the Five Presidents Report (above) – deeper and further integration because, after all, as Delors said the whole point of doing EMU before political integration was that the problems with the former would force the creation of the latter, like Athena from the head of Zeus.

(I have written elsewhere about the fundamental problem that the 21st century global system has become too complex for traditional states with traditional bureaucracies to cope with, and the need for very different kinds of political institution and very different training for political decision-makers. In other fields, there is innovation: for example, JSOC and UK special forces have completely changed how they operate to cope with networked enemies. In politics, there is almost no innovation partly because incentives are set up that reward people for failing conventionally, a political equivalent to ‘nobody gets fired for buying IBM’.)


The combination of no Treaty change and no substantial demands is no surprise to many. The EU timetable always made Treaty change practically impossible before the end of 2017 (other than an Article 48 minor change to lower order things, which NB. could allow DC to claim ‘I’ve got treaty change’). However, Cameron’s position since his Bloomberg speech has relied on teasing the media, his MPs, donors and others that he intended to fight for treaty change to show he was ‘serious’.

He has decided to drop this pretence at the start of the process. His approach since the election has helped rally support for the NO campaign as it becomes increasingly clear that his talk of ‘fundamental change’ was just to keep people onside before the election. Some Conservative Party donors, who have suspected for a long time that the Cameron-Llewellyn team does not know how to negotiate, can see there is no serious attempt to reshape our membership. This is particularly striking given that the combination of the Greek crisis and the African/Med migrant crisis is pushing the EU itself to consider a new Treaty and new arrangements for the non-euro countries.

This approach is also unpopular with some of Cameron’s spads in No10 and Cabinet ministers who think that Llewellyn – a diehard pro-euro/EU campaigner – is harming the prime minister with his approach. Some of them think that rushing it also means rushing the day of DC’s departure as the leadership campaign will effectively start after the referendum. Others have pointed out that rushing so transparently to get trivial changes is hardly the best way to win a YES vote – or to maintain good faith in, and good will for, the prime minister. So far Llewellyn is ignoring such criticism – if he hears it, which is perhaps unlikely given how No10 works. No10 spads are keen to stress to journalists that ‘Ed is completely in charge of this, my responsibilities are domestic only if you know what I mean’.

On the Referendum #7: Transparency for our Potemkin government – Memo to ministers and spads thinking about how you could help the NO campaign

There have been many attempts to quantify the extent to which EU law (primary, secondary, Regulations, Directives, ECJ judgements etc) really determines what happens in the UK. It is inherently hard to come to an agreed answer given the combination of a) the sheer scale and complexity of EU law’s entanglement with domestic law over decades including things like domestic court interpretations of ECJ judgements, b) different definitions of regulation and the units of measurement, c) the desire of the civil service to obscure the issue, and so on.

You – ministers and spads – can contribute something valuable to this debate in a way that will help the NO campaign at a crucial time.

For those not in government reading this… One of the basic mechanisms of government is the ‘Cabinet write round’ system. This involves Secretaries of State being given lots of documents every night in their box from other departments. The SoS is supposed to read these documents and tick the relevant box on the attached form signalling assent, disagreement, comments etc. (When I find a copy of one in my papers I’ll post a photo.)

For entirely domestic things, this process can lead to disagreement and negotiation. An interesting aspect of our membership of the EU is that a large fraction of the documents concerning future law and administrative action come from the EU. For reasons that are opaque, the civil service continues with the write round system. It is, of course, a Potemkin system as ministers do not have a real power to oppose anything – the document in question will become law regardless of how the minister fills in the chitty. Still, the chitties are sent around so everybody can pretend they are in charge. This is a depressing process for some ministers but perhaps the Cabinet Office regards it as a Pavlovian exercise – ministers become habituated to simply tick everything without engaging their brains or ethics.

When occasionally a SoS refuses, the first step is the Private Office asks whether a mistake has been made. No? Are you sure minister? Off the chitty goes to the Cabinet Office (‘very courageous minister’). Step 2 is that the Cabinet Office emails to say – ‘Was your SoS drunk again last night, he seems to have rejected the EU Directive on XXX, better go and tell him to withdraw his objection pronto or Jeremy [Heywood] will be on it.’ This is normally enough to get SoS scuttling to retract his objection. Stage 3 is unusual – it involves the SoS not giving in at Stage 2. What happens then is that the SoS is informed by the private office that Ed Llewellyn has said that the Prime Minister agrees with Jeremy and insists on measure X. This flattens practically all objections. I have witnessed the very unusual Stage 4 – the SoS sends back a message asking for a meeting with Jeremy. Jeremy arrived. ‘This is EU law so there is no basis for us to object.’ Gove: ‘Why do we get sent these stupid forms to fill out then if we can’t stop these awful things, this is going to waste hundreds of millions of pounds for nothing?’ Jeremy [a chuckle]: ‘Haha, yes, so I’ll inform the Prime Minister that you agree after all, we will mention to European officials that ministers have grave concerns, I’m sure Oliver will look at it further, goodbye Michael.’ Game Over: ‘All your base belong to us’, as the old video game said…

The fairy tale that Britain still has Cabinet government involves maintaining this Potemkin process.

I have asked No10 spads a few times over the years what proportion of things they see come from the EU. The estimates have been 50-60%. When I was in the DfE, I would occasionally do surveys of Gove’s box, going through every single paper in it, to see the proportion of EU stuff. I would estimate the same – typically about half, though sometimes much more (though obviously volume does not equate to importance).

With the referendum coming, this will be an important question. The usual surveys will not answer the question. So what could you do?

From now, start collecting stats on a daily basis of the proportion of EU stuff in the box. Spad, create a GoogleDoc – obviously do not use the official system – so that the minister can simply fill in the box on the grid for each day. For the minister (or you) to jot ‘x%’ each day and fill in a GoogleDoc grid will take no more than a few seconds per day, less than a minute per week, less than an hour in a year. (If they are technically hopeless just get him to jot a figure and you fill it in.)  Also, you could take a few photos of some of the boxes (‘This one was 80% EU stuff’) and save them to Dropbox for future use. Keep copies of the 1% most stupid, irrational, and wasteful things. Add ECHR/HRA stuff too – that is all relevant particularly given Cameron is going to do nothing at all about the Charter of Fundamental Rights (NB. this is the EU thing, not the ECHR). No officials will know you are doing it. Neither Heywood nor Llewellyn will be able to know you are doing it.

After Cameron returns from the EU proclaiming triumph and some of you resign, you will then have a record of contemporaneously collected stats on the real importance of EU affairs in Government. You will be able to publish this. It will be recorded over a year or so and therefore have hundreds of data points. Much more than other surveys on this question, people will take it seriously – particularly when you explain it at a press conference holding up some photos and copies of the most stupid documents. It will be impossible for No10 to rebut it effectively. They will not be able to publish documents that could refute it. Heywood will give a statement saying that your claims are wrong but nobody will believe him.

This is a simple thing that could have a significant impact at the right time. You all know how much EU stuff is hidden by Whitehall and how much effort goes into pretending that ministers decide things that were really decided by some lobbyist in a Brussels hotel years ago. You know these Kafka-esque bureaucratic processes, redolent of the dying days of the Austro-Hungarian Empire, that characterise modern Whitehall. DO SOMETHING ABOUT IT!

Lots of you now won’t know whether you are going to resign but you can do this without anybody knowing so you have something useful if you do decide to resign; if you don’t you can delete it all, no harm done. Boris, we know you read this blog, you could do the same thing in the Mayor’s office and surely there will be some committees Cameron puts you on shortly to try to keep you quiet…

Please suggest ideas about how to improve this process,


On the Referendum #5: reports of an anti-EU advertising campaign; Greece, the euro, and predictions

The Sunday Telegraph reports that a group of businessmen, including Arron Banks (the UKIP donor), plan a £20 million advertising campaign in September as part of an effort to win the referendum on the EU.

Brief thoughts…

Contrary to some phone calls I’ve had, this is not an example of ‘Eurosceptic infighting’.

Now, there are no spending limits as the Bill has not gone through Parliament. If a group of rich businessmen want to use this period to spend money persuading people of the problems of the EU and how Britain can do better, good luck to them. The important question is: does their campaign have the right messages so it is persuasive?

The Exploratory Committee that was announced last week is not the NO campaign. As I explained on Friday (here), it is a vehicle to coordinate discussions, raise money etc so that a professional NO campaign can exist. I am talking to people about how this can best be done, raising money, and trying to persuade appropriate people to leave their jobs to do this campaign.

As I have said hundreds of times over the past few weeks and will say thousands of times in the next few months, in order to win the referendum many people with very different views will have to find ways to cooperate. Libertarians, socialists and others have to find common ground. Also, all sorts of people and groups will, quite reasonably, want to do their own thing.

It is understandable that in the absence of an official NO campaign, motivated businesspeople are looking to do useful things. My concern is building the foundations of an official NO campaign in the right way such that it can grow into what will be an unprecedented organisational network over the next year. Scale and complexity require organisational innovations.

Greece, the euro, prediction, accountability

On a different subject, Greece and the euro is much in the news… When I was working on the campaign against Britain joining the euro, we did many debates/events/TV shows etc with people like Adair Turner, Ken Clarke, Heseltine, Peter Mandelson, Chris Patten et al.

Our businessmen, such as Stanley Kalms and Simon Wolfson, argued that the euro had been badly constructed and would cause problems for the existing members particularly Greece and Ireland. Turner, Clarke et al breezily wafted away such fears and said we would be proved wrong.

Almost the only extended conversation I have had with Ed Llewellyn, David Cameron’s ‘chief of staff’, was in a restaurant around 2002 when, I think, he was working for Ashdown. He of course said that the euro was a great idea, would work out brilliantly, and we would inevitably join. He is leading the No.10 renegotiation team.

As people who follow this blog know, one of the themes I have explored a lot is the issue of predictions in politics. Physics is so successful because it has an architecture for correcting errors of prediction. Politics has lacked this. Tetlock’s Good Judgement Project (with IARPA) is the most interesting project I know of to inject rigour into the issue of political prediction in order to improve performance radically.

Now that a referendum is coming, Clarke, Heseltine and others are all over the BBC making predictions about the ‘chaos’ and ‘loss of jobs’ that would come from leaving the EU. Because politics does not operate on the basis of being held accountable for predictions, they are almost never asked anything like – ‘but given your false predictions on the euro, why should we have confidence in your predictions on the EU, perhaps you simply have an emotional attachment to the EU that is not susceptible to evidence?’ In politics, ‘Bayesian updating’ is not fashionable particularly when moral signalling is so strong. Many in the BBC see the EU debate, as they saw the euro debate, simply as ‘internationalists v racists’ which makes them even less inclined to challenge people like Ken Clarke who is routinely allowed to make factually wrong assertions without challenge on the Today programme. I blogged about this in an earlier blog in this series HERE.

In comments below, please leave the best examples of quotes from the likes of Clarke, Turner, Mandelson etc along the lines of ‘don’t worry about Greece and Ireland, the euro will be great for them’. 


The Hollow Men II: Some reflections on Westminster and Whitehall dysfunction

Mistah Kurtz—he dead.

A penny for the Old Guy

We are the hollow men

We are the stuffed men

Leaning together

Headpiece filled with straw. Alas!

Our dried voices, when

We whisper together

Are quiet and meaningless

As wind in dry grass

Or rats’ feet over broken glass

In our dry cellar

Shape without form, shade without colour,

Paralysed force, gesture without motion…

… Between the idea

And the reality

Between the motion

And the act

Falls the Shadow…’

The Hollow Men, T.S. Eliot.


“Tiger, one day you will come to a fork in the road,” he said. “And you’re going to have to make a decision about which direction you want to go.” He raised his hand and pointed. “If you go that way you can be somebody. You will have to make compromises and you will have to turn your back on your friends. But you will be a member of the club and you will get promoted and you will get good assignments.” Then Boyd raised his other hand and pointed another direction. “Or you can go that way and you can do something – something for your country and for your Air Force and for yourself. If you decide you want to do something, you may not get promoted and you may not get the good assignments and you certainly will not be a favorite of your superiors. But you won’t have to compromise yourself. You will be true to your friends and to yourself. And your work might make a difference.” He paused and stared into the officer’s eyes and heart. “To be somebody or to do something. In life there is often a roll call. That’s when you will have to make a decision. To be or to do. Which way will you go?” Colonel ’60 second’ Boyd.


‘You’re a mutant virus, I’m the immune system and it’s my job to expel you from the organism.’ DfE official re Gove’s team.


Are you fed up with the Hollow Men in charge of everything and do you want to change things more than the three party leaders do? I am and I do.

There are three parts to this blog.

Part I: My overall argument

Part II: Four stories

Part III: Analysis


Part I: ‘A combustible mixture of ignorance and power’

1. Complexity makes prediction hard.

Our world is based on extremely complex, nonlinear, interdependent networks (physical, mental, social). Properties emerge from feedback between vast numbers of interactions: for example, the war of ant colonies, the immune system’s defences, market prices, and abstract thoughts all emerge from the interaction of millions of individual agents. Interdependence, feedback, and nonlinearity mean that systems are fragile and vulnerable to nonlinear shocks: ‘big things come from small beginnings’ and problems cascade, ‘they come not single spies / But in battalions’. Prediction is extremely hard even for small timescales. Effective action and (even loose) control are very hard and most endeavours fail.

At the beginning of From Russia With Love (the movie not the book), Kronsteen, a Russian chess master and SPECTRE strategist, is summoned to Blofeld’s lair to discuss the plot to steal the super-secret ‘Lektor Decoder’ and kill Bond. Kronsteen outlines to Blofeld his plan to trick Bond into stealing the machine for SPECTRE.

Blofeld: Kronsteen, you are sure this plan is foolproof?

Kronsteen: Yes it is, because I have anticipated every possible variation of counter-move.

Political analysis is full of chess metaphors, reflecting an old tradition of seeing games as models of physical and social reality. A game which has ten different possible moves at each turn and runs for two turns has 102 possible ways of being played; if it runs for fifty turns it has 1050 possible ways of being played, ‘a number which substantially exceeds the number of atoms in the whole of our planet earth’ (Holland); if it runs for eighty turns it has 1080 possible ways of being played, which is about the estimated number of atoms in the Universe. Chess is merely 32 pieces on an 8×8 grid with a few simple rules but the number of possible games is much greater than 1080.

Kronsteen’s confidence, often seen in politics, is therefore misplaced even in chess yet chess is simple compared to the systems that scientists or politicians have to try to understand, predict, and control. These themes of uncertainty, nonlinearity, complexity and prediction have been ubiquitous motifs of art, philosophy, and politics. We see them in Homer, where the gift of an apple causes the Trojan War; in Athenian tragedy, where a chance meeting at a crossroads settles the fate of Oedipus; in Othello’s dropped handkerchief; and in War and Peace with Nikolai Rostov, playing cards with Dolohov, praying that one little card will turn out differently, save him from ruin, and allow him to go happily home to Natasha.

‘I know that men are persuaded to go to war in one frame of mind and act when the time comes in another, and that their resolutions change with the changes of fortune…  The movement of events is often as wayward and incomprehensible as the course of human thought; and this is why we ascribe to chance whatever belies our calculation.’ Pericles to the Athenians.

2. Science and markets have reliable mechanisms for coping with complexity.

In two previous blogs (Complexity and Prediction, HERE), I explored some of the reasons why and how science and markets have developed institutional mechanisms for error-correction that allow the building of reliable knowledge and some control over this complexity. Market institutions allow decentralised experimentation amid astronomical complexity and evolutionary processes allow learning in a way similar to the learning of biological immune systems. Science has built an architecture that helps correct errors and normal human failings. For example, after Newton the system of open publishing and peer review developed. This encouraged scientists to make their knowledge public, confident that they would get credit. Experiments must be replicated and scientists are expected to provide their data honestly so that others can test their claims, however famous, prestigious, or powerful they are. The legendary physicist Richard Feynman described the process in physics as involving, at its best, ‘a kind of utter honesty … [Y]ou should report everything that you think might make [your experiment or idea] invalid… [Y]ou must also put down all the facts which disagree with it, as well as those that agree with it… The easiest way to explain this idea is to contrast it … with advertising.’

When the institutional architectures of science and markets are working normally, they display self-correction at the edges of the network – they do not require wise chiefs at the top to decide and fix everything. Catching errors, we inch forward ‘standing on the shoulders of giants’ as Newton put it.

3. Politics lacks reliable mechanisms for coping with complexity.

This progress in science and markets contrasts with ‘political experts’ and their predictions as explored in Tetlock’s cutting-edge research, sadly ignored in Westminster, and the failures of prediction in economics (see this previous blog). There is an obvious gulf between a) our ability to solve certain narrowly defined problems in science and the ability of markets to solve certain types of problem and b) our ability to make accurate political predictions and solve social problems. The extraordinary progress with the former has occurred largely without affecting the ancient problems of the latter.

The processes for selecting, educating, and training those at the apex of politics are between inadequate and disastrous, and political institutions suffer problems that are very well known but are very hard to fix – there are entangled vicious circles that cause repeated predictable failure.

A) The people at the apex of political power (elected and unelected) are far from the best people in the world in terms of goals, intelligence, ethics, or competence.

B) Their education and training is such that almost nobody has the skills needed to cope with the complexity they face or even to understand the tools (such as Palantir) that might help them. Political ‘experts’ are usually hopeless at predictions and routinely repeat the same sorts of errors without being forced to learn. While our ancestor chiefs understood bows, horses, and agriculture, our contemporary chiefs (and those in the media responsible for scrutiny of decisions) generally do not understand their equivalents, and are often less experienced in managing complex organisations than their predecessors.

C) Government institutions (national and international) within which they operate, and which select people for senior positions, tend to have reliably poor performance compared with what we know humans are capable of doing. Westminster and Whitehall train people to fail, predictably and repeatedly. The EU and UN do not have the effectiveness or legitimacy we need for international cooperation.

In The Hollow Men I, I set out a long view of the failure of British elite decision-making since the 1860s. In 2014, it is particularly appropriate to consider the fact that during the entire period of 1906-1914, the British Prime Minister, Foreign Secretary, and the senior military leaders had one proper meeting (23 August 1911) to discuss the interaction of foreign and military policy, and in particular what Britain would do in various scenarios involving a German invasion of France via Belgium, and the unresolved issues from this meeting were left hanging until disaster struck in July 1914. This failure echoed the failure to consider these issues properly in 1870 and it echoed again in the late 1930s. Given how shattering for civilisation World War I was, how the most senior people took decisions in the preceding crises now seems almost beyond comprehension, particularly if one studies the details.

Their equivalents today are making similar mistakes. All parties and the media are locked into a game that to outsiders is obviously broken – a set of implicit rules about the conduct of politics, and definitions of effective action, that tie them to behaviour that seems awful to the public, which is objectively failing, but from which they cannot free themselves. Because the system is stuck in a vicious circle – held in place by feedback loops between people, ideas, and institutions – whatever the outcome of the next election, the big problems will remain, No10 will continue to hurtle from crisis to crisis with no priorities and no understanding of how to get things done, the civil service will fail repeatedly and waste billions, the media will continue obsessing on the new rather than the important, and the public will continue to fume with rage.

In this blog, I expand on these problems. It is long and few will be interested in the twists and turns but  those who want to understand the detail of why Westminster and Whitehall do not work will, I hope, find it useful even if they strongly disagree.

4. Traditional politics collides with markets and technology: ‘a combustible mixture of ignorance and power.’

We therefore face a profound mismatch between the scale of threats and the nature of our institutions.

a) The spread of markets and science increases the reach of technology and is driving a series of profound economic, cultural, political, and intellectual transitions, such as the spread of machine intelligence, massive increases in resource requirements, two billion Asians joining the global economy, another two billion born soon living mainly in new cities (but very mobile), the ‘internet of things’ with ubiquitous connected sensors, the mobile internet, drones, genetic engineering, and so on. These transitions already are and will continue to disrupt all institutions and traditional beliefs.

b) Traditional politics over six million years of hominid evolution involved an attempt to secure in-group cohesion, prosperity and strength in order to dominate or destroy nearby out-groups in competition for scarce resources.

c) Our civilisation now depends on science and technology underlying complex interdependent networks in the economy, food, medicine, transport, communications and so on. The structure (topology) of these networks makes them fragile and therefore vulnerable to nonlinear shocks.

d) Markets and technology enhance the power of individuals and small groups (as well as traditional militaries and intelligence agencies) to inflict such shocks in the physical, virtual, or psychological worlds. Technology can inflict huge physical destruction and help manipulate the feelings and ideas of many people (including, sometimes particularly, the best educated) through ‘information operations’. Further, technology makes it easier to do these things potentially without detection which could render conventional deterrence obsolete.

There is therefore a mismatch between a) the growing reach of technology and the fragility of our civilisation, and b) the quality of elite decision makers and their institutions’ capacity to cope with these technologies and fragilities. Carl Sagan called this mismatch ‘a combustible mixture of ignorance and power’. If this mismatch persists, if we continue to pursue ‘traditional politics’ in the context of contemporary civilisation, it will sooner or later blow up in our faces. We will not keep catching breaks such as Hitler scuppering the Nazi nuclear programme or wriggling through the Cuban Missile Crisis. A.Q. Khan has spread nuclear technology far and wide and many of those who worked on the Soviet biowar programme (which so shocked everyone when it became public) disappeared after 1991. (* See endnote)

My essay explores many of these dangers. This blog HERE summarises some of them. I will return to this.

5. We need new education, training, and institutions such as ‘artificial immune systems’.

We need A) to select, educate and train people differently. I have suggested in particular that we need what Murray Gell Mann, the discoverer of the quark, calls ‘an Odyssean education‘ that integrates knowledge from maths and science, the humanities and social sciences, and training in effective action. For a sketch of what this might involve, look at the reading list at the end of my essay.

We need B) new institutions, such as artificial immune systems, that enable decentralised coordination to tackle hard problems much more effectively than existing institutions are capable of doing. We need institutions that i) help markets and science continue to bring dramatic improvements and ii) help us take decisions better so that we can 1) foresee and avoid some disasters, 2) turn some disasters into mere problems, and 3) adapt effectively to the disasters and problems we cannot avoid. Alternatives to the EU and UN are vital if we are to develop the international cooperation on big problems that we need.

We also need institutional change to allow a re-organisation of expert attention on important problems. Academia and markets are not aiming the most able people at our biggest problems. For example, sucking a huge proportion of the cleverest and most expensively educated people in the world into high-frequency algorithmic trading (in which, for example, advanced physics is used to calculate relativistic effects that bring nanosecond trading advantages) is an obvious extreme mismatch between talent and priority. Michael Nielsen has written brilliantly about the potential for technological and incentive changes to transform this situation. When struggling with General Relativity, Einstein caught a big break – his friend Grossman introduced him to ideas in non-Euclidean geometry that were needed for Relativity. The restructuring of expert attention – ‘a scientific social web that directs scientists’ attention where it is most valuable’ (Nielsen) plus data-driven intelligence – will enable a transition from the haphazard serendipity of ‘Grossman moments’ to ‘designed serendipity’.

Underlying both A and B, I have suggested C) a new national goal and organising principle. After 1945, Dean Acheson famously quipped that Britain had lost its empire and failed to find a new role. I suggest that this role should focus on making ourselves the leading country for education and science: Pericles described Athens as ‘the school of Greece’, we could be the school of the world. This would provide an organising principle for a new policy agenda and focus resources. It would give us a central role in building the new international institutions we need. It would require and enable fundamental changes to how the constitution, Parliament, and Whitehall work (for example, embedding evidence in the policy process). Because it is a noble goal that reflects the best in human nature, it is something that can help transcend differences and mobilise very large efforts (though it is no panacea and education increases some problems). We already have a head start. We lack focus, perhaps the hardest thing to hold in politics.

This could help us make progress with a necessary transition from (i) largely incompetent political decision-makers making the same sort of mistake repeatedly and wasting vast resources while trying to ‘manage’ things they cannot, and should not try to, ‘manage’, to (ii) largely competent political decision-makers who embed some simple lessons, grasp what it is reasonable to attempt to ‘manage’ and have the ability to do it reasonably well while devolving other things and adapting fast to inevitable errors.

There is a telling example of institutional change. From the middle of the 19th century, the Prussian army established the ‘General Staff’ and a new training system, complete with wargames and honest ‘Red Teams’ to analyse performance. Unfortunately for the world, this coincided in 1862 with Roon manoeuvring into power someone with skills in the political sphere equivalent to Newton or Einstein in the scientific sphere – the diabolical genius Otto von Bismarck. The world changed very rapidly. British and French institutions could not cope. Fortunately, both in 1914-18 and 1939-45 the operational superiority of this machine was undermined by Bismarck’s successors’  profound blunders.

This shows the dangers we face. (Do we want China’s version of the General Staff to dominate?) It also shows how we could improve the world if we build similarly effective training systems in the service of different goals and ethics.

I will also return to this: What Is to Be Done and How?

Can we change course? There is a widespread befuddled defeatism that nothing much in Westminster can really change and most people inside the Whitehall system think major change is impossible even if it were necessary. This is wrong. Change is possible. We do not have to live with the permanent omnishambles that we have become acclimatised to. Monnet created the EU by exploiting crises – sometimes nothing happens in decades, and sometimes decades happen in weeks. Big changes are possible if people are prepared.


Part II: Four stories

A preface to these remarks.

1. Obviously there are many great officials. I made many mistakes and was saved from the consequences of them usually by quiet calm capable women aged 23-35 paid a fraction of the senior management, and without whom the entire DfE, and probably most of Whitehall, would collapse. Also, the DfE has changed for the better in many ways since 2010 so don’t take the atmosphere of early 2011 as a reflection of the atmosphere now, particularly since all but one of the senior people are different.

My point is not ‘the DfE / Whitehall is filled with rubbish people’ – it is that Whitehall is a bureaucratic system that has gone wrong, so that duff people are promoted to the most senior roles and the thousands of able people who could do so much better cannot because of how they are managed and incentivised, hence lots of the best younger people leave and the duffers are promoted. I have been encouraged to explain the problems by many great officials particularly younger ones who are fed up of watching the farces that recur in such predictable, and avoidable, ways.

2. My role in DfE. Most of my job was converting long-term goals into reality via policy, operational planning, and project management. This requires focus on daily, weekly, monthly, and quarterly steps, and management to make sure people are doing what is needed to get there. (My most valuable experience was not in politics but in small businesses / startups in England and Russia that taught me about organisational dynamics and management amid ‘the fog of war’.) It is interesting that of the 12 tasks listed in the spad Code of Conduct, the things that took up by far the most of my time do not appear. The Code clearly regards spads as almost 100% party creatures but I spent almost no time on anything to do with party matters. Nick Hillman (former spad) describes three roles: ‘policy wonks; spinners; and bag carriers’. Although I spent a lot of time on policy, none of these categories covers the project management that took up most of my time. (This is not criticism of NH but just to point out that there is obviously no agreement or clarity about spad roles.) I usually got involved in communications stuff only if it involved something big and bad. Overall communications took up less than 1% of my time because I regarded it, for reasons explained elsewhere, as almost entirely a waste of time given the management of No10.

My main purpose here is as explained above. It is not to defend what we did in the DfE which I will discuss separately. It does, however, provide context for debate about ‘the Gove reforms’, including our methods, and it shows the scale of problems that Gove personally had to cope with. I would prefer not to have to be critical of individuals such as Cameron and Llewellyn (and I have named very few individuals) but it is necessary for these things to be discussed openly if things are to improve.

Four stories

Story 1. Day 1. Bedlam, a sign of things to come…

My first day in the DfE was in January 2011. Between 8ish and 11ish, roughly every half hour officials knocked on the spad office door and explained a new cockup – we had accidentally closed an institution because we’d forgotten to renew a contract, the latest capital figures briefed to the media were out by miles, a procurement process had blown up, letters had gone out with all the wrong numbers in them (this happened maybe monthly over the three years I was there), and so on – meanwhile people were trying to organise the launch of the National Curriculum Review in documents full of typos and umpteen other things were going wrong simultaneously. It seemed extraordinary at the time but soon it was normal.

At about 11, I walked into Michael’s office to go through some of these horrors with him. While he was talking, I noticed on the TV behind him (muted) words scrolling across the bottom of the BBC News 24 screen – something like ‘New disaster as Gove announces XXX…’ (I can’t remember the XXX.)

Me [pointing]: Michael, we just agreed we weren’t going to announce anything else, we’re going dark until we get a grip of this madhouse, what the…

MG [turning to stare at the screen]: I haven’t authorised any new announcement and certainly not that. I haven’t a clue what they’re on about.

Me: Arghhh.

For the first few months, all sorts of things spewed from the Department causing chaos. The organisation was in meltdown. Everything that could go wrong went wrong. It was often impossible to distinguish between institutionalised incompetence and hostile action. Things were reported as ‘Gove announces…’ that he did not even know about, never mind agree with. Then pundits and bloggers would spin to themselves elaborate tales of how the latest leak was ‘really’ deliberate spin, preparing the ground for some diabolical scheme. (I would guess that <5% of the things people thought we leaked actually came from us – maybe <1%.)

From that day for over a year, about every 2 hours, officials would knock at our door bearing news of the latest cockup, disaster, leak, and shambles, all compounded with intermittent ‘ideas for announcements’ from Downing Street. The last one would be at about 9ish on Friday evening – thump, thump, thump down the corridor, the door opens, ‘Dominic, bad news I’m afraid…’ One measure of ‘success’ was that the frequency of episodes fell from hourly towards a few per day, then daily, then, by the last quarter of 2012, a few days with nothing important obviously blowing up.

For all of these problems, Gove was held ‘responsible’. With all of them, regardless of how incompetently they had been handled – nobody was ever fired.

Story 2. Maxwell’s Demon, correspondence, and the DfE’s lifts.

For the first year of Gove’s time in the DfE (May 2010 – spring 2011), ministers were up until the early hours proofreading officials’ drafts of letters and rejecting about nine out of ten because of errors with basic facts, spelling, or grammar. When I got embroiled in rows about this in Q1 2011, some MPs had been sent no reply for six months. Despite several complaints to senior officials, nothing happened, shoulders were shrugged – ‘cuts, we need more resource, lack of core skills, all very difficult’ and so on.

This problem was only (partly) solved when we insisted that the five most senior officials in the DfE including the Permanent Secretary had to start proofreading all ministerial letters themselves. ‘What? I can’t waste my time doing this.’ ‘Well right now all the ministers are so you’ll have to until you sort it out.’ This persuaded the Permanent Secretary to take more serious action though it remained the case that a) the correspondence team could not reliably answer letters with the right information, correctly spelled, without errors, and b) the Permanent Secretary admitted that this was due to ‘basic skills deficiencies’ in the Department. (It’s better now but it still isn’t right.)

Similarly the DfE’s lifts were knackered from the start and still are. There were dozens of attempts to have them fixed. All failed. At one point the Permanent Secretary himself took on the task of fixing the lifts, so infuriated had he become. He retired licking his wounds. ‘It’s impossible, impossible!’ It turned out that fixing an appointment is much easier than fixing a lift.

Given this failure over four years (and counting), people should reflect on the wisdom of constantly demanding ‘the DfE must do X to solve Y’. One of the most interesting psychological aspects of Whitehall is that their inability to fix their own lifts in no way dents their confidence in advocating that they manage some incredibly complicated process. If one says, ‘given we’ve failed to fix the bloody lift in four years, maybe we should leave X alone’, they tend to look either mystified or as if you have made a particularly bad taste joke.

There is a famous problem in physics first formulated in the 19th century known as Maxwell’s Demon. Maxwell, one of the handful of the most important scientists in history, asked whether the application of intelligence (an intelligent ‘demon’) could allow an escape from the inexorable increase in entropy mandated by the Second Law of Thermodynamics. It was an extremely subtle problem and took about a century to vanquish (the answer is No, intelligence cannot provide an escape) and the solution revealed all sorts of connections between the concepts of energy, entropy and information/intelligence. There is an analogous problem in politics: how best to apply intelligence to reduce local entropy? The insuperable problem of the lifts shows how hard this can be and gives a clue to what is really happening in Whitehall: most of everybody’s day is spent just battling entropy – it is not pursuing priorities and building valuable things.

For at least the period January 2011 – July 2012, it took a huge effort to think seriously about priorities other than after 10pm or at weekends and many of the meetings I set up to advance them got cancelled to deal with something ludicrous. Priorities slip unless you remain dementedly focused and demented focus is an alien concept in Westminster. Because ministers can never explain the truth about ‘crises’, and the official story is that any glitches are occasional aberrations for ‘a Rolls Royce machine’, there is a tendency for the baffled public to infer ministerial conspiracies, rather than chronic dysfunction, everywhere.

Story 3. Gogol’s Dead Souls in the DfE – or ‘priority movers’ and Whitehall HR…

In my first fortnight in January 2011, there was a terrible blunder with capital. We were told one Sunday that a senior official had made mistakes that had cost the taxpayer many millions of pounds. I said, naively, to one of the four most senior officials ‘so who will be replacing X [the official who had blundered]?’ Shock.

‘Dominic, you’re a spad, you’re not allowed even to discuss personnel matters.’

Me: ‘Michael will certainly want to know what is happening with this official and so do I.’

Official: ‘Errr, I’ll get back to you.’

Sure enough, they fixed the meeting to discuss it with MG without informing my office so I didn’t twig that it had happened for a while, by which time the Permanent Secretary had made his decision. When I first arrived, I thought they would not do something like this given it would obviously diminish my trust in them. I soon realised they did not care about this much – certainly not as much as they cared about keeping spads out of personnel issues. I soon learned the ‘set the meeting but ensure spads aren’t invited’ trick was a standard one. I developed countermeasures.

The official was, of course, not fired. He had an extended paid holiday then was promoted into a non-job for another few months before being pensioned off with a gong in the next honours list. Over the next few years, the capital team would bounce from debacle to debacle. We forced out various people, closed a quango, forced out more people. There were some improvements but blunders costing millions remained endemic because of a collapse of core skills and the HR system made it impossible to recruit the right people, as I explain below. (But SA-H: you are great, thank God you came, and you’ve saved millions, more power to you!)

A later example… I won’t go into details (unless they leak in which case I’ll clarify) but in a nutshell, something very important that the DfE had contracted was completely botched. Like opening Russian matrioshki, each meeting revealed a new absurdity and after seeing dozens of such episodes I now knew what would happen. First, I knew that the official who had signed the contract would have signed a stupid contract. Second, I knew that the contract had been signed three years earlier so the official would have long gone and the new people would shrug and say ‘not me’. (When I insisted that a particular inquiry into a cockup be pursued to a senior official in another department who’d left DfE, so mad was I at this trick, there was a panicked reaction: ‘we can’t go around demanding answers from officials who’ve moved, Dominic, where would it all end?!’)

Third, I knew that their bosses would all have changed too, so they could also say ‘very regrettable, but of course I wasn’t here then’. Fourth, I knew that EU procurement rules would be partly responsible for complicating everything unnecessarily. Fifth, I knew that some officials would instinctively cover it up while a tiny number would push for a serious ‘lessons learned’ exercise and get nowhere. Sixth, I had to make a decision about how hard to push for an internal investigation or use it as leverage to force officials to do something else I wanted done (‘SoS might be persuaded not to pursue this too hard, but we are very keen that X happens’, where X is something important and much resisted). Seventh, I knew that the first version of the scale of the problem would not be right and all the numbers would be wrong.

This time there was an added twist – the DfE had used (at the direction of the Cabinet Office, officials said) an EU Framework that actually forbade the DfE from clawing back the money from the company that had screwed up. This I had not predicted, it was a new twist though not a surprising one. ‘How many other contracts have been signed under this EU Framework which stop us from clawing back money?’ ‘Err, we’ll get back to you…’

Some people who make blunders like those described above are then deemed by the HR system to be ‘priority movers’. This means that a) they are regarded as among the worst performers but also means b) they have to be interviewed for new jobs ahead of people who are better qualified. It is a very bizarre system, made more bizarre by the fact that there are great efforts to keep it hidden from ministers and the outside world. These people float around in the HR system, both dead and alive, removed from ‘full time employee’ lists but still employed, like Gogol’s Dead Souls. ‘We need someone to do SEN funding.’ ‘Ahh, what about Y, they could do it.’ ‘But Y has been a rubbish press officer all his life, he’d be a disaster!’ ‘Yes, but it would be one less priority mover on my books.’ (‘Look, too, at Probka Stepan, the carpenter. I will wager my head that nowhere else would you find such a workman. What a strong fellow he was!’ ‘Why do you list the talents of the deceased, seeing that they are all of them dead? What is a dead soul worth, and is it of any use to any one?’ ‘It is of use to YOU, or you would not be buying such articles.’) This connection between core skills and the nightmare world of ‘HR’ is vital but practically ignored in all analyses of the civil service (see below).

Story 4. New blood learns the ropes. 

To every new person who would arrive (minister, spad, official, outsider coming in for a project, NED), I would give them roughly this advice:

‘There’ll be the odd exception but it’s safest to assume this… Every process will be mismanaged unless it involves one of these officials [XYZ]. No priority you have will happen unless spads and private office make it a priority. Trust private office – they’re the only reliable thing between you and disaster. Every set of figures will be wrong. Every financial model will be wrong. Every bit of legal advice will be wrong. Every procurement will blow up. Every contract process will have been mismanaged. Every announcement will go wrong unless Zoete [my fellow spad], Frayne [director of communications], or [names withheld to protect the innocent] is in charge – let them sort it out and never waste your time having meetings about communications. Never trust Clegg and Laws who only care about party politics, though you can trust Leunig who is honest. Never make an announcement on a Monday [see below]. Never announce budgets without Sam [Freedman] checking. Every process described as ‘cross-Whitehall’ will be a fiasco – especially if it is being coordinated by Number Ten. Don’t tell Number Ten anything about anything – leave that to us. Don’t give Ofsted anything else to do as it can’t do its core functions now. In short, assume that everything that can go wrong will go wrong and when you catch yourself thinking ‘someone MUST have done X or it would be crazy’, stop, because X will not be happening. Your only hope is to focus on a few priorities relentlessly and chase every day and every week. When you cock something up, tell us straight away, and when you think we or Michael are cocking something up, tell us straight away.’

People had the same reaction. A sort of nervous laughter and a ‘mmm yes sounds ghastly, well we’ll see.’

Within two weeks they would rush through the spads door gabbling something like: ‘OhmyGOD you won’t believe this meeting I’ve just been to in the Cabinet Office, this place is crazy, I can’t believe it, it’s Alice in Wonderland.’

Me: You’re through the looking glass.

Them: The oddest thing is nobody seems to realise how weird it is, I kept looking around the table waiting for someone else to explode but everyone just nodded as if it’s normal.

Me: It is normal. Zoete, add it to the list. [Zoete reaches over and scribbles on a bit of paper, while talking on the phone with exaggerated calmness, ‘No no that’s not what it means, you can’t write that… No no our announcement is the opposite, the leak was a spoiler by Clegg, yeah yeah I KNOW IT’S CONFUSING… No I don’t know why they used those figures, they might be lying they probably just screwed up. No, listen, forget it it’s a rubbish story and anyway Paton had it 6 months ago. Now listen to this, much more important and you can have it exclusive…’]

[Bang bang on the glass door, an official enters looking nervous…] Err, I need to speak to Zoete, I’m afraid we sent out hundreds of funding letters and all the numbers are wrong, the press office is already taking calls, thing is, the letters went out without private office seeing them so SoS doesn’t know anything about them.

Me: Give him two minutes, he’s just dealing with the Clegg thing this morning…

[Bang bang bang on the glass door, a PS enters looking mad.] DPM’s office on the phone. They say that because we didn’t consult with him on the latest Ofqual thing Clegg’s had a strop and HA [Clegg’s Home Affairs Committee] won’t clear your GCSE announcement.

Me: Doesn’t matter, we’re not sending it to HA or telling No10, we’re just announcing it and it’s already briefed for tomorrow. Just reply saying ‘OK, we’ll get back to you, SoS is pondering’.

Official: God, not again. [Leaves.]

[Bang bang on the glass door, another official enters looking nervous, glances at the second official…] Err, Dom, you know that contract we were talking about yesterday?

Me: Don’t tell me the tests have gone haywire.

Official: Yes they have but that’s not what I mean – I mean that Academy procurement process.

Me: Yes.

Official: Well, the legal advice says – if we go ahead, we’ll get JRd [judicially reviewed] and lose but if we stop and reboot we’ll also get JRd and lose.

Me: So we’re screwed whatever we do.

Official: Seems like it.

Me: Tell the Perm Sec’s office I’ll need ten minutes with him.

Official [lowering voice]: I think he wants to talk to you anyway about [XXX] getting moved.

Me: Make it 15.

[Bang bang on the glass door, another official enters…] No10’s been on the phone, XXX [a private secretary] says the PM is ‘bored of fighting with Clegg on childcare’ so he’s told us to give in.

Me: That was always doomed, better tell Truss, she’s about to give a speech promising it will happen.

[Bang bang on the glass door, another official enters…] Err, the DPM’s office just called sounding contrite, he’s just had a meeting with black community leaders, sounds like he’s blurted out that Mary Seacole will be kept in the National Curriculum, so officials are saying ‘really sorry, we know we promised no curriculum gimmicks but DPM’s spads think this will now have to happen.’ Also, the press have got wind of it so…’

Me: They probably don’t realise she isn’t in the Curriculum now, she’s in the Notes. Clegg, I’ll tell you what we’re going to do about Clegg –

[Bang bang on the glass door, another official enters…] Zoete’s meeting on the National Pupil Database is going in. Zoete’s trying to force [XXX] to publish more data but if he isn’t there bugger all will happen.

Me: I’ll do it, poor Zoete’s swamped. [NB. Zoete was ‘media spad’ but unlike most media spads he spent a huge amount of his time on policy and management issues.]

[Bang bang on the glass door, another official enters looking nervous…] Err, I need to speak to Zoete, the latest iteration of the School Food Plan involves SoS, the PM, and Henry Dimbleby zipwiring into a bouncy castle, and No10 is asking if we should get Boris along, but we thought we’d better check with you guys, it sounds TOTALLY CRAZY but officials say the PM is desperate to be involved in a food stunt.

Me: Great, that’s the perfect way to launch this fuc –

[Zoete covering the phone with his hand.] ARGHHHH, WHAT ZIPWIRES?! – hang on, Shippers, hang on, I’ll call you back… ZIPWIRES, what the…

I leave with the new person, ‘you’ll get used to it, gotta have priorities, keep your focus, or you’ll just blunder around in this chaos all day…’

(NB. I’ve left out the best stories.)

Why is this not an unusual 20 minutes?


Part III: Analysis

The failures of Westminster & Whitehall: wrong people, bad education and training, dysfunctional institutions with no architecture for fixing errors

‘The man of system, on the contrary, is apt to be very wise in his own conceit; and is often so enamoured with the supposed beauty of his own ideal plan of government, that he cannot suffer the smallest deviation from any part of it. He goes on to establish it completely and in all its parts, without any regard either to the great interests, or to the strong prejudices which may oppose it. He seems to imagine that he can arrange the different members of a great society with as much ease as the hand arranges the different pieces upon a chess-board. He does not consider that the pieces upon the chess-board have no other principle of motion besides that which the hand impresses upon them; but that, in the great chess-board of human society, every single piece has a principle of motion of its own, altogether different from that which the legislature might choose to impress upon it.’ Adam Smith.

The selection, education, and training, of those making crucial decisions about our civilisation are between inadequate and disastrous. The institutions they work in are generally dysfunctional.

First, our mentality. We often are governed by ‘fear, honour and interest’ (Thucydides). We attribute success to skill and failure to luck: ‘The movement of events is often as wayward and incomprehensible as the course of human thought; and this is why we ascribe to chance whatever belies our calculation,’ said Pericles to the Athenians. We prefer to enhance prestige rather than face reality and admit ignorance or error. ‘So little trouble do men take in the search after truth, so readily do they accept whatever comes first to hand’ (Thucydides); ‘men may construe things after their fashion / Clean from the purpose of the things themselves’ (Cicero, Julius Caesar). As Feynman said, if you want to understand reality, ‘The first principle is that you must not fool yourself – and you are the easiest person to fool.’

Robert Trivers, one of the most influential evolutionary thinkers of the last fifty years, has described how evolutionary dynamics can favour not just deception but self-deception: conflict for resources is ubiquitous; deception helps win; a classic evolutionary ‘arms race’ encourages both deception detection and ever-better deception; perhaps humans evolved to deceive themselves because this fools others’ detection systems (for example, self-deception suppresses normal clues we display when lying). This is, perhaps, one reason why most people consistently rate themselves as above average.

Children display deception when just months old (e.g. fake crying). There is ‘clear evidence that natural variation in intelligence is positively correlated with deception… We seek out information and then act to destroy it… Together our sensory systems are organized to give us a detailed and accurate view of reality, exactly as we would expect if truth about the outside world helps us to navigate it more effectively. But once this information arrives in our brains, it is often distorted and biased to our conscious minds. We deny the truth to ourselves … We repress painful memories, create completely false ones, rationalize immoral behavior, act repeatedly to boost positive self-opinion, and show a suite of ego-defense mechanisms’ (Trivers). Roberta Wohlstetter wrote in ‘Slow Pearl Harbors’ regarding ignoring threats, ‘Not to be deceived was uncomfortable. Self-deception, if not actually pleasurable, at least can avoid such discomforts.’

Tales of such self-deception are legendary. ‘I don’t know how Nixon won, no one I know voted for him’ (Pauline Kael, famous movie critic, responding to news of Nixon’s 1972 landslide victory). ‘The basic mechanism explaining the success of Ponzi schemes is the tendency of humans to model their actions, especially when dealing with matters they don’t fully understand, on the behavior of other humans,’ said Psychiatry Professor Stephen Greenspan in The Annals of Gullibility (2008), which he wrote just before he lost more than half his retirement investments in Madoff’s ponzi. ‘But for self-deception, you can hardly beat academics. In one survey, 94 percent placed themselves in the top half of their profession’ (Trivers). ‘Academics, like teenagers, sometimes don’t have any sense regarding the degree to which they are conformists’ (Bouchard, Science 3/7/09). Even physical scientists who know that teleological explanations are false can revert to them under time pressure, suggesting that such ideas are hardwired and are masked, not replaced, by specific training.

Also, it is depressingly possible that those who climb to the top of the hierarchy are more likely to focus only on their own interests. Studies such as ‘Higher Social Class Predicts Increased Unethical Behavior’ claim that the rich are much more likely ‘to prioritize their own self-interests above the interests of other people’ (Piff) and even just thinking about money makes people more self-centred. Not only are richer people healthier (less likely to have heart attacks or suffer mood disorders), but they also produce less cortisol (suggesting lower stress levels; cf. studies suggest those at the top of hierarchies suffer less stress because they feel a greater sense of control), they are less attentive to pedestrians when driving, and less compassionate when watching videos of children suffering with cancer.  This article touches on these studies though it should be remembered that many studies of such things are not replicated. Further, one of the most important studies on IQ, personality and scientific and financial success also shows a negative correlation between earnings and agreeableness. (Cf. piece by Mary Wakefield HERE.)

Most of our politics is still conducted with the morality and the language of the simple primitive hunter-gatherer tribe: ‘which chief shall we shout for to solve our problems?’ Our ‘chimp politics’ has an evolutionary logic: our powerful evolved instinct to conform to a group view is a flip-side of our evolved in-group solidarity and hostility to out-groups (and keeping in with the chief could lead to many payoffs, while making enemies could lead to death, so going along with leaders’ plans was incentivised). This partly explains the persistent popularity of collectivist policies and why ‘groupthink’ is a recurring disaster. Such instincts, which evolved in relatively simple prehistoric environments involving relatively small numbers of known people and relatively simple problems (like a few dozen enemies a few miles away), cause disaster when the problem is something like ‘how to approach an astronomically complex system such as health provision for millions.’

Second, our education and training. The education of the majority even in rich countries is between awful and mediocre. In England, few are well-trained in the basics of extended writing or mathematical and scientific modelling and problem-solving. Less than 10 percent per year leave school with formal training in basics such as exponential functions, ‘normal distributions’ (‘the bell curve’), and conditional probability. Only about 2-3 percent are taught about matrices and ‘complex numbers’ (which many children can grasp between the age of 10-14 but they are not given the chance unless they do Further Maths A Level). Less than one percent learn hard skills necessary to grasp how the ‘unreasonable effectiveness of mathematics’ provides the language of nature and a foundation for our scientific civilisation. Only a small subset of that fraction of one percent then study trans-disciplinary issues concerning vital complex systems the failure of which cause chaos.

This small subset has approximately zero overlap with powerful decision-makers. Generally, these people are badly or narrowly educated and trained. Courses offered by elite universities are thought to prepare future leaders well but are clearly inadequate and in some ways are damaging (see below). Those who scramble to the apex of power are sometimes relatively high scorers in tests of verbal ability (like Cameron) but are rarely high scorers in tests of mathematical ability or have good problem-solving skills in cognitively hard areas such as physics or computer science.

MPs and officials have to make constant forecasts but have little idea about how to make them, how the statistics and computer models underlying these forecasts work, or how to judge the reliability of their own views. A recent survey of 100 MPs by the Royal Statistical Society found that only 40% of MPs correctly answered a simple probability question (much simpler than the type of problem they routinely opine on): ‘what is the probability of getting two heads from flipping a fair coin twice?’ Despite their failures on a beginner question, about three-quarters nevertheless said they are confident in their ability to deal with numbers. Issues such as ‘how financial models contributed to the 2008 crisis’ or ‘intelligence and genetics’ cannot be understood in even a basic way without some statistical knowledge, such as normal distribution and standard deviation, yet most MPs do not understand much simpler concepts than these. They also have little knowledge of evolutionary systems (biological or cultural), and little understanding of technology. (How many of those at a senior level dealing with Ebola discussions or financial market disasters recently have any idea about the topology of ‘scale free networks’, cf. HERE? The basic concepts, as opposed to detailed modelling, are not hard to grasp but they do not appear in the typical education of ministers or senior officials.)

A mismatch on a scale of 104 between the experience of MPs and the responsibilities of ministers. Further, ministers have little experience in well-managed complex organisations and their education and training does not fill this huge gap. Even most of the ones who have good motives – and there are many, though they struggle to advance – have a fundamental problem of scale. The apex of the political system is full of people who have never managed employees on the scale of 102 people or budgets on the scale of 106 pounds, yet their job is to reshape bureaucracies on scales of 104 (DfE) – 106 (NHS) employees and 1010-1011 pounds. The scale of their experience of management is therefore often at least 104 off from  what they are trying to control. Some unusual people can make jumps like this. Most cannot. For example, Cameron never worked in a highly functioning entity before suddenly acquiring large responsibilities  – he went straight from PPE to Conservative Central Office – and never had responsibility for anything on a significant scale so he could not acquire the experience that he so needs now (and, perhaps more importantly, he has never understood how unprepared he and his gang were). The only minister in the DfE team 2010-14 who had significant experience of dealing with budgets on a scale of £108-109 was Nash – unsurprisingly, he was the most effective minister at dealing with DfE budgets / capital / property deals and so on. John Holland, the inventor of ‘genetic algorithms’, points out that ‘changes of three orders of magnitude or more usually require a new science’. It should be no surprise that politics is a story of repeated administrative failure.

Many of these problems can be seen particularly starkly in those who did courses like Politics, Philosophy, and Economics (PPE). PPE is treated as a cross-disciplinary course suitable to educate future leaders. It is failing. Part of the reason for this is that the conventional economics that is taught often gives students a greatly misplaced confidence in their understanding of the world. They are taught to treat some economic theories as if they are similar to physical theories, and there is often spurious precision involving mathematical models but no explanation of the conceptual problems with these models, or the critique of them by physical scientists. I have watched many PPE graduates give presentations of forecasts, complete with decimal points, of economic numbers years into the future, then dismiss arrogantly those who point out the repeated failure of such predictions. PPE also teaches nothing about project management in complex organisations so they have little feel for how decisions will ripple through systems (including bureaucracies) into the real world.

At its worst, therefore, students leave university for politics and the civil service with degrees that reward verbal fluency, some fragments of philosophy, little knowledge of maths or science, and confidence in a sort of arrogant bluffing combined with ignorance about how to get anything done. They think they are prepared to ‘run the country’ but many cannot run their own diaries. In the absence of relevant experience, people naturally resort to destructive micromanagement rather than trusting to Auftragstaktik (give people a goal and let them work out the means rather than issue detailed instructions) which requires good training of junior people. This combination of arrogant incompetence is very widespread in Westminster and responsible for many problems. When such people surround themselves largely or solely with advisers who are very similar to themselves, we know from large amounts of research that the odds are high that groupthink will make these errors and problems even worse.

NB. These gaps in education and training are not a ‘natural’ product of the concepts’ difficulty but because of deep flaws in a) school and university education and b) training programmes.


Third, our institutions and tools. Unlike science and markets, politics has no comparable institutional architecture that provides reliable processes for limiting the predictable trouble caused by our mentality combined with a lack of education and training.

Large bureaucracies, including political parties, operate with very predictable dynamics. They have big problems with defining goals, selecting and promoting people, misaligned incentives, misaligned timescales, a failure of ‘information aggregation’, and a lack of competition (in normal environments). These problems produce two symptoms: a) errors are not admitted and b) the fast adaptation needed to cope with complexity does not happen. More fundamentally, unlike in successful entities, there is no focus of talented and motivated people on important problems. People externally ask questions like ‘how could X go wrong?’, assuming that millions are spent on X so everyone must be thinking about X, but the inquiries usually reveal that nobody senior was thinking about X – they spent their time on endless trivia, or actually stopping people working on X.

These dynamics are well-understood but are very hard to change. Bureaucratic institutions tend to change significantly only in the event of catastrophic failure (e.g. 1914, 1929, 1945, 1989) – catastrophes that they themselves often contribute to. However, these dynamics are so deep that even predictable failures that lead to significant loss of life can often leave bureaucracies largely untouched other than a soon-forgotten media frenzy.

Goals. First, in political institutions, it is usually much harder than in science or business to formulate and agree clear goals like ‘make a profit’ or ‘search for a new particle within these parameters’. Often, the official public definition of the goal is not even properly defined or is so vague as to be useless. This problem is entangled with the problem of incentives (below) – often defining goals wisely is disincentivised. Often in politics, officially stated goals are, taken literally, nonsensical and could not possibly be serious but are worded to sound vaguely friendly (e.g. ‘this must never happen again’, which I must have deleted dozens of times from draft documents).


Personality and ‘human resources’.

Second, political institutions tend to become dominated by narcissists and bureaucrats.

What sort of people are selected by parties to be MPs and (in the UK) form the pool from which ministers are chosen? MPs are seldom selected for their ability to devise policy, prioritise, manage complex organisations, or admit and fix errors. Elected representatives are often chosen from a subset of people who have very high opinions of themselves and who really enjoy social networking. While some who seek election are motivated at least partly by genuine notions of public service, many representative bodies are full of people motivated mainly by ambition, vanity, and a strong desire that others watch them talking. The social aspect of being an elected representative inevitably repels some personality types and attracts others – some are energised by parties and public speaking, others are drained by it. Often watching MPs one sees a group of people looking at their phones listening only for a chance to interrupt, dreaming of the stage and applause. They are often persuasive in meetings (with combinations of verbal ability, psychological cunning, and ‘chimp politics’) and can form gangs. Parliaments seem to select for such people despite the obvious dangers. This basic aspect inevitably repels a large fraction of entrepreneurs and scientists who are externally oriented – that is, focused on building things, not social networking and approval.

Many political parties and governments reinforce the problem of publicity-seeking MPs by promoting people up the greasy pole on the basis of their success in self-publicity and on the basis of having helped their ‘in-group’ (i.e. their own party) and harmed their ‘out-group’ (other parties). If you watch junior ministers as they approach reshuffles, you will see what I mean. They select for those who pursue prestige and suppress honesty (a refusal to admit errors can be a perverse ‘asset’ in politics) and against those with high IQs, a rational approach to problem-solving, honesty and selflessness; they are not trying to recruit those most able to solve problems in the public interest. Politics therefore suffers from a surfeit of narcissists.

Further, consider the plight of an MP, probably without sufficient training or experience, suddenly made Secretary of State of a department spending  £1010-1011. This poor minister does not have any of the most basic tools of a CEO regarding their organisation: they cannot hire, fire, promote, or train their team. (A typical SoS is allowed to hire and fire 2-3 ‘special advisers’, so perhaps one in 103-105 of the employees they are ‘accountable’ for, and formally, as a peeved David Cameron likes to remind people occasionally, these spads are formally hired by him not by the SoS.) Not only are ministers 1) often the wrong people with the wrong education, and 2) they are operating in institutions more on a scale of Bill Gates’ experience than their own, 3) they are trying to do this without any of the basic tools Bill Gates uses. Further, the supposed experts whose job it is to manage on their behalf are often similarly inexperienced and no better at managing the organisation (see below).

The biggest contrast in personality type and outlook of relevance to politics is not between ‘business’ and ‘politics / civil service’. The real contrast is between ‘bureaucrats‘ (private and public sector) and venture capitalists, start-up entrepreneurs, and small businesspeople (‘startups‘ for short). Many of those who dominate FTSE-100 companies and organisations like the CBI are much more similar to the worst sort of bureaucrats than they are to startups. This blog by physicist Steve Hsu, Creators and Rulers, discusses the differences between genuinely intelligent and talented ‘creators’ (e.g. scholars, tech start-ups) and the ‘ruler’ type that dominates politics and business organisations (e.g. CEOs with a history in law). The ‘Ruler’ described there represents with few exceptions the best end of those in politics, many of whom are far below the performance level of a successful ‘political’ CEO.

It is the startups who, generally, make breakthroughs and solve hard problems – not bureaucrats – but it is the bureaucrats who dominate the upper echelons of large public companies, politics, and public service HR systems. Civil service bureaucracies at senior levels generally select for the worst aspects of chimp politics and against those skills seen in rare successful organisations (e.g the ability to simplify, focus, and admit errors). They recruit ‘people who won’t rock the boat’ but of course the world advances exactly because of the efforts of people who do ‘rock the boat’. They recruit a lot of lawyers, who are trained to focus on process rather than outcome, reinforcing one of the worst aspects of bureaucracies. Further, consider how easy it is for a) a lawyer and b) a cutting-edge scientist to become an MP or senior official without sacrificing their career. We do not make the system welcoming for our best problem-solvers.

Further, when someone with a startup mentality strays into the bureaucratic world, the bureaucracy reacts like an immune system to expel the intruder. This is one of the reasons why young talented people who want to get things done more than they want to get ahead – they want ‘to do’ rather than ‘to be’ – soon leave the civil service. This in turn explains why bureaucracies are the way they are – they filter out people with a startup approach so the dominant culture at senior levels is so distasteful for someone with a startup mentality that they leave and the institution becomes even harder to change. If your entire institutional structure selects against the skills of entrepreneurs or scientists, do not be surprised when the people in charge cannot solve problems like entrepreneurs or scientists.

The true Jedi skills of officials are revealed in battles over appointments. This is the lifeblood of Whitehall. This is where favours are traded and a lot of personal money rides on decisions; ‘a post now for Charlie, and I get one back in a few years’. Spads are theoretically 100% (and practically near 100%) excluded from appointments. When you want to appoint someone, they insist on an ‘open competition’. When they want to appoint someone – say a senior official has someone who needs to be moved and they don’t want any arguments – then miraculously an ‘open competition’ is no longer needed. When there is a ‘competition’, the Cabinet Office always has its candidate and sometimes more than one. It will usually spy who your candidate is if you have one (and if you haven’t you should not let the process start).

They usually only gave Gove a choice of two so ideally (for them) they weed out your candidate at an early stage so you are left choosing between their two candidates. But even if your candidate survives to the last two, that is no guarantee of victory. In extremis, they will find a way to exclude your candidate by post facto altering the criteria, or they will ‘discover’ some bit of evidence ‘that cannot be shared for legal reasons’, or any one of a number of tricks in the hidden wiring. (They control the process for the process – and, if necessary, the inquiry into the process for the process – so they can always change whatever they want, while maintaining the facade of ‘open and fair’, of course, without anybody realising.) Sometimes you can trade. ‘You know the department badly let us down on X. You owe us. I want Y to get this job and I don’t want to hear anything about “impartial processes” that will spit out the Cabinet Office candidate who we both know is clueless. You give me this, I’ll drop Z. Deal?’ (I.e. an implicit threat to secure a trade.) Unsurprisingly, the best method is a mild form of blackmail – get an official who knows you could get them chopped to act as your agent inside the system. The Cabinet Office is watching for overt enemies – like anybody, it is more vulnerable to ‘traitors’. (NB. I’m not claiming to have done this.)

The same attitude extends to the basic issue of officials being fired for incompetence. In my entire time in the DfE, I never encountered a single person fired for incompetence. What tends to happen when an official has badly dropped the ball? In general, when officials know they have cocked up, a simple default mechanism is to insist that a) ‘it is very sensitive involving legal / personnel issues we’re not allowed to discuss with spads’, so b) ‘we must discuss this with just SoS’. Since they also write the minutes of the meeting, they can then claim later that ‘SoS agreed it would be unfair to take action against X’. Often spads are not even told about such meetings or ‘decisions’ for ages and by the time they find out, it’s too late.

They tried this repeatedly with me in the early days, particularly as they realised that I would pursue such issues while it is almost impossible for a SoS to pursue such matters without help from spads (officials simply string it out, using ‘legal issues’ if necessary, and the SoS will have so many other problems running concurrently he has to let it, or other things, go). Making clear that such tactics may be repaid with determination to have them moved and/or give them a career blot is vital to limit such tricks (you also need an effective private office). As the DfE gradually changed in 2011/12, some officials realised it would be easier for them to take me into their confidence on personnel issues but it was persistently very hard to deal with this. Moving and swapping (never firing) officials via trades with their bosses is vital if you want to change anything, but ministerial teams that intrude on personnel and management issues encounter very strong resistance and not-so-coded messages to ‘leave us alone or else, this isn’t your business’.

There is a very basic problem with the selection of senior officials: confusion between policy, management, and ‘fixing’. In markets and science, the world is specialising. Of course you get rare people who are great at more than one thing. However, it is obvious that the skills required for doing great policy – e.g. Michael Quinlan’s famous work on nuclear policy – are not the same as the skills of Bill Gates or Steve Jobs in managing. It is also obvious that one can be great at one and awful at the other other. A third skill prized in Whitehall is ‘fixer‘: this is neither policy nor management, strictly speaking. Permanent Secretaries are generally recruited supposedly to be the lead policy adviser to the Secretary of State but the people who appoint Permanent Secretaries also know that being a ‘fixer’ is vital and it is the ‘fixer’ role that is highly prized as a fixer is almost always ‘one of us’ – you rarely get maverick fixers. Management is not seen as nearly so important. E.g. Chris Wormald in the DfE knows that his chances of promotion do not rest on him turning the nightmare of DfE capital into an exemplar of good government. Unsurprisingly, many Permanent Secretaries are more interested in policy, politics, and fixing – and neglect management. They in turn hire people in their image. The outcome? Ministers are not allowed to manage departments and Permanent Secretaries are not interested in managing and/or can’t do it. One of the many ways in which Whitehall refuses to face reality is that it largely ignores this dilemma. (Please do not take this as criticism of Chris Wormald, I am making a general point.)

Flexi-time and holiday chaos. Why did I say (above) ‘Never make an announcement on a Monday’? We pretty much banned Monday announcements unless they were routine because we discovered that it was impossible to assemble the responsible team on a Friday to discuss Monday. Some of the people would be on what were called ‘compressed hours’ (work an extra hour for a few days and you earn a day off), others would be on ‘flexitime’ (‘working from home’). Even worse, the lead official who you have been working with on a project – say, GCSE changes – will often vanish. For example, you email them on Thursday night saying ‘can we meet tomorrow to discuss X for the announcement on Monday’ and you CC in their team. Immediately, you get a bunch of pingbacks, many related to compressed hours or flexitime, one of which will be from the lead official and say, ‘I am now on annual leave until X’. WHAT? you shout at the computer, IMPOSSIBLE, I talked to you only two hours ago!? But no – it is all too possible. While you are making the announcement about X for which they have been the lead official for months and about which you already have a queasy feeling, they are on the beach and they have gone on holiday without telling you – they’ve set their auto-responder and fled. Further, nobody you complain to will think there is anything wrong with this. Why? Because failure is normal, not something to strive to avoid.

This relates to another HR nightmare. People are constantly moving jobs. Often you have a team in which there is one person clearly better than all the others. Before you know it, the one person who understands a subset of funding decisions has been moved to be in charge of SEN and you know you are going to have even more funding nightmares than usual for the next few months. These things happen without reference to ministers and spads. After we had been there for a while, we sometimes got warning that such moves were in the offing but we could rarely head off a problem. ‘Give X a pay rise to keep her in the job and save the money by getting rid of her boss who is rubbish and more expensive – everyone’s a winner’, I would plead, obviously to no avail.

Priority movers’, Whitehall’s Dead Souls. I mentioned above the system called ‘priority movers‘ that reminded me of Gogol’s Dead Souls. This is a pool of people who have been identified for the axe by a review process looking to reduce headcount. However, they are not actually axed. They are labelled ‘priority movers’. This means that whenever someone needs to hire someone, they have to look through the pile of ‘priority movers’ first. But the ‘priority movers’ include, by definition, people regarded as the worst in the department (though actually the worst officials in the DfE always escaped the axe). Senior managers therefore spend huge amounts of time interviewing ‘priority movers’ for roles so that they do not spark an employment grievance. For example, the press office has to interview priority movers for the role of ‘senior press officer’ even though they have never talked to a journalist in their life, or a team recruiting for someone to ‘manage’ a complicated process has to interview people even though they have spent their entire career in the press office and have no relevant experience. Imagine how much money is wasted having senior officials waste hours interviewing people they already know they will never give the job to simply in order to tick a HR box.

In a further twist, whenever we found that something important was being screwed up because of a delay involving this process, I would go and complain and every time I would be told – ‘Dominic, there is no such thing as priority movers, you’ve misunderstood, naturally you’re not an expert on Whitehall HR, why would you be hahaha, X has explained it badly to you, I’ll investigate’. Mmm, I thought, early on, weird. Then you would find that poor old X had been given a bollocking for letting you in on the ‘priority mover’ scam. Then you would be told that ‘it did exist but it’s finished now’. Then a few weeks later, the same thing happened again. For three years, officials kept telling me that the priority mover scam had been ditched and repeatedly I discovered it had not. Finally, the Permanent Secretary came clean: yes it exists, yes it’s normal across Whitehall, yes I agree it’s mad, no I cannot stop it unless the Cabinet Office change HR rules Whitehall-wide. And this was the bottom line on all Whitehall HR. Everybody knows that Cameron hasn’t the faintest interest in fighting over such issues, not least because he doesn’t grasp the connection between such systems and why things he wants to happen don’t happen, and without his support there are strict limits on what Secretaries of State can do. Maude’s team has tried to change things but major changes are impossible when senior officials know that the prime minister’s heart, and his chief of staff, are with them.

I have seen startup people change politics then run away in disgust, and I have seen young people with a startup mentality bang their heads against brick walls then leave in disgust, to be replaced by the worst sort of apparatchik who cares nothing for the public interest but is regarded as ‘one of us’. I saw some excellent civil servants in the DfE, particularly women 25-35 in private office who kept the show on the road, but the HR system generally promoted middle-aged male conservative mediocre apparatchiks. In 2013, I sent this presentation on how Netflix’s ‘human resources’ system works – something you should read if you want an example of the difference between a ‘startup’ and bureaucratic culture – to a few of the most senior officials in Whitehall (inside and outside the DfE). One replied, ‘This is fascinating… The culture described here … is not in the legal framework, civil service rules or the working culture here.’ Exactly.

Colonel Boyd, the revolutionary fighter pilot who helped design the F-16 and was the bane of the USAF bureaucracy, talked often of the choice between ‘to be’ or ‘to do’ – whether to focus on climbing the greasy pole or serving the public. Insiders tend to choose the former, partly because of natural human selfishness but also because the combination of the promotion system and internal organisational incentives strongly encourages them to do so and follow corrupted assumptions contrary to the public interest.

(PS. One of the ways we tried to get around the crazy Whitehall HR system was to bring in expertise from outside (which sometimes required overcoming strong internal opposition, given the determination to control appointments). E.g. Without Rachel Wolf and the New Schools Network, there would have been no Free Schools in 2011 and the whole programme may well have collapsed in 2010/11 (NSN also developed a huge amount of the detailed processes that were needed, and they were more influential than all think tanks combined). We split the school minister job into two so that Jonathan Hill (then Nash) could focus just on Academies and Free Schools (and we divided DfE empires to fit this change). We brought in Alison Wolf who did great work on vocational education. We brought in people from outside with skills the civil service needed but did not have, such as Tom Shinner. James Frayne both greatly reduced the headcount and budget of the communications department and transformed its performance. We invited Ben Goldacre, who had been publicly critical of us, to analyse the DfE’s approach to evidence-based policy and data, against initially very strong opposition (credit to Wormald for siding with us on this) and his report has helped changed attitudes to ‘cargo cult science’ in education. We asked a very successful head teacher, Charlie Taylor, to help us dig through the bureaucracy to the facts about behaviour problems in schools. We made great use of (unpaid) non-executive directors such as Theo Agnew, Paul Marshall, and David Meller who have each saved the taxpayer many millions. All of these people got involved because their priority was improving schools – not party politics – and they all had the virtue of telling MG and spads what they really thought and where we were wrong, which helped increase cognitive diversity since we all disagreed about all sorts. If I didn’t think they would do that, I would not have wanted them involved.)


Incentives and institutions.

Third, even if a goal is well defined, it is usually not at all what is incentivised internally. Unlike open systems such as Silicon Valley which does incentivise solving hard problems, Westminster and Whitehall do not incentivise people to solve useful problems or even to avoid obvious waste and failure. Incentives tend to enforce groupthink, coverups, and the defence of the status quo because that is where the power and money is. Incentives encourage people to stay within the current broken rules but solving hard problems is extremely hard to do in such circumstances. Westminster’s incentive system pushes people to spend their time trying to manipulate the media and help their party against the other. Between parties, MPs focus on small differences between each other in order to gain power for themselves – they are not focused on important problems facing the public.

Bureaucracies lack the institutional mechanisms of markets and science that allow relatively quick adaptation to errors. Bureaucracies tend to be closed or opaque rather than transparent, unlike the scientific peer review system at its best. Bureaucracies, such as the Department for Education or Health, have to operate without a functioning price system which is so fundamental to the decentralised coordination of markets. Instead of clear goals, a price system, and (theoretically) financial transparency for shareholders, and instead of the institutional mechanisms of the scientific method, there are unclear goals and often distorting ‘targets’.

Markets and scientific prizes incentivise goals while letting decentralised cooperation figure out methods. For example, DARPA’s recent Grand Challenge sparked the breakthroughs in autonomous vehicles now changing the world. It operated by having a carefully defined performance goal but leaving competing teams to decide on methods. Bureaucracies start off with unclear goals and then set many targets involving methods. These targets therefore rapidly pervert incentives internally. Further, bureaucracies suck decisions ever-upwards to ‘wise chiefs of the tribe’. Most people feel disempowered, sullen, and unappreciated (rationally, because they often are unappreciated). They are dominated by the feeling that most of one’s effort is just battling entropy – not advancing.

What feedback that happens is often slow, confused, and corrupted by dodgy incentives. This lack of transparency and feedback means it is easier for senior people to fiddle targets than admit the targets were wrong. People lower down the hierarchy fiddle targets because they have (often accidentally) been incentivised to do so, hence many of the NHS scandals and why schools game league tables. In extremis, you get peasants melting down ploughs for scrap metal to hit Mao’s ‘Great Leap Forward’ steel targets, leading to famine. In Soviet Russia, quotas for steel sheets ‘by the ton’ were made too heavy, and quotas ‘by area’ were made too thin. Instead of admitting failure, it is easy to shovel more and more money into failing systems – particularly since one does not have to persuade sceptical investors and one can fiddle the books in ways that public companies cannot. Instead of admitting failure, it is easier to accuse your political opponents of bad motives – ‘you want X to fail because you don’t care‘, and so on. In extremis, the failure of a Great Leap Forward leads not to retreat but to a Cultural Revolution.

Officials are not incentivised to ask ‘who in the world has already solved problem X by doing Y and how could we implement Y here as cheaply and quickly as possible?’ In meeting after meeting, I would ask this question. Whitehall is very parochial and officials hate the idea of just taking an idea from elsewhere, something successful companies do routinely. Repeatedly, officials would come back in a fortnight with some rubbish idea. ‘Did you look at how they’ve solved this in Switzerland or XXX?’ No. ‘Why not since I asked you to?’ Err… ‘Do it now.’ A week later. ‘We’ve looked, there’s nothing.’ ‘I’ve looked too – I found this, go and work on it’. ‘It won’t work here because – ‘. ‘Go and work on it and I want to see it in 48 hours.’ ‘We can’t do it in 48 hours, I have to look after my kids / I’m on holiday / I’m on compressed hours, it’ll take us a month at least’. ‘You’ve already had three weeks, get it to me in 48 hours or…’, etc. It is hard to avoid the conclusion that officials often prefer a process involving months of meetings and a long implementation timetable as this provides easy, no-pressure work long into the future.

This connects to the issue of record-keeping and institutional memory. The DfE destroyed its own library some time before 2010. It was a sign of how abysmal Whitehall has become that such things – and the much worse destruction of the Foreign Office library – happen and nobody really cares. It is also abysmal at record-keeping. Partly because everybody can email everybody with huge CC lists and attachments, nobody keeps accurate files (apart from private office). The situation is so bad that many Ministers have been reduced to FOI-ing their own departments (though this is not only an issue of competence – it is also an issue of trust).

Whitehall is not only parochial about other countries, it is parochial about its own past. One of the most useful questions one can ask is not only ‘who has already solved this problem?’ but ‘have we already tried to do X and failed?’ In the DfE there is no system to answer this question reliably. Unless you get lucky with an old-timer, you cannot know and because they abolished their own library you can’t even go and study it. (All the emails, files, papers etc are supposedly archived somewhere but obviously they would never let spads or a spad appointment into it to do analysis.) An obvious thing that is desperately needed in Whitehall is the creation of a network of ‘libraries plus internal historians’ connected to departments’ analysis teams that could not only answer the question ‘did we already fail with X?’ but would also be able to make public, on proper websites, as much information as possible for researchers and the general public to examine. This is one of the few aspects of the civil service that, to me, obviously needs to be ‘permanent’ yet it is now neglected by a civil service desperate to maintain its permanence in many fields where it is not necessary.

Officials are not incentivised to cooperate across Whitehall. Where there is a cross-Whitehall issue, there will be a turf war. Here is an example of how perverse incentives work. We regarded many cross-Whitehall plans (often appalling gimmicks from No10) as an excellent opportunity to give a bit of the DfE away to another department in pursuit of a smaller and better focused department. Why? Officials regard it as a ‘win’ to take over control of some policy or process regardless of how doomed it is. This makes it surprisingly easy to ditch various bits of a department or swerve involvement in some dreadful idea. Our modus operandi was to drop a hint to officials from the other department that we had no interest in X, they would suggest that they should control X and dig in for a fight, we would say ‘great idea, take X as far as we’re concerned though I doubt [junior minister Y] will agree, check with private office,’ they would go to private office, private office would say ‘reluctantly our minister is prepared to give you control of X if you will do Z [something we actually cared about]’, the ‘opposing’ officials would do the deal and collect a pat on the head from their superiors, while our private office got what we really wanted in return for what they presented as a ‘concession’ to the other department. Win-win for us, though in conventional Whitehall terms we had ‘lost’. Because Whitehall is full of people trying to snaffle new territory like a game of RISK, rather than thinking about whether X is a good idea, it is quite easy to slim one’s department down in minor areas. A year later, one would come across some miserable minister in another department muttering in a corridor to an official, ‘I dunno how I got lumbered with this troubled families fiasco but it’s totally knackered, the 120,000 figure itself is off the back of a fag packet for god’s sake, and I’ve got No10 badgering me about a PM announcement on it, I mean my God what can we say…’ We would hurry past – there but for the grace of God and fancy private office footwork…

Officials are not even incentivised to avoid embarrassment for the department. Most officials have been through a cycle of a parliament, usually including quite a few different ministers. They know that disaster, cockup, failure, humiliation, and firing of ministers is normal. They also know that it rarely puts the slightest dent in their day – never mind their career. Many times, we would be leading the news with ‘Gove’s incompetence denounced’ headlines while the lead official for the issue would be spotted pottering home at 4 o’clock, entirely unperturbed. Officials are incentivised to avoid embarrassment for other officials – but embarrassment for ministers is quite another matter, and is often quite handy. After all, a minister weakened is a minister more easily controlled.

They are not incentivised to cut ‘red tape’. Apart from undermining their own role, that would also risk blame when something goes wrong, whereas nobody will blame you for imposing stupid bureaucracy that indirectly kills people or slows everything to a snail’s pace. This is exactly the opposite of how the best organisations, including startups, work. Warren Buffet, who has a HQ of two dozen people, explains the difference:

‘We tend to let our many subsidiaries operate on their own, without our supervising and monitoring them to any degree. That means we are sometimes late in spotting management problems and that both operating and capital decisions are occasionally made with which Charlie and I would have disagreed had we been consulted. Most of our managers, however, use the independence we grant them magnificently, rewarding our confidence by maintaining an owner-oriented attitude that is invaluable and too seldom found in huge organizations. We would rather suffer the visible costs of a few bad decisions than incur the many invisible costs that come from decisions made too slowly – or not at all – because of a stifling bureaucracy… We will never allow Berkshire to become some monolith that is overrun with committees, budget presentations and multiple layers of management. Instead, we plan to operate as a collection of separately-managed medium-sized and large businesses, most of whose decision-making occurs at the operating level. Charlie and I will limit ourselves to allocating capital, controlling enterprise risk, choosing managers and setting their compensation.’

Officials are not incentivised to save money. Some might expect that financial scrutiny would catch out many errors. No. When ministers get clobbered for something, the amount of money wasted is often made public. However, when officials screw something up and are caught before they can turn it into a ministerial screw up, the figures are often hidden. The opaque Whitehall accountancy system is used to shuffle a few million around. Suddenly, from a budget you were told during the spending review could not be cut by one million or the heavens would fall, mysterious millions are found to plug the gap. Ah, the famous Treasury scrutiny, you say? Officials in the Treasury, contra myths, are not interested in controlling costs. HMT officials are interested in their control over Whitehall – not saving taxpayers’ money. 

Apart from the obvious fact that in bureaucracies people do not think about saving money the way startups do, there is also the problem that almost nobody in Whitehall can remember the last time they had to make real cuts – they lived for two decades with ever more money. If you have worked in small businesses (as I have) it is striking how in Whitehall there is no similar mentality about reducing costs. One of the ways this manifests itself is the grotesque over-paying of almost everybody – and the sometimes even more grotesque pay-off culture in which people are given six-figure ‘payoff’ pots of cash for no good reason, and sometimes are swiftly rehired anyway. This drove me mad. It is also hard to tackle except across Whitehall, as there is an obvious collective action problem, and again Cameron showed no interest in action, treating it as ‘like the weather’. This culture of excessive pay not only wastes money but deepens public resentment as the public rightly suspects there is a general attitude of ‘jobs for boys’ in which everyone thinks their turn will come for a cushy berth.

This brings us to a fundamental issue. If they are not incentivised to devise good policy, implement it effectively and rapidly, save taxpayers money and so on – what are they incentivised to do? The answer? Obsess on process. In his new book, the legendary venture capitalist Peter Thiel writes:

‘In the most dysfunctional organizations, signaling that work is being done becomes a better strategy for career advancement than actually doing work (if this describes your company, you should quit now).’

It is sobering to reflect that this definition of ‘dysfunctional organisations’ encompasses a vast amount – maybe the majority – of the work done in the civil service. In good organisations outside Whitehall, people obsess on the quality of their products or service or idea. Inside Whitehall, officials obsess on process. Provided the right people are CCd into emails, the forms are filled in, the (absurd) risk assessment process stuck to etc, all is fine! Shambles on TV? Forget it, normal! Millions wasted? Daily occurrence! Kids are dead? Tragedy – did we fill the forms in right? Minister gone? Who cares, we’re all here! But if you don’t get the process right and instead focus on something irrelevant – say you prioritise rapid exam reform or learning from the latest SCR fiasco rather than keeping the Cabinet Office in the loop – woe betide you, your colleagues will drop you down a hole fast, if people start behaving like that where will it all end! Many officials across Whitehall care far more about not being CCd in to an email than they do about millions of pounds being wasted or thousands of people’s lives being inconvenienced – the former is an insult to their status, while the latter is normal daily life. Many were the complaints to private office that ‘Cummings is cutting us out of decisions again by not CCing us into emails’ from an official whose blunders meant we were again leading the news.

They are also incentivised to stay friends with powerful special interests. It was obvious that many officials regarded staying friends with the unions, campaign groups like NSPCC, and quangos like Ofsted as much more important than doing what we wanted. After all, a minister will probably only last 1-2 years but they might have to deal with Chris Keates for a decade. (Though there are also some heroes on this front who I obviously could not name without blowing up, you know who you are…)

When bureaucracies are in a major crisis and feel they must deliver, they usually do not change their basic wiring. If they are really in a panic, they tend to create systems to subvert their own rules rather than change the rules. For example, in order to get around crazy procurement rules, the US Joint Special Operations Command (the classified end of US special forces) created a separate equipment procurement system (the Special Capabilities Office) working in a silo separate from the usual dysfunctional systems that remained in place – then they classified it so Congress had to leave it alone. Most parts of government do not have these sort of options to escape the horror.


Timescales, planning horizons, and pace.

Fourth, serious problems are caused by a mismatch between the timescale of politicians’ and civil servants’ career demands and the timescale of the problems they are supposed to deal with, which causes a mismatch between two very different planning horizons.

The systems politicians are trying to change, such as pensions or the NHS, usually only display significant changes on a timescale of say 103-104 days (i.e. 3-30 years), partly because a) they often require a mix of substantially different new people and large-scale re-training of existing people, and b) bureaucracies are really bad at ‘training’ even though they discuss it as if it is a magic bullet.

However, the effective planning horizon of No10 is ten days at best (often less than 72 hours) – again about a 104 difference (see above for a similar 104 scale gap facing many MPs and officials). Within a month, supposedly new and ‘top’ priorities can be created and almost forgotten, such as with the riots in 2011 or Scotland recently. Even if you are unwise enough to believe No10’s planning horizon is 102 days my point stands. Even if one considers the timescale of five years between elections, it is too short to make a dint in many big hard problems.

This tension causes problems for business as well as politics. As Larry Page (co-founder of Google) has observed, big public companies are under a lot of pressure to focus on quarterly results and most CEOs don’t survive for more than about five years, while many of the problems they face require a planning horizon beyond this. Rare companies like Google that are able to ignore such pressures and focus on the long-term can, perhaps, only do so because they have an effective monopoly and are not struggling in life-and-death competition. On the other hand, it is interesting that capitalism is often a byword for ‘short-termism’ in the media yet the venture capital industry – about which most in Westminster know nothing – is based on often taking bets with substantially longer planning horizons than the five years of politicians, given that the cash flow required to make a VC investment strike gold often requires significantly more than five years. For example, Peter Thiel, Elon Musk, and Larry Page invest in companies like Palantir, SpaceX , and Planetary Resources on the basis of expected returns that are mostly beyond a decade away.

Parliament has found very few mechanisms to escape this problem and many of the mechanisms that have been found are quiet, very discrete Whitehall fixes on security issues that are anyway inevitably handled differently from normal politics.

Further, nobody is incentivised to solve problems fast. Ministers acquire a reputation for ‘wisdom’ simply by saying about everything ‘sounds very risky let’s not do that’ or ‘let’s add another two years to the timetable’. This limits the chances of embarrassment for the civil service but also means the problem is not solved. Officials are adept at psychologically reinforcing this, by praising ministers as ‘very wise’ whenever they demand delays and ‘very brave’ whenever they demand an aggressive timetable. The cost of going quickly is harder work by, and potential embarrassment for, officials; the costs of going slowly fall on the public. Who do you think weighs more in decisions taken confidentially in Whitehall, without the tradeoffs ever having to be crassly articulated?

Questions about the speed of management are fundamental: Whitehall uses pace to control form. One of our most fundamental problems in the DfE involved the issue of pace and it is intimately connected to the issue of core skills. Sometimes incompetence put planned timescales in doubt. Often, senior officials who did not want to do X fought rearguard campaigns to slow things down and sabotage certain crucial nonlinear milestones – all sorts of things have to happen by date X or else they can’t happen for a year. Stopping last minute attempts by some officials to push something over the timetable edge required constant vigilance. A ‘threat of an EU/ECHR judicial review’ in general and ‘EU procurement rules’ in particular are tools regularly deployed to slow things down.

But one cannot just blame officials – ultimately MPs set their incentives, or allow officials to set their own.


The failure of aggregation.

Fifth, while markets and science have effective methods to aggregate information, aggregation in politics is far from guaranteed to improve decisions and can be destructive. For example, so-called ‘brainstorming’ is proven not to work in politics, partly because psychological aspects of how we evolved to deal with status pervert useful discussion and encourage groupthink. High status people tend to dominate discussion and common information is over-discussed while information unique to an individual, especially a lower status individual, is routinely ignored. The wisdom of crowds only works if many independent judgements are aggregated; if social influence distorts the process, one gets disastrous mobs – not the wisdom of crowds.

Parliaments do not necessarily or reliably perform the same alchemy as the wonders of successful ‘information markets’. As Bismarck reflected on his experience before becoming Prussian prime minster, ‘Looked at individually these people [parliamentary representatives] are in part very shrewd, mostly educated, regular German university culture … [A]s soon as they assemble in corpore, they are dumb in the mass, though individually intelligent.’

The Good Judgement Project and other initiatives are exploring how we might effectively use in politics those aggregation techniques successfully used in other fields.


The failure of core skills.

Sixth, core skills have disintegrated in large parts of the civil service.

Politicians usually operate within institutions, including government departments, that have vastly more ambitious formal goals than the dysfunctional management could possibly achieve. Nevertheless, these dysfunctional entities, in the DfE’s case spending a billion pounds per week, acquire more and more goals in response to media pressure, lobbying from the ecosystem in which they live (and which is fed by them), and MPs’ incentives to maintain the flow of gimmicks. One of the most interesting features of politics is the way in which Insiders see failures daily yet it almost never stops them continuing to expand the organisation’s formal goals.

Many of these bureaucracies cannot reliably do the simplest things. I explained above about the inability to do basic correspondence or fix. Basic spreadsheet skills were so lacking that financial models and budgets could never be trusted and almost every figure released to the media or Parliament was wrong. Legal advice was unreliable and government lawyers are also given the wrong incentive (they are told to prioritise never going to court, which is stupid). Basic project management skills of the sort a world class engineering company routinely deploys are practically non-existent among senior officials. In short, core skills are as healthy in Whitehall as they are in English state schools and the days of Michael Quinlan are long gone.

These problems are compounded by a combination of the growth of public law, judicial review, EU regulation, and the ECHR/HRA, which have added cost, complexity, and uncertainty. There is no objective view of ‘what the law is’ in many circumstances so management decisions are undermined many times per day by advice to do things ‘to avoid losing a judicial review’ the risks of which are impossible to analyse clearly. Legal advice is offered saying that both doing X and not doing X could be ‘illegal’ leading to Kafka-esque discussions and pseudo-‘fair processes’ (like ‘consultations’) designed only to be evidence in court. Internal legal advice makes discussion of regulatory trade-offs tortuous and wasteful; it is always easier to urge ‘caution’ and ‘we’ll lose a JR’ is an easy way across Whitehall to delay or block change.

These problems are largely ignored in Whitehall.

Exhibit A: the former Cabinet Secretary Gus O’Donnell. Unintentionally, Gus O’Donnell often reveals the serious errors of senior mandarins when he gives interviews. He recently discussed problems in Whitehall. ‘Public servants are committed to improving services. They like nothing more than a satisfied customer.’ I’ve already explained why the mismatch of incentives shows this is a fantasy. He goes on: officials ‘would love to have more investment in their creaking IT systems’. As if the problem with Whitehall is not enough money spent on IT and ‘more investment’ would solve the problems! In this one quote, GO’D reveals how little he understands about management. He goes on, ‘Is the solution more bureaucrats and fewer elected politicians? In areas where there is a clear need for a long-term framework, such as energy, infrastructure and planning policy, there is much to be said for the former.’ Ahh, so for long-term policies the answer is ‘more bureaucrats’!

The fundamental reason for Whitehall’s failure is management, not a lack of bureaucrats or money. As Colonel Boyd used to shout, ‘People, ideas, machines – in that order!’ In the DfE, we cut the department’s headcount by more than a third and halved running costs. We more than halved the press office, and cut 95 percent of the communication budget. Performance improved rapidly. It would improve further if the DfE were halved again. The fact that the former head of the civil service could unintentionally reveal such deep misunderstandings about the problems with Whitehall and the nature of management shows how serious the problems are.

Exhibit B. The Institute of Government recently did a report on No10’s structure. It does not explore why implementation and project management is so poor, the huge failure of Whitehall HR policy, and it says nothing I noticed on the issue: how do you know if you’re going wrong? Amusingly, it assumes ‘the efficiency of the administrative machine in 10 Downing Street’ – an assumption that provokes a hollow laugh from those who have to deal with it.

An example that combines issues of transparency, legal issues, and timescale. Senior officials initially hated our commitment to put all the exam information in the National Pupil Database into the public domain and strip ‘equivalents’ out of the league tables. Why? Partly because they disagreed with us about equivalents but mainly because making the information transparent took power from Whitehall and gave it to the public, and they rightly knew that it would be practically impossible for them to reverse (Labour will struggle to argue that exam data should be secret again). When they came up with their first timetable for implementing this policy, it read ‘Delivery 2019-21‘. We said – do it now. They said – legal issues, data protection, judicial review, blah blah. We did it in 2011/12 (thanks to Henry de Zoete who pursued it relentlessly despite the fact that as media spad the effect of greater transparency was to destroy more of his weekends).


Lack of internal criticism and external competition.

Seventh, Whitehall suffers from a lack of internal mechanisms to enforce honesty about errors and a lack of external competition.

No Red Teams and ‘lessons learned’. There is rarely any serious formal process for testing rigorously before policies are launched. ‘Red Teams’ are a traditional answer. Often they have worked. For example, between the world wars the Germany Army examined British exercises with armoured divisions and asked themselves, ‘how might this affect future war?’ and insights helped develop von Manstein’s ‘Blitzkrieg’. Often, they have been ignored or even suppressed. Japan’s wargaming before Pearl Harbor assumed carriers would continue to be peripheral and in its planning for Midway, Rear Admiral Ugaki repeatedly overruled umpires whenever they made a decision that cast doubt on the ability of the Japanese navy to execute its plans. Classified Pentagon wargames 1963 – 1965 (the SIGMA series) predicted that the main elements of US strategy in Vietnam would fail. They were ignored. The report of the Senate Select Committee on Intelligence re the CIA, Iraq and WMD concluded: ‘The presumption that Iraq had active WMD programs was so strong that formalized mechanisms established to challenge assumptions and ‘group think’, such as ‘red teams’ … were not utilized.’

It is very hard to ‘learn lessons’. David Galula’s fascinating account of his successful counter-insurgency, ‘Pacification in Algeria 1956-8’, discussed how hard it was for armies to remember ancient and modern lessons in this field – and was itself promptly forgotten not only by the Americans (who commissioned it) in Vietnam but for the next forty years until 9/11. McMaster wrote a study of LBJ’s failures in Vietnam (‘Dereliction of Duty’); the suppression of bad news was central. McMaster fought in Iraq in 2003 and saw for himself similar errors repeated. He tried new tactics (small bases embedded in, and helping, the population). He was repeatedly passed over for promotion as superiors suppressed bad news. The reluctance of the NASA bureaucracy to face facts viz the Challenger disaster (1986), the ‘PowerPoint festival of bureaucratic hyperrationalism’, and Feynman’s famous pursuit of the facts and exposure of groupthink (which brought the comment from the head of the investigation that ‘Feynman is becoming a real pain’), were followed by the Columbia disaster (2003) and another report showing NASA had not learned lessons from the previous disaster, and that internal pressure to conform meant ‘it is difficult for minority and dissenting opinions to percolate up through the agency’s hierarchy’. Political disasters are rarely analysed carefully. E.g. Many doubted that the euro’s institutions would work (e.g. Feldstein HERE and HERE and even the ECB’s own Otmar Issing). European elites not only rejected such warnings but treated them as the views of the idiotic or malign, and such has been the emotional commitment (cf. Habermas’ ‘Why Europe Needs a Constitution’) that it is still hard for those elites to consider the euro’s / EU’s problems rationally.

In the DfE, officials would often refuse to have a proper look into the origins of a blunder. They would say that we could not look at all the documents on the grounds that ‘we must protect the convention that current ministers cannot look at the papers for previous governments’. This is very handy as under the cloak of ‘political impartiality’ officials prevent ministers getting to the bottom of complex long-term debacles. E.g. we were forbidden from seeing various documents about capital pre-2010 on the basis of ‘impartiality’ but when we insisted / tricked our way in, we found many cockups that Whitehall simply did not want revealed to anybody.

Many accident reports, from air crashes to Fukushima, show that reflexive obedience to the chief lies behind fatal errors. Work by surgeons such as Gawande on checklists, on the other hand, shows how they can change cultures profoundly so that everybody starts correcting lots of small errors, leading to large performance improvements. In the DfE I tried to introduce this idea and get people to consider the extensive literature. There was great hostility mainly from older people (some young officials were enthusiastic and helped): ‘Social work isn’t like flying a plane Dominic, it’s far more complicated’. Bad answer. It is normal for domains to resist being told told by outsiders – ‘your domain is bad at dealing with errors and you need to learn from others’. It is particularly damaging when the bureaucracy that sets rules for other domains thinks like this itself.

Warren Buffett has proposed institutionalising Red Teams to limit damage done by egomaniac CEOs pursuing flawed mergers and acquisitions: ‘it appears to me that there is only one way to get a rational and balanced discussion. Directors should hire a second advisor to make the case against the proposed acquisition, with its fee contingent on the deal not going through’. This seems to me to be a great idea and MPs and Permanent Secretaries should think hard about how to operationalise it in Whitehall.

High barriers of entry, little competition

Barriers of entry are so high in politics that there is little competition and the system is hard for outsiders to disrupt. It is implicit in our method of parliamentary democracy that the contest between the parties will roughly serve the public interest as parties are incentivised to correct the obvious errors of their opponents, offer the public what they want, and thereby gain power, so that atavistic instincts are roughly channelled in ways that help society. This works in the sense that, at least where democratic institutions and the rule of law are embedded, elections stop parties and their leaders from becoming too extreme in the sense of undermining the basic principles of a market-based democracy. However, this incentive system is very indirect and ineffective beyond this basic function.

Further, Whitehall has such a tight grip on the MPs that it chokes off attempts to change the basic wiring of the system. MPs have willingly handed control of vast powers to officials. For example, in a Jedi move, Heywood convinced Cameron and Llewellyn early that everything they do should be monitored by Sue Gray and her ‘ethics committee’ so No10 has now officially outsourced judgement of its own ethics. This of course gives officials huge, hidden, and unaccountable power. Heywood can give the thumbs up or thumbs down to Cameron himself on all sorts of sensitive issues (e.g. which billionaire came for dinner and was it all above board). In No10 now, Sue Gray herself decides what meetings she attends to monitor everyone’s ethics, forcing terrified spads and ministers to flee the building to have certain meetings. In my experience, these developments help dishonest coverups. Because MPs have such little moral authority and such little self-confidence (another vicious circle), they are easily beaten back if they kick up a fuss about something. (And remember, no press office or spin doctor lies to the media as routinely or successfully as the Cabinet Office does over ‘ethical’ issues.)

Given very high barriers to entry and little competition, profound failure can continue undisturbed for years in the absence of large shocks.


Given all this, what do MPs do all day? Media manipulation, not operational planning on priorities.

Unsurprisingly, most senior MPs in all three parties are locked into a game in which they spend most of their time on a) launching gimmicks, and b) coping with crises. These two forms of activity are closely related. The only widely understood model of activity in Westminster (and one which fits well psychologically with the desire for publicity) is a string of gimmicks aimed to manipulate the media (given the label ‘strategy’ to make it sound impressive) which are announced between, and in response to, media crises, some of which are trivial and some of which reflect structural problems. Many, drawing perhaps only on the bluffing skills rewarded by PPE, have no idea what else to do.

Powerful people rush from meetings about the latest gimmick they are to announce, to meetings about the latest cockup for which they need to try to dodge the blame (possibly caused directly by a previously announced gimmick), to the TV studio, to dinner parties, where they gossip about either a) the daily crisis, or b) vague speculations about the distant future (and give overconfident predictions that are usually wrong but which they later reimagined to have been right – ‘as I’ve always said…’). Ministers’ time is dominated by unfocused panic about the media environment – not focused urgency about the most important problems.

These gimmicks have obvious costs in the form of money wasted and the ostensible goal unfulfilled. They also have indirect costs that are often higher. 1) They divert the bandwidth of senior people from serious issues. (For example, dealing with No10 gimmicks diverted DfE ministers, spads, and officials from focusing on serious issues such as child protection.) 2) Once announced, they can easily trigger a set of further stupid decisions as the system attempts to evade the humiliation of the gimmick failing. While many outside Westminster assume there must be some ‘purpose’ or ‘strategy’ to the gimmick, often the truth is it exists purely to be briefed to the media – it is not even intended as a serious idea, and indeed such gimmicks are often soon forgotten even by their inventors.

Ironically, since their only purpose is usually ‘communication’ and ‘sending signals’, they are usually also useless as communication devices and are simply white noise to the public who are watching game shows or football instead. The tiny amount of political communication in Britain that gets through to the public is often accidental (e.g. ‘if it isn’t hurting it isn’t working’, ‘hug a hoodie’). Few phrases are more common than ‘we need a campaign to…’ but few things are rarer than a professional campaign that changes millions of people’s opinions or feelings. So-called ‘strategic communication’ is rarely attempted, never mind done, partly because it requires a lot of hard thinking, focus, priorities, facing weaknesses etc – i.e. many things that are psychologically difficult. Most of what people call ‘strategic communication’ is really just answering phone calls from journalists. In crises, almost everyone panics and spins stories about ‘strategy’ to journalists whilst its practice dissolves if it ever existed (unlikely). The subject is widely discussed in defence and intelligence circles but also rarely well executed. E.g. The Pentagon knows that the huge amount of effort it has put into ‘information operations’ did not work in Afghanistan and Iraq (Report).

Amazingly little Whitehall discussion ever involves concrete operational planning to advance priorities from A to Z (weekly / monthly / quarterly). Why? Because most senior people have no idea about how to go about such planning and it is not incentivised as I explained above. On one hand, many take pride in not having a plan, an attitude with deep roots in the Tory party: ‘I distrust anyone who foresees consequences and advocates remedies for the avoidance of them’ (foreign secretary Lord Halifax before the war). Many think that Macmillan’s ‘events, dear boy, events’ is a ‘how to prioritise’ guide.  On the other hand, politics is dominated by discussion of ‘strategy’ and ‘priorities’, but few know how to think strategically (or even what ‘strategy’ is) or how to make and stick to priorities. Misunderstanding of strategy, and the proliferation of rhetoric masquerading as strategy, causes huge problems, including with national leaderships’ attempts to define ‘national strategy’. (** See endnote.)

This is a huge gap in Whitehall but the system has gone so wrong few even realise the gap is there and those who do cannot do anything about it.

Most media commentary on politics therefore enormously overstates the extent to which news derives from ‘plans’ and understates the extent to which news derives either from, first, panic driven by chaos exacerbated by lack of operational grip, and, second, unthought out gimmicks aimed only at shaping the media environment for a day or two. Whenever I read commentators explaining to the public things involving Whitehall, particularly No10, that I have been involved in, they always assume an average level of ‘planning’ much higher than actually existed and they assume processes of analysis and discussion that seldom happened. Commentators are always looking for specific things as explanatory factors but the reality is that similar things keep happening in very similar ways because of general features of the political system. Often a focus on specifics clouds understanding. Events are over-interpreted because journalists do not want to face the idea that they are usually spectators of over-promoted people floundering amid chaos – actions must be intended (‘their strategy is…’), farcical reality must be tarted up. (I will explore this subject separately.)

To some extent democratic politics is always going to involve gimmickry. My point is that the British state has degenerated to the point where that’s about all there is and the public increasingly understand that’s all there is.


No10: The horror, the horror of the Random Announcement Generator…

There’s a wonderful scene in Book II of War and Peace. The cynical diplomat, Bilibin, is explaining the latest disaster against Napoleon, a tragicomic story in which the Austrians accidentally gave away the Tabor bridge to the French because of the manifestation of a general, systemic dysfunction in General Mack’s army.

“‘It’s not treason, or dastardliness, or stupidity: it’s the same as at Ulm… it is…’ – he seemed to be trying to find a suitable expression. ‘It’s … c’est du Mack. We’ve been Macked,’ he concluded, feeling that he had coined a word, a new word that would be repeated.’ (p. 186, Edmonds translation.)

Britain, too, gets ‘Macked’ every week.

Cameron requires no psychological analysis. He is one of the most straightforward people one will meet in politics. Pundits have wasted millions of words on what they regard as his ‘mystery’ but he is exactly what he seems – he is, as Bismarck said of Napoleon III, a ‘sphinx without a riddle’. He’s cleverer than most MPs and can hold his own in conversations with senior officials with whom he has a lot in common intellectually. He may be in the top two percent (+2 standard deviations) for verbal skills but has none of the expertise or experience necessary for managing very complex processes and solving hard problems. He does not dig into the details of policy. His self-assurance has some positive aspects (he is not intimidated or destroyed by the size of the job) but also big negative aspects. One could still be an OK prime minister with this combination of characteristics if one had great judgement about people but his worst characteristic as PM is his awful judgement about senior advisers (Coulson***, Llewellyn, Rock, Oliver) as even his closest friends accept. If he had the self-awareness to consider his senior appointments and hire alpha people, then faced with Miliband he would likely win easy.

Why is he there? Because 1) Cameron’s 2005 rival was David Davis who over a long campaign scared too many MPs about his temperament, 2) Blair blew up over the Middle East making Cameron’s rival Brown, 3) Cameron is superficially suitable for the job in the way that ‘experts’ often judge such things – i.e. basic chimp politics skills, height, glibness etc, so we can ‘shove him out to give a statement on X’. That’s it. In a dysfunctional institutional structure, someone without the skills we need in a prime minister can easily get the job with a few breaks like that.

Cameron regards his job as like a steward in charge of the ‘ship of state’ – his job is not to crash it into the rocks. His main method for doing this is to implement what he is told by senior civil servants who suffer a severe lack of cognitive diversity. This has the advantage of making life much easier, as the heels click and the salutes snap to attention even if everything is going to pot, whereas fighting official conventional wisdom has high costs. He has exasperated and depressed many with his ‘so what do I believe in this week’ approach. In doing this job, he regards his Party with a mix of contempt and anger. (He has thought that his many critics will not launch a coup because of a mixture of cowardice and greed for red boxes and chauffeurs – so far they have not.) Cameron and Llewellyn regard the optimal outcome of the next election as a similar outcome to last time – a hung Parliament with Clegg and Miliband weakened. They regard a large majority as impossible and a small majority as a nightmare. They do not have ambitions to ‘solve the EU problem’ or ‘make the NHS worldclass’ – it is not how they think about the world. This is not itself a criticism – it is not necessarily a virtue to have bold ambitions. Rather than criticising him for a lack of ambition, it is more accurate and fairer to criticise him simply for putting his own personal interests ahead of the public interest. His party regards him as untrustworthy and selfish – they suspect he does not want a majority and does not care if the Party implodes the day after he walks away, but they also worry no other current MP can give them a majority. As they say in Moscow, ‘everybody’s right and everybody’s unhappy’.

If you want to understand why the news is what it is, remember that Cameron and his two most senior advisers – Ed Llewellyn and Craig Oliver – are rushing from gimmick to dinner party to gimmick to dinner party. They do not engage in serious operational planning. Why? a) They have no idea what it looks like – it is an alien concept. b) Their model for political activity is as described above – a string of gimmicks. Oliver regards his job as fire-hosing stories at the lobby and coping with perpetual cockups. (I feel sorry for Oliver. He should never have been put in this job for which he is entirely unsuitable.) Llewellyn regards his job as helping Whitehall and the EU do what they want while keeping MPs quiet, keeping Clegg happy, and coping with perpetual cockups.

The hierarchy of problems that our DfE team faced was (biggest problems first): some of our own officials, Downing Street, the BBC, Labour and the unions. No10 is supposed to work now on the basis of controlling ‘The Grid’, a compilation of Whitehall’s announcements. However, their ‘grid’ was more like a malfunctioning Random Announcement Generator – input sense, output nonsense. If Cameron/Oliver got an iPad app for their Grid, they could shake the iPad up and down and all the different stories could randomly bounce into new slots. Shake shake shake – here’s a plan! Shake shake shake – here’s another plan! Just as good! Nobody would notice the difference with how it is done now. (This was not the fault of junior people like Ameet but of the most senior people.)

If we told them what we were doing, it would either leak or they would chime in with appalling ideas. Llewellyn only appeared on our radar to tell us to give in either to Whitehall or to Clegg. It was extremely difficult being stuck between a) internal opponents working with b) Clegg, Llewellyn, and the Cabinet Office, and meant that we were constantly faced with the need to adopt extreme measures in order to make progress. Many things we did were sub-optimal because of the need to smuggle them into existence without Cameron, Clegg, or Llewellyn knowing about them. (Some No10 people, such as James O’Shaughnessy, did help us and deserve credit.)

I will go into this in future blogs but here is an example of what I mean about the way No10 did not take school reform seriously and could not be engaged with in a serious way on policy. Between Gove getting the job in 2007 and January 2014, how many meetings do you think happened between a) Cameron and his senior policy advisers, and b) Gove and his senior policy advisers to discuss schools policy? If quarterly, then about 25-30? Answer: two. One in 2009, one in 2011. However, this was a good thing. It meant that No10 largely left us alone for long periods. Whenever No10 sent word that ‘the PM is thinking of making an intervention’, it guaranteed 100 percent that the horror, the horror, would descend.

One mechanism we devised to deal with this concerned The Grid / RAG. Once we established some grip of the DfE over 2011/12, I kept three timetables. 1) Our real plan. This was shared among less than 10 people. 2) An internal DfE plan which excluded only sensitive things like personnel moves. This was not shared with No10. 3) A ‘No10 friendly’ plan, which had everything important removed in order to keep them in the dark. (There were exceptions. We worked quietly with some No10 people who knew we were right about Llewellyn and Oliver and we shared information with them to help them out, but strictly on the basis that Llewellyn and Oliver would not be told.) The Random Announcement Generator can also be turned to good effect. Monnet created the EU by always having a plan in his pocket for when disaster hit. ‘Oh you’ve hit a crisis – here’s my plan for the European Coal and Steel Community.’ In a tiny way, we tried to do the same, as I will explain another time.

One last story that connects some of these themes. In summer 2013, Clegg and Danny Alexander tried to stop the next wave of Free Schools being announced. Clegg had become progressively keener on using this regular media event to spin stories suggesting he was hostile to Free Schools and Gove (all the time in private obviously telling us that ‘of course I support Free Schools but I’ve got to do something about the optics‘). He had tried to interfere with the process of selecting Free Schools but we had told him No Way (using some civil service jiujitsu with ‘judicial review’). Now, he used the Treasury to block the announcement with Danny Alexander as the instrument. No10 sided with Clegg and DA. ‘But this is long-arranged, if we cancel it it will hit thousands of people directly.’ ‘The PM wants to keep Clegg happy.’ ‘But it will be a disastrous story, “Government drops Free Schools”, surely he won’t want that.’ ‘Arghhhh, yes, but the PM thinks we can sneak through that story, and he’s promised Clegg.’ Ok. So I announced the Free School round anyway by the simple expedient of sending out the press release and it rolled out in the media in the usual way, sending Clegg and various mandarins into a meltdown. My logic: we won’t trash all the Free School groups we had encouraged to apply because of Clegg’s ‘optics’, and because Cameron is so desperate to prop him up and so careless of real things and people that he will not overrule him, as he easily could do if he had priorities. (The idea that Cameron had some amazing Grand Bargain in return, as Llewellyn would comically try to claim now and then, was obviously rubbish. When Cameron caved in on abolishing GCSEs in 2012, he didn’t even ask for anything in return.) There are many interesting aspects of this story that I’ll explore another time but it demonstrates various layers of problem and illustrates why I think so strongly that a priority must be to remove MPs’ whims from the management of schools.

No10 does not even realise it has to focus on priorities, so of course it does not notice that it cannot project manage them through the system, or that the senior officials they trust to do this for them also cannot do it. No10 and the Cabinet Office are themselves a major source of chaos so it no surprise that the rest of government is in permanent omnishambles. Cameron makes clear to Heywood and other Permanent Secretaries that he has no interest in civil service reform so of course nothing serious changes. Cameron’s time is spent on tactical media manipulation but the person he has hired to do this for him does not know how to do it and even someone who did know how to do it would be subject to the daily litany of cockups because they are an inevitable outcome of systemic dysfunction.

The occupants of No10, like Tolstoy’s characters in War and Peace, are blown around by forces they do not comprehend as they gossip, intrigue, and babble to the media. The MPs and spin doctors steer their priorities according to the rapidly shifting sands of the pundits who they are all spinning, while the pundits shift (to some extent unconsciously) according to the polls. The outcome? Everybody rushes around in tailspins assembling circular firing squads while the real dynamics of opinion play out largely untouched by their conscious actions. In terms of a method to ‘manage’ government, it is not far from tribal elders howling incantations around the camp fire after inspecting the entrails of slaughtered animals. It makes no sense because it is not based on the real world. Because of this systemic dysfunction, the rest of us get repeatedly ‘Macked’.


The combination of 1) evolved mental characteristics, 2) poor education and training, and 3) a dysfunctional institutional architecture, combined with a) inherent uncertainty and wrong predictions, and b) the inherent difficulty of adapting amid the stormy chaos of events where the simplest things are hard and failure is ubiquitous, creates a series of vicious feedback loops.

We do not have a problem with ‘too much cynicism’ – we have a problem with too much trust in people and institutions that are not fit to control so much. When faced with the ‘fog of war’ in nonlinear systems such as the financial system, disease outbreaks, or terrorism, the current system is absolutely bound to respond with sloth/panic, chaos, and blunders.

Our leaders are like 19th Century Germans who had lost religion of whom Nietzsche said, ‘they merely register their existence in the world with a kind of dumb amazement’. They get up every day and react to the media without questioning why: sometimes they are lauded, usually they are trashed, but they carry on in a state of ‘dumb amazement’ without realising how absurd their situation is. Meanwhile, the institutions within which they operate continue with their own momentum and dynamics, and they pretend to themselves that they are, in the phrase they love, ‘running the country’.

But the phrase is hollow, hollow, hollow…

[Coming soon… What is to be done?]

A Fermi estimate of the number of really dangerous people. The global population of people with an IQ four standard deviations above the average (i.e. >160) is ~250k. About 1% of the population are psychopaths so there are perhaps ~2-3,000 with an IQ roughly that of a Nobel physics or Fields prize winner. The psychopathic population with an IQ over three standard deviations (>145, where the average science PhD ~130) is 30 times bigger. A subset of these people will also be practically competent. Some of them may think, ‘Flectere si nequeo superos, / Acheronta movebo’ (‘If Heav’n thou can’st not bend, Hell thou shalt move’). Board et al (2005) showed that high-level business executives are more likely than inmates of Broadmoor to have one of three personality disorders (PDs): histrionic PD, narcissistic PD, and obsessive-compulsive PD. Mullins-Sweatt et al (2010) showed unsurprisingly that successful psychopaths are more conscientious than the unsuccessful.

** ‘Strategy’ is much mentioned but little studied. Strategy is not ‘goals’, ‘vision’ or rhetoric. Strategy focuses action on crucial problems to connect operations to aims; it requires diagnosis, a guiding policy, and coherent action. Good strategy requires choices, choices require not doing some things, and some people will be upset at not being ‘a priority’; therefore, good strategy is by definition hard for politicians to articulate even if they can develop it. Bad strategy is identified by: fluff (vague, grandiloquent rhetoric), ignoring important problems, mistaking goals for strategy, and setting bad (or contradictory) ‘strategic objectives’. It is not miscalculation. It is sometimes a substitution of belief for thought. Now it is often produced via a powerpoint template, with visions, mission statements, core values, strategic goals, lists of initiatives etc – all avoiding the hard questions (Rumelt, 2011).

Clausewitz described military strategy as ‘the use of the engagement for the purpose of the war’ and says the strategist ‘must therefore define an aim for the entire operational side of the war that will be in accordance with its purpose.’ Colin Gray defines military strategy as ‘the direction and use that is made of force and the threat of force for the ends of policy’. The first use of ‘strategy’ in a sense beyond narrow generalship was in 1777 in French and German, and prior to 1810 English dictionaries did not contain a ‘strategy’ entry. ‘Strategy was not recognized linguistically as a function distinctive from statecraft or generalship prior to the late 18th century. Polities did not have permanent or even temporary schools and military staff charged with “strategic” duties. Policy and strategy, though logically separable, usually were all but collapsed one into the other.’ (Gray, Schools for Strategy, 2009).

I think the word has become so confused and confusing that outside specialist groups it should be abandoned. In DfE meetings, I tried to stop people using the word ‘strategy’ as it was guaranteed to confuse discussion. If you watch people in Westminster using the word, it is used interchangeably for ‘goal’, ‘plan’, ‘tactics’ etc.

*** Coulson and ‘spin’. Recently quite a few commentators have said about Coulson ‘at least as he was very good at his job’, ‘he understood the dark arts’. This is wrong. (The ‘arts’ are not ‘dark’ in the sense of mysterious, but I’ll leave that for now.) The pro-Coulson argument is: he knew what ‘a story’ is, he was not a clown, and he did not go to Eton. This does not make him a good Director of Communications. I don’t think Coulson was even good at spinning stories but my point is different – it is that even being a good spin doctor is not at all the same as being a good campaign manager or director of communications. Further, being a good spin doctor is not even a necessary condition for being a good campaign manager. A good DoC has priorities, a plan, and an effective machine. Coulson had none of these things. A good DoC is not focused on the daily media but on long-term goals. Coulson encouraged Cameron in one of his worst traits – to obsess about press coverage and behave like a pundit surfing the news rather than a leader. Like with Oliver, I do not blame Coulson for this – he was the wrong person for the job as would have been obvious except Cameron himself does not understand what the job is and simply wanted a ‘spin doctor’ close to News International. Britain now has a tendency to hire journalists to run communications which is not what happens in the more professional US environment where they know that journalists seldom have the right skills to run a large communication operation. NB. I do not say this because of any personal grudge with Coulson. Contra many reports, I never had any arguments or fall-outs with him. I doubt we exchanged 1,000 words in three years. He objected to me going into the DfE not because of any row but because he thought that I would not take orders from him or Llewellyn. Llewellyn agreed with him. They were right.