‘If you don’t get this elementary, but mildly unnatural, mathematics of elementary probability into your repertoire, then you go through a long life like a one-legged man in an ass-kicking contest. You’re giving a huge advantage to everybody else. One of the advantages of a fellow like Buffett … is that he automatically thinks in terms of decision trees and the elementary math of permutations and combinations… It’s not that hard to learn. What is hard is to get so you use it routinely almost everyday of your life. The Fermat/Pascal system is dramatically consonant with the way that the world works. And it’s fundamental truth. So you simply have to have the technique…
‘One of the things that influenced me greatly was studying physics… If I were running the world, people who are qualified to do physics would not be allowed to elect out of taking it. I think that even people who aren’t [expecting to] go near physics and engineering learn a thinking system in physics that is not learned so well anywhere else… The tradition of always looking for the answer in the most fundamental way available – that is a great tradition.’ Charlie Munger, Warren Buffet’s partner.
During the ten week official campaign the implied probability from Betfair odds of IN winning ranged between 60-83% (rarely below 66%) and the probability of OUT winning ranged between 17-40% (rarely above 33%). One of the reasons why so few in London saw the result coming was that the use by campaigns of data is hard to track even if you know what to look for and few in politics or the media know what to look for yet. Almost all of Vote Leave’s digital communication and data science was invisible even if you read every single news story or column ever produced in the campaign or any of the books so far published (written pre-Shipman’s book).
Today we have made a software product available for download – Vote Leave’s ‘Voter Intention Collection System’ (VICS) – click HERE. It was named after Victoria Woodcock, Operations Director, known as Vics, who was the most indispensable person in the campaign. If she’d gone under a bus, Remain would have won. When comparing many things in life the difference between average and best is say 30% but some people are 50 times more effective than others. She is one of them. She had ‘meetings in her head’ as people said of Steve Wozniak. If she had been Cameron’s chief of staff instead of Llewellyn and Paul Stephenson had been director of communications instead of Oliver and he’d listened to them, then other things being equal Cameron would still be on the No10 sofa with a glass of red and a James Bond flick. They were the operational/management and communications foundation of the campaign. Over and over again, those two – along with others, often very junior – saved us from the consequences of my mistakes and ignorance.
Among the many brilliant things Vics did was manage the creation of VICS. When we started the campaign I had many meetings on the subject of canvassing software. Amazingly there was essentially no web-based canvassing software system for the UK that allowed live use and live monitoring. There have been many attempts by political parties and others to build such systems. All failed, expensively and often disastrously.
Unfortunately, early on (summer 2015) Richard Murphy was hired to manage the ground campaign. He wanted to use an old rubbish system that assumed the internet did not exist. This was one of the factors behind his departure and he decided to throw in his lot with Farage et al. He then inflicted this rubbish system on Grassroots Out which is one of the reasons why it was an organisational/management disaster and let down its volunteers. After Vote Leave won the official designation, many GO activists defected, against official instructions from Farage, and plugged into VICS. Once Murphy was replaced by Stephen Parkinson (now in No10) and Nick Varley, the ground campaign took off.
We created new software. This was a gamble but the whole campaign was a huge gamble and we had to take many calculated risks. One of our central ideas was that the campaign had to do things in the field of data that have never been done before. This included a) integrating data from social media, online advertising, websites, apps, canvassing, direct mail, polls, online fundraising, activist feedback, and some new things we tried such as a new way to do polling (about which I will write another time) and b) having experts in physics and machine learning do proper data science in the way only they can – i.e. far beyond the normal skills applied in political campaigns. We were the first campaign in the UK to put almost all our money into digital communication then have it partly controlled by people whose normal work was subjects like quantum information (combined with political input from Paul Stephenson and Henry de Zoete, and digital specialists AIQ). We could only do this properly if we had proper canvassing software. We built it partly in-house and partly using an external engineer who we sat in our office for months.
Many bigshot traditional advertising characters told us we were making a huge error. They were wrong. It is one of the reasons we won. We outperformed the IN campaign on data despite them starting with vast mounts of data while we started with almost zero, they had support from political parties while we did not, they had early access to the electoral roll while we did not, and they had the Crosby/Messina data and models from the 2015 election while we had to build everything from scratch without even the money to buy standard commercial databases (we found ways to scrape equivalents off the web saving hundreds of thousands of pounds).
If you want to make big improvements in communication, my advice is – hire physicists, not communications people from normal companies and never believe what advertising companies tell you about ‘data’ unless you can independently verify it. Physics, mathematics, and computer science are domains in which there are real experts, unlike macro-economic forecasting which satisfies neither of the necessary conditions – 1) enough structure in the information to enable good predictions, 2) conditions for good fast feedback and learning. Physicists and mathematicians regularly invade other fields but other fields do not invade theirs so we can see which fields are hardest for very talented people. It is no surprise that they can successfully invade politics and devise things that rout those who wrongly think they know what they are doing. Vote Leave paid very close attention to real experts. (The theoretical physicist Steve Hsu has a great blog HERE which often has stuff on this theme, e.g. HERE.)
More important than technology is the mindset – the hard discipline of obeying Richard Feynman’s advice: ‘The most important thing is not to fool yourself and you are the easiest person to fool.’ They were a hard floor on ‘fooling yourself’ and I empowered them to challenge everybody including me. They saved me from many bad decisions even though they had zero experience in politics and they forced me to change how I made important decisions like what got what money. We either operated scientifically or knew we were not, which is itself very useful knowledge. (One of the things they did was review the entire literature to see what reliable studies have been done on ‘what works’ in politics and what numbers are reliable.) Charlie Munger is one half of the most successful investment partnership in world history. He advises people – hire physicists. It works and the real prize is not the technology but a culture of making decisions in a rational way and systematically avoiding normal ways of fooling yourself as much as possible. This is very far from normal politics.
(One of the many ways in which Whitehall and Downing Street should be revolutionised is to integrate physicist-dominated data science in decision-making. There are really vast improvements possible in Government that could save hundreds of billions and avoid many disasters. Leaving the EU also requires the destruction of the normal Whitehall/Downing Street system and the development of new methods. A dysfunctional broken system is hardly likely to achieve the most complex UK government project since beating Nazi Germany, and this realisation is spreading – a subject I will return to.)
In 2015 they said to me: ‘If the polls average 50-50 at the end you will win because of differential turnout and even if the average is slightly behind you could easily win because all the pollsters live in London and hang out with people who will vote IN and can’t imagine you winning so they might easily tweak their polls in a way they think is making them more accurate but is actually fooling themselves and everybody else.’ This is what happened. Almost all the pollsters tweaked their polls and according to Curtice all the tweaks made them less accurate. Good physicists are trained to look for such errors. (I do not mean to imply that on 23 June I was sure we would win. I was not. Nor was I as pessimistic as most on our side. I will write about this later.)
VICS allows data to be input centrally (the electoral roll, which in the UK is a nightmare to gather from all the LAs) and then managed at a local level, whether that be at street level, constituency or wider areas. Security levels can be set centrally to ensure that no-one can access the whole database. During the campaign we used VICS to upload data models which predicted where we thought Leave voters were likely to be so that we could focus our canvassing efforts, which was important given limited time and resources on the ground. The model produced star ratings so that local teams could target the streets more likely to contain Leave voters.
Data flowed in on the ground and was then analysed by the data science team and integrated with all the other data streaming in. Data models helped us target the ground campaign resources and in turn data from the ground campaign helped test and refine the models in a learning cycle – i.e. VICS was not only useful to the ground campaign but also helped improve the models used for other things. (This was the point of our £50 million prize for predicting the results of the European football championships, which gathered data from people who usually ignore politics – I’m still frustrated we couldn’t persuade someone to insure a £350 million prize which is what I wanted to do.) In the official 10 week campaign we served about one billion targeted digital adverts, mostly via Facebook and strongly weighted to the period around postal voting and the last 10 days of the campaign. We ran many different versions of ads, tested them, dropped the less effective and reinforced the most effective in a constant iterative process. We combined this feedback with polls (conventional and unconventional) and focus groups to get an overall sense of what was getting through. The models honed by VICS also were used to produce dozens of different versions of the referendum address (46 million leaflets) and we tweaked the language and look according to the most reliable experiments done in the world (e.g. hence our very plain unbranded ‘The Facts’ leaflet which the other side tested, found very effective, and tried to copy). I will blog more about this.
These canvassing events represented 80-90% of our ground effort in the last few months, hence some of the reports by political scientists derived from Events pages on the campaign websites, which did not include canvassing sessions, are completely misleading about what actually happened (this includes M Goodwin who is badly confused and confusing, and kept telling the media duff information after he was told it was duff). There was also a big disinformation campaign by Farage’s gang, including Bone and Pursglove, who told the media ‘Vote Leave has no interest in the ground campaign’. This was the opposite of the truth. By the last 10 weeks we had over 12,000 people doing things every week (we had many more volunteers than this but the 12,000 were regularly active). When Farage came to see me for the last time (as always fixated only on his role in the debates and not the actual campaign which he was sure was lost) he said that he had 7,000 activists who actually did anything. He was stunned when I said that we had over 12,000. I think Farage et al believe their own spin on this subject and were deluded not lying. (Obviously there was a lot of overlap between these two figures.) These volunteers delivered about 70 million leaflets out of a total ~125 million that were delivered one way or another.
While there were some fantastic MPs who made huge efforts on the ground – e.g. Anne Marie Trevelyan – it was interesting how many MPs, nominally very committed to Leave, did nothing useful in their areas nor had any interest in ground campaigning and data. Many were far more interested in trying to get on TV and yapping to hacks than in gathering useful data, including prominent MPs on our Board and Campaign Committee, some of whom contributed ZERO useful data in the entire campaign. Some spent much of the campaign having boozy lunches with Farage gossiping about what would happen after we lost. Because so many of them proved untrustworthy and leaked everything I kept the data science team far from prying eyes – when in the office, if asked what they did they replied ‘oh I’m just a junior web guy’. It would have been better if we could have shared more but this was impossible given some of the characters.
VICS is the first of its kind in the UK and provided new opportunities. It is, of course, far from ideal. It was developed very quickly, we had to cut many corners, and it could be improved on. But it worked. Many on the ground, victims of previous such attempts, assumed it would blow up under the pressure of GOTV. It did not. It worked smoothly right through peak demand. This was also because we solved the hardware problem by giving it to Rackspace which did a great job – they have a system that allows automatic scaling depending on the demand so you don’t have to worry about big surges overwhelming the system.
There were many things we could have done much better. Our biggest obstacle was not the IN campaign and its vast resources but the appalling infighting on our own side driven by all the normal human motivations described in Thucydides – fear, interest, the pursuit of glory and so on. Without this obstacle we would have done far more on digital/data. Having seen what is offered by London’s best communications companies, vast improvements in performance are clearly possible if you hire the right people. A basic problem for people in politics is that approximately none have the hard skills necessary to distinguish great people from charlatans. It was therefore great good fortune that I was friends with our team before the campaign started.
During the campaign many thousands of people donated to Vote Leave. They paid for VICS. Given we spent a lot of money developing it and there is nothing equivalent available on the market and Vote Leave is no more (barring a very improbable event), we thought that we would make VICS available for anybody to use and improve though strictly on the basis that nobody can claim any intellectual property rights over it. It is being made available in the spirit of the open source movement and use of it should be openly acknowledged. Thanks again to the thousands of people who made millions of sacrifices – because of you we won everywhere except London, Scotland and Northern Ireland against the whole Government machine supported by almost every organisation with power and money.
I will write more about the campaign once the first wave of books is published.
PS. Do not believe the rubbish peddled by Farage and the leave.EU team about social media. E.g. a) They boasted publicly that they paid hundreds of thousands of pounds for over half a million Facebook ‘Likes’ without realising that b) Facebook’s algorithms no longer optimised news feeds for Likes (it is optimised for paid advertising). Leave.EU wasted hundreds of thousands just as many big companies spent millions building armies of Likes that were rendered largely irrelevant by Facebook’s algorithmic changes. This is just one of their blunders. Vote Leave put our money into targeted paid adverts, not buying Likes to spin stories to gullible hacks, MPs, and donors. Media organisations should have someone on the political staff who is a specialist in data or have a route to talk to their organisation’s own data science teams to help spot snake oil merchants.
PPS. If you are young, smart, and interested in politics, think very hard before studying politics / ‘political science’ / PPE at university. You will be far better off if you study maths or physics. It will be easy to move into politics later if you want to and you will have more general skills with much wider application and greater market value. PPE does not give such useful skills – indeed, it actually causes huge problems as it encourages people like Cameron and Ed Balls to ‘fool themselves’ and spread bad ideas with lots of confidence and bluffing. You can always read history books later but you won’t always be able to learn maths. If you have these general skills, then you will be much more effective than the PPE-ers you will compete against. In a few years, this will be more obvious as data science will be much more visible. A new interdisciplinary degree is urgently needed to replace PPE for those who want to go into politics. It should include the basics of modelling and involve practical exposure to people who are brilliant at managing large complex organisations.
PPPS. One of the projects that the Gove team did in the DfE was funding the development of a ‘Maths for Presidents’ course, in the same spirit as the great Berkeley course ‘Physics for Presidents’, based on ideas of Fields Medallist Tim Gowers. The statistics of polling would be a good subject for this course. This course could have a big cultural effect over 20 years if it is supported wisely.
Hi Dom. This is a worthy way of dealing with this development software but isn’t it slightly premature to give away our trade secrets? I fear VoteLeave may need this tool again before too long.
Sent from my iPad
A fascinating article! My physics is weak but just about comprehended your argument . We are indebted to you and yours. (I did contribute!) I hope that you are involved with Theresa May, strictly from a what is best for the country point of view, as Brexit should be, all things being equal! (Which we know is not always the case!)
Incredible insight on the past – the Leave campaign, and the future – the power and significance of iteration in communications. Confirms my plea for the PR industry to preach ‘Measurement, Evaluation, and Iteration’.
I deliver a Campaign Analytics module at Cardiff University’s Masters degree in Global Communications where I teach part-time. Do you have any slides or further info so I can create a Powerpoint case study on this incredible campaign story.
It’s a shame that young, intelligent mathematically literate people would waste their skills supporting a campaign based on lies and driven by hatred. Anyone mathematically literate would see that leaving the EU was (and still is) a terrible idea – I can’t possibly imagine why they would have stooped so low as to work for your campaign. Perhaps Trump will get elected and we’ll have them to thank.
LikeLiked by 1 person
Are you only going to tease us rather than provide us with any actual details?
How did the targeted advertising through Facebook work? What gave you an edge on others using the same platform? Iterative testing and refinement of the ads being served isn’t that complicated. The problems with buying likes are already well known. At what point did advanced analytics and the big brains need to get involved?
Could you say more about the “vast improvements possible in Government that could save hundreds of billions and avoid many disasters”? You have hinted at these possibilities before but still are yet to give any actual examples. “Data Science” works well in certain areas where there is sufficient data to provide answers to narrow problems – for example, predicting insurance risks, website click-through rates or even the results of elections. I’m not sure there are many similar such opportunities in government. Predictive analytics won’t help with the organisational problems of a large IT project. Even when the problem is more obviously analytical, “Data science” is unlikely to assist. It won’t help assessing the economic value of infrastructure investment because model uncertainty dominates other factors. It won’t help assess the likely effectiveness of a change to classroom teaching methods because that question can only be answered reliably through well designed experimental trials.
I’m not sure how exactly studying mathematics or physics would change the personality of ambitious young would-be politicians. Success in politics requires you to appear confident – the confidence of Cameron and Balls wasn’t caused by studying PPE. It isn’t fair to repeatedly single this one subject out for criticism. It is at least a fairly technical degree because of the material from economics and philosophy, so there is a higher IQ barrier than other “soft” degrees. Famous PPE graduates normally find success in politics. In contrast to famous history graduates who are usually journalists/bullshitters.
Question 47 from the Gowers blog post asks “How should a government determine tax rates if it wants to maximize the amount of tax that it collects? What about if it has other objectives?” How do you think this question could be carefully answered without using some concepts from economics? There are also plenty of questions about probability/statistics in that post – you do realize that Economics students typically study both subjects?
It seems to me that you vastly exaggerate the value of studying technical subjects for people who then work in areas where the knowledge cannot be directly applied. It would be interesting to see some real examples of how the knowledge obtained through their degree would then have a useful impact in politics or managing organisations.
You also largely miss the point of Steve Hsu’s posts. Smart people can succeed in many different areas because they are smart. Not because they studied quantum physicists.
As a scientist, with little and less knowledge of formal economics, politics, philosophy or the law – though I like reading and am always interested to learn more – I thought I’d put some things to you about how a straight-up scientist would think of strategy applied outside of their field, such as political campaigning.
I fell across this blog post, I find it interesting – the actual leave/remain debate isn’t relevant here – we’re talking strategy only to maximise the number of votes to leave. Here’s now my mind would work – and you can disregard, disagree and think it stupid and irrelevant and nothing new.
There are concepts within science that can be slid sideways. You, no doubt, know of the Schrodinger’s Cat thing – the closed box, the cat either dead, alive or both, as you like. The opening to ascertain the status. We like systems and we like to know everything we can about them: limits, constraints, cause and effect. You need an observer if you want to know what’s up – and your observer is human. Consider humans outside of the impartial, unbiased, reliable, theoretical textbook. Observers are human and humans can lie. They see what they want to. Sometimes they cannot bear to look. You cannot control a regular human’s perception of a situation, but you can look to influence it beforehand. That’s where your strategic effort goes in.
Things look different depending on the perspective. I’ve read my Feynman too, and his insight into QED was based on looking to understand how an electron behaves in the field from the point of view of the electron. Looks obvious, but it isn’t: if it were then there were plenty of great minds knocking around at that time and none of them thought of it that way.
Shift it sideways. Your electron is the political strategist seeking a vote to leave – not the leave elected politicians, not UKIP, not the campaign body itself, not the media commentator – the field is everything relevant in the country, which changes over time. You are looking for the path that maximises the number of votes to leave and the amount of influence over the process of leaving, post-result – and that means you disregard an awful lot of stuff that looks like it should be important, but it isn’t.
Choices made by Cameron about the referendum conditions from his POV looked to be in his favour and detrimental to leave. They weren’t. The vague nature, the non-binding ‘advisory’ wording suited Leave far more – which is counter-intuitive. The court decision yesterday, I would predict – based on no real knowledge whatsoever save reading the piece above – was anticipated and is actually a better outcome for the political strategist than if it went the other way. Teresa May had to give it a go because the inverse ruling leaves her with more control over the parameters of the leaving process.
When Cameron went for Remain, Vote Leave strategists must have wanted to send him a cake. From that point onwards, the Remain campaign was hugely constrained.
You might think that what I’ve written is total bullshit. You might think it obvious. I don’t know, though I’d be interested in your comments. ‘Smart people succeed because they are smart’ – perhaps. However, I know many very intelligent people who do not give equal weighting to what they know and what they don’t. They are far too concerned with being ‘right’.
Apologies, but I find this super interesting and have an urge to get across a scientist’s perspective ( – admittedly my own, with all the inherent subjectivity). My motivation is that of an enthusiast who loves science and the value it has in how we think. Again, this is a direct response to the blog post: I have no political affiliation or background. I studied chemistry at Oxford in the early 90’s from a comprehensive, council house background, a family on benefits after my father was disabled in an industrial accident. If anyone understands what education is worth, it is me. I mention Oxford – not as a rank pulling thing or to make my opinion more important, that’s eminence-based argument and ridiculous – but I think it’s more transparent when considering my response.
‘Are you only going to tease us rather than provide us with any actual details?’ – I think you have them, abstract though they are.
‘It seems to me that you vastly exaggerate the value of studying technical subjects for people who then work in areas where the knowledge cannot be directly applied’ – totally disagree. And it’s not the kind of thing where you can have one, say, physicist, and they apply it and then tell you about it. It’s not the conclusion, it’s the getting there: the former is not adaptable, the latter is and it lives inside your head. There is no shortcut – nobody can learn this stuff for you – you have to discover it for yourself and you have to live it. The thought processes have to be your own.
The biggest variable in the referendum was time and how it was anticipated and used. It’s about pathways and they change over time. Usually side A says blah, side B says no, it’s another blah….and so on; it’s not about that and it’s a waste of time responding to it. You choose your information upfront and hold the line with minimal effort: whilst your opponents are scrabbling around with rhetoric and evidence, trying to persuade, you are spending that time communicating based on the assumption that the reader already agrees. Eminence-based, evidence-based, whatever: it does not matter. The internet is wonderful, it’s equality of information and it matters not where it’s from, who it’s from: day and night, all hyperlinks are grey. What matters is predisposing people to like it and prompting them to find it, as if for themselves. Debate on the internet is just throwing words and a waste of time: it’s not the path that optimises your success.
Maybe politics, economics thinks like this, I don’t know, but I’ve never read any article that uses this kind of language and perspective.
We are not dealing simple physical events – we are dealing with free will and the consequences of human choices.
In his blog Dominic described the factors that were considered most essential to swing voters – who, like rich people – have different preferences and values to ordinary voters.
In shaping the leave campaign to take cognisance of their interests, which were different to those you might imagine, LEAVE were able to refine their campaign IN REAL TIME according to the feedback they were receiving.
This is standard practice in US elections where advertising is seen as an essential feature of US pragmatism.
So why didn’t US analysts foresee Trump’s victory?
That’s a different question but the short answer is – they did.
P.S. To all those many people who worked to accomplish Brexit – please accept my sincere thanks. Your victory will prove a blessing for all mankind.
It would be life-changing if 100 members of parliament had studied science or maths at university. The policy errors that one witnesses are never-ending, largely because lawyers, accountants and economists are arrogant enough to think they understand science and even appear on the Today programme to pontificate about say, energy policy, whereas scientists like me never pretend that they are high-flying legal experts and appear on R4 to discuss a high-profile case at the Old Bailey.
I’m actually a remainer but I enjoy the blog because people from any side in a debate should think like that and try to seek out the best results for our country. That sometimes necessitates sidelining ‘arrogant fools’ (to be polite).
An education system that still forces children to go for ‘arts’ or ‘sciences’ at age 14-15, as I had to do 50 years ago, is one of many, many ways to ensure that the country will not be as well-run as it could be.
Pingback: A review of Tetlock’s ‘Superforecasting’ (2015) – Dominic Cummings's Blog
Pingback: How the right got its hands on all the best data - and paid Facebook to call the tunes
Pingback: Hubris->Nemesis->Pathos. Or the Incredible Story of Admitting I Got It Wrong and What is To Be Done | Simon Radford
Good work. You’re right – the PPE brigade are boneheads.
Pingback: On the referendum #21: Branching histories of the 2016 referendum and ‘the frogs before the storm’ – Dominic Cummings's Blog
…You will be far better off if you study maths or physics. It will be easy to move into politics later if you want to and you will have more general skills with much wider application and greater market value. …
Is engineering any good…?
Pingback: Facebook pimping for politicos despite fake news 'purge' - THE BAY AREA NEWS CENTER
Pingback: Facebook pimping for politicos despite fake news 'purge' - GIZMODOM
Pingback: Facebook pimping for politicos despite fake news 'purge' - THE CLEANTECH CRASH
Pingback: Facebook pimping for politicos despite fake news 'purge' - The Los Angeles Digital Times
Pingback: Facebook pimping for politicos despite fake news 'purge' - WASHINGTON DC NEWS
Pingback: Facebook pimping for politicos despite fake news 'purge' - SILICON VALLEY
Pingback: Facebook pimping for politicos despite fake news 'purge' - The San Francisco Bay Guard-Eon Newspaper
Pingback: Dominic Cummings: how the Brexit referendum was won | Coffee House
Pingback: Understanding Big Data, AI and computer science is the biggest long-term challenge for communication professionals | Change Opinion | Westbourne Communications | Westbourne Communications