March 4, 2019 by dominiccummings

The most secure bio-labs routinely make errors that could cause a global pandemic & are about to re-start experiments on pathogens engineered to make them mammalian-airborne-transmissible

‘Although the institutions of our culture are so amazingly good that they have been able to manage stability in the face of rapid change for hundreds of years, the knowledge of what it takes to keep civilization stable in the face of rapidly increasing knowledge is not very widespread. In fact, severe misconceptions about several aspects of it are common among political leaders, educated people, and society at large. We’re like people on a huge, well-designed submarine, which has all sorts of lifesaving devices built in, who don’t know they’re in a submarine. They think they’re in a motorboat, and they’re going to open all the hatches because they want to have a nicer view.’ David Deutsch, the physicist who extended Alan Turing’s 1936 paper on classical computation to quantum computation.

Experiments on viruses that could cause a global pandemic killing many millions were halted but were recently cleared to resume and will be conducted in these ‘top security’ labs.

The new Bulletin of Atomic Scientists carries research showing how the supposedly most secure bio-labs have serious security problems and clearly present an unacceptable risk of causing a disastrous pandemic:

‘Incidents causing potential exposures to pathogens occur frequently in the high security laboratories often known by their acronyms, BSL3 (Biosafety Level 3) and BSL4. Lab incidents that lead to undetected or unreported laboratory-acquired infections can lead to the release of a disease into the community outside the lab; lab workers with such infections will leave work carrying the pathogen with them. If the agent involved were a potential pandemic pathogen, such a community release could lead to a worldwide pandemic with many fatalities. Of greatest concern is a release of a lab-created, mammalian-airborne-transmissible, highly pathogenic avian influenza virus, such as the airborne-transmissible H5N1 viruses created in the laboratories of Ron Fouchier in the Netherlands and Yoshihiro Kawaoka In Madison Wisconsin.

‘Such releases are fairly likely over time, as there are at least 14 labs (mostly in Asia) now carrying out this research. Whatever release probability the world is gambling with, it is clearly far too high a risk to human lives. Mammal-transmissible bird flu research poses a real danger of a worldwide pandemic that could kill human beings on a vast scale.

‘Human error is the main cause of potential exposures of lab workers to pathogens. Statistical data from two sources show that human error was the cause of, according to my research, 67 percent and 79.3 percent of incidents leading to potential exposures in BSL3 labs…

‘A key observation is that human error in the lab is mostly independent of pathogen type and biosafety level. Analyzing the likelihood of release from laboratories researching less virulent or transmissible pathogens therefore can serve as a reasonable surrogate for how potential pandemic pathogens are handled. (We are forced to deal with surrogate data because, thank goodness, there are little data on the release of potentially pandemic agents.) Put another way, surrogate data allows us to determine with confidence the probability of release of a potentially pandemic pathogen into the community. In a 2015 publication, Fouchier describes the careful design of his BSL3+ laboratory in Rotterdam and its standard operating procedures, which he contends should increase biosafety and reduce human error. Most of Fouchier’s discussion, however, addresses mechanical systems in the laboratory.

‘But the high percentage of human error reported here calls into question claims that state-of-the-art design of BSL3, BSL3+ (augmented BSL3), and BSL4 labs will prevent the release of dangerous pathogens. How much lab-worker training might reduce human error and undetected or unreported laboratory acquired infections remains an open question. Given the many ways by which human error can occur, it is doubtful that Fouchier’s human-error-prevention measures can eliminate release of airborne-transmissible avian flu into the community through undetected or unreported lab infections…

‘In its 2016 study for the NIH, “Risk and Benefit Analysis of Gain of Function Research,” Gryphon Scientific looked to the transportation, chemical, and nuclear sectors to define types of human error and their probabilities. As Gryphon summarized in its findings, the three types of human error are skill-based (errors involving motor skills involving little thought), rule-based (errors in following instructions or set procedures accidentally or purposely), and knowledge-based (errors stemming from a lack of knowledge or a wrong judgment call based on lack of experience). Gryphon claimed that “no comprehensive Human Reliability Analysis (HRA) study has yet been completed for a biological laboratory… . This lack of data required finding suitable proxies for accidents in other fields.”

‘But mandatory incident reporting to FSAP and NIH actually does provide sufficient data to quantify human error in BSL3 biocontainment labs…

‘Among other things, the GAO report called attention to a well-publicized incident in which a Defense Department laboratory “inadvertently sent live Bacillus anthracis, the bacterium that causes anthrax, to almost 200 laboratories worldwide over the course of 12 years. The laboratory believed that the samples had been inactivated.” The report describes yet another well-publicized incident in China in which “two researchers conducting virus research were exposed to severe acute respiratory syndrome (SARS) coronavirus samples that were incompletely inactivated. The researchers subsequently transmitted SARS to others, leading to several infections and one death in 2004.”

‘The GAO identified three recent releases of Ebola and Marburg viruses from BSL4 to lower containment labs due to incomplete inactivation.

‘A fourth release in 2014 from the CDC labs occurred when “Scientists inadvertently switched samples designated for live Ebola virus studies with samples intended for studies with inactivated material. As a result, the samples with viable Ebola virus, instead of the samples with inactivated Ebola virus, were transferred out of a BSL-4 laboratory to a laboratory with a lower safety level for additional analysis. While no one contracted Ebola virus in this instance, the consequences could have been dire for the personnel involved as there are currently no approved treatments or vaccines for this virus.”…

‘ In an analysis circulated at the 2017 meeting for the Biological Weapons Convention, a conservative estimate shows that the probability is about 20 percent for a release of a mammalian-airborne-transmissible, highly pathogenic avian influenza virus into the community from at least one of 10 labs over a 10-year period of developing and researching this type of pathogen… Analysis of the FOIA NIH data gives a much higher release probability — that is, a factor five to 10 times higher…

‘The avian flu virus H5N1 kills 60 percent of people who become infected from direct contact with infected birds. The mammalian-airborne-transmissible, highly pathogenic avian influenza created in the Fouchier and Kawaoka labs should be able to infect humans through the air, and the viruses could be deadly.

‘A release into the community of such a pathogen could seed a pandemic with a probability of perhaps 15 percent. This estimate is from an average of two very different approaches…

‘Combining release probability with the not insignificant probability that an airborne-transmissible influenza virus could seed a pandemic, we have an alarming situation…

‘Those who support mammalian-airborne-transmissible, highly pathogenic avian influenza experiments either believe the probability of community release is infinitesimal or the benefits in preventing a pandemic are great enough to justify the risk. For this research, it would take extraordinary benefits and significant risk reduction via extraordinary biosafety measures to correct such a massive overbalance of highly uncertain benefits to too-likely risks.

‘Whatever probability number we are gambling with, it is clearly far too high a risk to human lives. There are experimental approaches that do not involve live mammalian-airborne-transmissible, highly pathogenic avian influenza which identify mutations involved in mammalian airborne transmission. These “safer experimental approaches are both more scientifically informative and more straightforward to translate into improved public health…” Asian bird flu virus research to develop live strains transmissible via aerosols among mammals (and perhaps some other potentially pandemic disease research as well), should for the present be restricted to special BSL4 laboratoriesor augmented BSL3 facilities where lab workers are not allowed to leave the facility until it is certain that they have not become infected.’

This connects to my blog last week on nuclear/AGI safety and how to turn government institutions responsible for decisions about billions of lives and trillions of dollars from hopeless to high performance. Science is ‘a blind search algorithm’. New institutions are needed that incentivise hard thinking about avoiding disasters…

As the piece above stresses and lessons from nuclear safety also show, getting the physical security right is only one hard problem. Most security failings happen because of human actions that are not envisaged when designing systems. This is why Red Teams are so vital but they cannot solve the problem of broken political institutions. Remember: Red Teams told the federal government all about the failures of airline security at the airports used by the 9/11 attackers before 9/11. Those who wrote the reports were DEMOTED and the Red Team was CLOSED: those with power did not want to hear.

The problems considered above are ‘accidents’ — what if these systems were subject to serious penetration testing by the likes of Chris Vickery? (Also consider that there is a large network of Soviet scientists that participated in the covert Soviet bio-weapons program that the West was almost completely ignorant about until post-1991. Many of these people have scattered to places unknown with who knows what.)

Pop Quiz…

A. How many MPs understand security protocols in UK facilities rated ‘most secure’?

B. Does the minister responsible? Have they ever had a meeting with experts about this? Is the responsible minister even aware of this very recent research above? Are they aware that these experiments are about to restart? When was the last time a very high level Red Team test of supposedly ‘top secret/secure’ UK facilities was conducted using teams with expertise in breaking into secure facilities by any means necessary, legal or illegal (i.e a genuine ‘free play’ exercise, not a scripted game where the Red Team is prohibited from being too ‘extreme’)? Has this happened at all in the last 10 years? How bad were the results? Were any ministers told? Have any asked? Does any minister even know who is responsible for such things? Are officials of the calibre of those who routinely preside over procurement disasters in charge (back in SW1) of the technical people working on such issues (after all, some play senior roles in Brexit negotiations)?

C. How much coverage of the above finding has appeared in newspapers like the FT?

My answers would be: A. ~0. B. Near total general failure. C. ~0.

A hypothesis that should be tested: With a) <£1million to play with, b) the ability to recruit a team from among special forces/intel services/specialist criminals/whoever, and c) no rules (so for example they could deploy honey traps on the head of security), a Red Team would break into the most secure UK bio-research facilities and acquire material that could be released publicly in order to cause deaths on the scale of millions. A serious test will also reveal that there is no serious attempt to incentivise the stars of Whitehall to work on such important issues or involve extremely able people from outside Whitehall.

As I wrote last week, it was clear years ago that a smart teen could take out any world leaders using a drone in Downing Street — they can’t even install decent CCTV and audio — but we should be much more worried about bio-facilities.

February 21, 2019 by dominiccummings

On the referendum #30: Genetics, genomics, predictions & ‘the Gretzky game’ — a chance for Britain to help the world

Britain could contribute huge value to the world by leveraging existing assets, including scientific talent and how the NHS is structured, to push the frontiers of a rapidly evolving scientific field — genomic prediction — that is revolutionising healthcare in ways that give Britain some natural advantages over Europe and America. We should plan for free universal ‘SNP’ genetic sequencing as part of a shift to genuinely preventive medicine — a shift that will lessen suffering, save money, help British advanced technology companies in genomics and data science/AI, make Britain more attractive for scientists and global investment, and extend human knowledge in a crucial field to the benefit of the whole world.

‘SNP’ sequencing means, crudely, looking at the million or so most informative markers or genetic variants without sequencing every base pair in the genome. SNP sequencing costs ~$50 per person (less at scale), whole genome sequencing costs ~$1,000 per person (less at scale). The former captures most of the predictive power now possible at 1/20th of the cost of the latter.

Background: what seemed ‘sci fi’ ~2010-13 is now reality

In my 2013 essay on education and politics, I summarised the view of expert scientists on genetics (HERE between pages 49-51, 72-74, 194-203). Although this was only a small part of the essay most of the media coverage focused on this, particularly controversies about IQ.

Regardless of political affiliation most of the policy/media world, as a subset of ‘the educated classes’ in general, tended to hold a broadly ‘blank slate’ view of the world mostly uninformed by decades of scientific progress. Technical terms like ‘heritability’, which refers to the variance in populations, caused a lot of confusion.

When my essay hit the media, fortunately for me the world’s leading expert, Robert Plomin, told hacks that I had summarised the state of the science accurately. (I never tried to ‘give my views on the science’ as I don’t have ‘views’ — all people like me can try to do with science is summarise the state of knowledge in good faith.) Quite a lot of hacks then spent some time talking to Plomin and some even wrote about how they came to realise that their assumptions about the science had been wrong (e.g Gaby Hinsliff).

Many findings are counterintuitive to say the least. Almost everybody naturally thinks that ‘the shared environment’ in the form of parental influence ‘obviously’ has a big impact on things like cognitive development. The science says this intuition is false. The shared environment is much less important than we assume and has very little measurable effect on cognitive development: e.g an adopted child who does an IQ test in middle age will show on average almost no correlation with the parents who brought them up (genes become more influential as you age). People in the political world assumed a story of causation in which, crudely, wealthy people buy better education and this translates into better exam and IQ scores. The science says this story is false. Environmental effects on things like cognitive ability and education achievement are almost all from what is known as the ‘non-shared environment’ which has proved very hard to pin down (environmental effects that differ for children, like random exposure to chemicals in utero). Further, ‘The case for substantial genetic influence on g [g = general intelligence ≈ IQ] is stronger than for any other human characteristic’ (Plomin) and g/IQ has far more predictive power for future education than class does. All this has been known for years, sometimes decades, by expert scientists but is so contrary to what well-educated people want to believe that it was hardly known at all in ‘educated’ circles that make and report on policy.

Another big problem is that widespread ignorance about genetics extends to social scientists/economists, who are much more influential in politics/government than physical scientists. A useful heuristic is to throw ~100% of what you read from social scientists about ‘social mobility’ in the bin. Report after report repeats the same clichés, repeats factual errors about genetics, and is turned into talking points for MPs as justification for pet projects. ‘Kids who can read well come from homes with lots of books so let’s give families with kids struggling to read more books’ is the sort of argument you read in such reports without any mention of the truth: children and parents share genes that make them good at and enjoy reading, so causation is operating completely differently to the assumptions. It is hard to overstate the extent of this problem. (There are things we can do about ‘social mobility’, my point is Insider debate is awful.)

A related issue is that really understanding the science requires serious understanding of statistics and, now, AI/machine learning (ML). Many social scientists do not have this training. This problem will get worse as data science/AI invades the field.

A good example is ‘early years’ and James Heckman. The political world is obsessed with ‘early years’ such as Sure Start (UK) and Head Start (US). Politicians latch onto any ‘studies’ that seem to justify it and few have any idea about the shocking state of the studies usually quoted to justify spending decisions. Heckman has published many papers on early years and they are understandably widely quoted by politicians and the media. Heckman is a ‘Nobel Prize’ winner in economics. One of the world’s leading applied mathematicians, Professor Andrew Gelman, has explained how Heckman has repeatedly made statistical errors in his papers but does not correct them: cf. How does a Nobel-prize-winning economist become a victim of bog-standard selection bias? This really shows the scale of the problem: if a Nobel-winning economist makes ‘bog standard’ statistical errors that confuse him about studies on pre-school, what chance do the rest of us in the political/media world have?

Consider further that genomics now sometimes applies very advanced mathematical ideas such as ‘compressed sensing’. Inevitably few social scientists can judge such papers but they are overwhelmingly responsible for interpreting such things for ministers and senior officials. This is compounded by the dominance of social scientists in Whitehall units responsible for data and evidence. Many of these units are unable to provide proper scientific advice to ministers (I have had personal experience of this in the Department for Education). Two excellent articles by Duncan Watts recently explained fundamental problems with social science and what could be done (e.g a much greater focus on successful prediction) but as far as I can tell they have had no impact on economists and sociologists who do not want to face their lack of credibility and whose incentives in many ways push them towards continued failure (Nature paper HERE, Science paper HERE — NB. the Department for Education did not even subscribe to the world’s leading science journals until I insisted in 2011).

1) The problem that the evidence for early years is not what ministers and officials think it is is not a reason to stop funding but I won’t go into this now. 2) This problem is incontrovertible evidence, I think, of the value of an alpha data science unit in Downing Street, able to plug into the best researchers around the world, and ensure that policy decisions are taken on the basis of rational thinking and good science or, just as important, everybody is aware that they have to make decisions in the absence of this. This unit would pay for itself in weeks by identifying flawed reasoning and stopping bad projects, gimmicks etc. Of course, this idea has no chance with those now at the top of Government and the Cabinet Office would crush such a unit as it would threaten the traditional hierarchy. One of the arguments I made in my essay was that we should try to discover useful and reliable benchmarks for what children of different abilities are really capable of learning and build on things like the landmark Study of Mathematically Precocious Youth. This obvious idea is anathema to the education policy world where there is almost no interest in things like SMPY and almost everybody supports the terrible idea that ‘all children must do the same exams’ (guaranteeing misery for some and boredom/time wasting for others). NB. Most rigorous large-scale educational RCTs are uninformative. Education research, like psychology, produces a lot of what Feynman called ‘cargo cult science’.

Since 2013, genomics has moved fast and understanding in the UK media has changed probably faster in five years than over the previous 35 years. As with the complexities of Brexit, journalists have caught up with reality much better than MPs. It’s still true that almost everything written by MPs about ‘social mobility’ is junk but you could see from the reviews of Plomin’s recent book, Blueprint, that many journalists have a much better sense of the science than they did in 2013. Rare good news, though much more progress is needed…

What’s happening now?

In 2013 it was already the case that the numbers on heritability derived from twin and adoption studies were being confirmed by direct inspection of DNA — therefore many of the arguments about twin/adoption studies were redundant — but this fact was hardly known.

I pointed out that the field would change fast. Both Plomin and another expert, Steve Hsu, made many predictions around 2010-13 some of which I referred to in my 2013 essay. Hsu is a physics professor who is also one of the world’s leading researchers on genomics.

Hsu predicted that very large samples of DNA would allow scientists over the next few years to start identifying the actual genes responsible for complex traits, such as diseases and intelligence, and make meaningful predictions about the fate of individuals. Hsu gave estimates of the sample sizes that would be needed. His 2011 talk contains some of these predictions and also provides a physicist’s explanation of ‘what is IQ measuring’. As he said at Google in 2011, the technology is ‘right on the cusp of being able to answer fundamental questions’ and ‘if in ten years we all meet again in this room there’s a very good chance that some of the key questions we’ll know the answers to’. His 2014 paper explains the science in detail. If you spend a little time looking at this, you will know more than 99% of high status economists gabbling on TV about ‘social mobility’ saying things like ‘doing well on IQ tests just proves you can do IQ tests’.

In 2013, the world of Westminster thought this all sounded like science fiction and many MP said I sounded like ‘a mad scientist’. Hsu’s predictions have come true and just five years later this is no longer ‘science fiction’. (Also NB. Hsu’s blog was one of the very few places where you would have seen discussion of CDOs and the 2008 financial crash long BEFORE it happened. I have followed his blog since ~2004 and this from 2005, two years before the crash started, was the first time I read about things like ‘synthetic CDOs’: ‘we have yet another ill-understood casino running, with trillions of dollars in play’. The quant-physics network had much better insight into the dynamics behind the 2008 Crash than high status mainstream economists like Larry Summers responsible for regulation.)

His group and others have applied machine learning to very large genetic samples and built predictors of complex traits. Complex traits like general intelligence and most diseases are ‘polygenic’ — they depend on many genes each of which contributes a little (unlike diseases caused by a single gene).

‘There are now ~20 disease conditions for which we can identify, e.g, the top 1% outliers with 5-10x normal risk for the disease. The papers reporting these results have almost all appeared within the last year or so.’

For example, the height predictor ‘captures nearly all of the predicted SNP heritability for this trait — actual heights of most individuals in validation tests are within a few cm of predicted heights.’ Height is similar to IQ — polygenic and similar heritability estimates.

These predictors have been validated with out-of-sample tests. They will get better and better as more and more data is gathered about more and more traits.

This enables us to take DNA from unborn embryos, do SNP genetic sequencing costing ~$50, and make useful predictions about the odds of the embryo being an outlier for diseases like atrial fibrillation, diabetes, breast cancer, or prostate cancer. NB. It is important that we do not need to sequence the whole genome to do this (see below). We will also be able to make predictions about outliers in cognitive abilities (the high and low ends). (My impression is that predicting Alzheimers is still hampered by a lack of data but this will improve as the data improves.)

There are many big implications. This will obviously revolutionise IVF. ~1 million IVF embryos per year are screened worldwide using less sophisticated tests. Instead of picking embryos at random, parents will start avoiding outliers for disease risks and cognitive problems. Rich people will fly to jurisdictions offering the best services.

Forensics is being revolutionised. First, DNA samples can be used to give useful physical descriptions of suspects because you can identify ethnic group, height, hair colour etc. Second, ‘cold cases’ are now routinely being solved because if a DNA sample exists, then the police can search for cousins of the perpetrator from public DNA databases, then use the cousins to identify suspects. Every month or so now in America a cold case murder is solved and many serial killers are being found using this approach — just this morning I saw what looks to be another example just announced, a murder of an 11 year-old in 1973. (Some companies are resisting this development but they will, I am very confident, be smashed in court and have their reputations trashed unless they change policy fast. The public will have no sympathy for those who stand in the way.)

Hsu recently attended a conference in the UK where he presented some of these ideas to UK policy makers. He wrote this blog about the great advantages the NHS has in developing this science.

‘The UK could become the world leader in genomic research by combining population-level genotyping with NHS health records… The US private health insurance system produces the wrong incentives for this kind of innovation: payers are reluctant to fund prevention or early treatment because it is unclear who will capture the ROI [return on investment]… The NHS has the right incentives, the necessary scale, and access to a deep pool of scientific talent. The UK can lead the world into a new era of precision genomic medicine.

‘NHS has already announced an out-of-pocket genotyping service which allows individuals to pay for their own genotyping and to contribute their health + DNA data to scientific research. In recent years NHS has built an impressive infrastructure for whole genome sequencing (cost ~$1k per individual) that is used to treat cancer and diagnose rare genetic diseases. The NHS subsidiary Genomics England recently announced they had reached the milestone of 100k whole genomes…

‘At the meeting, I emphasized the following:

1. NHS should offer both inexpensive (~$50) genotyping (sufficient for risk prediction of common diseases) along with the more expensive $1k whole genome sequencing. This will alleviate some of the negative reaction concerning a “two-tier” NHS, as many more people can afford the former.

2. An in-depth analysis of cost-benefit for population wide inexpensive genotyping would likely show a large net cost savings: the risk predictors are good enough already to guide early interventions that save lives and money. Recognition of this net benefit would allow NHS to replace the $50 out-of-pocket cost with free standard of care.’ (Emphasis added)

NB. In terms of the short-term practicalities it is important that whole genome sequencing costs ~$1,000 (and falling) but is not necessary: a version 1/20th of the cost, looking just at the most informative genetic variants, captures most of the predictive benefits. Some have incentives to distort this, such as companies like Illumina trying to sell expensive machines for whole genome sequencing, which can distort policy — let’s hope officials are watching carefully. These costs will, obviously, keep falling.

This connects to an interesting question… Why was the likely trend in genomics clear ~2010 to Plomin, Hsu and others but invisible to most? Obviously this involves lots of elements of expertise and feel for the field but also they identified FAVOURABLE EXPONENTIALS. Here is the fall in the cost of sequencing a genome compared to Moore’s Law, another famous exponential. The drop over ~18 years has been a factor of ~100,000. Hsu and Plomin could extrapolate that over a decade and figure out what would be possible when combined with other trends they could see. Researchers are already exploring what will be possible as this trend continues.

Identifying favourable exponentials is extremely powerful. Back in the early 1970s, the greatest team of computer science researchers ever assembled (PARC) looked out into the future and tried to imagine what could be possible if they brought that future back to the present and built it. They were trying to ‘compute in the future’. They created personal computing. (Chart by Alan Kay, one of the key researchers — he called it ‘the Gretzky game’ because of Gretzky’s famous line ‘I skate to where the puck is going to be, not where it has been.’ The computer is the Alto, the first personal computer that stunned Steve Jobs when he saw a demo. The sketch on the right is of children using a tablet device that Kay drew decades before the iPad was launched.)

Hopefully the NHS and Department for Health will play ‘the Gretzky game’, take expert advice from the likes of Plomin and Hsu and take this opportunity to make the UK a world leader in one of the most important frontiers in science.

We can imagine everybody in the UK being given valuable information about their health for free, truly preventive medicine where we target resources at those most at risk, and early (even in utero) identification of risks.
This would help bootstrap British science into a stronger position with greater resources to study things like CRISPR and the next phase of this revolution — editing genes to fix problems, where clinical trials are already showing success.
It would also give a boost to British AI/data science companies — the laws, rules on data etc should be carefully shaped to ensure that British companies (not Silicon Valley or China) capture most of the financial value (though everybody will gain from the basic science).
These gains would have positive feedback effects on each other, just as investment in basic AI/ML research will have positive feedback effects in many industries.
I have argued many times for the creation of a civilian UK ‘ARPA’ — a centre for high-risk-high-payoff research that has been consistently blocked in Whitehall (see HERE for an account of how ARPA-PARC created the internet and personal computing). This fits naturally with Britain seeking to lead in genomics/AI. Thinking about this is part of a desperately needed overall investigation into the productivity of the British economy and the ecosystem of universities, basic science, venture capital, startups, regulation (data, intellectual property etc) and so on.

There will also be many controversies and problems. The ability to edit genomes — and even edit the germline with ‘gene drives’ so all descendants have the same copy of the gene — is a Promethean power implying extreme responsibilities. On a mundane level, embracing new technology is clearly hard for the NHS with its data infrastructure. Almost everyone I speak to using the NHS has had similar problems that I have had — nightmares with GPs, hospitals, consultants et al being able to share data and records, things going missing, etc. The NHS will be crippled if it can’t fix this, but this is another reason to integrate data science as a core ‘utility’ for the NHS.

On a political note…

Few scientists and even fewer in the tech world are aware of the EU’s legal framework for regulating technology and the implications of the recent Charter of Fundamental Rights (the EU’s Charter, NOT the ECHR) which gives the Commission/ECJ the power to regulate any advanced technology, accelerate the EU’s irrelevance, and incentivise investors to invest outside the EU. In many areas, the EU regulates to help the worst sort of giant corporate looters defending their position against entrepreneurs. Post-Brexit Britain will be outside this jurisdiction and able to make faster and better decisions about regulating technology like genomics, AI and robotics. Prediction: just as Insiders now talk of how we ‘dodged a bullet’ in staying out of the euro, within ~10 years Insiders will talk about being outside the Charter/ECJ and the EU’s regulation of data/AI in similar terms (assuming Brexit happens and UK politicians even try to do something other than copy the EU’s rules).

China is pushing very hard on genomics/AI and regards such fields as crucial strategic ground for its struggle for supremacy with America. America has political and regulatory barriers holding it back on genomics that are much weaker here. Britain cannot stop the development of such science. Britain can choose to be a backwater, to ignore such things and listen to MPs telling fairy stories while the Chinese plough ahead, or it can try to lead. But there is no hiding from the truth and ‘for progress there is no cure’ (von Neumann). We will never be the most important manufacturing nation again but we could lead in crucial sub-fields of advanced technology. As ARPA-PARC showed, tiny investments can create entire new industries and trillions of dollars of value.

Sadly most politicians of Left and Right have little interest in science funding with tremendous implications for future growth, or the broader question of productivity and the ecosystem of science, entrepreneurs, universities, funding, regulation etc, and we desperately need institutions that incentivise politicians and senior officials to ‘play the Gretzky game’. The next few months will be dominated by Brexit and, hopefully, the replacement of the May/Hammond government. Those thinking about the post-May landscape and trying to figure out how to navigate in uncharted and turbulent waters should focus on one of the great lessons of politics that is weirdly hard for many MPs to internalise: the public rewards sustained focus on their priorities!

One of the lessons of the 2016 referendum (that many Conservative MPs remain desperate not to face) is the political significance of the NHS. The concept described above is one of those concepts in politics that maximises positive futures for the force that adopts it because it draws on multiple sources of strength. It combines, inter alia, all the political benefits of focus on the NHS, helping domestic technology companies, incentivising global investment, doing something that shows the world that Britain is (contra the May/Hammond outlook) open to science and high skilled immigrants, it is based on intrinsic advantages that Europe and America will find hard to overcome over a decade, it supplies (NB. MPs/spads) a never-ending string of heart-wrenching good news stories, and, very rarely in SW1, those pushing it would be seen as leading something of global importance. It will, therefore, obviously be rejected by a section of Conservative MPs who much prefer to live in a parallel world, who hate anything to do with science and who are ignorant about how new industries and wealth are really created. But for anybody trying to orient themselves to reality, connect themselves to sources of power, and thinking ‘how on earth could we clamber out of this horror show’, it is an obvious home run…

NB. It ought to go without saying that turning this idea into a political/government success requires focus on A) the NHS, health, science, NOT getting sidetracked into B) arguments about things like IQ and social mobility. Over time, the educated classes will continue to be dragged to more realistic views on (B) but this will be a complex process entangled with many hysterical episodes. (A) requires ruthless focus…

Please leave comments, fix errors below. I have not shown this blog in draft to Plomin or Hsu who obviously are not responsible for my errors.

Further reading

Plomin’s excellent new book, Blueprint. I would encourage journalists who want to understand this subject to speak to Plomin who works in London and is able to explain complex technical subjects to very confused arts graduates like me.

On the genetic architecture of intelligence and other quantitative traits, Hsu 2014.

Cf. this thread by researcher Paul Pharaoh on breast cancer.

Hsu blogs on genomics.

Some recent developments with AI/ML, links to papers.

On how ARPA-PARC created the modern computer industry and lessons for high-risk-high-payoff science research.

My 2013 essay.

January 13, 2017 by dominiccummings

Unrecognised simplicities of effective action #1: expertise and a quadrillion dollar business

‘The combination of physics and politics could render the surface of the earth uninhabitable.’ John von Neumann.

Introduction

This series of blogs considers:

the difference between fields with genuine expertise, such as fighting and physics, and fields dominated by bogus expertise, such as politics and economic forecasting;
the big big problem we face – the world is ‘undersized and underorganised’ because of a collision between four forces: 1) our technological civilisation is inherently fragile and vulnerable to shocks, 2) the knowledge it generates is inherently dangerous, 3) our evolved instincts predispose us to aggression and misunderstanding, and 4) there is a profound mismatch between the scale and speed of destruction our knowledge can cause and the quality of individual and institutional decision-making in ‘mission critical’ institutions – our institutions are similar to those that failed so spectacularly in summer 1914 yet they face crises moving at least ~10³ times faster and involving ~10⁶ times more destructive power able to kill ~10¹⁰ people;
what classic texts and case studies suggest about the unrecognised simplicities of effective action to improve the selection, education, training, and management of vital decision-makers to improve dramatically, reliably, and quantifiably the quality of individual and institutional decisions (particularly 1) the ability to make accurate predictions and b) the quality of feedback);
how we can change incentives to aim a much bigger fraction of the most able people at the most important problems;
what tools and technologies can help decision-makers cope with complexity.

[I’ve tweaked a couple of things in response to this blog by physicist Steve Hsu.]

Summary of the big big problem

The investor Peter Thiel (founder of PayPal and Palantir, early investor in Facebook) asks people in job interviews: what billion (10⁹) dollar business is nobody building? The most successful investor in world history, Warren Buffett, illustrated what a quadrillion (10¹⁵) dollar business might look like in his 50th anniversary letter to Berkshire Hathaway investors.

‘There is, however, one clear, present and enduring danger to Berkshire against which Charlie and I are powerless. That threat to Berkshire is also the major threat our citizenry faces: a “successful” … cyber, biological, nuclear or chemical attack on the United States… The probability of such mass destruction in any given year is likely very small… Nevertheless, what’s a small probability in a short period approaches certainty in the longer run. (If there is only one chance in thirty of an event occurring in a given year, the likelihood of it occurring at least once in a century is 96.6%.) The added bad news is that there will forever be people and organizations and perhaps even nations that would like to inflict maximum damage on our country. Their means of doing so have increased exponentially during my lifetime. “Innovation” has its dark side.

‘There is no way for American corporations or their investors to shed this risk. If an event occurs in the U.S. that leads to mass devastation, the value of all equity investments will almost certainly be decimated.

‘No one knows what “the day after” will look like. I think, however, that Einstein’s 1949 appraisal remains apt: “I know not with what weapons World War III will be fought, but World War IV will be fought with sticks and stones.”’

Politics is profoundly nonlinear. (I have written a series of blogs about complexity and prediction HERE which are useful background for those interested.) Changing the course of European history via the referendum only involved about 10 crucial people controlling ~£10⁷ while its effects over ten years could be on the scale of ~10⁸ – 10⁹people and ~£10¹²: like many episodes in history the resources put into it are extremely nonlinear in relation to the potential branching histories it creates. Errors dealing with Germany in 1914 and 1939 were costly on the scale of ~100,000,000 (10⁸) lives. If we carry on with normal human history – that is, international relations defined as out-groups competing violently – and combine this with modern technology then it is extremely likely that we will have a disaster on the scale of billions (10⁹) or even all humans (~10¹⁰). The ultimate disaster would kill about 100 times more people than our failure with Germany. Our destructive power is already much more than 100 times greater than it was then: nuclear weapons increased destructiveness by roughly a factor of a million.

Even if we dodge this particular bullet there are many others lurking. New genetic engineering techniques such as CRISPR allow radical possibilities for re-engineering organisms including humans in ways thought of as science fiction only a decade ago. We will soon be able to remake human nature itself. CRISPR-enabled ‘gene drives’ enable us to make changes to the germ-line of organisms permanent such that changes spread through the entire wild population, including making species extinct on demand. Unlike nuclear weapons such technologies are not complex, expensive, and able to be kept secret for a long time. The world’s leading experts predict that people will be making them cheaply at home soon – perhaps they already are. These developments have been driven by exponential progress much faster than Moore’s Law reducing the cost of DNA sequencing per genome from ~$10⁸ to ~$10³in roughly 15 years.

screenshot-2017-01-16-12-24-13

It is already practically possible to deploy a cheap, autonomous, and anonymous drone with facial-recognition software and a one gram shaped-charge to identify a relevant face and blow it up. Military logic is driving autonomy. For example, 1) the explosion in the volume of drone surveillance video (from 71 hours in 2004 to 300,000 hours in 2011 to millions of hours now) requires automated analysis, and 2) jamming and spoofing of drones strongly incentivise a push for autonomy. It is unlikely that promises to ‘keep humans in the loop’ will be kept. It is likely that state and non-state actors will deploy low-cost drone swarms using machine learning to automate the ‘find-fix-finish’ cycle now controlled by humans. (See HERE for a video just released for one such program and imagine the capability when they carry their own communication and logistics network with them.)

In the medium-term, many billions are being spent on finding the secrets of general intelligence. We know this secret is encoded somewhere in the roughly 125 million ‘bits’ of information that is the rough difference between the genome that produces the human brain and the genome that produces the chimp brain. This search space is remarkably small – the equivalent of just 25 million English words or 30 copies of the King James Bible. There is no fundamental barrier to decoding this information and it is possible that the ultimate secret could be described relatively simply (cf. this great essay by physicist Michael Nielsen). One of the world’s leading experts has told me they think a large proportion of this problem could be solved in about a decade with a few tens of billions and something like an Apollo programme level of determination.

Not only is our destructive and disruptive power still getting bigger quickly – it is also getting cheaper and faster every year. The change in speed adds another dimension to the problem. In the period between the Archduke’s murder and the outbreak of World War I a month later it is striking how general failures of individuals and institutions were compounded by the way in which events moved much faster than the ‘mission critical’ institutions could cope with such that soon everyone was behind the pace, telegrams were read in the wrong order and so on. The crisis leading to World War I was about 30 days from the assassination to the start of general war – about 700 hours. The timescale for deciding what to do between receiving a warning of nuclear missile launch and deciding to launch yourself is less than half an hour and the President’s decision time is less than this, maybe just minutes. This is a speedup factor of at least 10³.

Economic crises already occur far faster than human brains can cope with. The financial system has made a transition from people shouting at each other to a a system dominated by high frequency ‘algorithmic trading’ (HFT), i.e. machine intelligence applied to robot trading with vast volumes traded on a global spatial scale and a microsecond (10^-6) temporal scale far beyond the monitoring, understanding, or control of regulators and politicians. There is even competition for computer trading bases in specific locations based on calculations of Special Relativity as the speed of light becomes a factor in minimising trade delays (cf. Relativistic statistical arbitrage, Wissner-Gross). ‘The Flash Crash’ of 9 May 2010 saw the Dow lose hundreds of points in minutes. Mini ‘flash crashes’ now blow up and die out faster than humans can notice. Given our institutions cannot cope with economic decisions made at ‘human speed’, a fortiori they cannot cope with decisions made at ‘robot speed’. There is scope for worse disasters than 2008 which would further damage the moral credibility of decentralised markets and provide huge chances for extremist political entrepreneurs to exploit. (* See endnote.)

What about the individuals and institutions that are supposed to cope with all this?

Our brains have not evolved much in thousands of years and are subject to all sorts of constraints including evolved heuristics that lead to misunderstanding, delusion, and violence particularly under pressure. There is a terrible mismatch between the sort of people that routinely dominate mission critical political institutions and the sort of people we need: high-ish IQ (we need more people >145 (+3SD) while almost everybody important is between 115-130 (+1 or 2SD)), a robust toolkit for not fooling yourself including quantitative problem-solving (almost totally absent at the apex of relevant institutions), determination, management skills, relevant experience, and ethics. While our ancestor chiefs at least had some intuitive feel for important variables like agriculture and cavalry our contemporary chiefs (and those in the media responsible for scrutiny of decisions) generally do not understand their equivalents, and are often less experienced in managing complex organisations than their predecessors.

The national institutions we have to deal with such crises are pretty similar to those that failed so spectacularly in summer 1914 yet they face crises moving at least ~10³ times faster and involving ~10⁶ times more destructive power able to kill ~10¹⁰ people. The international institutions developed post-1945 (UN, EU etc) contribute little to solving the biggest problems and in many ways make them worse. These institutions fail constantly and do not – cannot – learn much.

If we keep having crises like we have experienced over the past century then this combination of problems pushes the probability of catastrophe towards ‘overwhelmingly likely’.

What Is To be Done? There’s plenty of room at the top

‘In a knowledge-rich world, progress does not lie in the direction of reading information faster, writing it faster, and storing more of it. Progress lies in the direction of extracting and exploiting the patterns of the world… And that progress will depend on … our ability to devise better and more powerful thinking programs for man and machine.’ Herbert Simon, Designing Organizations for an Information-rich World, 1969.

‘Fascinating that the same problems recur time after time, in almost every program, and that the management of the program, whether it happened to be government or industry, continues to avoid reality.’ George Mueller, pioneer of ‘systems engineering’ and ‘systems management’ and the man most responsible for the success of the 1969 moon landing.

Somehow the world has to make a series of extremely traumatic and dangerous transitions over the next 20 years. The main transition needed is:

Embed reliably the unrecognised simplicities of high performance teams (HPTs), including personnel selection and training, in ‘mission critical’ institutions while simultaneously developing a focused project that radically improves the prospects for international cooperation and new forms of political organisation beyond competing nation states.

Big progress on this problem would automatically and for free bring big progress on other big problems. It could improve (even save) billions of lives and save a quadrillion dollars (~$10¹⁵). If we avoid disasters then the error-correcting institutions of markets and science will, patchily, spread peace, prosperity, and learning. We will make big improvements with public services and other aspects of ‘normal’ government. We will have a healthier political culture in which representative institutions, markets serving the public (not looters), and international cooperation are stronger.

Can a big jump in performance – ‘better and more powerful thinking programs for man and machine’ – somehow be systematised?

Feynman once gave a talk titled ‘There’s plenty of room at the bottom’ about the huge performance improvements possible if we could learn to do engineering at the atomic scale – what is now called nanotechnology. There is also ‘plenty of room at the top’ of political structures for huge improvements in performance. As I explained recently, the victory of the Leave campaign owed more to the fundamental dysfunction of the British Establishment than it did to any brilliance from Vote Leave. Despite having the support of practically every force with power and money in the world (including the main broadcasters) and controlling the timing and legal regulation of the referendum, they blew it. This was good if you support Leave but just how easily the whole system could be taken down should be frightening for everybody .

Creating high performance teams is obviously hard but in what ways is it really hard? It is not hard in the same sense that some things are hard like discovering profound new mathematical knowledge. HPTs do not require profound new knowledge. We have been able to read the basic lessons in classics for over two thousand years. We can see relevant examples all around us of individuals and teams showing huge gains in effectiveness.

The real obstacle is not financial. The financial resources needed are remarkably low and the return on small investments could be incalculably vast. We could significantly improve the decisions of the most powerful 100 people in the UK or the world for less than a million dollars (~£10⁶) and a decade-long project on a scale of just ~£10⁷could have dramatic effects.

The real obstacle is not a huge task of public persuasion – quite the opposite. A government that tried in a disciplined way to do this would attract huge public support. (I’ve polled some ideas and am confident about this.) Political parties are locked in a game that in trying to win in conventional ways leads to the public despising them. Ironically if a party (established or new) forgets this game and makes the public the target of extreme intelligent focus then it would not only make the world better but would trounce their opponents.

The real obstacle is not a need for breakthrough technologies though technology could help. As Colonel Boyd used to shout, ‘People, ideas, machines – in that order!’

The real obstacle is that although we can all learn and study HPTs it is extremely hard to put this learning to practical use and sustain it against all the forces of entropy that constantly operate to degrade high performance once the original people have gone. HPTs are episodic. They seem to come out of nowhere, shock people, then vanish with the rare individuals. People write about them and many talk about learning from them but in fact almost nobody ever learns from them – apart, perhaps, from those very rare people who did not need to learn – and nobody has found a method to embed this learning reliably and systematically in institutions that can maintain it. The Prussian General Staff remained operationally brilliant but in other ways went badly wrong after the death of the elder Moltke. When George Mueller left NASA it reverted to what it had been before he arrived – management chaos. All the best companies quickly go downhill after the departure of people like Bill Gates – even when such very able people have tried very very hard to avoid exactly this problem.

Charlie Munger, half of the most successful investment team in world history, has a great phrase he uses to explain their success that gets to the heart of this problem:

‘There isn’t one novel thought in all of how Berkshire [Hathaway] is run. It’s all about … exploiting unrecognized simplicities… It’s a community of like-minded people, and that makes most decisions into no-brainers. Warren [Buffett] and I aren’t prodigies. We can’t play chess blindfolded or be concert pianists. But the results are prodigious, because we have a temperamental advantage that more than compensates for a lack of IQ points.’

The simplicities that bring high performance in general, not just in investing, are largely unrecognised because they conflict with many evolved instincts and are therefore psychologically very hard to implement. The principles of the Buffett-Munger success are clear – they have even gone to great pains to explain them and what the rest of us should do – and the results are clear yet still almost nobody really listens to them and above average intelligence people instead constantly put their money into active fund management that is proved to destroy wealth every year!

Most people think they are already implementing these lessons and usually strongly reject the idea that they are not. This means that just explaining things is very unlikely to work:

‘I’d say the history that Charlie [Munger] and I have had of persuading decent, intelligent people who we thought were doing unintelligent things to change their course of action has been poor.’ Buffett.

Even more worrying, it is extremely hard to take over organisations that are not run right and make them excellent.

‘We really don’t believe in buying into organisations to change them.’ Buffett.

If people won’t listen to the world’s most successful investor in history on his own subject, and even he finds it too hard to take over failing businesses and turn them around, how likely is it that politicians and officials incentivised to keep things as they are will listen to ideas about how to do things better? How likely is it that a team can take over broken government institutions and make them dramatically better in a way that outlasts the people who do it? Bureaucracies are extraordinarily resistant to learning. Even after the debacles of 9/11 and the Iraq War, costing many lives and trillions of dollars, and even after the 2008 Crash, the security and financial bureaucracies in America and Europe are essentially the same and operate on the same principles.

Buffett’s success is partly due to his discipline in sticking within what he and Munger call their ‘circle of competence’. Within this circle they have proved the wisdom of avoiding trying to persuade people to change their minds and avoiding trying to fix broken institutions.

This option is not available in politics. The Enlightenment and the scientific revolution give us no choice but to try to persuade people and try to fix or replace broken institutions. In general ‘it is better to undertake revolution than undergo it’. How might we go about it? What can people who do not have any significant power inside the system do? What international projects are most likely to spark the sort of big changes in attitude we urgently need?

This is the first of a series. I will keep it separate from the series on the EU referendum though it is connected in the sense that I spent a year on the referendum in the belief that winning it was a necessary though not sufficient condition for Britain to play a part in improving the quality of government dramatically and improving the probability of avoiding the disasters that will happen if politics follows a normal path. I intended to implement some of these ideas in Downing Street if the Boris-Gove team had not blown up. The more I study this issue the more confident I am that dramatic improvements are possible and the more pessimistic I am that they will happen soon enough.

Please leave comments and corrections…

* A new transatlantic cable recently opened for financial trading. Its cost? £300 million. Its advantage? It shaves 2.6 milliseconds off the latency of financial trades. Innovative groups are discussing the application of military laser technology, unmanned drones circling the earth acting as routers, and even the use of neutrino communication (because neutrinos can go straight through the earth just as zillions pass through your body every second without colliding with its atoms) – cf. this recent survey in Nature.

August 19, 2014 by dominiccummings

‘Standin’ by the window, where the light is strong’: de-extinction, machine intelligence, the search for extra-solar life, autonomous drone swarms bombing Parliament, genetics & IQ, science & politics, and much more @ SciFoo 2014

‘SciFoo’ 8-10 August 2014, the Googleplex, Silicon Valley, California.

On Friday 8 August, I woke up in Big Sur (the coast of Northern California), looked out over the waves breaking on the wild empty coastline, munched a delicious Mexican breakfast at Deetjen’s, then drove north on Highway 1 towards Palo Alto where a few hours later I found myself looking through the windows of Google’s HQ at a glittering sunset in Silicon Valley.

I was going to ‘SciFoo’. SciFoo is a weekend science conference. It is hosted by Larry Page at Google’s HQ in Silicon Valley and organised by various people including the brilliant Timo Hannay from Digital Science.

I was invited because of my essay that became public last year (cf. HERE). Of the 200+ people, I was probably the only one who made zero positive contribution to the fascinating weekend and therefore wasted a place, so although it was a fantastic experience for me the organisers should not invite me back and I feel guilty about the person who could not go because I was there. At least I can let others know about some of the things discussed… (Although it was theoretically ‘on the record unless stated otherwise’, I could tell that many scientists were not thinking about this and so I have left out some things that I think they would not want attributed. Given they were not experienced politicians being interviewed but scientists at a scientific conference, I’m erring on the side of caution, particularly given the subjects discussed.)

It was very interesting to see many of the people whose work I mentioned in my essay and watch them interacting with each other – intellectually and psychologically / physically.

I will describe some of the things that struck me though, because there are about 7-10 sessions going on simultaneously, this is only a small snapshot.

In my essay, I discuss some of the background to many of these subjects so I will put references [in square brackets] so people can refer to it if they want.

Please note that below I am reporting what I think others were saying – unless it is clear, I am not giving my own views. On technical issues, I do not have my ‘own’ views – I do not have relevant skills. All I can do is judge where consensus lies and how strong it is. Many important issues involve asking at least 1) is there a strong scientific consensus on X among physical scientists with hard quantitative data to support their ideas (uber-example, the Standard Model of particle physics), b) what are the non-science issues, such as ‘what will it cost, who pays/suffers and why?’ On A, I can only try to judge what technically skilled people think. B is a different matter.

Whether you were there or not, please leave corrections / additions / questions in the comments box. Apologies for errors…

In a nutshell, a few likely scenarios / ideas, without spelling out caveats… 1) Extinct species are soon going to be brought back to life and the same technology will be used to modify existing species to help prevent them going extinct. 2) CRISPR – a new gene editing technology – will be used to cure diseases and ‘enhance’ human performance but may also enable garage bio-hackers to make other species extinct. 3) With the launch of satellites in 2017/18, we may find signs of life by 2020 among the ~10¹¹ exoplanets we now know exist just in our own galaxy though it will probably take 20-30 years, but the search will also soon get crowdsourced in a way schools can join in. 4) There is a reasonable chance we will have found many of the genes for IQ within a decade via BGI’s project, and the rich may use this information for embryo selection. 5) ‘Artificial neural networks’ are already outperforming humans on various pattern-recognition problems and will continue to advance rapidly. 6) Automation will push issues like a negative income tax onto the political agenda as millions lose their jobs to automation. 7) Autonomous drones will be used for assassinations in Europe and America shortly. 8) Read Neil Gershenfeld’s book ‘FAB’ if you haven’t and are interested in science education / 3D printing / computer science (or at least watch his TED talks). 9) Scientists are desperate to influence policy and politics but do not know how.

Biological engineering / computational biology / synthetic biology [Section 4]

George Church (Harvard), a world-leading biologist, spoke at a few sessions and his team’s research interests were much discussed. (Don’t assume he said any specific thing below.)

The falling cost of DNA sequencing continues to spur all sorts of advances. It has fallen from a billion dollars per genome a decade ago to less than a thousand dollars now (a million-fold improvement), and the Pentagon is planning on it reaching $100 soon. We can also sequence cancer cells to track their evolution in the body.

CRISPR. CRISPR is a new (2012) and very hot technology that is a sort of ‘cut and paste’ gene editing tool. It allows much more precise and effective engineering of genomes. Labs across America are rushing to apply it to all sorts of problems. In March this year, it was used to correct faulty genes in mice and cure them of a liver condition. It plays a major part in many of the biological issues sketched below.

‘De-extinction’ (bringing extinct species back to life). People are now planning the practical steps for de-extinction to the extent that they are scoping out land in Siberia where woolly mammoths will roam. As well as creating whole organisms, they will also grow organs modified by particular genes to test what specific genes and combinations do. This is no longer sci-fi – it is being planned and is likely to happen. The buffalo population was recently re-built (Google serves buffalo burgers in its amazing kitchens) from a tiny population to hundreds of thousands and there seems no reason to think it is impossible to build a significant population from scratch.

What does this mean? You take the DNA from an animal, say a woolly mammoth buried in the ground, sequence it, then use the digitised genome to create an embryo and either grow it in a similar animal (e.g. elephant for a mammoth) or in an artificial womb. (I missed the bit explaining the rationale for some of the proposed projects but, apart from the scientific reasons, one rationale for the mammoth was described as a conservation effort to preserve the frozen tundra and prevent massive amounts of greenhouse gases being released from beneath it.)

There are also possibilities of using this technology for conservation. For example, one could re-engineer the Asian elephant so that it could survive in less hospitable climates (e.g. modify the genes that produce haemoglobin so it is viable in colder places).

Now that we have sequenced the genome for Neanderthals (and learned that humans interbred with them, so you have traces of their DNA – unless you’re an indigenous sub-Saharan African), there is no known physical reason why we could not bring a Neanderthal back to life once the technology has been refined on other animals. This obviously raises many ethical issues – e.g. if we did it, they would have to be given the same legal rights as us (one distinguished person said that if there were one in the room with us we would not notice, contra the pictures often used to illustrate them). It is assumed by many that this will happen (nobody questioned the assumption) – just as it seemed to be generally assumed that human cloning will happen – though probably not in a western country but somewhere with fewer legal restrictions, after the basic technologies have been refined. (The Harvard team gets emails from women volunteering to be the Neanderthal’s surrogate mum.)

‘Biohacking’. Biohacking is advancing faster than Moore’s Law. CRISPR editing will allow us to enhance ourselves. E.g. Tibetans have evolved much more efficient systems for coping with high altitude, and some Africans have much stronger bones than the rest of us (see below). Will we reengineer ourselves to obtain these advantages? CRISPR obviously also empowers all sorts of malevolent actors too – cf. this very recent paper (by Church et al). It may soon be possible for people in their garages to edit genomes and accidentally or deliberately drive species to extinction as well as attempt to release deadly pathogens. I could not understand why people were not more worried about this – I hope I was missing a lot. (Some had the attitude that ‘nature already does bio-terrorism’ so we should relax. I did not find this comforting and I’m sure I am in the majority so for anybody influential reading this I would strongly advise you not to use this argument in public advocacy or it is likely to accelerate calls for your labs to be shut down.)

‘Junk’. There is more and more analysis of what used to be called ‘junk DNA’. It is now clear that far from being ‘junk’ much of this has functions we do not understand. This connects to the issue that although we sequenced the human genome over a decade ago, the quality of the ‘reference’ version is not great and (it sounded like from the discussions) it needs upgrading.

‘Push button’ cheap DNA sequencers are around the corner. Might such devices become as ubiquitous as desktop printers? Why doesn’t someone create a ‘gene web browser’ that can cope with all the different data formats for genomes?

Privacy. There was a lot of talk about ‘do you want your genome on the web?’. I asked a quick informal pop quiz (someone else’s idea): there was unanimity that ‘I’d much rather my genome was on the web than my browsing history’. [UPDATE: n<10 and perhaps they were tongue in cheek!? One scientist pointed out in a session that when he informed his insurance company, after sequencing his own genome, that he had a very high risk of getting colon cancer, they raised his premiums. There are all sorts of reasons one would want to control genomic information and I was being a bit facetious.]

In many ways, computational biology and synthetic biology have that revolutionary feeling of the PC revolution in the 1970s – huge energy, massive potential for people without big resources to make big contributions, the young crowding in, the feeling of dramatic improvements imminent. Will this all seem ‘too risky’? It’s hard to know how the public will respond to risk. We put up with predictable annual carnage from car accidents but freak out over trivia. We ignore millions of deaths in the Congo but freak out over a handful in Israel/Gaza. My feeling is some of the scientists are too blasé about how the public will react to the risks, but I was wrong about how much fear there would be about the news that scientists recently deliberately engineered a much more dangerous version of an animal flu.

AI / machine learning / neuroscience [Section 5].

Artificial neural networks (NNs), now often referred to as ‘deep learning’, were first created 50 years ago but languished for a while when progress slowed. The field is now hot again. (Last year Google bought some companies leading the field, and a company, Boston Dynamics, that has had a long-term collaboration with DARPA.)

Jurgen Schmidhuber explained progress and how NNs have recently approached or surpassed human performance in various fields. E.g. recently NNs have surpassed human performance in recognising traffic signals (0.56% error rate for the best NN versus 1.16% for humans). Progress in all sorts of pattern recognition problems is clearly going to continue rapidly. E.g. NNs are now being used to automate a) the analysis of scans for cancer cells and b) the labelling of scans of human brains – so artificial neural networks are now scanning and labelling natural neural networks.

Steve Hsu has blogged about this session here:

http://infoproc.blogspot.co.uk/2014/08/neural-networks-and-deep-learning.html?m=1

Michael Nielsen is publishing an education project online for people to teach themselves the basics of neural networks. It is brilliant and I would strongly advise teachers reading this blog to consider introducing it into their schools and doing the course with the pupils.

http://neuralnetworksanddeeplearning.com

Neil Gershenfeld (MIT) gave a couple of presentations. One was on developments in computer science connecting: non-‘von Neumann architecture’, programmable matter, 3D printing, ‘the internet of things’ etc. [Cf. Section 3.] NB. IBM announced this month substantial progress in their quest for a new computer architecture that is ‘non-Von Neumann’: cf. this –

http://venturebeat.com/2014/08/07/ibms-synapse-marshals-the-power-of-the-human-brain-in-a-computer/view-all/

Another was on the idea of an ‘interspecies internet’. We now know many species can recognise each other, think, and communicate much better than we realised. He showed bonobos playing music with Peter Gabriel and dolphins communicating. He and others are plugging them into the internet. Some are doing this to help the general goal of figuring out how we might communicate with intelligent aliens – or how they might communicate with us.

(Gershenfeld’s book FAB led me to push 3D printing into the new National Curriculum and I would urge school science teachers to watch his TED talks and read this book. [INSERTED LATER: Some people have asked about this point. I (I thought obviously) did not mean I wrote the NC document. I meant – I pushed the subject into the discussions with the committees/drafters who wrote the NC. Experts in the field agreed it belonged. When it came out, this was not controversial. We also funded pilots with 3D printers so schools could get good advice about how to teach the subject well.] His point about 3D printers restoring the connection between thinking and making – lost post-Renaissance – is of great importance and could help end the foolishly entrenched ‘knowledge’ vs ‘skills’ and academic vs vocational trench wars. Gove actually gave a speech about this not long before he was moved and as far as I could tell it got less coverage than any speech he ever gave, thus proving the cliché about speeches on ‘skills’.)

There were a few presentations about ‘computational neuroscience’. I could not understand anything much as they were too technical. It was clear that there is deep concern among EU neuroscientists about the EU’s huge funding for Henry Markram’s Human Brain Project. One leading neuroscientist said to me that the whole project is misguided as it does not have clear focused goals and the ‘overhype’ will lead to public anger in a few years. Apparently, the EU is reconsidering the project and its goals. I have no idea about the merits of these arguments. I have a general prejudice that, outside special circumstances, experience suggests that it is better to put funding into many pots and see what works, as DARPA does.

There are all sorts of crossovers between: AI / neuroscience / big data / NNs / algorithmic pattern recognition in other fields.

Peter Norvig, a leader in machine intelligence, said that he is more worried about the imminent social implications of continued advances making millions unemployed than he is about a sudden ‘Terminator / SKYNET’ scenario of a general purpose AI bootstrapping itself to greater than human intelligence and exterminating us all. Let’s hope so. It is obvious that this field is going to keep pushing boundaries – in open, commercial, and classified projects – so we are essentially going to be hoping for the best as we make more and more advances in AI. The idea of a ‘negative income tax’ – or some other form of essentially paying people X just to live – seems bound to return to the agenda. I think it could be a way around all sorts of welfare arguments. The main obstacle, it seems to me, is that people won’t accept paying for it if they think uncontrolled immigration will continue as it is now.

Space [Section 2]

There was great interest in various space projects and some senior people from NASA. There is much sadness at how NASA, despite many great people, has become a normal government institution – ie. caught in DC politics, very bureaucratic, and dysfunctional in various ways. On the other hand, many private ventures are now growing. E.g. Elon Musk is lowering the $/kg of getting material into orbit and planning a non-government Mars mission. As I said in my essay, really opening up space requires a space economy – not just pure science and research (such as putting telescopes on the far side of the moon, which we obviously should do). Columbus opened up America – not the Vikings.

There is another obvious motive. As Carl Sagan said, if the dinosaurs had had a space programme, they’d still be here. In the long-term we either develop tools for dealing with asteroids or we will be destroyed. We know this for sure. I think I heard that NASA is planning to park a small asteroid close to the moon around 2020 but I may have misheard / misunderstood.

Mario Livio led a great session on the search for life on exoplanets. The galaxy has ~10¹¹ stars and there is ~1 planet on average per star. There are ~10¹¹ galaxies, so a Fermi estimate is there are ~10²² planets – 10 billion trillion planets – in the observable universe (this number is roughly 1,000 times bigger than the number you get in the fable of putting a grain of rice on the first square of a chessboard and doubling on each subsequent square). Many of them are in the ‘habitable zone’ around stars.

In 2017/18, there are two satellites launching that will be able to do spectroscopy on exoplanets – i.e. examine their atmospheres and detect things like oxygen and water. ‘If we get lucky’, these satellites will find ‘bio-signatures’ of life. If they find life having looked at only a few planets, then it would mean that life is very common. ‘More likely’ is it will take 20-30 years and a new generation of space-based telescopes to find life. If planets are found with likely biosignatures, then it would make sense to turn SETI’s instruments towards them to see if they find anything. (However, we are already phasing out the use of radio waves for various communications – perhaps the use of radio waves is only a short window in the lifetime of a civilisation.) There are complex Bayesian arguments about what we might infer about our own likely future given various discoveries but I won’t go into those now. (E.g. if we find life is common but no traces of intelligent life, does this mean a) the evolution of complex life is not a common development from simple life; b) intelligent life is also common but it destroys itself; c) they’re hiding, etc.)

A very impressive (and helpful towards the ignorant like me) young scientist working on exoplanets called Oliver Guyon demonstrated a fascinating project to crowdsource the search for exoplanets by building a global network of automated cameras – PANOPTES (www.projectpanoptes.org). His team has built a simple system that can find exoplanets using normal digital cameras costing less than $1,000. They sit in a box connected to a 12V power supply, automatically take pictures of the night sky every few seconds, then email the data to the cloud. There, the data is aggregated and algorithms search for exoplanets. These units are cheap (can’t remember what he said but I think <$5,000). Everything is open-source, open-hardware. They will start shipping later this year and will make a brilliant school science project. Guyon has made the project with schools in mind so that assembling and operating the units will not require professional level skills. They are also exploring the next move to connect smartphone cameras.

Building the >15m diameter space telescopes we need to search for life seems to me an obvious priority for scientific budgets – it is one of the handful of the most profound questions facing us.

There was an interesting cross-over discussion about ‘space and genetics’ in which people discussed various ways in which space exploration would encourage / require genetic modification. E.g.1 some sort of rocket fuel has recently been discovered to exist in large quantities on Mars. This is very handy but the substance is toxic. It might therefore make sense to modify humans going to live on Mars to be resistant. E.g.2 Space travel weakens bones. It has been discovered that mutations in the human population can improve bone strength by 8 standard deviations. This is a massive improvement – for comparison, 8 SDs in IQ covers people from severely mentally disabled to Nobel-winners. This was discovered by a team of scientists in Africa who noticed that people in a local tribe who got hit by cars did not suffer broken bones, so they sequenced the locals’ genomes. (Someone said there have already been successful clinical trials testing this discovery in a real drug to deal with osteoporosis.) E.g.3 Engineering E. Coli shows that just four mutations can improve resistance to radiation by ?1,000 times (can’t read my note).

Craig Venter and others are thinking about long-term projects to send ‘von Neumman-bots’ (self-replicating space drones) across the universe containing machines that could create biological life once they arrive somewhere interesting, thus avoiding the difficult problems of keeping humans alive for thousands of years on spaceships. (Nobel-winning physicist Gerard t’ Hooft explains the basic principles of this in his book Playing with planets.)

This paper (August 2014) summarises issues in the search for life:

http://www.pnas.org/content/early/2014/08/01/1304213111.full.pdf

Finding the genes for IQ and engineering possibilities [Section 5].

When my essay came out last year, there was a lot of mistaken reporting that encouraged many in the education world to grab the wrong end of the stick about IQ, though the BBC documentary about the controversy (cf. below) was excellent and a big step forward. It remains the case that very few people realise that in the last couple of years direct examination of DNA has now vindicated the consistent numbers on IQ heritability from decades of twin/adoption studies.

The rough heritability numbers for IQ are no longer in doubt among physical scientists who study this field: it is roughly 50% heritable at age ~18-20 and this number rises towards 70-80% for older adults. This is important because IQ is such a good predictor of the future – it is a better predictor than social class. E.g. The long-term Study of Mathematically Precocious Youth, which follows what has happened to children with 1:10,000 ability, shows among many things that a) a simple ‘noisy’ test administered at age 12-13 can make amazingly accurate predictions about their future, and b) achievements such as scientific breakthroughs correlate strongly with IQ. (If people looked at the data from SMPY, then I think some of the heat and noise in the debate would fade but it is a sad fact that approximately zero senior powerful people in the English education world had even heard of this study before the furore over Plomin last year.)

Further, the environmental effects that are important are not the things that people assume. If you test the IQ of an adopted child in adulthood and the parents who adopted it, you find approximately zero correlation – all those anguished parenting discussions had approximately no measurable impact on IQ. (This does not mean that ‘parenting doesn’t matter’ – parents can transfer narrow skills such as playing the violin.) In the technical language, the environmental effects that are important are ‘non-shared’ environmental effects – i.e. they are things that two identical twins do not experience in the same way. We do not know what they are. It is reasonable to think that they are effectively random tiny events with nonlinear effects that we may never be able to track in detail – cf. this paper for a discussion of this issue in the context of epidemiology: http://ije.oxfordjournals.org/content/40/3/537.full.pdf+html

There remains widespread confusion on this subject among social scientists, education researchers, and the worlds of politics and the media where people were told misleading things in the 1980s and 1990s and do not realise that the debates have been transformed. To be fair, however, it was clear from this weekend that even many biologists do not know about new developments in this field so it is not surprising that political journalists and education researchers do not.

(An example of confusion in the political/media world… In my essay, I used the technical term ‘heritable’ which is a population statistic – not a statement about an individual. I also predicted that media coverage would confuse the subject (e.g. by saying things like ‘70% of your IQ comes from genes’). Sure enough some journalists claimed I said the opposite of what I actually said then they quoted scientists attacking me for making a mistake that not only did I not make but which I actually warned about. Possibly the most confused sentence of all those in the media about my essay was the line ‘wealth is more heritable than genes’, which was in Polly Toynbee’s column and accompanying headline in the Guardian. This sentence is a nonsense sentence as it completely mangles the meaning of the term ‘heritable’. Much prominent commentary from politicians and sociologists/economists on ‘social mobility’ is gibberish because of mistaken assumptions about genes and environment. The Endnote in my essay has links to work by Plomin, Hsu et al that explains it all properly. This interview with Plomin is excellent: http://www.spectator.co.uk/features/8970941/sorry-but-intelligence-really-is-in-the-genes/. This recent BBC radio programme is excellent and summarises the complex issues well: http://www.bbc.co.uk/programmes/b042q944/episodes/guide)

I had a fascinating discussion/tutorial at SciFoo with Steve Hsu. Steve Hsu is a professor of theoretical physics (and successful entrepreneur) with a long interest in IQ (he also runs a brilliant blog that will keep you up to speed on all sorts). He now works part time on the BGI project in China to discover the genes responsible for IQ.

IQ is very similar to height from the perspective of behavioural genetics. Height has the advantage that it is obviously easier to measure than IQ but it has roughly the same heritability. Large scale GWAS are already identifying some of the genes responsible for height. Hsu recently watched a talk by Fields Medallist Terry Tao and realised that a branch of maths could be used to examine the question – how many genomes do we need to scan to identify a substantial number of the genes for IQ? His answer: ‘roughly 10k moderately rare causal variants of mostly negative effect are responsible for normal population variation’ and finding them will require sequencing roughly a million genomes. The falling cost of sequencing DNA means that this is within reach. ‘At the time of this writing SNP genotyping costs are below $50 USD per individual, meaning that a single super-wealthy benefactor could independently fund a crash program for less than $100 million’ (Hsu).

The BGI project to find these genes has hit some snags recently (e.g. a US lawsuit between the two biggest suppliers of gene sequencing machines). However, it is now expected to start again soon. Hsu thinks that within a decade we could find many of the genes responsible for IQ. He has just put his fascinating paper on this subject on his blog (there is also a Q&A on p.27 that will be very useful for journalists):

http://infoproc.blogspot.co.uk/2014/08/genetic-architecture-of-intelligence.html

Just discovering a substantial fraction of the genes would be momentous in itself but there is more. It is already the case that farmers use genomes to make predictions about cows’ properties and behaviour (‘genotype to phenotype’ predictions). It is already the case that rich people could use in vitro fertilisation to select the egg which they think will be most advantageous, because they can sequence genomes of multiple eggs and examine each one to look for problems then pick the one they prefer. Once we identify a substantial number of IQ genes, there is no obvious reason why rich people will not select the egg that has the highest prediction for IQ.

This clearly raises many big questions. If the poor cannot do the same, then the rich could quickly embed advantages and society could become not only more unequal but also based on biological classes. One response is that if this sort of thing does become possible, then a national health system should fund everybody to do this. (I.e. It would not mandate such a process but it would give everybody a choice of whether to make use of it.) Once the knowledge exists, it is hard to see what will stop some people making use of it and offering services to – at least – the super-rich.

It is vital to separate two things: a) the basic science of genetics and cognition (which must be allowed to develop), and b) the potential technological applications and their social implications. The latter will rightly make people deeply worried, given our history, and clearly require extremely serious public debate. One of the reasons I wrote my essay was to try to stimulate such debate on the biggest – and potentially most dangerous – scientific issues. By largely ignoring such issues, Westminster, Whitehall, and the political media are wasting the time we have to discuss them so technological breakthroughs will be unnecessarily shocking when they come.

Hsu’s contribution to this research – and his insight when listening to Tao about how to apply a branch of mathematics to a problem – is also a good example of how the more abstract fields of maths and physics often make contributions to the messier study of biology and society. The famous mathematician von Neumann practically invented some new fields outside maths and made many contributions to others. The physicist-mathematician Freeman Dyson recently made a major contribution to Game Theory which had lain unnoticed for decades until he realised that a piece of maths could be applied to uncover new strategies (Google “Dyson zero determinant strategies” and cf. this good piece: http://www.americanscientist.org/issues/id.16112,y.0,no.,content.true,page.1,css.print/issue.aspx).

However, this also raises a difficult issue. There is a great deal of Hsu’s paper – and the subject of IQ and heritability generally – that I do not have the mathematical skills to understand. This will be true of a large fraction of education researchers in education departments – I would bet a large majority. This problem is similar for many other vital issues (and applies to MPs and their advisers) and requires general work on translating such research into forms that can be explained by the media.

Kathryn Ashbury also did a session on genes and education but I went to a conflicting one with George Church so unfortunately I missed it.

‘Big data’, simulations, and distributed systems [Section 6&7]

The rival to Markram’s Brain Project for mega EU funding was Dirk Helbing (ETH Zurich) and his project for new simulations to aid policy-making. Helbing was also at SciFoo and gave a couple of presentations. I will write separately about this.

Helbing says convincingly: ‘science must become a fifth pillar of democracies, besides legislation, executive, jurisdiction, and the public media’. Many in politics hope that technology will help them control things that now feel out of control. This is unlikely. The amount of data is growing at a faster rate than the power of processing and the complexity of networked systems grows factorially therefore top-down control will become less and less effective.

The alternative? ‘Distributed (self-)control, i.e. bottom-up self-regulation’. E.g. Helbing’s team has invented self-regulating traffic lights driven by traffic flows that can ‘outperform the classical top-down control by a conventional traffic center.’

‘Can we transfer and extend this principle to socio-economic systems? Indeed, we are now developing mechanisms to overcome coordination and cooperation failures, conflicts, and other age-old problems. This can be done with suitably designed social media and sensor networks for real-time measurements, which will eventually weave a Planetary Nervous System. Hence, we can finally realize the dream of self-regulating systems… [S]uitable institutions such as certain social media – combined with suitable reputation systems – can promote other-regarding decision-making. The quick spreading of social media and reputation systems, in fact, indicates the emergence of a superior organizational principle, which creates collective intelligence by harvesting the value of diversity…’

His project’s website is here:

http://www.futurict.eu

I wish MPs and spads in all parties would look at this project and Helbing’s work. It provides technologically viable and theoretically justifiable mechanisms to avoid the current sterile party debates about delivery of services. We must move from Whitehall control to distributed systems…

Science and politics

Unsurprisingly, there was a lot of grumbling about politicians, regulation, Washington gridlock, bureaucracy and so on.

Much of it is clearly justified. Some working in genetics had stories about how the regulations forbid them to tell people about imminently life threatening medical problems they discover. Others were bemoaning the lack of action on asteroid defence and climate change.

Some of these problems are inherently extremely difficult, as I discuss in my essay. On top of this, though, is the problem that many (most?) scientists do not know how to go about changing things.

It was interesting that some very eminent scientists, all much cleverer than ~100% of those in politics [INSERT: better to say ‘all with higher IQ than ~100% of those in politics’], have naive views about how politics works. In group discussions, there was little focused discussion about how they could influence politics better even though it is clearly a subject that they care about very much. (Gershenfeld said that scientists have recently launched a bid to take over various local government functions in Barcelona, which sounds interesting.)

A few times I nearly joined in the discussion but I thought it would disrupt things and distract them. In retrospect, I think this may have been a mistake and I should have spoken up. But also I am not articulate and I worried I would not be able to explain their errors and it would waste their time.

I will blog on this issue separately. A few simple observations…

To get things changed in politics, scientists need mechanisms a) to agree priorities in order to focus their actions on b) roadmaps with specifics. Generalised whining never works. The way to influence politicians is to make it easy for them to fall down certain paths without much thought, and this means having a general set of goals but also a detailed roadmap the politicians can apply, otherwise they will drift by default to the daily fog of chaos and moonlight.

Scientists also need to be prepared to put their heads above the parapet and face controversy. Many comments amounted to ‘why don’t politicians do the obviously rational thing without me having to take a risk of being embroiled in media horrors’. Sorry guys but this is not how it works.

Many academics are entirely focused on their research and do not want to lose time to politics. This is entirely reasonable. But if you won’t get involved you can have little influence other than lending your name to the efforts of others.

Working in the Department for Education, I have experienced in England that very few scientists were prepared to face controversy over the issue of A Levels (exams at 18) and university entry / undergraduate standards even though this problem directly affected their own research area. Many dozens sought me out 2007-14 to complain about existing systems. I can count on the fingers of one hand those who rolled the dice and did things in the public domain that could have caused them problems. I have heard many scientists complain about media reports but when I’ve said – ‘write a blog explaining why they’re wrong’, the answer is almost invariably ‘oh, the VC’s office would go mad’. If they won’t put their heads above the parapet on an issue that directly touches their own subject and career, how much are they likely to achieve in moving political debate in areas outside their own fields?

Provided scientists a) want to avoid controversy and b) are isolated, they cannot have the leverage they want. The way to minimise controversy is to combine in groups – for the evolutionary biologists reading this, think SHOALS! – so that each individual is less exposed. But you will only join a shoal if you agree a common purpose.

I’m going to do a blog on ‘How scientists can learn from Bismarck and Jean Monnet to influence politics‘. Monnet avoided immediate battles for power in favour of ‘preparing the future’ – i.e. having plans in his pocket for when crises hit and politicians were desperate. He created the EEC in this way. In the same way people find it extremely hard to operationalise the lessons of Thucydides or Bismarck, they do not operationalise the lessons from Monnet. It would be interesting if scientists did this in a disciplined way. In some ways, it seems to me vital if we are to avoid various disasters. It is also necessary, however, to expose scientists to the non-scientific factors in play.

Anyway, it would be worth exploring this question: can very high IQ people with certain personality traits (like von Neumann, not like Gödel) learn enough in half a day’s exposure to case studies of successful political action to enable them to change something significant in politics, provided someone else can do most of the admin donkey work? I’m willing to bet the answer is YES. Whether they will then take personal risks by ACTING is another question.

A physicist remarked: ‘we’re bitching about politicians but we can’t even sort out our own field of scientific publishing which is a mess’.

NB. for scientists who haven’t read anything I’ve read before, do not make the mistake of thinking I am defending politicians. If you read other stuff I’ve written you will see that I have made all the criticisms that you have. But that doesn’t mean that scientists cannot do much better than they are at influencing policy.

A few general comments

1. It has puzzled me for over a decade that a) one of the few things the UK still has that is world class is Oxbridge, b) we have the example of Silicon Valley and our own history of post-1945 bungling to compare it with (e.g. how the Pentagon treated von Neumann and how we treated Turing viz the issue of developing computer science), yet c) we persistently fail to develop venture capital-based hubs around Oxbridge on the scale they deserve. As I pottered down University Avenue in Palo Alto looking for a haircut, past venture capital offices that can provide billions in start-up investment, I thought: you’ve made a few half-hearted attempts to persuade people to do more on this, when you get home try again. So I will…

2. It was interesting to see how physicists have core mathematical skills that allow them to grasp fundamentals of other fields without prior study. Watching them reminded me of Mandelbrot’s comment that:

‘It is an extraordinary feature of science that the most diverse, seemingly unrelated, phenomena can be described with the same mathematical tools. The same quadratic equation with which the ancients drew right angles to build their temples can be used today by a banker to calculate the yield to maturity of a new, two-year bond. The same techniques of calculus developed by Newton and Leibniz two centuries ago to study the orbits of Mars and Mercury can be used today by a civil engineer to calculate the maximum stress on a new bridge… But the variety of natural phenomena is boundless while, despite all appearances to the contrary, the number of really distinct mathematical concepts and tools at our disposal is surprisingly small… When we explore the vast realm of natural and human behavior, we find the most useful tools of measurement and calculation are based on surprisingly few basic ideas.’

3. High status people have more confidence in asking basic / fundamental / possibly stupid questions. One can see people thinking ‘I thought that but didn’t say it in case people thought it was stupid and now the famous guy’s said it and everyone thinks he’s profound’. The famous guys don’t worry about looking stupid and they want to get down to fundamentals in fields outside their own.

4. I do not mean this critically but watching some of the participants I was reminded of Freeman Dyson’s comment:

‘I feel it myself, the glitter of nuclear weapons. It is irresistible if you come to them as a scientist. To feel it’s there in your hands. To release the energy that fuels the stars. To let it do your bidding. And to perform these miracles, to lift a million tons of rock into the sky, it is something that gives people an illusion of illimitable power, and it is in some ways responsible for all our troubles, I would say, this is what you might call ‘technical arrogance’ that overcomes people when they see what they can do with their minds.’

People talk about rationales for all sorts of things but looking in their eyes the fundamental driver seems to be – am I right, can I do it, do the patterns in my mind reflect something real? People like this are going to do new things if they can and they are cleverer than the regulators. As a community I think it is fair to say that outside odd fields like nuclear weapons research (which is odd because it still requires not only a large collection of highly skilled people but also a lot of money and all sorts of elements that are hard (but not impossible) for a non-state actor to acquire and use without detection), they believe that pushing the barriers of knowledge is right and inevitable. Fifteen years on from the publication by Silicon Valley legend Bill Joy of his famous essay (‘Why the future doesn’t need us’), it is clear that many of the things he feared have proceeded and there remains no coherent government approach or serious international discussion. (I am not suggesting that banning things is generally the way forward.)

5. The only field where there was a group of people openly lobbying for something to be made illegal was the field of autonomous lethal drones. (There is a remorseless logic that means that countermeasures against non-autonomous drones (e.g. GPS-spoofing) incentivises one to make one’s drones autonomous. They can move about waiting to spot someone’s face then destroy them without any need for human input.) However, the discussion confirmed my view that even if this might be a good idea – it is doomed, in the short-term at least. I wonder what is to stop someone sending a drone swarm across the river and bombing Parliament during PMQs. Given it will be possible to deploy autonomous drones anonymously, it seems there may be a new era of assassinations coming, apart from all the other implications of drones. Given one may need a drone swarm to defend against drone swarm, I can’t see them being outlawed any time soon. (Cf. Suarez’s Kill Decision for a great techno-thriller on the subject.)

(Also, I thought that this was an area where those involved in cutting edge issues could benefit from talking to historians. E.g. my understanding is that we filmed the use of anthrax on a Scottish island and delivered the footage to the Nazis with the message that we would anthrax Germany if they used chemical weapons – i.e. the lack of chemical warfare in WWII was a case of successful deterrence, not international law.)

6. A common comment is – ‘technology X [e.g. in vitro fertilisation] was denounced at the time but humans adapt to such changes amazingly fast, so technology Y will be just the same’. This is a reasonable argument in some ways but I cannot help but think that many will think de-extinction, engineered bio-weapons, or human clones are going to be perceived as qualitative changes far beyond things like in vitro fertilisation.

7. Daniel Suarez told me what his next techno-thriller is about but if I put it on my blog he will deploy an autonomous drone with face recognition AI to kill me, so I’m keeping quiet. If you haven’t read Daemon, read it – it’s a rare book that makes you laugh out loud about how clever the plot is.

8. Von Neumann was heavily involved not only in the Manhattan Project but also the birth of the modern computer, the creation of the hydrogen bomb, and nuclear strategy. Before his tragic early death, he wrote a brilliant essay about the political problem of dealing with advanced technology which should be compulsory reading for all politicians aspiring to lead. It summarises the main problems that we face – ‘for progress, there is no cure…’

http://features.blogs.fortune.cnn.com/2013/01/13/can-we-survive-technology/

As I said at the top, any participants please tell me where I went wrong, and thanks for such a wonderful weekend.

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this: