A Conversation with Nicholas Wright on Cognitive Defense, the Joint Force in a Digitizing World

Clear and Present Podcast: Episode 1

---

---

Transcript

Dr. James Giordano (00:00):

Welcome to Clear and Present, a biweekly podcast of the Institute for Biodefense Research, where we bring subject matter experts to the fore to discuss their views and insights to current topics and issues at the interface of the biomedical sciences and technology, biosecurity, and biodefense.

Dr. James Giordano (00:38):

Hello, and welcome to this week’s podcast. I’m your host, Dr. James Giordano, from the Institute of Biodefense Research, and today I’m speaking with Dr. Nicholas Wright, a true polymath, Dr. Wright is an affiliated scholar with Georgetown university, the University College of London, Intelligent Biology and New America. He combines neuroscientific, behavioral, and technological insights to understand decision making in a variety of different settings: politics, international confrontations, in ways that are viable and useful for policy, for international governance, regulation and oversight. Again, a true polymath, Dr. Wright is both a physician and a neurocognitive scientist.

Dr. James Giordano (01:20):

I’ve had the pleasure of working Dr. Wright and the privilege of working with him before the strategic multi-layer assessment group at the Joint Staff of the Pentagon, where his ongoing work has really been focused upon that area of intersection between the biological sciences, cognitive sciences, and how they can interface international relations, national security intelligence and defense. And Nick was a practicing neurologist in Oxford and in London before receiving his PhD from the Laboratory at University College, London that spun out Google’s pioneering AI unit, DeepMind. He worked on nuclear strategy as a fellow for the Carnegie Endowment, Washington, DC, and he regularly advises government and a variety of businesses. He is the author of numerous academic and general publications and is editor of the book, Artificial Intelligence, China, Russia, and the Global Order. He received his medical degree from UCL and received a bachelor’s in health policy from Imperial college, London, a Master’s in science and neuroscience, and a PhD from neuroscience, both from UCL and is an elected member of the Royal College of Physicians. Nick, it is always a pleasure. Welcome.

Dr. Nicholas Wright (02:24):

Hi, thank you very much, Jim. And thank you very much for having me.

Dr. James Giordano (02:28):

It’s a pleasure once again. So let’s jump right into the discussion, because I think what’s so important is that late last year, you penned a report for the joint staff of the Pentagon entitled Cognitive Defense, the Joint Force in a Digitizing World. Well, I guess the key question here is knowing the work of the Pentagon, not only as it relates domestically, but internationally, what’s the interest there with regard to cognitive defense?

Dr. Nicholas Wright (02:55):

Thanks, Jim. I think that what we’ve seen over the past few years is that adversaries have become ever more sophisticated and capable in using information. And I think we can assume that those weapons are going to be wielded against the Joint Force in a future conflict. And so what they asked me to think about was the interaction of the human side and the technological side. So service personnel, their families, and their friends, they’re humans, you know. We’re all humans and what adversaries and other destabilizing forces are threatening to do is to sow discord and disruption amongst those humans, in order to degrade their collective capability, to stop them cooperating to achieve the missions that they need, that we need them to achieve. And in order to do that, adversaries can harness the powerful, new digital technologies that are immersing our lives.

Dr. Nicholas Wright (04:00):

I’m literally, obviously speaking to you now across the internet, I’m surrounded by digital devices. So are you, and people will be listening to the podcast on digital technologies. So the question was, or the thing they asked me to do was to think about how can we protect the humans in the Joint Force from adversary information campaigns. And just to say, I think if you put yourselves in the shoes of an adversary and you think about the Joint Force, in many ways, it’s an enormously capable system of systems, but it is also, if you think about it, millions of humans who comprise a large number of target audiences, and you could think of those as a very tempting smorgasbord of potential target audiences for information operations and fortunately, or unfortunately, depending on whether you’re the adversary or not.

Dr. Nicholas Wright (05:02):

Fortunately, for example, certainly last year, Facebook was very kindly able to neatly identify, characterize, and research, the members of the Joint Force, for example. You could specifically look to target members of the Joint Force and members of the US military. And if you think about China’s AI power TikTok as of last year, that was used by about 21% of US adults and by a staggering 48% of those 18 to 29. And of course, as you know, a lot of the US military are younger people. So we can assume, and a lot of people who in the future will become members of the US military are now on TikTok, which is fundamentally a Chinese platform. So this is a real challenge. We are not the only ones thinking about this. If you think about it from the point of view of an adversary, this is a very tempting smorgasbord. And our challenge is how can we defend these humans within the Joint Force?

Dr. James Giordano (06:19):

Let’s jump on that challenge because as you’ve well put it, it’s an opportunity for an adversary or, let’s back that up. It’s an opportunity even for a peer competitor. I mean, let’s face it, what you’re dealing with is narratives. Narratives can be influential and by influencing individuals, both tacitly and explicitly, which you’re really doing is you’re dictating bits of their cognitions. You’re setting particular biases, predispositions, attitudes, and values. That’s pretty strong. I mean, the level of control, again, both subtly and explicitly that that could exercise could be profound, clearly providing at least something of a plus some advantage to, again, an adversary or a peer competitor. But I guess the question then is, we’re talking about a smorgasbord of targets. Who is targetable? I mean, in your situation being asked to write a report such as this, I think one of the key issues is recommendations. Part of those recommendations have to be recommendations for whom and about what? Who are we defending, Nick?

Dr. Nicholas Wright (07:21):

I mean, this gets to the heart of some of the difficulties of working on this space. So just to give you some numbers. So there are currently, and these things were current last year. So there are currently around 1.3 million active duty service personnel in the US. In addition to that about some 4 million hold clearances at secret or above. Now, these US people, in addition to that, you’ve got to think about their families. Often the most effective way to get to somebody, to influence somebody, is through their families, or their social networks. This isn’t anything new, this isn’t anything that people haven’t known for millennia, but that’s often the best way to get to people. So you are now talking about many millions of people, and given the geographical dispersion of the US forces, they are distributed across the world. So it is a very difficult, it is a very large and amorphous group of people to defend. And inevitably, whenever you’re dealing with that just sheer number of people, there will be people who have mental health problems. There will be people who have gambling problems. There will be people who are political extremists of a wide variety of different flavors. That’s just the nature of when you’re dealing with any kind of large population, however well you screen it. And these are all people who can be targeted.

Dr. James Giordano (08:53):

So I guess the question then is twofold. Number one, is there some way of producing a safe zone, a fail safe within these programs? And number two, if the idea is to fail safe, which creates safe zones, then it becomes very important to ask the key question, well, what is it exactly we’re defending them from? I mean, obviously information that could be capricious nefarious, influential, in negativistic ways, but how do we identify that information? And of course, given the ubiquity of that information on the internet and these new platforms we have to ask, what’s old and what’s new? And how do we then create that scenario to identify things that are going to be problematic in terms of either being burdensome, risky or morbidly threatening?

Dr. Nicholas Wright (09:41):

I mean that, again, that’s a great question and this is part of the challenge. So, and if there is no, I don’t think we can provide 100% security in these, against these challenges for a wide variety of reasons. One of them is just the sheer scale. And the other is that, which we’re going to come on to later hopefully is that there will be significant issues around Democratic compatibility. So, the fact that you are the son or daughter of a US Joint Force member, does that mean that your private communications should be monitored by the US government when you are just sitting there in Arkansas, wherever it is, or St. Louis or at Chicago, whatever? And the answer is probably not, in most circumstances. Definitely not in a variety of circumstances, perhaps in other circumstances. So that’s, you are already raising very difficult issues. The not inseparable way-

Dr. James Giordano (10:55):

Let me just jump in for a quick question here, because I think it literally begs the question. If we’re asking, who are we defending, what are we defending them from? You do realize particularly domestically, that there is a bit of suspicion that exists in the United States. And I think that increasingly globally, with regard to the proliferation of surveillance, the idea that Big Brother is watching and here there’s something of a paradox, perhaps even a dilemma: we’re trying to protect people, agreed, by virtue of the paradigm that you propose. And what that involves is, as you may say, having to watch over them. But watching over them is just that: it’s watching over them. And it conjures images of, if you will, a cognitive or computational panopticon.

Dr. Nicholas Wright (11:40):

Yeah.

Dr. James Giordano (11:41):

How might we address that?

Dr. Nicholas Wright (11:43):

So again, there are different ways to think about it. So if we look, for example, at different parts of this group, so different target audiences within the Joint Force and all those support networks that enable the Joint Force to function. So those who have security clearances, higher level security clearances, they necessarily have agreed to a level of surveillance. So you are not allowed. We are not supposed to in a wide variety of different ways to run up vast gambling debts and not tell anybody about them and so on and so forth. So there is an element of surveillance that those people have agreed to undertake and it is not, I’m not saying anything secret here, when I say that there are methods that people are now employing to conduct more ongoing surveillance of those groups of people.

Dr. Nicholas Wright (12:48):

So the question then becomes, how do you do that in a way that is compatible with living in a free democracy? And so in some ways for those people, the challenge then you can think about is counter-espionage at scale. So you’re talking about, say around about 4 million people with security clearances. So this is a counter-espionage, or one way of thinking about it for those people is this is a counter-espionage at scale challenge in a digital world. And again, I could say you could read the report, but there are obviously specific things you can do. Like for example, testing small groups within those much larger populations also, and then using known adversarial or competitor contacts, a seed that they enable you to conduct further examination of networks within the Joint Force and so on. Now, if you are then talking about the family members of the Joint Force, to go to the other extreme, that is obviously a very different type of target population, just to say we should assume that those populations will be targeted.

Dr. Nicholas Wright (14:01):

And so there, you are thinking much more about how can we help them to protect themselves often. So it’s about things like enabling, making sure that they understand that these risks occur, giving them the tools often to protect themselves and certainly short of war time that is going to be the best we can do with that part of the population. And then between these two extremes between security clearances, people of particularly high level security people on the one hand, and the family on the other hand, you then have a range of different groups within the Joint Force and supporting the Joint Force.

Dr. James Giordano (14:43):

You’ve thrown out the phrase “short of war time”, but the reality of it is that we are in a state of global war time. It may not be quote a world war per se, but the issue here is that there’s explicit actions for example, Russia in the Ukraine, there are those things far more tacit, clandestine, perhaps even covert. Just today’s newspaper headlines are warning of the possibility of some cyber influence from our transatlantic peer competitor, Russia, within the United States and within the EU. And although I think that’s the sort of low hanging fruit, you’ve done a wonderful job in describing a China-US escalation scenario during a crisis that brings out a lot of the issues that you’re bringing to the fore. So let’s refocus the lens. I think that right now, probably most conspicuous is our trans-Atlantic peer competitor, but let’s look in the other direction. Clearly the global power is balancing a little bit differently now in light of what’s happening in the Ukraine, let’s talk about China’s role. Let’s talk about that escalation scenario that you’ve…

Dr. Nicholas Wright (15:44):

Exactly. I mean, think this is one of the other things that we talked a little bit earlier about TikTok and so on, but now exactly let’s talk about how would one potentially use some of the new technologies, particularly AI power, digital technologies in a China-US escalation scenario. Someone can paint a variety of different sort of escalation scenarios. For example, Taiwan, catalyzed by Taiwan or catalyzed by the South China sea the [inaudible 00:16:13] islands and so on. But let’s just take a relatively, put that to one side for a moment and just think about, there is now a China-US escalation scenario, which has either gone kinetic or is about to go kinetic. So we’re really at the, a lot of pieces are moving. A lot of things in the Joint Force are in movement. So what would happen? So the first thing to say is that we know already. So what are the technologies? So we already know that there is mass personalization of influence operations in retail. So things like Amazon spend billions of pounds, billions of dollars, understanding specific target audiences, right down to the level of the individual and personalizing recommendations to individuals. The Chinese also do this, and they’re very, very good at it. They have corporations worth hundreds of billions of dollars who are very, very good at mass personalization, for example, within retail.

Dr. Nicholas Wright (17:22):

So a question would be why would mass personalization not also be applied to information operations, okay? So if you then think about the personal data on members of the Joint Force, right. Things like, for example, a destroyer in the Pacific and personal data on the people who are on that destroyer, but also personal data about their families and a variety of different support networks for that destroyer. So you’ve got things like medical data, TikTok use, financial data, romantic dating sites. Yet, you have a lot of data about the individuals on that ship and in addition to that, about their families and friends and so on. There are a huge number of ways that you can then use that data to, in a personalized way. Affect the people on that ship. So to give you an example, you could inject information about adultery, financial impropriety, often things people are just embarrassed about. Things about eating disorders, all sorts of…

Dr. Nicholas Wright (18:41):

There are so many different members of families. There are so many different things that are potentially leveragable within that kind of scenario and that, and in addition to things that really happen, so you can then inject that into their social networks, the people on the ship. Can you tell their family, basically put it into the feed of people who are the friends of their families, for example. All of this is eminently doable and in addition to actual real data, you can also obviously put fake data in. So for every one real adulterous relationship that you expose of the captain of a destroyer in the Pacific, that then affects his family, right. You can also add in fake data as we saw with more exquisite attacks that the Russians for example were using over the past sort of decade or so with higher profile people in the US security apparatus. So I think we can think about it as mass personalization, the type of things that the Russians were doing at a very high level can now be mass personalized across the Joint Force.

Dr. James Giordano (19:57):

Now let’s work on that. Mass personalized data. And again, it goes back to the initial premise we’re dealing with ubiquity of information, data, data systems, how those are uptaken and used. I mean, each individual for example, is a multidimensional data portal, both in terms of data intake and data output. So I guess the question then is what can we do about it? I mean, you wrote about the 3D paradigm: detect, defend and democratic compatibility. Let’s dive into each of these in brief. What do we mean by that?

Dr. Nicholas Wright (20:34):

So I think the first thing, I mean, in terms of detect, the US must build the capabilities to detect and characterize adversary influence operations against the Joint Force. So before you can really think about defending, although that’s obviously critical, you’ve got to be able to detect a lot of these campaigns. And if you’re thinking about that now, in some ways you can think what would adversaries like us to do, right? What would adversaries like the US to do? They’d probably like the US to try and build, try and do essentially what a lot of the big tech companies wanted to do, which was to build AI heavy, use a lot of technology, a lot of AI heavy responses that also have a lot of difficulty going, looking at influence campaigns that cross over from things that are happening outside the US, into the domestic US information set of networks. So things that go from side to inside the US, that’s probably what you’d want if you were an adversary. And so I think what the US needs to do is build counterintelligence at scale, to detect as the detect part, to build counterintelligence scale. And that’s going to require integrated human AI and organizational capabilities. And it’s going to have to think about detecting adversary information operations at scales of relevance, going all the way from a few individuals all the way through to millions of individuals.

Dr. Nicholas Wright (22:20):

So I think that’s really, you know, that’s the first thing to say. And the second thing to say is that part of this is about detecting how adversaries will also shape the terrain over years. And I’ve said this, and it becomes very unfashionable, but I think it is absolutely bonkers that TikTok is allowed to operate so freely in the US and its allies.

Dr. James Giordano (22:49):

What defense?

Dr. Nicholas Wright (22:50):

So in terms of defense, I think a lot of this is about making the individuals tougher targets and making at now. But a lot of where the problem then comes down is that’s not saying it’s all up to the individual, because it’s not. A lot of the things that will need to be done to defend individuals can only be done organizational scales, and can only also be done when you think over the course of years. So let’s think about defending an individual at the sort of human, the individual human scale. So one thing is to make the Joint Force less vulnerable, to enhance social family and mental support. For example, with particular focus on predictable periods of vulnerability, like moving between postings and so on. Now, this isn’t very fashionable, but it is important. Another thing would be to give individuals a technological means to defend themselves online. Low cost, practical options exist, things like NewsGuard.

Dr. Nicholas Wright (23:52):

Other people have tried to produce them. I myself tried to produce one in some work that I was doing with other academics and originally sort of with some interest from DARPA, but it’s hard, there is no money going into helping individuals to defend themselves.

Dr. Nicholas Wright (24:14):

So then you need to think about defending individuals at the organizational scale. And one of the things is, for example, do we really think that the medical data of the service person on the Joint Force is well protected? Do we think that a wide variety of other things about data of individuals on the Joint Force are well protected? I would say probably not. And in addition to that, against what is now the conventional wisdom, I think we need to build silos for data. One of the great problems with the Office of Personnel Management hack was that there a giant data leak with too much information in it. That information should have been siloed. And one of the advantages, like the reason why the CIA wasn’t so badly affected is because their data was held separately. So I think data silos separating data is critical. That’s something that happens on the organizational scale. Then the last thing I’d say again on the defense side is we need to defend against the shaping of the information terrain over years. And part of that’s building in defensive advantages for the US and its allies in the information terrain, and preventing competitors developing strategic advantage via platforms like TikTok.

Dr. James Giordano (25:24):

But there’s a balance to be struck here for sure. I mean in open society, such as we have in the United States and those societies of many of the United States’ international military and economic allies, personal freedoms are valued. And so I guess the question here is these ideas of detection, which involves surveillance, defense, which in some cases can join certain restrictions and constraints on what might see as their freedoms and or the insertion of some oversight, governmental oversight, institutional oversight. How do you make this compatible with democratic institutions?

Dr. Nicholas Wright (26:00):

That’s a critical question, because again, you say what is it that an adversary would definitely want us in the west to do? One of the things they would like us to do is to massively overreact, build our own surveillance state. And then they could point to us and say, “oh, isn’t this a disaster that those guys are no different to us.” And in addition to that will damage our democratic systems and our freedoms. And it will be basically, overreacting is a very bad thing to do. So we need to think very carefully about how we build these systems in ways that are democratically compatible. And so let me just give you four ways that can be done.

Dr. Nicholas Wright (26:43):

So one thing is that we need to maintain the scene between domestic and foreign, right? Which is critical. We can’t just say that, for example, the NSA needs to have different rules for looking at people in the United States, US citizens of the United States, versus looking at people who are, for example, in China or Russia. The rules need to be different. But on the other hand, we also need to think about how we bridge the seam between domestic and foreign, because that is a seam that we know adversaries will try and exploit. So the first thing would be to maintain the seam between domestic and foreign and manage the vulnerabilities, the inevitable vulnerabilities that that brings. A second thing would be that some detection and defensive capabilities carry fewer risks to democracy, such as enhancing social and mental health support than others, such as building offensive information capabilities that can then be turned inwards within the United States. So secondly, focus on those, greater emphasis should be placed on safer options.

Dr. Nicholas Wright (27:47):

Third is to build robust ethics into the cultures and processes of the organizations and individuals who are charged with detecting and defending against adversary information operations, okay. There are ways of doing that. And fourth, I’d say that to ensure existing frameworks for democratic oversight are fit for current and near future technologies, such as digital counterintelligence scale. And this can be done: to give you an example, GCHQ, which is the UK equivalent of the NSA is in some ways a bit more aggressive than the NSA, even in some of the things that it does. And yet we had, with SMA at the Pentagon, we had a former director of GCHQ who has worked long and hard on the ethics of digital espionage and counter-espionage and has produced a series of practical guidelines that can help guide these types of organizations.

Dr. James Giordano (28:48):

I think the practical applications are needing to be not only put in place, but evaluated as we go. But clearly what you’re describing is a work in progress. And I think what becomes important for our listeners to understand is that progress reports on that work in progress need on relatively real time. And it needs to be responsive and flexible and in some cases, fungible, so as to be able to remain on the cutting edge of not only capability, but of sensibility, or an evermore capable set of international peer competitors. Nick, it’s always a pleasure. You’ll have to come back and talk to us some more about this in the future. We’ll welcome you back.

Dr. Nicholas Wright (29:25):

Thanks very much, Jim. It’s been a pleasure.

Dr. James Giordano (29:28):

Thank you.

Speaker 3 (29:38):

Special thanks to our guests this week and to you, our listening audience, subscribe to your favorite podcast channel to join us next time for another episode of Clear and Present.