The President sits in the Situation Room with members of her Cabinet. They have assembled to make a final decision about whether to initiate a drone strike against someone they believe to be a senior member of al Qaeda. The Secretary of Defense briefs the others in the room about the information that the Department of Defense (DOD) has compiled. Using advanced facial recognition software, he notes, the Department is 90% certain that the individual in question is al Qaeda’s third most senior operative, even though the enhanced image he puts on the screen is so hazy that the Cabinet members can’t identify him when shown the still frame.  Further, the Secretary tells the President, a different Defense Department machine learning algorithm that was trained on huge volumes of telephonic data indicates that the operative is very likely to lead an attack against the U.S. embassy in Algiers in the coming week. A third algorithm has predicted that, based on his pattern of life, there is an 87% chance he’ll be alone in a particular house tomorrow. The CIA Director chimes in: “If we kill him, the U.S. intelligence community believes—based on our algorithms—that there is a 62% probability that it will take al Qaeda at least six months to replace him.”

“How do you know all of this?” the President asks. “Our algorithms made these predictions,” the Secretary replies, “and we trust them.” What the Secretary knows is that DOD’s facial recognition software can identify subjects at a higher rate of accuracy than humans can. He also knows that other machine learning algorithms used by his Department and the various agencies in the intelligence community are making predictions about individuals’ identities, locations, and behaviors based on connections that humans would take years to find, using millions of data points on which the engineers have trained the systems. The computer’s predictions rely on the quality of information from sensors, electronic surveillance, and models—things that the President and his senior advisers cannot see and may not fully understand. Yet the President may authorize the Defense Department to kill this alleged senior al Qaeda member based on these algorithmic recommendations, especially if the situation requires an immediate decision. And the Defense Department might decide to use an autonomous drone swarm that, once launched, requires no further human involvement in executing the operation. Because these types of operations are classified, we—the American people—likely would not know that the Defense Department has these types of machine learning algorithms at its disposal, or that the President has ordered the killing on the basis of predictions made by algorithms, or that the Defense Department is using autonomous drones to perform the mission.

The computer’s predictions rely on the quality of information from sensors, electronic surveillance, and models—things that the President and his senior advisers cannot see and may not fully understand. Yet the President may authorize the Defense Department to kill this alleged senior al Qaeda member based on these algorithmic recommendations, especially if the situation requires an immediate decision.

(. . .)

The use of AI/ML [artificial intelligence/machine learning] algorithms in the administrative state is complicated and raises a host of important issues that the Executive, Congress, and the federal courts must work hard to resolve in the coming decades. This book, however, is about an even thornier challenge: the use of AI/ML algorithms to protect U.S. national security. U.S. national security and homeland security agencies are using more of these algorithms every day, raising fundamental questions about how the U.S. public can ensure that its government only adopts and uses these tools in a way that reflects the values and virtues of a democracy. These questions are hard not simply because it is difficult to decide when, whether, and how to use AI, but also because so much national security activity takes place behind the veil of classification and because the geopolitical stakes of national security decisions are very high.

Many of the leading voices on AI in the United States are urging the United States to move forward with alacrity. For instance, the National Security Commission on AI (NSCAI), set up by Congress, argues that the United States must do what it takes to retain its technological advantages in the AI competition with China and to responsibly use AI to defend democracies. In the NSCAI’s words, “AI is going to reorganize the world. America must lead the charge.” More recently, some of the NSCAI’s members issued a more sharply worded report that argued, “Absent targeted action, the United States is unlikely to close the growing technology gaps with China” and will fall behind in developing critical AI tools.

These questions are hard not simply because it is difficult to decide when, whether, and how to use AI, but also because so much national security activity takes place behind the veil of classification and because the geopolitical stakes of national security decisions are very high.

Notwithstanding this hard-charging approach to AI, the NSCAI recognizes that the competition between the United States and China is a “values competition,” and it urges the United States to “work with fellow democracies to . . . advance democratic norms to guide AI uses so that democracies can responsibly use AI tools for national security purposes.” Both of these approaches seem reasonable: the United States should continue apace to develop national security AI tools, but in doing so must make sure that it preserves its values. The hard question remains: How will we—the public—know whether it is doing so?

Let’s assume that actors within the U.S. Defense Department and the intelligence community fully intend to follow the seven or eight basic norms that most agree are requisites for responsible AI—norms such as reliability, governability, and accountability. How can we know if they are following through on those intentions? What if they confront a new, highly classified national security threat that causes them to revisit those basic norms? What legal and moral guidelines will they use to decide whether and how to use AI to handle the new threat? These types of questions are real, and they are pressing. The United States made some poor legal, policy, and operational decisions in secret after the September 11 attacks, even as those inside the government were acting with good intentions to protect the country. The rise of AI amid broader geopolitical tensions with China, Russia, and Iran triggers similar pressures to protect the United States, even at high costs.

Let’s assume that actors within the U.S. Defense Department and the intelligence community fully intend to follow the seven or eight basic norms that most agree are requisites for responsible AI—norms such as reliability, governability, and accountability. How can we know if they are following through on those intentions?

This book makes six arguments. First, governments are introducing AI/ML and autonomy into their national security operations today. This means we are at a key inflection point, both for setting expectations around the world for what responsible uses of AI look like and for setting granular rules and standards for the U.S. government. Second, there is rough consensus (especially in the United States, including inside the U.S. government) about what basic principles AI/ML should follow. Signs are favorable regarding the government’s commitment to the ethical and responsible use of AI (as reflected in recent DOD and intelligence community policies). But these policies are written at a high level of generality, and it matters a lot how these principles cash out in specific cases.

Third, and critically for this book, much of the use of national security AI/ML will happen in secret. This secrecy, coupled with the opaque nature of the AI/ML tools themselves, will significantly complicate democratic oversight, including in the United States. I term this the “double black box” problem. We have already seen the U.S. government adopt uses of AI that, when they come to light, turn out to be tools that the public rejects. Although U.S. officials generally are competent and well-intentioned, we should not leave the executive branch (or private sector actors that sell the government AI/ML tools) entirely to their own devices in deciding what types of AI systems to develop and deploy.

This secrecy, coupled with the opaque nature of the AI/ML tools themselves, will significantly complicate democratic oversight, including in the United States. I term this the “double black box” problem. We have already seen the U.S. government adopt uses of AI that, when they come to light, turn out to be tools that the public rejects.

Fourth, we will need to rely heavily on both the traditional set of actors that check and balance classified executive policymaking—Congress, the courts, executive branch lawyers, inspectors general, and whistleblowers—and on alternatives to these traditional surrogates to ensure that the U.S. government complies with the public law values of legality, competence, accountability, and justification. These actors will need to be creative about how to do so and will bear an antecedent burden of educating themselves about AI tools and their pathologies. Fifth, it is very unlikely that the full set of states pursuing advanced AI will agree on new, binding international rules to regulate those tools, but the United States is potentially well aligned with European and other allies to develop AI tools that are palatable to democratic populations. This group should band together to create appropriate norms and tools that comport with their own values, regardless of what AI/ML tools countries such as China and Russia adopt. Finally, there is no one silver bullet to address the double black box problem. We must pursue many avenues simultaneously.

(…)

This book’s focus on ensuring that the U.S. process of deciding to use, build, and deploy AI systems reflects public law values means that many of its proposed solutions focus on institutional design and process, rather than the contents or parameters of the algorithms themselves. Process and substance are deeply intertwined, though: if we can ensure that the Executive complies with our public law values, it is more likely to produce substantively reliable and legally defensible algorithms. Unless we find a set of satisfactory answers to the questions raised earlier—how do we hold the executive branch accountable for its use of AI in national security settings and how can we be sure that it is adopting only safe, transparent, and reliable AI?—we will end up with a national security state that is less responsive to its citizens, harder to hold to account for serious errors, operated by an ever-shrinking set of players, and setting precedents that we may not want other countries to follow.

We will need to rely heavily on both the traditional set of actors that check and balance classified executive policymaking—Congress, the courts, executive branch lawyers, inspectors general, and whistleblowers—and on alternatives to these traditional surrogates to ensure that the U.S. government complies with the public law values of legality, competence, accountability, and justification. 

A key theme of this book is that ensuring executive compliance with public law values will require diligence by a wide range of actors. We will need to rely on our traditional surrogates, especially the congressional committees that oversee the military, intelligence community, and diplomats. But we will also need to rely on our nontraditional secrecy surrogates, including foreign allies and U.S. technology companies. Front-end input will be important. By virtue of how machine learning algorithms work, it will be critical to ensure early policy and legal input into whether to use a machine learning algorithm at all in a given context and what kinds of data and parameters that algorithm should use (or avoid). This will be particularly challenging when the government purchases AI tools (including large language models) from the private sector with the goal of adapting them to national security work. Drawing from Lawrence Lessig’s seminal early insight that “code is law,” there is important work for lawyers to do to introduce legal guardrails into the code that helps produce national security outcome.

Excerpted from The Double Black Box published by Oxford University Press ©2025