AI could break legal systems by finding previously hidden loopholes

If AI can find the holes in programming code, it can do the same with legal code, paving the way for chaos.

Felix Ringe, a visiting fellow at Stanford Law School’s liftlab and legal scholar, suggests large language models (LLMs) have become potent enough that they could break legal systems by identifying exploitable inconsistencies at scale.

“Legal systems encode the foundations of social order in natural language,” Ringe and collaborator Linus Folkerts argued in a blog post last week. “Similar to computer systems, the written law can contain vulnerabilities that invite exploitation.”

Ringe contends that while courts tend to pave over these text vulnerabilities through statutory interpretation, the ability to discover adversarially useful inconsistencies in the law in an automated way risks overwhelming the legal system’s ability to bridge the gaps.

To buttress his thesis, he points to recent work by other legal scholars that frames laws as vulnerable code, such as “Legal Zero-Days: A Novel Risk Vector for Advanced AI Systems” and “Hardware and Software Bugs in the Legal System” [PDF].

The analogy dates back at least to 1999, with Lawrence Lessig’s Code and Other Laws of Cyberspace [PDF] and Code Is Law: On Liberty in Cyberspace. Lessig credits William J. Mitchell, in City of Bits: Space, Place, and the Infobahn (1995), with the idea that code is cyberspace’s law, adding that he “drew the metaphor from others as well.”

Ringe has turned this on its head by proposing that law is code.

The analogy between programming code and legal code works reasonably well, up to a point. Law and code both have their share of formatting pedantry and ritual. The mapping breaks down when courts get cast as a runtime environment – non-deterministic trial outcomes don’t work for production applications.

Discontinuities aside, Ringe’s point is that AI models can surface statutory gaps at a scale that may bring the chaos currently seen in the software security ecosystem to the legal world.

To demonstrate that, Ringe and Folkerts cite several potential gaps in German law that they uncovered with the help of LLMs, noting that they’re only sharing legal exploits they believe could not cause serious harm.

For example, an amendment to Germany’s Occupational Diseases Regulation (BKV) is subject to a constitutional requirement to cite its statutory authority. But Ringe and Folkerts argue that the amendment omits a required reference. So, in theory, a journalist could request an unpublished statement submitted to Germany’s occupational-diseases advisory council under freedom of information rules, challenging a confidentiality provision that the researchers argue may be invalid.

Another example cited describes a transit visa rule that may be invalid because the government failed to identify the authority under which the rule was made.

“We believe that with cheap legal reasoning proliferating, AI systems could jeopardize the functioning of today’s legal systems,” they wrote. “In such a world, lawmakers would need to significantly harden the process of legislation, for example by leveraging advanced AI themselves.”

Wendy Musell, a San Francisco-based employment law attorney and partner, told The Register that the use of AI in the legal field is expanding rapidly.

“Some California Courts, including LA Superior Court, are testing the use of AI as law clerks,” said Musell. “Generally law clerks would assist the Court by researching and drafting opinions for the judge’s review. That means AI is serving that role in certain courts. Litigators now must consider how AI is analyzing motions and cases.

“This is the case even where I personally have found different AI sources to appear to have a slant or bias, and often fail to accurately analyze legal issues. It was just a matter of time before AI came for the legislative process.”

Eric Goldman, associate dean for research and professor of law at Santa Clara University School of Law, told The Register that Ringe’s argument has some merit.

“I think the methodology of using AI to identify potential statutory gaps is logical,” said Goldman. “However, it will require careful human review to determine if the AI-identified gaps are in fact actually gaps. Statutory gaps also may have no legal consequence, or at least none of significance; and legislatures will often correct any major gaps once they are identified.”

Goldman said the idea that legislators should use AI during the drafting process is ambitious, at least in the short term.

“Legislators often draft laws with a range of goals and priorities, and avoiding statutory gaps is rarely a high priority,” he explained. “But many legislatures have legislative counsel who are responsible for converting draft bills into final legislative text, and I could see where legislative counsel could benefit from using AI to identify drafting gaps as part of their standard cleanup protocols.” ®