Skip to content
Artificial Intelligence

An OpenAI Model ‘Disproved’ a Famous Math Conjecture. This Mathematician Couldn’t Leave It Alone

Mathematician Will Sawin discusses his experience reviewing and refining a mathematical proof devised by OpenAI's internal model—and what that could mean for mathematics.
By

Reading time 10 minutes

Comments (0)

Will Sawin got OpenAI’s email on a Friday night. Or Saturday morning. Either way, Sawin, a professional mathematician, spent his entire weekend thinking about that email. By next Monday, he decided to write up a paper that essentially improved what was given to him—an AI’s “proof” of Paul Erdős’s unit-distance problem, an infamous conjecture from 1946.

Last week, OpenAI published a blog post on the AI’s proof. The paper came with a companion piece containing comments from nine renowned mathematicians uninvolved with OpenAI, including Sawin. Many prominent mathematicians praised the work, with Fields medalist Tim Gowers calling it a “milestone in AI mathematics.” This result is just one of dozens of AI-derived solutions to long-time mathematical riddles. All this has us asking: Could AI usher in a new era of mathematical advancements?

The answer, if one even exists, is likely a nuanced one. There are certainly computational advantages that AI brings to the equation (no pun intended). But what does this really mean? Does that represent some tangible revolution, or is it a “misconception” stemming from AI’s data-driven imitation of human intelligence, to quote Pope Leo’s recent encyclical?

We’ll for sure continue to see more AI solutions pop up—after all, impossible math conjectures come by the hundreds—and each time, human mathematicians will be summoned to check the computer’s work. To OpenAI’s credit, its blog post closes with this pleasant sentiment: “People choose the problems that matter, interpret the results, and decide what questions to pursue next.” So, Gizmodo reached out to Will Sawin, who appears to have done just that: interpret the results and pursue a relevant question. We wanted to know what the experience was like.

Sawin is a Fernholz Professor of mathematics at Princeton University. He began his academic career at Yale when he was 10 years old and has since worked in number theory, algebraic geometry, and combinatorics. During the conversation, Gizmodo asked Sawin about his own experience reviewing AI-derived mathematical proofs, the reality of using AI in mathematics—and, most importantly, what it is and is not doing.

The following conversation has been edited for grammar and clarity.

Gayoung Lee, Gizmodo: Let’s start with the headline of OpenAI’s blog post: “an OpenAI model has disproved a central conjecture in discrete geometry.” What’s the conjecture we’re talking about here?

Will Sawin: We start with the problem in simple form: if you have a set of points in the plane, how many pairs of points can have a distance exactly 1 [unit distance] from each other? You can play around with different constructions. If you try it by hand, the best you’re going to find is some kind of grid, like a triangular grid of points. Each point will be the next unit distance from 6 other points. That’s pretty good for small numbers of points.

The mathematical question is for each n number of points, “What’s the greatest number of pairs of unit distances you can get with that many points?” For Erdős, the problem was, how does this grow as a function of n? And the conjecture he made was that it can grow slower than every power of n > 1. So, slower than n1+1/2, and slower than n1+1/4, and so on.

This is a purely asymptotic question, so not about any particular value of n. I think one thing that people were disappointed by is that OpenAI’s paper, our paper explaining the proof, and my paper on an optimized version—none of these papers had an example of the construction for a particular value of n. That’s because of the asymptotic nature of the problem.

When Erdős first brought up the problem 80 years ago, he was not sure it should be true. But he seems to have gotten a little more confident in this over time, as nobody figured out a way to make it grow faster than that. There are some conjectures in mathematics that, if you disproved them, it would be a really big shock to the mathematical community. It wasn’t such a huge shock that this statement wasn’t true, but it was the opposite of what people generally believed.

Gizmodo: You mentioned that Erdős grew more confident in his question. To ask a more metaphysical question—what does it mean to prove or disprove a conjecture in mathematics? What does it mean exactly for a problem to be “solved”?

Sawin: To answer your first question, a proof or a disproof in mathematics is an argument that is completely convincing, leaving no room for doubt. What mathematicians have considered to be a proof has changed over time. Hundreds of years ago, you might see people use some kind of physical reasoning and consider that to be about the real world and consider that to be a mathematical proof.

Now people have different standards. One standard is a formal proof. There exist formal proof systems with logical rules for what statement you’re allowed to introduce from another statement. One common belief is that a proof really should be a formal proof. If you have an informal proof using English words, it’s only a valid proof if it’s an explanation of why there exists a formal proof. Other people disagree and say there’s something about informal proofs that is not completely captured by formal proofs.

Paul Erdos Terrence Tao
Paul Erdős and Terence Tao in 1985. Credit: Billy and Grace Tao via Wikimedia Commons

Let me not take a position on that philosophical question right now (laughs). But the [OpenAI] proof in question is an informal proof that you could eventually turn into a formal proof if you wanted. And this is usually what mathematicians mean when they say something’s been “proved.”

Gizmodo: So OpenAI’s proof is an informal proof?

Sawin: It’s an informal proof that looks very similar to the informal proofs that mathematicians produce. There are some things about the way it’s organized that, if you’ve seen a lot of mathematical proofs, you can tell aren’t exactly the same as how a mathematician would write them. But I would say someone who has not read a lot of mathematical proofs might not be able to tell the difference.

Gizmodo: Could you unpack for me the content of this informal proof? How did the AI arrive at its conclusions, from what you can tell?

Sawin: If you were trying to prove what Erdős said—an upper bound for the number of unit distances—you’d have to reason about any possible collection of points on the plane to make some argument that’s valid for any set of points on the plane. Which would be hard. So the disproof, in some sense, is easier, because you have to come up with a specific sequence of points in the plane. So it’s a question about the limit as the size goes to infinity, and you need to construct a sequence and show it has a lot of unit distances.

The way the AI did this was to use algebraic number theory to use a ring of integers in an algebraic number field. When I describe this idea at that level of generality, it’s not so foreign to what mathematicians had already tried. I would say the key thing that AI realized that humans didn’t is not just that you can use algebraic number fields but that you can use algebraic number fields of growing degrees. You let the degree of the field grow, which basically increases the kind of complexity of the numbers you’re working with. And that makes the number of unit distances grow very rapidly and more rapidly than Erdős expected.

Gizmodo: I’m admittedly somewhat skeptical about news that AI did this impossible thing in math, since my thought tends to go to how AI is just a really great computing device. But here, it sounds like OpenAI’s model looked at what human mathematicians were doing and kind of… made a logical decision as opposed to pure computation. Is that something we’ve seen before?

Sawin: It depends on what you mean by “before.” In terms of the cleverness of the mathematical reasoning, there’s not a big gap between this and the most impressive previous cases of OpenAI and mathematics that I have seen. Erdős had a huge number of problems, and Thomas Bloom collected over a thousand on a website. And he has not collected all the problems that Erdős asked; there are even more than that.

Over the last few months, a lot of people have been trying to use AI to generate solutions to Erdős problems. Some solutions aren’t incredibly impressive. Some of them are, like, the AI discovers there’s a paper where somebody solved the problem already, but they just didn’t know about the problem. So they didn’t know, but the [AI realizes] their paper just immediately solves the problem.

But in some cases, AI introduces some new ideas that humans didn’t use. Sometimes, the idea isn’t very interesting. Other times, the idea is interesting and leads humans to wonder what they could do with this idea. That is not too dissimilar to what happened here [with OpenAI]. I think this was a more technically intricate idea than previous problems that I’ve seen. It definitely was a bigger problem that more people knew about and more people had worked on.

I can tell you reasons for skepticism. So certainly, this is an idea that, as far as we can tell, humans did not come up with. This is not an idea that humans couldn’t have come up with. I can see why it was hard for somebody to come up with that idea, but people come up with ideas that are hard for people to come up with all the time. It’s definitely not a situation where AI is doing something that humans can’t do.

We don’t know the amount of computing power that was used, the amount of problems that OpenAI tried it on, or how the AI system is set up. I mean, OpenAI did say some things about this, but certainly not at the level of detail that they used to have when AI was not such a big deal. So exactly how hard it is for the AI to do this is something we don’t fully know.

Gizmodo: On that note, let’s talk about your own preprint. This “refined” the AI’s proof. What does that mean?

Sawin: Yeah. So the conjecture is that this function grows slower than any power of n > 1. The OpenAI proof said that it actually grows faster than a power of n > 1. But it doesn’t tell you which power of n > 1, just “some” power of n > 1. I wanted to try getting an explicit value that’s reasonably good, that’s not a really tiny distance away from 1. The value I ended up getting is a little more than 1.01, so not a big difference from 1. Other people have improved it, and it’s now more like 1.03.

And as I was reading the argument of the AI and trying to understand each step, I was thinking, ‘How would I do each step in an efficient way?’ A lot of things about the AI’s argument were inefficient in some way, which I think is not surprising. Like, a human wouldn’t have written it that way. But the AI was clearly not trying to be efficient. There was definitely nothing wrong with the original argument. It achieved this goal. I had a somewhat different goal. But basically, almost every step of the argument had to be changed in some way to support this goal of getting a reasonable, explicit constant.

Gizmodo: And OpenAI was fine that you did this?

Sawin: Oh yeah, I told them. And they were fine with it. Their announcement mentioned my paper. No concerns.

Gizmodo: Has this experience influenced at all how you view AI’s impact on your work, or the perception of your work, as a mathematician?

Sawin: There are two things that these AIs are very effective at now. One is searching the literature. If you want to know if a certain theorem exists, you’re much more likely to get the result by asking an LLM than you are by typing it into a mathematics-specific search engine. The other thing it’s good at is reading and proofreading a paper—not just typos or grammatical errors but also mathematical errors that could affect the result.

I mean, I do still read and proofread my paper as a human, because, you know, that’s what’s appropriate. And I think most other mathematicians don’t want to let it write their papers for them, either, for reasons of personal credibility and accountability, but also because one still shouldn’t fully trust the AI. They’ve gotten a lot better, but there are still blind spots. If you’re using an AI-generated idea, you should still express your understanding of it as a human.

As for generating ideas, I haven’t found AI very effective at that. I think that’s partly because it varies from one field of mathematics to another. It’s easier for AI to generate new ideas in combinatorics than in some areas where there’s more technical background and fewer prior works to use as a point of comparison.

Gizmodo: And as you mentioned, sometimes it’s because an AI having an idea doesn’t necessarily mean that it’s an idea that humans would never have thought about.

Sawin: Yeah. Like, if I were to ask AI for an idea, I wouldn’t be asking so I could throw out the ones I already came up with for a question.

Gizmodo: So to you, a mathematician, is AI a partner or a search engine tool?

Sawin: Currently, I’d say tool. The people that are having the most success are using AI as a tool. It’s completely conceivable that soon somebody will prompt an AI to look at the recent literature, come up with its own problems, and solve those problems. And it’ll come up with interesting stuff. But that hasn’t happened yet, as far as I’m aware. I don’t want to try to make any predictions about the future, but that’s what’s true currently.

Gizmodo: Okay. So I’d like to ask you for some advice. When we see another AI-written, fantastical proof, what do you think is the most important question we should ask ourselves?

Sawin: I’d say that it is probably less exciting than it sounds (laughs). But it’s probably still somewhat exciting. If somebody’s like, “Oh, AI solved this impossible math problem,” it’s definitely not that. It’s not that AI solved an impossible math problem, but it’s not nothing. It’s somewhere in between.

I’ve definitely seen people whose first reaction is to assume that if AI solved an impossible math problem, like, human mathematics is over. And I’ve definitely seen people whose reaction is to assume that it’s nothing, that AI really didn’t do anything.

It’s definitely somewhere in between those two. And I’m not sure exactly where it is, but it’s somewhere in between those two.

Explore more on these topics

Share this story

Sign up for our newsletters

Subscribe and interact with our community, get up to date with our customised Newsletters and much more.