When the Climate TRACE database came online in 2020, Time Magazine named it one of the 100 best inventions of the year. Backed by leading universities, environmental non-profits, and former Vice President Al Gore, this independent organization helped put AI-powered carbon monitoring on the map. Now, researchers claim there’s a major error in its data.
Research published Tuesday in the journal Environmental Research Letters compared vehicle CO2 emissions data from Climate TRACE with estimates from the Vulcan Project, a government-funded research effort that maps carbon emissions across North America in fine detail. The study, led by Northern Arizona University professor Kevin Gurney—who also leads the lab behind Vulcan—found that Climate TRACE underestimates vehicle CO2 emissions in U.S. cities by an average of 70%.
“While the Vulcan on-road data is not perfect, with uncertainty of about 14%, this is far lower than the differences found when we compared 260 city vehicle CO2 emissions in the U.S. to the Climate TRACE database,” co-author Bilal Aslam, a postdoctoral researcher in Gurney’s lab, said in an NAU release.
In an emailed statement, Climate TRACE co-founder Gavin McCormick told Gizmodo that the organization has compared its data to official city datasets around the world and has not found results consistent with the study’s claims.
A contentious data discrepancy
Climate TRACE is built on one of the first carbon emissions analytics tools to incorporate machine learning. Its algorithms are trained on satellite imagery and direct measurements of greenhouse gases to produce a global inventory of emissions. When the organization officially released its database, it claimed it was the world’s first comprehensive accounting of global carbon emissions based primarily on direct, independent observations.
The Vulcan Project takes a different approach. Funded by multiple U.S. agencies and spawned under the North American Carbon Program, this monitoring effort uses government datasets from agencies including the EPA, the Department of Energy, and the Federal Highway Administration, combined with land-use and infrastructure data, to estimate CO2 emissions at very high spatial resolution.
Comparing these two databases isn’t exactly apples to apples, but according to the study, Vulcan’s accuracy makes it a useful benchmark for evaluating other databases, like Climate TRACE. Vulcan’s on-road emissions uncertainty has been independently estimated at ±14.2% for all road types—which is relatively small. The study also states that Vulcan’s estimate of total U.S. emissions is within about 1.4% of an independent estimate based on atmospheric measurements of carbon radioisotopes, which scientists use to infer CO2 emissions.
When Gurney’s team compared the two on urban vehicle emissions, it found an average relative difference of -70.4%, suggesting Climate TRACE may be drastically underestimating these emissions.
The organization refutes that claim. “If that were true across the board, our calculation of total U.S. road emissions would be significantly out of line with existing inventories,” McCormick said. “In reality, our data aligns closely with the official U.S. inventory. Our U.S. road transport total in 2021 was 1.5 billion tonnes of CO2 equivalent and the official U.S. inventory as submitted to the UNFCCC was 1.45 billion tonnes [CO2 equivalent].
McCormick added that Climate TRACE passes the same accuracy tests that Vulcan does, claiming that its estimate for total U.S. CO2 emissions is also within 1.4% of the same radioisotope-based estimate.
AI analytics falling short?
While McCormick disagrees with the study’s findings, he said Climate TRACE will examine the data carefully. “If there are improvements to our dataset that we can make based on this analysis, we’ll be delighted to incorporate them,” he said.
Gurney and his colleagues believe that biases in Climate TRACE’s machine learning model, fuel economy values, and fleet distribution values may be behind the discrepancy they identified. Based on their findings, they urge policymakers and climate scientists to use them on-road CO2 emissions estimates “with caution.”
“Individual cities such as Indianapolis and Nashville were lower by more than 90%,” co-author Pawlok Dass, another postdoctoral researcher in Gurney’s lab, said in the NAU release. While these local underestimations are alarming, he and his colleagues suspect the problem is present at varying levels throughout the global database.
The study raises questions about the reliability of AI-based emissions analytics. Such tools have emerged as a faster, cheaper way to deliver precise data, but Gurney and his colleagues emphasize that scientific rigor, transparency, and expert review remain essential to ensuring accuracy. As these tools begin to inform real-world climate decisions, understanding their potential limitations will be critical.
