People’s trust in AI is taking a dangerous new turn. Researchers are saying that AI-generated photos and summaries are successfully distorting people’s own memories.
MIT scientist Pattie Maes spoke to the Washington Post about her team’s research. Participants viewed a series of photographs, and after a break, some of the participants were shown the exact same photographs while others were shown versions altered by AI. When facing a memory test later, the people who viewed the AI-altered images were not only more likely to misremember the original image, but they were also overly confident in their false recollections.
Human memory is fickle, unreliable and highly open to suggestion. This has been covered extensively in psychological experiments dating back as far as the 19th century. That’s what makes misinformation particularly dangerous, with some studies showing that it can continue to influence decisionmaking, including political decisions, even after corrections have been issued.
The introduction of artificial intelligence only makes that dynamic more precarious. AI-generated slop is only becoming more convincing and very pervasive online, and it has been enabling and turbocharging a new wave of fraud and disinformation.
Numerous studies besides Maes’ have shown that these AI-generated pieces of misinformation have been successful at planting false memories in participants’ brains, changing what the participants know to be true. A study from last year found that both deepfakes and AI-generated texts effectively planted false memories about completely made-up political scandals, going so far as to reduce voting intention.
Even when AI is not used maliciously, it can have a potentially catastrophic influence on truth. That’s because generative AI is prone to making mistakes in its output, called hallucinations. As government agencies, banks, militaries, hospitals and an ever-expanding list of more organizations race to deploy AI, you are likely to hear a professed importance on “humans in the loop,” aka a human, often a senior personnel with expertise on the issue, auditing the output of the AI system. The thinking is that widespread AI deployment with human oversight will speed up any process, from military strikes to record-keeping, while ensuring that any AI-generated errors will be caught before they can translate into action and cause real-life harms.
But research shows that users are not particularly good at spotting these hallucinations. And with our spotty memories and tendency to believe AI systems, even when something an AI system puts out contradicts a piece of information the human expert in the loop knows to be true, the expert is vulnerable to having their judgement distorted.
There may be ways to go about solving this problem. Currently, an average person’s ability to correctly identify AI-generated content is basically a coin toss, studies claim, but numerous researchers are working to find interventions and training methods to help humans get better at spotting deepfakes.
But there is no foolproof method yet on fixing this faulty tendency. While experts are struggling to find one, the AI industry is racing ahead to cut humans out of the loop with agentic AI systems they hope to make completely autonomous.
This discrepancy has already led to catastrophic outcomes. An American military strike on an elementary school in Iran killed more than 150 people, including at least 123 children, earlier this year. An internal Department of Defense investigation, reported by Bloomberg last month, concluded that the strike was partially due to an overreliance on a Palantir-developed AI system, which recommended the elementary school as a day-one strike target due to outdated data.
