How cognitive security can protect judgment in AI use
Dr. Kimberly Ferguson-Walter's research reveals why experience and confidence with AI tools do not always prevent errors.
Why it matters
- AI-generated summaries can introduce distortions that influence how people interpret source information
- Intelligence analysis depends on independent judgment, making these risks especially consequential
- Research suggests targeted cues can help users recognize errors before incorporating them into their work
AI can turn a stack of documents into a summary in seconds.
For the person reading it though, a harder task remains: deciding whether that summary preserves what the sources actually say.
A change in meaning that goes unnoticed can become part of the reader’s own analysis and confidence in the answer may make that change harder to question.
Dr. Kimberly Ferguson-Walter, director of research for cyber technologies for Leidos Intelligence, leads a cognitive security team studying this interaction between AI-generated information and human judgment. The team's work examines how people can retain their ability to evaluate information independently as large language models (or LLMs) become embedded in everyday workflows.
“The hazard lives at the nexus in the interaction between how these systems generate language and how human cognition processes it,” Ferguson-Walter said.
For intelligence and national security professionals, who must assess information and communicate uncertainty constantly, that interaction has particular significance. An AI tool may accelerate the work while introducing distortions that are difficult to spot.
What does cognitive security mean for AI use?
Cognitive security concerns how information affects people’s reasoning and their ability to make independent decisions. In AI-assisted work, that includes understanding how a system’s responses shape the conclusions a user reaches.
The risk can even arise without an adversary deliberately manipulating the information. A user may trust an AI-generated summary and accept a claim that differs from the underlying source.
Ferguson-Walter’s team is investigating where interventions can make a difference: when people encounter information, before a distorted claim becomes part of their judgment.
That research informed the development of CogSLLaM, short for Cognitive Security for Large Language Models. The system uses cognitive cues designed to help users recognize distortions in AI-generated information, reducing the effort needed to spot problems.
The team describes how this approach works in a paper published by the Association for the Advancement of Artificial Intelligence.
What did research reveal about AI summaries?
In a study involving 151 participants, the cognitive security team asked people to read short source documents and then LLM-generated summaries. Participants were tested on their ability to recognize claims that distorted the original information.
An important finding from this study concerned what happened after someone spotted a problem. When participants recognized a distortion, they did not carry it into their work product.
“This tells us that the information processing before judgment is made, is where the intervention is needed,” Ferguson-Walter said during her presentation at Information Professionals Association’s APEX 2026 forum, where the submission received second-place recognition in the open category.
Several other patterns were revealed:
- Cues improved performance: Participants responded faster and made fewer errors when cues identified both accurate and distorted claims.
- Experience did not consistently prevent errors: Participants reporting greater LLM experience often made more errors when cues highlighted accurate claims, a pattern consistent with automation complacency.
- Confidence did not reliably reflect accuracy: Participants consistently reported high confidence, even when their answers were wrong.
Why can experienced AI users still make mistakes?
Familiarity with LLMs did not consistently protect participants from errors.
People who reported greater LLM experience, through frequent use or understanding of the underlying algorithms, often made more errors when given cues identifying accurate claims. Ferguson-Walter described this pattern as consistent with automation complacency: trust in a system can reduce a person’s vigilance.
“When a trusted system is visibly checking claims, high trust users may reduce their own vigilance for the rest,” she said.
The result raises a practical question for those designing AI tools. A cue that reassures one user may encourage another to accept unchecked information. Experienced users and novices may need different forms of assistance.
Participants also consistently reported high confidence, even when their answers were inaccurate. Their sense of certainty did not reliably indicate how well they had performed.
You can't self-correct a mistake you're sure you didn't make.
Dr. Kimberly Ferguson-Walter
Director of Research for Cyber Technologies, Leidos
How can AI tools preserve independent judgment?
The team’s findings point toward interventions that help people recognize specific distortions at the moment they encounter them.
Ferguson-Walter points to the pacemaker as a useful analogy for this design goal. It monitors the heart and applies a correction when needed, allowing the heart to continue functioning. In a similar way, cognitive security informed technology could identify a problem in the information a person receives and provide a timely cue.
This goal is a promising direction for future development. The study showed that simple interventions could reduce errors in a document summarization task; further research is needed to understand how those approaches perform across different users and working environments.
For intelligence organizations, the findings offer a useful starting point for evaluating AI-assisted analysis: examine how the tool affects the person using it, including whether its design helps them question a claim and check the evidence.
These questions also matter in dynamic missions like information operations, where AI-informed platforms such as Leidos’ Imperium™ help teams rapidly interpret contested information environments and recognize shifting narratives.
“We can harness the speed and scale that AI provides us and still optimize human cognition and decision making, if we do it carefully,” Ferguson-Walter said.
Watch: Dr. Kimberly Ferguson-Walter
Key takeaways
- Cognitive security examines how AI-generated information affects independent reasoning and decision-making
- Leidos research found that targeted cues can reduce errors in document summarization study
- User experience and confidence with AI did not consistently protect participants from accepting distorted claims
