RAG is only as good as its context
Key insights from AWS Summit
Why it matters
- Trustworthy AI depends on trustworthy context
- Many AI failures begin before generation
- Context quality is becoming a competitive advantage
Artificial intelligence has a trust problem.
Ask almost any technology leader about generative AI, and they can recall a moment when an AI system delivered an answer that sounded authoritative, only to discover it was wrong. The issue isn’t simply inaccuracy—it’s confidence. A hallucinated answer presented with certainty can be far more detrimental than an obvious failure, particularly in mission environments where decisions carry real operational consequences.
That challenge was at the heart of Leidos’ AWS Summit Washington, D.C. session, “Context is Everything: Building Smarter RAG Pipelines on AWS.”
While discussions about AI reliability often focus on large language models (LLMs), real-world experience suggests that many failures originate elsewhere.
The model may not be the problem. The context pipeline often is.
The hidden reality of RAG
Retrieval-Augmented Generation (RAG) has become the preferred approach for grounding AI responses in enterprise data. Yet many organizations still think of RAG as a retrieval feature layered onto an LLM.
In practice, RAG is an interconnected system.
Before a model generates a single word, content must be extracted, transformed, chunked, embedded, stored, retrieved, ranked, governed, refreshed, and calibrated. Weaknesses at any stage can undermine the quality of the final answer.
A common pattern emerges in AI deployments:
- Source data is fragmented or poorly structured
- Content is chunked without considering retrieval behavior
- Embeddings fail to capture domain-specific meaning
- Search returns information that is similar, but not relevant
- Outdated content remains accessible long after it should have been refreshed
When that happens, teams often blame the model. In reality, the model may be working with incomplete or misleading context.
The lesson: trustworthy AI begins with trustworthy context.
Think of it like sending someone a message but forgetting to include the actual message content.
Context architecture is becoming a competitive advantage
As organizations scale AI, the conversation is shifting from model selection to context architecture.
Embeddings, vector databases, semantic search, hybrid retrieval, and graph technologies are no longer niche concepts reserved for AI specialists. They are becoming foundational design decisions that directly influence business outcomes.
Consider a simple distinction:
- Traditional databases excel at finding exact matches
- Vector stores excel at finding similar meaning
- Graph technologies excel at understanding relationships
As organizations scale AI, the conversation is shifting from model selection to context architecture. Traditional databases excel at exact matches, vector stores find similar meaning, and graph technologies uncover relationships.
Together, they form the foundation for more reliable and explainable AI systems.
Why context should be treated as a living system
One of the biggest misconceptions in enterprise AI is that context management ends once content has been indexed. In reality, context continuously changes—and so must the systems that depend on it.
Many organizations view RAG as a process that ends when an answer is generated. Production-grade RAG does not. The most reliable systems continuously govern, refresh, and calibrate context.
Governance ensures the right information is retrieved by the right users with appropriate permissions and controls. Refresh keeps information current as policies, documents, data, and mission needs evolve. Calibration tunes responses, source citations, uncertainty handling, and alignment with the requirements of each use case.
Together, governance, refresh, and calibration keep context useful after deployment. Without them, even a well-designed RAG pipeline can become stale, inconsistent, or unreliable.
Context is not a static asset but an operational capability that must be continuously managed to deliver trusted, explainable, mission-relevant answers.
Lessons from real-world deployments
Building AI systems for production environments often reveals a gap between proof-of-concept success and operational reality. Leidos has encountered these challenges firsthand while developing ManagedX, an AI document intelligence platform built on AWS that transforms complex enterprise content into structured, searchable, and cited context.
The lessons below reflect practical experience designing RAG systems for commercial and government environments.
1. Smaller AI specialists often outperform one large prompt
Many organizations initially attempt to solve complex workflows with a single, massive prompt. Over time, these systems become difficult to optimize and harder to evaluate.
An emerging alternative is the use of specialized agents focused on distinct tasks such as classification, extraction, validation, and review.
This modular approach enables teams to independently improve each stage while increasing transparency and control. As each stage improves, value compounds.
2. Users want answers, not search results
Enterprise users rarely want a list of documents. They want trusted answers backed by evidence. That principle shaped the design of ManagedX, where users interact through a RAG assistant that delivers source-grounded responses with citations rather than simply returning relevant documents.
This shift is driving growing demand for AI systems that provide source-grounded responses with citations and traceability. Explainability is becoming just as important as fluency, particularly in regulated and mission-critical environments.
Five questions to ask before building RAG
Organizations considering AI-enabled knowledge retrieval can benefit from asking a few foundational questions before selecting tools or architectures.
- Does the embedding model understand your domain?
- What retrieval architecture best fits the problem?
- How frequently does the information change?
- What level of precision does the use case require?
- Is similarity enough, or are relationships essential?
These questions help organizations focus less on technology trends and more on operational outcomes.
The future of enterprise AI is context engineering
As generative AI matures, competitive advantage will increasingly come from how organizations manage context—not simply which model they choose. The real challenge is ensuring answers are relevant, reliable, governed, and explainable. Don't just build a RAG pipeline.
Build the context your mission can depend on.
Three things to remember
- RAG performance depends on context quality, retrieval strategy, and data freshness
- AI hallucinations often stem from weak context pipelines, not the model
- Trustworthy enterprise AI requires grounded, governed, and explainable answers
