Is your technical team developing an AI agent that will answer questions using safety-critical information? Could the agent’s responses influence a design decision or the direction that an engineering program takes? If so, how do we know if its answer can be trusted? Modern AI tools become increasingly capable every day, but in technical and safety-related work, confidence is not evidence of correctness. The challenge doesn’t lie in building an agent that can provide an answer. Rather, it comes from building one whose answers can be traced, verified, and challenged when the evidence used to produce them is insufficient. In many ways, developing a trustworthy AI agent resembles developing a safety-critical system. Hazards must be identified, boundaries must be established, requirements must be defined, and behavior must be verified before pushing it to users. The model itself is just a small part of the solution – trust is established from what is built around the model.

What it Takes to Build Something You Can Trust
If we take a step back, the process of building an agent takes the functional safety mindset and points it at a new kind of system. Before the first prompt is written, a team naturally begins by identifying potential hazards. For example, among other questions, we asked, “What happens if the agent provides an answer without supporting evidence?” and “What happens if it cites outdated or conflicting information?” while developing our AI agent.

After answering such questions, a team defines the boundaries of the system. This includes which information the agent can use, what it can say, how it cites its answers, and what questions it should refuse to answer. These decisions become requirements that shape the design of the knowledge base, retrieval process, permissions, citations, and audit trail.

The technologies may be new, but the approach behind building them is a familiar engineering one. Rather than trying to construct the smartest model, the most valuable effort consists of establishing layers that determine safeguards, evidence, and governance that allow the user to trust the system. Once implemented, those controls must be tested and validated to ensure that the agent behaves as intended.

Why this matters now
In functional safety, the point of a system’s lifecycle is to prove that it can hold up outside the conditions for which it was designed. The same applies to AI agents. A polished demonstration creates an illusion of readiness, but in production, the agent doesn’t field carefully selected test questions. It responds to real users making real decisions in a language that sounds just as ready, even if the evidence behind its results is outdated or missing.

Consumer-facing apps generally have low stakes. A wrong answer could mean a bad movie recommendation or a clunky, robotic interaction. In engineering or safety work, a wrong answer could slip into a decision with real consequences. That’s why “Can this agent give me an answer?” is the wrong metric to pursue. Any modern AI model can provide an answer, but what we must consider is whether the answer can be trusted and verified. The verification check is an engineering problem and where most of the effort lies.

The model is the easy part
Today’s foundational models are very strong out of the box and the platforms around them handle the work of pulling the right documents and fitting them to a prompt. The model that someone selects has become one of the most trivial elements of the whole project.

What actually determines whether the agent can be trusted are the many layers built around it. Two can be easily specified: the sources it draws from and the things it refuses to say. The governance layer tends to be ignored, which is significant because it represents the ability to trace and defend what the agent told a user long after the conversation is over. Building an agent with the belief that you can choose a model with minimal setup will inevitably lead to poor results. In our experience, the model selection amounts to at most 20% of the effort, and the remaining resources are required for building the layers around it.

Grounding is the decision that matters most
Before building a reliable agent, one question needs to be answered: “Will the agent only answer from an approved set of sources with a required citation after every answer, or is it able to draw on anything it picked up during training?” This decision determines whether the agent will be reliable or a liability. In a technical field, a confidently wrong answer can have bigger consequences than no answer at all, because a confidently wrong answer is plausible enough that users act on it without verifying the information.

The standard technique to avoid crossing this line is called retrieval augmented generation, or RAG. Instead of the model generating an answer from everything it was trained on, it first searches a defined set of documents, pulls the relevant sections, and builds a concise answer from that information. Every answer comes back with a citation that you can open and read for yourself. This is a functional safety concept put into practice: you decide what the system can rely on and make the reliance inspectable. A claim in a safety case is only worth as much as the source used to substantiate it, and the same applies to an agent’s answer.

The knowledge base is the real work
“Point it at our documents” is an instruction that hides most of the agent’s work because it assumes that the documents are clean and machine-readable. In practice, they rarely are.

Scanned PDFs tend to catch an agent first. To the human eye, a scanned PDF looks identical to a normal page. An agent interprets it as an image with no readable text, making it invisible until you run optical character recognition (OCR) to extract the words. If that step is skipped, the whole document drops from the agent’s view without the user’s knowledge.

Another example of an overlooked source is one that was never written down, such as an educational webinar or an undocumented tool walkthrough. The information has to be transcribed before the agent can interpret any of it, which requires the human judgement call of deciding if the content is worth transcribing.

The usage of consistency over completeness must also be considered when building an agent. Three slightly different versions of the same document or one document saved under several different file names will cause more problems for ingestion than a file that does not even exist. In such cases, the agent has to determine which document takes precedence, and if given enough tries, it will guess incorrectly. A small, well-organized library will reliably outperform a large and messy one. This is why naming conventions and file structure are an engineering practice that may seem negligible but is critical in the long term.

On most platforms, another factor that catches teams off guard is the retrieval of information as the person using the agent – in other words, it can only see what the user is permitted to see. This is a sensible design decision because it prevents the agent from citing a file that the user should not be able to access. However, this permission structure can affect how the agent behaves. For example, two people with different levels of access could ask the same question but receive different answers. The practical takeaway is to test using an ordinary user’s access rather than that of the builder. Builders are the worst test cases because they usually have unrestricted access to all facets of the agent.

The most valuable thing you can teach it is to say no
Every safety-critical system needs a defined safe state, which is the behavior it falls back to when a hazard is detected. For an agent, that safe state is a refusal. It’s common for a builder to invest a significant amount of time in making the agent better at answering questions, but the more difficult task is teaching it when to refuse.

Refining this skill happens in layers, starting with plain refusal. Without an approved source, there is no answer. The agent should explicitly state so and point the user to a different direction for further help. The agent’s ability to say “I can’t answer this question from the knowledge that I have” should be treated as a strong feature rather than a limitation.

Judgement is the ultimate boundary for an agent. The agent can lay out what an approved document says but whether a given approach is sufficient for a specific task is not its call to make. Its job is to inform the decision and leave the professional judgement to the user, who carries the accountability for it.

Someone building an agent must consider these factors to answer the question: “Why is restraint hard to build?” Above all else, the model is trained to be helpful, but without the refusal guardrail, the agent will want to simply find relevant information and reconcile disagreements before providing a clean and resolved answer. Practicing restraint means working against the agent’s original purpose, and that is where trust begins to build. This strategy of designing a fallback mechanism that competes with the default behavior is also a standard in safety engineering.

What You Should Know When Building
Every project has residual risk because no set of controls can eliminate all risks. Therefore, several noteworthy limitations accompany the building of AI agents.

For example, guardrails are probabilistic safeguards that should be treated as risk reductions rather than a guarantee. An audit log of guardrail failures provides explanations of which issues occurred and when, facilitating traceability.

Most importantly, the agent is a complete reflection of its sources. If any content in the agent’s knowledge base is wrong or obsolete, those errors will be reflected in the answers delivered to users. For better or worse, the quality of the answers is directly related to the quality of the sources.

However, these limitations should not discourage building. Rather, they proactively help build and support the framework under the intelligence before the agent becomes user-facing.

Conclusion
Building an agent that can answer questions is the easy part – the more difficult task is creating an agent that is verifiable and auditable when the answer actually matters.

While the technology is fairly new, the methodology behind building the agent is not. The standard steps include identifying the hazards, defining what the system relies on, turning those decisions into requirements, and testing the system. It’s the same discipline that engineers already practice daily, but it’s aimed at a system that produces language instead of motion or physical outputs.

The model provides the intelligence, but the surrounding engineering decisions determine if that intelligence can be trusted. So, if your agent answered a critical question today, could you identify the evidence behind its answer and confidently defend it? If not, closing that gap is the next engineering problem to solve.