Knowledge Retrieval Agent: AI That Only Knows What You Tell It

When you ask a standard AI model a question it does not know the answer to, it does not say “I don’t know.” It makes something up. For a casual conversation that is annoying. For a business using AI to answer questions about its own policies, contracts, or procedures, it can become a serious problem.

The Knowledge Retrieval Agent reduces this risk by grounding answers in your own documents. It reads your private documents, converts every text chunk into a mathematical format that can be searched by meaning rather than by keyword, and then forces the AI to answer using only what it found in your files. If the answer is not in your documents, the agent says so clearly instead of inventing one.

What This Agent Does

Meridian Advisory Group, a fictional consulting firm I use as my example organization throughout this, has two internal policy documents: a client service policy covering refunds, confidentiality, and engagement terms, and a data handling policy covering retention schedules and breach response.

A new employee asks: “What is our refund policy for cancelled engagements?”

Without this agent, an AI model might answer from its training data, which contains nothing about Meridian’s internal policies. It would either guess, hallucinate a plausible-sounding policy that does not exist, or simply make something up that contradicts the actual document.

With the Knowledge Retrieval Agent running, the question is converted into a mathematical vector, the agent searches its index of Meridian’s actual policy documents, finds the most relevant text chunks, hands only those chunks to the AI, and instructs it strictly: answer from this text only. The AI then summarizes what the document actually says.

The answer is grounded in the retrieved document chunks because the prompt tells the AI to use only the provided context.

Why I Built This

Every AI deployment I have reviewed hits the same wall. The AI is confident, fluent, and wrong. Not maliciously wrong, just statistically wrong in the way that language models are when they do not have the right information and have not been constrained from filling the gap with their own knowledge.

The standard industry response is to say “use RAG.” Retrieval-Augmented Generation. Every tutorial explains the concept. Almost none of them show you what it looks like when the framework dependency breaks, when the embedding model gets deprecated by the provider, or when the routing architecture you assumed was working is actually bypassing your proxy entirely.

I built this agent to have a clean, LangChain-free, production-honest reference implementation. One that documents what actually breaks and why.

The Architecture: How It Works

The agent operates in two phases: a build phase and a query phase.

In the build phase, the agent reads every text file in a designated folder, slices each document into overlapping character-based chunks so no information gets cut off at an arbitrary boundary, sends each chunk to an embedding model that converts the text into a list of floating-point numbers representing its meaning, and stores those numbers in a FAISS index, which is a local search engine designed specifically for this kind of mathematical comparison.

In the query phase, the user types a question. The agent converts that question into the same mathematical format using the same embedding model. It then searches the FAISS index for the chunks whose numbers are closest to the question’s numbers, meaning the chunks most similar in meaning to what was asked. It takes the top results, builds a prompt that contains only those chunks plus the question, sends that to the AI, and instructs the AI to answer using only the provided text.

The AI never sees anything except what the agent gives it.

Core Components

The FAISS index is the search engine at the center of this build. FAISS stands for Facebook AI Similarity Search. It was developed by Meta and released as open source. It is built for one specific job: given a list of vectors and a new vector, find the ones that are mathematically closest. In this agent, the vectors are chunk embeddings and the query is the question embedding. FAISS handles the comparison at speeds that would be impossible with a standard database query.

The embedding model is what converts text into vectors. Think of it as a translation layer. Every retrieved chunk from your documents, and every question the user asks, gets translated into the same coordinate system. Two sentences that mean the same thing end up near each other in that coordinate system even if they use completely different words. This is why the agent finds relevant chunks even when the user’s question uses different vocabulary than the document.

LiteLLM is the unified gateway that routes every API call. The chat model, the embedding model, the provider behind each, all of it is configured in one place. Swapping providers requires changing two lines in the .env file and nothing else.

The Model-Agnostic Layer

For this verification, I ran the agent through DeepSeek V4 Pro behind the local LiteLLM proxy. The chat model and embedding model are both configuration variables, so the code stays model-agnostic. To switch providers, you update MODEL_NAME and EMBEDDING_MODEL_NAME in the .env file instead of rewriting the agent.

The Build: Step by Step

Step 1: Document chunking without a framework

I used a Python function that slices text into overlapping segments of a configurable character length.

The overlap is the important part. If a critical piece of information sits at the boundary between two chunks, a non-overlapping chunker will cut it in half and the retrieval will miss it. With overlap, the same text appears at the end of one chunk and the beginning of the next, giving the index two chances to find it.

Both chunk size and overlap are environment variables. A team processing dense documents might need smaller chunks with more overlap. A team processing internal memos might need larger chunks. Neither should require touching the code.

def chunk_text(text, chunk_size, chunk_overlap):
    if chunk_overlap >= chunk_size:
        chunk_overlap = chunk_size // 2

    step = chunk_size - chunk_overlap
    return [text[start:start + chunk_size] for start in range(0, len(text), step)]

What this code is actually doing: The function starts at position zero in the text and cuts a segment of chunk_size characters. Then instead of moving forward by the full chunk size, it moves forward by chunk_size minus chunk_overlap, so the next chunk starts a little bit back from where the previous one ended. This creates the overlap. The loop continues until it reaches the end of the document.

Step 2: Building the vector index

Once the documents are chunked, every chunk gets sent to the embedding model. The response comes back as a list of floating-point numbers, one list per chunk. Those lists get loaded into a numpy array and added to the FAISS index.

python

def build_index(chunks, config):
    print("Embedding chunks for the knowledge base...")
    embeddings = np.array(embed_texts(chunks, config), dtype=np.float32)
    index = faiss.IndexFlatL2(embeddings.shape[1])
    index.add(embeddings)
    print(f"Total chunks embedded and indexed: {len(chunks)}")
    return index

What this code is actually doing: Imagine you have a library of 6 chunks and you need to store each one as a set of coordinates so you can find the closest ones later. The embedding model already translated each chunk into a list of numbers, like GPS coordinates but with hundreds of dimensions instead of just latitude and longitude. NumPy takes all those lists and arranges them into a clean grid, one row per chunk, so the math can be done efficiently.

FAISS then takes that grid and builds a searchable index from it. Before it can do that, it needs to know one thing: how many coordinates does each chunk have? It reads that number from the grid automatically. The embedding size depends on the configured embedding model. That number never changes because the same embedding model always produces the same size output.

IndexFlatL2 is the search method. The name sounds technical but the concept is simple. When you ask a question, FAISS converts it into the same kind of coordinate list. Then it measures the distance between your question’s coordinates and every chunk’s coordinates. The chunks with the smallest distance are the ones most similar in meaning to your question, the same way the nearest coffee shop is the one with the smallest distance from where you are standing. FAISS returns the closest ones and those become the context the AI uses to answer.

Step 3: The retrieval and answer loop

When the user types a question, the agent runs it through the same embedding model that processed the documents. This converts the question into the same kind of coordinate list that every chunk in the index already has. FAISS then does one job: it looks at every chunk’s coordinates and finds the ones that are mathematically closest to the question’s coordinates. Those are the chunks most likely to contain the answer. The agent takes the top four, by default, and hands them to the AI as the only text it is allowed to use.

Think of it like this. You walk into a library and ask a librarian a question. Instead of reading every book, the librarian goes straight to the shelf where the most relevant books are clustered together because the library is organized by meaning, not by alphabet. The agent is that librarian. FAISS is the organizational system that makes it possible.

system_prompt = (
    "Answer using only the provided context. If the answer is not "
    "in the context, say you could not find it in the documents."

)

What this code is actually doing: This one line of instruction is the most important part of the entire build. Without it, the AI would read your documents and then add its own knowledge on top, mixing what you gave it with what it already believes, and you would have no way to tell which parts of the answer came from your files and which parts the AI invented.

The system prompt changes the AI’s role from a thinker to a reader. It is told: here is some text, here is a question, your only job is to summarize what the text says. Do not add anything else. If the text does not contain the answer, say so.

That single constraint is what makes this agent trustworthy for business use. A customer service team can point it at their policy documents and know that every answer it gives came directly from those documents, not from the AI’s best guess about what a reasonable policy might look like.

The Bug That Took an Hour: What Nobody Documents About LiteLLM Embeddings

When I started building this agent, I followed the standard approach that every LiteLLM tutorial and documentation page describes. There is a built-in function called litellm.embedding() that is supposed to handle embedding calls the same way the rest of LiteLLM handles chat calls. You pass it your model name, your text, and the URL of your proxy, and it is supposed to route the request through your setup just like everything else does.

It failed. Every time. Generic connection error. No useful message. The proxy was running fine, chat completions were working perfectly, but embeddings would not go through no matter what I tried.

Here is what was actually happening beneath the surface. Every piece of documentation points you there. I tried it. It failed with a generic connection error every time, even though the proxy was running perfectly and chat completions were working fine.

Here is what was actually happening.

litellm.embedding() is designed to call AI providers directly using their native authentication systems. When you pass api_base to it pointing at your own proxy, it does not simply forward the request to the proxy and let the proxy handle authentication. Instead, it tries to authenticate directly with the provider, in this case Google Gemini, using Google’s credential system rather than your LiteLLM proxy key. My computer does not have Google credentials configured. The call fails before it ever reaches my proxy.

The second problem compounded this. Google deprecated the Gemini embedding models text-embedding-004 and embedding-001 from their v1beta API endpoint in early 2026. LiteLLM routes Gemini embedding calls to v1beta. This meant every Gemini embedding model string I tried returned a 404 not found error at the Google API level, regardless of how I configured the LiteLLM version or the config file.

The model that works as of May 2026 is gemini-embedding-001, configured in the LiteLLM proxy config with the gemini/gemini-embedding-001 model string and referenced in the agent by its proxy alias text-embedding. Not by the provider model name.

After about 45 minutes of hitting the same wall, I went back to basics. I opened my VPS terminal and ran a curl command directly against the proxy. Curl is the simplest possible way to test an API: you type a command, it sends a request, you see exactly what comes back. The curl command worked perfectly. Embeddings came back in under a second.

That told me something important. The proxy was not broken. The Python function was broken. The proxy could handle embedding requests just fine when something talked to it correctly. The question was what curl was doing that the Python function was not.

Curl sends a plain HTTP request. It does not know anything about Google, or Gemini, or LiteLLM’s internal routing logic. It just sends a request to a URL with a key and gets a response back. It has no opinion about which AI provider is on the other end.

That is when I looked for a Python library that works the same way curl does. Just send an HTTP request to a URL, include the key, get the response. No assumptions about providers. No internal routing logic. No trying to authenticate with Google directly.

That library is httpx. It is one of the most widely used HTTP libraries in Python, the equivalent of curl but in code. Instead of asking LiteLLM’s Python function to manage the embedding call, I used httpx to talk directly to my proxy’s /v1/embeddings endpoint, the same endpoint curl was hitting, with the same LiteLLM API key, and let the proxy handle everything else internally.

It worked on the first try.

The proxy exists to sit between your code and the AI providers and handle all the authentication complexity. The correct way to use it from your application is to talk to the proxy directly, not to use a library that tries to talk to the provider on your behalf. Curl figured that out automatically. Once I understood what curl was doing, the fix took about two minutes to write.

response = httpx.post(
    f"{base_url}/v1/embeddings",
    headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
    json={"model": embedding_model_name, "input": texts},
    timeout=60,
)

Instead of asking LiteLLM’s Python library to call Google for you, you ask your proxy directly, and your proxy calls Google. One extra hop. The proxy exists precisely to be that intermediary.

This fix is visible in the simplified embedding function: the code calls the LiteLLM /v1/embeddings endpoint directly with httpx. If you clone the repo and wonder why the embedding function looks different from every other LiteLLM example you have seen, that simplified function is your answer.

What I Would Do Differently

The knowledge base in this build lives entirely in memory. The moment the agent stops running, the index is gone. Every restart re-embeds every document from scratch. For the two small sample text files in this repo, this takes only a few seconds. For a thousand-page legal archive it would take minutes and cost real API money on every restart.

A production version would save the FAISS index to disk after the first build and reload it on subsequent runs, only rebuilding when the documents change. That is a two-function addition to this agent and the natural next step for anyone deploying this at scale.

The document format is also limited to plain text files. Real business data lives in PDFs, Word documents, SharePoint sites, and email threads. Extending the ingestion pipeline to handle multiple formats is the gap between a demonstration and a deployable product.

The way this agent slices documents into pieces is simple and it works well for getting started. It counts characters and cuts at a fixed number, with a small overlap so nothing important gets split in half. Clean, fast, and no chunking framework.

But here is the problem that fixed-size cutting creates in the real world.

Imagine a legal contract with a clause that reads: “The client may terminate this agreement with 30 days written notice, provided that all outstanding invoices have been settled in full and no active deliverables are pending review.” That clause is the answer to the question “how do I cancel this contract?” It needs to stay together in one chunk for the agent to retrieve it correctly.

If that clause happens to start near the end of a 1000-character chunk, the chunker cuts it in the middle of the sentence. The first half ends up in one chunk. The second half starts the next chunk. When a user asks about cancellation, the agent retrieves whichever half is most similar to the question and returns an incomplete answer. The user gets half a policy and thinks they understand the full picture.

This is not a theoretical problem. It happens constantly with fixed-size chunking on real business documents, and it is invisible. The agent does not tell you it returned an incomplete answer. It returns what it found and the AI summarizes it confidently.

Semantic chunking solves this by reading the document for meaning rather than counting characters. Instead of cutting every 1000 characters, it detects when the topic shifts. A paragraph about payment terms ends. A new paragraph about termination begins. That is the cut point. Each chunk contains one complete idea, not one arbitrary slice of text.

The practical impact is significant. A RAG system with good chunking retrieves complete, coherent answers. A RAG system with poor chunking retrieves fragments and the AI fills the gaps with its own knowledge, which is exactly the hallucination problem we built this agent to eliminate in the first place.

Semantic chunking is the next meaningful upgrade to this architecture and the reason retrieval quality in enterprise RAG systems varies so dramatically between implementations that look identical on the surface.

Use Case

If your organization uses AI to answer questions about your own policies, procedures, contracts, or products, you currently have one of two problems. Either the AI is guessing, which means your staff or customers are sometimes receiving wrong answers delivered with complete confidence. Or you are not using AI for these tasks at all because you do not trust it, which means your team is spending time answering questions that a well-architected system could handle reliably.

This agent represents the architecture that makes AI trustworthy for document-grounded tasks. It does not require a cloud service, a managed RAG platform, or a vendor contract. It runs on your own infrastructure, the application reads local documents and routes model requests through your configured LiteLLM proxy, so deployment privacy depends on how that proxy and provider are configured., and the model behind it can be swapped without touching the code.

The open source foundation this is built on would cost a team of developers weeks to research, evaluate, and assemble from scratch. The build documented here took one working session plus one hour of debugging a routing issue that is now documented so you never have to hit it yourself.

Tools Used

FAISS (MIT): github.com/facebookresearch/faiss
LiteLLM (MIT): github.com/BerriAI/litellm
httpx (BSD): github.com/encode/httpx
python-dotenv (BSD): github.com/theskumar/python-dotenv
NumPy (BSD): github.com/numpy/numpy
NosisTech Agent Engineering — github.com/nosistech/agent-engineering