10decodersMost enterprise RAG pilots fail at the PDF, not the prompt. How to build it right with NVIDIA NeMo Retriever and NIM.
NVIDIA's RAG Blueprint treats extraction, retrieval and generation as separate, swappable services. Here is what each one does and the order to build them in.
September 17, 2026 · 9 min read
Why RAG pilots stall
A RAG demo on clean text takes an afternoon. Then someone uploads a 90-page scanned contract with tables, a benefits schedule saved as an image, or a slide deck full of charts.
The text extractor flattens the tables into noise. The chunks lose their meaning. The LLM answers confidently from the wrong page. The team blames the model and tries a bigger one, and the answers stay wrong, because the problem sits upstream of the model.
NVIDIA's RAG Blueprint, now at version 2.6.0, is built around that failure. You do not have to use all of it. You should understand what each part is for before you replace any of it.
When a RAG system gives wrong answers, check extraction first, retrieval second, and the model last.
The reference architecture in one view
| Component | Default in Blueprint v2.6.0 | Why it matters |
|---|---|---|
| Extraction | NeMo Retriever page elements, table structure, graphic elements and OCR NIMs | Keeps tables and charts structured instead of scrambled |
| Embedding | llama-nemotron-embed-vl-1b-v2, a vision-language model, now the default | Pages with images and charts become searchable, not just their text |
| Vector store | Elasticsearch by default; Milvus with GPU-accelerated cuVS as an option | Weighted hybrid search catches exact terms like policy codes |
| Reranker | Llama Nemotron rerank NIM, with VLM reranker support | Re-scores candidates so the LLM sees chunks that answer the question |
| LLM | nemotron-3-super-120b-a12b with reasoning enabled | OpenAI, AWS, Azure and open models are supported since v2.4.0 |
| Safety and evaluation | Optional NeMo Guardrails; RAGAS-based evaluation and performance tooling | A quality score before your users give you one |
Step by step: from prototype to self-hosted
Step 1. Prove retrieval quality on hosted endpoints
Before deploying a single container, test the three models that decide quality: embedding, reranking and generation. NVIDIA's LangChain connector talks to hosted endpoints today and to your own NIM containers later by changing one URL.
# pip install langchain-nvidia-ai-endpoints
# export NVIDIA_API_KEY=... (from build.nvidia.com)
# Model IDs change often. Copy current IDs from the API catalogue.
from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings, NVIDIARerank, ChatNVIDIA
embedder = NVIDIAEmbeddings(model="<embedding-model-id>")
reranker = NVIDIARerank(model="<rerank-model-id>", top_n=5)
llm = ChatNVIDIA(model="<llm-model-id>")
# Later, point the same classes at self-hosted NIM:
# llm = ChatNVIDIA(base_url="http://your-nim-host:8000/v1", model="...")Build a test set of 50 to 100 real questions, each tagged with the page its answer should come from. Measure how often the right page lands in the top five results, with and without the reranker. That number tells you more than any demo. We covered how to run that comparison in our guide to adding re-ranking to an enterprise RAG pipeline.
Step 2. Fix extraction before you tune anything else
Run your hardest documents through NeMo Retriever extraction and inspect the output by hand. Tables should keep their rows and columns. Chart captions should survive. Scanned pages should come through OCR readable. If extraction is wrong here, no embedding model will rescue it.
Step 3. Deploy the Blueprint with Docker Compose
For a single node, the repository at github.com/NVIDIA-AI-Blueprints/rag ships Compose files for the ingestion server, RAG server, NIMs and vector store. Follow the quick-start for your GPU. Expect the LLM microservice to take five to six minutes to start, and note that single uploads are capped at 400 MB. If you would rather embed the pipeline in your own Python service, there is a library mode and a containerless lite notebook.
Step 4. Tune retrieval for your content
- Turn on weighted hybrid search if users search by codes, product names or clause numbers
- Use collection-level metadata and dynamic filters to scope queries by department, date or document type
- Test query rewriting and decomposition on your question set before leaving them switched on
Step 5. Add guardrails, evaluation and monitoring
Enable the content safety and topic control NIMs if the assistant faces customers or regulated staff. Run RAGAS evaluation on every change to models, chunking or prompts. Log retrieval and reranker scores per query so drift shows up before users report it.
Step 6. Add agentic RAG only after basic RAG works
Version 2.6.0 adds a plan-and-execute agentic pipeline for complex queries, and 2.4.0 brought an MCP server integration. Both help with questions that span several documents. Both also add latency and failure points. Earn them with a working baseline first.
Stuck between a RAG demo and production?
We build RAG and knowledge base systems for healthcare, BFSI and mid-market teams, including on-premise deployments where data cannot leave the building.
Book a Free AI Assessment →Hardware: what to plan for
| Stage | Typical setup | Watch out for |
|---|---|---|
| Prototype | Hosted NVIDIA endpoints, or one workstation-class GPU running smaller NIMs | Hosted model IDs and rate limits change. Pin versions in config |
| Private pilot | On-premise node running the full Blueprint; RTX PRO 6000-class GPUs gained MIG support in v2.5.0 | Audio ingestion is not supported on Kubernetes with RTX 6000 MIG |
| Production | Kubernetes or OpenShift with Helm; H100 or A100 recommended for advanced features | B200 is not supported for image captioning, Guardrails, VLM inference or Nemotron Parse |
A 120B-parameter default LLM needs a lot of GPU memory. Many pilots keep NVIDIA's extraction, embedding and reranking on-premise and swap in a smaller model or an external API for generation. Size GPU counts against NVIDIA's published support matrix for your version.
Where this is not the right answer
- Your corpus is clean text, like a help centre or wiki export. Multimodal extraction adds cost for little gain. Basic embedding and reranking will do.
- You have no GPU budget and no residency requirement. A managed RAG service on your existing cloud gets you live faster.
- You need an answer in two weeks with no DevOps capacity. The full Blueprint runs many containers. Start with hosted endpoints and a minimal pipeline.
RAG readiness checklist
- A scored test set of 50 or more real user questions, each with its source page marked
- Extraction output on your worst scanned and table-heavy documents checked by a person
- Top-5 retrieval accuracy measured with and without reranking
- A written, signed-off rule on which data can leave your network
- Evaluation that runs on every model, chunking or prompt change
- A named engineer who owns the pipeline after launch and can debug GPU and container issues
What to do this week
- Pull 20 of your messiest documents and 50 questions users have actually asked.
- Run the questions against hosted embedding and reranking endpoints and record your top-5 hit rate.
- Inspect extraction output on five scanned or table-heavy files.
- Decide which components must run inside your network, and deploy only those first.
Frequently asked questions
What is the NVIDIA RAG Blueprint?
It is NVIDIA's open reference implementation of a retrieval-augmented generation pipeline. Version 2.6.0 combines NeMo Retriever extraction, embedding and reranking NIMs, Elasticsearch or Milvus, a Nemotron LLM, optional NeMo Guardrails and RAGAS-based evaluation, deployable with Docker Compose or Helm.
Can I use OpenAI or Azure models with the NVIDIA RAG Blueprint?
Yes. Since version 2.4.0 the Blueprint supports non-NIM models, including OpenAI, AWS, Azure and open-source models, so you can keep NVIDIA's extraction and retrieval while using another LLM for generation.
Which vector database does the NVIDIA RAG Blueprint use?
Elasticsearch is the default as of version 2.6.0. Milvus with GPU-accelerated cuVS indexing remains a supported option. Weighted hybrid search combining keyword and vector retrieval is available.
Why does my RAG system give wrong answers from tables and scanned PDFs?
Usually because text extraction flattens tables and loses chart content before anything is embedded. Structured extraction of page elements, tables, charts and OCR, as in NeMo Retriever, fixes the problem upstream of the model.
