LlamaIndex 2026: Build Production RAG in 30 Lines of Python

What Is LlamaIndex?

LlamaIndex is a Python (and TypeScript) data framework purpose-built for connecting custom data sources to large language models. It is the layer that sits between your raw data — PDFs, databases, APIs, Notion pages, Slack archives — and the LLM that answers questions about that data. Created by Jerry Liu in late 2022 and now maintained by LlamaIndex Inc., it has become one of the two most-used frameworks in the retrieval-augmented generation (RAG) space.

Where a general-purpose orchestration framework like LangChain spans chains, agents, tools, and integrations across many use cases, LlamaIndex is opinionated about one thing: turning data into something an LLM can reason over. Its primitives — Documents, Nodes, Indexes, Retrievers, Query Engines, Response Synthesizers — are deeper and more specialized for retrieval than what you find in general frameworks.

The original framework was named GPT Index. It was renamed to LlamaIndex in early 2023 to signal the broader ambition: not just indexing, but the whole data-to-LLM lifecycle. By 2026 it spans data ingestion via 160+ connectors, multiple index structures, sophisticated retrieval strategies, agent loops, and an event-driven Workflows runtime for production applications.

Why LlamaIndex in 2026

Three things make LlamaIndex the default choice for serious RAG projects in 2026: the depth of its retrieval primitives, the breadth of its data connectors, and the maturity of the Workflows API for agentic applications.

Depth of RAG Primitives

Most teams arrive at RAG through tutorials that show a 20-line "load-embed-query" pipeline and stop there. Real production RAG needs more: hybrid retrieval combining vector and keyword search, sub-question decomposition for complex queries, recursive retrieval across document hierarchies, reranking with Cohere or Voyage, post-retrieval transformations, and multi-step response synthesis. LlamaIndex ships all of these as first-class abstractions, not glue code you write yourself.

Breadth of Data Connectors

The LlamaHub registry — at llamahub.ai — provides over 160 readers for ingesting data from real-world sources. PDF and DOCX with layout preservation. Notion with hierarchical block parsing. Slack with channel-aware threading. Google Drive, Confluence, Jira, GitHub, Airtable, SQL databases of every flavor, S3, GCS, Azure Blob. Most production RAG projects fail not at the LLM step but at the data step. LlamaHub eliminates 80 percent of that friction.

Workflows for Agentic Applications

The Workflows API, introduced in 2024 and matured through 2025, is LlamaIndex's event-driven framework for multi-step LLM applications. Each step is a Python function decorated with @step that listens for specific events and emits new events. This model expresses agent loops, branching, parallel execution, and human-in-the-loop checkpoints more cleanly than the older DAG-based pipeline abstraction. By 2026 Workflows has become the recommended way to build any LlamaIndex application beyond a simple query engine.

Core Concepts: Documents, Nodes, Indexes

To use LlamaIndex effectively, you need a clear mental model of its core abstractions. Five concepts cover almost everything: Documents, Nodes, Indexes, Retrievers, and Query Engines. Each layer transforms the data progressively from raw bytes to a queryable knowledge surface.

Documents

A Document is the input unit — a single source-of-truth object representing one PDF, one Notion page, one row from a SQL query, one webpage. It carries the raw text plus metadata (filename, author, URL, timestamps, custom tags). Documents come from data connectors. They are the framework's "what you started with" object.

Nodes

A Node is a chunk of a Document — typically a few hundred tokens of text plus inherited metadata. The chunking strategy matters for retrieval quality: too-small chunks lose context, too-large chunks dilute relevance scores. LlamaIndex offers multiple node parsers including SentenceSplitter (the default), SemanticSplitterNodeParser (semantic boundary detection), and structure-aware parsers for Markdown, JSON, and HTML.

Indexes

An Index is the searchable structure built over Nodes. LlamaIndex provides several index types because different data shapes call for different retrieval strategies:

• VectorStoreIndex — embeds each Node and stores vectors for similarity search. The default and the workhorse for most RAG.
• SummaryIndex — stores Nodes in a flat list and synthesizes answers from all of them. Useful when you want the LLM to see every chunk.
• TreeIndex — builds a hierarchical tree where parent nodes summarize their children. Good for very large corpora where you traverse top-down.
• KeywordTableIndex — maps keywords to Nodes for keyword-based retrieval, complementary to vector search.
• KnowledgeGraphIndex — extracts entities and relationships from Nodes and stores them in a graph for relationship-aware queries.

Retrievers and Query Engines

A Retriever takes a query string and returns relevant Nodes. A QueryEngine wraps a Retriever and adds the LLM call: it retrieves Nodes, synthesizes a response over them, and returns the final answer with optional source citations. Both are obtainable from any Index via index.as_retriever() and index.as_query_engine() — the most-used method calls in the entire framework.

Response Synthesizers

The Response Synthesizer is the strategy for turning retrieved Nodes plus the user query into a final answer. Options include refine (iteratively refine an answer across nodes), compact (concatenate nodes into the largest prompt that fits), tree_summarize (recursively summarize across nodes), and simple_summarize (truncate-and-answer). Choosing the right synthesizer for your use case is one of the highest-leverage tuning decisions in production RAG.

Installation & First RAG Pipeline

LlamaIndex requires Python 3.9 or higher. The core package installs with a single pip command. For specific LLM providers, embedding models, or vector stores, install the corresponding integration package.

Step 1: Install the Core Package

# Core framework + default OpenAI integrations
pip install llama-index

# Common provider integrations (install only what you need)
pip install llama-index-llms-anthropic
pip install llama-index-embeddings-cohere
pip install llama-index-vector-stores-qdrant
pip install llama-index-readers-file

Step 2: Set Your API Key

By default LlamaIndex uses OpenAI for both embeddings and LLM calls. Set your OPENAI_API_KEY environment variable, or configure a different provider via the Settings object.

Step 3: Build Your First RAG Pipeline

The canonical "load some files, ask questions" pipeline is genuinely five lines of code. This is the demo that hooks every new user.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# 1. Load every file in ./data into Documents
documents = SimpleDirectoryReader("./data").load_data()

# 2. Chunk into Nodes, embed, and build the vector index
index = VectorStoreIndex.from_documents(documents)

# 3. Get a query engine (retriever + LLM synthesis)
query_engine = index.as_query_engine()

# 4. Ask a question
response = query_engine.query(
    "What were the main findings in the 2025 report?"
)

print(response)
print("Sources:", [n.metadata for n in response.source_nodes])

That snippet does five things behind the scenes: reads files via SimpleDirectoryReader, chunks them with the default sentence splitter, calls OpenAI's embedding API for each Node, stores embeddings in an in-memory vector store, and on query retrieves the top-k similar Nodes and synthesizes an answer with the default LLM. To persist the index for reuse, call index.storage_context.persist(persist_dir="./index_store") and reload with load_index_from_storage.

Advanced RAG with LlamaIndex

The five-line pipeline is the demo. Production systems need more: query decomposition for complex questions, hybrid retrieval that combines vector and keyword signals, reranking to push the most relevant chunks to the top, and recursive retrieval across hierarchical data. LlamaIndex ships all of these as composable building blocks.

Sub-Question Query Engine

For questions that span multiple topics or documents, the SubQuestionQueryEngine uses an LLM to decompose the user's question into smaller sub-questions, routes each to the right query engine, and synthesizes a final answer. This pattern dramatically improves accuracy on multi-hop questions.

Hybrid Retrieval and Reranking

Vector search alone misses queries dominated by rare terms or exact-match requirements. Pairing vector retrieval with BM25 keyword retrieval — and reranking the combined candidates with a cross-encoder like Cohere Rerank or Voyage Rerank — typically lifts retrieval recall by 15–30 percent on real corpora.

from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.postprocessor.cohere_rerank import CohereRerank
from llama_index.core.query_engine import RetrieverQueryEngine

Settings.chunk_size = 512

# Build vector + BM25 retrievers over the same nodes
index = VectorStoreIndex.from_documents(documents)
vector_retriever = index.as_retriever(similarity_top_k=10)
bm25_retriever   = BM25Retriever.from_defaults(
    docstore=index.docstore, similarity_top_k=10
)

# Fuse with reciprocal rank fusion
hybrid_retriever = QueryFusionRetriever(
    [vector_retriever, bm25_retriever],
    similarity_top_k=10,
    num_queries=1,
    mode="reciprocal_rerank",
)

# Cross-encoder rerank to keep only the top 3
reranker = CohereRerank(top_n=3, model="rerank-english-v3.0")

query_engine = RetrieverQueryEngine.from_args(
    retriever=hybrid_retriever,
    node_postprocessors=[reranker],
)

response = query_engine.query(
    "How does the proposed method compare to the 2024 baseline?"
)
print(response)

Recursive Retrieval

For data with hierarchy — say, a document collection where each document has a summary node and many leaf chunks — recursive retrieval first matches against summaries, then drills into the leaves of the matched documents. This dramatically improves retrieval over very large corpora because you do not need every chunk in the same flat vector space.

LlamaIndex Agents & Workflows

LlamaIndex provides two agent patterns and an event-driven runtime. The simpler pattern is FunctionAgent and ReActAgent — a single agent loop wrapping tools. The more powerful pattern is the Workflows API — event-driven steps that compose into arbitrary multi-agent and multi-step applications.

FunctionAgent and ReActAgent

FunctionAgent uses native function-calling on models that support it (OpenAI, Anthropic, Mistral, recent open-weights). ReActAgent uses the ReAct prompting pattern (Reason, Act, Observe) and works with any chat-capable LLM. Both expose tools — Python functions or LlamaIndex query engines wrapped as tools — and run an iterative loop until the agent decides it is done.

from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.tools import QueryEngineTool
from llama_index.llms.anthropic import Anthropic

# Wrap an existing query engine as a tool
research_tool = QueryEngineTool.from_defaults(
    query_engine=query_engine,
    name="research_2025_reports",
    description="Search the 2025 industry research corpus.",
)

# A regular Python function also works as a tool
def calculate_growth(start: float, end: float) -> float:
    """Compute year-over-year growth percent."""
    return ((end - start) / start) * 100

agent = FunctionAgent(
    tools=[research_tool, calculate_growth],
    llm=Anthropic(model="claude-sonnet-4-6"),
    system_prompt="You are a careful research analyst.",
)

response = await agent.run(
    "What was the cloud spend growth from 2024 to 2025 per the reports?"
)
print(response)

The Workflows API

For anything beyond a single agent loop, Workflows is the modern path. Define a class that subclasses Workflow; each method decorated with @step consumes an event and emits one or more new events. The runtime handles dispatching, parallel execution where steps don't depend on each other, and produces an event log for debugging. Workflows is also the foundation for human-in-the-loop patterns — a step can wait for an external event before continuing, making approval gates and tool-confirmation flows natural to express.

LlamaIndex vs LangChain vs Haystack

LlamaIndex, LangChain, and Haystack are the three most-used Python frameworks for building LLM applications. They overlap but have meaningfully different design centers. Choosing the right one comes down to whether your problem is fundamentally retrieval, fundamentally orchestration, or fundamentally enterprise-search.

The pragmatic answer: if your application is fundamentally "answer questions over my documents and data," LlamaIndex is the cleaner fit and you will write less glue code. If your application is fundamentally "an agent with many tools, only one of which is retrieval," LangChain (with LangGraph) gives you more orchestration surface. If your application is fundamentally "build a search engine over enterprise content," Haystack's pipeline model maps closely to the problem.

Many production teams use both LlamaIndex and LangChain together — LlamaIndex for the data and retrieval layer, LangChain or LangGraph for higher-level agent orchestration. The two frameworks interoperate cleanly through the tool abstraction.

Production Deployment

Going from a notebook prototype to production RAG involves three things: a real vector store, observability, and an evaluation loop. LlamaIndex has first-class integrations for all three.

Vector Stores

The in-memory vector store is fine for prototypes and small corpora. For production, swap to a managed or self-hosted vector database. The main integrations:

• Pinecone — fully managed, low-friction, the default for most teams that want zero operations
• Weaviate — open-source with strong hybrid search built in, deployable self-hosted or as a managed service
• Qdrant — open-source, Rust-based, fast filtering, popular with teams that want self-hosted control
• Chroma — embedded-first, Python-native, great for local development
• Milvus — open-source, designed for very large scale (billions of vectors), favored in enterprise
• pgvector — Postgres extension, perfect when your team already runs Postgres and wants to avoid a new datastore

Wiring any of these into LlamaIndex is a one-line swap. Build the index against the chosen vector store and the rest of your code is unchanged.

Observability

LlamaIndex emits structured events and integrates with multiple observability platforms: Arize Phoenix for LLM tracing, LangSmith for trace inspection and evaluation, and OpenTelemetry for sending traces to whatever APM you already use (Datadog, Honeycomb, Grafana Tempo). Turn on tracing before you ship — debugging RAG without traces is painful and unnecessary.

Evaluation

Production RAG without an evaluation harness drifts. LlamaIndex provides evaluators for faithfulness (does the answer use the retrieved sources?), relevancy (is the retrieved content actually relevant?), context recall, and answer correctness. Build a small eval set of 50–200 representative queries with reference answers, and run it on every meaningful change to chunking, embeddings, retrieval, or prompts. The teams that ship reliable RAG run their eval set on a CI cadence the same way they run unit tests.

Best Practices & Common Pitfalls

Eight practices separate teams that ship reliable LlamaIndex applications from teams that endlessly tweak prototypes:

1. Tune chunk size against an eval set, not by feel. Defaults are conservative. Run a 256 / 512 / 1024 / 1500 token sweep against your eval set and let the numbers decide.
2. Choose embeddings deliberately. The default OpenAI text-embedding-3-small is fine for many use cases but not all. For long-context retrieval try Voyage voyage-3-large; for cost-sensitive deployments evaluate Cohere embed-english-v3.0 or open-weights like BGE-M3. Embedding choice affects retrieval quality more than most prompt changes.
3. Add a reranker before you tune anything else. Cross-encoder reranking on the top 20–50 candidates almost always helps and rarely hurts. It is the single highest-ROI addition after a baseline pipeline works.
4. Evaluate retrieval separately from generation. When the system gives a wrong answer, you need to know whether retrieval missed the right chunk or the LLM ignored a chunk it received. LlamaIndex's separate RetrievalEvaluator and answer evaluators make this clean.
5. Cache embeddings aggressively. Re-embedding documents on every index rebuild is wasted money and time. Persist the docstore and re-embed only changed nodes.
6. Watch the tokens. A SubQuestionQueryEngine with 5 sub-questions and refine synthesis can easily issue 10+ LLM calls per user query. Profile token usage early and decide consciously where the spend is going.
7. Use Workflows over QueryPipeline for new code. The older QueryPipeline abstraction still exists but Workflows is the recommended pattern for any non-trivial application going forward. Migrating later is harder than starting there.
8. Don't fight the framework on simple cases. If VectorStoreIndex.from_documents() + .as_query_engine() answers your need, ship that. Most production RAG starts simple and adds complexity only where evaluation shows a gap. Premature complexity is the most common LlamaIndex anti-pattern.

Sources: LlamaIndex Official Documentation, LlamaHub Connector Registry, LlamaIndex GitHub. API surface and connector counts reflect framework versions current as of April 2026 and are subject to change.