Skip to main content

LlamaIndex and Knowledge Graphs: Retrieval-Augmented Generation

LlamaIndex is an open-source framework that simplifies the process of connecting large language models to external knowledge sources through retrieval-augmented generation (RAG). Knowledge graphs extend this by modeling relationships between entities, enabling semantic search that understands context, not just keyword matching. Teams using LlamaIndex + knowledge graphs report 45–60% improvements in answer relevance and 35–50% reductions in hallucination rates compared to LLMs without grounding.

Why This Matters Now​

Raw language models have a fundamental limitation: their knowledge was frozen at training time, and they often hallucinate plausible but false information. The solution is retrieval-augmented generation (RAG): you index your documents, retrieve the most relevant passages when a user asks a question, and feed those passages to the LLM so it answers grounded in real evidence.

LlamaIndex automates the messy parts of RAG:

  • Indexing: Breaking documents into chunks, embedding them, and storing them efficiently
  • Retrieval: Finding relevant context from millions of documents in milliseconds
  • Ranking: Re-ranking results to surface the most relevant passages first
  • Integration: Connecting retrieval seamlessly to your LLM application

Knowledge graphs add semantic structure: instead of treating documents as flat text, you model entities (people, companies, concepts) and their relationships. This enables richer queries like "show me all customers in the legal industry who bought our enterprise product in 2024" instead of keyword-only search.

Core Concepts: Document Indexing and Retrieval​

The RAG Pipeline​

A typical RAG system has three stages:

  1. Indexing (offline, run once or periodically):

    • Read your documents (PDFs, web pages, databases)
    • Split them into chunks (300–1000 tokens, with overlap to preserve context)
    • Embed each chunk into a vector (semantic representation)
    • Store vectors + metadata in a vector database

  2. Retrieval (at query time):

    • Embed the user's query using the same embedding model
    • Search the vector database for chunks similar to the query
    • Retrieve top-K (typically 3–5) most relevant chunks

  3. Generation:

    • Feed the user's question + retrieved chunks to an LLM
    • LLM generates an answer grounded in the retrieved evidence
    • Output includes citations linking claims back to source documents

Why Embeddings Matter​

Embeddings convert text into high-dimensional vectors. Text with similar meaning have vectors close together in space. This enables semantic search—finding chunks by meaning, not just keyword matching.

Example:

Query: "What are the side effects of Aspirin?"
Keyword match would only find documents containing "side effects" + "Aspirin"
Semantic match finds documents about adverse reactions, contraindications, safety
profiles—all semantically related to "side effects" even if the words differ

Modern embedding models (e.g., OpenAI's text-embedding-3, Anthropic's embeddings) are trained to recognize these semantic relationships.

Getting Started with LlamaIndex​

Installation and Setup​

# Install LlamaIndex
pip install llama-index

# Install supporting libraries
pip install llama-index-embeddings-openai  # or your preferred embedding provider
pip install llama-index-vector-stores-pinecone  # or Weaviate, Qdrant, etc.
pip install pypdf  # for PDF parsing

A Minimal RAG Example​

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

# Step 1: Load documents
documents = SimpleDirectoryReader(input_dir="./docs").load_data()

# Step 2: Create index (embeds documents and stores vectors)
embed_model = OpenAIEmbedding(model="text-embedding-3-small")
llm = OpenAI(model="gpt-4")

index = VectorStoreIndex.from_documents(documents, embed_model=embed_model)

# Step 3: Create a query engine
query_engine = index.as_query_engine(llm=llm)

# Step 4: Ask a question
response = query_engine.query("What is the company's return policy?")
print(response)

This minimal example:

  1. Loads all .pdf and .txt files from ./docs/
  2. Embeds each chunk using OpenAI's embedding model
  3. Stores vectors in an in-memory index
  4. Retrieves relevant chunks for the query
  5. Feeds the question + chunks to GPT-4 to generate an answer

Production-Grade Setup: Custom Vector Store and Persistence​

For production, replace the in-memory index with a persistent vector database:

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.pinecone import PineconeVectorStore
import pinecone

# Initialize Pinecone (scales to billions of vectors)
pinecone.init(api_key="your-api-key", environment="prod-1")
pinecone_index = pinecone.Index("documents-index")

# Create vector store
vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Index documents and persist
documents = SimpleDirectoryReader(input_dir="./docs").load_data()
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    embed_model=embed_model
)

# Later: Load the index without re-embedding
index = VectorStoreIndex.from_existing_index(
    pinecone_index,
    storage_context=storage_context,
    embed_model=embed_model
)

Knowledge Graphs: Modeling Entity Relationships​

A knowledge graph represents entities (nouns: people, places, companies, concepts) and relationships (verbs: "works at", "founded", "is located in"). This structure enables richer queries.

Simple Knowledge Graph Example​

Entity: Apple Inc.
  - Founded: 1976
  - Founder: Steve Jobs
  - Headquarters: Cupertino, CA
  - Products: iPhone, Mac, iPad

Relationships:
  - Apple.founder → Steve Jobs
  - Apple.headquarters → Cupertino, CA
  - iPhone.manufacturer → Apple Inc.
  - Steve Jobs.founded_company → Apple Inc.

A knowledge graph query like "Show me companies founded by Steve Jobs" traverses these relationships:

  1. Find entity "Steve Jobs"
  2. Follow founded_company relationship
  3. Return "Apple Inc.", "NeXT Computer", "Pixar"

Building a Knowledge Graph with LlamaIndex​

LlamaIndex can extract entities and relationships from documents and build a knowledge graph automatically:

from llama_index.core import KnowledgeGraphIndex
from llama_index.core import SimpleLLM

# Create a knowledge graph index
kg_index = KnowledgeGraphIndex.from_documents(
    documents,
    llm=llm,
    storage_context=storage_context,
    embed_model=embed_model
)

# Query the knowledge graph
response = kg_index.query(
    "What companies were founded by entrepreneurs in Silicon Valley?",
    include_text=True  # Include source document text in response
)

# The LLM traverses the knowledge graph to find relevant entities and relationships
print(response)

Hybrid Search: Combining Vector and Knowledge Graph Retrieval​

For better results, combine vector search (semantic similarity) with knowledge graph search (entity relationships):

from llama_index.core import KnowledgeGraphQueryEngine
from llama_index.core import RetrieverQueryEngine

# Create two retrievers
vector_retriever = index.as_retriever(similarity_top_k=5)
kg_retriever = kg_index.as_retriever()

# Create a hybrid query engine
hybrid_engine = RetrieverQueryEngine(
    retriever=vector_retriever,
    knowledge_graph_retriever=kg_retriever,
    node_postprocessor=[...],  # Optional: re-rank results
)

response = hybrid_engine.query("What are Apple's main products and markets?")

This approach:

  1. Vector search finds all documents mentioning "Apple" and "products"
  2. Knowledge graph search finds entities related to Apple (iPhone, iPad, etc.)
  3. Hybrid engine combines both, providing more complete answers

Advanced Patterns: Chunking and Ranking​

Chunking Strategy​

How you split documents into chunks significantly impacts retrieval quality:

from llama_index.core import SimpleNodeParser

# Strategy 1: Fixed-size chunks with overlap
parser = SimpleNodeParser.from_defaults(
    chunk_size=512,  # tokens per chunk
    chunk_overlap=50  # tokens of overlap between chunks
)
nodes = parser.get_nodes_from_documents(documents)

# Strategy 2: Semantic chunking (chunks at sentence/paragraph boundaries)
from llama_index.core.node_parser import SemanticSplitterNodeParser

semantic_parser = SemanticSplitterNodeParser(
    buffer_size=1,  # number of sentences to buffer
    breakpoint_percentile_threshold=95,  # sensitivity: higher = larger chunks
)
semantic_nodes = semantic_parser.get_nodes_from_documents(documents)

# Strategy 3: Hierarchical chunking (multi-level)
# Small chunks for detail, larger chunks for context
from llama_index.core.node_parser import HierarchicalNodeParser

hierarchical_parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[512, 1024, 2048],  # three levels
)
hierarchical_nodes = hierarchical_parser.get_nodes_from_documents(documents)

Guidance:

  • Start with fixed-size chunks (512 tokens, 50 overlap)
  • If results feel fragmented, try semantic chunking
  • For very long documents, use hierarchical chunking

Re-Ranking Retrieved Results​

The initial retrieval returns K chunks, but not all are equally relevant. Re-rankers improve quality by scoring each chunk and re-ordering:

from llama_index.core.postprocessor import SentenceTransformerRerank

reranker = SentenceTransformerRerank(
    model="cross-encoder/ms-marco-MiniLM-L-12-v2",  # Fast, accurate
    top_n=3,  # Return top 3 after re-ranking (was 5 from vector search)
)

# Use in query engine
query_engine = index.as_query_engine(
    node_postprocessors=[reranker],
    similarity_top_k=5,  # Retrieve top 5 initially
)

# Re-ranker filters to top 3 before feeding to LLM
response = query_engine.query("What are our service guarantees?")

Re-ranking trades a small amount of latency (re-ranking is fast) for significantly better accuracy.

Production Considerations: Monitoring and Evaluation​

Measuring RAG Quality​

Track these metrics in production:

# Relevance: Are retrieved chunks relevant to the query?
def measure_relevance(query, retrieved_chunks, human_relevance_labels):
    relevant_count = sum(
        1 for chunk in retrieved_chunks 
        if human_relevance_labels.get(chunk.id, False)
    )
    recall = relevant_count / len([c for c in human_relevance_labels.values() if c])
    return recall

# Grounding: Does the answer cite the retrieved sources?
def measure_grounding(answer, retrieved_chunks):
    cited_chunks = extract_citations(answer)
    grounded_count = len(cited_chunks)
    total_claims = count_factual_claims(answer)
    grounding_ratio = grounded_count / total_claims
    return grounding_ratio

# Hallucination: Does the answer contain facts not in the retrieved chunks?
def measure_hallucination(answer, retrieved_chunks):
    external_facts = extract_facts_not_in_chunks(answer, retrieved_chunks)
    hallucination_rate = len(external_facts) / count_factual_claims(answer)
    return hallucination_rate

Run these metrics on a small test set weekly. Regressions in relevance or grounding signal that your index or retrieval strategy needs adjustment.

Handling Retrieval Failures​

Sometimes no relevant chunks exist (out-of-domain question) or all chunks are poor quality. Design graceful failure:

def safe_rag_query(query, query_engine, threshold=0.5):
    """
    Query with a confidence threshold. Refuse low-confidence answers.
    """
    response = query_engine.query(query)
    
    # Check if top retrieved chunk is similar enough
    retrieval_score = response.source_nodes[0].score if response.source_nodes else 0
    
    if retrieval_score < threshold:
        return {
            "answer": "I don't have relevant information to answer this question.",
            "confidence": "low",
            "suggestion": "Please refine your question or contact support."
        }
    
    return {
        "answer": response.response,
        "confidence": "high",
        "sources": [node.source_node.metadata["source"] for node in response.source_nodes]
    }

This prevents answering questions without evidence, reducing hallucination in production.

Try This Yourself​

Exercise 1: Index Your Own Documents (20 minutes)​

  1. Create a folder with 3–5 .pdf or .txt files (any topic: your blog posts, documentation, research papers)
  2. Run the minimal RAG example above
  3. Ask 3 test questions: one easy (exact match), one moderate (synonym search), one hard (inferential)
  4. Evaluate: Does it retrieve the right chunks? Does the answer feel grounded?

Exercise 2: Build a Knowledge Graph (30 minutes)​

  1. Take a document about your domain (e.g., company handbook, product docs)
  2. Create a knowledge graph using LlamaIndex
  3. Write 3 entity-relationship queries (e.g., "What departments report to the CEO?")
  4. Compare knowledge graph results to vector search results

Which retrieval method was better for each query type?

Key Takeaways​

  • RAG solves hallucination: Grounding LLM outputs in retrieved documents dramatically reduces false information and improves user trust.
  • LlamaIndex automates the plumbing: Indexing, vector storage, retrieval, and ranking are handled by the framework so you focus on application logic.
  • Knowledge graphs add semantic richness: Modeling entity relationships enables richer queries that pure keyword or semantic search cannot answer.
  • Chunking and re-ranking matter: Small details in document splitting and result ranking significantly impact final answer quality; test and measure.
  • Monitoring is essential: Track retrieval relevance, grounding ratio, and hallucination rate weekly; regressions indicate your index or retrieval strategy needs adjustment.

Frequently Asked Questions​

What embedding model should I use?​

For general use, OpenAI's text-embedding-3-small (cost-effective) or text-embedding-3-large (higher quality) are reliable defaults. For privacy-sensitive data, use open-source models (e.g., all-MiniLM-L6-v2). Never use different embedding models for indexing vs. retrieval; they must match.

How many documents can I index?​

In-memory indexes (good for learning) handle up to ~1,000 documents. For production, use a vector database (Pinecone, Weaviate, Qdrant). Pinecone and Weaviate scale to millions of documents and support real-time indexing.

Should I use vector search, knowledge graphs, or both?​

Start with vector search—it's simple and effective. Add knowledge graphs if your data is highly structured (entities and relationships are clear) and you have entity-relationship queries. For most applications, hybrid search (both) provides the best results.

How do I update my index when documents change?​

Use incremental indexing: only re-embed and re-index changed documents, not the entire corpus. Most vector databases support delta updates efficiently.

What if a user asks an out-of-domain question?​

Set a relevance threshold (e.g., retrieval score > 0.5). If no chunk exceeds the threshold, refuse to answer and suggest rephrasing or contacting support. This prevents hallucination on questions your index doesn't contain relevant information for.

Further Reading​

LlamaIndex and knowledge graphs transform LLMs from isolated models that hallucinate into grounded systems that answer questions with evidence. Start simple, measure quality, and iterate.