RAG & GraphRAG

What is RAG and Why Does It Matter?

Retrieval-Augmented Generation (RAG) has become the gold standard for giving large language models access to specific, up-to-date information. Unlike fine-tuning, which bakes knowledge into the model’s weights, RAG retrieves relevant information on-demand and includes it in the prompt context.

Think of it this way:

Fine-tuning teaches the model new skills and behaviors RAG gives the model access to a dynamic knowledge base

This makes RAG ideal for:

• Customer support systems that need product documentation
• Internal knowledge bases with frequently changing information
• Research assistants accessing large document collections
• Q&A systems over proprietary data

The RAG Pipeline: How It Works

A production RAG system typically consists of 5 key components:

1. Document Processing

The first step is ingesting and preparing your documents:

from langchain.text_splitter import RecursiveCharacterTextSplitter

# Split documents into chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", " ", ""]
)

chunks = splitter.split_documents(documents)

Key considerations:

• Chunk size affects both retrieval accuracy and context limits
• Overlap ensures important information isn’t split
• Semantic chunking (by section/paragraph) often works better than fixed sizes

2. Embedding Generation

Each chunk gets converted to a vector representation:

from openai import OpenAI

client = OpenAI()

def get_embeddings(texts):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    return [item.embedding for item in response.data]

Popular embedding models:

• text-embedding-3-small (1536 dimensions, cost-effective)
• text-embedding-3-large (3072 dimensions, higher accuracy)
• Cohere embed-v3 (excellent multilingual support)

3. Vector Storage

Embeddings are stored in a vector database for fast similarity search:

4. Retrieval

When a user asks a question, we:

1. Convert the question to an embedding
2. Find similar chunks using vector search
3. Optionally re-rank results for better relevance

# Simple similarity search
results = vector_store.similarity_search(
    query="How do I reset my password?",
    k=5
)

# Advanced: Hybrid search (vector + keyword)
results = vector_store.search(
    query="password reset",
    search_type="hybrid",
    k=5,
    alpha=0.5  # Balance between vector and keyword
)

5. Generation

Finally, retrieved chunks are added to the LLM prompt:

prompt = f"""Answer the question based on the following context:

Context:
{retrieved_chunks}

Question: {user_question}

Answer:"""

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": prompt}]
)

When RAG Isn’t Enough: Enter GraphRAG

Traditional RAG works great for direct fact retrieval, but struggles with:

• Multi-hop questions (“Who is the CEO of the company that acquired Instagram?”)
• Thematic queries (“What are the main challenges discussed across all documents?”)
• Relationship-heavy domains (organizational charts, knowledge graphs)

What is GraphRAG?

GraphRAG enhances traditional RAG by building a knowledge graph from your documents, capturing:

• Entities (people, organizations, concepts)
• Relationships (works for, located in, related to)
• Communities (clusters of related entities)

This enables:

# Traditional RAG: Returns chunks mentioning "CEO"
results = rag.search("Who is the CEO?")

# GraphRAG: Traverses relationships
results = graph_rag.search("Who reports to the CEO's direct reports?")
# Returns: CEO → Directors → Managers (multi-hop traversal)

GraphRAG Architecture

GraphRAG adds two key components:

1. Knowledge Graph Construction

from langchain.graphs import Neo4jGraph

# Extract entities and relationships
extractor = EntityRelationshipExtractor(llm=llm)
graph_data = extractor.process_documents(documents)

# Store in graph database
graph = Neo4jGraph(url="bolt://localhost:7687")
graph.add_graph_documents(graph_data)

2. Community Detection

GraphRAG identifies communities (clusters) in your knowledge graph:

“By identifying communities of related entities, we can answer questions about themes and patterns that span multiple documents.”

This enables global queries like:

• “What are the main research themes?”
• “Which projects are related to AI safety?”
• “Summarize the company’s strategic initiatives”

RAG vs GraphRAG: Decision Matrix

Here’s when to use each approach:

Use Traditional RAG When:

• ✅ Queries are fact-based and direct
• ✅ Documents are independent (not highly interconnected)
• ✅ Fast implementation is priority
• ✅ Budget is limited (GraphRAG requires more compute)

Examples: Product docs, FAQ systems, simple Q&A

Use GraphRAG When:

• ✅ Queries require multi-hop reasoning
• ✅ Documents have rich relationships
• ✅ Need thematic analysis across corpus
• ✅ Dealing with structured knowledge domains

Examples: Research papers, organizational knowledge, legal documents

Consider Hybrid (Both) When:

• ✅ Need both specific facts AND thematic insights
• ✅ Large, complex knowledge base
• ✅ Budget allows for comprehensive solution

Production Implementation Checklist

Ready to build a RAG system? Here’s our battle-tested checklist:

Phase 1: Foundation (Week 1-2)

• Document ingestion pipeline

  • Support for PDF, DOCX, HTML, markdown
  • Metadata extraction (author, date, source)
  • Error handling and retry logic

• Chunking strategy

  • Test different chunk sizes (500, 1000, 1500 tokens)
  • Implement semantic chunking for long documents
  • Add chunk metadata (document title, section)

• Embedding generation

  • Select embedding model (start with text-embedding-3-small)
  • Implement batch processing for efficiency
  • Cache embeddings to avoid recomputation

• Vector database setup

  • Choose database (recommend Pinecone for MVP)
  • Design collection schema with metadata
  • Set up monitoring and backups

Phase 2: Retrieval Quality (Week 2-3)

• Retrieval evaluation

  • Create test dataset (20-30 question-answer pairs)
  • Measure retrieval accuracy (MRR, NDCG)
  • Tune k (number of results) parameter

• Query enhancement

  • Implement query rewriting for clarity
  • Add query expansion (synonyms, related terms)
  • Test hybrid search (vector + keyword)

• Re-ranking

  • Add cross-encoder re-ranker (Cohere, Jina)
  • Measure impact on answer quality
  • Balance latency vs accuracy

Phase 3: Generation & Polish (Week 3-4)

• Prompt engineering

  • Design system prompt for your domain
  • Add citation requirements
  • Test different instruction styles

• Response generation

  • Implement streaming for better UX
  • Add source citations (chunk IDs)
  • Handle “I don’t know” cases

• Observability

  • Log queries, retrievals, and responses
  • Track latency at each stage
  • Monitor embedding and LLM costs

Common Pitfalls and Solutions

Pitfall #1: Poor Chunking

Problem: Chunks split in the middle of important information

Solution: Use semantic chunking based on document structure:

# Bad: Fixed-size chunking
chunks = split_every_1000_chars(document)

# Good: Semantic chunking
chunks = split_by_sections_and_paragraphs(document)

Pitfall #2: Irrelevant Retrievals

Problem: Retrieved chunks don’t actually answer the question

Solutions:

1. Re-ranking: Add a cross-encoder to re-score results
2. Hybrid search: Combine vector and keyword search
3. Metadata filtering: Filter by document type, date, or other metadata
4. Query expansion: Rewrite queries to capture user intent

Pitfall #3: Hallucinations Despite RAG

Problem: Model still makes up information even with retrieved context

Solutions:

# Add explicit instruction
prompt = f"""CRITICAL: Only use information from the context below.
If the answer is not in the context, say "I don't have enough information."

Context:
{retrieved_chunks}

Question: {question}"""

# Verify citations
response = llm.generate(prompt)
verify_citations_exist(response, retrieved_chunks)

Pitfall #4: Scaling Issues

Problem: Performance degrades as document count grows

Solutions:

• Namespace/partition data by category (e.g., separate collections per product)
• Use metadata filtering to reduce search space
• Pre-filter with keyword search before vector search
• Upgrade vector database tier for better performance

Advanced RAG Techniques

Once you have a basic RAG system working, consider these enhancements:

1. Contextual Retrieval

Add a summary or context to each chunk:

for chunk in chunks:
    context = f"""Document: {chunk.document_title}
Section: {chunk.section}
Previous content: {previous_chunk_summary}"""

    chunk.embedding = embed(f"{context}\n\n{chunk.content}")

This helps the model understand chunk context even when retrieved in isolation.

2. Query Routing

Route different query types to specialized retrievers:

def route_query(query):
    if is_factual(query):
        return vector_search(query)
    elif requires_computation(query):
        return sql_search(query)
    elif needs_multi_hop(query):
        return graph_search(query)

3. Iterative Retrieval

Retrieve, analyze, then retrieve again if needed:

# First retrieval
initial_results = retrieve(query)

# Check if more info needed
if needs_more_context(initial_results, query):
    followup_query = generate_followup_query(initial_results, query)
    additional_results = retrieve(followup_query)
    results = merge(initial_results, additional_results)

Cost Optimization

RAG systems can get expensive at scale. Here’s how to optimize:

Embedding Costs

• Batch processing: Embed 100+ chunks at once
• Smaller models: text-embedding-3-small is 4x cheaper than large
• Cache embeddings: Never re-embed the same content
• Incremental updates: Only embed new/changed documents

Cost comparison (1M tokens):

• text-embedding-3-small: $0.02
• text-embedding-3-large: $0.13
• Cohere embed-v3: $0.10

LLM Costs

• Smaller context windows: Use fewer retrieved chunks
• Cheaper models: Try GPT-3.5 or Claude Haiku for simple queries
• Prompt caching: Reuse cached system prompts (Anthropic Claude)
• Streaming: Better UX, but same cost

Vector Database Costs

• Right-size storage: Don’t over-provision
• Use quantization: 8-bit quantization saves 75% storage
• Archive old data: Move inactive documents to cold storage
• Choose wisely: Pinecone serverless is cheaper for variable load

Real-World Case Study

Client: B2B SaaS company with 10,000+ help articles Challenge: Support team spent 60% of time finding information Solution: RAG-powered internal knowledge assistant

Results:

• ⏱️ Response time: 15 min → 2 min (87% reduction)
• 📉 Support tickets: -35% (users self-serve with AI assistant)
• 💰 ROI: 300% in first year (saved 2 FTE worth of time)
• 🎯 Accuracy: 94% of AI-generated responses were correct

Key success factors:

1. Invested in chunking strategy: Spent 2 weeks optimizing
2. Continuous evaluation: Weekly review of low-confidence answers
3. User feedback loop: Thumbs up/down improved retrieval
4. Metadata rich: Added product, feature, and version tags

Getting Started with Suvegasoft

Ready to implement RAG or GraphRAG for your organization?

We offer three engagement models:

1. POC/MVP (4-6 weeks)

• Architecture design
• Basic RAG pipeline
• Evaluation framework
• Demo application

2. Production Implementation (8-12 weeks)

• Full-featured RAG system
• Production deployment
• Monitoring and observability
• Documentation and training

3. GraphRAG Advanced (12-16 weeks)

• Knowledge graph construction
• Multi-hop reasoning
• Community detection
• Hybrid RAG + GraphRAG system

All engagements include:

• ✅ Source code and full ownership
• ✅ Architecture documentation
• ✅ Team training and knowledge transfer
• ✅ 3 months post-launch support

Next Steps

Want to dive deeper? Check out these resources:

• 🛠️ Vector Database Comparison - Choosing the right database

• 📖 Glossary: RAG - Quick reference

Have questions about implementing RAG for your use case? Book a consultation and we’ll help you design the right solution.