Chunking Strategy for RAG Systems

Overview

Provides chunking strategies for RAG systems, vector databases, and document processing. Recommends chunk sizes, overlap percentages, and boundary detection methods; validates semantic coherence; evaluates retrieval metrics.

When to Use

Use when building or optimizing RAG systems, vector search pipelines, document chunking workflows, or performance-tuning existing systems with poor retrieval quality.

Instructions

Choose Chunking Strategy

Select based on document type and use case:

1. Fixed-Size Chunking (Level 1)

• Use for simple documents without clear structure
• Start with 512 tokens and 10-20% overlap
• Adjust: 256 for factoid queries, 1024 for analytical

1. Recursive Character Chunking (Level 2)

• Use for documents with structural boundaries
• Hierarchical separators: paragraphs → sentences → words
• Customize for document types (HTML, Markdown, JSON)

1. Structure-Aware Chunking (Level 3)

• Use for structured content (Markdown, code, tables, PDFs)
• Preserve semantic units: functions, sections, table blocks
• Validate structure preservation post-split

1. Semantic Chunking (Level 4)

• Use for complex documents with thematic shifts
• Embedding-based boundary detection with 0.8 similarity threshold
• Buffer size: 3-5 sentences

1. Advanced Methods (Level 5)

• Late Chunking for long-context models
• Contextual Retrieval for high-precision requirements
• Monitor computational cost vs. retrieval gain

Reference: references/strategies.md.

Implement Chunking Pipeline

1. Pre-process documents

• Analyze structure, content types, information density
• Identify multi-modal content (tables, images, code)

1. Select parameters

• Chunk size: embedding model context window / 4
• Overlap: 10-20% for most cases
• Strategy-specific settings

1. Process and validate

• Apply chunking strategy
• Validate coherence: run evaluate_chunks.py --coherence (see below)
• Test with representative documents

1. Evaluate and iterate

• Measure precision and recall
• If precision < 0.7: reduce chunk_size by 25% and re-evaluate
• If recall < 0.6: increase overlap by 10% and re-evaluate
• Monitor latency and memory usage

Reference: references/implementation.md.

Validate Chunk Quality

Run validation commands to assess chunk quality:

# Check semantic coherence (requires sentence-transformers)
python -c "
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
chunks = [...]  # your chunks
embeddings = model.encode(chunks)
similarity = (embeddings @ embeddings.T).mean()
print(f'Cohesion: {similarity:.3f}')  # target: 0.3-0.7
"

# Measure retrieval precision
python -c "
relevant = sum(1 for c in retrieved if c in relevant_chunks)
precision = relevant / len(retrieved)
print(f'Precision: {precision:.2f}')  # target: >= 0.7
"

# Check chunk size distribution
python -c "
import numpy as np
sizes = [len(c.split()) for c in chunks]
print(f'Mean: {np.mean(sizes):.0f}, Std: {np.std(sizes):.0f}')
print(f'Min: {min(sizes)}, Max: {max(sizes)}')
"

Reference: references/evaluation.md.

Examples

Fixed-Size Chunking

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=256,
    chunk_overlap=25,
    length_function=len
)
chunks = splitter.split_documents(documents)

Structure-Aware Code Chunking

import ast

def chunk_python_code(code):
    tree = ast.parse(code)
    chunks = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
            chunks.append(ast.get_source_segment(code, node))
    return chunks

Semantic Chunking

def semantic_chunk(text, similarity_threshold=0.8):
    sentences = split_into_sentences(text)
    embeddings = generate_embeddings(sentences)
    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = cosine_similarity(embeddings[i-1], embeddings[i])
        if sim < similarity_threshold:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])
    chunks.append(" ".join(current))
    return chunks

Best Practices

Core Principles

• Balance context preservation with retrieval precision
• Maintain semantic coherence within chunks
• Optimize for embedding model context window constraints

Implementation

• Start with fixed-size (512 tokens, 15% overlap)
• Iterate based on document characteristics
• Test with domain-specific documents before deployment

Pitfalls to Avoid

• Over-chunking: context-poor small chunks
• Under-chunking: missing information in oversized chunks
• Ignoring semantic boundaries and document structure
• One-size-fits-all for diverse content types

Constraints and Warnings

Resource Considerations

• Semantic methods require significant compute resources
• Late chunking needs long-context embedding models
• Complex strategies increase processing latency
• Monitor memory for large document batches

Quality Requirements

• Validate semantic coherence post-processing
• Test with representative documents before deployment
• Ensure chunks maintain standalone meaning
• Implement error handling for malformed content

References

• strategies.md - Detailed strategies
• implementation.md - Implementation guidelines
• evaluation.md - Performance metrics
• tools.md - Libraries and frameworks
• research.md - Research papers
• advanced-strategies.md - 11 advanced methods
• semantic-methods.md - Semantic approaches
• visualization-tools.md - Visualization tools