Practical RAG chunking strategies for reliable RAG pipelines

If you want your Retrieval-Augmented Generation (RAG) system to answer questions quickly and accurately, the first thing you need to get right is how you split the source material into chunks. Good RAG chunking strategies keep the vector store small enough for fast similarity search while preserving enough context for the language model to generate coherent answers.

In this post I’ll walk through why chunking matters, how to pick a size and overlap that works for text and tabular data, the Python libraries that make chunking painless, and how to measure chunk quality with RAGAS and other metrics. I’ll also share code snippets that I use in production, and point out the traps that cost me time and compute.

Why is chunking critical for RAG performance?

Chunking directly influences three core metrics of a RAG pipeline: latency, relevance, and hallucination risk. Smaller chunks mean more vectors, which inflates index size and search time. Larger chunks reduce the number of vectors but increase the chance that a retrieved passage contains unrelated material, prompting the generator to hallucinate. In my experience, a 10-15 % latency increase per 10 % growth in vector count is a real concern when you serve hundreds of requests per second behind FastAPI.

Chunking also determines how well the retriever can surface the exact piece of information a query needs. If a paragraph is split mid-sentence, the embedding may lose the semantic cue that ties the sentence to the query. Overlap mitigates this, but too much overlap duplicates work and wastes storage.

How do I choose the right chunk size and overlap?

The sweet spot depends on your document type, model context window, and the cost of embedding calls. Here’s the rule of thumb I follow:

Why those numbers? A 4096-token context window (the limit of many LLM APIs) leaves room for the user query and the system prompt. If you allocate ~300 tokens per chunk, you can retrieve 3-4 chunks without hitting the limit, which is enough for most answers. Overlap of ~10 % of the chunk size preserves sentence boundaries.

In practice I run a quick benchmark: embed a sample of 1 000 chunks at different sizes, record the average latency, and check the recall of a known ground-truth answer set. The size that gives the highest recall per millisecond is the one I lock in.

When not to use overlap

If you’re indexing millions of rows from a structured database, the extra storage from overlap can be prohibitive. In that case I switch to a row-level chunker that treats each row as an atomic unit and rely on column-level embeddings to capture relationships.

What chunking techniques work for text vs. tabular data?

Text chunking

For free-form text I usually start with a sentence splitter (NLTK, spaCy) and then roll up sentences until I reach the target token count. The code below uses LangChain’s RecursiveCharacterTextSplitter, which handles overlap automatically.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.document_loaders import TextLoader

loader = TextLoader("data/article.txt")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=300,          # tokens approx.
    chunk_overlap=40,        # keep a bit of context
    length_function=lambda txt: len(txt.split())
)

chunks = splitter.split_documents(documents)
print(f"Generated {len(chunks)} chunks")

If you need to respect markdown headings, set separator="\n# " and adjust the chunk_size accordingly. This prevents a heading from being split across two vectors, which would confuse the retriever.

Tabular chunking

Tabular data needs a different approach. Splitting rows arbitrarily can break the relational meaning. I group rows by a logical key (e.g., a customer ID) and then concatenate the selected columns into a single string before embedding.

import pandas as pd

df = pd.read_csv("sales.csv")
grouped = df.groupby("customer_id")

def make_chunk(group):
    # Join selected columns, keep it under 200 tokens
    text = " ".join(
        f"{col}: {val}" for col, val in group[["date", "product", "amount"]].itertuples(index=False, name=None)
    )
    return text[:1500]  # rough token limit

chunks = [make_chunk(g) for _, g in grouped]
print(f"Created {len(chunks)} customer chunks")

If you have a very wide table, consider embedding each column separately with LlamaIndex’s NodeParser and then combine the vectors at query time. This keeps the index size manageable while still allowing cross-column search.

Which Python libraries should I reach for when chunking?

LangChain

LangChain ships with a suite of splitters (CharacterTextSplitter, RecursiveCharacterTextSplitter, MarkdownHeaderTextSplitter). It also integrates directly with most embedding providers, so you can feed the chunks straight into a vector store.

Pros:

• Simple API, good docs.
• Built-in overlap handling.
• Works out-of-the-box with LangChain retrievers.

Cons:

• Limited to text; you have to roll your own tabular logic.

LlamaIndex (formerly GPT Index)

LlamaIndex treats any data source as a “node” and provides parsers for PDFs, CSVs, databases, and even HTML. Its SimpleNodeParser and SentenceSplitterNodeParser let you specify chunk size in characters or tokens and automatically add metadata.

from llama_index import SimpleDirectoryReader, GPTVectorStoreIndex, ServiceContext

documents = SimpleDirectoryReader("data/").load_data()
service_context = ServiceContext.from_defaults(chunk_size=300, chunk_overlap=30)

index = GPTVectorStoreIndex.from_documents(documents, service_context=service_context)

Pros:

• First-class support for structured data.
• Metadata propagation (source file, page number) is automatic.
• Tight integration with various vector stores.

Cons:

• Slightly heavier dependency graph.
• Some APIs still in beta, so expect breaking changes.

Other utilities

• Haystack: Provides PreProcessor with similar token-based chunking.
• sentence-transformers: If you need custom token counting, you can use its tokenizer directly.

In production I often combine LangChain for raw text ingestion and LlamaIndex for the tabular side, then push everything into a single Pinecone index. The hybrid approach keeps the codebase modular and lets me swap out one side without touching the other.

How do I evaluate chunk quality with RAGAS and other metrics?

Chunk quality isn’t just “does it fit the token limit?” It’s about whether the retrieved chunks enable the generator to answer correctly. RAGAS (Retrieval Augmented Generation Assessment Suite) gives you a set of quantitative scores: Faithfulness, Answer Relevance, Contextual Recall, and Contextual Precision.

A minimal RAGAS workflow looks like this:

from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness

# Assume you have a list of (query, ground_truth, retrieved_chunks) tuples
samples = [
    {
        "question": "What is the quarterly revenue for Q2 2024?",
        "ground_truth": "The revenue was $12.3M.",
        "contexts": ["...chunk 1...", "...chunk 2..."]
    },
    # more samples …
]

results = evaluate(
    samples,
    metrics=[answer_relevancy, faithfulness],
    llm="gpt-4o-mini",  # cheap model for evaluation
)

print(results.mean())

The output gives you a score between 0 and 1 for each metric. If your chunk size is too small, you’ll see Contextual Recall drop because the relevant sentence is split across two chunks. If overlap is missing, Faithfulness suffers as the model hallucinates missing connections.

Other sanity checks

1. Embedding similarity distribution – Plot the cosine similarity of a query against its top-k chunks. A long tail indicates the index is too noisy.
2. Chunk length histogram – Verify that you didn’t accidentally create a few mega-chunks that dominate the index size.
3. Round-trip test – For a set of known Q&A pairs, run the full RAG pipeline and check the exact match rate.

When a metric falls below a threshold (e.g., answer relevancy < 0.7), I go back to the splitter settings, adjust chunk_size or chunk_overlap, and rerun the evaluation. It’s an iterative loop, but the cost is far lower than debugging a production outage where the model returns unrelated answers.

When not to over-engineer chunking

If you’re building a proof-of-concept that only handles a few hundred documents, the simplest splitter (split on newline) is fine. Adding complex overlap logic or custom parsers adds maintenance overhead that outweighs performance gains. Also, if your downstream model runs on a local GPU with a 32k context window, you can afford larger chunks and skip overlap altogether.

In a recent project I tried to micro-optimize chunk size for a dataset of 2 M product reviews. The engineering time spent tuning the splitter was longer than the latency savings we actually realized. The lesson: benchmark early, stop when marginal gains dip below the cost of extra code.

FAQ

What token count should I aim for when using OpenAI embeddings?
OpenAI’s text-embedding-3-large expects inputs under 8191 tokens. I keep chunks under 300 tokens to stay well within that limit and leave room for the query and system prompt in the generation step.

Can I reuse the same chunker for both text and CSV files?
Not directly. Text splitters work on strings, while CSV data usually needs row-level grouping. You can wrap a CSV reader in a small adapter that feeds each row group to the same RecursiveCharacterTextSplitter after converting it to a plain string.

How often should I re-evaluate chunk quality after the index is live?
At least once per major data refresh. If you add new documents, run the RAGAS suite on a sample of the new material. For high-traffic services, schedule a nightly sanity check that records similarity distribution; alert if the median drops more than 10 % from the baseline.

Do I need to store overlap tokens twice in the vector store?
Yes, each overlapped segment becomes its own vector. That duplicates some embeddings but guarantees that queries hitting the boundary still retrieve the full context. The storage cost is usually negligible compared to the overall index size.

Key Takeaways

• Chunking is the first performance gate in any RAG pipeline; bad chunks cause latency spikes and hallucinations.
• Start with 200-400 token chunks for prose, add 10-15 % overlap, and adjust based on embedding latency and recall benchmarks.
• Use LangChain for pure text, LlamaIndex for structured data, and keep the two pipelines separate but feed them into a common vector store.
• Validate chunk quality with RAGAS metrics; low answer relevance or faithfulness usually points to chunk size or overlap issues.
• Don’t over-engineer for small datasets - simple newline splitting often suffices.
• Re-evaluate after each data ingest or major model upgrade to keep the system reliable.

If you’re curious how these practices fit into a full FastAPI service, check out my recent posts on FastAPI vs Flask in 2026: Is Flask Finally Dead? and the guide on fixing async session errors in FastAPI, which both touch on the same production constraints that make good chunking essential.