RAG Chunking & Keyword Scoring
See how text splits into chunks (sizes in characters), then rank chunks against a query with lexical BM25 and TF-IDF scoring. Keyword scoring only, not embeddings, so it cannot diagnose a dense-retrieval pipeline.
Rank chunks against a query with lexical BM25 and TF-IDF cosine similarity, matched terms highlighted. Not embeddings, and no stemming, so related word forms do not match.
Lexical scoring only: BM25 and TF-IDF on exact words, with no stemming, not embeddings. Production RAG usually retrieves with embeddings, so a good score here does not mean your vector search is working.
Compare fixed-size, sentence, paragraph and recursive chunking with adjustable size and overlap. Sizes are in characters; many RAG libraries size chunks in tokens.
Chunk size and overlap are in characters, not tokens. Many RAG libraries (LangChain, LlamaIndex) size chunks in tokens, so sizes here will not match theirs one to one.