Inspecting chunks
cd.inspect_chunks(splitter, documents) splits documents and reports on the
chunks. cd.compare_chunkers({name: splitter}, documents) does the same for
several splitters.
"""Compare two ways of chunking the sample documents."""
import complydoc as cd
text = cd.extract_text("src/complydoc/sample", ocr=False, mask=False)documents = [ {"page_content": chunk.text, "metadata": {"source": chunk.document, "page": chunk.page - 1}} for chunk in text.chunks]
def by_paragraph(docs): return [ {"page_content": part, "metadata": doc["metadata"]} for doc in docs for part in doc["page_content"].split("\n\n") if part.strip() ]
def by_line(docs): return [ {"page_content": line, "metadata": doc["metadata"]} for doc in docs for line in doc["page_content"].splitlines() if line.strip() ]
comparison = cd.compare_chunkers( {"paragraph": by_paragraph, "line": by_line}, documents, facts=["Two administrator accounts have no multi-factor authentication"],)for row in comparison.rows(): print( row["chunker"], row["chunks"], "chunks; tiny:", row["tiny"], "split sentences:", row["split_sentence"], "facts split:", row["facts_split"], )
report = comparison.reports["paragraph"]print(report.stats)print(report.repeated_identifiers)splitter |
Called as |
|---|---|
| LangChain text splitter | split_documents(documents) |
| LlamaIndex node parser | get_nodes_from_documents(documents) |
| Callable | splitter(documents) |
A LlamaIndex node parser is given LlamaIndex documents, made from whatever
documents holds.
documents is anything inspect_documents accepts, including a loader. With
documents omitted, splitter is taken to be chunks that were already made.
Per chunk
Section titled “Per chunk”report.chunks holds an InspectedChunk per chunk: source document and page,
characters, tokens (from model’s encoding, or the headline model’s),
identifiers as masked values, the number of hidden passages of medium severity or
above, metadata keys, a masked preview, and flags:
| Flag | Set when |
|---|---|
tiny |
Fewer than min_tokens tokens (20) |
oversized |
More than max_tokens tokens, when given |
split_sentence |
The chunk ends without closing punctuation and the next chunk of the same document starts in lowercase |
split_table |
The last line and the next chunk’s first line both look like table rows |
heading_at_end |
The last line is a Markdown heading or a short line in capitals |
duplicate |
The same text, ignoring whitespace, appeared in an earlier chunk |
path_metadata |
A metadata value is an absolute file path |
Per splitter
Section titled “Per splitter”report.stats: count, total, minimum, median, 95th percentile and maximum tokensreport.flag_counts: chunks per flagreport.repeated_identifiers: identifiers found in more than one chunkreport.facts: aFactLocationper expected fact, with statuswhole(inside one chunk, listed inchunks),split(only in the joined document text, so a chunk boundary cuts it) ormissing
report.to_pandas() returns one row per chunk; comparison.to_pandas() one row
per splitter.
Retrieval check
Section titled “Retrieval check”questions pairs each question with a passage that answers it. Every question is
ranked against every chunk with BM25, a keyword ranking built in, with no
embedding model or index:
"""Check whether questions retrieve the chunk that answers them, for two chunkers."""
import complydoc as cd
text = cd.extract_text("src/complydoc/sample", ocr=False, mask=False)documents = [ {"page_content": chunk.text, "metadata": {"source": chunk.document, "page": chunk.page}} for chunk in text.chunks]
def by_page(docs): return list(docs)
def by_sentence(docs): return [ {"page_content": sentence, "metadata": doc["metadata"]} for doc in docs for sentence in doc["page_content"].split(". ") if sentence.strip() ]
comparison = cd.compare_chunkers( {"page": by_page, "sentence": by_sentence}, documents, questions=[ cd.Question( "Which accounts have no multi-factor authentication?", "Two administrator accounts have no multi-factor authentication", document="vendor-assessment.pdf", ), cd.Question( "When are invoices payable?", "Invoices are payable within", document="terms-and-conditions.pdf", ), ], top_k=3,)
for row in comparison.rows(): print(row["chunker"], row["chunks"], row["retrieval_hit_rate"], row["mean_reciprocal_rank"])
for result in comparison.reports["sentence"].retrieval: print(result.status, result.rank, result.question)| Status | Meaning |
|---|---|
retrieved |
A chunk holding the fact ranks within top_k (5) |
ranked_low |
A chunk holds the fact and ranks below top_k, or shares no word with the question |
split |
No single chunk holds the fact; the text before splitting does |
missing |
The text does not contain the fact |
A question is a cd.Question(text, fact, document=None), a mapping with
question, fact and document, or a (question, fact) pair. The fact is
matched as expected facts are, and document limits the answering chunks to one
file.
report.retrieval holds a QuestionResult per question: the rank of the first
chunk holding the fact, the chunks holding it, and the top_k best-ranked
chunks. report.retrieval_hit_rate is the share retrieved, and
report.mean_reciprocal_rank averages 1/rank over retrieved questions and 0 for
the rest. comparison.rows() has both per splitter.
A split or missing fact fails with any retriever. BM25 ranks by shared words, so a question worded differently from its answer ranks low here and may still be found by an embedding model. Use the ranks to compare splitters on the same questions.
From the command line
Section titled “From the command line”complydoc chunks ./contracts \ --splitter "langchain_text_splitters:RecursiveCharacterTextSplitter chunk_size=800" \ --splitter "langchain_text_splitters:RecursiveCharacterTextSplitter chunk_size=1500" \ --fact "Payment is due within thirty days" --max-tokens 1000Each page’s text is read as extract_text(mask=False) reads it and passed to the
splitter as documents with source and page metadata. --splitter takes a
module:attribute followed by key=value arguments, read as YAML values. A class is
created with the arguments; a function is called with the documents and the
arguments. Repeating --splitter compares them.
The command writes complydoc-chunks.json to --out (.complydoc by default),
which complydoc ui opens on its Chunks page. It holds masked previews and
identifiers, not the document text.
The JSON is a report like an audit’s, with the same run record, no per-document
entries, and each splitter’s ChunkReport under chunks. complydoc ui lists it
with the folder’s other runs and opens it on the Chunks page; cd.load_report
reads it back.
--questions questions.yaml runs the retrieval check, and --top-k sets top_k:
- question: Which accounts have no multi-factor authentication? fact: Two administrator accounts have no multi-factor authentication document: vendor-assessment.pdf- question: When are invoices payable? fact: Invoices are payable withinSeveral splitters at once
Section titled “Several splitters at once”complydoc chunks --preset common runs six splitters over a folder and compares
them: LangChain’s recursive splitter at 500, 1,000 and 2,000 characters, its
character and token splitters, and LlamaIndex’s sentence splitter. A splitter
whose library is not installed is left out, and the run says so. Add your own
with --splitter.