Skip to content

Extracting masked text

extract_text returns a folder’s text with detected identifiers replaced by mask characters, split into chunks, each with a token count.

text_for_a_pipeline.py
"""Get a folder's text with the identifiers covered over, ready to send on."""
import complydoc as cd
result = cd.extract_text("src/complydoc/sample", ocr=False, max_tokens=1000)
for chunk in result.chunks:
print(f"{chunk.document} p{chunk.page}: {chunk.tokens} tokens ({chunk.token_fidelity})")
print(f" {chunk.masked} masked, {chunk.masked_confirmed} of them checksum-backed")
print(f"\n{result.tokens} tokens in total across {len(result.chunks)} chunks")
# Read these before using the text. `complete` is false when something could
# not be read; the best-effort warning is there every time masking runs.
if not result.complete:
print("\nsomething is missing:")
for warning in result.warnings:
where = f"{warning.document or 'run'}" + (f" p{warning.page}" if warning.page else "")
print(f" [{warning.kind}] {where}: {warning.detail}")
employee-record.pdf p1: 191 tokens (approximate)
15 masked, 8 of them checksum-backed
financial-summary.pdf p1: 74 tokens (approximate)
0 masked, 0 of them checksum-backed
[unreadable_page] invoice-scan.pdf p1: nothing could be read off this page; OCR was not run
[masking_best_effort] run: organisation name, person name are recognised by a statistical
model, so some will have been missed

result.warnings is a list of ExtractionWarning. Five kinds:

Warning Means
masking_best_effort Raised every time masking runs. Names have no checksum, so a model finds them and models miss
masking_incomplete A category could not be scanned at all — none of that kind were covered, anywhere
unreadable_page Nothing could be read off a page. Usually a scan with ocr=False
unreadable_document A file would not open. None of its content is here
estimated_tokens No local encoding, so counts are a character estimate

One chunk per page, unless max_tokens is set. A page over the budget is split at paragraph breaks; a single paragraph exceeding the budget is emitted whole.

chunk.token_fidelity is exact when the count came from the model’s own encoding, approximate from another provider’s encoding, and estimated when no encoding was available and the count is characters divided by four.

The report truncates page text at 20,000 characters. extract_text loads and scans documents directly, so the text it returns is not truncated.

For a format complydoc does not handle:

bring_your_own_loader.py
"""Teach complydoc a format it does not handle."""
from pathlib import Path
import complydoc as cd
class MarkdownLoader:
extensions = (".md",)
format = cd.DocumentFormat.OTHER
def load(self, path: Path, options: cd.IngestOptions) -> cd.Document:
document = cd.Document(path=path, sha256=cd.sha256_of(path), format=self.format)
page = cd.Page(number=1, width_pt=595.0, height_pt=842.0)
page.text = path.read_text(encoding="utf-8")
page.text_source = "native"
document.pages.append(page)
return document
cd.register_loader(MarkdownLoader())
print(".md is now readable:", ".md" in cd.supported_extensions())

Discovery, the scan, masking, the signals and the report then treat it as any other document. register_extractor registers a reader for a PDF text layer; register_engine registers an OCR engine.