Extracting masked text
extract_text returns a folder’s text with detected identifiers replaced by
mask characters, split into chunks, each with a token count.
"""Get a folder's text with the identifiers covered over, ready to send on."""
import complydoc as cd
result = cd.extract_text("src/complydoc/sample", ocr=False, max_tokens=1000)
for chunk in result.chunks: print(f"{chunk.document} p{chunk.page}: {chunk.tokens} tokens ({chunk.token_fidelity})") print(f" {chunk.masked} masked, {chunk.masked_confirmed} of them checksum-backed")
print(f"\n{result.tokens} tokens in total across {len(result.chunks)} chunks")
# Read these before using the text. `complete` is false when something could# not be read; the best-effort warning is there every time masking runs.if not result.complete: print("\nsomething is missing:")for warning in result.warnings: where = f"{warning.document or 'run'}" + (f" p{warning.page}" if warning.page else "") print(f" [{warning.kind}] {where}: {warning.detail}")employee-record.pdf p1: 191 tokens (approximate) 15 masked, 8 of them checksum-backedfinancial-summary.pdf p1: 74 tokens (approximate) 0 masked, 0 of them checksum-backed
[unreadable_page] invoice-scan.pdf p1: nothing could be read off this page; OCR was not run [masking_best_effort] run: organisation name, person name are recognised by a statistical model, so some will have been missedWarnings
Section titled “Warnings”result.warnings is a list of ExtractionWarning. Five kinds:
| Warning | Means |
|---|---|
masking_best_effort |
Raised every time masking runs. Names have no checksum, so a model finds them and models miss |
masking_incomplete |
A category could not be scanned at all — none of that kind were covered, anywhere |
unreadable_page |
Nothing could be read off a page. Usually a scan with ocr=False |
unreadable_document |
A file would not open. None of its content is here |
estimated_tokens |
No local encoding, so counts are a character estimate |
Chunks
Section titled “Chunks”One chunk per page, unless max_tokens is set. A page over the budget is split
at paragraph breaks; a single paragraph exceeding the budget is emitted whole.
chunk.token_fidelity is exact when the count came from the model’s own
encoding, approximate from another provider’s encoding, and estimated when
no encoding was available and the count is characters divided by four.
Source of the text
Section titled “Source of the text”The report truncates page text at 20,000 characters. extract_text loads and
scans documents directly, so the text it returns is not truncated.
Registering a loader
Section titled “Registering a loader”For a format complydoc does not handle:
"""Teach complydoc a format it does not handle."""
from pathlib import Path
import complydoc as cd
class MarkdownLoader: extensions = (".md",) format = cd.DocumentFormat.OTHER
def load(self, path: Path, options: cd.IngestOptions) -> cd.Document: document = cd.Document(path=path, sha256=cd.sha256_of(path), format=self.format) page = cd.Page(number=1, width_pt=595.0, height_pt=842.0) page.text = path.read_text(encoding="utf-8") page.text_source = "native" document.pages.append(page) return document
cd.register_loader(MarkdownLoader())print(".md is now readable:", ".md" in cd.supported_extensions())Discovery, the scan, masking, the signals and the report then treat it as any
other document. register_extractor registers a reader for a PDF text layer;
register_engine registers an OCR engine.