Extracting masked text¶

extract_text returns a folder's text with detected identifiers replaced by mask characters, split into chunks, each with a token count.

text_for_a_pipeline.py

"""Get a folder's text with the identifiers covered over, ready to send on."""

import complydoc as cd

result = cd.extract_text("src/complydoc/sample", ocr=False, max_tokens=1000)

for chunk in result.chunks:
    print(f"{chunk.document} p{chunk.page}: {chunk.tokens} tokens ({chunk.token_fidelity})")
    print(f"   {chunk.masked} masked, {chunk.masked_confirmed} of them checksum-backed")

print(f"\n{result.tokens} tokens in total across {len(result.chunks)} chunks")

# Read these before using the text. `complete` is false when something could
# not be read; the best-effort warning is there every time masking runs.
if not result.complete:
    print("\nsomething is missing:")
for warning in result.warnings:
    where = f"{warning.document or 'run'}" + (f" p{warning.page}" if warning.page else "")
    print(f"  [{warning.kind}] {where}: {warning.detail}")

employee-record.pdf p1: 191 tokens (approximate)
   15 masked, 8 of them checksum-backed
financial-summary.pdf p1: 74 tokens (approximate)
   0 masked, 0 of them checksum-backed

  [unreadable_page] invoice-scan.pdf p1: nothing could be read off this page; OCR was not run
  [masking_best_effort] run: organisation name, person name are recognised by a statistical
  model, so some will have been missed

Warnings¶

result.warnings is a list of ExtractionWarning. Five kinds:

Warning	Means
`masking_best_effort`	Raised every time masking runs. Names have no checksum, so a model finds them and models miss
`masking_incomplete`	A category could not be scanned at all — none of that kind were covered, anywhere
`unreadable_page`	Nothing could be read off a page. Usually a scan with `ocr=False`
`unreadable_document`	A file would not open. None of its content is here
`estimated_tokens`	No local encoding, so counts are a character estimate

Masking is best effort

Categories detected by the statistical model will be missed at some rate. On the shipped sample the model finds John Smith and does not find Jane Doe on the preceding line.

chunk.masked counts replacements. chunk.masked_confirmed counts the subset that passed a checksum. The difference is the model-detected part.

Chunks¶

One chunk per page, unless max_tokens is set. A page over the budget is split at paragraph breaks; a single paragraph exceeding the budget is emitted whole.

chunk.token_fidelity is exact when the count came from the model's own encoding, approximate from another provider's encoding, and estimated when no encoding was available and the count is characters divided by four.

Source of the text¶

The report truncates page text at 20,000 characters. extract_text loads and scans documents directly, so the text it returns is not truncated.

Registering a loader¶

For a format complydoc does not handle:

bring_your_own_loader.py

"""Teach complydoc a format it does not handle."""

from pathlib import Path

import complydoc as cd


class MarkdownLoader:
    extensions = (".md",)
    format = cd.DocumentFormat.OTHER

    def load(self, path: Path, options: cd.IngestOptions) -> cd.Document:
        document = cd.Document(path=path, sha256=cd.sha256_of(path), format=self.format)
        page = cd.Page(number=1, width_pt=595.0, height_pt=842.0)
        page.text = path.read_text(encoding="utf-8")
        page.text_source = "native"
        document.pages.append(page)
        return document


cd.register_loader(MarkdownLoader())
print(".md is now readable:", ".md" in cd.supported_extensions())

Discovery, the scan, masking, the signals and the report then treat it as any other document. register_extractor registers a reader for a PDF text layer; register_engine registers an OCR engine.