Python API¶
Everything on this page is public. A breaking change to it changes the minor version. Anything in the package that is not on this page is internal.
Audits and inspection¶
async
¶
aiter_audit(
target: str | PathLike[str],
*,
components: Sequence[str] = COMPONENTS,
**options: Unpack[AuditOptions],
) -> AsyncIterator[DocumentReport | SkipRecord]
iter_audit for asynchronous code, reading each document in a worker thread.
check_facts(
report: AuditReport,
facts: Iterable[Fact | str],
*,
threshold: float = FUZZY_THRESHOLD,
) -> list[FactCheck]
Whether each fact appears in the report's extracted text.
The report must have been produced with extracted_text=True. Results are keyed
by the loader's name for loader output, and by report otherwise.
compare_chunkers(
chunkers: Mapping[str, Any],
documents: Any,
**options: Any,
) -> ChunkComparison
inspect_chunks for each named splitter on the same documents.
compare_loaders(
loaders: Mapping[str, Any] | Sequence[Any],
*,
config: Config | None = None,
components: Sequence[str] = COMPONENTS,
reveal: bool = False,
extracted_text: bool = True,
models: Sequence[str] | None = None,
allow_network: bool = False,
paths: str
| PathLike[str]
| Sequence[str | PathLike[str]]
| None = None,
facts: Iterable[Fact | str] | None = None,
fact_threshold: float = FUZZY_THRESHOLD,
cache_dir: str | PathLike[str] | None = None,
verify_with: str | VisionModel | None = None,
verify_scope: str = "flagged",
) -> AuditReport
Run several loaders on the same input and report where their output differs.
loaders maps a name to a loader, or is a sequence of loaders, each named
after its class. Anything inspect_documents accepts as a source is
accepted as a loader, including documents already loaded. The first is the
baseline the report's findings are built from.
The comparison is in report.loader_comparison. Each document's
extractions lists every loader's reading of it, first the baseline's, and
with extracted_text on, each page carries the other loaders' text in
readings.
allow_network applies to every loader. Each one's connections are
recorded in its row of report.loader_comparison.loaders.
With paths, a folder, a file or a list of files, each loader is a callable
taking a file path, such as a loader class or a complydoc.loaders.parsers preset, and
runs once per file. Files a loader raises on are recorded in its row.
cache_dir stores each loader's output per file when paths is given, so a later
run does not parse unchanged files again; see complydoc.loaders.cache.
facts are passages the documents are expected to contain. Each is checked
against every loader's text; see complydoc.extraction.facts.
verify_with reads the baseline's pages again with a vision model of the
caller's, as full_audit does, rendered from the files the loaders read.
The vision reading joins each page's readings, so it can be put beside any
loader's, with what each one cost.
cost_audit(
target: str | PathLike[str],
**options: Unpack[AuditOptions],
) -> AuditReport
Estimate what these documents would cost an LLM to read.
Per model and per architecture — the text layer, the text layer with OCR behind it, and every page sent as an image.
extract_text(
target: str | Path,
*,
config: Config | None = None,
mask: bool = True,
reveal: bool = False,
ocr: bool = True,
recurse: bool = True,
password: str = "",
extractor: str | None = None,
model: str | None = None,
max_tokens: int | None = None,
) -> TextResult
Read a folder's text, masked, with token counts and what went wrong.
ocr is on by default here, unlike the audit, so scanned pages are read.
mask=False returns the text as the page says it, with no scan run at all.
reveal=True runs the scan and reports the counts while leaving the values
in place.
full_audit(
target: str | PathLike[str],
**options: Unpack[AuditOptions],
) -> AuditReport
Audit a file or folder on every component.
Cost, readiness and the identifier scan, with the global readiness score and the quick wins that follow from all three.
inspect_chunks(
splitter: Any,
documents: Any = None,
*,
name: str | None = None,
config: Config | None = None,
model: str | None = None,
min_tokens: int = 20,
max_tokens: int | None = None,
facts: Iterable[Fact | str] | None = None,
fact_threshold: float = FUZZY_THRESHOLD,
questions: Iterable[
Question | Mapping[str, Any] | Sequence[str]
]
| None = None,
top_k: int = 5,
) -> ChunkReport
Split documents with splitter and inspect the chunks.
With documents omitted, splitter is taken to be the chunks themselves.
max_tokens enables the oversized flag. Token counts use model's encoding,
or the headline model's.
inspect_documents(
source: Any,
*,
name: str | None = None,
config: Config | None = None,
components: Sequence[str] = COMPONENTS,
reveal: bool = False,
extracted_text: bool = True,
models: Sequence[str] | None = None,
allow_network: bool = False,
) -> AuditReport
Report on the documents a loader produced.
source is a loader, a callable returning documents, a list of documents,
or a single document. A loader is run inside the network guard, and any
connection it attempts is recorded in report.loader.network_attempts; a
loader that fails because a connection was refused produces a report with
no documents and the failure in report.loader.error. Any other exception
a loader raises is not caught.
allow_network=True lets the loader's connections through, for loaders
that call a hosted service. Each lookup and connection is still recorded,
the report states that network access was allowed, and everything
complydoc does after loading stays behind the guard.
extracted_text is on by default here, unlike full_audit.
iter_audit(
target: str | PathLike[str],
*,
components: Sequence[str] = COMPONENTS,
**options: Unpack[AuditOptions],
) -> Iterator[DocumentReport | SkipRecord]
Yield each document's report entry as it is read, and each skipped file.
Takes the options of full_audit. The network guard is armed while a document
is read and released between documents. Raises FileNotFoundError for a missing
path when called.
readiness_audit(
target: str | PathLike[str],
**options: Unpack[AuditOptions],
) -> AuditReport
Measure how ready these documents are to extract data from.
report.documents[i].readiness.signals holds the signals. A signal that could
not be measured is reported as not applicable.
security_audit(
target: str | PathLike[str],
**options: Unpack[AuditOptions],
) -> AuditReport
Find personal and financial identifiers, and nothing else.
Values arrive masked. report.documents[i].sensitive.matches carries each
one with its severity and its evidence tier — confirmed where a
checksum passed, down to model for a statistical guess.
Strings¶
Tokens in text for model, or for the headline model when none is given.
find_hidden(
text: str,
*,
config: Config | None = None,
reveal: bool = False,
) -> list[ContentFinding]
Invisible characters and instruction-like passages in text.
A string has no rendering to check, so visibility is not_measured except for
characters that are invisible by definition, such as Unicode tag characters.
mask_text(
text: str, *, config: Config | None = None
) -> MaskedText
text with each identifier replaced, strongest evidence first.
scan_text(
text: str,
*,
config: Config | None = None,
reveal: bool = False,
) -> TextScan
Personal and financial identifiers in text, masked unless reveal.
Reports and tests¶
diff_reports(
old: AuditReport,
new: AuditReport,
*,
score_tolerance: float = 0.5,
) -> ReportDiff
Changes from old to new. Score changes smaller than score_tolerance are ignored.
Load the three config files from config_dir, defaulting to the shipped set.
load_report(
source: str | PathLike[str] | Mapping[str, Any],
) -> AuditReport
An AuditReport from a JSON file written by write_json, or from its parsed data.
Raises ValueError for a schema_version not in READABLE_SCHEMA_VERSIONS.
Fields missing from an older report take their default values.
write_chunks_html(
result: ChunkReport | ChunkComparison,
path: str | Path,
*,
source: str = "",
) -> Path
Write a chunk report or splitter comparison as one self-contained HTML file.
source names what was split, for the page heading. Previews and identifiers
are masked, as they are in the report.
write_diff_html(
diff: ReportDiff,
path: str | Path,
*,
old: str = "",
new: str = "",
) -> Path
Write the result of diff_reports as one self-contained HTML file.
old and new label the two reports in the heading, such as their file names.
write_html(
report: AuditReport,
path: str | PathLike[str],
*,
config: Config | None = None,
) -> Path
Write the report as one self-contained HTML file, and return where.
Pass the same config the audit used if it was not the default one: the
page echoes parts of it, so a report written against a different
configuration would describe settings that did not produce it.
write_json(
report: AuditReport,
path: str | PathLike[str],
*,
detail: Literal["summary", "full"] = "summary",
) -> Path
Write the report as JSON, and return where. Same shape the CLI writes.
detail="summary" leaves out the price of every document on every model and
the page geometry the HTML draws with, keeping the folder's cost per model.
detail="full" writes every field; use it for anything that reprocesses
reports, since a summary reads back without the parts it left out.
Pipeline steps¶
Bases: Step
Removes invisible characters and instruction-like sentences from the text.
With drop_document, a document containing either is dropped instead. This works
on text: hidden formatting in the source file is detected by an audit, not here.
Invisible characters include zero-width joiners, which some scripts use.
Base class: subclasses implement apply on one document's text and metadata.
The new text and metadata, or None to drop the document.
dataclass
¶
Extending¶
Bases: Protocol
dataclass
¶
Everything a detector is given about the category it is running for.
Bases: Protocol
Limit the engine's own threading, before it reads its first page.
Bases: Protocol
Reads the text layer of one page.
dataclass
¶
Bases: Protocol
What every format loader must provide.
dataclass
¶
How to build a loader for one file, and what running it involves.
instance-attribute
¶
Called with a file path; returns a loader or a list of documents.
class-attribute
instance-attribute
¶
True when the parser sends documents to a hosted service.
class-attribute
instance-attribute
¶
Entry under parsers in pricing.yaml, for the cost per page.
class-attribute
instance-attribute
¶
Framework and library names shown beside the loader in reports.
dataclass
¶
What a signal returns.
class-attribute
instance-attribute
¶
Supporting numbers, shown under the row so the value can be checked.
Bases: Protocol
Implemented by every module under readiness/signals/.
instance-attribute
¶
The default plain-English sentence. readiness.yaml can override it.
Presets for document parsers, for use with compare_loaders(..., paths=...).
import complydoc as cd
report = cd.compare_loaders(
{
"pypdf": PyPDFLoader,
"docling": cd.parsers.docling(),
"llamaparse": cd.parsers.llamaparse(tier="cost_effective"),
},
paths="./contracts",
allow_network=True,
)
Each preset returns a LoaderSpec: a factory that builds a loader for one file,
whether the parser sends documents to a hosted service, and the parsers entry in
pricing.yaml used to estimate its cost per page. The parser libraries are not
dependencies; each preset imports its library when it runs and says what to install
when it is missing.
A hosted preset only runs with allow_network=True.
dataclass
¶
How to build a loader for one file, and what running it involves.
instance-attribute
¶
Called with a file path; returns a loader or a list of documents.
class-attribute
instance-attribute
¶
True when the parser sends documents to a hosted service.
class-attribute
instance-attribute
¶
Entry under parsers in pricing.yaml, for the cost per page.
class-attribute
instance-attribute
¶
Framework and library names shown beside the loader in reports.
docling(
export: Literal["markdown", "chunks"] = "markdown",
**options: Any,
) -> LoaderSpec
Docling through langchain-docling, run locally.
export="markdown" returns one document per file; "chunks" returns Docling's
own chunks. options are passed to DoclingLoader.
unstructured(
*,
api: bool = False,
api_key: str | None = None,
strategy: Literal[
"fast", "hi_res", "ocr_only", "auto"
] = "fast",
**options: Any,
) -> LoaderSpec
Unstructured through langchain-unstructured.
api=True partitions through the hosted Unstructured API. options are passed
to UnstructuredLoader.
llamaparse(
*,
api_key: str | None = None,
tier: Literal[
"fast", "cost_effective", "agentic", "agentic_plus"
] = "cost_effective",
**options: Any,
) -> LoaderSpec
LlamaParse, hosted, through the llama-parse package.
tier selects the price entry; parsing options such as the mode are passed in
options to LlamaParse.
azure_document_intelligence(
*,
endpoint: str,
api_key: str,
model: str = "prebuilt-layout",
mode: Literal[
"markdown", "page", "single"
] = "markdown",
**options: Any,
) -> LoaderSpec
Azure AI Document Intelligence, hosted, through langchain-community.
Add an identifier detector to this process.
A category uses it when its detector is this detector's id; add the category
to sensitive.categories, for example with Config.override.
Class decorator. Instantiates the extractor once and registers it by id.
The same arrangement the signals use, so the two read alike.
Score passages with a classifier as well as the patterns.
classifier takes a passage and returns how likely it is to be an injected
instruction, from 0 to 1. A passage at or above
instructions.classifier_threshold in hidden.yaml is reported at the
model tier. Pass None to remove it.
complydoc ships and downloads no model. The classifier runs inside the
network guard, so it has to load from files already on the machine. It is
registered in this process only: audits with jobs above 1 run their
documents in worker processes that do not have it.
Add a readiness signal to every run in this process.
The signal is measured and shown unrated until readiness.signals.<id> is
configured, for example with Config.override.
Errors¶
Bases: RuntimeError
Raised when a configuration file is missing or fails validation.
Bases: AssertionError
A report did not meet an expectation.
Bases: RuntimeError
A file could not be opened or parsed. Always caught; never fatal to a run.
Bases: RuntimeError
Raised when any code attempts to open a network connection.
Bases: ValueError
A model was asked for by name and is not in the pricing config.
Results and types¶
Bases: TypedDict
Everything the four entry points accept beyond the folder itself.
One set for all of them. An option that does not apply to the components being run is ignored.
instance-attribute
¶
A loaded configuration. load_config() is used when this is absent.
instance-attribute
¶
Read pages with no text layer. Slower, and finds what a scan is hiding.
instance-attribute
¶
Put identifiers in the report in full. Off, and the report says which it was.
instance-attribute
¶
Price against these models instead of the configured comparison.
instance-attribute
¶
Audit this many documents, keeping each file type's share of the folder.
instance-attribute
¶
Seconds to give each document. A document still being read when the time passes is stopped and reported as skipped. Enforced by reading in a worker process, so a run with a timeout always uses one.
instance-attribute
¶
Keep the text read off each page. It is the document, so it is off by default.
instance-attribute
¶
Block outbound sockets for the duration. On, and only off for a caller who knows their process needs the network while this runs.
instance-attribute
¶
Called with (finished, total, path) as each document completes.
instance-attribute
¶
verify_with: VisionModel | str | None
Read pages again with a vision model of your own, and report where it disagrees.
A callable taking a VisionPage and returning a VisionReading or the
text, or a "vision:module:function" spec naming a factory for one. The
page images go wherever that code sends them, and the report names the
hosts. A callable keeps the run in one process; a spec crosses into
workers. Nothing is verified by default.
instance-attribute
¶
flagged, the default: the pages routing sent to vision, the pages with no
usable reading, and the pages two readers disagreed about. all: every page.
dataclass
¶
class-attribute
instance-attribute
¶
Printed in the report whenever a score is shown.
class-attribute
instance-attribute
¶
Echoed so the report can state how much of a value is shown.
class-attribute
instance-attribute
¶
Global readiness: content, cost and exposure combined.
Both this and quick_wins are derived from a finished report, so their
modules import this one. The annotations are resolved only by a type
checker, which keeps the dependency one-way at runtime.
class-attribute
instance-attribute
¶
What to do next, ranked. See complydoc.report.quickwins.
class-attribute
instance-attribute
¶
loader: LoaderRun | None = None
Set when the documents came from an external loader.
class-attribute
instance-attribute
¶
loader_comparison: LoaderComparison | None = None
Set by compare_loaders. loader is then the baseline's run.
class-attribute
instance-attribute
¶
Pages per route for the folder, and what that mix costs. See
complydoc.report.routing.
class-attribute
instance-attribute
¶
How many pages an independent vision read agreed with. None unless --verify.
One table of this report as a pandas DataFrame.
Table names are in complydoc.report.tables.TABLES. Requires the notebook extra.
dataclass
¶
dataclass
¶
One piece of a document's text, and what is known about it.
instance-attribute
¶
'exact', 'approximate' (a real encoding, another provider's) or 'estimated'.
instance-attribute
¶
How many of those passed a checksum.
The difference between the two is the part of the masking that is best effort. A chunk where they are equal had nothing guessed in it.
instance-attribute
¶
'native' where the text was on the page, 'ocr' where it was recognised.
dataclass
¶
dataclass
¶
class-attribute
instance-attribute
¶
Identifiers found in more than one chunk, with the number of chunks.
class-attribute
instance-attribute
¶
retrieval: list[QuestionResult] = field(
default_factory=list
)
One result per question, when questions were given.
property
¶
Share of questions retrieved within top_k. None without questions.
property
¶
Mean of 1/rank for questions retrieved within top_k, 0 for the rest.
dataclass
¶
dataclass
¶
What was written, and what was done to it.
class-attribute
instance-attribute
¶
Identifiers replaced by their masked form.
class-attribute
instance-attribute
¶
The subset of those that passed a checksum.
class-attribute
instance-attribute
¶
Metadata keys and parts taken out of the copy.
class-attribute
instance-attribute
¶
Categories nothing was looked for, so none of that kind were masked.
class-attribute
instance-attribute
¶
Each identifier covered over, with where it was.
class-attribute
instance-attribute
¶
What the copy does not cover, in the caller's own terms.
class-attribute
instance-attribute
¶
Set when no copy was written, with the reason.
dataclass
¶
A passage that is hidden from a reader, reads as an instruction to a model, or both.
instance-attribute
¶
visible, not_measured, suspected (one signal) or confirmed.
instance-attribute
¶
confirmed (decoded from hidden characters), pattern, model or none.
instance-attribute
¶
The passage, identifiers masked unless the run used reveal, cut at 240 characters.
class-attribute
instance-attribute
¶
The registered classifier's score, when there is one.
class-attribute
instance-attribute
¶
For loader output: whether the loader's text contains this passage.
dataclass
¶
Bases: StrEnum
class-attribute
instance-attribute
¶
For a third-party loader of a format complydoc does not know.
The registry is public, so a third party can teach complydoc a format without changing it. They need a name for what they are loading, and inventing enum members for formats we have never seen is not possible — so there is one to share. The reports group by this, so a folder of them appears together as "other".
dataclass
¶
class-attribute
instance-attribute
¶
The route each page needs: its text layer, OCR, or a vision model.
class-attribute
instance-attribute
¶
Per-page wireframes. Geometry only — never document content.
class-attribute
instance-attribute
¶
The text itself. Only populated with --extracted-text: it is the document.
class-attribute
instance-attribute
¶
One per extractor the run was asked for. The first is the one kept.
class-attribute
instance-attribute
¶
metadata_findings: list[MetadataFinding] = field(
default_factory=list
)
Identifiers in the metadata a loader returned. Empty for files read directly.
class-attribute
instance-attribute
¶
content_findings: list[ContentFinding] = field(
default_factory=list
)
Hidden passages and instruction-like text. See complydoc.hidden.
class-attribute
instance-attribute
¶
Pages read again by a vision model. None unless the run used --verify.
class-attribute
instance-attribute
¶
Whether hidden text could be checked for. None when the scan did not run.
class-attribute
instance-attribute
¶
Metadata keys whose value is an absolute filesystem path.
Not an identifier category, and no detector would flag it, but a home directory path names the account it belongs to and the layout around it.
property
¶
Whether the extractors read this document differently enough to say so.
Measured against the one whose output was kept. A tenth of the text is the line: below that the difference is line endings and whitespace, and reporting it would be noise on every document.
property
¶
The kind of disagreement between extractors, or None.
instance-attribute
¶
Gather results instead of raising, so every check runs. Used by policies.
no_identifiers(
*,
severity: str | None = None,
evidence: str | None = None,
categories: Iterable[str] | None = None,
) -> Expectation
No identifiers in text or metadata at or above severity and evidence.
no_hidden(*, severity: str = 'medium') -> Expectation
No hidden or instruction-like passages at or above severity.
readiness_at_least(score: float) -> Expectation
Every scored document has a readiness score of at least score.
facts_found(
facts: Iterable[Fact | str] | None = None,
*,
threshold: float = FUZZY_THRESHOLD,
) -> Expectation
Every fact found by every loader.
Without facts, uses the facts of a compare_loaders report.
no_network() -> Expectation
Nothing in the run reached the network: no loader, and no classifier.
The loader attempts were the whole of this check, which made it pass on a run that sent every judged passage to a hosted service — the one case where a gate asserting "no network" most needs to fail.
no_regressions(
baseline: AuditReport | str | PathLike[str],
*,
score_tolerance: float = 0.5,
) -> Expectation
Nothing worse than in baseline, a report or a path to its JSON.
dataclass
¶
One page as one extractor read it.
class-attribute
instance-attribute
¶
Characters before unicode normalisation, where the extractor exposes them.
Empty when it does not: the garbled-character signal needs un-normalised text and reports as not measured without it.
class-attribute
instance-attribute
¶
False when this extractor cannot look for tables at all, which is not the same as looking and finding none.
Share of the page covered by text boxes, or None if none were returned.
A reader that returns text without geometry has no coverage measurement.
dataclass
¶
Something the caller needs to know before using the text.
property
¶
Whether this warning means text is missing from the result.
dataclass
¶
A passage a document is expected to contain.
class-attribute
instance-attribute
¶
File name or relative path to check. None checks every document.
dataclass
¶
Whether one expected fact appears in the text of each loader.
instance-attribute
¶
The document the fact was checked in, or None for every document.
instance-attribute
¶
Loader name to the best similarity found, from 0 to 1.
instance-attribute
¶
Loader name to the document the fact was found in.
class-attribute
instance-attribute
¶
Loader name to the passage that came closest, where the fact was not found exactly.
What a missing fact looks like in that loader's text, which is usually the reason it went missing.
dataclass
¶
instance-attribute
¶
whole when one chunk contains it, split when only the joined document text
does, missing otherwise.
dataclass
¶
An identifier in some loaders' output and not in others'.
Matched by where it was found, its category and its masked value, so the
same identifier read by two loaders is one row. Metadata findings are
matched regardless of key name, because loaders spell the same field
differently: producer in one, Producer in another.
dataclass
¶
class-attribute
instance-attribute
¶
Run local OCR on pages with no usable text layer.
class-attribute
instance-attribute
¶
Below this many native characters, a page counts as having no text layer.
class-attribute
instance-attribute
¶
Resolution used when rasterising a page for OCR or skew measurement.
class-attribute
instance-attribute
¶
Cap on how many pages of one document are rasterised, to bound memory.
class-attribute
instance-attribute
¶
Which extractor's output the report is built from.
class-attribute
instance-attribute
¶
Others to run alongside, for comparison only. They never change a finding.
class-attribute
instance-attribute
¶
OCR engines to read every rasterised page with, beside the one in use.
class-attribute
instance-attribute
¶
Whether to keep each reader's text for the page.
class-attribute
instance-attribute
¶
Tried on encrypted files before falling back to an empty password.
class-attribute
instance-attribute
¶
Also OCR pages that already have a text layer, so the two can be compared.
class-attribute
instance-attribute
¶
Rasterise every page, including pages no signal needs.
Set when the report is going to show the page next to what was extracted from it. Off by default: rasterising costs time and memory, and the default report carries no page images.
dataclass
¶
dataclass
¶
Where several loaders' output differed, measured against the first.
class-attribute
instance-attribute
¶
Keys not returned by every loader, matched ignoring case, and which returned them.
class-attribute
instance-attribute
¶
Documents not returned by every loader, and which loaders returned them.
class-attribute
instance-attribute
¶
facts: list[FactCheck] = field(default_factory=list)
Expected facts, checked against every loader's text.
class-attribute
instance-attribute
¶
The loader to use, where the run could tell. None when it could not.
class-attribute
instance-attribute
¶
What decided the recommendation, or what stopped it being decided.
class-attribute
instance-attribute
¶
Every loader, best first, by what the run could measure.
dataclass
¶
What an external loader did when complydoc ran it.
instance-attribute
¶
None when documents were passed in already loaded.
class-attribute
instance-attribute
¶
Connections the loader tried to open: refused by the guard, or made, when
network_allowed is true.
class-attribute
instance-attribute
¶
Every metadata key the loader returned, across all documents.
class-attribute
instance-attribute
¶
Files the loader raised on, with the error, when it ran over several files.
class-attribute
instance-attribute
¶
Files whose output came from the cache instead of the loader.
class-attribute
instance-attribute
¶
The caller passed allow_network=True, so the loader's connections went
through. complydoc's own processing stays behind the guard either way.
class-attribute
instance-attribute
¶
The framework and library the loader comes from, such as LangChain and pypdf.
dataclass
¶
One loader's totals, in a comparison of several.
instance-attribute
¶
Documents with at least one metadata key holding an absolute path.
class-attribute
instance-attribute
¶
Files the loader raised on, with the error.
class-attribute
instance-attribute
¶
Expected facts found in this loader's text, when facts were given.
class-attribute
instance-attribute
¶
Estimated parser cost for these pages, from parsers in pricing.yaml.
class-attribute
instance-attribute
¶
The framework and library the loader comes from, such as LangChain and pypdf.
dataclass
¶
A string with its identifiers replaced by mask characters.
dataclass
¶
An identifier found in a document's metadata.
Loaders attach metadata to every document they return, and it is usually stored beside the content — in a vector store, next to each chunk — so an identifier there travels exactly as far as one in the text.
property
¶
Whether this finding raises limitations and quick wins.
A low-severity category found by the name model is excluded. Loaders put the producing software in metadata — "ReportLab PDF Library", "Microsoft Word" — and the model labels it an organisation, which would otherwise flag every PDF. It stays in the list; it does not become a limitation or a quick win on its own.
dataclass
¶
class-attribute
instance-attribute
¶
What OCR read, kept separately from text.
Normally OCR only runs where there is no text layer, and its output becomes
text. With ocr_compare it runs on every page as well, so the text layer
and the recognised text can be put side by side — which is how you tell a
document that extracts badly from an extractor that reads it badly.
class-attribute
instance-attribute
¶
The engine's mean confidence in what it read, 0 to 1. None if it did not run.
class-attribute
instance-attribute
¶
How long OCR took on this page, on the machine that ran it. None if it did not run.
class-attribute
instance-attribute
¶
Page characters as stored, before any unicode normalisation.
text comes from the extractor's own text assembly, which NFKC-normalises —
turning a fi ligature into two plain letters. That is usually helpful, but it
destroys exactly the evidence the garbled-character signal needs, so the
un-normalised characters are kept alongside. Word spacing is absent here, so
this is only useful for character-level questions.
class-attribute
instance-attribute
¶
One per extractor asked for, including the one whose output was kept.
class-attribute
instance-attribute
¶
What each extractor and OCR engine made of this page, by name.
Only populated for the readers a run was asked to compare, and only when the run is keeping text at all. It is the page's words several times over, so it is the largest thing a comparison adds to a report.
class-attribute
instance-attribute
¶
Populated only for pages that need to be looked at as pixels.
property
¶
Rounded page size, for counting distinct sizes within one document.
Effective scan resolution, from the largest embedded image on the page.
Only meaningful where the page really is a scan; a page whose images are small logos will report a number that means nothing, so callers check image coverage first.
dataclass
¶
The handles a page can be read through.
One per library, opened once by the loader. An extractor takes the handle it understands and ignores the rest, so adding a third does not mean opening the file a third time for the two that were already there.
dataclass
¶
dataclass
¶
dataclass
¶
What an engine read, and how sure it was.
An engine reports a confidence for every box it recognises. Throwing those away would leave OCR text asserted with the same authority as a native text layer, which it has not earned.
dataclass
¶
dataclass
¶
dataclass
¶
A file complydoc could not open, or could not finish. Reported; the run continues.
dataclass
¶
dataclass
¶
Everything read, in order, with what could not be read alongside it.
property
¶
Whether everything in the folder reached this result.
False when a file or a page could not be read.
property
¶
Whether every configured identifier category ran.
True means no category was skipped. Categories found by a model can
still miss values; MASKING_BEST_EFFORT says so every time masking runs.
dataclass
¶
Identifiers found in a string.
class-attribute
instance-attribute
¶
Categories that could not be scanned, with the reason.
Bases: Protocol
Anything called with a page that returns what it read.
dataclass
¶
dataclass
¶
What a vision model read off one page.
class-attribute
instance-attribute
¶
The provider's own count, from the response. Makes the cost actual.
class-attribute
instance-attribute
¶
What the call cost, where the caller already knows. Taken as given.
class-attribute
instance-attribute
¶
The model that answered, where it differs from the one the reader named.
clean_document(
path: str | Path,
out_dir: str | Path,
config: Config | None = None,
*,
rasterise: bool = False,
) -> CleanResult
Write a safe copy of path into out_dir, and say what was done to it.
rasterise applies to PDFs only: each page is rendered to an image and the
file rebuilt from those, so the text layer does not survive.