Skip to content

Baselines and tests

A report written with cd.write_json can be read back, compared with a later report, and checked with assertions in a test suite or CI.

baseline_and_expect.py
"""Save a baseline, compare a later run with it, and assert on a report."""
import tempfile
from pathlib import Path
import complydoc as cd
document = "src/complydoc/sample/vendor-assessment.pdf"
with tempfile.TemporaryDirectory() as folder:
baseline = cd.write_json(cd.security_audit(document), Path(folder) / "baseline.json")
later = cd.security_audit(document)
changes = cd.diff_reports(cd.load_report(baseline), later)
print(changes.summary())
cd.expect(later).no_regressions(baseline).no_failures()
try:
cd.expect(later).no_hidden(severity="high")
except cd.ExpectationError as error:
print(error)

cd.load_report(path) returns the same AuditReport the run produced. It also accepts the parsed JSON as a dictionary. It reads every schema_version in cd.report.json_reader.READABLE_SCHEMA_VERSIONS, currently 5 to 15; fields added since a report was written take their defaults. Any other version raises ValueError.

cd.diff_reports(old, new) returns a ReportDiff. Documents are matched by relative path. Each Change has an area, a kind (added, removed or changed), the document, a subject, the values before and after, and worse.

Area Worse when
documents A document is missing
identifiers, metadata An identifier appears
hidden A hidden or instruction-like passage appears
readiness, global A score falls by at least score_tolerance (0.5)
signals A signal’s rating falls
similarity A loader’s text similarity to the baseline loader falls by 0.01 or more
facts A loader no longer finds a fact
loaders Network attempts or failed files increase
limitations An important limitation appears

changes.regressions and changes.improvements split the changes; changes.summary() prints one line each, and changes.to_pandas() returns a table.

Terminal window
complydoc audit ./documents --out build --name current -q
complydoc diff baseline.json build/current.json --out build

complydoc diff OLD NEW prints each change and exits with status 1 when there is a regression, 2 when either file cannot be read, and 0 otherwise. --no-fail-on-regression always exits 0 for readable files. --tolerance sets score_tolerance. --out also writes complydoc-diff.json; --print-json writes the JSON to stdout.

cd.expect(report) starts a chain of checks. A failed check raises cd.ExpectationError, a subclass of AssertionError, listing up to 20 items that failed it.

Check Fails when
no_identifiers(severity=, evidence=, categories=) An identifier in text or metadata matches the filters
no_hidden(severity="medium") A hidden or instruction-like passage is at or above the severity
readiness_at_least(score) A document scores below score
global_score_at_least(score) Global readiness is below score
facts_found(facts=None) A loader misses a fact; without facts, uses a comparison’s facts
no_network() A loader attempted or made a connection, or a registered classifier or --verify vision model sent document content to a host
no_failures() A loader failed on a file, or a file was skipped
all_categories_scanned() An identifier category could not be scanned
no_regressions(baseline) diff_reports(baseline, report) has a regression