Baselines and tests
A report written with cd.write_json can be read back, compared with a later
report, and checked with assertions in a test suite or CI.
"""Save a baseline, compare a later run with it, and assert on a report."""
import tempfilefrom pathlib import Path
import complydoc as cd
document = "src/complydoc/sample/vendor-assessment.pdf"
with tempfile.TemporaryDirectory() as folder: baseline = cd.write_json(cd.security_audit(document), Path(folder) / "baseline.json")
later = cd.security_audit(document) changes = cd.diff_reports(cd.load_report(baseline), later) print(changes.summary())
cd.expect(later).no_regressions(baseline).no_failures()
try: cd.expect(later).no_hidden(severity="high")except cd.ExpectationError as error: print(error)Reading a report
Section titled “Reading a report”cd.load_report(path) returns the same AuditReport the run produced. It also
accepts the parsed JSON as a dictionary. It reads every schema_version in
cd.report.json_reader.READABLE_SCHEMA_VERSIONS, currently 5 to 15; fields
added since a report was written take their defaults. Any other version raises
ValueError.
Comparing reports
Section titled “Comparing reports”cd.diff_reports(old, new) returns a ReportDiff. Documents are matched by
relative path. Each Change has an area, a kind (added, removed or
changed), the document, a subject, the values before and after, and worse.
| Area | Worse when |
|---|---|
documents |
A document is missing |
identifiers, metadata |
An identifier appears |
hidden |
A hidden or instruction-like passage appears |
readiness, global |
A score falls by at least score_tolerance (0.5) |
signals |
A signal’s rating falls |
similarity |
A loader’s text similarity to the baseline loader falls by 0.01 or more |
facts |
A loader no longer finds a fact |
loaders |
Network attempts or failed files increase |
limitations |
An important limitation appears |
changes.regressions and changes.improvements split the changes;
changes.summary() prints one line each, and changes.to_pandas() returns a
table.
complydoc audit ./documents --out build --name current -qcomplydoc diff baseline.json build/current.json --out buildcomplydoc diff OLD NEW prints each change and exits with status 1 when there is
a regression, 2 when either file cannot be read, and 0 otherwise.
--no-fail-on-regression always exits 0 for readable files. --tolerance sets
score_tolerance. --out also writes complydoc-diff.json; --print-json
writes the JSON to stdout.
Assertions
Section titled “Assertions”cd.expect(report) starts a chain of checks. A failed check raises
cd.ExpectationError, a subclass of AssertionError, listing up to 20 items that
failed it.
| Check | Fails when |
|---|---|
no_identifiers(severity=, evidence=, categories=) |
An identifier in text or metadata matches the filters |
no_hidden(severity="medium") |
A hidden or instruction-like passage is at or above the severity |
readiness_at_least(score) |
A document scores below score |
global_score_at_least(score) |
Global readiness is below score |
facts_found(facts=None) |
A loader misses a fact; without facts, uses a comparison’s facts |
no_network() |
A loader attempted or made a connection, or a registered classifier or --verify vision model sent document content to a host |
no_failures() |
A loader failed on a file, or a file was skipped |
all_categories_scanned() |
An identifier category could not be scanned |
no_regressions(baseline) |
diff_reports(baseline, report) has a regression |