Comparing loaders
compare_loaders runs several loaders on the same input and reports where their
output differs: the text, the identifiers found in it, the metadata, the
documents returned and the network connections attempted.
"""Compare a retired langchain-community loader with its standalone replacement."""
# requires: langchain_community, langchain_pymupdf4llm
from langchain_community.document_loaders import PyPDFLoader # archived June 2026from langchain_pymupdf4llm import PyMuPDF4LLMLoader
import complydoc as cd
path = "src/complydoc/sample/vendor-assessment.pdf"report = cd.compare_loaders( {"pypdf": PyPDFLoader(path), "pymupdf4llm": PyMuPDF4LLMLoader(path)}, facts=["Two administrator accounts have no multi-factor authentication"],)
comparison = report.loader_comparisonfor row in comparison.loaders: print( f"{row.name:<12} {row.characters:>6,} characters " f"{row.identifiers_in_text:>2} identifiers in text " f"{len(row.metadata_keys):>2} metadata keys " f"network attempts: {len(row.network_attempts)}" )
for document in report.documents: for reading in document.extractions[1:]: print( f"\n{document.relative_path}: {reading.extractor} matches the baseline " f"{reading.similarity:.0%}" )
for difference in comparison.identifier_differences: where = "text" if difference.location == "text" else f"metadata ({', '.join(difference.keys)})" print( f"{difference.label} in {where}: found by {', '.join(difference.found_by)}, " f"missed by {', '.join(difference.missed_by)}" )
for check in comparison.facts: print(check.fact, check.found)
# `complydoc ui` opens what is in .complydoc, where the command line writes too.cd.write_json(report, ".complydoc/loaders.json")Moving off langchain-community? Replacing a langchain-community loader
says where each loader went and what to compare before switching.
loaders is either a mapping of names to loaders, or a sequence of loaders
named after their class. A second loader of the same class is named
PyPDFLoader (2). Each entry is anything
inspect_documents accepts, including a
list of documents already loaded.
At least two are required. Every loader runs with the same config,
components, reveal, models and allow_network.
Loader tags
Section titled “Loader tags”Each loader is tagged with the framework and library it comes from, read from its
module and class: PyMuPDF4LLMLoader from langchain_pymupdf4llm is tagged LangChain
and PyMuPDF4LLM, PyPDFLoader from langchain_community LangChain and pypdf, and a
LlamaIndex reader LlamaIndex. Parser presets declare their
tags, and hosted presets are also tagged hosted. Loaders from other modules get no
tags. The tags are in LoaderSummary.tags and LoaderRun.tags, in the tags column of
report.to_pandas("loaders"), and beside each loader’s name in the viewer.
Baseline
Section titled “Baseline”The first loader is the baseline. The report’s findings, readiness score,
global score, cost estimate and quick wins are built from its output;
report.loader is its run. The other loaders are measured against it.
A document the first loader did not return, because it failed on the file or was
not given its type, is measured against the next loader that returned it, and the
report is built from that loader’s reading of it. These documents are listed in
loader_comparison.baselines.
Each document’s extractions has one entry per loader that returned it, the
baseline first. similarity is how closely that loader’s words match the
baseline’s, in order, from 0 to 1. Where both loaders return page numbers it is
the lowest page’s similarity, and a page only one loader returned counts as 0.
Where either loader returns no page numbers, the whole document’s text is
compared.
reordered is true when every page that differed held the same words in a
different order.
With extracted_text on, the default, each page’s readings holds the other
loaders’ text for that page, and the viewer’s document view shows the words
that differ. Documents without page numbers get readings only when
the baseline returned them as a single page.
Identifiers
Section titled “Identifiers”report.loader_comparison.identifier_differences lists identifiers found in
some loaders’ output for a document and not in others’. Only loaders that
returned the document are compared on it.
An identifier is matched by where it was found (text or metadata), its
category and its masked value. Metadata findings are matched regardless of key,
so producer and Producer holding the same value are one identifier; the keys
are listed in keys.
Values are masked unless reveal=True. A difference in extracted text can also
change the masked form, for example when one loader drops the spaces inside a
number; that is reported as two differences.
A folder of files
Section titled “A folder of files”With paths, each loader is a callable taking a file path, such as a LangChain
loader class, and runs once per file:
"""Compare two settings of one LangChain loader over a folder, with expected facts."""
# requires: langchain_pymupdf4llm
import functools
from langchain_pymupdf4llm import PyMuPDF4LLMLoader
import complydoc as cd
report = cd.compare_loaders( { "layout": PyMuPDF4LLMLoader, "no-layout": functools.partial(PyMuPDF4LLMLoader, use_layout=False), }, paths=[ "src/complydoc/sample/employee-record.pdf", "src/complydoc/sample/vendor-assessment.pdf", ], facts=[ cd.Fact( "Two administrator accounts have no multi-factor authentication", document="vendor-assessment.pdf", ), ],)
for row in report.loader_comparison.loaders: print( f"{row.name:<10} documents {row.documents} failed {len(row.failures)} " f"facts found {row.facts_found}" )
for check in report.loader_comparison.facts: print(check.fact, check.found)paths is a folder (searched as complydoc audit searches it), a file, or a list
of files. A file a loader raises on is recorded in its row’s failures and the
report lists it as a limitation; the other files are still loaded.
Several file types
Section titled “Several file types”A folder of PDFs, Word files and spreadsheets is usually read by a different loader for each. Each loader is given only the file types it is meant for, and the comparison is repeated for each type:
"""Compare loaders over a folder of PDFs and Word files, one file type at a time."""
# requires: langchain_community, docx2txt
from langchain_community.document_loaders import Docx2txtLoader, PDFPlumberLoader, PyPDFLoader
import complydoc as cd
# PyPDFLoader and PDFPlumberLoader are given the PDFs, Docx2txtLoader the Word file.report = cd.compare_loaders( {"pypdf": PyPDFLoader, "pdfplumber": PDFPlumberLoader, "docx2txt": Docx2txtLoader}, paths="src/complydoc/sample", facts=["Two administrator accounts have no multi-factor authentication"],)
comparison = report.loader_comparisonfor file_type in comparison.formats: skipped = ", ".join(file_type.skipped_by) or "none" print(f"{file_type.label} ({file_type.documents}), skipped by {skipped}") for row in file_type.loaders: print(f" {row.name:<11} read {row.documents}, failed on {len(row.failures)}") print(f" use {file_type.recommended or '—'}: {file_type.verdict}")
print(comparison.verdict)# `complydoc ui` opens what is in .complydoc, where the command line writes too.cd.write_json(report, ".complydoc/loaders.json")A loader’s file types come from, in order:
formats={"name": [...]}passed tocompare_loaders, as extensions (.pdf) or format names (pdf,docx,xlsx,pptx,html,markdown,text,email,image);- a parser preset’s own list:
docling,markitdown,opendataloader_pdfandazure_document_intelligenceknow theirs,unstructuredandllamaparsetake every file; - the class name, for loaders known to read one type:
PyPDFLoader,PDFPlumberLoader,PyMuPDFLoader,Docx2txtLoader,UnstructuredExcelLoader, LlamaIndex’sPDFReaderand others.
Any other loader, such as PyMuPDF4LLMLoader, which opens Office files too, is
given every file. A file not given to a loader is listed in its row’s skipped,
and is not a failure.
loader_comparison.formats has one entry per file type:
| Field | Contents |
|---|---|
label, documents |
The type, and how many files of it the run had |
loaders |
A row per loader given the type: documents read, pages, characters, load time, failures, facts found, and similarity, the mean of its documents’ similarity to their baseline |
skipped_by |
Loaders not meant for the type |
recommended, verdict |
The choice among the loaders given the type, decided as for the whole run |
A fact is checked only by the loaders given its document’s type, so a Word
loader is not counted as missing a sentence from a PDF. Metadata keys and
documents “returned by some loaders only” are likewise compared between loaders
of the same type. report.to_pandas("loader_formats") has a row per type and
loader.
Expected facts
Section titled “Expected facts”facts are passages the documents should contain, as strings or
cd.Fact(text, document=...). Each is checked in every loader’s text:
| Result | Meaning |
|---|---|
exact |
The passage appears, ignoring case, whitespace, invisible characters and words split at a line break |
fuzzy |
The most similar run of words scores at least fact_threshold (0.9 by default) |
| none | Not found |
Results are in report.loader_comparison.facts, each loader’s count in
facts_found, and loaders that miss a fact are listed in the limitations.
cd.check_facts(report, facts) runs the same check on any report produced with
extracted_text=True.
Parser presets
Section titled “Parser presets”complydoc.loaders.parsers returns a LoaderSpec for each parser, used with paths:
| Preset | Library | Hosted | Price entry |
|---|---|---|---|
parsers.docling(export="markdown") |
langchain-docling[local] |
no | docling |
parsers.markitdown() |
markitdown |
no | none |
parsers.opendataloader_pdf() |
langchain-opendataloader-pdf (Java 11+ on PATH) |
no | none |
parsers.unstructured(api=False) |
langchain-unstructured |
when api=True |
unstructured_api |
parsers.llamaparse(tier="cost_effective") |
llama-parse |
yes | llamaparse_<tier> |
parsers.azure_document_intelligence(endpoint=..., api_key=...) |
azure-ai-documentintelligence |
yes | azure_read, azure_layout |
The libraries are not dependencies; a preset reports what to install when its
library is missing. A hosted preset raises unless allow_network=True.
OpenDataLoader PDF reads PDFs locally by default; its safety filters do not replace
an audit of the extracted text before indexing.
A preset’s price comes from parsers in pricing.yaml. The comparison reports
parser_usd, the page count times the price per page. The shipped prices are not
verified, and the report says so for each one used.
Other differences
Section titled “Other differences”| Field | Contents |
|---|---|
loaders |
Per loader: documents, pages, characters, load time, network, metadata keys, identifier counts, readiness and global scores, text-path cost |
metadata_keys |
Keys not returned by every loader, matched ignoring case, and which loaders returned them. A key spelled differently by different loaders is shown as author / Author |
documents |
Documents not returned by every loader, and which loaders returned them |
Documents are matched by the file named in their metadata, as described in grouping.
Limitations in the report
Section titled “Limitations in the report”In addition to the baseline’s own limitations, a comparison adds:
- which loader is the baseline
- the number of identifiers found by some loaders only, excluding low-severity findings from the name model
- documents not returned by every loader
- network attempts, failures and metadata findings from the other loaders
From the command line
Section titled “From the command line”complydoc compare-loaders FILE runs a comparison described in YAML and writes
the same JSON report as complydoc audit, which complydoc ui opens:
loaders: pymupdf4llm: langchain_pymupdf4llm:PyMuPDF4LLMLoader one-per-file: loader: langchain_pymupdf4llm:PyMuPDF4LLMLoader options: {mode: single} docling: preset: docling options: {export: markdown}paths: ./contractsfacts: - Payment is due within thirty days| Key | Default | Contents |
|---|---|---|
loaders |
required | Two or more names mapped to module:attribute, or to loader or preset with keyword options and formats |
paths |
required | A folder, a file or a list, relative to the YAML file |
facts |
none | Passages each loader’s text should contain |
fact_threshold |
0.9 | Fuzzy match threshold |
allow_network |
false |
Let the loaders connect; the command prints a warning when set |
cache_dir |
none | Cache loader output between runs |
components |
all | cost, readiness and sensitive |
models |
all priced | Models to price |
The first loader is the baseline. A loader is called with each file path and its
options; a preset is one of the parser presets above. formats, such as
[docx], limits the files a loader is given. With several file types, the output
adds a table of each type and what each loader did with it. An invalid file, or a
loader that cannot be imported, exits with status 2.
Which one to use
Section titled “Which one to use”The comparison ends with a recommendation on the command line and in the report’s
loader_comparison. The viewer leaves the choice to you: open a document and its
Diff sets any loader’s reading beside another’s. Three things can decide the
recommendation:
| What decided it | Example |
|---|---|
| Files a loader could not open | Use pypdf, it opened all 9 documents; pdfplumber raised on 1 |
| An expected fact one loader kept | Use pypdf, it kept 5 of 5 expected facts, pdfplumber kept 4 |
| Time, where the loaders read the same text | Use pypdf, every loader read the same text and it was the quickest |
Where the loaders read the same documents differently and no fact was given, there is no recommendation, because nothing in the text says which reading is right:
No recommendation — the loaders read 1 document differently and nothing heresays which reading is right. Pass a passage these documents contain, as a fact,and the comparison can say which reading holds it.A two-column page is the usual cause: one library walks the columns and another
reads straight across, and both return the same words. facts is what settles
it, which is why it is worth filling in.
With several file types, each type is decided among the loaders given it. One loader is recommended for the folder only when it is the choice for every type; otherwise the verdict names one for each:
No recommendation — Each file type is decided on its own. Use pypdf for PDF,docx2txt for Word. None of the loaders is meant for Excel files.