Skip to content

Comparing loaders

compare_loaders runs several loaders on the same input and reports where their output differs: the text, the identifiers found in it, the metadata, the documents returned and the network connections attempted.

compare_langchain_loaders.py
"""Compare a retired langchain-community loader with its standalone replacement."""
# requires: langchain_community, langchain_pymupdf4llm
from langchain_community.document_loaders import PyPDFLoader # archived June 2026
from langchain_pymupdf4llm import PyMuPDF4LLMLoader
import complydoc as cd
path = "src/complydoc/sample/vendor-assessment.pdf"
report = cd.compare_loaders(
{"pypdf": PyPDFLoader(path), "pymupdf4llm": PyMuPDF4LLMLoader(path)},
facts=["Two administrator accounts have no multi-factor authentication"],
)
comparison = report.loader_comparison
for row in comparison.loaders:
print(
f"{row.name:<12} {row.characters:>6,} characters "
f"{row.identifiers_in_text:>2} identifiers in text "
f"{len(row.metadata_keys):>2} metadata keys "
f"network attempts: {len(row.network_attempts)}"
)
for document in report.documents:
for reading in document.extractions[1:]:
print(
f"\n{document.relative_path}: {reading.extractor} matches the baseline "
f"{reading.similarity:.0%}"
)
for difference in comparison.identifier_differences:
where = "text" if difference.location == "text" else f"metadata ({', '.join(difference.keys)})"
print(
f"{difference.label} in {where}: found by {', '.join(difference.found_by)}, "
f"missed by {', '.join(difference.missed_by)}"
)
for check in comparison.facts:
print(check.fact, check.found)
# `complydoc ui` opens what is in .complydoc, where the command line writes too.
cd.write_json(report, ".complydoc/loaders.json")

Moving off langchain-community? Replacing a langchain-community loader says where each loader went and what to compare before switching.

loaders is either a mapping of names to loaders, or a sequence of loaders named after their class. A second loader of the same class is named PyPDFLoader (2). Each entry is anything inspect_documents accepts, including a list of documents already loaded.

At least two are required. Every loader runs with the same config, components, reveal, models and allow_network.

Each loader is tagged with the framework and library it comes from, read from its module and class: PyMuPDF4LLMLoader from langchain_pymupdf4llm is tagged LangChain and PyMuPDF4LLM, PyPDFLoader from langchain_community LangChain and pypdf, and a LlamaIndex reader LlamaIndex. Parser presets declare their tags, and hosted presets are also tagged hosted. Loaders from other modules get no tags. The tags are in LoaderSummary.tags and LoaderRun.tags, in the tags column of report.to_pandas("loaders"), and beside each loader’s name in the viewer.

The first loader is the baseline. The report’s findings, readiness score, global score, cost estimate and quick wins are built from its output; report.loader is its run. The other loaders are measured against it.

A document the first loader did not return, because it failed on the file or was not given its type, is measured against the next loader that returned it, and the report is built from that loader’s reading of it. These documents are listed in loader_comparison.baselines.

Each document’s extractions has one entry per loader that returned it, the baseline first. similarity is how closely that loader’s words match the baseline’s, in order, from 0 to 1. Where both loaders return page numbers it is the lowest page’s similarity, and a page only one loader returned counts as 0. Where either loader returns no page numbers, the whole document’s text is compared.

reordered is true when every page that differed held the same words in a different order.

With extracted_text on, the default, each page’s readings holds the other loaders’ text for that page, and the viewer’s document view shows the words that differ. Documents without page numbers get readings only when the baseline returned them as a single page.

report.loader_comparison.identifier_differences lists identifiers found in some loaders’ output for a document and not in others’. Only loaders that returned the document are compared on it.

An identifier is matched by where it was found (text or metadata), its category and its masked value. Metadata findings are matched regardless of key, so producer and Producer holding the same value are one identifier; the keys are listed in keys.

Values are masked unless reveal=True. A difference in extracted text can also change the masked form, for example when one loader drops the spaces inside a number; that is reported as two differences.

With paths, each loader is a callable taking a file path, such as a LangChain loader class, and runs once per file:

compare_loaders_over_folder.py
"""Compare two settings of one LangChain loader over a folder, with expected facts."""
# requires: langchain_pymupdf4llm
import functools
from langchain_pymupdf4llm import PyMuPDF4LLMLoader
import complydoc as cd
report = cd.compare_loaders(
{
"layout": PyMuPDF4LLMLoader,
"no-layout": functools.partial(PyMuPDF4LLMLoader, use_layout=False),
},
paths=[
"src/complydoc/sample/employee-record.pdf",
"src/complydoc/sample/vendor-assessment.pdf",
],
facts=[
cd.Fact(
"Two administrator accounts have no multi-factor authentication",
document="vendor-assessment.pdf",
),
],
)
for row in report.loader_comparison.loaders:
print(
f"{row.name:<10} documents {row.documents} failed {len(row.failures)} "
f"facts found {row.facts_found}"
)
for check in report.loader_comparison.facts:
print(check.fact, check.found)

paths is a folder (searched as complydoc audit searches it), a file, or a list of files. A file a loader raises on is recorded in its row’s failures and the report lists it as a limitation; the other files are still loaded.

A folder of PDFs, Word files and spreadsheets is usually read by a different loader for each. Each loader is given only the file types it is meant for, and the comparison is repeated for each type:

compare_loaders_across_file_types.py
"""Compare loaders over a folder of PDFs and Word files, one file type at a time."""
# requires: langchain_community, docx2txt
from langchain_community.document_loaders import Docx2txtLoader, PDFPlumberLoader, PyPDFLoader
import complydoc as cd
# PyPDFLoader and PDFPlumberLoader are given the PDFs, Docx2txtLoader the Word file.
report = cd.compare_loaders(
{"pypdf": PyPDFLoader, "pdfplumber": PDFPlumberLoader, "docx2txt": Docx2txtLoader},
paths="src/complydoc/sample",
facts=["Two administrator accounts have no multi-factor authentication"],
)
comparison = report.loader_comparison
for file_type in comparison.formats:
skipped = ", ".join(file_type.skipped_by) or "none"
print(f"{file_type.label} ({file_type.documents}), skipped by {skipped}")
for row in file_type.loaders:
print(f" {row.name:<11} read {row.documents}, failed on {len(row.failures)}")
print(f" use {file_type.recommended or '—'}: {file_type.verdict}")
print(comparison.verdict)
# `complydoc ui` opens what is in .complydoc, where the command line writes too.
cd.write_json(report, ".complydoc/loaders.json")

A loader’s file types come from, in order:

  1. formats={"name": [...]} passed to compare_loaders, as extensions (.pdf) or format names (pdf, docx, xlsx, pptx, html, markdown, text, email, image);
  2. a parser preset’s own list: docling, markitdown, opendataloader_pdf and azure_document_intelligence know theirs, unstructured and llamaparse take every file;
  3. the class name, for loaders known to read one type: PyPDFLoader, PDFPlumberLoader, PyMuPDFLoader, Docx2txtLoader, UnstructuredExcelLoader, LlamaIndex’s PDFReader and others.

Any other loader, such as PyMuPDF4LLMLoader, which opens Office files too, is given every file. A file not given to a loader is listed in its row’s skipped, and is not a failure.

loader_comparison.formats has one entry per file type:

Field Contents
label, documents The type, and how many files of it the run had
loaders A row per loader given the type: documents read, pages, characters, load time, failures, facts found, and similarity, the mean of its documents’ similarity to their baseline
skipped_by Loaders not meant for the type
recommended, verdict The choice among the loaders given the type, decided as for the whole run

A fact is checked only by the loaders given its document’s type, so a Word loader is not counted as missing a sentence from a PDF. Metadata keys and documents “returned by some loaders only” are likewise compared between loaders of the same type. report.to_pandas("loader_formats") has a row per type and loader.

facts are passages the documents should contain, as strings or cd.Fact(text, document=...). Each is checked in every loader’s text:

Result Meaning
exact The passage appears, ignoring case, whitespace, invisible characters and words split at a line break
fuzzy The most similar run of words scores at least fact_threshold (0.9 by default)
none Not found

Results are in report.loader_comparison.facts, each loader’s count in facts_found, and loaders that miss a fact are listed in the limitations. cd.check_facts(report, facts) runs the same check on any report produced with extracted_text=True.

complydoc.loaders.parsers returns a LoaderSpec for each parser, used with paths:

Preset Library Hosted Price entry
parsers.docling(export="markdown") langchain-docling[local] no docling
parsers.markitdown() markitdown no none
parsers.opendataloader_pdf() langchain-opendataloader-pdf (Java 11+ on PATH) no none
parsers.unstructured(api=False) langchain-unstructured when api=True unstructured_api
parsers.llamaparse(tier="cost_effective") llama-parse yes llamaparse_<tier>
parsers.azure_document_intelligence(endpoint=..., api_key=...) azure-ai-documentintelligence yes azure_read, azure_layout

The libraries are not dependencies; a preset reports what to install when its library is missing. A hosted preset raises unless allow_network=True. OpenDataLoader PDF reads PDFs locally by default; its safety filters do not replace an audit of the extracted text before indexing.

A preset’s price comes from parsers in pricing.yaml. The comparison reports parser_usd, the page count times the price per page. The shipped prices are not verified, and the report says so for each one used.

Field Contents
loaders Per loader: documents, pages, characters, load time, network, metadata keys, identifier counts, readiness and global scores, text-path cost
metadata_keys Keys not returned by every loader, matched ignoring case, and which loaders returned them. A key spelled differently by different loaders is shown as author / Author
documents Documents not returned by every loader, and which loaders returned them

Documents are matched by the file named in their metadata, as described in grouping.

In addition to the baseline’s own limitations, a comparison adds:

  • which loader is the baseline
  • the number of identifiers found by some loaders only, excluding low-severity findings from the name model
  • documents not returned by every loader
  • network attempts, failures and metadata findings from the other loaders

complydoc compare-loaders FILE runs a comparison described in YAML and writes the same JSON report as complydoc audit, which complydoc ui opens:

loaders.yaml
loaders:
pymupdf4llm: langchain_pymupdf4llm:PyMuPDF4LLMLoader
one-per-file:
loader: langchain_pymupdf4llm:PyMuPDF4LLMLoader
options: {mode: single}
docling:
preset: docling
options: {export: markdown}
paths: ./contracts
facts:
- Payment is due within thirty days
Key Default Contents
loaders required Two or more names mapped to module:attribute, or to loader or preset with keyword options and formats
paths required A folder, a file or a list, relative to the YAML file
facts none Passages each loader’s text should contain
fact_threshold 0.9 Fuzzy match threshold
allow_network false Let the loaders connect; the command prints a warning when set
cache_dir none Cache loader output between runs
components all cost, readiness and sensitive
models all priced Models to price

The first loader is the baseline. A loader is called with each file path and its options; a preset is one of the parser presets above. formats, such as [docx], limits the files a loader is given. With several file types, the output adds a table of each type and what each loader did with it. An invalid file, or a loader that cannot be imported, exits with status 2.

The comparison ends with a recommendation on the command line and in the report’s loader_comparison. The viewer leaves the choice to you: open a document and its Diff sets any loader’s reading beside another’s. Three things can decide the recommendation:

What decided it Example
Files a loader could not open Use pypdf, it opened all 9 documents; pdfplumber raised on 1
An expected fact one loader kept Use pypdf, it kept 5 of 5 expected facts, pdfplumber kept 4
Time, where the loaders read the same text Use pypdf, every loader read the same text and it was the quickest

Where the loaders read the same documents differently and no fact was given, there is no recommendation, because nothing in the text says which reading is right:

No recommendation — the loaders read 1 document differently and nothing here
says which reading is right. Pass a passage these documents contain, as a fact,
and the comparison can say which reading holds it.

A two-column page is the usual cause: one library walks the columns and another reads straight across, and both return the same words. facts is what settles it, which is why it is worth filling in.

With several file types, each type is decided among the loaders given it. One loader is recommended for the folder only when it is the choice for every type; otherwise the verdict names one for each:

No recommendation — Each file type is decided on its own. Use pypdf for PDF,
docx2txt for Word. None of the loaders is meant for Excel files.