data_quality_manager.py — Data-Quality Engine
BCHPR · used by 11 projects · 2023 – present
REDCap validates one field at a time, at entry. It cannot express “this specimen date precedes the enrolment date on a different form”, or “these two record IDs are the same participant”. This 14,274-line engine does, and it remembers — every issue it raises has an open date, a close date, and a resolution time.
Highlights
- Seven-dimension quality taxonomy (completeness, validity, accuracy, consistency, timeliness, uniqueness, integrity) with 35 error categories mapped onto it.
- Fluent QueryBuilder over a declarative, immutable schema — referencing an undeclared field fails at build time rather than silently yielding null.
- Six selectable scoring algorithms, defaulting to defect density (failures per opportunity to fail) rather than ticket-closure velocity.
- SQLite tuned for SharePoint-synced drives: WAL where safe, DELETE journaling where it is not, busy timeouts, and configurable integrity-check depth — a full check on a multi-GB tracker can stall for tens of minutes.
- History auto-pruned to the three most recent runs after one tracker database reached 35 million rows and 8.8 GB.
- Duplicate-record analysis is a quarter of the file: pairwise field comparison, concordance scoring, and generated merge plans for same-participant / different-record-ID collisions.
- Carries no study-specific checks. Each project writes its own against the engine — including a non-clinical inventory use, which is the genuine test of the schema abstraction.
Related projects
Data Manager
MANAGER.py — Manager.io Accounting API Client
15,661-line Manager.io API client. Full create / update / delete across 18 document types — sales quotes, orders, invoices, credit notes, delivery notes and receipts; purchase quotes, orders, invoices and debit notes; inventory items, locations, transfers, write-offs, goods receipts, production orders and starting balances; and payments. Customers, suppliers, projects, accounts, assets and tax codes are exposed read-only.
Data Manager
data_utilities.py — Shared DataFrame Toolbox
The dependency root of the library — 9,746 lines that import nothing else in the codebase and that everything else imports. 84 public callables covering REDCap activity-log parsing, data-dictionary recoding, checkbox column expansion, export diffing, and file I/O that survives a SharePoint sync.
Data Manager
redcap_api.py — Typed REDCap API Facade
REDCap exposes one POST endpoint per project, dispatched by form parameters rather than typed routes, with no retry, no pagination, and no bulk file endpoint. This 6,561-line facade wraps 16 of those endpoints behind a typed surface of 42 exported names.