dump_upload_pipeline.py — Mobile-Dump Ingestion
BCHPR · NPOC and Wave 11 · 2024 – present
Built to end a specific failure: the notebook it replaced created duplicate patients three ways. Repeated QR codes were suffixed −1, −2 and uploaded as separate people; a field phone syncing straight to REDCap produced a second unlinked record for someone already dumped; and merging forms before upload multiplied rows. Duplicate enrolment inflates the denominator, which is an endpoint-validity problem, not a tidiness one.
Highlights
- Participant identity is a canonical QR code per row; change detection is a SHA-256 hash over the form's key columns, normalised so that “5.0” and “5” hash identically.
- Idempotency enforced structurally — SQLite primary keys plus INSERT OR IGNORE, so a crashed and restarted run cannot double-write.
- Nine pipeline stages across 10 tables and two databases, with a review queue holding six kinds of conflict for human adjudication rather than guessing.
- File-change detection is size and mtime, not content hash — a SharePoint touch costs one cheap CSV re-read, where hashing every file every run costs far more.
- A salvage path for dumps arriving as JSON-wrapped CSV, added after six files in one site produced between 46,000 and 123,000 phantom columns each.
- Journal mode chosen by filesystem: WAL normally, DELETE on network and NTFS mounts where shared memory is unavailable.
- Uploads run thread-parallel over HTTP while every SQLite write stays on the main thread, keeping a single writer.
Related projects
Data Manager
data_quality_manager.py — Data-Quality Engine
REDCap validates one field at a time, at entry. It cannot express “this specimen date precedes the enrolment date on a different form”, or “these two record IDs are the same participant”. This 14,274-line engine does, and it remembers — every issue it raises has an open date, a close date, and a resolution time.
Data Manager
MANAGER.py — Manager.io Accounting API Client
15,661-line Manager.io API client. Full create / update / delete across 18 document types — sales quotes, orders, invoices, credit notes, delivery notes and receipts; purchase quotes, orders, invoices and debit notes; inventory items, locations, transfers, write-offs, goods receipts, production orders and starting balances; and payments. Customers, suppliers, projects, accounts, assets and tax codes are exposed read-only.
Data Manager
data_utilities.py — Shared DataFrame Toolbox
The dependency root of the library — 9,746 lines that import nothing else in the codebase and that everything else imports. 84 public callables covering REDCap activity-log parsing, data-dictionary recoding, checkbox column expansion, export diffing, and file I/O that survives a SharePoint sync.