data_utilities.py — Shared DataFrame Toolbox
BCHPR · imported by 22 pipeline scripts · 2023 – present
The dependency root of the library — 9,746 lines that import nothing else in the codebase and that everything else imports. 84 public callables covering REDCap activity-log parsing, data-dictionary recoding, checkbox column expansion, export diffing, and file I/O that survives a SharePoint sync.
Highlights
- RedcapLogVectorOps is a third of the file and holds one rule: no row-wise Python loops, so it stays usable at production log volumes.
- Three implementations of duplicate-filling, and a router that measures the actual duplicate ratio to choose between them — sampling was rejected in a comment as statistically biased.
- Polars on the hot path with a pandas fallback, but deliberately reversed for compressed CSVs: single-pass decompress-and-parse holds peak memory at 1× file size where Polars would need 2×.
- Encoding fallback ordered utf-8 → cp1252 → latin-1, because latin-1 decodes every byte sequence and would silently mojibake Windows-1252 files if it came earlier.
- Atomic writes with a real OS file lock across platforms: hidden temp file, fsync, then os.replace.
- Worker count capped at 4 — more parallel CSV loads means more files in RAM at once, the primary cause of OOM under WSL.
- Log statements refuse to print row content, because the rows are medical records and scheduled-task logs retain stdout.
Related projects
Data Manager
data_quality_manager.py — Data-Quality Engine
REDCap validates one field at a time, at entry. It cannot express “this specimen date precedes the enrolment date on a different form”, or “these two record IDs are the same participant”. This 14,274-line engine does, and it remembers — every issue it raises has an open date, a close date, and a resolution time.
Data Manager
MANAGER.py — Manager.io Accounting API Client
15,661-line Manager.io API client. Full create / update / delete across 18 document types — sales quotes, orders, invoices, credit notes, delivery notes and receipts; purchase quotes, orders, invoices and debit notes; inventory items, locations, transfers, write-offs, goods receipts, production orders and starting balances; and payments. Customers, suppliers, projects, accounts, assets and tax codes are exposed read-only.
Data Manager
redcap_api.py — Typed REDCap API Facade
REDCap exposes one POST endpoint per project, dispatched by form parameters rather than typed routes, with no retry, no pagination, and no bulk file endpoint. This 6,561-line facade wraps 16 of those endpoints behind a typed surface of 42 exported names.