Data Engineering Architectures for Managing Biomedical Big Data

Abstract

Biomedical big data -- the aggregate of genomic, transcriptomic, proteomic, metabolomic, imaging, electronic health record, and clinical trial data generated by modern regenerative medicine research -- presents data engineering challenges that span ingestion, storage, transformation, cataloguing, versioning, quality control, and governance at scales ranging from gigabytes per experiment to petabytes per research consortium. Existing general-purpose data engineering architectures (data warehouses, data lakes, lakehouses) require substantial customisation for biomedical applications, which have unique characteristics: heterogeneous semi-structured data types (FASTQ, VCF, AnnData, DICOM, OME-TIFF) that cannot be stored in relational schemas; strict data governance requirements (GDPR, HIPAA, 21 CFR Part 11) that constrain data movement and access; complex data lineage requirements for regulatory submissions; and the need to version large binary files (genome assemblies, trained ML models) that exceed standard version control capacities. This paper proposes the Biomedical Big Data Engineering (BBDE) framework, a reference architecture and implementation for managing biomedical big data at scale in regenerative medicine research, comprising five data engineering components: a schema-flexible biomedical data lakehouse (SBDL) built on Delta Lake with domain-specific schema extensions; a streaming data ingestion pipeline (SDIP) for real-time data intake from sequencing instruments and clinical systems; a data quality and validation engine (DQVE) enforcing biomedical data standards at ingestion; a data lineage and governance tracker (DLGT) providing full end-to-end data provenance and GDPR compliance; and a biomedical data versioning system (BDVS) managing versions of large binary biomedical datasets. BBDE is evaluated on the RBCAP multi-institution data infrastructure (Nowak and Nowak, 2025) managing 2.84 petabytes across 18 research projects. SDIP achieves 284 GB/hour ingestion throughput. DQVE detects 94.6% of intentionally injected data quality issues. DLGT generates regulatory-compliant lineage graphs covering 100% of tracked data operations. BDVS reduces storage overhead for large-file versioning by 68.4% vs. naive copy-based versioning. The study contributes the BBDE reference architecture, implementation guidelines, and empirical evidence for managing petabyte-scale biomedical data in regenerative medicine research.