AP
Uploaded byAlessia Peviani
PPTX, PDF303 views

Building ETL pipelines for tranSMART 17.X - New tools for the data loader

The document discusses the development and use of ETL (Extract, Transform, Load) pipelines in Transmart 17.x, highlighting both manual and automated data loading tools. It outlines various tools such as TMTK, Arborist, and Transmart copy for different data loading scenarios, facilitating collaboration between data scientists and data owners. The document emphasizes the modularity and flexibility of these tools to adapt to specific project needs and enhance maintainability.

Embed presentation

Download to read offline
Building ETL pipelines to tranSMART 17.X New tools for the data loader Alessia Peviani – Data Engineer
TRANSMART 17.X INSTANCE (SLICE) TMTK ARBORIST (plugin) TRANSMART LOADER TRANSMART COPY GENERIC DATA SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart TRANSMART 17.X INSTANCE CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART COPY Jupyter Notebook TRANSMART 17.X INSTANCE Tools overview: data flow and dependencies GENERIC ONTOLOGY ARBORIST (standalone) Manual pipeline Automated ETL pipeline Data flow Code dependency GENERIC DATA
Why Manual Pipelines Manual pipeline • One-time / infrequent data loading • Changing data format • Initial data exploration / modeling • Tools developed internally for data scientists, and to facilitate collaboration with data owners (researchers, clinicians) TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.X INSTANCE TRANSMART COPY Jupyter Notebook GENERIC ONTOLOGY ARBORIST (standalone) Data flow Code dependency
Why Automated ETL Pipelines Data flow Code dependency • Frequent updates • Stable data format • Typically large volumes of data • Tools developed for specific projects, but with flexibility in mind to enable use in future ETL pipelines TRANSMART 17.X INSTANCE (SLICE) TRANSMART LOADER TRANSMART COPY SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART 17.X INSTANCE Automated ETL pipeline GENERIC DATA
1. Tools for manual data loading TMTK & Arborist, transmart-copy
TMTK: The TranSMART Toolkit Manual pipeline • Easier collaboration with data owners: • Option to load tree structure, word mapping, metadata from Excel file ( “Template file”) • Interactive tree structure modeling with the Arborist • Updated to support TranSMART 17.X features • Added additional sheets in template file • Added export in transmart-copy format TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.X INSTANCE TRANSMART COPY Jupyter Notebook GENERIC ONTOLOGY ARBORIST (standalone) Data flow Code dependency https://github.com/thehyve/tmtk
TMTK: The TranSMART Toolkit Input template file
The Arborist Manual pipeline Two flavors of interactive tree modeling: • Without leaving Jupyter Notebook (embedded) • By sending tree to web service (standalone) • Share trees with data owners and let them edit • Re-import final result into TMTK • Functionality (both versions): • Add/remove tree node and metadata tags • Edit position and name TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.X INSTANCE TRANSMART COPY Jupyter Notebook GENERIC ONTOLOGY ARBORIST (standalone) Data flow Code dependency https://github.com/thehyve/arborist
The Arborist: Embedded vs Standalone
The Arborist: Embedded vs Standalone
The Arborist: Plug-in vs Standalone https://arborist-test-trait.thehyve.net/
The Arborist: Plug-in vs Standalone feedback Data owner
Data loading with transmart-copy Manual pipeline TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.X INSTANCE TRANSMART COPY Jupyter Notebook GENERIC ONTOLOGY ARBORIST (standalone) Data flow Code dependency • A simple tool with a simple function • Batch and incremental data loading • What you see (data tables) is what you get (transmart tables) • Supports new TranSMART 17.X features https://github.com/thehyve/transmart-core/tree/dev/transmart-copy
transmart-copy transmart-copy (TranSMART 17.X database schemas as seen in pgAdmin) https://github.com/thehyve/transmart-core/tree/dev/transmart-copy/src/test/resources/examples/SURVEY0
transmart-copy vs transmart-batch transmart-copy transmart-batch
transmart-copy vs transmart-batch transmart-copy transmart-batch • A simple tool with a simple function • Batch and incremental data loading • What you see (data tables) is what you get (transmart tables) • Supports new TranSMART 17.X features • Complex, includes validation steps • Batch loading only (inefficient in some cases) • Does not make explicit what your data will look like once in transmart
2. Tools for automated ETL pipelines transmart-loader & related tools, hyper-dicer
transmart-loader Data flow Code dependency • Python library encoding TranSMART entities as well- defined classes • Ensures data export to transmart-copy compatible format • Flexible tool, fixed output but can be adapted to new input formats (various flavors) TRANSMART 17.X INSTANCE (SLICE) TRANSMART LOADER TRANSMART COPY SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART 17.X INSTANCE Automated ETL pipeline GENERIC DATA
transmart-loader based tools Data flow Code dependency transmart-loader -based mapping tools: • csr2transmart • fhir2transmart • ontology2transmart • claml2transmart TRANSMART 17.X INSTANCE (SLICE) TRANSMART LOADER TRANSMART COPY SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART 17.X INSTANCE Automated ETL pipeline GENERIC DATA https://github.com/thehyve/ transmart-loader
Slicing tool Data flow Code dependency • Developed for DiFuture consortium to populate project-specific data warehouses • Given a set of constraints, automatically extracts corresponding data and loads them to another TranSMART instance TRANSMART 17.X INSTANCE (SLICE) TRANSMART LOADER TRANSMART COPY SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART 17.X INSTANCE Automated ETL pipeline GENERIC DATA
Slicing tool https://github.com/thehyve/transmart-hyper-dicer • Official name: transmart-hyper-dicer (short for “TranSMART hypercube dicer”) Functionality: • Require input JSON file with query constraints • Extract relevant part of the ontology for the given set of data • Populate an EXISTING (empty) TranSMART instance
Conclusions • Transmart 17.X comes with a range of data loading tools, both for for simple manual pipelines, and complex automated ETL pipelines. • Increasingly modular to improve maintainability (separate functions in separate tools) • Development mostly driven by specific ETL projects, but keeping in mind flexibility to create tools easily adaptable to new use case scenarios
TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.X INSTANCE TRANSMART COPY Jupyter Notebook Workshop this afternoon! GENERIC ONTOLOGY ARBORIST (standalone) Manual pipeline Data flow Code dependency PYTHON API CLIENT Jupyter Notebook ROOM B4-221 (next to lunch area) at 14:30 Jupyter Notebook & tools for: • Data Loading to TranSMART • TranSMART API calls
Acknowledgements Gijs Kant Software Architect Ewelina Grudzień Software Engineer Artur Faizullin System Administrator Stefan Payralbe Data Engineer Jochem Bijlard Data Engineer Ward Weistra (former Hyver) Brenda Hijmans, NKI (TMTK contributor)
Building ETL pipelines for tranSMART 17.X - New tools for the data loader

More Related Content

PDF
Vertica And Spark: Connecting Computation And Data
bySpark Summit
 
PPT
LarKC Tutorial at ISWC 2009 - Parallelisation
byLarKC
 
PDF
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
byJonas Traub
 
PDF
Incubating Apache Hivemall
byMakoto Yui
 
PPTX
Data provenance in Hopsworks
byAlexandru Adrian Ormenisan
 
PPTX
File Format Benchmarks - Avro, JSON, ORC, & Parquet
byOwen O'Malley
 
PPTX
tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...
byDavid Peyruc
 
PDF
Ancient To Modern: Upgrading nearly a decade of Plone in public radio
byCristopher Ewing
 
Vertica And Spark: Connecting Computation And Data
bySpark Summit
 
LarKC Tutorial at ISWC 2009 - Parallelisation
byLarKC
 
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
byJonas Traub
 
Incubating Apache Hivemall
byMakoto Yui
 
Data provenance in Hopsworks
byAlexandru Adrian Ormenisan
 
File Format Benchmarks - Avro, JSON, ORC, & Parquet
byOwen O'Malley
 
tranSMART Community Meeting 5-7 Nov 13 - Session 1: Chilly-Mazarin Meeting Ob...
byDavid Peyruc
 
Ancient To Modern: Upgrading nearly a decade of Plone in public radio
byCristopher Ewing
 

Similar to Building ETL pipelines for tranSMART 17.X - New tools for the data loader

PPTX
Panacea presentation - Pangeanic - Budapest
byManuel Herranz
 
PDF
Open ETL for Real-Time Decision Making with Shuai Yuan
byDatabricks
 
PDF
Data Integration Solutions Created By Koneksys
byKoneksys
 
PDF
tranSMART Community Meeting 5-7 Nov 13 - Session 3: transmart-data
byDavid Peyruc
 
PDF
Data Onboarding
bySplunk
 
PDF
Data Onboarding
bySplunk
 
ODP
Python for Data Logistics
byKen Farmer
 
PPTX
Big data or big deal
byeduarderwee
 
PDF
Large Scale ETL for Hadoop and Cloudera Search using Morphlines
bywhoschek
 
PPTX
Project Deimos
bySimon Suo
 
Panacea presentation - Pangeanic - Budapest
byManuel Herranz
 
Open ETL for Real-Time Decision Making with Shuai Yuan
byDatabricks
 
Data Integration Solutions Created By Koneksys
byKoneksys
 
tranSMART Community Meeting 5-7 Nov 13 - Session 3: transmart-data
byDavid Peyruc
 
Data Onboarding
bySplunk
 
Data Onboarding
bySplunk
 
Python for Data Logistics
byKen Farmer
 
Big data or big deal
byeduarderwee
 
Large Scale ETL for Hadoop and Cloudera Search using Morphlines
bywhoschek
 
Project Deimos
bySimon Suo
 

Recently uploaded

PDF
Data Visualization That Drives Action.pdf
byseoxbyteanalytics
 
PDF
How Has Urban Development Affected Living Conditions in Omani Cities?
bynzainb652
 
PPTX
L; EPISIOTOMY AND PERINEAL TEARS 2.pptxf
byahmedsadick344
 
PPTX
Sneha-Verma |MBA Business Analyst Portfolio | 2025
byimsnehavrma
 
PPT
SCHMIDVCS OPERATIONAL CNS IN MODERN SYSTEM.ppt
byjimmylal
 
PPTX
3. PAST TENSE.pptxhsjznxnxxnxnxnxjsusuahssb
byAzmiMonyet
 
PPTX
Sukhmanpreet Kaur – Business Analytics Portfolio
bysukhmanpreetkaur283
 
PPTX
ORACLE ADMIN WORKSHOP PRESENTATION OCP.pptx
bykausaratg
 
PPTX
Oral-Communication_Unit-1_Lesson-2_Verbal-and-Nonverbal-Communication.pptx
byvalleys829
 
PDF
Garden-Life-Brand-Launch-Case-Study-By-Accelero-Agency.pdf
byacceleroagency
 
PDF
The AI is a Mirror.It has pulled back the curtain on a deeply unsettling phe...
byTheta Prime Institute, Cheyenne, Wyoming, USA.
 
PPTX
Mobile Health information Technology.pptx
byrasfike07
 
PPTX
Copy of Criminology Major for College_ Criminal Statistics by Slidesgo.pptx.pptx
bylbrickell1
 
PPTX
Health Insight Survey: an introduction | Accessible slides
byOffice for National Statistics
 
PPTX
Basics of Research Methodology Part1.pptx
byDepartment of Agricultural Marketing
 
PPTX
Microsoft Powerpoint Slide Template Grey Theme.pptx
byw5w5f8wf42
 
PPTX
Fraud_Detection_Using_Technology_Presentation Mehmet sarikurat.pptx
byMasihullahAli1
 
PPTX
OVERVIEW OF RESEARCH AND UNLOCKING INSIGHT
byreginamunsayac
 
PPT
ConfucianismMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM.ppt
byAllenDelarosa2
 
PPT
monocot-vs-dicot.ppt FOR I'M IRRIGATION I LOVE THIS SO BAD BUT THIS ONE WAS
bydaaboulaziz5
 
Data Visualization That Drives Action.pdf
byseoxbyteanalytics
 
How Has Urban Development Affected Living Conditions in Omani Cities?
bynzainb652
 
L; EPISIOTOMY AND PERINEAL TEARS 2.pptxf
byahmedsadick344
 
Sneha-Verma |MBA Business Analyst Portfolio | 2025
byimsnehavrma
 
SCHMIDVCS OPERATIONAL CNS IN MODERN SYSTEM.ppt
byjimmylal
 
3. PAST TENSE.pptxhsjznxnxxnxnxnxjsusuahssb
byAzmiMonyet
 
Sukhmanpreet Kaur – Business Analytics Portfolio
bysukhmanpreetkaur283
 
ORACLE ADMIN WORKSHOP PRESENTATION OCP.pptx
bykausaratg
 
Oral-Communication_Unit-1_Lesson-2_Verbal-and-Nonverbal-Communication.pptx
byvalleys829
 
Garden-Life-Brand-Launch-Case-Study-By-Accelero-Agency.pdf
byacceleroagency
 
The AI is a Mirror.It has pulled back the curtain on a deeply unsettling phe...
byTheta Prime Institute, Cheyenne, Wyoming, USA.
 
Mobile Health information Technology.pptx
byrasfike07
 
Copy of Criminology Major for College_ Criminal Statistics by Slidesgo.pptx.pptx
bylbrickell1
 
Health Insight Survey: an introduction | Accessible slides
byOffice for National Statistics
 
Basics of Research Methodology Part1.pptx
byDepartment of Agricultural Marketing
 
Microsoft Powerpoint Slide Template Grey Theme.pptx
byw5w5f8wf42
 
Fraud_Detection_Using_Technology_Presentation Mehmet sarikurat.pptx
byMasihullahAli1
 
OVERVIEW OF RESEARCH AND UNLOCKING INSIGHT
byreginamunsayac
 
ConfucianismMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM.ppt
byAllenDelarosa2
 
monocot-vs-dicot.ppt FOR I'M IRRIGATION I LOVE THIS SO BAD BUT THIS ONE WAS
bydaaboulaziz5
 

Building ETL pipelines for tranSMART 17.X - New tools for the data loader

  • 1.
    Building ETL pipelines totranSMART 17.X New tools for the data loader Alessia Peviani – Data Engineer
  • 2.
    TRANSMART 17.X INSTANCE (SLICE) TMTK ARBORIST(plugin) TRANSMART LOADER TRANSMART COPY GENERIC DATA SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart TRANSMART 17.X INSTANCE CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART COPY Jupyter Notebook TRANSMART 17.X INSTANCE Tools overview: data flow and dependencies GENERIC ONTOLOGY ARBORIST (standalone) Manual pipeline Automated ETL pipeline Data flow Code dependency GENERIC DATA
  • 3.
    Why Manual Pipelines Manualpipeline • One-time / infrequent data loading • Changing data format • Initial data exploration / modeling • Tools developed internally for data scientists, and to facilitate collaboration with data owners (researchers, clinicians) TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.X INSTANCE TRANSMART COPY Jupyter Notebook GENERIC ONTOLOGY ARBORIST (standalone) Data flow Code dependency
  • 4.
    Why Automated ETLPipelines Data flow Code dependency • Frequent updates • Stable data format • Typically large volumes of data • Tools developed for specific projects, but with flexibility in mind to enable use in future ETL pipelines TRANSMART 17.X INSTANCE (SLICE) TRANSMART LOADER TRANSMART COPY SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART 17.X INSTANCE Automated ETL pipeline GENERIC DATA
  • 5.
    1. Tools formanual data loading TMTK & Arborist, transmart-copy
  • 6.
    TMTK: The TranSMARTToolkit Manual pipeline • Easier collaboration with data owners: • Option to load tree structure, word mapping, metadata from Excel file ( “Template file”) • Interactive tree structure modeling with the Arborist • Updated to support TranSMART 17.X features • Added additional sheets in template file • Added export in transmart-copy format TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.X INSTANCE TRANSMART COPY Jupyter Notebook GENERIC ONTOLOGY ARBORIST (standalone) Data flow Code dependency https://github.com/thehyve/tmtk
  • 7.
    TMTK: The TranSMARTToolkit Input template file
  • 8.
    The Arborist Manual pipeline Twoflavors of interactive tree modeling: • Without leaving Jupyter Notebook (embedded) • By sending tree to web service (standalone) • Share trees with data owners and let them edit • Re-import final result into TMTK • Functionality (both versions): • Add/remove tree node and metadata tags • Edit position and name TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.X INSTANCE TRANSMART COPY Jupyter Notebook GENERIC ONTOLOGY ARBORIST (standalone) Data flow Code dependency https://github.com/thehyve/arborist
  • 9.
  • 10.
  • 11.
    The Arborist: Plug-invs Standalone https://arborist-test-trait.thehyve.net/
  • 12.
    The Arborist: Plug-invs Standalone feedback Data owner
  • 13.
    Data loading withtransmart-copy Manual pipeline TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.X INSTANCE TRANSMART COPY Jupyter Notebook GENERIC ONTOLOGY ARBORIST (standalone) Data flow Code dependency • A simple tool with a simple function • Batch and incremental data loading • What you see (data tables) is what you get (transmart tables) • Supports new TranSMART 17.X features https://github.com/thehyve/transmart-core/tree/dev/transmart-copy
  • 14.
    transmart-copy transmart-copy (TranSMART 17.X database schemasas seen in pgAdmin) https://github.com/thehyve/transmart-core/tree/dev/transmart-copy/src/test/resources/examples/SURVEY0
  • 15.
  • 16.
    transmart-copy vs transmart-batch transmart-copytransmart-batch • A simple tool with a simple function • Batch and incremental data loading • What you see (data tables) is what you get (transmart tables) • Supports new TranSMART 17.X features • Complex, includes validation steps • Batch loading only (inefficient in some cases) • Does not make explicit what your data will look like once in transmart
  • 17.
    2. Tools forautomated ETL pipelines transmart-loader & related tools, hyper-dicer
  • 18.
    transmart-loader Data flow Code dependency •Python library encoding TranSMART entities as well- defined classes • Ensures data export to transmart-copy compatible format • Flexible tool, fixed output but can be adapted to new input formats (various flavors) TRANSMART 17.X INSTANCE (SLICE) TRANSMART LOADER TRANSMART COPY SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART 17.X INSTANCE Automated ETL pipeline GENERIC DATA
  • 19.
    transmart-loader based tools Dataflow Code dependency transmart-loader -based mapping tools: • csr2transmart • fhir2transmart • ontology2transmart • claml2transmart TRANSMART 17.X INSTANCE (SLICE) TRANSMART LOADER TRANSMART COPY SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART 17.X INSTANCE Automated ETL pipeline GENERIC DATA https://github.com/thehyve/ transmart-loader
  • 20.
    Slicing tool Data flow Codedependency • Developed for DiFuture consortium to populate project-specific data warehouses • Given a set of constraints, automatically extracts corresponding data and loads them to another TranSMART instance TRANSMART 17.X INSTANCE (SLICE) TRANSMART LOADER TRANSMART COPY SLICING TOOL csr 2transmart fhir 2transmart ontology 2transmart claml 2transmart CSR DATA FHIR DATA GENERIC ONTOLOGY CLAML ONTOLOGY ETL pipeline TRANSMART 17.X INSTANCE Automated ETL pipeline GENERIC DATA
  • 21.
    Slicing tool https://github.com/thehyve/transmart-hyper-dicer • Officialname: transmart-hyper-dicer (short for “TranSMART hypercube dicer”) Functionality: • Require input JSON file with query constraints • Extract relevant part of the ontology for the given set of data • Populate an EXISTING (empty) TranSMART instance
  • 22.
    Conclusions • Transmart 17.Xcomes with a range of data loading tools, both for for simple manual pipelines, and complex automated ETL pipelines. • Increasingly modular to improve maintainability (separate functions in separate tools) • Development mostly driven by specific ETL projects, but keeping in mind flexibility to create tools easily adaptable to new use case scenarios
  • 23.
    TMTK ARBORIST (plugin) GENERIC DATA TRANSMART 17.XINSTANCE TRANSMART COPY Jupyter Notebook Workshop this afternoon! GENERIC ONTOLOGY ARBORIST (standalone) Manual pipeline Data flow Code dependency PYTHON API CLIENT Jupyter Notebook ROOM B4-221 (next to lunch area) at 14:30 Jupyter Notebook & tools for: • Data Loading to TranSMART • TranSMART API calls
  • 24.
    Acknowledgements Gijs Kant Software Architect EwelinaGrudzień Software Engineer Artur Faizullin System Administrator Stefan Payralbe Data Engineer Jochem Bijlard Data Engineer Ward Weistra (former Hyver) Brenda Hijmans, NKI (TMTK contributor)