How Data Commons are Changing the Way that Large Datasets Are Analyzed and Shared
1 like646 views
The document discusses the evolution and impact of data commons in genomic and clinical datasets from 2014 to 2024, highlighting their role in enhancing data analysis and sharing within the research community. It emphasizes the integration of data, storage, and computing resources with user-defined applications, facilitating collaborative analysis and encouraging the development of new tools through APIs. The NCI Genomic Data Commons is presented as a case study, illustrating how data commons are transforming the landscape of biomedical research through open-source infrastructure and governance models.
More Related Content
Similar to How Data Commons are Changing the Way that Large Datasets Are Analyzed and Shared (20)
How Data Commons are Changing the Way that Large Datasets Are Analyzed and Shared
- 5. Deep Learning (DL) Dogma • Use Deep Neural Networks (DNN) with lots of layers (10’s to 100’s of layers). • Try using Convolutional Neural Networks (CNN), even if the problem is not translation invariant. • Represent the inputs and internal states as long vectors of numbers (even if the input is an image, text, spoken voice, etc. that has structure) • Train with very large amounts of labeled data. • Don’t worry about the internal structure of the model, just its accuracy and coverage.
- 6. DNN (2016) Data Mining (1996) Use lots of parameters 10’s to 100’s of hidden layers Large trees, large ensembles Long vectors for inputs Part of the dogma Generally the case Use as much data as possible Yes Yes Do we care about the internal structure of the model? No No Hardware GPU & custom chips Clusters of workstations Labeled data Often the limiting factor Often the limiting factor Features Not needed Generally the hard part
- 7. • In some sense, Deep Learning is eating the world. • Compare: Marc Andressen, Why Software Is Eating The World, Wall Street Journal, August 20, 2011. • From a broader perspective, Machine Learning (ML) continues to eat the world, as it has been doing for the last 20 years drive by the exponentially growth in the amount of data and the computational power available to estimate parameters.
- 10. NCI Genomic Data Commons* • Launched in 2016 with over 4 PB of data. • Joint project with OICR. • Used by 1500 - 2000+ users per day. • Based upon an open source software stack that can be used to build other data commons.*See: NCI Genomic Data Commons: Grossman, Robert L., et al. "Toward a shared vision for cancer genomic data." New England Journal of Medicine 375.12 (2016): 1109-1112.
- 13. • MuSE (MD Anderson) • VarScan2 (Washington Univ.) • SomaticSniper (Washington Univ.) • MuTect2 (Broad Institute) Source: Zhenyu Zhang, et. al. and the GDC Project Team, Uniform Genomic Data Analysis in the NCI Genomic Data Commons, to appear. System 2: Data Harmonization System To Analyze all of the Submitted Data with a Common Pipelines
- 14. System 3: User Defined Applications and Notebooks to Create a Data Ecosystem https://gdc-api.nci.nih.gov/files/5003adf1-1cfd-467d-8234-0d396422a4ee?fields=state • The GDC has a REST API so that researchers can develop their own applications. • There are third party applications that use the REST API for Python, R, Jupyter notebooks and Shiny. • The REST API drives the GDC data portal, data submission system, etc.
- 15. GDC Application Programming Interface (API) – To Build Applications https://gdc-api.nci.nih.gov/files/5003adf1-1cfd-467d-8234-0d396422a4ee?fields=state API URL Endpoint Optional Entity ID Query parameters • Based upon data model • Drives internally developed applications, e.g. data portal • Allows third parties to develop their own applications • Can be used by other commons, by workspaces, by other systems, by user-developed applications and notebooks
- 17. • Supports big data with cloud computing • Researchers can analyze data with collaborative tools (workspaces) – i. e. data does not have to be downloaded) • Data repository • Researchers download data. Databases Data Clouds Data Commons • Supports big data • Workspaces • Common data models • Core data services • Harmonized data • Governance 1982 - present 2010 - 2020 2014 - 2024
- 19. OCC Open Science Data Cloud (2010) OCC – NASA Project Matsu (2009) NCI Genomic Data Commons* (2016) OCC-NOAA Environmental Data Commons (2016) OCC Blood Profiling Atlas in Cancer (2017) Bionimbus Protected Data Cloud* (2013) *Operated under a subcontract from NCI / Leidos Biomedical to the University of Chicago with support from the OCC. Brain Commons (2017) Kids First Data Resource (2017) Gen3 Gen2 Gen1
- 20. The Gen3 Data Model Is Customizable & Extensible • BloodPAC • Brain Commons • Wellness Commons • Kids First Data Resource
- 22. Core Data Commons Framework Services • Digital ID services • Metadata services • Authentication services • Authorization services • Designed to span multiple data commons • Designed to support multiple private and commercial clouds • In the future, we will support portable workspaces
- 24. Data Commons Framework Services Private Academic Cloud Univ. of Chicago CSOC (ops center) Cross Cloud Services
- 26. Summary 1. Designed to support disease specific, project specific or consortium specific data commons, including governance model. 2. Designed to support multiple data commons that peer and interoperate. 3. Designed to support an ecosystem of FAIR-based applications. 4. The core underlying software stack is open source. 5. Data commons governance model in which data is public and you “pay for compute”. 6. Supported by the independent not-for-profit Open Commons Consortium.
- 28. Data Commons 2014 - 2024 Data Clouds 2010 - 2020 Data Ecosystems 2018 - 2028 Databases 1982 - present
- 29. Three Large Scale Data Commons That are Working Towards Common APIs 1. NCI GDC / Cloud Resources (UChicago / Broad) 2. NIH All of Us (Broad / Verily) 3. CZI HCA Data Platform (UCSC/Broad) For more information, see: Josh Denny, David Glazer, Robert L. Grossman, Benedict Paten & Anthony Philippakis, A Data Biosphere for Biomedical Research, https://medium.com/@benedictpaten/a-data-biosphere-for-biomedical-research- d212bbfae95d. Also available at: https://goo.gl/9CySeo
- 31. 100% 75% 70% 50% 0% 25% 25% 25% 0% 0% 5% 25% 0% 20% 40% 60% 80% 100% 120% 1995 2005 2015 2025 Hardware / data production Software / data analysis Data ecosystem / data reuse
- 34. For more information: • To learn more about data commons: Robert L. Grossman, et. al. A Case for Data Commons: Toward Data Science as a Service, Computing in Science & Engineering 18.5 (2016): 10-20. Also https://arxiv.org/abs/1604.02608 • To large more about large scale, secure compliant cloud based computing environments for biomedical data, see: Heath, Allison P., et al. "Bionimbus: a cloud for managing, analyzing and sharing large genomics datasets." Journal of the American Medical Informatics Association 21.6 (2014): 969-975. This article describes Bionimbus Gen1. • To learn more about the NCI Genomic Data Commons: Grossman, Robert L., et al. "Toward a shared vision for cancer genomic data." New England Journal of Medicine 375.12 (2016): 1109-1112. The GDC was developed using Bionimbus Gen2. • To learn more about BloodPAC, Grossman, R. L., et al. "Collaborating to compete: Blood Profiling Atlas in Cancer (BloodPAC) Consortium." Clinical Pharmacology & Therapeutics (2017). BloodPAC was developed using the GDC Community Edition (CE) aka Bionimbus Gen3