{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T23:28:36Z","timestamp":1774999716443,"version":"3.50.1"},"reference-count":38,"publisher":"Wiley","issue":"16","license":[{"start":{"date-parts":[[2021,3,2]],"date-time":"2021-03-02T00:00:00Z","timestamp":1614643200000},"content-version":"am","delay-in-days":0,"URL":"https:\/\/2.zoppoz.workers.dev:443\/http\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"},{"start":{"date-parts":[[2021,3,2]],"date-time":"2021-03-02T00:00:00Z","timestamp":1614643200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/2.zoppoz.workers.dev:443\/http\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000015","name":"U.S. Department of Energy","doi-asserted-by":"publisher","award":["DE\u2010AC02\u201005CH11231"],"award-info":[{"award-number":["DE\u2010AC02\u201005CH11231"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Concurrency and Computation"],"published-print":{"date-parts":[[2021,8,25]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>Scientific datasets are growing rapidly and becoming critical to next\u2010generation scientific discoveries. The validity of scientific results relies on the quality of data used and data are often subject to change, for example, due to observation additions, quality assessments, or processing software updates. The effects of data change are not well understood and difficult to predict. Datasets are often repeatedly updated and recomputing derived data products quickly becomes time consuming and resource intensive and may in some cases not even be necessary, thus delaying scientific advance. Despite its importance, there is a lack of systematic approaches for best comparing data versions to quantify the changes, and ad\u2010hoc or manual processes are commonly used. In this article, we propose a novel hierarchical approach for analyzing data changes, including real\u2010time (online) and offline analyses. We employ a variety of fast\u2010to\u2010compute numerical analyses, graphical data change representations, and more resource\u2010intensive recomputations of a subset of the data product. We illustrate the application of our approach using three scientific diverse use cases, namely, satellite, cosmological, and x\u2010ray data. The results show that a variety of data change metrics should be employed to enable a comprehensive representation and qualitative evaluation of data changes.<\/jats:p>","DOI":"10.1002\/cpe.6245","type":"journal-article","created":{"date-parts":[[2021,3,3]],"date-time":"2021-03-03T06:26:30Z","timestamp":1614752790000},"update-policy":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Assessing data change in scientific datasets"],"prefix":"10.1002","volume":"33","author":[{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0001-8627-1992","authenticated-orcid":false,"given":"Juliane","family":"M\u00fcller","sequence":"first","affiliation":[{"name":"Computational Research Division Lawrence Berkeley National Laboratory  Berkeley California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Boris","family":"Faybishenko","sequence":"additional","affiliation":[{"name":"Earth Sciences Division Lawrence Berkeley National Laboratory  Berkeley California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Deborah","family":"Agarwal","sequence":"additional","affiliation":[{"name":"Computational Research Division Lawrence Berkeley National Laboratory  Berkeley California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephen","family":"Bailey","sequence":"additional","affiliation":[{"name":"Physics Division Lawrence Berkeley National Laboratory  Berkeley California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chongya","family":"Jiang","sequence":"additional","affiliation":[{"name":"Seoul National University  Seoul Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Youngryel","family":"Ryu","sequence":"additional","affiliation":[{"name":"Seoul National University  Seoul Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Craig","family":"Tull","sequence":"additional","affiliation":[{"name":"Computational Research Division Lawrence Berkeley National Laboratory  Berkeley California"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lavanya","family":"Ramakrishnan","sequence":"additional","affiliation":[{"name":"Computational Research Division Lawrence Berkeley National Laboratory  Berkeley California"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2021,3,2]]},"reference":[{"key":"e_1_2_8_2_1","volume-title":"Surfacing Data Change in Scientific Work","author":"Paine D","year":"2018"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rse.2016.12.018"},{"key":"e_1_2_8_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rse.2016.08.030"},{"key":"e_1_2_8_5_1","doi-asserted-by":"crossref","unstructured":"GhoshalD RamakrishnanL AgarwalD. DAC\u2010MAN: data change management for scientific datasets on HPC systems;2018.","DOI":"10.1109\/SC.2018.00075"},{"key":"e_1_2_8_6_1","doi-asserted-by":"crossref","unstructured":"PaineD RamakrishnanL. Surfacing data change in scientific work. Paper presented at: Proceedings of the International Conference on Information;2019:15\u201026; Springer New York NY.","DOI":"10.1007\/978-3-030-15742-5_2"},{"key":"e_1_2_8_7_1","first-page":"707","article-title":"Binary codes capable of correcting deletions, insertions, and reversals","volume":"10","author":"Levenshtein VI","year":"1966","journal-title":"Sov Phys Dokl"},{"key":"e_1_2_8_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1950.tb00463.x"},{"key":"e_1_2_8_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/375360.375365"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.1137\/1115049"},{"key":"e_1_2_8_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2004.837550"},{"key":"e_1_2_8_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-2070(00)00057-1"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijforecast.2006.03.001"},{"key":"e_1_2_8_14_1","volume-title":"Bioinformatics: Sequence and Genome Analysis","author":"Mount DM","year":"2004"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.3354\/cr030079"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611970586"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v008.i18"},{"key":"e_1_2_8_18_1","unstructured":"NASAMODIS website;https:\/\/2.zoppoz.workers.dev:443\/https\/modis.gsfc.nasa.gov\/."},{"key":"e_1_2_8_19_1","volume-title":"Python and HDF5: Unlocking Scientific Data","author":"Collette A","year":"2013"},{"key":"e_1_2_8_20_1","doi-asserted-by":"publisher","DOI":"10.1029\/2011GB004053"},{"key":"e_1_2_8_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2013.09.023"},{"key":"e_1_2_8_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0034-4257(02)00135-9"},{"key":"e_1_2_8_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rse.2016.11.004"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rse.2014.09.023"},{"key":"e_1_2_8_25_1","unstructured":"Sloan Digital Sky Survey websitehttps:\/\/2.zoppoz.workers.dev:443\/http\/www.sdss.org\/."},{"key":"e_1_2_8_26_1","first-page":"53","article-title":"The FITS image extension","volume":"105","author":"Ponz J","year":"1994","journal-title":"Astronomy Astrophys Suppl Ser"},{"key":"e_1_2_8_27_1","doi-asserted-by":"publisher","DOI":"10.1088\/1475-7516\/2018\/11\/015"},{"key":"e_1_2_8_28_1","doi-asserted-by":"publisher","DOI":"10.3847\/2041-8213\/ab133f"},{"key":"e_1_2_8_29_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature25180"},{"key":"e_1_2_8_30_1","doi-asserted-by":"publisher","DOI":"10.3847\/0004-637X\/832\/2\/103"},{"key":"e_1_2_8_31_1","unstructured":"Advanced light source websitehttps:\/\/2.zoppoz.workers.dev:443\/https\/als.lbl.gov\/."},{"key":"e_1_2_8_32_1","doi-asserted-by":"publisher","DOI":"10.1107\/S0909049502017739"},{"key":"e_1_2_8_33_1","doi-asserted-by":"publisher","DOI":"10.1107\/S1600577514015033"},{"key":"e_1_2_8_34_1","doi-asserted-by":"publisher","DOI":"10.1107\/S1600577514007000"},{"key":"e_1_2_8_35_1","doi-asserted-by":"publisher","DOI":"10.1111\/gcb.13787"},{"issue":"12","key":"e_1_2_8_36_1","first-page":"27","article-title":"The eleventh and twelfth data releases of the Sloan digital sky survey: final data from SDSS\u2010III","volume":"219","author":"Alam S","year":"2015","journal-title":"Astrophys J Suppl Ser"},{"key":"e_1_2_8_37_1","article-title":"Astropy: a community Python package for astronomy","volume":"558","author":"Robitaille TP","year":"2013","journal-title":"Astron Astrophys"},{"key":"e_1_2_8_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/0013-4694(92)90009-7"},{"key":"e_1_2_8_39_1","doi-asserted-by":"publisher","DOI":"10.1107\/S0021889813000150"}],"container-title":["Concurrency and Computation: Practice and Experience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cpe.6245","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/cpe.6245","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cpe.6245","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cpe.6245","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,29]],"date-time":"2023-08-29T15:16:28Z","timestamp":1693322188000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/10.1002\/cpe.6245"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,2]]},"references-count":38,"journal-issue":{"issue":"16","published-print":{"date-parts":[[2021,8,25]]}},"alternative-id":["10.1002\/cpe.6245"],"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1002\/cpe.6245","archive":["Portico"],"relation":{},"ISSN":["1532-0626","1532-0634"],"issn-type":[{"value":"1532-0626","type":"print"},{"value":"1532-0634","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,2]]},"assertion":[{"value":"2019-12-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-02","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-03-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e6245"}}