{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T21:55:25Z","timestamp":1779141325853,"version":"3.51.4"},"reference-count":31,"publisher":"Wiley","issue":"8","license":[{"start":{"date-parts":[[2015,8,4]],"date-time":"2015-08-04T00:00:00Z","timestamp":1438646400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/2.zoppoz.workers.dev:443\/http\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/501100003593","name":"CNPq","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100002322","name":"CAPES","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002322","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004586","name":"FAPERJ","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004586","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100012950","name":"Inria","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100012950","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Concurrency and Computation"],"published-print":{"date-parts":[[2016,6,10]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>Computer simulations may ingest and generate high numbers of raw data files. Most of these files follow a <jats:italic>de facto<\/jats:italic> standard format established by the application domain, for example, Flexible Image Transport System for astronomy. Although these formats are supported by a variety of programming languages, libraries, and programs, analyzing thousands or millions of files requires developing specific programs. Database management systems (DBMS) are not suited for this, because they require loading the raw data and structuring it, which becomes heavy at large scale. Systems like NoDB, RAW, and FastBit have been proposed to index and query raw data files without the overhead of using a database management system. However, these solutions are focused on analyzing one single large file instead of several related files. In this case, when related files are produced and required for analysis, the relationship among elements within file contents must be managed manually, with specific programs to access raw data. Thus, this data management may be time\u2010consuming and error\u2010prone. When computer simulations are managed by a scientific workflow management system (SWfMS), they can take advantage of provenance data to relate and analyze raw data files produced during workflow execution. However, SWfMS registers provenance at a coarse grain, with limited analysis on elements from raw data files. When the SWfMS is dataflow\u2010aware, it can register provenance data and the relationships among elements of raw data files altogether in a database, which is useful to access the contents of a large number of files. In this paper, we propose a dataflow approach for analyzing element data from several related raw data files. Our approach is complementary to the existing single raw data file analysis approaches. We use the Montage workflow from astronomy and a workflow from Oil and Gas domain as data\u2010intensive case studies. Our experimental results for the Montage workflow explore different types of raw data flows like showing all linear transformations involved in projection simulation programs, considering specific mosaic elements from input repositories. The cost for raw data extraction is approximately 3.7% of the total application execution time. Copyright \u00a9 2015 John Wiley &amp; Sons, Ltd.<\/jats:p>","DOI":"10.1002\/cpe.3616","type":"journal-article","created":{"date-parts":[[2015,8,4]],"date-time":"2015-08-04T21:05:57Z","timestamp":1438722357000},"page":"2528-2545","source":"Crossref","is-referenced-by-count":12,"title":["Analyzing related raw data files through dataflows"],"prefix":"10.1002","volume":"28","author":[{"given":"V\u00edtor","family":"Silva","sequence":"first","affiliation":[{"name":"Department of Computer Science Federal University of Rio de Janeiro\/COPPE  Rio de Janeiro Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"de Oliveira","sequence":"additional","affiliation":[{"name":"Institute of Computing Fluminense Federal University  Niter\u00f3i Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Patrick","family":"Valduriez","sequence":"additional","affiliation":[{"name":"Zenith Team Inria and LIRMM  Montpellier France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marta","family":"Mattoso","sequence":"additional","affiliation":[{"name":"Department of Computer Science Federal University of Rio de Janeiro\/COPPE  Rio de Janeiro Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2015,8,4]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.331.6018.666"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1051\/0004-6361:20021326"},{"key":"e_1_2_8_4_1","unstructured":"Unidata Program Center.NetCDF 2014. (Available from:https:\/\/2.zoppoz.workers.dev:443\/http\/www.unidata.ucar.edu\/software\/netcdf\/.) (accessed on 30 June 2015)"},{"key":"e_1_2_8_5_1","unstructured":"The HDF Group.HDF5 2014. (Available from:https:\/\/2.zoppoz.workers.dev:443\/http\/www.hdfgroup.org\/HDF5\/.) (accessed on 30 June 2015)"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJCSE.2009.026999"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/180\/1\/012053"},{"key":"e_1_2_8_8_1","doi-asserted-by":"crossref","unstructured":"ChouJ RyneRD HowisonM AustinB WuK QiangJ BethelEW ShoshaniA R\u00fcbelOet al.Parallel index and query for large scale data analysis.2011 International Conference for High Performance Computing Networking Storage and Analysis Seattle 2011; p. 1.","DOI":"10.1145\/2063384.2063424"},{"key":"e_1_2_8_9_1","doi-asserted-by":"crossref","unstructured":"BlanasS WuK BynaS DongB ShoshaniA.Parallel data analysis directly on scientific file formats.2014 ACM SIGMOD International Conference on Management of Data Snowbird 2014;385\u2013396.","DOI":"10.1145\/2588555.2612185"},{"key":"e_1_2_8_10_1","doi-asserted-by":"crossref","unstructured":"AlagiannisI BorovicaR BrancoM IdreosS AilamakiA.NoDB: efficient query execution on raw data files 2012; p. 241.","DOI":"10.1145\/2213836.2213864"},{"key":"e_1_2_8_11_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732977.2732986"},{"key":"e_1_2_8_12_1","doi-asserted-by":"crossref","unstructured":"VahiK RyngeM JuveG MayaniR DeelmanE.Rethinking data management for big data scientific workflows.Workshop on Big Data and Science:Infrastructure and Services Silicon Valley 2013;27\u201335.","DOI":"10.1109\/BigData.2013.6691724"},{"key":"e_1_2_8_13_1","doi-asserted-by":"crossref","unstructured":"AssuncaoL CunhaJC.Enabling global experiments with interactive reconfiguration and steering by multiple users.14th International Conference on Computational Science Cairns vol.29 2014;2137\u20132144.","DOI":"10.1016\/j.procs.2014.05.198"},{"key":"e_1_2_8_14_1","doi-asserted-by":"crossref","unstructured":"BowersS McphillipsT RiddleS AnandMK Lud\u00e4scherB.Kepler\/pPOD: scientific workflow and provenance support for assembling the tree of life.2nd International Provenance and Annotation Workshop Salt Lake City 2008;70\u201377.","DOI":"10.1007\/978-3-540-89965-5_9"},{"key":"e_1_2_8_15_1","doi-asserted-by":"crossref","unstructured":"IkedaR ChoJ FangC SalihogluS TorikaiS WidomJ.Provenance\u2010based debugging and drill\u2010down in data\u2010oriented workflows.IEEE 28th International Conference on Data Engineering(ICDE) Arlington 2012;1249\u20131252.","DOI":"10.1109\/ICDE.2012.118"},{"issue":"16","key":"e_1_2_8_16_1","first-page":"2327","article-title":"Chiron: a parallel engine for algebraic scientific workflows","volume":"25","author":"Ogasawara E","year":"2013","journal-title":"CCPE"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2014.11.017"},{"key":"e_1_2_8_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2008.79"},{"key":"e_1_2_8_19_1","doi-asserted-by":"crossref","unstructured":"SilvaV OliveiraD MattosoM.Exploratory analysis of raw data files through dataflows.Workshop on Parallel and Distributed Computing for Big Data Applications(WPBA 2014) Paris 2014;114\u2013119.","DOI":"10.1109\/SBAC-PADW.2014.32"},{"key":"e_1_2_8_20_1","unstructured":"KimJ AbbasiH ChaconL DocanC KlaskyS LiuQ PodhorszkiN ShoshaniA WuK.Parallelin situindexing for data\u2010intensive computing.IEEE Symposium on Large Data Analysis and Visualization(LDAV) Providence 2011;65\u201372."},{"key":"e_1_2_8_21_1","doi-asserted-by":"crossref","unstructured":"MaB ShoshaniA SimA WuK ByunY HahmJ ShinM\u2010S.Efficient attribute\u2010based data access in astronomy analysis.SC Companion:High Performance Computing Networking Storage and Analysis(SCC) 2012;562\u2013571.","DOI":"10.1109\/SC.Companion.2012.80"},{"key":"e_1_2_8_22_1","unstructured":"Amazon EC2.Amazon Elastic Compute Cloud (Amazon EC2) 2010. (Available from:https:\/\/2.zoppoz.workers.dev:443\/http\/aws.amazon.com\/ec2\/.) (accessed on 30 June 2015.)"},{"key":"e_1_2_8_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2010.9"},{"key":"e_1_2_8_24_1","doi-asserted-by":"crossref","unstructured":"OgasawaraE DiasJ OliveiraD PortoF ValduriezP MattosoM.An algebraic approach for data\u2010centric scientific workflows.37th International Conference on Very Large Data Bases(PVLDB) Seattle vol.4(12) 2011;1328\u20131339.","DOI":"10.14778\/3402755.3402766"},{"key":"e_1_2_8_25_1","doi-asserted-by":"crossref","unstructured":"IkedaR Das SarmaA WidomJ.Logical provenance in data\u2010oriented workflows?2013 IEEE International Conference on Data Engineering(ICDE 2013) Brisbane 2013;877\u2013888.","DOI":"10.1109\/ICDE.2013.6544882"},{"key":"e_1_2_8_26_1","doi-asserted-by":"crossref","unstructured":"CostaF SilvaV deOliveiraD Oca\u00f1aK OgasawaraE DiasJ MattosoM.Capturing and querying workflow runtime provenance with PROV: a practical approach.Joint EDBT\/ICDT 2013 \u2010 Workshops on EDBT'13 Genoa 2013;282\u2013289.","DOI":"10.1145\/2457317.2457365"},{"key":"e_1_2_8_27_1","doi-asserted-by":"crossref","unstructured":"MissierP BelhajjameK CheneyJ.The W3C PROV family of specifications for modelling provenance metadata.16th International Conference on Extending Database Technology Genoa 2013;773\u2013776.","DOI":"10.1145\/2452376.2452478"},{"key":"e_1_2_8_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0168-9002(97)00048-X"},{"key":"e_1_2_8_29_1","doi-asserted-by":"crossref","unstructured":"DeelmanE SinghG LivnyM BerrimanB GoodJ.The cost of doing science on the cloud: the Montage example.SC'08:2008 ACM\/IEEE Conference on Supercomputing Austin 2008;1\u201312.","DOI":"10.1109\/SC.2008.5217932"},{"key":"e_1_2_8_30_1","unstructured":"NASA\/IPAC.Two Micron All Sky Survey (2MASS) 2014. (Available from:https:\/\/2.zoppoz.workers.dev:443\/http\/irsa.ipac.caltech.edu.) (accessed on 30 June 2015.)"},{"key":"e_1_2_8_31_1","doi-asserted-by":"crossref","unstructured":"ChirigatiF SilvaV OgasawaraE OliveiraD DiasJ PortoF ValduriezP MattosoM.Evaluating parameter sweep workflows in high performance computing.1st International Workshop on Scalable Workflow Enactment Engines and Technologies(SWEET'12) Scottsdale 2012; p. 10.","DOI":"10.1145\/2443416.2443418"},{"key":"e_1_2_8_32_1","unstructured":"NACAD.NACAD: high performance computing center 2015. (Available from:https:\/\/2.zoppoz.workers.dev:443\/http\/www.nacad.ufrj.br\/accessed on 30 June 2015.)"}],"container-title":["Concurrency and Computation: Practice and Experience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fcpe.3616","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cpe.3616","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,12]],"date-time":"2023-09-12T21:58:09Z","timestamp":1694555889000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/10.1002\/cpe.3616"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,8,4]]},"references-count":31,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2016,6,10]]}},"alternative-id":["10.1002\/cpe.3616"],"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1002\/cpe.3616","archive":["Portico"],"relation":{},"ISSN":["1532-0626","1532-0634"],"issn-type":[{"value":"1532-0626","type":"print"},{"value":"1532-0634","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,8,4]]}}}