{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T09:45:20Z","timestamp":1753868720377,"version":"3.41.2"},"reference-count":40,"publisher":"Wiley","issue":"9","license":[{"start":{"date-parts":[[2019,12,19]],"date-time":"2019-12-19T00:00:00Z","timestamp":1576713600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/2.zoppoz.workers.dev:443\/http\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["U1611261"],"award-info":[{"award-number":["U1611261"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Concurrency and Computation"],"published-print":{"date-parts":[[2020,5,10]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>As data analysis scenarios keep increasing on high\u2010performance computing systems, the ability to select a small fraction of data from a large volume of scientific data sets is vital to accelerate scientific discovery. However, parallel file systems lack the ability to provide efficient data locating services at the granularity of both a file and a record. Existing methods for identifying and indexing data are often domain\u2010specific and do not scale to large scientific data sets. In this paper, we describe the design and implementation of UniIndex framework, which combines our proposed techniques for user\u2010annotation extraction, in\u2010memory cache layer, in\u2010situ indexing, and parallel query processing. Acting as middleware on top of production file systems, UniIndex enables efficient data locating services with minimal user effort. Our evaluations show that UniIndex can locate target files from directories containing millions of files in microseconds. By applying in situ indexing and the lightweight range\u2010bitmap index, record\u2010level index building time can be dramatically reduced while maintaining up to two orders of magnitude query speedup than scanning the entire data set.<\/jats:p>","DOI":"10.1002\/cpe.5609","type":"journal-article","created":{"date-parts":[[2019,12,20]],"date-time":"2019-12-20T05:36:09Z","timestamp":1576820169000},"update-policy":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["UniIndex: An index and query middleware for parallel file systems"],"prefix":"10.1002","volume":"32","author":[{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0001-5126-9956","authenticated-orcid":false,"given":"Peng","family":"Cheng","sequence":"first","affiliation":[{"name":"College of Computer National University of Defense Technology  Changsha China"},{"name":"State Key Laboratory of High Performance Computing  Changsha China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yong","family":"Wang","sequence":"additional","affiliation":[{"name":"National Supercomputer Center in Guangzhou School of Data and Computer Science, Sun Yat\u2010Sen University  Guangzhou China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yutong","family":"Lu","sequence":"additional","affiliation":[{"name":"National Supercomputer Center in Guangzhou School of Data and Computer Science, Sun Yat\u2010Sen University  Guangzhou China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yunfei","family":"Du","sequence":"additional","affiliation":[{"name":"National Supercomputer Center in Guangzhou School of Data and Computer Science, Sun Yat\u2010Sen University  Guangzhou China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiguang","family":"Chen","sequence":"additional","affiliation":[{"name":"National Supercomputer Center in Guangzhou School of Data and Computer Science, Sun Yat\u2010Sen University  Guangzhou China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2019,12,19]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"crossref","unstructured":"SewellC HeitmannK FinkelH et al.Large\u2010scale compute\u2010intensive analysis via a combined in\u2010situ and co\u2010scheduling workflow approach. In: Proceedings of the International Conference for High Performance Computing Networking Storage and Analysis;2015;Austin TX.","DOI":"10.1145\/2807591.2807663"},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2016.2602560"},{"key":"e_1_2_9_4_1","doi-asserted-by":"crossref","unstructured":"TaftR VartakM SatishNR SundaramN MaddenS StonebrakerM.GenBase: a complex analytics genomics benchmark. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data;2014;Snowbird UT.","DOI":"10.1145\/2588555.2595633"},{"key":"e_1_2_9_5_1","unstructured":"SchwanP.Lustre: building a file system for 1000\u2010node clusters. In: Proceedings of the 2003 Linux Symposium;2003;Ottawa Canada."},{"key":"e_1_2_9_6_1","unstructured":"SchmuckFB HaskinRL.GPFS: a shared\u2010disk file system for large computing clusters. In: Proceedings of the 1st USENIX Conference on File and Storage Technologies;2002;Monterey CA."},{"key":"e_1_2_9_7_1","unstructured":"The HDF Group.The HDF5 Library and File Format.https:\/\/2.zoppoz.workers.dev:443\/https\/www.hdfgroup.org\/solutions\/hdf5. Accessed June 12 2019."},{"key":"e_1_2_9_8_1","doi-asserted-by":"crossref","unstructured":"LiJ LiaoW\u2010K ChoudharyAN et al.Parallel netCDF: a high\u2010performance scientific I\/O interface. In: Proceedings of the 2003 ACM\/IEEE Conference on Supercomputing;2003;Phoenix AZ.","DOI":"10.1145\/1048935.1050189"},{"key":"e_1_2_9_9_1","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/180\/1\/012053"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2008.10.003"},{"key":"e_1_2_9_11_1","doi-asserted-by":"crossref","unstructured":"WangDL MonkewitzSM LimK\u2010T BeclaJ.Qserv: a distributed shared\u2010nothing database for the LSST catalog. In: Conference on High Performance Computing Networking Storage and Analysis \u2010 State of the Practice Reports;2011;Seattle WA.","DOI":"10.1145\/2063348.2063364"},{"key":"e_1_2_9_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2013.19"},{"key":"e_1_2_9_13_1","doi-asserted-by":"crossref","unstructured":"SimH KimY VazhkudaiSS Vall\u00e9eGR LimS\u2010H ButtAR.Tagit: an integrated indexing and search service for file systems. In: Proceedings of the International Conference for High Performance Computing Networking Storage and Analysis;2017;Denver CO.","DOI":"10.1145\/3126908.3126929"},{"key":"e_1_2_9_14_1","doi-asserted-by":"crossref","unstructured":"TangH BynaS DongB LiuJ KoziolQ.SoMeta: scalable object\u2010centric metadata management for high performance computing. Paper presented at: 2017 IEEE International Conference on Cluster Computing (CLUSTER);2017;Honolulu HI.","DOI":"10.1109\/CLUSTER.2017.53"},{"key":"e_1_2_9_15_1","unstructured":"JohnsonC KeetonK MorreyCBIII et al.From research to practice: experiences engineering a production metadata database for a scale out file system. In: Proceedings of the 12th USENIX conference on File and Storage Technologies;2014;Santa Clara CA."},{"key":"e_1_2_9_16_1","doi-asserted-by":"crossref","unstructured":"ChouJ HowisonM AustinB et al.Parallel index and query for large scale data analysis. In: Proceedings of 2011 International Conference for High Performance Computing Networking Storage and Analysis;2011;Seattle WA.","DOI":"10.1145\/2063384.2063424"},{"key":"e_1_2_9_17_1","doi-asserted-by":"crossref","unstructured":"WuT ChouJ HaoS DongB KlaskyS WuK.Optimizing the query performance of block index through data analysis and I\/O modeling. In: Proceedings of the International Conference for High Performance Computing Networking Storage and Analysis;2017;Denver CO.","DOI":"10.1145\/3126908.3126934"},{"key":"e_1_2_9_18_1","doi-asserted-by":"crossref","unstructured":"GuJ KlaskyS PodhorszkiN QiangJ WuK.Querying large scientific data sets with adaptable IO system ADIOS. In: Proceedings of the Supercomputing Frontiers \u2010 4th Asian Conference (SCFA'18);2018;Singapore.","DOI":"10.1007\/978-3-319-69953-0_4"},{"key":"e_1_2_9_19_1","doi-asserted-by":"crossref","unstructured":"KimJ AbbasiH Chac\u00f3nL. et al.Parallel in situ indexing for data\u2010intensive computing. Paper presented at: 2011 IEEE Symposium on Large Data Analysis and Visualization;2011;Providence RI.","DOI":"10.1109\/LDAV.2011.6092319"},{"key":"e_1_2_9_20_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.3125"},{"issue":"10","key":"e_1_2_9_21_1","first-page":"133","article-title":"OSG\u2010GEM: gene expression matrix construction using the open science grid","volume":"10","author":"Poehlman WL","year":"2016","journal-title":"Bioinform Biol Insights"},{"key":"e_1_2_9_22_1","unstructured":"Leveldb a light\u2010weight single\u2010purpose library for persistence with bindings to many platforms.https:\/\/2.zoppoz.workers.dev:443\/http\/leveldb.org\/. Accessed October 25 2018."},{"key":"e_1_2_9_23_1","doi-asserted-by":"crossref","unstructured":"RenK ZhengQ PatilS GibsonG.Indexfs: Scaling file system metadata performance with stateless caching and bulk insertion. In: Proceedings of the International Conference for High Performance Computing Networking Storage and Analysis;2014;New Orleans LA.","DOI":"10.1109\/SC.2014.25"},{"key":"e_1_2_9_24_1","doi-asserted-by":"crossref","unstructured":"GhoshalD RamakrishnanL AgarwalDA.Dac\u2010Man: data change management for scientific datasets on HPC systems. In: Proceedings of the International Conference for High Performance Computing Networking Storage and Analysis;2018;Dallas TX.","DOI":"10.1109\/SC.2018.00075"},{"key":"e_1_2_9_25_1","doi-asserted-by":"crossref","unstructured":"ZhengQ CranorCD GuoD GangerGR.Scaling embedded in\u2010situ indexing with deltaFS. Paper presented at: International Conference for High Performance Computing Networking Storage and Analysis;2018;Dallas TX.","DOI":"10.1109\/SC.2018.00006"},{"key":"e_1_2_9_26_1","doi-asserted-by":"crossref","unstructured":"WuT ChouJ PodhorszkiN et al.Apply block index technique to scientific data analysis and I\/O systems. In: Proceedings of the 17th IEEE\/ACM International Symposium on Cluster Cloud and Grid Computing;2017;Madrid Spain.","DOI":"10.1109\/CCGRID.2017.37"},{"key":"e_1_2_9_27_1","doi-asserted-by":"crossref","unstructured":"LiH GhodsiA ZahariaM ShenkerS StoicaI.Tachyon: reliable memory speed storage for cluster computing frameworks. In: Proceedings of the ACM Symposium on Cloud Computing;2014;Seattle WA.","DOI":"10.1145\/2670979.2670985"},{"key":"e_1_2_9_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-014-3501-3"},{"key":"e_1_2_9_29_1","unstructured":"mdtest HPC Benchmark.https:\/\/2.zoppoz.workers.dev:443\/https\/sourceforge.net\/projects\/mdtest\/. Accessed December 25 2018."},{"key":"e_1_2_9_30_1","unstructured":"PhilippeC\u2010M HideakiK Kian\u2010TatL JennieR.Ss\u2010db: a standard science DBMS benchmark.https:\/\/2.zoppoz.workers.dev:443\/http\/www-conf.slac.stanford.edu\/xldb10\/docs\/SSDB_benchmark.pdf. Accessed December 25 2018. Under submission."},{"key":"e_1_2_9_31_1","unstructured":"Adios 1.13.1 User's Manual.https:\/\/2.zoppoz.workers.dev:443\/https\/users.nccs.gov\/pnorbert\/ADIOS-UsersManual-1.13.1.pdf. Accessed June 13 2019."},{"key":"e_1_2_9_32_1","doi-asserted-by":"crossref","unstructured":"BaruC MooreR RajasekarA WanM.The SDSC storage resource broker. In: Proceedings of CASCON First Decade High Impact Papers (CASCON '10);2010;Toronto Canada.","DOI":"10.1145\/1925805.1925816"},{"key":"e_1_2_9_33_1","unstructured":"BlanasS BynaS.Towards exascale scientific metadata management. arXiv:1503.08482.2015."},{"key":"e_1_2_9_34_1","doi-asserted-by":"crossref","unstructured":"KlaskyS WolfM AinsworthM et al.A view from ORNL: scientific data research opportunities in the big data age. Paper presented at: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS);2018;Vienna Austria.","DOI":"10.1109\/ICDCS.2018.00136"},{"key":"e_1_2_9_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2016.2640301"},{"key":"e_1_2_9_36_1","doi-asserted-by":"crossref","unstructured":"ChiuH\u2010T ChouJerryC\u2010Y VishwanathV WuK.In\u2010memory query system for scientific dataseis. Paper presented at: 2015 IEEE 21st International Conference on Parallel and Distributed Systems (ICPADS);2015;Melbourne Australia.","DOI":"10.1109\/ICPADS.2015.53"},{"key":"e_1_2_9_37_1","doi-asserted-by":"crossref","unstructured":"AlagiannisI BorovicaR BrancoM IdreosS AilamakiA.NoDB: efficient query execution on raw data files. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data;2012;Scottsdale AZ.","DOI":"10.1145\/2213836.2213864"},{"key":"e_1_2_9_38_1","doi-asserted-by":"publisher","DOI":"10.14778\/3115404.3115415"},{"key":"e_1_2_9_39_1","unstructured":"SchnaitterK AbiteboulS MiloT PolyzotisN.COLT: continuous on\u2010line tuning. In: Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data;2006;Chicago IL."},{"key":"e_1_2_9_40_1","unstructured":"IdreosS KerstenML ManegoldS.Database cracking. In: Proceedings of the Third Biennial Conference on Innovative Data Systems Research;2007;Pacific Grove CA."},{"key":"e_1_2_9_41_1","doi-asserted-by":"publisher","DOI":"10.14778\/2002938.2002944"}],"container-title":["Concurrency and Computation: Practice and Experience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fcpe.5609","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cpe.5609","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/cpe.5609","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cpe.5609","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,7]],"date-time":"2023-09-07T08:49:32Z","timestamp":1694076572000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/10.1002\/cpe.5609"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,19]]},"references-count":40,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2020,5,10]]}},"alternative-id":["10.1002\/cpe.5609"],"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1002\/cpe.5609","archive":["Portico"],"relation":{},"ISSN":["1532-0626","1532-0634"],"issn-type":[{"type":"print","value":"1532-0626"},{"type":"electronic","value":"1532-0634"}],"subject":[],"published":{"date-parts":[[2019,12,19]]},"assertion":[{"value":"2019-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-24","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-12-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e5609"}}