{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,20]],"date-time":"2026-01-20T02:14:13Z","timestamp":1768875253276,"version":"3.49.0"},"reference-count":52,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2018,3,30]],"date-time":"2018-03-30T00:00:00Z","timestamp":1522368000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/2.zoppoz.workers.dev:443\/http\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/501100001866","name":"Fonds National de la Recherche Luxembourg","doi-asserted-by":"publisher","award":["FNR\/P10\/03"],"award-info":[{"award-number":["FNR\/P10\/03"]}],"id":[{"id":"10.13039\/501100001866","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Software Testing Verif &amp; Rel"],"published-print":{"date-parts":[[2018,6]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>An important aim in software testing is constructing a test suite with high structural code coverage, that is, ensuring that most if not all of the code under test have been executed by the test cases comprising the test suite. Several search\u2010based techniques have proved successful at automatically generating tests that achieve high coverage. However, despite the well\u2010established arguments behind using evolutionary search algorithms (eg, genetic algorithms) in preference to random search, it remains an open question whether the benefits can actually be observed in practice when generating unit test suites for object\u2010oriented classes. In this paper, we report an empirical study on the effects of using evolutionary algorithms (including a genetic algorithm and chemical reaction optimization) to generate test suites, compared with generating test suites incrementally with random search. We apply the E<jats:sc>VO<\/jats:sc>S<jats:sc>UITE<\/jats:sc>unit test suite generator to 1000 classes randomly selected from the SF110 corpus of open\u2010source projects. Surprisingly, the results show that the difference is much smaller than one might expect: While evolutionary search covers more branches of the type where standard fitness functions provide guidance, we observed that, in practice, the vast majority of branches do not provide any guidance to the search. These results suggest that, although evolutionary algorithms are more effective at covering complex branches, a random search may suffice to achieve high coverage of most object\u2010oriented classes.<\/jats:p>","DOI":"10.1002\/stvr.1660","type":"journal-article","created":{"date-parts":[[2018,3,30]],"date-time":"2018-03-30T14:32:21Z","timestamp":1522420341000},"update-policy":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Random or evolutionary search for object\u2010oriented test suite generation?"],"prefix":"10.1002","volume":"28","author":[{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0003-3745-3508","authenticated-orcid":false,"given":"Sina","family":"Shamshiri","sequence":"first","affiliation":[{"name":"Department of Computer Science University of Sheffield Sheffield UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jos\u00e9 Miguel","family":"Rojas","sequence":"additional","affiliation":[{"name":"Department of Informatics University of Leicester Leicester UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luca","family":"Gazzola","sequence":"additional","affiliation":[{"name":"Department of Informatics, Systems and Communication University of Milano\u2010Bicocca Milano Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gordon","family":"Fraser","sequence":"additional","affiliation":[{"name":"Chair of Software Engineering II University of Passau Passau Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Phil","family":"McMinn","sequence":"additional","affiliation":[{"name":"Department of Computer Science University of Sheffield Sheffield UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Leonardo","family":"Mariani","sequence":"additional","affiliation":[{"name":"Department of Informatics, Systems and Communication University of Milano\u2010Bicocca Milano Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrea","family":"Arcuri","sequence":"additional","affiliation":[{"name":"Westerdals Oslo ACT Oslo Norway and University of Luxembourg"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2018,3,30]]},"reference":[{"key":"e_1_2_10_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.602"},{"key":"e_1_2_10_3_1","doi-asserted-by":"crossref","unstructured":"C.PachecoandM. D.Ernst Randoop: Feedback\u2010directed random testing for Java ACM SIGPLAN International Conference on Object\u2010Oriented Programming Systems Languages and Applications (OOPSLA) ACM 2007 pp.815\u2013816.","DOI":"10.1145\/1297846.1297902"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.93"},{"key":"e_1_2_10_5_1","doi-asserted-by":"crossref","unstructured":"T.Xie Augmenting automatically generated unit\u2010test suites with regression oracle checking European Conference on Object\u2010Oriented Programming (ECOOP). Springer Nantes France 2006 pp.380\u2013403.","DOI":"10.1007\/11785477_23"},{"key":"e_1_2_10_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2699688"},{"key":"e_1_2_10_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2363479"},{"key":"e_1_2_10_8_1","doi-asserted-by":"crossref","unstructured":"I. W. B.Prasetya T3 a combinator\u2010based random testing tool for Java: Benchmarking FITTEST'13 Springer Turkey Istanbul 2014 pp.101\u2013110.","DOI":"10.1007\/978-3-319-07785-7_7"},{"key":"e_1_2_10_9_1","doi-asserted-by":"crossref","unstructured":"M.OriolandS.Tassis Testing .NET code with YETI ICECCS IEEE Oxford UK 2010 pp.264\u2013265.","DOI":"10.1109\/ICECCS.2010.58"},{"key":"e_1_2_10_10_1","doi-asserted-by":"crossref","unstructured":"G.FraserandA.Arcuri EvoSuite: Automatic test suite generation for object\u2010oriented software Proceedings of the Symposium on the Foundations of Software Engineering (FSE) ACM Szeged Hungary 2011 pp.416\u2013419.","DOI":"10.1145\/2025113.2025179"},{"key":"e_1_2_10_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1013886.1007528"},{"key":"e_1_2_10_12_1","doi-asserted-by":"crossref","unstructured":"J. H.Andrews F. C.Li andT.Menzies Nighthawk: A two\u2010level genetic\u2010random unit test data generator Proceedings of the International Conference on Automated Software Engineering (ASE) ACM Atlanta Georgia USA 2007 pp.144\u2013153.","DOI":"10.1145\/1321631.1321654"},{"key":"e_1_2_10_13_1","doi-asserted-by":"crossref","unstructured":"L.Baresi P. L.Lanzi andM.Miraz Testful: An evolutionary test approach for Java Proceedings of the International Conference on Software Testing Verification and Validation (ICST). IEEE Paris France 2010 pp.185\u2013194.","DOI":"10.1109\/ICST.2010.54"},{"key":"e_1_2_10_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.294"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.52"},{"key":"e_1_2_10_16_1","unstructured":"T.B\u00e4ck F.Hoffmeister andH.Schwefel A survey of evolution strategies Proceedings of the International Conference on Genetic Algorithms (ICGA) Morgan Kaufmann San Diego CA 1991 pp.2\u20139."},{"key":"e_1_2_10_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCI.2006.329691"},{"key":"e_1_2_10_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10898-007-9149-x"},{"key":"e_1_2_10_19_1","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1007\/978-0-387-30164-8_630","volume-title":"Encyclopedia of Machine Learning","author":"Kennedy J.","year":"2011"},{"key":"e_1_2_10_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2009.2033580"},{"key":"e_1_2_10_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2012.2227973"},{"key":"e_1_2_10_22_1","doi-asserted-by":"crossref","unstructured":"A.Lam J.Xu andV. O.Li Chemical reaction optimization for population transition in peer\u2010to\u2010peer live streaming IEEE Congress on Evolutionary Computation (CEC) IEEE Barcelona Spain 2010 pp.1\u20138.","DOI":"10.1109\/CEC.2010.5585933"},{"key":"e_1_2_10_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2011.35"},{"key":"e_1_2_10_24_1","unstructured":"J.Xu A. Y.Lam andV. O.Li Stock portfolio selection using chemical reaction optimization Proceedings of the International Conference on Operations Research and Financial Engineering (ICORFE) Paris France 2011 pp.458\u2013463."},{"key":"e_1_2_10_25_1","doi-asserted-by":"crossref","unstructured":"J. J.Yu A.Lam andV. O.Li Evolutionary artificial neural network based on chemical reaction optimization 2011 IEEE Congress on Evolutionary Computation (CEC) New Orleans LA USA IEEE 2011 pp.2083\u20132090.","DOI":"10.1109\/CEC.2011.5949872"},{"key":"e_1_2_10_26_1","doi-asserted-by":"crossref","unstructured":"U.Rueda R.Just J. P.Galeotti andT. E. J.Vos.Unit testing tool competition: Round four Proceedings of the International Workshop on Search\u2010Based Software Testing (SBST) ACM Austin TX USA 2016 pp.19\u201328.","DOI":"10.1145\/2897010.2897018"},{"key":"e_1_2_10_27_1","doi-asserted-by":"crossref","unstructured":"S.Shamshiri R.Just J. M.Rojas G.Fraser P.McMinn andA.Arcuri.Do automatically generated unit tests find real faults? An empirical study of effectiveness and challenges Proceedings of the International Conference on Automated Software Engineering (ASE). IEEE Lincoln NE USA 2015 pp.201\u2013211.","DOI":"10.1109\/ASE.2015.86"},{"key":"e_1_2_10_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2685612"},{"key":"e_1_2_10_29_1","doi-asserted-by":"crossref","unstructured":"S.Shamshiri J. M.Rojas G.Fraser andP.McMinn.Random or genetic algorithm search for object\u2010oriented test suite generation?Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) ACM Madrid Spain 2015 pp.1367\u20131374.","DOI":"10.1145\/2739480.2754696"},{"key":"e_1_2_10_30_1","doi-asserted-by":"crossref","unstructured":"N.AlshahwanandM.Harman Automated web application testing using search based software engineering Proceedings of the International Conference on Automated Software Engineering (ASE) IEEE Lawrence KS USA 2011 pp.3\u201312.","DOI":"10.1109\/ASE.2011.6100082"},{"key":"e_1_2_10_31_1","doi-asserted-by":"crossref","unstructured":"G.FraserandA.Arcuri The seed is strong: Seeding strategies in search\u2010based software testing Proceedings of the International Conference on Software Testing Verification and Validation (ICST) IEEE Montreal QC Canada 2012 pp.121\u2013130.","DOI":"10.1109\/ICST.2012.92"},{"key":"e_1_2_10_32_1","doi-asserted-by":"crossref","unstructured":"P.McMinn M.Shahbaz andM.Stevenson Search\u2010based test input generation for string data types using the results of web queries Proceedings of the International Conference on Software Testing Verification and Validation (ICST) IEEE Montreal QC Canada 2012 pp.141\u2013150.","DOI":"10.1109\/ICST.2012.94"},{"key":"e_1_2_10_33_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1601"},{"key":"e_1_2_10_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.14"},{"key":"e_1_2_10_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.57624"},{"key":"e_1_2_10_36_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1495"},{"key":"e_1_2_10_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9424-2"},{"key":"e_1_2_10_38_1","unstructured":"M.Harman L.Hu R.Hierons A.Baresel andH.Sthamer.Improving evolutionary testing by flag removal Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) MK Pub. New York City New York USA 2002 pp.1359\u20131366."},{"key":"e_1_2_10_39_1","doi-asserted-by":"crossref","unstructured":"A.BareselandH.Sthamer Evolutionary testing of flag conditions Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) Springer Chicago IL USA 2003 pp.2442\u20132454.","DOI":"10.1007\/3-540-45110-2_148"},{"key":"e_1_2_10_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9249-9"},{"key":"e_1_2_10_41_1","doi-asserted-by":"crossref","unstructured":"A.Lam L.Victor andJ.Xu Chemical reaction optimization: A tutorial Memetic Computing (2012) Springer 2012 pp.3\u201317.","DOI":"10.1007\/s12293-012-0075-1"},{"key":"e_1_2_10_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/4235.585893"},{"key":"e_1_2_10_43_1","doi-asserted-by":"publisher","DOI":"10.3102\/10769986025002101"},{"key":"e_1_2_10_44_1","doi-asserted-by":"crossref","unstructured":"A.Aleti I.Moser andL.Grunske Analysing the fitness landscape of search\u2010based software testing problems Proceedings of the International Conference on Automated Software Engineering (ASE) 2016 pp.603\u2013621.","DOI":"10.1007\/s10515-016-0197-7"},{"key":"e_1_2_10_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2004.1265732"},{"key":"e_1_2_10_46_1","doi-asserted-by":"crossref","unstructured":"Y.LiandG.Fraser Bytecode testability transformation International Symposium on Search Based Software Engineering (SSBSE) Springer Szeged Hungary 2011 pp.237\u2013251.","DOI":"10.1007\/978-3-642-23716-4_21"},{"key":"e_1_2_10_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.71"},{"key":"e_1_2_10_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(01)00190-2"},{"key":"e_1_2_10_49_1","doi-asserted-by":"crossref","unstructured":"R.Sharma M.Gligoric A.Arcuri G.Fraser andD.Marinov.Testing container classes: Random or systematic? Proceedings of the international conference on fundamental approaches to software engineering (FASE) Springer Saarbr\u00fccken Germany 2011 pp.262\u2013277.","DOI":"10.1007\/978-3-642-19811-3_19"},{"key":"e_1_2_10_50_1","doi-asserted-by":"crossref","unstructured":"M.Eler A.Endo andV.Durelli Quantifying the characteristics of Java programs that may influence symbolic execution from a test data generation perspective Proceedings of the International Conference on Computer Software and Applications Conference (COMPSAC) IEEE Vasteras Sweden 2014 pp.181\u2013190.","DOI":"10.1109\/COMPSAC.2014.26"},{"key":"e_1_2_10_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2012.11.048"},{"key":"e_1_2_10_52_1","doi-asserted-by":"crossref","unstructured":"J. P.Galeotti G.Fraser andA.Arcuri Improving search\u2010based test suite generation with dynamic symbolic execution Proceedings of the International Symposium on Software Reliability Engineering (ISSRE) IEEE Pasadena CA USA 2013 pp.360\u2013369.","DOI":"10.1109\/ISSRE.2013.6698889"},{"key":"e_1_2_10_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2014.05.032"}],"container-title":["Software Testing, Verification and Reliability"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fstvr.1660","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/stvr.1660","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,3]],"date-time":"2025-07-03T06:46:18Z","timestamp":1751525178000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/10.1002\/stvr.1660"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,3,30]]},"references-count":52,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,6]]}},"alternative-id":["10.1002\/stvr.1660"],"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1002\/stvr.1660","archive":["Portico"],"relation":{},"ISSN":["0960-0833","1099-1689"],"issn-type":[{"value":"0960-0833","type":"print"},{"value":"1099-1689","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,3,30]]},"assertion":[{"value":"2016-12-22","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-12-14","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-03-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e1660"}}