{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:43:39Z","timestamp":1782848619559,"version":"3.54.5"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100021171","name":"Guangdong Basic and Applied Basic Research Foundation","doi-asserted-by":"crossref","award":["2024A1515010145"],"award-info":[{"award-number":["2024A1515010145"]}],"id":[{"id":"10.13039\/501100021171","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>While cloud-native microservice architectures have revolutionized software development, their inherent operational complexity makes failure Root Cause Analysis (RCA) a critical yet challenging task. Numerous data-driven RCA models have been proposed to address this challenge. However, we find that the benchmarks used to evaluate these models are often too simple to reflect real-world scenarios. Our preliminary study reveals that simple rule-based methods can achieve performance comparable to or even surpassing state-of-the-art (SOTA) models on four widely used public benchmarks. This finding suggests that the oversimplification of existing benchmarks might lead to an overestimation of the performance of RCA methods.<\/jats:p>\n                  <jats:p>To further investigate the oversimplification issue, we conduct a systematic analysis of popular public RCA benchmarks, identifying key limitations in their fault injection strategies, call graph structures, and telemetry signal patterns. Based on these insights, we propose an automated framework for generating more challenging and comprehensive benchmarks that include complex fault propagation scenarios. Our new dataset contains 1,430 validated failure cases from 9,152 fault injections, covering 25 fault types across 6 categories, dynamic workloads, and hierarchical ground-truth labels that map failures from services down to code-level causes. Crucially, to ensure the failure cases are relevant to IT operations, each case is validated to have a discernible impact on user-facing SLIs.<\/jats:p>\n                  <jats:p>\n                    Our re-evaluation of 11 SOTA models on this new benchmark shows that they achieve low\n                    <jats:italic toggle=\"yes\">Top@1<\/jats:italic>\n                    accuracies, averaging 0.21, with the best-performing model reaching merely 0.37, and execution times escalating from seconds to hours. From this analysis, we identify three critical failure patterns common to current RCA models:\n                    <jats:italic toggle=\"yes\">scalability issues<\/jats:italic>\n                    ,\n                    <jats:italic toggle=\"yes\">observability blind spots<\/jats:italic>\n                    , and\n                    <jats:italic toggle=\"yes\">modeling bottlenecks<\/jats:italic>\n                    . Based on these findings, we provide actionable guidelines for future RCA research. We emphasize the need for robust algorithms and the co-development of challenging benchmarks. To facilitate further research, we publicly release our benchmark generation framework, the new dataset, and our implementations of the evaluated SOTA models.\n                  <\/jats:p>","DOI":"10.1145\/3797100","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"1605-1627","source":"Crossref","is-referenced-by-count":0,"title":["Rethinking the Evaluation of Microservice RCA with a Fault Propagation-Aware Benchmark"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0006-7116-5613","authenticated-orcid":false,"given":"Aoyang","family":"Fang","sequence":"first","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0002-2626-6563","authenticated-orcid":false,"given":"Songhan","family":"Zhang","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0008-3952-9207","authenticated-orcid":false,"given":"Yifan","family":"Yang","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0001-6682-5799","authenticated-orcid":false,"given":"Haotong","family":"Wu","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0001-1516-103X","authenticated-orcid":false,"given":"Junjielong","family":"Xu","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0009-4811-5900","authenticated-orcid":false,"given":"Xuyang","family":"Wang","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0002-5284-4435","authenticated-orcid":false,"given":"Rui","family":"Wang","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0003-0802-1674","authenticated-orcid":false,"given":"Manyi","family":"Wang","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0008-3715-9265","authenticated-orcid":false,"given":"Qisheng","family":"Lu","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0003-3377-8129","authenticated-orcid":false,"given":"Pinjia","family":"He","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shenzhen, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Gaia-dataset. https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/CloudWise-OpenSource\/GAIA-DataSet. Accessed: 2024-09-14."},{"key":"e_1_2_1_2_1","unstructured":"2024. AIOps Competition Dataset. https:\/\/2.zoppoz.workers.dev:443\/https\/www.aiops.cn\/gitlab\/aiops-nankai\/data\/trace\/aiops2021. Accessed: 2025-07- 01."},{"key":"e_1_2_1_3_1","unstructured":"2024. DeathStarBench. https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/delimitrou\/DeathStarBench\/tree\/master. Accessed: 2025-07-01."},{"key":"e_1_2_1_4_1","unstructured":"2024. Eadro Dataset. https:\/\/2.zoppoz.workers.dev:443\/https\/zenodo.org\/records\/7615394. Accessed: 2025-07-01."},{"key":"e_1_2_1_5_1","unstructured":"2024. GAIA Dataset. https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/CloudWise-OpenSource\/GAIA-DataSet. Accessed: 2025-07-01."},{"key":"e_1_2_1_6_1","volume-title":"The implementation of multimodal observability data root cause analysis approach Nezha in FSE","author":"Nezha GitHub","year":"2023","unstructured":"2024. GitHub -IntelligentDDS\/Nezha: The implementation of multimodal observability data root cause analysis approach Nezha in FSE 2023. https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/IntelligentDDS\/Nezha. Accessed: 2024-10-27."},{"key":"e_1_2_1_7_1","unstructured":"2024. Online Boutique. https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/GoogleCloudPlatform\/microservices-demo. Accessed: 2025-07-01."},{"key":"e_1_2_1_8_1","unstructured":"2024. Opentelemetry eBPF Profiler. https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/open-telemetry\/opentelemetry-ebpf-profiler. Accessed: 2024-08-21."},{"key":"e_1_2_1_9_1","unstructured":"2024. Train Ticket. https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/FudanSELab\/train-ticket. Accessed: 2025-07-01."},{"key":"e_1_2_1_10_1","unstructured":"2025. Hubble. https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/cilium\/hubble. Accessed: 2025-07-01."},{"key":"e_1_2_1_11_1","unstructured":"2025. OpenTelemetry Collector. https:\/\/2.zoppoz.workers.dev:443\/https\/opentelemetry.io\/docs\/collector\/. Accessed: 2025-07-01."},{"key":"e_1_2_1_12_1","volume-title":"A dynamic fault tolerance model for microservices architecture","author":"Addeen Hajar Hameed","unstructured":"Hajar Hameed Addeen. 2019. A dynamic fault tolerance model for microservices architecture. South Dakota State University. https:\/\/2.zoppoz.workers.dev:443\/https\/openprairie.sdstate.edu\/etd\/3410"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458336.3465286"},{"key":"e_1_2_1_14_1","unstructured":"CloudZero. 2025. Cloud Computing Statistics For 2025 And Beyond. https:\/\/2.zoppoz.workers.dev:443\/https\/www.cloudzero.com\/blog\/cloud-computingstatistics\/ Accessed: 2025-07-01."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613864"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","unstructured":"Aoyang Fang Songhan Zhang Yifan Yang Haotong Wu Junjielong Xu Xuyang Wang Rui Wang Manyi Wang Qisheng Lu and Pinjia He. 2026. Artifact for Rethinking the Evaluation of Microservice RCA with a Fault Propagation -Aware Benchmark. https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.5281\/zenodo.19494725. doi:10.5281\/zenodo.19494725 Accessed: 2026-04-23. 10.5281\/zenodo.19494725","DOI":"10.5281\/zenodo.19494725"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3623278.3624758"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2023.3241299"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695475"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556904"},{"key":"e_1_2_1_21_1","first-page":"73","volume-title":"16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)","author":"Huang Lexiang","year":"2022","unstructured":"Lexiang Huang, Matthew Magnusson, Abishek Bangalore Muralikrishna, Salman Estyak, Rebecca Isaacs, Abutalib Aghayev, Timothy Zhu, and Aleksey Charapko. 2022. Metastable failures in the wild. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 73-90. https:\/\/2.zoppoz.workers.dev:443\/https\/www. usenix.org\/conference\/osdi22\/presentation\/huang-lexiang"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.17105973"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.52202\/068431-2259"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3543507.3583338"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-024-00445-w"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/icse48619.2023.00150"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539041"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3623278.3624771"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549092"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i1.27772"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/icse-seip52600.2021.00043"},{"key":"e_1_2_1_32_1","first-page":"559","volume-title":"17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20)","author":"Lou Chang","year":"2020","unstructured":"Chang Lou, Peng Huang, and Scott Smith. 2020. Understanding, detecting and localizing partial failures in large system software. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). USENIX Association, Santa Clara, CA, 559-574. https:\/\/2.zoppoz.workers.dev:443\/https\/dl.acm.org\/doi\/10.5555\/3388242.3388284"},{"key":"e_1_2_1_33_1","unstructured":"OperationsPAI Team. 2026. Revisiting RCA Evaluation Project Page. https:\/\/2.zoppoz.workers.dev:443\/https\/operationspai.github.io\/revisiting-rcaevaluation\/. Accessed: 2026-04-23."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464805"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3660805"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695065"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3701716.3715290"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3695999"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695495"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","unstructured":"Lei Tao Xianglin Lu Shenglin Zhang Jiaqi Luan Yingke Li Mingjie Li Zeyan Li Qingyang Yu Hucheng Xie Ruijie Xu et al. 2024. Diagnosing performance issues for large-scale microservice systems with heterogeneous graph. IEEE Transactions on Services Computing (2024). doi:10.1109\/tsc.2024.3402172 10.1109\/tsc.2024.3402172","DOI":"10.1109\/tsc.2024.3402172"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599392"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599849"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695485"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.5555\/3691825.3691895"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/noms47738.2020.9110353"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","unstructured":"Zhe Xie Shenglin Zhang Yitong Geng Yao Zhang Minghua Ma Xiaohui Nie Zhenhe Yao Longlong Xu Yongqian Sun Wentao Li et al. 2024. Microservice Root Cause Analysis With Limited Observability Through Intervention Recognition in the Latent Space. (2024). doi:10.1145\/3637528.3671530 10.1145\/3637528.3671530","DOI":"10.1145\/3637528.3671530"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023"},{"key":"e_1_2_1_49_1","volume-title":"The Thirteenth International Conference on Learning Representations. https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=M4qNIzQYpd","author":"Xu Junjielong","year":"2025","unstructured":"Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. 2025. OpenRCA: Can large language models locate the root cause of software failures?. In The Thirteenth International Conference on Learning Representations. https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=M4qNIzQYpd"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449905"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616249"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3543507.3583302"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639088"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/tsc.2023.3290018"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3715005"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","unstructured":"Shenglin Zhang Yongxin Zhao Sibo Xia Shirui Wei Yongqian Sun Chenyu Zhao Shiyu Ma Junhua Kuang Bolin Zhu Lemeng Pan et al. 2024. No More Data Silos: Unified Microservice Failure Diagnosis with Temporal Knowledge Graph. IEEE Transactions on Services Computing (2024). doi:10.1109\/tsc.2024.3489444 10.1109\/tsc.2024.3489444","DOI":"10.1109\/tsc.2024.3489444"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3663529.3663834"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3589334.3645442"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3674726"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/dl.acm.org\/doi\/pdf\/10.1145\/3797100","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:46:26Z","timestamp":1782845186000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/3797100"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":59,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3797100"],"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1145\/3797100","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}