Decoding Safety Feedback from Diverse Raters: A Data-driven Lens on Responsiveness to Severity.

Charvi Rastogi

Pushkar Mishra Charvi Rastogi Stephen R. Pfohl Alicia Parrish Tian Huey Teh Roma Patel Mark Diaz Ding Wang 0006 Michela Paganini Vinodkumar Prabhakaran Lora Aroyo Verena Rieser Decoding Safety Feedback from Diverse Raters: A Data-driven Lens on Responsiveness to Severity. 2026 2026 Trans. Mach. Learn. Res. https://openreview.net/forum?id=vx6qrM7VB5 db/journals/tmlr/tmlr2026.html#MishraRPPTPDWPPAR26 streams/journals/tmlr

Charvi Rastogi Mukul Bhutani Minsuk Kahng Shamsuddeen Hassan Muhammad Evgeniia Razumovskaia Priyanka Suresh Ibrahim Said Ahmad Charu Kalia Yaaseen Mahomed Madhurima Maji Minjae Lee Alicia Parrish Jessica Quaye Vijay Janapa Reddi Aishwarya Verma Lora Aroyo Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South. 2608-2642 2026 FAccT https://doi.org/10.1145/3805689.3812296 conf/fat/2026 db/conf/fat/facct2026.html#RastogiBKMRSAKM26 Jessica Quaye Alicia Parrish Charvi Rastogi Minsuk Kahng Oana Inel Lora Aroyo Vijay Janapa Reddi Bridging the Scale Gap: Augmenting Human Red-Teaming to Uncover Latent Risks in T2I Models. 7422-7449 2026 FAccT https://doi.org/10.1145/3805689.3812210 conf/fat/2026 db/conf/fat/facct2026.html#QuayePRKIAR26

Canfer Akbulut Rasmi Elasmar Abhishek Roy Anthony Payne Priyanka Suresh Lujain Ibrahim Seliem El-Sayed Charvi Rastogi Ashyana Kachra Will Hawkins Kristian Lum Laura Weidinger Evaluating Language Models for Harmful Manipulation. 2026 March abs/2603.25326 CoRR https://doi.org/10.48550/arXiv.2603.25326 db/journals/corr/corr2603.html#abs-2603-25326 streams/journals/corr

Arkadiy Saakyan Charvi Rastogi Lora Aroyo Quantifying the Salience of Geo-Cultural Values for Pluralistic Safety Alignment. 2026 June abs/2606.00369 CoRR https://doi.org/10.48550/arXiv.2606.00369 db/journals/corr/corr2606.html#abs-2606-00369 streams/journals/corr

Yi Hao Ming Wang Luke Guerdan Charvi Rastogi Kenneth Holstein Evaluating Human-AI Decision-Making Policies Under Imperfect Proxy Labels. 2025 AutomationXP@CHI https://ceur-ws.org/Vol-4101/paper9.pdf conf/automationxp/2025 db/conf/automationxp/automationxp2025.html#WangGRH25 Charvi Rastogi Tian Huey Teh Pushkar Mishra Roma Patel Ding Wang 0006 Mark Díaz Alicia Parrish Aida Mostafazadeh Davani Zoe Ashwood Michela Paganini Vinodkumar Prabhakaran Verena Rieser Lora Aroyo Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models. 2025 NeurIPS http://papers.nips.cc/paper_files/paper/2025/hash/da835e893e7ec381fd8ef7809a7c0e1d-Abstract-Datasets_and_Benchmarks_Track.html conf/nips/2025 db/conf/nips/neurips2025.html#RastogiTMPWDPDA25

Lujain Ibrahim Canfer Akbulut Rasmi Elasmar Charvi Rastogi Minsuk Kahng Meredith Ringel Morris Kevin R. McKee Verena Rieser Murray Shanahan Laura Weidinger Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Models. 2025 February abs/2502.07077 CoRR https://doi.org/10.48550/arXiv.2502.07077 db/journals/corr/corr2502.html#abs-2502-07077 streams/journals/corr

Pushkar Mishra Charvi Rastogi Stephen R. Pfohl Alicia Parrish Roma Patel Mark Diaz Ding Wang 0006 Michela Paganini Vinodkumar Prabhakaran Lora Aroyo Verena Rieser Nuanced Safety for Generative AI: How Demographics Shape Responsiveness to Severity. 2025 March abs/2503.05609 CoRR https://doi.org/10.48550/arXiv.2503.05609 db/journals/corr/corr2503.html#abs-2503-05609 streams/journals/corr

Charvi Rastogi Tian Huey Teh Pushkar Mishra Roma Patel Ding Wang 0006 Mark Díaz Alicia Parrish Aida Mostafazadeh Davani Zoe Ashwood Michela Paganini Vinodkumar Prabhakaran Verena Rieser Lora Aroyo Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models. 2025 July abs/2507.13383 CoRR https://doi.org/10.48550/arXiv.2507.13383 db/journals/corr/corr2507.html#abs-2507-13383 streams/journals/corr

Ding Wang 0006 Mark Díaz Charvi Rastogi Aida Mostafazadeh Davani Vinodkumar Prabhakaran Pushkar Mishra Roma Patel Alicia Parrish Zoe Ashwood Michela Paganini Tian Huey Teh Verena Rieser Lora Aroyo "Just a strange pic": Evaluating 'safety' in GenAI Image safety annotation tasks from diverse annotators' perspectives. 2025 July abs/2507.16033 CoRR https://doi.org/10.48550/arXiv.2507.16033 db/journals/corr/corr2507.html#abs-2507-16033 streams/journals/corr

Jessica Quaye Charvi Rastogi Alicia Parrish Oana Inel Minsuk Kahng Lora Aroyo Vijay Janapa Reddi From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Models. 2025 July abs/2507.17922 CoRR https://doi.org/10.48550/arXiv.2507.17922 db/journals/corr/corr2507.html#abs-2507-17922 streams/journals/corr

Jessica Quaye Alicia Parrish Oana Inel Charvi Rastogi Hannah Rose Kirk Minsuk Kahng Erin van Liemt Max Bartolo Jess Tsang Justin White Nathan Clement Rafael Mosquera Juan Ciro Vijay Janapa Reddi Lora Aroyo Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation. 388-406 2024 FAccT https://doi.org/10.1145/3630106.3658913 https://www.wikidata.org/entity/Q130933694 conf/fat/2024 db/conf/fat/facct2024.html#QuayePIRKKLBTWC24

Charvi Rastogi Xiangchen Song Zhijing Jin 0001 Ivan Stelmakh Hal Daumé III Kun Zhang 0001 Nihar B. Shah A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions. 2024 abs/2403.01015 CoRR https://doi.org/10.48550/arXiv.2403.01015 db/journals/corr/corr2403.html#abs-2403-01015

Jessica Quaye Alicia Parrish Oana Inel Charvi Rastogi Hannah Rose Kirk Minsuk Kahng Erin van Liemt Max Bartolo Jess Tsang Justin White Nathan Clement Rafael Mosquera Juan Ciro Vijay Janapa Reddi Lora Aroyo Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation. 2024 abs/2403.12075 CoRR https://doi.org/10.48550/arXiv.2403.12075 db/journals/corr/corr2403.html#abs-2403-12075

Charvi Rastogi Tian Huey Teh Pushkar Mishra Roma Patel Zoe Ashwood Aida Mostafazadeh Davani Mark Diaz Michela Paganini Alicia Parrish Ding Wang 0006 Vinodkumar Prabhakaran Lora Aroyo Verena Rieser Insights on Disagreement Patterns in Multimodal Safety Perception across Diverse Rater Groups. 2024 abs/2410.17032 CoRR https://doi.org/10.48550/arXiv.2410.17032 db/journals/corr/corr2410.html#abs-2410-17032 streams/journals/corr

Charvi Rastogi Marco Túlio Ribeiro Nicholas King Harsha Nori Saleema Amershi Supporting Human-AI Collaboration in Auditing LLMs with LLMs. 913-926 2023 AIES https://doi.org/10.1145/3600211.3604712 https://www.wikidata.org/entity/Q130896496 conf/aies/2023 db/conf/aies/aies2023.html#RastogiRKNA23 Charvi Rastogi Investigating the Relative Strengths of Humans and Machine Learning in Decision-Making. 987-989 2023 AIES https://doi.org/10.1145/3600211.3604738 https://www.wikidata.org/entity/Q130985436 conf/aies/2023 db/conf/aies/aies2023.html#Rastogi23 Mark Mazumder Colby R. Banbury Xiaozhe Yao Bojan Karlas William Gaviria Rojas Sudnya Frederick Diamos Greg Diamos Lynn He Alicia Parrish Hannah Rose Kirk Jessica Quaye Charvi Rastogi Douwe Kiela David Jurado David Kanter Rafael Mosquera Will Cukierski Juan Ciro Lora Aroyo Bilge Acun Lingjiao Chen Mehul Raje Max Bartolo Evan Sabri Eyuboglu Amirata Ghorbani Emmett D. Goodman Addison Howard Oana Inel Tariq Kane Christine R. Kirkpatrick D. Sculley Tzu-Sheng Kuo Jonas W. Mueller Tristan Thrush Joaquin Vanschoren Margaret Warren Adina Williams Serena Yeung Newsha Ardalani Praveen K. Paritosh Ce Zhang 0001 James Y. Zou Carole-Jean Wu Cody Coleman Andrew Y. Ng Peter Mattson Vijay Janapa Reddi DataPerf: Benchmarks for Data-Centric AI Development. 2023 NeurIPS http://papers.nips.cc/paper_files/paper/2023/hash/112db88215e25b3ae2750e9eefcded94-Abstract-Datasets_and_Benchmarks.html conf/nips/2023 db/conf/nips/neurips2023.html#MazumderBYKRDDH23

Charvi Rastogi Marco Túlio Ribeiro Nicholas King Saleema Amershi Supporting Human-AI Collaboration in Auditing LLMs with LLMs. 2023 abs/2304.09991 CoRR https://doi.org/10.48550/arXiv.2304.09991 db/journals/corr/corr2304.html#abs-2304-09991

Alicia Parrish Hannah Rose Kirk Jessica Quaye Charvi Rastogi Max Bartolo Oana Inel Juan Ciro Rafael Mosquera Addison Howard Will Cukierski D. Sculley Vijay Janapa Reddi Lora Aroyo Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models. 2023 abs/2305.14384 CoRR https://doi.org/10.48550/arXiv.2305.14384 db/journals/corr/corr2305.html#abs-2305-14384

Charvi Rastogi Sivaraman Balakrishnan Nihar B. Shah Aarti Singh Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions. 225:1-225:48 2022 23 J. Mach. Learn. Res. https://jmlr.org/papers/v23/20-1304.html db/journals/jmlr/jmlr23.html#RastogiBSS22

Charvi Rastogi Yunfeng Zhang Dennis Wei Kush R. Varshney Amit Dhurandhar Richard Tomsett Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making. 83:1-83:22 2022 6 Proc. ACM Hum. Comput. Interact. CSCW1 https://doi.org/10.1145/3512930 db/journals/pacmhci/pacmhci6.html#RastogiZWVDT22

Charvi Rastogi Ivan Stelmakh Nihar B. Shah Sivaraman Balakrishnan No Rose for MLE: Inadmissibility of MLE for Evaluation Aggregation Under Levels of Expertise. 3168-3173 2022 ISIT https://doi.org/10.1109/ISIT50566.2022.9834340 conf/isit/2022 db/conf/isit/isit2022.html#RastogiSSB22

Ivan Stelmakh Charvi Rastogi Ryan Liu 0001 Shuchi Chawla 0001 Federico Echenique Nihar B. Shah Cite-seeing and Reviewing: A Study on Citation Bias in Peer Review. 2022 abs/2203.17239 CoRR https://doi.org/10.48550/arXiv.2203.17239 db/journals/corr/corr2203.html#abs-2203-17239

Charvi Rastogi Ivan Stelmakh Xinwei Shen Marina Meila Federico Echenique Shuchi Chawla 0001 Nihar B. Shah To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online. 2022 abs/2203.17259 CoRR https://doi.org/10.48550/arXiv.2203.17259 db/journals/corr/corr2203.html#abs-2203-17259

Charvi Rastogi Liu Leqi Kenneth Holstein Hoda Heidari A Unifying Framework for Combining Complementary Strengths of Humans and ML toward Better Predictive Decision-Making. 2022 abs/2204.10806 CoRR https://doi.org/10.48550/arXiv.2204.10806 db/journals/corr/corr2204.html#abs-2204-10806

Charvi Rastogi Ivan Stelmakh Alina Beygelzimer Yann N. Dauphin Percy Liang Jennifer Wortman Vaughan Zhenyu Xue Hal Daumé III Emma Pierson Nihar B. Shah How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions? 2022 abs/2211.12966 CoRR https://doi.org/10.48550/arXiv.2211.12966 db/journals/corr/corr2211.html#abs-2211-12966

Charvi Rastogi Sivaraman Balakrishnan Nihar B. Shah Aarti Singh Two-Sample Testing on Pairwise Comparison Data and the Role of Modeling Assumptions. 1271-1276 2020 ISIT https://doi.org/10.1109/ISIT44484.2020.9174055 conf/isit/2020 db/conf/isit/isit2020.html#RastogiBSS20

Charvi Rastogi Sivaraman Balakrishnan Nihar B. Shah Aarti Singh Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions. 2020 abs/2006.11909 CoRR https://arxiv.org/abs/2006.11909 db/journals/corr/corr2006.html#abs-2006-11909

Charvi Rastogi Yunfeng Zhang Dennis Wei Kush R. Varshney Amit Dhurandhar Richard Tomsett Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making. 2020 abs/2010.07938 CoRR https://arxiv.org/abs/2010.07938 db/journals/corr/corr2010.html#abs-2010-07938

Ivan Stelmakh Charvi Rastogi Nihar B. Shah Aarti Singh Hal Daumé III A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions. 2020 abs/2011.15083 CoRR https://arxiv.org/abs/2011.15083 db/journals/corr/corr2011.html#abs-2011-15083

Bhavya Kailkhura Jayaraman J. Thiagarajan Charvi Rastogi Pramod K. Varshney Peer-Timo Bremer A Spectral Approach for the Design of Experiments: Design, Analysis and Algorithms. 2018 19 J. Mach. Learn. Res. https://jmlr.org/papers/v19/17-735.html db/journals/jmlr/jmlr19.html#KailkhuraTRVB18 34:1-34:46

Charvi Rastogi Animesh Kumar Mobile Sensing of Two-Dimensional Bandlimited Fields on Random Paths. 2017 abs/1711.04114 CoRR http://arxiv.org/abs/1711.04114 db/journals/corr/corr1711.html#abs-1711-04114

Bhavya Kailkhura Jayaraman J. Thiagarajan Charvi Rastogi Pramod K. Varshney Peer-Timo Bremer A Spectral Approach for the Design of Experiments: Design, Analysis and Algorithms. 2017 abs/1712.06028 CoRR http://arxiv.org/abs/1712.06028 db/journals/corr/corr1712.html#abs-1712-06028

Bilge Acun Ibrahim Said Ahmad Canfer Akbulut Saleema Amershi Newsha Ardalani Lora Aroyo Zoe Ashwood Sivaraman Balakrishnan Colby R. Banbury Max Bartolo Alina Beygelzimer Mukul Bhutani Peer-Timo Bremer Shuchi Chawla 0001 Lingjiao Chen Juan Ciro Nathan Clement Cody Coleman Will Cukierski Hal Daumé III Yann N. Dauphin Aida Mostafazadeh Davani Amit Dhurandhar Gregory Frederick DiamosGreg Diamos Sudnya Frederick Diamos Mark DiazMark Díaz Federico Echenique Seliem El-Sayed Rasmi Elasmar Sabri EyubogluEvan Sabri Eyuboglu Amirata Ghorbani Emmett D. Goodman Luke Guerdan Will Hawkins Lynn He Hoda Heidari Kenneth Holstein Addison Howard Lujain Ibrahim Oana Inel Zhijing Jin 0001 David Jurado Ashyana Kachra Minsuk Kahng Bhavya Kailkhura Charu Kalia Tariq Kane David Kanter Bojan Karlas Douwe Kiela Nicholas King Hannah KirkHannah Rose Kirk Christine R. Kirkpatrick Animesh Kumar Tzu-Sheng Kuo Minjae Lee Liu Leqi Percy Liang Erin van Liemt Ryan Liu 0001 Kristian Lum Yaaseen Mahomed Madhurima Maji Peter Mattson Mark Mazumder Kevin R. McKee Marina Meila Pushkar Mishra Meredith Ringel Morris Rafael Mosquera Jonas Mueller 0001Jonas W. Mueller Shamsuddeen Hassan Muhammad Andrew Y. Ng Harsha Nori Michela Paganini Praveen K. Paritosh Alicia Parrish Roma Patel Anthony Payne Stephen PfohlStephen R. Pfohl Emma Pierson Vinodkumar Prabhakaran Jessica Quaye Mehul Raje Evgeniia Razumovskaia Vijay Janapa Reddi Marco Túlio Ribeiro Verena Rieser William Gaviria Rojas Abhishek Roy Arkadiy Saakyan D. Sculley Nihar B. Shah Murray Shanahan Xinwei Shen Aarti Singh Xiangchen Song Ivan Stelmakh Priyanka Suresh Tian Huey Teh Jayaraman J. Thiagarajan Tristan Thrush Richard Tomsett Jess Tsang Joaquin Vanschoren Kush R. Varshney Pramod K. Varshney Jennifer Wortman Vaughan Aishwarya Verma Ding Wang 0006 Yi Hao Ming Wang Margaret Warren Dennis Wei Laura Weidinger Justin White Adina Williams Carole-Jean Wu Zhenyu Xue Xiaozhe Yao Serena Yeung-LevySerena Yeung Ce Zhang 0001 Kun Zhang 0001 Yunfeng Zhang James Zou 0001James Y. Zou