{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T16:37:10Z","timestamp":1781887030398,"version":"3.54.5"},"reference-count":155,"publisher":"Association for Computing Machinery (ACM)","issue":"10","funder":[{"DOI":"10.13039\/501100001459","name":"Ministry of Education, Singapore","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001459","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Academic Research Fund Tier 3","award":["MOET32020-0004"],"award-info":[{"award-number":["MOET32020-0004"]}]},{"name":"Key R&D Program of Zhejiang","award":["2025C01083"],"award-info":[{"award-number":["2025C01083"]}]},{"DOI":"10.13039\/100018735","name":"Ant Group","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100018735","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:p>\n                    Since the launch of ChatGPT, a powerful AI Chatbot developed by OpenAI, large language models (LLMs) have made significant advancements in both academia and industry, bringing about a fundamental engineering paradigm shift in many areas. While LLMs are powerful, it is also crucial to best use their power where \u201cprompt\u201d plays a core role. However, the booming LLMs themselves, including excellent APIs like ChatGPT, have several inherent limitations: (1) temporal lag of training data, and (2) the lack of physical capabilities to perform external actions. Recently, we have observed the trend of utilizing prompt-based tools to better utilize the power of LLMs for downstream tasks, but a lack of systematic literature and standardized terminology, partly due to the rapid evolution of this field. Therefore, in this work, we survey related prompting tools and promote the concept of the \u201cPrompting Framework\u201d (PF), i.e. the framework for managing, simplifying, and facilitating interaction with LLMs. We define the lifecycle of the PF as a hierarchical structure, from bottom to top, namely: Data Level, Base Level, Execute Level, and Service Level. We also systematically depict the overall landscape of the emerging PF field and discuss potential future research and challenges. To continuously track the developments in this area, we maintain a repository at\n                    <jats:ext-link xmlns:xlink=\"https:\/\/2.zoppoz.workers.dev:443\/http\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/lxx0628\/Prompting-Framework-Survey\">https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/lxx0628\/Prompting-Framework-Survey<\/jats:ext-link>\n                    , which can be a useful resource sharing platform for both academic and industry in this field.\n                  <\/jats:p>","DOI":"10.1145\/3789253","type":"journal-article","created":{"date-parts":[[2026,2,21]],"date-time":"2026-02-21T09:47:47Z","timestamp":1771667267000},"page":"1-38","update-policy":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Prompting Frameworks for Large Language Models: A Survey"],"prefix":"10.1145","volume":"58","author":[{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0003-6662-9020","authenticated-orcid":false,"given":"Xiaoxia","family":"Liu","sequence":"first","affiliation":[{"name":"Zhejiang University","place":["Hangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0001-7113-7635","authenticated-orcid":false,"given":"Jingyi","family":"Wang","sequence":"additional","affiliation":[{"name":"Zhejiang University","place":["Hangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0009-0003-4866-245X","authenticated-orcid":false,"given":"Xiaohan","family":"Yuan","sequence":"additional","affiliation":[{"name":"Zhejiang University","place":["Hangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0002-3545-1392","authenticated-orcid":false,"given":"Jun","family":"Sun","sequence":"additional","affiliation":[{"name":"Singapore Management University School of Information Systems","place":["Singapore, Singapore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0002-4146-1749","authenticated-orcid":false,"given":"Guoliang","family":"Dong","sequence":"additional","affiliation":[{"name":"Singapore Management University","place":["Singapore, Singapore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0002-5799-5876","authenticated-orcid":false,"given":"Peng","family":"Di","sequence":"additional","affiliation":[{"name":"Ant Group CO Ltd","place":["Hangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0002-1936-2840","authenticated-orcid":false,"given":"Wenhai","family":"Wang","sequence":"additional","affiliation":[{"name":"Zhejiang University","place":["Hangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/2.zoppoz.workers.dev:443\/https\/orcid.org\/0000-0001-9812-3911","authenticated-orcid":false,"given":"Dongxia","family":"Wang","sequence":"additional","affiliation":[{"name":"Zhejiang University","place":["Hangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,4,1]]},"reference":[{"key":"e_1_3_3_2_2","volume-title":"Awesome ChatGPT Prompts","author":"Ak\u0131n Fatih Kadir","year":"2022","unstructured":"Fatih Kadir Ak\u0131n. 2022. Awesome ChatGPT Prompts. Retrieved December 20, 2022 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/f\/awesome-chatgpt-prompts"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2023.3254179"},{"key":"e_1_3_3_4_2","volume-title":"AutoRAG","year":"2024","unstructured":"AutoRAG. 2024. AutoRAG. Retrieved May 1, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/topics\/open-source"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","unstructured":"Stephen H. Bach Victor Sanh Zheng-Xin Yong Albert Webson Colin Raffel Nihal V. Nayak Abheesht Sharma Taewoon Kim M. Saiful Bari Thibault Fevry et\u00a0al. 2022. PromptSource: An integrated development environment and repository for natural language prompts. (May 2022) 93\u2013104. DOI:10.18653\/v1\/2022.acl-demo.9","DOI":"10.18653\/v1\/2022.acl-demo.9"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","unstructured":"Yejin Bang Samuel Cahyawijaya Nayeon Lee Wenliang Dai Dan Su Bryan Wilie Holy Lovenia Ziwei Ji Tiezheng Yu Willy Chung et\u00a0al. 2023. A multitask multilingual multimodal evaluation of chatgpt on reasoning hallucination and interactivity. (Nov. 2023) 675\u2013718. DOI:10.18653\/v1\/2023.ijcnlp-main.45","DOI":"10.18653\/v1\/2023.ijcnlp-main.45"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","unstructured":"Clark Barrett Brad Boyd Ellie Burzstein Nicholas Carlini Brad Chen Jihye Choi Amrita Roy Chowdhury Mihai Christodorescu Anupam Datta Soheil Feizi et\u00a0al. 2023. Identifying and mitigating the security risks of generative AI. Foundations and Trends in Privacy and Security 6 1 (12 2023) 1\u201352. DOI:10.1561\/3300000041","DOI":"10.1561\/3300000041"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2013.50"},{"key":"e_1_3_3_9_2","article-title":"A neural probabilistic language model","volume":"13","author":"Bengio Yoshua","year":"2000","unstructured":"Yoshua Bengio, R\u00e9jean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in Neural Information Processing Systems 13 (2000). https:\/\/2.zoppoz.workers.dev:443\/https\/proceedings.neurips.cc\/paper_files\/paper\/2000\/file\/728f206c2a01bf572b5940d7d9a8fa4c-Paper.pdf","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_10_2","unstructured":"Amanda Bertsch Uri Alon Graham Neubig and Matthew R. Gormley. 2023. Unlimiformer: Long-range transformers with unlimited length input. Article 1543 (2023) 22."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3591300"},{"key":"e_1_3_3_12_2","unstructured":"Rishi Bommasani Percy Liang Tony Lee Dimitris Tsipras Dilara Soylu Michihiro Yasunaga Yian Zhang Deepak Narayanan Yuhuai Wu Ananya Kumar Benjamin Newman Binhang Yuan Bobby Yan Ce Zhang Christian Cosgrove Christopher D. Manning Christopher R\u00e9 Diana Acosta-Navas Drew A. Hudson Eric Zelikman Esin Durmus Faisal Ladhak Frieda Rong Hongyu Ren Huaxiu Yao Jue Wang Keshav Santhanam Laurel Orr Lucia Zheng Mert Yuksekgonul Mirac Suzgun Nathan Kim Neel Guha Niladri Chatterji Omar Khattab Peter Henderson Qian Huang Ryan Chi Sang Michael Xie Shibani Santurkar Surya Ganguli Tatsunori Hashimoto Thomas Icard Tianyi Zhang Vishrav Chaudhary William Wang Xuechen Li Yifan Mai Yuhui Zhang and Yuta Koreeda. 2023. Holistic Evaluation of Language Models. (2023). arXiv:2211.09110 [cs.CL] https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2211.09110"},{"key":"e_1_3_3_13_2","volume-title":"Botpress","year":"2023","unstructured":"botpress. 2023. Botpress. Retrieved March 1, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/botpress\/botpress"},{"key":"e_1_3_3_14_2","article-title":"Language models are few-shot learners","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et\u00a0al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33, Article 159 (2020), 25.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_15_2","volume-title":"gpt-jargon","author":"Brukhman Jake","year":"2023","unstructured":"Jake Brukhman. 2023. gpt-jargon. Retrieved March 11, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/jbrukh\/gpt-jargon"},{"key":"e_1_3_3_16_2","unstructured":"Aydar Bulatov Yuri Kuratov and Mikhail S. Burtsev. 2023. Scaling transformer to 1M tokens and beyond with RMT. arXiv abs\/2304.11062 (2023). https:\/\/2.zoppoz.workers.dev:443\/https\/api.semanticscholar.org\/CorpusID:258291566"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","unstructured":"Yihan Cao Siyu Li Yixin Liu Zhiling Yan Yutong Dai Philip Yu and Lichao Sun. 2025. A Survey of AI-Generated Content (AIGC). ACM Comput. Surv. 57 5 Article 125 (Jan. 2025) 38. DOI:10.1145\/3704262","DOI":"10.1145\/3704262"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","unstructured":"Yong Cao Li Zhou Seolhwa Lee Laura Cabello Min Chen and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. (May 2023) 53\u201367. DOI:10.18653\/v1\/2023.c3nlp-1.7","DOI":"10.18653\/v1\/2023.c3nlp-1.7"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-023-01925-4"},{"key":"e_1_3_3_20_2","volume-title":"Chainlit","year":"2024","unstructured":"Chainlit. 2024. Chainlit. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/docs.chainlit.io\/get-started\/overview"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","unstructured":"Yupeng Chang Xu Wang Jindong Wang Yuan Wu Kaijie Zhu Hao Chen Linyi Yang Xiaoyuan Yi Cunxiang Wang Yidong Wang et\u00a0al. 2023. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol. 15 3 Article 39 (March 2024) 45. DOI:10.1145\/3641289","DOI":"10.1145\/3641289"},{"key":"e_1_3_3_22_2","volume-title":"LangChain","author":"Chase Harrison","year":"2022","unstructured":"Harrison Chase. 2022. LangChain. Retrieved December 10, 2022 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/hwchase17\/langchain"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","unstructured":"Le Chen Pei-Hung Lin Tristan Vanderbruggen Chunhua Liao Murali Emani and Bronis de Supinski. 2023. LM4HPC: Towards effective language model application in high-performance computing. (2023) 18\u201333. DOI:10.1007\/978-3-031-40744-4_2","DOI":"10.1007\/978-3-031-40744-4_2"},{"key":"e_1_3_3_24_2","unstructured":"Lingjiao Chen Matei Zaharia and James Zou. 2023. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research (2024). https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=cSimKw5p6R"},{"key":"e_1_3_3_25_2","unstructured":"Weize Chen Yusheng Su Jingwei Zuo Cheng Yang Chenfei Yuan Chen Qian Chi-Min Chan Yujia Qin Yaxi Lu Ruobing Xie et\u00a0al. 2023. AgentVerse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=EHg5GDnyq1"},{"key":"e_1_3_3_26_2","unstructured":"Xuanting Chen Junjie Ye Can Zu Nuo Xu Rui Zheng Minlong Peng Jie Zhou Tao Gui Qi Zhang and Xuanjing Huang. 2023. How robust is GPT-3.5 to predecessors? A comprehensive study on language understanding tasks. arXiv abs\/2303.00293 (2023). https:\/\/2.zoppoz.workers.dev:443\/https\/api.semanticscholar.org\/CorpusID:257255121"},{"key":"e_1_3_3_27_2","volume-title":"LLMStack: A Platform to Build and Deploy LLM Applications","author":"Chintala Ajay Kumar","unstructured":"Ajay Kumar Chintala and Vignesh Aigal. [n. d.]. LLMStack: A Platform to Build and Deploy LLM Applications. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/trypromptly\/llmstack"},{"key":"e_1_3_3_28_2","unstructured":"Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann et\u00a0al. 2023. PaLM: Scaling language modeling with pathways. (2023) 113."},{"key":"e_1_3_3_29_2","unstructured":"Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay William Fedus Eric Li Xuezhi Wang Mostafa Dehghani Siddhartha Brahma et\u00a0al. 2024. Scaling instruction-finetuned language models. J. Mach. Learn. Res. 25 1 Article 70 (Jan. 2024) 53."},{"key":"e_1_3_3_30_2","first-page":"2493","article-title":"Natural language processing (almost) from scratch","volume":"12","author":"Collobert Ronan","year":"2011","unstructured":"Ronan Collobert, Jason Weston, L\u00e9on Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12, ARTICLE (2011), 2493\u20132537.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","unstructured":"Wei Dai Jionghao Lin Flora Jin Tongguang Li Yi-Shan Tsai Dragan Gasevic and Guanliang Chen. 2023. Can large language models provide feedback to students? A case study on ChatGPT. (2023) 323\u2013325. DOI:10.1109\/ICALT58122.2023.00100","DOI":"10.1109\/ICALT58122.2023.00100"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","unstructured":"Gelei Deng Yi Liu Yuekang Li Kailong Wang Ying Zhang Zefeng Li Haoyu Wang Tianwei Zhang and Yang Liu. 2024. Jailbreaker: Automated jailbreak across multiple large language model chatbots. (2024). DOI:10.14722\/ndss.2024.24188","DOI":"10.14722\/ndss.2024.24188"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","unstructured":"Ameet Deshpande Vishvak Murahari Tanmay Rajpurohit Ashwin Kalyan and Karthik Narasimhan. 2023. Toxicity in ChatGPT: Analyzing persona-assigned language models. (Dec. 2023) 1236\u20131270. DOI:10.18653\/v1\/2023.findings-emnlp.88","DOI":"10.18653\/v1\/2023.findings-emnlp.88"},{"key":"e_1_3_3_34_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. (2019). https:\/\/2.zoppoz.workers.dev:443\/https\/api.semanticscholar.org\/CorpusID:52967399"},{"key":"e_1_3_3_35_2","volume-title":"Dify","year":"2024","unstructured":"Dify. 2024. Dify. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/dify.ai\/"},{"key":"e_1_3_3_36_2","volume-title":"Weaviate","author":"Dilocker Etienne","unstructured":"Etienne Dilocker, Bob van Luijt, Byron Voorbach, Mohd Shukri Hasan, Abdel Rodriguez, Dirk Alexander Kulawiak, Marcin Antas, and Parker Duckworth. [n. d.]. Weaviate. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/weaviate\/weaviate"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","unstructured":"Qingxiu Dong Lei Li Damai Dai Ce Zheng Zhiyong Wu Baobao Chang Xu Sun Jingjing Xu and Zhifang Sui. 2024. A survey for in-context learning. (Nov. 2024) 1107\u20131128. DOI:10.18653\/v1\/2024.emnlp-main.64","DOI":"10.18653\/v1\/2024.emnlp-main.64"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","unstructured":"Hongyang Du Zonghang Li Dusit Niyato Jiawen Kang Zehui Xiong Xuemin Shen and Dong In Kim. 2024. Enabling AI-generated content (AIGC) services in wireless edge networks. IEEE Wireless Communications 31 3 (2024) 226\u2013234. DOI:10.1109\/MWC.004.2300015","DOI":"10.1109\/MWC.004.2300015"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2023.102642"},{"key":"e_1_3_3_40_2","volume-title":"SudoLang","author":"Elliott Eric","year":"2023","unstructured":"Eric Elliott. 2023. SudoLang. Retrieved January 30, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/paralleldrive\/sudolang-llm-support"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","unstructured":"Emilio Ferrara. 2023. Should chatgpt be biased? Challenges and risks of bias in large language models. First Monday 28 11 (Nov. 2023). DOI:10.5210\/fm.v28i11.13346","DOI":"10.5210\/fm.v28i11.13346"},{"key":"e_1_3_3_42_2","volume-title":"Flappy","year":"2024","unstructured":"Flappy. 2024. Flappy. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/flappy.pleisto.com\/"},{"key":"e_1_3_3_43_2","volume-title":"Floom","year":"2024","unstructured":"Floom. 2024. Floom. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/floomai.github.io\/"},{"key":"e_1_3_3_44_2","volume-title":"AutoChain","year":"2023","unstructured":"Forethought-Technologies. 2023. AutoChain. Retrieved March 10, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/Forethought-Technologies\/AutoChain"},{"key":"e_1_3_3_45_2","volume-title":"gpt-Prompt-Engineer","author":"engineer gpt-prompt","year":"2024","unstructured":"gpt-prompt engineer. 2024. gpt-Prompt-Engineer. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/mshumer\/gpt-prompt-engineer"},{"key":"e_1_3_3_46_2","volume-title":"GPT4All","year":"2023","unstructured":"GPT4All. 2023. GPT4All. Retrieved June 10, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/www.nomic.ai\/gpt4all"},{"key":"e_1_3_3_47_2","unstructured":"Kai Greshake Sahar Abdelnabi Shailesh Mishra Christoph Endres Thorsten Holz and Mario Fritz. 2023. More than you\u2019ve asked for: A comprehensive analysis of novel prompt injection threats to application-integrated large language models. arXiv abs\/2302.12173 (2023). https:\/\/2.zoppoz.workers.dev:443\/https\/api.semanticscholar.org\/CorpusID:257102404"},{"key":"e_1_3_3_48_2","volume-title":"Griptape","author":"ai griptape","year":"2023","unstructured":"griptape ai. 2023. Griptape. Retrieved March 1, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/griptape-ai\/griptape"},{"key":"e_1_3_3_49_2","volume-title":"LinGoose","year":"2023","unstructured":"henomis. 2023. LinGoose. Retrieved March 1, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/henomis\/lingoose"},{"key":"e_1_3_3_50_2","unstructured":"Sirui Hong Mingchen Zhuge Jonathan Chen Xiawu Zheng Yuheng Cheng Jinlin Wang Ceyao Zhang zili wang Steven Yau Zijuan Lin Liyang Zhou Chenyu Ran Lingfeng Xiao Chenglin Wu and J\u00fcrgen Schmidhuber. 2024. MetaGPT: Meta programming for multi-agent collaborative framework. 23247\u201323275. https:\/\/2.zoppoz.workers.dev:443\/https\/proceedings.iclr.cc\/paper_files\/paper\/2024\/file\/6507b115562bb0a305f1958ccc87355a-Paper-Conference.pdf"},{"key":"e_1_3_3_51_2","unstructured":"Edward J. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. arXiv abs\/2106.09685 (2021). https:\/\/2.zoppoz.workers.dev:443\/https\/api.semanticscholar.org\/CorpusID:235458009"},{"key":"e_1_3_3_52_2","volume-title":"PromptFlow","year":"2023","unstructured":"InsuranceToolkits. 2023. PromptFlow. Retrieved March 1, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/InsuranceToolkits\/promptflow"},{"key":"e_1_3_3_53_2","unstructured":"Neel Jain Khalid Saifullah Yuxin Wen John Kirchenbauer Manli Shu Aniruddha Saha Micah Goldblum Jonas Geiping and Tom Goldstein. 2024. Bring your own data! Self-supervised evaluation for large language models. (2024). https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=zH6zBoktYO"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.5555\/280484"},{"key":"e_1_3_3_55_2","article-title":"An LLM compiler for parallel function calling","author":"Kim Sehoon","year":"2023","unstructured":"Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael Mahoney, Kurt Keutzer, and Amir Gholami. 2023. An LLM compiler for parallel function calling. In Proceedings of the International Conference on Machine Learning (ICML) 2025.","journal-title":"Proceedings of the International Conference on Machine Learning (ICML) 2025"},{"key":"e_1_3_3_56_2","article-title":"Large language models are zero-shot reasoners","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems 35, Article 1613 (2022), 15.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"1","key":"e_1_3_3_57_2","first-page":"3","article-title":"Supervised machine learning: A review of classification techniques","volume":"160","author":"Kotsiantis Sotiris B.","year":"2007","unstructured":"Sotiris B. Kotsiantis, Ioannis Zaharakis, P. Pintelas, et\u00a0al. 2007. Supervised machine learning: A review of classification techniques. Emerging Artificial Intelligence Applications in Computer Engineering 160, 1 (2007), 3\u201324.","journal-title":"Emerging Artificial Intelligence Applications in Computer Engineering"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_3_3_59_2","unstructured":"John Lafferty Andrew McCallum and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. (2001) 282\u2013289."},{"key":"e_1_3_3_60_2","volume-title":"Langroid","year":"2024","unstructured":"Langroid. 2024. Langroid. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/langroid.github.io\/langroid\/"},{"key":"e_1_3_3_61_2","volume-title":"LangStream","year":"2024","unstructured":"LangStream. 2024. LangStream. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/langstream.ai\/"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","unstructured":"Md Tahmid Rahman Laskar M. Saiful Bari Mizanur Rahman Md Amran Hossen Bhuiyan Shafiq Joty and Jimmy Xiangji Huang. 2023. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. (July 2023) 431\u2013469. DOI:10.18653\/v1\/2023.findings-acl.29","DOI":"10.18653\/v1\/2023.findings-acl.29"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","unstructured":"Mike Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdelrahman Mohamed Omer Levy Ves Stoyanov and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation translation and comprehension. (July 2020) 7871\u20137880. DOI:10.18653\/v1\/2020.acl-main.703","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"e_1_3_3_64_2","first-page":"3843","article-title":"Solving quantitative reasoning problems with language models","volume":"35","author":"Lewkowycz Aitor","year":"2022","unstructured":"Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et\u00a0al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.). 35 (2022), 3843\u20133857. https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=IFXTZERXdM7","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","unstructured":"Xinzhe Li Ming Liu Shang Gao and Wray Buntine. 2023. A survey on out-of-distribution evaluation of neural nlp models. Article 749 (2023) 9. DOI:10.24963\/ijcai.2023\/749","DOI":"10.24963\/ijcai.2023\/749"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","unstructured":"Yen-Ting Lin and Yun-Nung Chen. 2023. LLM-Eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. CoRR abs\/2305.13711 (2023). 10.48550\/arXiv.2305.13711","DOI":"10.48550\/arXiv.2305.13711"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.1234"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3560815"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocad072"},{"key":"e_1_3_3_70_2","unstructured":"Yi Liu Gelei Deng Yuekang Li Kailong Wang Tianwei Zhang Yepang Liu Haoyu Wang Yan Zheng and Yang Liu. 2025. Prompt injection attack against LLM-integrated applications. arXiv:2306.05499. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2306.05499"},{"key":"e_1_3_3_71_2","unstructured":"Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv:1907.11692. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/1907.11692"},{"key":"e_1_3_3_72_2","volume-title":"Swiss Army Llama","author":"Llama Swiss Army","year":"2024","unstructured":"Swiss Army Llama. 2024. Swiss Army Llama. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/Dicklesworthstone\/swiss_army_llama"},{"key":"e_1_3_3_73_2","volume-title":"llama.cpp","year":"2023","unstructured":"llama.cpp. 2023. llama.cpp. Retrieved March 10, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/ggerganov\/llama.cpp"},{"key":"e_1_3_3_74_2","unstructured":"Pan Lu Baolin Peng Hao Cheng Michel Galley Kai-Wei Chang Ying Nian Wu Song-Chun Zhu and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. Article 1882 (2023) 32."},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1002\/asi.24750"},{"key":"e_1_3_3_76_2","volume-title":"Magentic","year":"2024","unstructured":"Magentic. 2024. Magentic. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/magentic.dev\/"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","unstructured":"Kai Mei Zheng Li Zhenting Wang Yang Zhang and Shiqing Ma. 2023. NOTABLE: Transferable backdoor attacks against prompt-based NLP models. (July 2023) 15551\u201315565. DOI:10.18653\/v1\/2023.acl-long.867","DOI":"10.18653\/v1\/2023.acl-long.867"},{"key":"e_1_3_3_78_2","volume-title":"MemFree","year":"2024","unstructured":"MemFree. 2024. MemFree. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/memfreeme\/memfree"},{"key":"e_1_3_3_79_2","volume-title":"txtai: the all-in-one embeddings database","author":"Mezzetti David","year":"2020","unstructured":"David Mezzetti. 2020. txtai: the all-in-one embeddings database. Retrieved June 10, 2020 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/neuml\/txtai"},{"key":"e_1_3_3_80_2","unstructured":"Gr\u00e9goire Mialon Roberto Dess\u00ec Maria Lomeli Christoforos Nalmpantis Ram Pasunuru Roberta Raileanu Baptiste Rozi\u00e8re Timo Schick Jane Dwivedi-Yu Asli Celikyilmaz et\u00a0al. 2023. Augmented language models: A survey. arXiv:2302.07842. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2302.07842"},{"key":"e_1_3_3_81_2","volume-title":"Guidance","year":"2023","unstructured":"microsoft. 2023. Guidance. Retrieved December 30, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/guidance-ai\/guidance"},{"key":"e_1_3_3_82_2","volume-title":"TypeChat","year":"2023","unstructured":"microsoft. 2023. TypeChat. Retrieved December 30, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/microsoft\/TypeChat"},{"key":"e_1_3_3_83_2","unstructured":"Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In International Conference on Learning Representations. https:\/\/2.zoppoz.workers.dev:443\/https\/api.semanticscholar.org\/CorpusID:5959482"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2010-343"},{"key":"e_1_3_3_85_2","article-title":"Distributed representations of words and phrases and their compositionality","volume":"26","author":"Mikolov Tomas","year":"2013","unstructured":"Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems 26 (2013), 3111\u20133119.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3605943"},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","unstructured":"Sewon Min Xinxi Lyu Ari Holtzman Mikel Artetxe Mike Lewis Hannaneh Hajishirzi and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? (Dec. 2022) 11048\u201311064. DOI:10.18653\/v1\/2022.emnlpmain.759","DOI":"10.18653\/v1\/2022.emnlpmain.759"},{"key":"e_1_3_3_88_2","volume-title":"MindSQL","year":"2024","unstructured":"MindSQL. 2024. MindSQL. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/Mindinventory\/MindSQL"},{"key":"e_1_3_3_89_2","volume-title":"ModelFusion","year":"2024","unstructured":"ModelFusion. 2024. ModelFusion. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/modelfusion.dev\/guide\/"},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.3390\/computers12080153"},{"key":"e_1_3_3_91_2","volume-title":"BabyAGI","author":"Nakajima Yohei","year":"2023","unstructured":"Yohei Nakajima. 2023. BabyAGI. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/yoheinakajima\/babyagi"},{"key":"e_1_3_3_92_2","unstructured":"OpenAI. 2024. Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et\u00a0al. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_3_93_2","volume-title":"NeMo-Guardrails","year":"2023","unstructured":"NVIDIA. 2023. NeMo-Guardrails. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/NVIDIA\/NeMo-Guardrails"},{"key":"e_1_3_3_94_2","first-page":"161","volume-title":"Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004","author":"Och Franz Josef","year":"2004","unstructured":"Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alexander Fraser, Shankar Kumar, Libin Shen, David A. Smith, Katherine Eng, et\u00a0al. 2004. A smorgasbord of features for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004. 161\u2013168."},{"key":"e_1_3_3_95_2","volume-title":"Ollama","year":"2023","unstructured":"Ollama. 2023. Ollama. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/ollama.com\/"},{"key":"e_1_3_3_96_2","volume-title":"Evals","year":"2023","unstructured":"OpenAI. 2023. Evals. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/openai\/evals"},{"key":"e_1_3_3_97_2","unstructured":"OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774 [cs.CL]. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_3_98_2","volume-title":"OpenDAN","year":"2023","unstructured":"OpenDAN.ai. 2023. OpenDAN. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/www.opendan.ai\/"},{"key":"e_1_3_3_99_2","unstructured":"Jonas Oppenlaender and Joonas H\u00e4m\u00e4l\u00e4inen. 2023. Mapping the challenges of HCI: An application and evaluation of ChatGPT and GPT-4 for cost-efficient question answering."},{"key":"e_1_3_3_100_2","article-title":"Training language models to follow instructions with human feedback","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et\u00a0al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, Article 2011 (2022), 15.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_101_2","unstructured":"Ankit Pal. 2022. Promptify: Structured output from LLMs. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/promptslab\/Promptify. Prompt-Engineering components for NLP tasks in Python."},{"key":"e_1_3_3_102_2","unstructured":"Rodrigo Pedro Daniel Castro Paulo Carreira and Nuno Santos. 2023. From prompt injections to SQL injection attacks: How protected is your LLM-integrated web application? arXiv:2308.01990. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2308.01990"},{"key":"e_1_3_3_103_2","unstructured":"Baolin Peng Chunyuan Li Pengcheng He Michel Galley and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv:2304.03277. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2304.03277"},{"key":"e_1_3_3_104_2","unstructured":"F\u00e1bio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv:2211.09527. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2211.09527"},{"key":"e_1_3_3_105_2","volume-title":"Haystack: The End-to-End NLP Framework for Pragmatic Builders","author":"Pietsch Malte","year":"2019","unstructured":"Malte Pietsch, Timo M\u00f6ller, Bogdan Kostic, Julian Risch, Massimiliano Pippi, Mayank Jobanputra, Sara Zanzottera, Silvano Cerza, Vladimir Blagojevic, Thomas Stadelmann, et\u00a0al. 2019. Haystack: The End-to-End NLP Framework for Pragmatic Builders. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/deepset-ai\/haystack"},{"key":"e_1_3_3_106_2","volume-title":"Pinecone","year":"2023","unstructured":"Pinecone. 2023. Pinecone. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/pinecone-io"},{"key":"e_1_3_3_107_2","volume-title":"PromptHub","year":"2023","unstructured":"PromptHub. 2023. PromptHub. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/deepset-ai\/prompthub"},{"key":"e_1_3_3_108_2","volume-title":"QA-Pilot","year":"2024","unstructured":"QA-Pilot. 2024. QA-Pilot. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/reid41\/QA-Pilot"},{"key":"e_1_3_3_109_2","volume-title":"QAnything","year":"2024","unstructured":"QAnything. 2024. QAnything. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/qanything.ai\/"},{"key":"e_1_3_3_110_2","unstructured":"Chengwei Qin Aston Zhang Zhuosheng Zhang Jiaao Chen Michihiro Yasunaga and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? In The 2023 Conference on Empirical Methods in Natural Language Processing. https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=u03xn1COsO"},{"key":"e_1_3_3_111_2","doi-asserted-by":"publisher","unstructured":"Yujia Qin Shengding Hu Yankai Lin Weize Chen Ning Ding Ganqu Cui Zheni Zeng Xuanhe Zhou Yufei Huang Chaojun Xiao Chi Han Yi Ren Fung Yusheng Su Huadong Wang Cheng Qian Runchu Tian Kunlun Zhu Shihao Liang Xingyu Shen Bokai Xu Zhen Zhang Yining Ye Bowen Li Ziwei Tang Jing Yi Yuzhang Zhu Zhenning Dai Lan Yan Xin Cong Yaxi Lu Weilin Zhao Yuxiang Huang Junxi Yan Xu Han Xian Sun Dahai Li Jason Phang Cheng Yang Tongshuang Wu Heng Ji Guoliang Li Zhiyuan Liu and Maosong Sun. 2024. Tool learning with foundation models. ACM Comput. Surv. 57 4 Article 101 (Dec. 2024) 40. DOI:10.1145\/3704435","DOI":"10.1145\/3704435"},{"key":"e_1_3_3_112_2","unstructured":"Alec Radford Jeff Wu Rewon Child David Luan Dario Amodei and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. (2019)."},{"key":"e_1_3_3_113_2","doi-asserted-by":"publisher","DOI":"10.5555\/3455716.3455856"},{"key":"e_1_3_3_114_2","volume-title":"ReLLM","author":"Rickard Matt","year":"2023","unstructured":"Matt Rickard. 2023. ReLLM. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/r2d4\/rellm"},{"key":"e_1_3_3_115_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.880083"},{"key":"e_1_3_3_116_2","volume-title":"PromptLang","year":"2023","unstructured":"ruvnet. 2023. PromptLang. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/ruvnet\/promptlang"},{"key":"e_1_3_3_117_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.psychres.2021.114135"},{"key":"e_1_3_3_118_2","volume-title":"TermGPT","year":"2023","unstructured":"Sentdex. 2023. TermGPT. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/Sentdex\/TermGPT"},{"key":"e_1_3_3_119_2","volume-title":"Guardrails","year":"2023","unstructured":"ShreyaR. 2023. Guardrails. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/ShreyaR\/guardrails"},{"key":"e_1_3_3_120_2","volume-title":"Auto-GPT","author":"Gravitas Significant","unstructured":"Significant Gravitas. [n. d.]. Auto-GPT. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/Significant-Gravitas\/Auto-GPT"},{"key":"e_1_3_3_121_2","volume-title":"Embedchain","author":"Singh Taranjeet","year":"2023","unstructured":"Taranjeet Singh. 2023. Embedchain. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/embedchain\/embedchain"},{"key":"e_1_3_3_122_2","volume-title":"llm-chain","year":"2023","unstructured":"sobelio. 2023. llm-chain. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/sobelio\/llm-chain"},{"key":"e_1_3_3_123_2","doi-asserted-by":"publisher","DOI":"10.3115\/1034678.1034712"},{"key":"e_1_3_3_124_2","volume-title":"Hyv","year":"2023","unstructured":"TimPietrusky. 2023. Hyv. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/failfa-st\/hyv"},{"key":"e_1_3_3_125_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et\u00a0al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_3_126_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et\u00a0al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_3_127_2","volume-title":"SuperAGI","year":"2023","unstructured":"TransformerOptimus. 2023. SuperAGI. Retrieved January 30, 2024 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/TransformerOptimus\/SuperAGI"},{"key":"e_1_3_3_128_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017), 6000\u20136010.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_129_2","volume-title":"Semantic-kernel","author":"Perdana Gil LaHaye Vic","year":"2023","unstructured":"Gil LaHaye Vic Perdana. 2023. Semantic-kernel. Retrieved March 1, 2023 from https:\/\/2.zoppoz.workers.dev:443\/https\/github.com\/microsoft\/semantic-kernel"},{"key":"e_1_3_3_130_2","unstructured":"Jindong Wang Xixu Hu Wenxin Hou Hao Chen Runkai Zheng Yidong Wang Linyi Yang Haojun Huang Wei Ye Xiubo Geng et\u00a0al. 2023. On the robustness of chatgpt: An adversarial and out-of-distribution perspective. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models. https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=uw6HSkgoM29"},{"key":"e_1_3_3_131_2","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457550"},{"key":"e_1_3_3_132_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00660"},{"key":"e_1_3_3_133_2","doi-asserted-by":"publisher","unstructured":"Yizhong Wang Yeganeh Kordi Swaroop Mishra Alisa Liu Noah A. Smith Daniel Khashabi and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language model with self generated instructions. (July 2023) 13484\u201313508. DOI:10.18653\/v1\/2023.acl-long.754","DOI":"10.18653\/v1\/2023.acl-long.754"},{"key":"e_1_3_3_134_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-023-01659-w"},{"key":"e_1_3_3_135_2","doi-asserted-by":"crossref","unstructured":"Alexander Wei Nika Haghtalab and Jacob Steinhardt. 2023. Jailbroken: How does LLM safety training fail? 36 (2023) 80079\u201380110. https:\/\/2.zoppoz.workers.dev:443\/https\/proceedings.neurips.cc\/paper_files\/paper\/2023\/file\/fd6613131889a4b656206c50a8bd7790-Paper-Conference.pdf","DOI":"10.52202\/075280-3508"},{"key":"e_1_3_3_136_2","unstructured":"Jason Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph Sebastian Borgeaud Dani Yogatama Maarten Bosma Denny Zhou Donald Metzler et\u00a0al. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research (2022). https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=yzkSU5zdwD"},{"key":"e_1_3_3_137_2","article-title":"Chain-of-thought prompting elicits reasoning in large language models","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou, et\u00a0al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, Article 1800 (2022), 14.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_138_2","unstructured":"Jerry Wei Jason Wei Yi Tay Dustin Tran Albert Webson Yifeng Lu Xinyun Chen Hanxiao Liu Da Huang Denny Zhou et\u00a0al. 2023. Larger language models do in-context learning differently. arXiv:2303.03846. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2303.03846"},{"key":"e_1_3_3_139_2","doi-asserted-by":"publisher","unstructured":"Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Perric Cistac Clara Ma Yacine Jernite Julien Plu et\u00a0al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations Qun Liu and David Schlangen (Eds.). Association for Computational Linguistics 38\u201345. DOI:10.18653\/v1\/2020.emnlp-demos.6","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_3_3_140_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.bea-1.52"},{"key":"e_1_3_3_141_2","unstructured":"Jun Yan Vikas Yadav Shiyang Li Lichang Chen Zheng Tang Hai Wang Vijay Srinivasan Xiang Ren and Hongxia Jin. 2024. Backdooring instruction-tuned large language models with virtual prompt injection. https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=b3KgHQos7P"},{"key":"e_1_3_3_142_2","doi-asserted-by":"publisher","unstructured":"Jingfeng Yang Hongye Jin Ruixiang Tang Xiaotian Han Qizhang Feng Haoming Jiang Bing Yin and Xia Hu. 2024. Harnessing the power of LLMs in practice: A survey on ChatGPT and beyond. ACM Trans. Knowl. Discov. Data 18 6 Article 160 (April 2024) 32. DOI:10.1145\/3649506","DOI":"10.1145\/3649506"},{"key":"e_1_3_3_143_2","unstructured":"Xianjun Yang Yan Li Xinlu Zhang Haifeng Chen and Wei Cheng. 2023. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv:2302.08081. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2302.08081"},{"key":"e_1_3_3_144_2","article-title":"Xlnet: Generalized autoregressive pretraining for language understanding","volume":"32","author":"Yang Zhilin","year":"2019","unstructured":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems 32, Article 517 (2019), 11.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_145_2","doi-asserted-by":"publisher","unstructured":"Zhilin Yang Peng Qi Saizheng Zhang Yoshua Bengio William W. Cohen Ruslan Salakhutdinov and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse explainable multi-hop question answering. (Oct.\u2013Nov. 2018) 2369\u20132380. 10.18653\/v1\/D18-1259","DOI":"10.18653\/v1\/D18-1259"},{"key":"e_1_3_3_146_2","unstructured":"Shunyu Yao Dian Yu Jeffrey Zhao Izhak Shafran Thomas L. Griffiths Yuan Cao and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems. https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=5Xc1ecxO1h"},{"key":"e_1_3_3_147_2","unstructured":"Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. arXiv:2210.03629. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2210.03629"},{"key":"e_1_3_3_148_2","doi-asserted-by":"publisher","unstructured":"Xiaohan Yuan Jinfeng Li Dongxia Wang Yuefeng Chen Xiaofeng Mao Longtao Huang Hui Xue Wenhai Wang Kui Ren and Jingyi Wang. 2025. S-Eval: Automatic and adaptive test generation for benchmarking safety evaluation of large language models. In Proceedings of the ACM Softw. Eng. 2 ISSTA Article ISSTA094 (June 2025) 22. DOI:10.1145\/3728971","DOI":"10.1145\/3728971"},{"key":"e_1_3_3_149_2","unstructured":"Zhanpeng Zeng Cole Hawkins Mingyi Hong Aston Zhang Nikolaos Pappas Vikas Singh and Shuai Zheng. 2023. VCC: Scaling transformers to 128K tokens or more by prioritizing important tokens. Article 889 (2023) 27."},{"key":"e_1_3_3_150_2","unstructured":"Chaoning Zhang Chenshuang Zhang Chenghao Li Yu Qiao Sheng Zheng Sumit Kumar Dam Mengchun Zhang Jung Uk Kim Seong Tae Kim Jinwoo Choi et\u00a0al. 2023. One small step for generative AI one giant leap for AGI: A complete survey on ChatGPT in AIGC era. arXiv:2304.06488. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2304.06488"},{"key":"e_1_3_3_151_2","unstructured":"Chaoning Zhang Chenshuang Zhang Sheng Zheng Yu Qiao Chenghao Li Mengchun Zhang Sumit Kumar Dam Chu Myaet Thwal Ye Lin Tun Le Luang Huy et\u00a0al. 2023. A complete survey on generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 all you need? arXiv:2303.11717. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2303.11717"},{"key":"e_1_3_3_152_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong Yifan Du Chen Yang Yushuo Chen Zhipeng Chen Jinhao Jiang Ruiyang Ren Yifan Li Xinyu Tang Zikang Liu Peiyu Liu Jian-Yun Nie and Ji-Rong Wen. 2026. A survey of large language models. arXiv:2303.18223. Retrieved from https:\/\/2.zoppoz.workers.dev:443\/https\/arxiv.org\/abs\/2303.18223"},{"key":"e_1_3_3_153_2","unstructured":"Lianmin Zheng Liangsheng Yin Zhiqiang Xie Jeff Huang Chuyue Sun Cody_Hao Yu Shiyi Cao Christos Kozyrakis Ion Stoica Joseph E. Gonzalez Clark Barrett and Ying Sheng. 2024. Efficiently programming large language models using sglang. Article 2000 (2024) 27."},{"key":"e_1_3_3_154_2","unstructured":"Denny Zhou Nathanael Sch\u00e4rli Le Hou Jason Wei Nathan Scales Xuezhi Wang Dale Schuurmans Claire Cui Olivier Bousquet Quoc V. Le and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations. https:\/\/2.zoppoz.workers.dev:443\/https\/openreview.net\/forum?id=WZH7099tgfM"},{"issue":"2","key":"e_1_3_3_155_2","first-page":"7","article-title":"Exploring chatGPT\u2019S potential for consultation, recommendations and report diagnosis: Gastric cancer and gastroscopy reports\u2019 case","volume":"8","author":"Zhou Jiaming","year":"2023","unstructured":"Jiaming Zhou, Tengyue Li, Simon James Fong, Nilanjan Dey, and Rub\u00e9n Gonz\u00e1lez Crespo. 2023. Exploring chatGPT\u2019S potential for consultation, recommendations and report diagnosis: Gastric cancer and gastroscopy reports\u2019 case. International Journal of Interactive Multimedia & Artificial Intelligence 8, 2 (2023), 7\u201313.","journal-title":"International Journal of Interactive Multimedia & Artificial Intelligence"},{"key":"e_1_3_3_156_2","doi-asserted-by":"publisher","unstructured":"Kaijie Zhu Jindong Wang Jiaheng Zhou Zichen Wang Hao Chen Yidong Wang Linyi Yang Wei Ye Yue Zhang Neil Gong and Xing Xie. 2024. PromptBench: Towards evaluating the robustness of large language models on adversarial prompts. (2024) 57\u201368. DOI:10.1145\/3689217.3690621","DOI":"10.1145\/3689217.3690621"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/dl.acm.org\/doi\/pdf\/10.1145\/3789253","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T20:42:23Z","timestamp":1775853743000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/3789253"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,1]]},"references-count":155,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2026,7,31]]}},"alternative-id":["10.1145\/3789253"],"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1145\/3789253","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,1]]},"assertion":[{"value":"2023-11-18","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-26","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}