FACE: A Fine-Grained Reference-Free Evaluator for Conversational Information AccessDOI: XXXXXXX.XXXXXXXConference: The 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, AustraliaProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, AustraliaISBN: 978-1-4503-XXXX-X/2026/07

Hideaki Joko Affiliation: Radboud University , The Netherlands email: hideaki.joko@ru.nl and Faegheh Hasibi Affiliation: Radboud University , The Netherlands email: faegheh.hasibi@ru.nl

2026© , 2026;

Abstract.

A systematic, reliable, and low-cost evaluation of Conversational Information Access (CIA) systems remains an open challenge. Existing reference-based evaluation methods are proven insufficient for evaluating the dynamic nature of information access conversations, while existing LLM-based reference-free methods suffer from evaluation bias and limited generalizability. This work proposes FACE: a Fine-grained, Aspect-based Conversation Evaluation method that provides evaluation scores for diverse turn and dialogue-level aspects of conversations. FACE leverages beam search and bandit optimization to select optimized LLM instructions per evaluation aspect. It assigns scores to atomic information units (particles) using the selected instructions and then aggregates them into a single score. We show that FACE achieves strong correlation with human judgments, achieving system correlation of 0.9 outperforming state-of-the-art conversation evaluation methods by a large margin. We further demonstrate its optimized instructions are transferable across various LLMs and datasets. Additionally, unlike existing LLM-based methods that provide single uninterpretable scores, FACE provides insights into the system performance and enables identifying and locating problems within conversations.

https://github.com/informagi/face

Refer to caption — Figure 1. Illustration of FACE for a turn-level aspect. Instruction optimization generates a set of diverse evaluation instructions for the given aspect (e.g., relevance) based on an initial instruction; see example instructions in Section 5. For evaluation, each conversation is decomposed into particles containing a dialogue act, an atomic statement (mention), and user feedback from the following turn. A score distribution is created for each instruction-particle pair and the weighted summation of the scores is computed. The final score is obtained by aggregating the scores across all instructions and particles. For turn-level aspects, aggregation is performed per turn, while for dialogue-level aspects, scores of particles across the entire dialogue are aggregated.

1. Introduction

Recent advances in generative large language models (LLMs) have reshaped information access systems, enabling rich human-system conversational interactions. In parallel, offline evaluation of information access systems is also evolving with the increasing adoption of LLM–based approaches (54). In industrial settings, it is common practice to design evaluation prompts for specific aspects of a system, to reduce the high cost of human evaluation, and to help early identification of system issues during the development phase (14; 18). The research community, while maintaining skepticism, has valued the potential of LLM-based evaluation, and various LLM-based evaluation methods have been proposed for information access systems (15; 5; 16; 56; 22; 53; 4; 55).

Research gap. Most research on LLM-based evaluation for information access systems revolves around assessing the relevance of retrieved passages (55; 56; 16) or comparing generated texts against gold nuggets (42; 53). Similar efforts have been made for the offline evaluation of Conversational Information Access (CIA), where gold nuggets are generated for predefined trajectories of conversations (1; 4; 3). However, evaluation of conversational systems requires going beyond measuring factual accuracy over constrained conversational trajectories. 46 identified twenty evaluation criteria (or aspects) beyond factual correctness for conversational systems (e.g., modesty and conciseness), along with a set of requirements that a conversational evaluation method should satisfy. To date, this broader perspective has not been fully realized. As highlighted in the latest IR strategic meeting (SWIRL), developing “reliable and informative methods for evaluating dynamic trajectories of conversations at scale” is still an open question (54).

Research goal. In this paper, we aim to investigate “how dynamic conversational trajectories can be evaluated beyond factual accuracy.” Developing such an evaluation method poses several challenges. Reference-based evaluation methods restrict the evaluation process to predefined conversational trajectories. Even recent simulation-based approaches, such as those used in TREC iKAT 2025, remain constrained by predefined rubric questions (4). Reference-free conversational evaluation methods, on the other hand, mostly rely on LLM-based prompting and produce a single, uninterpretable score (39; 75; 36), making it difficult to trace the score back to the underlying contributing factors. This violates the Specificity requirement of conversational evaluation methods, which requires evaluation methods to “be specific when locating problem(s) within conversations” (46).

Method. We propose FACE, a Fine-grained Aspect-based Conversation Evaluation method, which is a reference-free approach for evaluating conversations across multiple evaluation aspects. We make the following departures from existing LLM-based conversation evaluation methods. First, we mimic the human evaluation process by modeling the diverse thought processes of multiple human assessors through a pool of optimized evaluation instructions. These optimized instructions, obtained using beam search and bandit optimization, reduce the arbitrary nature of prompt engineering and make the evaluation process robust. Second, we introduce the concept of “conversation particle,” which encompasses a composite nugget design consisting of an atomic statement, a dialogue act, and user feedback. These conversation particles enable evaluating an atomic statement, while providing a broader view of the overall conversational state. By evaluating these conversation particles using a set of optimized instructions, our method achieves specificity. Additionally, by aggregating these scores at the turn or dialogue level, FACE supports both turn-level and dialogue-level evaluation aspects. An overall overview of FACE is illustrated in Figure 1.

Meta-Evaluation. To evaluate FACE, we collect 20,962 human annotations for 467 human-system conversations of the CRSArena-Dial dataset (8). These conversations are obtained from interaction of real users with nine diverse conversational recommender systems. Human annotation of these conversations cover evaluation scores for seven evaluation aspects that are shown to be predictive of response quality and user satisfaction (50; 49). These include two turn-level aspects: relevance and interestingness, and five dialogue-level aspects: understanding, task completion, conversation efficiency, interest arousal, and overall impression. FACE is then applied to these evaluation aspects and the obtained evaluation scores are compared to human evaluation scores.

Experiments. Our experiments demonstrate that FACE outperforms state-of-the-art evaluation methods by a large margin, achieving system and turn/dialogue-level Spearman of 0.9 and 0.5, respectively, without observing any system-human conversations for instruction optimization. We demonstrate that FACE provides a reusable instruction pool that can be robustly used across different LLMs and conversation types. Remarkably, FACE outperforms strong baselines that utilize GPT-4 as their backbone on two chit- chat datasets, despite using an 8B-parameter Llama 3.1 as its backbone. Lastly, and importantly, we present a case study of two competitive conversational systems and demonstrate how FACE scores can diagnose issues and pinpoint problems within a conversation.

Key contributions of this paper are summarized as follows:

•

We propose FACE, a conversation evaluation method that evaluates dynamic trajectories of conversations on diverse evaluation aspects and significantly outperforms state-of-the-art evaluation methods (Sec. 6.1).
•

We introduce the notion of conversation particles and demonstrate their effectiveness in reducing bias (Sec. 7) and improving evaluation sample efficiency (Sec. 7).
•

We show that utilizing a pool of optimized instructions mitigates the arbitrary nature of prompt engineering for automatic evaluation and is robust across different LLMs and conversation types (Sec. 6.2).
•

We demonstrate that FACE scores are beyond a single uninterpretable score and can be used by humans to identify and locate potential system issues (Sec. 6.3).
•

We develop the CRSArena-Eval meta-evaluation dataset, containing 20,962 human annotations over 467 conversations, for evaluating conversational evaluators. The dataset and a meta-evaluation interface for evaluating an evaluator against CRSArena-Eval are available in our repository.

2. Related Work

Automatic Conversation Evaluation. Although human annotations are the gold standard for evaluating CIA systems, they are expensive and time-consuming. Therefore, automatic evaluation methods have been proposed to scale up the evaluation process. There are two main types of automatic evaluation methods: reference-based and reference-free (51).

Reference-based methods use gold references to evaluate system responses, which include Recall@K, BLEU (41), ROUGE (32), and BERTScore (72). While these methods are effective for machine translation and question-answering, they have limitations in conversation evaluation, as they overlook diverse response possibilities and various evaluation aspects. These limitations are supported by various studies showing a weak correlation between reference-based methods and human evaluations (39; 8; 35; 7).

Reference-free methods have been proposed to address these limitations (39; 75; 36). These methods evaluate system responses without relying on gold references, consider multiple aspects of system quality, and allow for the assessment of various response possibilities. However, these methods primarily focus on turn-level evaluation, limiting their ability to assess the whole conversation, which are crucial for evaluating system performance in real-world scenarios (49; 50). FACE addresses these limitations by evaluating the system based on the whole conversation and capturing multiple conversation trajectories from diverse user-system interactions.

LLMs for Evaluation. Recent advancements in large language models (LLMs) have led to various automatic evaluation methods, ranging from simple prompting to user simulation, that show strong performance (56; 18; 36; 34; 73; 14; 22; 4; 53). However, improved performance of LLM-based methods comes with challenges. Evaluation bias is one of them; LLMs tend to favor longer responses (length bias) (63; 18) and are biased toward texts from similar models (self-bias) (65; 36; 5). Generalizability is another challenge: LLMs are sensitive to handcrafted, arbitrary prompts, which are not reusable for different models (48; 45). FACE mitigates these biases with conversation particles and instruction optimization (see Sec. 7).

Instruction optimization. To address the arbitrary nature of handcrafted prompts, various instruction optimization (or prompt optimization) methods have been proposed (10; 29; 43; 68; 12; 23; 76; 67; 70; 66). 76 introduced APE, an automatic prompt engineering method that uses LLMs to generate prompt candidates, and perform a Monte Carlo search to find the optimal prompt. 29 proposed PRewrite, where prompts are optimized using proximal policy optimization (PPO) (47). 43 proposed an instruction optimization method using “textual gradient,” which provides natural language feedback for an LLM to optimize prompts. These studies focus on common tasks like QA and classification, leaving conversation evaluation unexplored. More importantly, optimized prompts are not generalizable between LLMs (76), which is crucial for evaluation tasks, where reusability is essential. In this work, we present a method for applying a textual gradient approach for conversation evaluation alongside effective strategies for transferring optimized prompts across different settings.

Nugget-based Evaluation. To assign granular evaluation scores, multiple nugget-based evaluation approaches are proposed (58; 38; 42; 69; 33; 20; 52; 44; 16). Nugget-based evaluation was proposed (58) to assign granular scores to system responses for non-binary queries, wherein a nugget, which is an atomic piece of information, serves as the unit of evaluation, enabling a more traceable assessment. Although the original nugget-based evaluation was intended as a manual method, many efforts have aimed to automate or semi-automate it (38; 42; 33; 20; 52; 44; 16; 1; 25). One of the earlier methods is POURPRE (33), an automatic nugget-based evaluation method that uses n-gram co-occurrences to assess nugget presence in system responses. More recent research employs LLMs for nugget matching; 42 introduced the AutoNuggetizer framework in TREC RAG 2024, where LLMs automatically create nuggets and assign them to system responses. However, these works are reference-based and/or focus on individual responses, and more importantly, they do not target information-seeking conversations.

46 proposes a conceptual framework to evaluate CIA systems by decomposing user-system interactions into units and assessing them on various aspects. While the concept is promising, its execution remains an open question, specifically how to automatically create and assess nuggets while ensuring the method’s generalizability. This work addresses these challenges.

3. Method

Our Fine-Grained Aspect-based Conversation Evaluation (FACE) approach handles the one-to-many nature of conversations and provides detailed scores for an evaluation aspect. Figure 1 illustrates the FACE method. Using beam search and bandit algorithms, FACE first optimizes a set of instructions for a given evaluation aspect. During evaluation, a dialogue is decomposed into conversation particles, each containing a dialogue act (e.g., “Recommendation”), mention (e.g., “How about Inception”), and corresponding user feedback from the user response (e.g., “Inception seems interesting”). Each particle is independently evaluated with optimized instructions via an LLM, generating score distributions and resulting in turn/dialogue-level scores for a given aspect.

We note, without detailed elaboration, that FACE is applicable to a broad range of evaluation aspects (46) and conversation types (e.g., recommendation, information seeking, task-oriented, and chit-chat dialogues). In this paper, we build on existing literature and available data resources (8) and apply FACE to conversational recommender systems for seven widely recognized turn- and dialogue-level evaluation aspects, following (49); see Section 4.1 for description of these aspects.

This section describes the evaluation steps of FACE: particle generation (Sec. 3.1) and evaluation score computation (Sec. 3.2), followed by the instruction optimization process (Sec. 3.3).

3.1. Conversation Particle Generation

FACE sets two goals: (1) enable reference-free evaluations to address the one-to-many nature of natural conversation evaluation, and (2) provide fine-grained scores at both turn- and dialogue-level to locate undesired system behavior within the conversation (46). To achieve these, we introduce conversation particle, a self-contained information unit decomposed from conversations. Each particle is composed of three parts: (i) Dialogue act is the system’s action associated with the particle, such as “recommendation” or “preference elicitation;” (ii) Mention denotes the atomic statement within the system’s response, like “How about the movie A?”; and (iii) Feedback is the user’s evaluative reply, for instance, “The movie A seems interesting.” Following (26), we use 5 dialogue acts: greetings, preference elicitation, recommendation, goodbye, and others.

We instruct an LLM to decompose system responses into particles, denoted as decomposer $\mathcal{D}$ in the rest of the paper. Let $r_{t}$ be the target system response at turn $t$ , $h$ the dialogue history preceding $r_{t}$ , and $r_{t+1}$ the user’s turn following $r_{t}$ . The decomposer $\mathcal{D}$ maps $(h,r_{t},r_{t+1})$ to conversation particles $\mathbf{P}_{r}$ :

\mathbf{P}_{r}=\mathcal{D}(h,r_{t},r_{t+1}),

where each particle $p\in\mathbf{P}_{r}$ is a triplet (act, mention, feedback). The full particle list for a dialogue, $\mathbf{P}_{d}$ , is the union of particles from all responses with the dialogue, i.e., $\mathbf{P}_{d}=\bigcup_{r\in d}\mathbf{P}_{r}$ .

The Prompts paragraph in Section 5 details prompts used for particle generation. It is shown that LLMs are more effective than traditional methods like dependency parsing and information extraction for decomposing texts into atomic units (42; 2).

3.2. Evaluation Score Computation

FACE utilizes optimized instructions to generate scores for each conversation particle. Formally, given a particle $p$ and the evaluation instruction $I^{a}$ for the aspect $a$ , an LLM generates a response $r_{p}^{a}$ . To address known issues with LLM-generated scores, such as low variance and their noise (36), we obtain a response distribution $\{r_{p,i}^{a}\}_{i=1}^{n}$ and compute a weighted sum over the response set:

(1)

\mathcal{E}_{\text{particle}}(I^{a},p)=\sum_{i=1}^{n}r_{p,i}^{a}P(r_{p,i}^{a}|I^{a},p;\theta),

where $P(.)$ is a probability of $r_{p,i}^{a}$ from an LLM parameterized by $\theta$ , and $n$ is the number of sampled evaluation responses. These particle scores are then aggregated per turn or conversation by taking their mean:

\mathcal{E}(I^{a},\mathbf{P}_{x})=\frac{1}{|\mathbf{P}_{x}|}\sum_{p\in\mathbf{P}_{x}}\mathcal{E}_{\text{particle}}(I^{a},p),

where $\mathbf{P}_{x}$ is the set of particles for a given turn or dialogue, depending on evaluation aspect $a$ ; e.g., for relevance, aggregation is performed over particles of a turn, and for task completion aggregation is done for all particles of the dialogue.

A unique feature of FACE is utilizing diverse reasoning paths for each evaluation aspect, which is obtained by selecting optimized chain-of-thought (CoT) instructions. The intuition is that evaluation requires complex reasoning, and an optimal answer can be obtained by marginalizing various thought paths (62). Here, a set of top-performing optimized instructions with various CoT instructions, $\mathbf{I}^{a}$ , are applied to particles, and the resulting scores are aggregated to obtain the final score $s^{a}$ :

s^{a}=\operatorname{FACE}(\mathbf{I}^{a},\mathbf{P}_{x})=\frac{1}{|\mathbf{I}^{a}|}\,\sum_{I^{a}\in\mathbf{I}^{a}}\mathcal{E}(I^{a},\mathbf{P}_{x}).

3.3. Instruction Optimization

Instruction optimizer generates diverse optimized CoT instructions for a given evaluation aspect. The goal is to obtain a representative set of thought processes (via CoT) for an evaluation aspect, and leverage them to evaluate unseen human-system conversations. The optimization process is performed on annotated human-human conversations to capture human thinking process and reasoning when assessing dialogue quality. We note, at the outset, that all the optimization and selection algorithms are performed independently for each aspect. For notational simplicity, we shall drop superscript $a$ from aspect-related instruction and evaluation scores in this section.

To optimize instructions, we assume access to human evaluated dialogues, $\mathbf{H}=\{(x_{i},l_{i})\}_{i=1}^{m}$ , where $x_{i}$ is either a turn or an entire dialogue depending on the aspect, and $l_{i}$ is its label. Similarly, we assume access to an LLM $\mathcal{L}_{1}$ that generates evaluation scores, $\mathbf{S_{I}}=\{(x_{i},\operatorname{FACE}(\mathbf{I},\mathbf{P}_{x_{i}}))\}_{i=1}^{m}$ , where $\mathbf{P_{x_{i}}}$ corresponds to particles of a specific turn or conversation and $\mathbf{P}=\bigcup_{x_{i}}\mathbf{P_{x_{i}}}$ is all particles in the dialogue collection.

The optimization objective is to identify a set of optimal instructions $\mathbf{I^{*}}$ that maximizes the correlation between human labels and the scores generated by the automatic evaluator:

\arg\max_{\mathbf{I^{*}}}\mathcal{C}(\mathbf{H},\mathbf{S}_{\mathbf{I^{*}}}),

where $\mathcal{C}(.)$ represents the correlation function.

The optimization process employs an LLM $\mathcal{L}_{2}$ to refine instructions based on the scores generated by the evaluator LLM $\mathcal{L}_{1}$ . FACE employs a non-parametric optimization algorithm using textual gradients (43). Here, natural language “gradients” (as opposed to numerical gradients) are generated to describe the shortcomings of instructions. The gradients are used to rewrite original instructions in the opposite semantic direction. The best instructions are iteratively selected using beam search and Upper Confidence Bound (UCB) bandits, based on correlations with human judgments. The process consists of three stages.

(1) Textual Gradient Generation. This is an iterative process, where a static prompt $\nabla$ is used for generating textual gradients (lines 6-7 of Algorithm 1). At each iteration $k$ , the prompt $\nabla$ takes an evaluation instruction $I_{k_{j}}$ from the current set of instructions $\mathbf{I}_{k}$ , its prediction score $s_{p,k_{j}}=\mathcal{E}_{\text{particle}}(I_{k_{j}},p)$ , and the corresponding human label $l_{p}$ for a given particle $p$ . A set of textual gradients $\mathbf{G}_{p,k_{j}}$ is then generated by:

(2)

\mathbf{G}_{p,k_{j}}=\mathcal{G}_{\nabla}(I_{k_{j}},s_{p,k_{j}},l_{p};\alpha),

where $\mathcal{G}_{\nabla}(.)$ is the gradient generation function, with parameter $\alpha$ denoting the number of gradients generated per instruction-score pair. Since human annotations are provided at the turn- or dialogue-level, $l_{p}$ represents human annotation for the turn or dialogue that contains the particle $p$ .

For the prompt $\nabla$ , we employ reasoning templates (68), which provide a set of items to be considered by $\mathcal{G}_{\nabla}$ . Our items include identifying inconsistencies between the predicted and human annotations, evaluating the correctness of the current task and CoT instructions, and suggesting edits to these instructions, if necessary.

(2) Instruction Rewriting. This step updates each instruction using textual gradients (lines 8-9 of Algorithm 1). For each particle $p$ at iteration $k$ , we use the rewriting function $\mathcal{R}_{\delta}$ with the prompt $\delta$ , which takes the current instruction $I_{k_{j}}$ and gradients $\mathbf{G}_{p,k_{j}}$ to obtain updated instructions $\mathbf{I}^{\prime}_{p,k_{j}}$ :

(3)

\mathbf{I}^{\prime}_{p,k_{j}}=\mathcal{R}_{\delta}(I_{k_{j}},\mathbf{G}_{p,k_{j}}).

An LLM $\mathcal{L}_{3}$ is used for the rewriting function $\mathcal{R}_{\delta}$ with the prompt $\delta$ guiding it to revise the current instruction, considering the provided feedback. We note that, while theoretically two distinct LLMs $\mathcal{L}_{2}$ and $\mathcal{L}_{3}$ are used for gradient generation and instruction rewriting, the two steps can be merged into a single LLM call by concatenating $\nabla$ and $\delta$ . This halves the number of LLM calls, resulting in a significant speed up of the optimization process.

(3) Instruction Selection. The step identifies the most promising instructions for the next iteration in two stages of selecting candidate instructions using UCB bandit, and identifying the top beam based on evaluation scores on the training data. This step corresponds to lines 13-16 of Algorithm 1.

Let $\mathbf{I}^{\prime}_{k}=\bigcup_{p\in\mathbf{P}}\bigcup_{j}\mathbf{I}^{\prime}_{p,k_{j}}$ be the set of all rewritten instructions at the iteration $k$ . The first stage identifies $b^{\prime}\geq b$ promising candidate instructions $\mathbf{I}^{\text{cand}}_{k}$ from the rewritten instructions $\mathbf{I}^{\prime}_{k}$ and stores them in an instruction pool:

(4)		$\displaystyle\mathbf{I}^{\text{cand}}_{k}$	$\displaystyle=\operatorname{Select}^{\text{UCB}}_{b^{\prime}}(\mathbf{I}^{\prime}_{k}),$
(5)		$\displaystyle\mathbf{I}^{\text{pool}}$	$\displaystyle\leftarrow\mathbf{I}^{\text{cand}}_{k}\cup\mathbf{I}^{\text{pool}}.$

The second stage then evaluates these candidate instructions by computing the correlation of the generated scores with human labels using the training set. Therefore, the instructions for the next iteration $k+1$ are generated by:

(6)

\displaystyle\mathbf{I}_{k+1}=\argmax_{\mathbf{I}_{k}\subseteq\mathbf{I}^{\text{pool}},|\mathbf{I}_{k}|=b}\mathcal{C}(\mathbf{H},\mathbf{S}_{\mathbf{I}_{k}}),

where $\mathbf{S}_{\mathbf{I}_{k}}$ denotes all predicted scores using instructions $\mathbf{I}_{k}$ . This process follows a beam search approach, where at each iteration we create a pool of candidates, assess their performance, and select the top ones to form the new beam for continued exploration. Once all iterations are complete, the final optimal instructions $\mathbf{I}^{*}$ are selected by their correlation scores on a validation set $\mathbf{H^{\prime}}$ (line 18 of Algorithm 1).

Algorithm 1 Instruction optimization of FACE

1: Human evaluations

\mathbf{H}

, initial instruction

I

, iterations

K

, beam width

b

, candidate size

b^{\prime}

, gradient samples

\alpha

2: Initialize

\mathbf{I}^{\text{pool}}\leftarrow\emptyset

\mathbf{I}_{1}\leftarrow\{I\}

3: for

k=1,...,K

4: for each instruction

I_{k_{j}}\in\mathbf{I}_{k}

5: for each particle

p\in\mathbf{P}

s_{p,k_{j}}\leftarrow\mathcal{E}_{\text{particle}}(I_{k_{j}},p)

\triangleright

Get score (Eq. 1)

7: // Textual Gradient Generation

\mathbf{G}_{p,k_{j}}\leftarrow\mathcal{G}_{\nabla}(I_{k_{j}},s_{p,k_{j}},l_{p};\alpha)

\triangleright

Eq. 2

9: // Instruction Rewriting

10:

\mathbf{I}^{\prime}_{p,k_{j}}\leftarrow\mathcal{R}_{\delta}(I_{k_{j}},\mathbf{G}_{p,k_{j}})

\triangleright

Eq. 3

11: end for

12: end for

13:

\mathbf{I}^{\prime}_{k}=\bigcup_{p\in\mathbf{P}}\bigcup_{j}\mathbf{I}^{\prime}_{p,k_{j}}

\triangleright

Collect all rewrites

14:

\mathbf{I}^{\text{cand}}_{k}=\operatorname{Select}^{\text{UCB}}_{b^{\prime}}(\mathbf{I}^{\prime}_{k})

\triangleright

Eq. 4 and Algorithm 2

15:

\mathbf{I}^{\text{pool}}\leftarrow\mathbf{I}^{\text{cand}}_{k}\cup\mathbf{I}^{\text{pool}}

\triangleright

Update pool (Eq. 5)

16: // Select top-b instructions

17:

\mathbf{I}_{k+1}\leftarrow\argmax_{\mathbf{I}_{k}\subseteq\mathbf{I}^{\text{pool}},|\mathbf{I}_{k}|=b}\mathcal{C}(\mathbf{H},\mathbf{S}_{\mathbf{I}_{k}})

\triangleright

Eq. 6

18: end for

19:

\mathbf{I}^{*}\leftarrow\arg\max_{\mathbf{I}^{*}\subseteq\mathbf{I}^{\text{pool}}}\mathcal{C}(\mathbf{H^{\prime}},\mathbf{S}_{\mathbf{I}^{*}})

20: return

\mathbf{I}^{*}

The UCB selection algorithm, denoted as $\operatorname{Select}^{\text{UCB}}_{b^{\prime}}(\cdot)$ , is presented in Algorithm 2. For each iteration $t$ , $N_{t}(I)$ denotes the number of evaluations of instruction $I$ on sampled particles and $Q_{t}(I)$ denotes its estimated correlation. Following (43), it samples a subset of particles and their corresponding human annotations, then selects the instruction that maximizes the UCB criterion $Q_{t}(I)+c\sqrt{\log t/N_{t}(I)}$ , where $c$ is an exploration constant. The selected instruction is evaluated on the sampled particles, and its estimated effectiveness is updated based on the correlation with human annotations. After $T$ iterations, the algorithm returns the candidate instructions $\mathbf{I}^{\text{cand}}_{k}$ containing the top $b^{\prime}$ instructions according to their final estimated effectiveness $Q_{T}$ with the $\operatorname{SelectTopInstructions}_{b^{\prime}}(Q_{T})$ function. This forms a set of promising candidates for the next stage of the selection process. We note that, for efficient execution of the UCB process, we approximate it by dividing $T$ into multiple small batches and processing each batch in parallel.

Algorithm 2

\operatorname{Select}^{\text{UCB}}_{b^{\prime}}(\cdot)

- Candidate Selection with UCB Bandits

1: All rewritten instructions

\mathbf{I}^{\prime}_{k}

, particles

\mathbf{P}

, human annotations

\mathbf{H}

, number of UCB iterations

T

, and the number of selected instructions

b^{\prime}

2: Initialize

N_{t}(I)\leftarrow 0,Q_{t}(I)\leftarrow 0,\forall I\in\mathbf{I}^{\prime}_{k}

3: for

t=1,...,T

4: // Sample particles and corresponding labels uniformly

\mathbf{P}_{\text{smp}}\subset\mathbf{P}

\mathbf{H}_{\text{smp}}\subset\mathbf{H}

6: // Select the instruction with the highest UCB criterion

I\leftarrow\arg\max_{I\in\mathbf{I}^{\prime}_{k}}\{Q_{t}(I)+c\sqrt{\frac{\log t}{N_{t}(I)}}\}

8: // Evaluate the instruction

I

on the sampled particles

\mathbf{S}_{\text{smp}}\leftarrow\{\mathcal{E}_{\text{particle}}(I,p)\}_{p\in\mathbf{P}_{\text{smp}}}

10: // Compute correlation and update UCB criterion

11: Observe reward

r\leftarrow\mathcal{C}(\mathbf{S}_{\text{smp}},\mathbf{H}_{\text{smp}})

12:

N_{t}(I)\leftarrow N_{t}(I)+|\mathbf{P}_{\text{smp}}|

13:

Q_{t}(I)\leftarrow Q_{t}(I)+\frac{r-Q_{t}(I)}{N_{t}(I)}

14: end for

15: return

\mathbf{I}^{\text{cand}}_{k}\leftarrow\operatorname{SelectTopInstructions}_{b^{\prime}}(Q_{T})

4. Human Annotation Collection

To assess the correlation of automatic conversation evaluation methods with human judgments, we develop a dataset and crowdsource human annotations on a set of human-system conversations. While FACE method can be applicable to any type of conversation (cf. Section 3), we build on an existing dataset of diverse human-system conversations and focus on recommendation conversations as a case study.

4.1. Dialogue Annotation

Human-System Dialogues. To evaluate automatic evaluation methods, we need to compute the correlation of their scores with human quality annotations of user-system conversations. This involves collecting user interactions with a variety of systems and annotating these conversations across multiple evaluation aspects. Similarly to 39, our goal in creating such a dataset is not to train and benchmark the best-performing systems, but rather to collect responses with varying quality of systems and obtain reliable human judgments of these responses. We select the CRSArena-Dial dataset (8) for this purpose, which contains, to the best of our knowledge, the most recent multi-turn dialogue dataset between human users and wide range of conversational recommender systems (CRSs), including LLM-based ones.

The CRSArena-Dial dataset consists of human conversations with nine state-of-the-art CRSs, including KBRD (11), BARCOR (59), UniCRS (61), ChatGPT-based CRS (60), and CRB-CRS (37), each developed based on OpenDialKG (40) and ReDial (31) datasets, except CRB-CRS, which is solely on the ReDial dataset. To ensure the quality of dialogues, we excluded seven dialogues that were unsuitable for our annotation, such as those with only a single user utterance, resulting in 467 dialogues with a total of 2,235 system responses for our annotations.

Evaluation Aspects. Over 50 aspects have been proposed for evaluating conversational systems in the literature.¹¹ 1 The full list of these aspects, compiled from 21 studies, is available in the GitHub repository of the paper. We follow 49 and collect annotations for seven evaluation aspects, whose effectiveness for CRS evaluation has been rigorously validated (49; 50). These aspects cover both system and user-centric features of a conversation. The aspects and their descriptions (used as instructions to annotators) are as follows:

Turn-level Aspects:

Table 1. Inter-annotator agreement of CRSArena-Eval and AB-ReDial (50) based on Pearson’s

r

, Spearman’s

\rho

, and Krippendorff’s

\alpha

correlations.

Aspect	CRSArena-Eval			AB-ReDial
Aspect	$r$	$\rho$	$\alpha$	$r$	$\rho$	$\alpha$
Turn-level
Relevance	0.613	0.611	0.612	0.527	0.502	0.526
Interestingness	0.386	0.386	0.375	0.209	0.217	0.209
Dialogue-level
Understanding	0.505	0.477	0.496	0.321	0.313	0.318
Task Completion	0.481	0.440	0.482	0.345	0.321	0.346
Efficiency	0.297	0.297	0.289	0.225	0.225	0.226
Interest Arousal	0.242	0.241	0.226	0.254	0.291	0.247
Overall Impression	0.573	0.526	0.572	0.321	0.300	0.321

Relevance (0–3): Does the assistant’s response make sense and meet the user’s interests?

Interestingness (0–2): Does the response make the user want to continue the conversation?

Dialogue-level Aspects:

Understanding (0–2): Does the assistant understand the user’s request and try to fulfill it?

Task Completion (0–2): Does the assistant make suggestions that the user finally accepts?

Efficiency (0–1): Does the assistant suggest items matching the user’s interests within the first three interactions?

Interest Arousal (0–2): Does the assistant try to spark the user’s interest in something new?

Overall Impression (0–4): What is the overall impression of the assistant’s performance?

These instructions and scales are based on (49), with minor adjustments from 46 for clarity. Turn-level aspects are evaluated for each system turn, while dialogue-level aspects are evaluated for the entire dialogue.

Although we select seven thoroughly assessed aspects supported by the existing literature (50), we note that FACE can be applied to a broad range of aspects and the choice of aspects is often context-dependent. Therefore, FACE algorithm allows researchers and practitioners to explore diverse aspects relevant to their own systems.

Table 2. Annotation correlations for reference-free evaluation methods. All columns show the correlations averaged over all aspects. Rel., Int., Und., Task, Eff., Int, and Overall denote relevance, interestingness, understanding, task completion, efficiency, interest arousal, and overall impression, respectively. All FACE correlations are statistically significant with

p<0.01

Methods	Turn-level				Dialogue-level										All
	Rel.		Int.		Und.		Task		Eff.		Int.		Overall		All
	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$
CRSArena-Eval (RD)
LLM^Direct	0.464	0.455	0.248	0.260	0.522	0.482	0.405	0.363	0.101	0.101	0.217	0.203	0.564	0.522	0.360	0.341
LLM^CoT+ICL	0.453	0.446	0.175	0.177	0.481	0.457	0.425	0.400	0.174	0.174	0.188	0.174	0.498	0.472	0.342	0.329
UniEval	0.311	0.288	0.182	0.242	0.246	0.225	–	–	–	–	–	–	0.395	0.387	–	–
G-Eval	0.490	0.471	0.302	0.289	0.490	0.444	0.351	0.364	0.488	0.482	0.332	0.325	0.577	0.577	0.433	0.422
FACE w/o train	0.468	0.462	0.279	0.290	0.605	0.574	0.482	0.392	0.339	0.423	0.235	0.255	0.617	0.555	0.432	0.422
FACE	0.549	0.550	0.443	0.437	0.650	0.635	0.570	0.453	0.484	0.534	0.447	0.430	0.712	0.668	0.551	0.530
CRSArena-Eval (KG)
LLM^Direct	0.452	0.452	0.238	0.231	0.599	0.546	0.538	0.481	0.137	0.137	0.425	0.378	0.655	0.557	0.435	0.397
LLM^CoT+ICL	0.419	0.408	0.203	0.190	0.562	0.520	0.475	0.434	0.114	0.114	0.309	0.279	0.599	0.521	0.383	0.352
UniEval	0.416	0.428	0.262	0.401	0.563	0.541	–	–	–	–	–	–	0.618	0.659	–	–
G-Eval	0.533	0.505	0.334	0.316	0.535	0.475	0.430	0.422	0.485	0.463	0.424	0.403	0.656	0.642	0.485	0.461
FACE w/o train	0.492	0.486	0.308	0.324	0.664	0.611	0.426	0.411	0.240	0.419	0.297	0.322	0.672	0.557	0.443	0.447
FACE	0.543	0.527	0.471	0.453	0.719	0.677	0.593	0.484	0.518	0.543	0.449	0.404	0.766	0.679	0.580	0.538

Annotation Interface. To crowdsource high-quality annotations, we built an interface, through multiple pilot experiments, to overcome the widely reported challenges in the literature (26; 6; 27; 19). This includes workers (1) skipping reading context, (2) misunderstanding aspect definitions, (3) getting distracted by overwhelming information on the annotation page, and (4) annotating randomly without focus. Our interface requires users to pass a quiz on aspect definitions and enforces the annotation of each turn before evaluating the entire dialogue. It further includes hidden tests with expert-verified answers, and workers who fail to meet the required agreement are dismissed.

Participants and Quality Control. We recruited Prolific²² 2 https://www.prolific.co/ workers from English-speaking countries with a $100\%$ approval rate and $\geq 1000$ previous submissions. Considering some workers exhibit behavior aimed at just maximizing financial gain (19), we filtered out those with a history of subpar submissions to ensure quality. Furthermore, workers with <30% agreement with experts on hidden tests were excluded. Each batch of work contained annotations for 20 dialogues, taking around 40 minutes to complete, at the cost of £6. Three annotations per annotation task were collected. In case of disagreement, additional annotations were collected until ties were resolved.

Figure 2. Distribution of human annotation scores for seven aspects across nine systems in CRSArena-Eval. The details of seven aspects and nine systems are described in Section 4.1.

4.2. Analysis

We now analyze our collected annotations, CRSArena-Eval.

Statistics. A total of 20,962 annotations were collected from 109 workers, spanning 467 dialogues and 2,235 system turns. On average, each dialogue has 14.6 annotation tasks, each annotated by three workers. Noteworthy, our annotation interface made the process highly efficient, requiring only 8 seconds per annotation. For 92% of the tasks, an agreement was achieved by the first three annotators and for the remaining 8% of tasks additional annotations were collected to resolve ties. In total, these processes produced 6,805 final labels.

Inter-annotator Agreement. Given the ordinal nature of judgments, we report inter-annotator agreement using Pearson’s $r$ and Spearman’s $\rho$ , following (39), along with Krippendorff’s $\alpha$ . Average scores across all aspects are $r=0.443$ , $\rho=0.425$ , and $\alpha=0.436$ , indicating moderate agreement. For comparison, the same aspects on the AB-ReDial dataset (49; 50) yield average agreements of $r=0.328$ , $\rho=0.325$ , and $\alpha=0.327$ , showing higher agreement of our CRSArena-Eval annotations.

Table 1 presents the breakdown of inter-annotator agreement for each aspect, showing that CRSArena-Eval consistently achieves higher agreement than AB-ReDial in all aspects except interest arousal. These results demonstrate that CRSArena-Eval provides higher quality annotations even compared to existing high-quality annotations of the AB-ReDial dataset, showing both the difficulty of the task and high quality annotations of our dataset.

System Score Distribution. Figure 2 shows the distribution of collected scores for the nine CRSs in CRSArena-Eval. It shows no system reaches the high end of the scale, indicating that existing CRSs do not fully satisfy users. This aside, the scores cover a broad range, reflecting differing system quality, which is crucial to assess the ability of automatic evaluators to distinguish system performance (39).

5. Experimental Setup

Datasets. For instruction optimization, we use the AB-ReDial (49; 50) dataset, which contains annotations of human-human conversations from the ReDial dataset (cf. Sec. 4.1). The annotations are obtained for the seven evaluation aspects and are per turn/dialogue. This ensures no human-system conversations are involved in the optimization process. We use 60% of AB-ReDial for training and the rest for validation. For evaluation, we use the CRSArena-Eval dataset (cf. Sec. 4). CRSArena-Eval (RD)/(KG) denote the subset of the dataset for systems developed using ReDial (31) and OpenDialKG (40), respectively.

Settings. Unless indicated otherwise Llama-3.1-8B-Instruct (17) is used for FACE. All experiments were performed using SGLang (74) for its inference efficiency. The temperature of 0.6 is set across all experiments unless otherwise stated. Following (43), we run the instruction optimization for $K=6$ iterations, and stored $b^{\prime}=16$ instructions in the instruction pool, resulting in $96$ instructions. We set parameters $\alpha=2$ , $c=1$ , $b=4$ , and use the batch size of $B=|\mathbf{I}^{\prime}_{k}|/2$ and $T=5B$ iterations. A sampling size of $n=5$ is used to create a score distribution (Sec. 3.2). All hyperparameters are obtained using the validation set or following (43). The final selection of the optimal instruction set $\mathbf{I^{*}}$ (line 18 of Algorithm 1) is done using the validation set and results in a set of 16 instructions.

Prompts. Inspired by TREC RAG 2024 (42), the decomposer prompt starts with: “Your task is to extract conversation nuggets […]” followed by CoT prompts and their format. The textual gradient prompt $\nabla$ begins with “Examine the original instructions, predicted nugget score, and gold score.” and then identifies inconsistency between predicted and gold scores, followed by suggestions to the instruction if necessary. The instruction rewriting prompt $\delta$ starts with “Propose new instructions of 50 words based on […]” followed by the guidance on how to rewrite based on the textual gradient, inspired by (68). The seed evaluation instruction $I$ is as follows “Given the dialogue, evaluate the quality of the target nugget based on the given aspect. Step 1: […]”. Examples of prompts are provided in our repository https://github.com/informagi/face.

Baselines. Multiple automatic evaluators are used as our baselines: LLM^Direct directly prompts the LLM to annotate the given turn/dialogue using the same instructions as human annotations for CRSArena-Eval; LLM^CoT+ICL adds CoT (28; 64) and in-context learning (ICL) (9) with two examples to the prompt of LLM^Direct; UniEval (75) and G-Eval (36) are state-of-the-art reference-free conversation evaluation methods. Since UniEval covers limited aspects, we only report those overlapping with ours. For fair comparison, we use Llama-3.1-8B-Instruct as the backbone for LLM-based methods LLM^Direct, LLM^CoT+ICL, and G-Eval in Section 6.1. To show generalizability to different LLMs in Section 6.2, other LLMs are used for our experiments.

Metrics. For correlation metrics, we use Pearson’s and Spearman’s to appropriately handle the annotation scales. Following (39), correlation significance is computed by p-value derived from t-distribution using Python’s SciPy library (57).

6. Results

We begin by outlining our key research questions and then present a series of experiments conducted to address them: RQ1: How does FACE correlate with human judgments? RQ2: How generalizable are FACE-optimized instructions across different LLMs and domains? RQ3: Can fine-grained evaluation scores of FACE provide insights about system’s issues?

Table 3. System ranking correlations on CRSArena-Eval, averaged over corresponding aspects. All FACE correlations are statistically significant with

p<0.05

Methods	Turn-level		Dial-level		All
	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$
R@1	-0.197	0.060	-0.120	0.081	-0.142	0.075
R@10	-0.192	0.048	-0.111	0.071	-0.134	0.064
iEvaLM^free	0.191	0.191	0.232	0.232	0.257	0.184
iEvaLM^attr	0.527	0.527	0.553	0.553	0.575	0.517
Distinct-3	0.716	0.841	0.665	0.780	0.680	0.798
Distinct-4	0.654	0.800	0.609	0.760	0.622	0.771
LLM^Direct	0.860	0.822	0.872	0.799	0.868	0.806
G-Eval	0.740	0.840	0.893	0.830	0.850	0.833
FACE	0.930	0.842	0.913	0.837	0.918	0.838

6.1. FACE Annotation Correlation

Annotation Correlation. Table 2 shows the annotation correlation results, demonstrating that FACE, on average, outperforms all baselines by a large margin. We note that FACE is not optimized on any human-system conversations (cf. Sec. 5), highlighting its strong generalization to unseen systems. Although one can argue that FACE can capture some information from human-human ReDial dataset during the instruction optimization process, results on CRSArena-Eval (KG) shows generalization and robustness of FACE to unseen recommendation datasets.

The results also demonstrate that even without instruction optimization (FACE w/o train), FACE remains competitive with the state-of-the-art method, G-Eval, suggesting the effectiveness of our particle-based approach. The benefit of training is more pronounced for challenging aspects such as Interestingness, Efficiency, and Interest Arousal, which indicates that FACE effectively optimizes instructions for aspects that LLMs struggle to capture using a single thought process.

Ranking Correlation. Table 3 shows the correlation of system rankings created by different automatic evaluation methods. System ranking correlations are calculated by averaging each system’s score, ranking systems, and measuring correlation with system rankings based on human judgments. We obtain correlation for reference-based metrics by computing system rankings from reported scores (60). As baselines, we report Recall, and Distinct-n (30) as reference-based metrics, iEvaLM (60) as state-of-the-art CRS specific evaluation method, and LLM^Direct and G-Eval as top-performing reference-free metrics from Table 2. iEvaLM has two variants: iEvaLM^free uses LLM-based free-form user simulation, while iEvaLM^attr employs attribute-based user simulation (60).

From the results, we notice that recall metrics are insufficient, which is in line with the literature (8). Surprisingly, iEvaLM baselines struggle to achieve high ranking correlations; this could be due to their reliance on a recall-based approach to evaluate the simulation results. While the results show that FACE outperforms all baselines, we note that system ranking correlations should be consumed with caution, as they are less informative than annotation correlation, especially for competitive systems (21; 15).

Overall, we can answer (RQ1): FACE achieves high annotation and system ranking correlations with human judgments, outperforming state-of-the-art methods by a large margin.

Table 4. Results on generalizability of FACE to other LLMs. CRS-RD and -KG represent CRSArena-Eval (RD) and (KG), respectively. All FACE annotation correlations are statistically significant with

p<0.01

Methods	LLM	Size	CRS-RD		CRS-KG		Avg.
Methods	LLM	Size	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$
LLM^Direct	Llama	8B	0.564	0.522	0.655	0.557	0.610	0.540
G-Eval	Llama	8B	0.577	0.577	0.656	0.642	0.617	0.610
FACE	Llama	8B	0.712	0.668	0.766	0.679	0.739	0.674
FACE*	Gemma	9B	0.689	0.687	0.718	0.703	0.704	0.695
FACE*	Gemma	2B	0.647	0.603	0.728	0.646	0.688	0.625
FACE*	Qwen	7B	0.698	0.664	0.764	0.693	0.731	0.679
FACE*	Qwen	3B	0.643	0.632	0.725	0.674	0.684	0.653
FACE*	Qwen	1.5B	0.557	0.606	0.605	0.635	0.581	0.621

Table 5. Results on generalizability of FACE to chit-chat conversations. All FACE correlations are statistically significant with

p<0.01

Methods	USR-Persona		USR-Topical		Avg.
Methods	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$
ROUGE-L	0.114	0.091	0.193	0.203	0.154	0.147
BLEU-4	0.147	0.151	0.131	0.235	0.139	0.193
METEOR	0.250	0.256	0.250	0.302	0.250	0.279
BERTScore	0.188	0.157	0.214	0.233	0.201	0.195
Dial-M	0.400	0.390	0.370	0.400	0.385	0.395
USR	0.607	0.528	0.416	0.377	0.512	0.453
UniEval	0.616	0.580	0.595	0.613	0.605	0.597
G-Eval^GPT-3.5	0.441	0.458	0.519	0.544	0.480	0.501
G-Eval^GPT-4	0.607	0.670	0.594	0.605	0.601	0.638
FACE	0.473	0.544	0.498	0.506	0.486	0.525
FACE ${\text{*}_{\hskip-2.45836pt\text{72B}}}$	0.681	0.697	0.570	0.582	0.625	0.639

6.2. Generalizability of FACE

We hypothesize that the pool of optimized instructions by FACE can be reused for different LLMs and domains. To assess this hypothesis, we take the instruction pool and re-select the top instructions (line 18 of Algorithm 1) for different LLMs and datasets. We denote this adapted FACE as FACE*.

Generalization to other LLMs. To assess generalizability of FACE to other LLMs, we use the AB-ReDial validation set and re-select instructions for five LLMs: Gemma 2 (9B and 2B) and Qwen 2.5 (7B, 3B, 1.5B). Table 4 shows annotation correlation results for top-performing baselines of Table 2. The results indicate that adapting to Gemma 9B and Qwen 7B achieves performance comparable to FACE. Interestingly, our method is highly effective for small models: FACE adapted to Gemma 2B outperforms G-Eval, which uses an LLM with 4x more parameters. A similar observation can be made for Qwen 3B and 1.5B.

Generalization to chit-chat conversations. To examine generalizability of FACE to another type of conversations, we evaluate FACE on the existing chit-chat datasets: (1) USR-Persona (39) (based on PersonaChat (71)), containing personalized chit-chats, and (2) USR-Topical (39) (based on Topical-Chat (24)), containing knowledge-grounded conversations. These datasets provide annotations for six evaluation aspects, of which “maintains context” is the only aspect that is similar to ours. We use the instruction pool for the relevance aspect and re-select optimal instructions using the validation set of USR-Persona for USR-Topical evaluation and vice versa, to ensure that the test set is completely unseen.

Table 5 presents the results of FACE generalizability to chit-chat conversations, with G-Eval results obtained using GPT-3.5 and 4. On average, FACE outperforms all baselines except G-Eval with GPT-4, while FACE ${\text{*}_{\hskip-2.45836pt\text{72B}}}$ outperforms all baselines. This is especially striking, considering that FACE is completely blind to category of conversations and uses an LLM with a lower number of parameters than GPT. Additionally, using an open model for evaluation has the added value of reproducibility. Based on the results of Tables 4 and 5, we answer our second research question (RQ2): FACE-optimized instructions are highly generalizable to different LLMs and domains, by performing a simple adaptation of FACE to a new LLM/domain. The adaptation to larger models can even surpass the state-of-the-art method with GPT-4 as a backbone on chit-chat conversations.

6.3. FACE Interpretability

We present a preliminary small-scale case study to demonstrate how FACE fine-grained scores can assist humans in identifying issues in CIA systems. Specifically, we compare FACE scores for two systems that are difficult to diagnose and receive contradictory evaluations from human and existing metrics: BARCOR (59) and UniCRS (61). Humans and FACE prefer BARCOR (cf. Fig. 2 and (8)), while recall-based metrics favor UniCRS (60). The left radar chart in Figure 3 shows FACE analysis for BARCOR and UniCRS for seven evaluation aspects. Although the overall impression indicates similar performance, FACE scores show that UniCRS excels in relevance and efficiency, while BARCOR is better in user understanding and keeping users interested. The right graph shows the detailed analysis of the user understanding aspect, where particle scores are aggregated for each turn. It is evident from the plot that BARCOR obtains higher scores in earlier turns. This indicates that BARCOR understands user preferences early on, which may explain its higher human preference. These insights suggest that, while UniCRS excels in recommendations, overall performance can be improved by focusing on user understanding.

To further understand each particle’s contribution, we analyze particle scores per dialogue act and find that “preference elicitation” scored higher for BARCOR than UniCRS (45.5 vs. 39.0; scaled 0-100%). This demonstrates that BARCOR’s superior user understanding stems from more effective preference elicitation, an insight overlooked by item-level recall metrics, which highlights the strength of FACE’s interpretability.

Overall, we answer (RQ3) positively: FACE can provide valuable insights into systems’ behavior, which are useful for system improvement.

Figure 3. Breakdown analysis comparing BARCOR and UniCRS. Left: FACE evaluation results for each aspect. Right: Scores of Understanding aspect per system turn.

Figure 4. Sample efficiency of different evaluation methods for the overall aspect on CRSArena-Eval.

7. Analysis

Sample Efficiency. To determine system scores and rankings, the evaluation method requires a sample of user-system conversations. To measure how many samples are needed to find a system ranking with a high correlation with human judgments, we plot system ranking correlations for various conversation counts per system in Figure 4. The results indicate that FACE has strong sample efficiency; it achieves a Spearman correlation of 0.8 with gold rankings using only 3 dialogues per system, making it twice as efficient as the best-performing existing method, G-Eval. Given that collecting human-system conversations require cost and effort, FACE’s sample efficiency significantly enhances actual usability.

Bias Analysis. We analyze whether FACE shows known LLM biases: length bias and self-bias (cf. Sec. 2). For length bias (18; 63), using CRSArena-Eval, we examine the correlation between a system’s average word count in conversations and the overall score. We find that Pearson’s correlations are 0.824 and 0.868 for FACE and humans, respectively. This indicates no sign of length bias compared to humans, which is in line with existing work (13; 18) that report humans also favour longer responses, highlighting the nuanced nature of the LLM length bias.

For self-bias, where LLMs prefer system responses over human ones, we use FACE to evaluate pairs of system- and human-generated responses and see if they show any preferences compared to gold human annotators. We examine two conversation types: USR-Persona for chit-chat, and a combination of CRSArena-Eval and AB-ReDial for a recommendation. We could not find evidence for self-bias in FACE; e.g., for USR-Persona, FACE aligns with human preferences 77.8% of the time when humans prefer human-generated responses and 71.4% of the time when they prefer system-generated responses.

8. Conclusion

We present FACE, a fine-grained, aspect-based evaluation method for conversational information access systems. It addresses the shortcomings of existing metrics, such as focusing on fixed dialogue history with reference-based metrics, limited generalizability of LLM-based metrics, and relying on non-granular scores with limited insights. FACE is shown to strongly correlate with human judgments, generalize across LLMs and domains, and provide insights for system improvement. Future work needs to address current limitations by further examining evaluation biases, assessing effectiveness across broad domains, and exploring how FACE can help expert evaluators.

References

Abbasiantaeb et al. (2025) Z. Abbasiantaeb, S. Lupart, L. Azzopardi, J. Dalton, and M. Aliannejadi Conversational gold: evaluating personalized conversational search system using gold nuggets. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §1, §2.
Alaofi et al. (2024) M. Alaofi, N. Arabzadeh, C. L. Clarke, and M. Sanderson Generative information retrieval evaluation. arXiv preprint arXiv:2404.08137. Cited by: §3.1.
Aliannejadi et al. (2024) M. Aliannejadi, Z. Abbasiantaeb, S. Chatterjee, J. Dalton, and L. Azzopardi TREC ikat 2023: the interactive knowledge assistance track overview. arXiv preprint arXiv:2401.01330. Cited by: §1.
Aliannejadi et al. (2025) M. Aliannejadi, S. Lupart, M. Gohsen, Z. Abbasiantaeb, N. Mirzakhmedova, J. Kiesel, and J. Dalton TREC ikat 2025: the interactive knowledge assistance track overview. In Proceedings of the 34th Text REtrieval Conference (TREC 2025), Cited by: §1, §1, §1, §2.
Balog et al. (2025) K. Balog, D. Metzler, and Z. Qin Rankers, judges, and assistants: towards understanding the interplay of llms in information retrieval evaluation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §1, §2.
Bernard and Balog (2023) N. Bernard and K. Balog MG-shopdial: a multi-goal conversational dataset for e-commerce. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 23–27, 2023, Taipei, Taiwan, Cited by: §4.1.
Bernard and Balog (2025) N. Bernard and K. Balog Limitations of current evaluation practices for conversational recommender systems and the potential of user simulation. In Proceedings of the 2025 International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, Cited by: §2.
Bernard et al. (2025) N. Bernard, H. Joko, F. Hasibi, and K. Balog CRS arena: crowdsourced benchmarking of conversational recommender systems. In Proceedings of the 18th ACM International Conference on Web Search and Data Mining, Cited by: §1, §2, §3, §4.1, §6.1, §6.3.
Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language models are few-shot learners. In Advances in Neural Information Processing Systems, Cited by: §5.
Chen et al. (2024a) L. Chen, J. Chen, T. Goldstein, H. Huang, and T. Zhou InstructZero: efficient instruction optimization for black-box large language models. In Proceedings of International Conference on Learning Representations, Cited by: §2.
Chen et al. (2019) Q. Chen, J. Lin, Y. Zhang, M. Ding, Y. Cen, H. Yang, and J. Tang Towards knowledge-based recommender dialog system. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Cited by: §4.1.
Chen et al. (2024b) W. Chen, S. Koenig, and B. Dilkina RePrompt: planning by automatic prompt engineering for large language models agents. arXiv preprint arXiv:2406.11132. Cited by: §2.
Chiang et al. (2025) W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. I. Jordan, J. E. Gonzalez, and I. Stoica Chatbot arena: an open platform for evaluating LLMs by human preference. In Proceedings of International Conference on Learning Representations, Cited by: §7.
Dey and Desarkar (2023) S. Dey and M. S. Desarkar Dial-m: a masking-based framework for dialogue evaluation. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Cited by: §1, §2.
Dietz et al. (2025) L. Dietz, O. Zendel, P. Bailey, C. L. A. Clarke, E. Cotterill, J. Dalton, F. Hasibi, M. Sanderson, and N. Craswell Principles and guidelines for the use of llm judges. Cited by: §1, §6.1.
Dietz (2024) L. Dietz A workbench for autograding retrieve/generate systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §1, §1, §2.
Dubey et al. (2024) A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.
Dubois et al. (2024) Y. Dubois, P. Liang, and T. B. Hashimoto Length-controlled alpacaeval: a simple way to debias automatic evaluators. In Conference on Language Modeling (COLM), Cited by: §1, §2, §7.
Eickhoff and de Vries (2011) C. Eickhoff and A. P. de Vries How crowdsourcable is your task?. In Proc. of CSDM ’11, pp. 11–14. Cited by: §4.1, §4.1.
Ekstrand-Abueg et al. (2013) M. Ekstrand-Abueg, V. Pavlu, M. Kato, T. Sakai, T. Yamamoto, and M. Iwata Exploring semi-automatic nugget extraction for japanese one click access evaluation. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §2.
Faggioli et al. (2023) G. Faggioli, L. Dietz, C. L. A. Clarke, G. Demartini, M. Hagen, C. Hauff, N. Kando, E. Kanoulas, M. Potthast, B. Stein, and H. Wachsmuth Perspectives on large language models for relevance judgment. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval, Cited by: §6.1.
Farzi and Dietz (2024) N. Farzi and L. Dietz Exam++: llm-based answerability metrics for ir evaluation. In Proceedings of LLM4Eval: The First Workshop on Large Language Models for Evaluation in Information Retrieval, Cited by: §1, §2.
Fernando et al. (2025) C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rocktäschel Promptbreeder: self-referential self-improvement via prompt evolution. In Proceedings of International Conference on Learning Representations, Cited by: §2.
Gopalakrishnan et al. (2019) K. Gopalakrishnan, B. Hedayatnia, Q. Chen, A. Gottardi, S. Kwatra, A. Venkatesh, R. Gabriel, D. Hakkani-Tur, and A. A. AI Topical-chat: towards knowledge-grounded open-domain conversations. In Proceedings of Interspeech 2019, Cited by: §6.2.
Joko et al. (2026) H. Joko, S. Amirshahi, C. L. A. Clarke, and F. Hasibi WildClaims: information access conversations in the wild (chat). In Proceedings of the 48th European Conference on Information Retrieval (ECIR 2026), Cited by: §2.
Joko et al. (2024) H. Joko, S. Chatterjee, A. Ramsay, A. P. de Vries, J. Dalton, and F. Hasibi Doing personal laps: LLM-augmented dialogue construction for personalized multi-session conversational search. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §3.1, §4.1.
Joko et al. (2021) H. Joko, F. Hasibi, K. Balog, and A. P. de Vries Conversational entity linking: problem definition and datasets. In Proceedings of the 44rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §4.1.
Kojima et al. (2024) T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Cited by: §5.
Kong et al. (2024) W. Kong, S. Hombaiah, M. Zhang, Q. Mei, and M. Bendersky PRewrite: prompt rewriting with reinforcement learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Cited by: §2.
Li et al. (2016) J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Cited by: §6.1.
Li et al. (2018) R. Li, S. Kahou, H. Schulz, V. Michalski, L. Charlin, and C. Pal Towards deep conversational recommendations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Cited by: §4.1, §5.
Lin (2004) C. Lin ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Cited by: §2.
Lin and Demner-Fushman (2005) J. Lin and D. Demner-Fushman Automatically evaluating answers to definition questions. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Cited by: §2.
Lin and Chen (2023) Y. Lin and Y. Chen LLM-Eval: unified multi-dimensional automatic evaluation for open-domain conversations with large language models. In Proceedings of the 5th Workshop on NLP for Conversational AI, Cited by: §2.
Liu et al. (2016) C. Liu, R. Lowe, I. Serban, M. Noseworthy, L. Charlin, and J. Pineau How NOT to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Cited by: §2.
Liu et al. (2023) Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu G-eval: nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Cited by: §1, §2, §2, §3.2, §5.
Manzoor and Jannach (2022) A. Manzoor and D. Jannach Towards retrieval-based conversational recommendation. Information Systems. Cited by: §4.1.
Mayfield et al. (2024) J. Mayfield, E. Yang, D. Lawrie, S. MacAvaney, P. McNamee, D. W. Oard, L. Soldaini, I. Soboroff, O. Weller, E. Kayi, K. Sanders, M. Mason, and N. Hibbler On the evaluation of machine-generated reports. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §2.
Mehri and Eskenazi (2020) S. Mehri and M. Eskenazi USR: an unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Cited by: §1, §2, §2, §4.1, §4.2, §4.2, §5, §6.2.
Moon et al. (2019) S. Moon, P. Shah, A. Kumar, and R. Subba OpenDialKG: explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Cited by: §4.1, §5.
Papineni et al. (2002) K. Papineni, S. Roukos, T. Ward, and W. Zhu Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Cited by: §2.
Pradeep et al. (2025) R. Pradeep, N. Thakur, S. Upadhyay, D. Campos, N. Craswell, I. Soboroff, H. T. Dang, and J. Lin The great nugget recall: automating fact extraction and rag evaluation with large language models. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), Cited by: §1, §2, §3.1, §5.
Pryzant et al. (2023) R. Pryzant, D. Iter, J. Li, Y. Lee, C. Zhu, and M. Zeng Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Cited by: §2, §3.3, §3.3, §5, §5.
Rajput et al. (2011) S. Rajput, V. Pavlu, P. B. Golbus, and J. A. Aslam A nugget-based test collection construction paradigm. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management, Cited by: §2.
Razavi et al. (2025) A. Razavi, M. Soltangheis, N. Arabzadeh, S. Salamat, M. Zihayat, and E. Bagheri Benchmarking prompt sensitivity in large language models. In Advances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6–10, 2025, Proceedings, Part III, Cited by: §2.
Sakai (2023) T. Sakai SWAN: a generic framework for auditing textual conversational systems. arXiv preprint arXiv:2305.08290. Cited by: §1, §1, §2, §3.1, §3, §4.1.
Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2.
Sclar et al. (2024) M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr Quantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting. In Proceedings of International Conference on Learning Representations, Cited by: §2.
Siro et al. (2022) C. Siro, M. Aliannejadi, and M. de Rijke Understanding user satisfaction with task-oriented dialogue systems. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §1, §2, §3, §4.1, §4.1, §4.2, §5.
Siro et al. (2023) C. Siro, M. Aliannejadi, and M. De Rijke Understanding and predicting user satisfaction with conversational recommender systems. ACM Trans. Inf. Syst.. Cited by: §1, §2, §4.1, §4.1, §4.2, Table 1, §5.
Soudani et al. (2026) H. Soudani, R. Petcu, E. Kanoulas, and F. Hasibi A survey on recent advances in conversational data generation. ACM Comput. Surv.. Cited by: §2.
Takehi et al. (2023) R. Takehi, A. Watanabe, and T. Sakai Open-domain dialogue quality evaluation: deriving nugget-level scores from turn-level scores. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, Cited by: §2.
Thakur et al. (2025) N. Thakur, R. Pradeep, S. Upadhyay, D. Campos, N. Craswell, I. Soboroff, H. T. Dang, and J. Lin Assessing support for the trec 2024 rag track: a large-scale comparative study of llm and human evaluations. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), Cited by: §1, §1, §2.
Trippas et al. (2025) J. R. Trippas, J. S. Culpepper, M. Aliannejadi, J. Allan, E. Amigó, J. Arguello, L. Azzopardi, P. Bailey, J. Callan, R. Capra, N. Craswell, B. Croft, J. Dalton, G. Demartini, L. Dietz, Z. Dou, C. Eickhoff, M. Ekstrand, N. Ferro, N. Fuhr, D. Glowacka, F. Hasibi, D. Hettiachchi, R. Jones, J. Kamps, N. Kando, S. Karimi, M. P. Kato, B. Koopman, Y. Liu, C. Ma, J. Mackenzie, M. Maistro, J. Mao, D. McKay, B. Mitra, S. Mizzaro, A. Moffat, J. Mothe, I. Ounis, L. Rashidi, Y. Ren, M. Sanderson, R. Santos, F. Scholer, C. Shah, L. Sitbon, I. Soboroff, D. Spina, P. Thomas, J. Urbano, A. de Vries, R. White, A. Yuan, H. Zamani, O. Zendel, M. Zhang, S. Zhuang, J. Zobel, and G. Zuccon Report from the 4th strategic workshop on information retrieval in lorne (swirl 2025). SIGIR Forum 59, pp. 1–68. External Links: Document, ISSN 0163-5840 Cited by: §1, §1.
Upadhyay et al. (2024a) S. Upadhyay, E. Kamalloo, and J. Lin LLMs can patch up missing relevance judgements in evaluation. arXiv preprint arXiv:2405.04727. Cited by: §1, §1.
Upadhyay et al. (2024b) S. Upadhyay, R. Pradeep, N. Thakur, D. Campos, N. Craswell, I. Soboroff, H. T. Dang, and J. Lin A large-scale study of relevance assessments with large language models: an initial look. arXiv preprint arXiv:2411.08275. Cited by: §1, §1, §2.
Virtanen et al. (2020) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, et al. SciPy 1.0: fundamental algorithms for scientific computing in python. Nature Methods. Cited by: §5.
Voorhees (2003) E. M. Voorhees Overview of the trec 2003 question answering track. In Proceedings of the Twelfth Text REtrieval Conference, Cited by: §2.
Wang et al. (2022a) T. Wang, S. Su, and Y. Chen Barcor: towards a unified framework for conversational recommendation systems. arXiv preprint arXiv:2203.14257. Cited by: §4.1, §6.3.
Wang et al. (2023a) X. Wang, X. Tang, X. Zhao, J. Wang, and J. Wen Rethinking the evaluation for conversational recommendation in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Cited by: §4.1, §6.1, §6.3.
Wang et al. (2022b) X. Wang, K. Zhou, J. Wen, and W. X. Zhao Towards unified conversational recommender systems via knowledge-enhanced prompt learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Cited by: §4.1, §6.3.
Wang et al. (2023b) X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. In Proceedings of International Conference on Learning Representations, Cited by: §3.2.
Wang et al. (2024) Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. R. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, and H. Hajishirzi How far can camels go? exploring the state of instruction tuning on open resources. In Proceedings of the 37th International Conference on Neural Information Processing Systems, Cited by: §2, §7.
Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Cited by: §5.
Xu et al. (2024) W. Xu, G. Zhu, X. Zhao, L. Pan, L. Li, and W. Wang Pride and prejudice: LLM amplifies self-bias in self-refinement. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Cited by: §2.
Yang et al. (2025) C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen Large language models as optimizers. In Proceedings of International Conference on Learning Representations, Cited by: §2.
Yang et al. (2024) C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen Large language models as optimizers. In Proceedings of International Conference on Learning Representations, Cited by: §2.
Ye et al. (2024) Q. Ye, M. Ahmed, R. Pryzant, and F. Khani Prompt engineering a prompt engineer. In Findings of the Association for Computational Linguistics ACL 2024, Cited by: §2, §3.3, §5.
Yu et al. (2025) F. Yu, N. Seedat, D. Herrmannova, F. Schilder, and J. R. Schwarz Beyond pointwise scores: decomposed criteria-based evaluation of LLM responses. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, Cited by: §2.
Yuksekgonul et al. (2024) M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou Textgrad: automatic “differentiation” via text. arXiv preprint arXiv:2406.07496. Cited by: §2.
Zhang et al. (2018) S. Zhang, E. Dinan, J. Urbanek, A. Szlam, D. Kiela, and J. Weston Personalizing dialogue agents: I have a dog, do you have pets too?. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Cited by: §6.2.
Zhang et al. (2020) T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi BERTScore: evaluating text generation with bert. In Proceedings of International Conference on Learning Representations, Cited by: §2.
Zheng et al. (2024a) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, Cited by: §2.
Zheng et al. (2024b) L. Zheng, L. Yin, Z. Xie, C. (. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. Gonzalez, C. Barrett, and Y. Sheng SGLang: efficient execution of structured language model programs. In The Thirty-Eighth Annual Conference on Neural Information Processing Systems, Cited by: §5.
Zhong et al. (2022) M. Zhong, Y. Liu, D. Yin, Y. Mao, Y. Jiao, P. Liu, C. Zhu, H. Ji, and J. Han Towards a unified multi-dimensional evaluator for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Cited by: §1, §2, §5.
Zhou et al. (2023) Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba Large language models are human-level prompt engineers. In Proceedings of International Conference on Learning Representations, Cited by: §2.