VISTA: A Visual Analytics Framework to Enhance Foundation Model-Generated Data Labels

A. Foundation Model-Involved Data Pipeline.

FMs’ capability to understand the diverse multi-modal world facilitates their applications in auto-labeling, which automatically generates textual descriptions of images/videos [4], [5]. This has spurred many efforts to leverage FMs in data pipelines, such as general-purpose image-language understanding [5], referring between noun phrases and bounding boxes [4], open-vocabulary image segmentation [3]. While extending data volumes has contributed to performance improvement, there are increased quality concerns. For instance, Dai et al. [6] prove that improving the quality of only a small data amount significantly boosts image generation performance. Touvron et al. [7] demonstrates that high-quality data acquired by supervised fine-tuning (SFT) can notably improve model performance. However, the efficiency of data quality enhancement remains a persistent challenge—Data validated by humans has verified quality with considerable costs, thus existing work often opts for validating a sampled subset [1], [7]. On the other hand, automatic methods utilize quality metrics to filter out unsatisfactory data [8], [9], which may miss the big picture and overlook many crucial issues [10]. VISTA aims to fill the existing gap with effective human-AI collaboration—bypassing the disadvantages including high manual labeling costs and limited capability of static filtering.

B. Human-in-the-Loop Data Labeling and Validation

Visual analytics (VA) plays a crucial role in labeling and validating expansive datasets [11]–[14], providing data exploration capabilities to expedite the costly annotation process [15], [16]. VA tools for label creation develop visualization techniques [17]–[20] or automated algorithms [21], [22] to facilitate label collection, which still demands substantial human involvement or fail to handle more complex multi-modal data. Label or model validation systems assume labels have already been produced and focus on validating their quality [23]–[34]. Techniques such as dimensionality reduction and uncertainty estimation have been employed to verify crowd-sourced image datasets and detect problematic samples [35]–[38], which can be resource-intensive. Hoque et al. [39] introduced Visual Concept Programming that uses semantic segmentation models and allows experts to infuse domain knowledge, scaling up data labeling through automated functions. Therefore, facing multi-modal data that are often more complex due to varied modalities, heterogeneous in structure, and massive in volume, we introduce VISTA to efficiently address data quality concerns with humans in the loop.

C. Open-Vocabulary Image Segmentation

OVIS marks a significant evolution in autolabeling, enabling models to recognize fine-grained semantics in the dynamic real world. Two primary approaches define the current landscape of this domain. Training-based approaches [40]–[45] use a web-scale dataset to train or fine-tune a sophisticated model, which can produce good results but are computationally intensive. The performance of these approaches is affected by multiple factors including model architectures, training data, loss functions, and optimization strategies [42]–[45]. On the other hand, training-free OVIS either explores the attention mechanisms of FMs [46]–[48] or constructs a reference set to perform semantic segmentation via information retrieval [49]–[53], while retrieval-based methods provide a valuable evaluation for our framework, whose performance is mainly determined by the quality of reference sets.

A. OVIS and Data Pipeline with FMs

Fig. 1 shows a typical data generation pipeline of OVIS. A multi-modal large language model (MLLM) processes the input image to produce enriched text descriptions. A grounding model then links noun phrases from descriptions to image regions using bounding boxes, according to which a segmentation model crafts segment masks.

B. Challenges for Quality Enhancement

Building upon the identified data quality considerations, we delineate the core challenges that must be addressed.

C. Requirements for Quality Enhancement

To overcome the identified challenges, combined with the expertise of our collaborated ML experts, we develop the essential requirements that our framework must meet:

D. VISTA Framework

To address the challenges and requirements, we introduce VISTA, a framework to enhance FM-generated data quality. An overview of VISTA is shown in Fig. 2. Our issue detection and summarization uses statistical or computational strategies (R1) to deconstruct this complex problem into coherent subgroups with clear summarization (R2, R3). Grounded on the summarized issues, the visual analytics system enables human validators to quickly locate, understand, and solve different issues to enhance data quality (R3, R4).

A. Data Structuring

B. Quality Quantification

Let $I$ denote an image, $S$ denote an image segment, $B$ denote the bounding box of the segment, and $L$ denote a textual label describing the segment, with root $r$. A multimodal model, such as clip_surgery [60], including an image encoder and a text encoder, encodes them into embeddings with the same dimension, $\mathit{i},\mathit{s},\mathit{b},\mathit{l}\in {\mathbb{R}}^n$, respectively. Besides, we use $\frac{⟨x,y⟩}{\Vert x\Vert \cdot \Vert y\Vert }$ to represent cosine similarity between $x$ and $y$. Now, we delve into our designed data quality metrics.

C. Data Issue Summarization

In this phase, our goal is to further sort through data, providing concise explanations for coherent groups and identifying consistent issue patterns from our initial findings (R3), making the information ready to be presented at our visual analytics interface (R4). Aligned with previous works [2], [8], we leverage segment-label similarity () as the foundation to identify problematic data, as it represents the kernel of this pairing problem. Additionally, with the consideration discussed in R1, other data issue perspectives are not overlooked.

D. Online Evaluation

To quantitatively evaluate the capability of VISTA in enhancing the quality of datasets with segment-label pairs, we adopt a training-free OVIS method [65] that performs semantic segmentation by retrieving a reference set with segment-label pairs, enabling real-time evaluation of data quality. Specifically, we compare the OVIS performance when using the original and VISTA-produced data, respectively.

Let ($S_{\mathit{train}},L_{\mathit{train}}$) denote segment-label pairs generated by FMs or enhanced by VISTA, with $N$ unique segments. ($S_{\mathit{text}},L_{\mathit{text}}$) be the ground-truth segment-label pairs in the test set, with $M$ labels. Firstly, we find similar segments and similar labels between $test$ and $train$ using similarity quantifications ${sim}_S$ and ${sim}_L$. We obtain the similarity between $S_{\mathit{text}}$ and $S_{\mathit{train}}$ by:

where ${\mathit{S}}_{\mathit{text}},{\mathit{S}}_{\mathit{train}}$ are embeddings of $S_{\mathit{text}},S_{\mathit{train}};{sim}_S\in {\mathbb{R}}^{1\times N}$. For all unique labels $l_{\mathit{train}}\in L_{\mathit{train}}$, we obtain their similarity with $L_{\text{test}}$ labels by:

where ${\mathit{o}}_{\mathit{train}}^i$ is the binary vector of labels for the $i$-th segment $s_{\mathit{train}}^i,{\mathit{L}}_{\mathit{train}},{\mathit{L}}_{\mathit{test}}$ are embeddings of unique $L_{\mathit{train}},L_{\mathit{test}}$, respectively. We have ${sim}_L\in {\mathbb{R}}^{N\times M}$. Note that ${\beta }_1$ and ${\beta }_2$ are temperatures for softmax, with a lower value considering more neighbors and a higher value leading to the focus on the top similar ones. An ablation study of parameters ${\beta }_1$ and ${\beta }_2$ is provided in the supplementary. Lastly, the predicted labels for $s_{\mathit{test}}$ is computed by:

This way, we obtain OVIS results by this referring process. Consequently, we can measure its performance with mloU and pixel accuracy, the common OVIS validation metrics, by:

A. Interface Overview

VISTA’s interface, as shown in Fig. 4, is designed data quality enhancement and consists of five coordinated views:

B. Analytical Workflow

The design of VISTA’s interface consolidates a user-centric analytical workflow for efficacy-driven data issue validation and mitigation (R4), where each view works coordinately to integrate human expertise in resolving critical label (I1) and alignment issues (I2). As illustrated in Fig. 5, for addressing issues fall into each category (I1 or I2), our workflow is structured into three stages: Explore & Detect, Interpret & Verify, and Annotate & Solve, each meticulously crafted to guide users through the data quality enhancement process. In the Explore & Detect stage, validators investigate the label grid or UpSet visualization to identify issues requiring validation. Progressing to the Interpret & Verify stage, following users’ selection, our interface displays corresponding distribution overviews or in-depth details, providing the context needed to make informed decisions. Upon verification of an issue, the Annotate & Solve stage provides intuitive interfaces for annotating them and proposing resolutions, ensuring a thorough and effective data refinement. Following these stages, validators can tackle different types of issues systematically, thereby upholding the integrity of the data throughout the validation and mitigation process. Next, we delve into the specifics of how VISTA assists validators in resolving label issues and alignment issues, respectively.

C. Data Issue Resolution

When human validators confirm and apply their annotations at the end of the analytical workflow, VISTA meticulously documents them to apply human-proposed solutions for data issue correction. In our pursuit to improve the quality of data generated by FMs, we consciously refrain from utilizing these models at the error-addressing stage. Instead, we have collaborated with ML experts to devise three principal remedial strategies: (1) removing irrelevant labels, (2) removing problematic segments, and (3) seeking alternative matching segments for misaligned labels. These are provided in our interface as candidate solutions. If humans select from these categories, they are automatically executed to rectify the specified issues. In situations where the solutions fall outside, our specialists step in to review and address them accordingly.

As a prototype framework for addressing this challenging problem, our emphasis is on exploring the efficacy of human intervention and validating its significance in data quality enhancement. This approach allows us to establish a closed-loop system with manageable solutions, underscoring the merit of human contribution. While our current focus is on lightweight interventions, we remain receptive to the adoption of more advanced techniques in future iterations of our framework.

Data Pipeline.

In our study, a data generation pipeline was employed to acquire image segment-label data, as depicted in Fig. 6. The process begins with the Kosmos-2 performing phrase grounding on paired images and captions, which results in preliminary labels grounded with bounding boxes, prompting SAM to generate corresponding segment masks. This pipeline represents a typical approach for generating segment-label pairs with FMs, serving as the basis for OVIS. With the generated data, we deploy our VISTA for two cases.

Investigating the validity of labels.

With the current data pipeline, labels are directly generated by the phase grounding model. Alice does not completely trust the model, doubting if the labels themselves are troublesome. Following the analytical workflow, Alice interacts with the grid visualization of labels in view of VISTA, as shown in Fig. 4. She brushed the top histogram to check labels with low segment-label similarity, where she found many cells corresponding to high label frequency, indicating both low-performed and dominant labels. By selecting them and investigating views and , Alice found many labels are not even nouns and should not be assigned to any segment, such as “front”, which is assigned to 14,350 image segments. Somehow many such labels were extracted and paired with segments by the FMs. In addition, Alice noticed a meaningful noun, “camera”, being spotlighted in a neighbor cell as well. By checking the corresponding segments, she found most of them have no camera included, with content ranging from animals to humans, but they are all “looking at the camera” (as shown in Fig. 4). After exploring neighboring cells, Alice verified 18 labels with similar issues, annotated them as “irrelevant labels”, and decided to “remove label” (refer to Fig. 5 for an additional example). By annotating them with a few clicks, Alice moved to other views for further data validation.

Analyzing segments and label pairing errors.

Given that the data originated from a singular pipeline rather than multiple resources, Alice speculated that there might be other recurring errors. Observing the view B Issue Pattern View (refer to Fig. 4), she first noticed the three groups of bars, where “large segment size” is the highest. She also found the “error rate” indicator and thought this was important for error detection. Alice sorted the data by “error rate” while grouping by “large segment size”, and found three groups at the top (refer to Fig. 4), characterized by “large segment size” with “high error rate”. By investigating further details with views , she found “Group-8” with “large segment size”, “small image-label similarity”, and “small bounding box-label similarity” includes segments highlighting background information, where their labels are pretty noisy (refer to Fig. 4). In addition, upon examining the segment distributions in view , she observed that the labels of the segments’ neighbors, which were not highlighted, were accurate. Consequently, she marked this group’s issue as “extremity-sized segmentation” and the solution as “find matching labels in neighbors”. Additionally, for “Group-12” with “large segment size”, “middle image-label similarity”, and “low bounding box-label similarity”, she found the segments frequently depicted background elements, with labels correctly describing the image but not the segments. For instance, an image of a bus station had its ground segment erroneously labeled as “a station”, indicating an issue of “misplaced label”. Alice designated the resolution to “re-assign label to other segments in image” for such cases. Continuing this methodical approach, Alice addressed other top-ranked groups, annotating and resolving verified issues.

Performance enhancement.

In total, Alice annotated 27 problematic labels and 20 misaligned groups. By checking the report generated by VISTA, Alice was impressed that errors corresponding to 1.1M segment-label pairs have been fixed, which indicates a significant speed-up over manual data curation. After issue correction, Alice conducted the online evaluation(refer to Sec. IV-D) to compare OVIS performance with mIoU and pixel accuracy. As shown in Tab. I, the model demonstrated improved performance with VISTA-enhanced data, confirming the premise that high-quality data is essential for superior model performance.

Pedestrian identification accuracy.

Correctly identifying pedestrians is paramount for autonomous driving. As illustrated in Fig. 7, Kevin investigated the Label View to check whether there are problematic labels related to pedestrians. Sifting through the “segment-label similarity” histogram, he discovered that “people” and “pedestrian” lied in two neighboring cells, showing low similarity scores. Diving into the corresponding distributions in the Embedding View, he found frequent errors for assigning the label “people” to “traffic light, sign” segments, or assigning the label “pedestrian” to “building, wall” segments. Curious about the underlying causes, Kevin then checked related samples in the Sample View in detail, which led to significant insights—(1) Misalignments involving the label “people” often occurred with small, indistinct traffic signs or traffic sign-shaped segments; and (2) For the misalignment between the label “pedestrian” and the segments “building, wall”, there were recurring situations where part of a building or wall is close to pedestrians. A common observation for both is the corresponding segments are vague, due to distance or lighting factors. After verification through the interface, Kevin categorized them as “vague segments” and “misplaced labels”, respectively. Moreover, a review of neighboring cells at the Label View exposed that the label “person” was also associated with low “segment-label similarity” scores. This led Kevin to conclude that human-related segments were frequently susceptible to mislabeling, warranting a more thorough investigation.

Other common traffic labeling failures.

Kevin proceeded to the Issue Pattern View to check the existence of other segment-label misalignment. He noticed that when ranking the groups decreasingly according to the error rate, many groups corresponding to “small segment size” stayed on the top. Initially, he thought this might correspond to the pedestrian identification issue due to the typically small segment size. However, by investigating the Embedding View, Kevin found a large portion of the segment cluster “road, gravel” was highlighted, which was an unexpected pattern, contradicting with the common sense that “road, gravel” should have large sizes. Upon examining the cases, Kevin found that the model only captured edges of roads or traffic lines rather than the complete road, yet labeled such edges as “road”. He decided to visit the Embedding View to check the entire cluster with semantics “road”, where he confirmed that for those not highlighted by our framework, the segments were correctly associated with the entire road. He annotated them with “correct labeling with road” and “wrong labeling with road”, respectively. This finding indicated VISTA’s accuracy in detecting hidden issues and distinguishing between correct and problematic cases.

Iterative data enhancement and performance gains.

Because of the safety concerns, Kevin decided to conduct iterative data enhancement. After mitigating the detected errors, he loaded the updated data back to VISTA for the next-round data validation, repeating this process until reaching a satisfactory stage. Lastly, he validated the resulting data with VISTA and verified that they had the desired quality, supported by two observations: (1) the overall distribution of “segment-label similarity” in the Label View was much higher; (2) he had to increase the threshold to 0.8 to flag data with “potential issues”, indicating a good alignment between segments and labels in general. By checking the automatically generated report, he found that VISTA helped him correct auto-labeling issues of 15k samples. He also conducted the quantitative evaluation to compare training-free OVIS performance with the original and enhanced data as shown in Tab. I, which confirmed the efficacy of VISTA in boosting model performance. He collected the final annotation record and decided to validate other auto-labeling technologies using VISTA.

Expert demographics.

Before the study, We collected detailed demographics of each expert, to contextualize their feedback and ensure that they belong to our target user—ML experts who do not necessarily have visualization/VA knowledge. Among them, 4 are PhD students and 6 have obtained PhD degrees. All experts major in Computer Science for their PhD studies. We asked them to self-report their expertise in two areas on a scale of [0, 10] (0 for “no knowledge” and 10 for “expert”) and report the statistics as follows:

• Expertise in ML/CV: $\mathrm{M}\mathrm{d}=9.0,\mathrm{I}\mathrm{Q}\mathrm{R}=1.7$;

• Expertise in Visualization/VA: $\mathrm{M}\mathrm{d}=0.0,\mathrm{I}\mathrm{Q}\mathrm{R}=1.0$.

For topics they are mostly familiar with, all experts have either used or conducted research on FMs, with other research focuses spanned among: object detection/segmentation ($E_0-\left.E_4,E_8\right)$, generative AI $\left(E_2,E_5-E_7,E_9\right)$, medical/biomedical topics ($E_1,E_3$), autonomous driving ($E_0,E_4,E_6$), and 3D reconstruction ($E_1,E_2,E_5-E_9$).

Study setup.

Each expert was invited to join a one-on-one session, lasting around 60 minutes. In the session, we started with a 5-minute introduction of the background, tasks, and interface. Then experts were asked to explore VISTA with the public domain case, COCO Stuff, following a think-aloud protocol and exploring as many functionalities as possible. We recorded the study process and noted their comments, paying particular attention to how well the design requirements were met. At the end, we gathered each expert’s feedback through a questionnaire with Likert-type questions and an interview.

Fig.8 provides the results of our post-questionnaire, which indicates all experts positively rated the usability and usefulness of VISTA. From the interview, we obtained more detailed feedback, and the summarization is provided below.

Detection of hidden errors (R1, R2).

All experts successfully identified a range of data errors with VISTA. $E_0$ and $E_1$ commented positively on VISTA for its analytical depth in dissecting complex multi-modal data. $E_1$ appreciated the framework’s contribution to improving model robustness, saying “The key concern of adapting FMs in the medical domain is its uncontrolled nature and you helped mitigate this gap”. $E_0$ shared the same view—“The thorough validation from different angles increases my confidence in the data,” highlighting the importance of multi-perspective data validation as “crucial but often neglected.”

Interpretation and verification of data issues (R3).

$E_0$ commented VISTA for providing sufficient information to make data issues interpretable. In addition, all experts agreed that our coordinated views facilitated their understanding, with $E_1$ and $E_7$ particularly enjoying the interactive feature. $E_7$ mentioned “The interface simultaneously showed corresponding information upon my interactions. This greatly helped my task.” $E_9$ talked about the potential of VISTA for synthetic data, indicating that our Embedding and Sample Views can help them validate whether their generated data was realistic.

Usability of VISTA (R4).

Following our brief introduction, all experts navigated the system smoothly. Regarding our designed analytical workflow, $E_1$, $E_5$, and $E_8$ described it as “a clear guidance”. $E_0$ mentioned the sorting function, “it allows me to group various features for a closer examination.” $E_2$ and $E_4$ commended our “Pattern Dice” design as “compact and helpful”, and $E_4$ said “it makes identifying patterns super easy.” We also noticed that although both $E_5$ and $E_{10}$ haven’t used any VA system, they promptly finished the data enhancement tasks, proving the usability of VISTA.

Other comments.

The experts also offered constructive feedback. $E_0$ expressed interest in metric customization. While praising the system’s capability in summarizing prevalent issues, she proposed that “in later exploring stages, allowing users to define new metrics might uncover trivial problems that are less apparent.” $E_9$ discussed the possibilities of applying VISTA in natural language validation. “I would be interested to see how to validate LLM-generated data, ” she mentioned. Despite some suggestions might be out-of-scope of this work, we plan to explore these invaluable comments in the future.