Integrating Task-Specific and Universal Adapters for
Pre-Trained Model-based Class-Incremental Learning

Abstract

Class-Incremental Learning (CIL) requires a learning system to continually learn new classes without forgetting. Existing pre-trained model-based CIL methods often freeze the pre-trained network and adapt to incremental tasks using additional lightweight modules such as adapters. However, incorrect module selection during inference hurts performance, and task-specific modules often overlook shared general knowledge, leading to errors on distinguishing between similar classes across tasks. To address the aforementioned challenges, we propose integrating Task-Specific and Universal Adapters (TUNA) in this paper. Specifically, we train task-specific adapters to capture the most crucial features relevant to their respective tasks and introduce an entropy-based selection mechanism to choose the most suitable adapter. Furthermore, we leverage an adapter fusion strategy to construct a universal adapter, which encodes the most discriminative features shared across tasks. We combine task-specific and universal adapter predictions to harness both specialized and general knowledge during inference. Extensive experiments on various benchmark datasets demonstrate the state-of-the-art performance of our approach. Code is available at https://github.com/LAMDA-CL/ICCV2025-TUNA

.

1 Introduction

The advent of deep learning leads to the remarkable performance of deep neural networks in practical applications [9, 8, 55, 14]. However, in real-world scenarios, data often arrive in a continuous stream, necessitating a learning system capable of progressively assimilating knowledge of emerging classes, a process known as class-incremental learning (CIL) [40]. CIL faces a formidable challenge: the process of acquiring new classes often results in the erosion of previously learned knowledge, precipitating a phenomenon known as catastrophic forgetting of established features [13]. Correspondingly, recent breakthroughs in pre-training [17] have prompted the research community to leverage pre-trained models (PTMs) as a means to mitigate the issue of forgetting [50, 49, 45]. Leveraging vast datasets and considerable computing resources, PTMs naturally produce features with strong generalization capabilities. As a result, the development of a robust CIL methodology that harnesses the power of PTMs while mitigating catastrophic forgetting has attracted considerable attention from the research community [48, 64, 65, 66].

Owing to the remarkable generalization properties of PTMs, existing approaches frequently involve freezing the pre-trained weights and adapting to incremental tasks through the integration of lightweight modules  [25, 33, 7]. Many of these methods rely on visual prompt tuning [49, 43]. During training, they learn task-specific prompt parameters and a set of keys. These keys are later used for task selection through query-key matching during inference. However, these methods suffer from two drawbacks. First, continual learning requires models to dynamically adapt to a sequence of tasks while maintaining stability on previous ones. Current prompt-based methods rely on accurate retrieval of task-specific prompts during inference. However, incorrect key matching, especially in scenarios with task ambiguity or distribution shifts, can lead to the selection of irrelevant prompts, degrading performance. Second, these methods predominantly concentrate on the acquisition of task-specific knowledge and ignore the general knowledge shared between different tasks. Thus, they tend to make mistakes when distinguishing between highly similar classes across different tasks.

To overcome the challenges outlined above, we introduce integrating Task-Specific and Universal Adapters (TUNA) in this paper, which explicitly disentangles continual learning into two complementary components: (1) specialized adapters that extract task-discriminative features, and (2) a universal adapter that consolidates cross-task shared knowledge through fusion. This decomposition not only mitigates task interference but also enables more robust generalization to semantically overlapping classes.

First, we use orthogonal loss to train task-specific adapters. To enhance the accuracy of module selection, we introduce an entropy-based adapter selection strategy that routes inputs to the most relevant task-specific adapter based on prediction uncertainty, eliminating reliance on brittle key-query matching. Second, for knowledge consolidation, we leverage an adapter fusion technique that merges task-specific adapters into a universal adapter, preserving shared features while minimizing redundancy. During inference, our framework leverages both task-specific and universal adapters in a coordinated manner. Our comprehensive experiments validate that the proposed method achieves state-of-the-art results across benchmark datasets, demonstrating notable improvements on challenging datasets such as ImageNet-A and ObjectNet.

2 Related Work

Class-Incremental Learning (CIL): necessitates a learning system capable of continuously assimilating new class information while preserving previously acquired knowledge without forgetting [16, 62, 61, 31, 34, 35]. This paradigm can be broadly categorized into several categories. Data rehearsal-based methods [3, 60, 41, 5, 6] involve carefully selecting and reintroducing exemplars from earlier classes during the acquisition of new classes. Knowledge distillation-based methods [32, 21, 27, 11, 42] try to establish a mapping between the model from previous stages and the current model through the process of knowledge distillation [20]. These mappings, represented as logits or feature representations, assit the incremental model in retaining essential characteristics from earlier phases during updating. Model rectification-based methods  [52, 56, 59, 39, 1] seek to rectify the inductive bias inherent in incremental models, ensuring unbiased predictions during the updating process. Moreover, parameter regularization-based methods [57, 2, 30, 29] impose regularization penalties on the drift of crucial parameters throughout model adaptation, thereby safeguarding earlier knowledge. Expandable networks have recently shown strong performance in incremental learning [12, 23, 46, 53]. These methods preserve the original backbone and initialize a new one for each task, combining their outputs into a large feature map and training a classifier with exemplars for calibration.

Pre-Trained Model-Based CIL: is now a hot topic in today’s CIL field. Most pre-trained model-based CIL methods utilize the parameter-efficient fine-tuning mechanism to adapt the model efficiently while keeping the pre-trained model frozen. L2P [50] leverages a pre-trained model and dynamically learns a prompt pool to guide the model in addressing specific tasks. DualPrompt [49] introduces a novel approach by learning two mutually independent prompt spaces: the general prompt and the expert prompt, which encode task-invariant and task-specific knowledge, respectively. CODA-Prompt [43] presents a decomposition-based, attention-driven continual learning prompting method, offering a significantly larger learning capacity compared to existing prompt-based techniques. Instead of directly optimizing prompt parameters, DAP [26] designs prompt generators to generate instance-specific information in prompts. SLCA [58] employs distinct learning rates for the backbone and classifier, it also models class-wise feature distributions [67] and replays them to calibrate the classifier. APER [64] proposes constructing the classifier by merging embeddings from both the pre-trained model and the adapted downstream model. EASE [65] innovatively concatenates feature representations from multiple task-specific backbones, further enhancing model capabilities. Furthermore, RanPAC [37] introduces a random projection approach that constructs robust high-dimensional randomized features, proving effective for continual learning tasks.

3 Preliminaries

In this section, we introduce the background of class-incremental learning and corresponding baselines.

3.1 Class-Incremental Learning

CIL focuses on continuously learning from evolving data streams that introduce new classes, while preserving the knowledge of previously encountered classes to construct a unified classifier [40]. Consider a series of $T$ training stages, represented as $\{\mathcal{D}^{1},\mathcal{D}^{2},\cdots,\mathcal{D}^{T}\}$,where $\mathcal{D}^{t}=\{(\mathbf{x}_{i}^{t},y_{i}^{t})\}_{i=1}^{n_{t}}$ denotes the $t$-th incremental stage containing $n_{t}$ instances. Correspondingly, the testing set is denoted as $\{\mathcal{D}_{t}^{1},\mathcal{D}_{t}^{2},\cdots,\mathcal{D}_{t}^{T}\}$. Each training instance $\mathbf{x}_{i}^{t}\in\mathcal{D}^{t}$ is associated with a class label $y_{i}^{t}\in Y_{t}$. Here, $Y_{t}$ defines the set of labels for training task $t$, and it is guaranteed that $Y_{t}\cap Y_{t^{\prime}}=\varnothing$ for any $t\neq t^{\prime}$. In this paper, we follow the exemplar-free setting in [65], which means that no historical exemplars from previous classes are used. Consequently, the model only has access to data from $\mathcal{D}^{t}$ for training during the $t$-th stage. The model’s performance is evaluated across all previously encountered classes, denoted as $\mathcal{Y}_{t}=Y_{1}\cup\cdots\cup Y_{t}$, after each incremental learning task. Specifically, our objective is to learn a model $f(\mathbf{x}):X\rightarrow\mathcal{Y}_{t}$ that minimizes empirical risk across all test classes:

$$f^{*}=\underset{f\in\mathcal{H}}{\operatorname*{argmin}}\ \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}_{t}^{1}\cup\cdots\mathcal{D}_{t}^{t}}\mathbb{I}\left(y\neq f(\mathbf{x})\right),$$ (1)

where $\mathcal{H}$ is the hypothesis space and $\mathbb{I}(\cdot)$ denotes the indicator function, $\mathcal{D}_{t}^{b}$ refers to the testing set for task $b$. An effective CIL model satisfying Eq. 1 demonstrates strong discriminative abilities across all classes. It strikes a balance between acquiring knowledge of new classes and preserving information from previously learned ones.

In line with typical PTM-based CIL works [50, 49, 65], we assume that a pre-trained Vision Transformer (ViT) [10] is available as the initialization for $f(\mathbf{x})$. To facilitate understanding, we decompose the PTM into two components: $f(\mathbf{x})=W^{\top}\phi(\mathbf{x})$, where $\phi(\cdot):\mathbb{R}^{D}\rightarrow\mathbb{R}^{d}$ is the feature extractor and $W\in\mathbb{R}^{d\times|\mathcal{Y}_{t}|}$ is the classifier. We denote the classifier for class $k$ as $\mathbf{w}_{k}$, so that $W=[\mathbf{w}_{1},\mathbf{w}_{2},\cdots,\mathbf{w}_{|\mathcal{Y}_{t}|}]$.

3.2 Baselines in PTM-Based CIL

In the era of PTMs, many methods seek to modify the PTM slightly to maintain the pre-trained knowledge [50, 49, 43, 47]. These methods usually involve freezing the pre-trained weights and training additional modules like prompt pool to incorporate task-specific information. A representative example is L2P [50], which proposes a key-query matching strategy. Specifically, every prompt $P_{i}\in\mathbb{R}^{L\times d}$, with $L$ denoting the prompt length, is associated with a learnable key vector $\mathbf{k_{i}}\in\mathbb{R}^{d}$. The prompt pool is defined as $\mathbf{P}=\{(\mathbf{k}_{1},P_{1}),(\mathbf{k}_{2},P_{2}),\cdots,(\mathbf{k}_{Q},P_{Q})\}$, where $Q$ is the pool size. The optimization objective is formulated as:

$$\min_{W,\mathbf{P}}\sum_{(\mathbf{x},y)\in D^{t}}\mathcal{L}_{ce}\left(W^{\top}\bar{\phi}\left(\mathbf{x};\mathbf{P}\right),y\right)+\mathcal{L}_{reg}(\mathbf{P}),$$ (2)

where $\bar{\phi}\left(\cdot\right)$ represents the frozen pre-trained backbone parameters, $\mathcal{L}_{ce}$ corresponds to the cross-entropy loss, and $\mathcal{L}_{reg}$ serves as the prompt selection regularization term. During inference, the most appropriate prompts are selected by identifying the top-$N$ keys:

$$\mathbf{K}=\underset{\left\{t_{i}\right\}_{i=1}^{N}\subseteq[1,Q]}{\operatorname{argmin}}\quad\sum_{i=1}^{N}d\left(\phi({\mathbf{x}}),\mathbf{k}_{t_{i}}\right)\,,$$ (3)

where $\left\{t_{i}\right\}_{i=1}^{N}$ is the selected index set, and $\mathbf{K}$ is the selected top-$N$ keys, $d(\cdot,\cdot)$ denotes the cosine distance.

This approach has two main limitations. First, it demands precise selection of the most appropriate lightweight modules during inference, as guided by Eq. 3. However, the key-query matching process is fragile, making it prone to selecting unsuitable modules, which in turn leads to performance degradation. Second, it primarily focuses on task-specific knowledge while neglecting shared general knowledge between tasks. For instance, if the model learns to classify dogs and cats in different tasks, it may confuse similar-looking classes like a fluffy dog or a cat with a long snout due to its narrow focus on task-specific features. Consequently, it tends to make mistakes when distinguishing highly similar classes across tasks.

4 Methodology

To address the aforementioned challenges, we introduce TUNA in this paper. First, we train task-specific adapters and use an entropy-based mechanism to select the best adapter for each input. Second, we fuse these adapters into a universal adapter to retain shared knowledge across tasks. During inference, we employ a dual-adapter strategy that simultaneously leverages both the selected task-specific adapter and the universal adapter to boost the accuracy.

Figure 1: Illustration of TUNA. Left: The training protocol of TUNA. We use orthogonal loss to train task-specific adapters. Middle: The fusing process. We construct an aggregated sign vector and a magnitude vector, which are combined to form the universal task vector. Right: During the inference phase, we select the most appropriate task-specific adapter based on entropy, and then combine the outputs from both the task-specific and universal adapters.

4.1 Learning Orthogonal Task-Specific Adapters

In this paper, we follow [64] to use adapter [7] to efficiently adapt the PTM to downstream tasks. An adapter is a bottleneck structure that can be incorporated into a pre-trained vision-transformer to facilitate transfer learning. Suppose we have $L$ transformer blocks in the pre-trained vision-transformer, each with a self-attention module and an MLP layer. We can insert an adapter into each block’s MLP via residual connections. An adapter comprises a down-projection layer $W_{down}\in\mathbb{R}^{d\times r}$, a non-linear activation function ReLU and an up-projection layer $W_{up}\in\mathbb{R}^{r\times d}$. The output formula of the MLP layer is formulated as follows:

$$\mathbf{x}_{o}=\text{MLP}(\mathbf{x}_{i})+\text{ReLU}(\mathbf{x}_{i}W_{down})W_{up},$$ (4)

where $\mathbf{x}_{i}$ and $\mathbf{x}_{o}$ are the input and output of the MLP, respectively. Eq. 4 illustrates how to inject task-specific information by adding residual connections of adapters to the original outputs. For a specific task $i$, we define the set of adapters across all transformer blocks as $\mathcal{A}_{i}$, which represents task-specific adapters. Furthermore, we denote the output embedding of a given $\mathcal{A}_{i}$ combined with the PTM as $\phi(\mathbf{x};\mathcal{A}_{i})$, the corresponding prediction as $f(\mathbf{x};\mathcal{A}_{i})=W^{\top}{\phi}\left(\mathbf{x};\mathcal{A}_{i}\right)$. During the learning process of task $t$, we initialize a new adapter $\mathcal{A}_{t}$, which is composed of $W_{down}^{t}$ and $W_{up}^{t}$, and then freeze the weights of the PTM, focus solely on optimizing the task-specific adapters and the corresponding classifier:

$$\mathcal{L}_{cls}=-\frac{1}{n_{t}}\sum_{(\mathbf{x},y)\in D^{t}}\log\frac{\exp(\mathbf{w}_{y}^{\top}\phi(\mathbf{x};\mathcal{A}_{t}))}{\sum_{i=1}^{|\mathcal{Y}_{t}|}\exp(\mathbf{w}_{i}^{\top}\phi(\mathbf{x};\mathcal{A}_{t}))},$$ (5)

where $n_{t}$ denotes the number of instances in $t$-th incremental stage. After the first task, we utilize an orthogonal loss function to ensure that the trainable weights remain orthogonal to those learned from previous tasks:

$$\mathcal{L}_{orth}=\sum_{i=1}^{t-1}\left\|W_{up}^{t}\cdot{W_{up}^{i}}^{\top}\right\|_{1},$$ (6)

where $\left\|\cdot\right\|_{1}$ represents the $L_{1}$ norm. The up-projection weights in the adapter module plays a key role in projecting intermediate features into a higher-dimensional space, which is essential for encoding task-specific information. By imposing orthogonality constraints on the up-projection weights, we ensure that the current adapter learns unique and non-redundant features, effectively differentiating it from previously learned adapters. The overall optimization target is formulated as:

$$\mathcal{L}=\mathcal{L}_{cls}+\lambda\mathcal{L}_{orth},$$ (7)

where $\lambda$ is a scalar to weight the loss. After training $T$ tasks by optimizing Eq. 7, we get a list of $T$ adapters: $\{\mathcal{A}_{1},\mathcal{A}_{2},\cdots,\mathcal{A}_{T}\}$. These adapters effectively capture the most salient features for their respective tasks.

Effect of task-specific adapters: Figure 1 (Left) shows the training protocol. We independently train and optimize adapter modules for each incremental task, ensuring each module extracts maximally discriminative task-specific features. This framework is general and can be seamlessly integrated with various parameter-efficient fine-tuning techniques like LoRA [22] and VPT [25]. Additionally, the lightweight architecture of adapters requires significantly fewer trainable parameters than full-model fine-tuning.

4.2 Multi-Stage Adapter Fusion

After training on $t$ tasks, we obtain a set of task-specific adapters $\{\mathcal{A}_{1},\mathcal{A}_{2},\cdots,\mathcal{A}_{t}\}$. These adapters are derived by optimizing the same PTM via Eq. 7, ensuring that each adapter is discriminative for its respective task and functions as a ‘task expert.’ For example, if the first task involves classifying ‘tigers,’ the first adapter will focus on features like furpatterns and stripes. If the next task contains ‘birds,’ the adapter will emphasize characteristics such as beaks and feathers. Thus, each adapter is typically limited to task-specific knowledge and struggles to differentiate between similar classes across tasks. In a simplified scenario where the task identity is known, we could directly use the corresponding expert adapter for prediction. However, in class-incremental learning, where task identity is not available, it is necessary to create a unified embedding space that accommodates all tasks. Drawing on insights from model merging techniques [36, 54, 24], we want to integrate these task-specific adapters into a universal adapter that can capture the high-level features shared across all tasks.

To achieve this, we begin by flattening the weights of the task-specific adapters into vectors: $\mathbf{v}^{i}=\text{Flatten}(\mathcal{A}_{i})$, resulting in a collection of task-specific vectors, denoted as $\{\mathbf{v}^{1},\mathbf{v}^{2},\cdots,\mathbf{v}^{t}\}$. Next, we construct the universal sign vector by determining the dominant sign for each parameter across all task-specific vectors. This is done by taking the sign of the sum of the corresponding parameters:

$$\mathbf{s}^{\text{uni}}=\text{sgn}\left(\sum_{i=1}^{t}\mathbf{v}^{i}\right),$$ (8)

where $\text{sgn}(\cdot)$ denotes the sign function. For each parameter, we then identify the maximum absolute value among all task vectors that maintain the consensus sign direction, forming the magnitude vector. Specifically, the $j$-th dimension of the magnitude vector $\mathbf{\epsilon}_{j}^{\text{uni}}$ is calculated as:

$$\mathbf{\epsilon}_{j}^{\text{uni}}=\begin{cases}abs\left(\max(v_{j}^{1},\cdots,v_{j}^{t})\right)&\text{if }s_{j}^{\text{uni}}>0\\ abs\left(\min(v_{j}^{1},\cdots,v_{j}^{t})\right)&\text{if }s_{j}^{\text{uni}}<0\end{cases},$$ (9)

where $s_{j}^{\text{uni}}$ denotes the $j$-th dimension of the sign vector. Then the universal task vector is generated through Hadamard (element-wise) multiplication:

$$\mathbf{v}^{\text{uni}}=\mathbf{\epsilon}^{\text{uni}}\odot\mathbf{s}^{\text{uni}}.$$ (10)

Finally, we reshape $\mathbf{v}^{\text{uni}}$ to match the original dimensions of the adapter, yielding the universal adapter $\mathcal{A}_{\text{uni}}$.

Effect of the universal adapter: Figure 1 (Middle) illustrates the fusion process, which employs two principled operations: sign summation and max-absolute-value selection. The sign summation operates as a voting system that maintains dominant feature orientations across tasks. Concurrently, the max-absolute-value selection with sign consistency suppresses noisy minor activations while preserving task-specific feature magnitudes without attenuation. This operation is theoretically grounded in max-out networks  [15] and has been shown to preserve discriminative features.

Thorough the sign and max operation, the resulting universal adapter captures high-level features common to all tasks, which may not be fully represented by individual task-specific adapter. By using the universal adapter, we can effectively leverage shared knowledge and ensure the model is better equipped to handle all encountered tasks.

4.3 Adapter Selection via Prediction Uncertainty

Suppose the model has progressively learned $t$ tasks and is now required to classify a test image, which may belong to any of the previously learned $t$ tasks. The primary challenge lies in selecting the most suitable task-specific adapter for this prediction. For a sample $\mathbf{x}$, the predictions of PTM combined with different task-specific adapters are denoted as $f(\mathbf{x};\mathcal{A}_{1}),f(\mathbf{x};\mathcal{A}_{2}),\cdots,f(\mathbf{x};\mathcal{A}_{t})$. Previous research have observed that minimizing entropy on test samples during optimization enables the pre-trained model to effectively adjust to previously unseen test data distributions. Nevertheless, it is still unclear whether entropy minimization can reliably function as a proxy objective for identifying the optimal task-specific adapter. To investigate this, we conduct a pilot study. Specifically, we choose Imagenet-A [19] and Imagenet-R [18] as the datasets and split them into 10 tasks. We assess the model’s performance when combined with each of the 10 different task-specific adapters: $f(\mathbf{x};\mathcal{A}_{1}),f(\mathbf{x};\mathcal{A}_{2}),\cdots,f(\mathbf{x};\mathcal{A}_{10})$ respectively. We compute the corresponding entropy and prediction accuracy. As illustrated in Figure 2(a) and Figure 2(b), lower entropy is associated with higher prediction accuracy. In other words, the greater the model’s confidence in its predictions, the more accurate it tends to be. Consequently, we conclude that entropy minimization effectively acts as a robust proxy objective for identifying the optimal task-specific adapter. When we have $t$ task-specific adapters, we can choose the most suitable adapter $\mathcal{A}^{*}$ according to the following formula:

$$\mathcal{A}^{*}=\operatorname*{arg\,min}_{\mathcal{A}_{i}\in\{\mathcal{A}_{1},\mathcal{A}_{2},\dots,\mathcal{A}_{t}\}}\left(-\sum_{c=1}^{\mathcal{Y}_{t}}f_{c}(\mathbf{x};\mathcal{A}_{i})\log f_{c}(\mathbf{x};\mathcal{A}_{i})\right),$$ (11)

where $f_{c}(\mathbf{x};\mathcal{A}_{i})$ denotes the predicted probability of class $c$ for input $\mathbf{x}$ using adapter $\mathcal{A}_{i}$.

Effect of entropy-based adapter selection: Entropy serves as a natural indicator of adapter-task alignment, when an adapter properly matches the input task, it generates confident, peaked predictions (low entropy), whereas mismatched adapters produce uncertain, flat distributions (high entropy). This intrinsic property makes entropy a reliable metric for selecting the most suitable adapter.

(a) ImageNet-A B0 inc20

(b) ImageNet-R B0 inc20

Figure 2: Relationship between accuracy and entropy.

4.4 Task-Specific and Universal Model Ensemble

While task-specific adapters excel at extracting discriminative features for individual tasks, their narrow focus often fails to capture transferable patterns that could aid in distinguishing visually similar classes across different tasks. Our objective is to leverage both specialized and general features effectively, enabling better discrimination between visually similar classes from distinct tasks. Building on this insight, we propose a novel inference strategy: given a test image, we not only select the most suitable task-specific adapter $\mathcal{A}^{*}$ according to Eq. 11 but also incorporate the predictions generated by the universal adapter to enhance classification robustness:

$$y^{*}=\operatorname*{arg\,max}_{y}\left(f_{y}(\mathbf{x};\mathcal{A}^{*})+f_{y}(\mathbf{x};\mathcal{A}_{\text{uni}})\right).$$ (12)

Table 1: Average and last performance comparison on four datasets with ViT-B/16-IN21K as the backbone. We report all compared methods with their source code. The best performance is highlighted in bold. None of the methods utilize exemplars in their implementation.

Method	CIFAR B0 Inc5		ImageNet-R B0 Inc20		ImageNet-A B0 Inc20		ObjectNet B0 Inc20
	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$
L2P [50]	85.94	79.93	75.46	69.77	49.39	41.71	63.78	52.19
DualPrompt [49]	87.87	81.15	73.10	67.18	53.71	41.67	59.27	49.33
CODA-Prompt [43]	89.11	81.96	77.97	72.27	53.54	42.73	66.07	53.29
SLCA [58]	92.49	88.55	81.17	77.00	68.66	58.74	72.55	61.30
SSIAT [45]	93.52	90.07	83.20	78.85	70.83	62.23	73.65	62.45
MOS [44]	93.30	89.25	82.96	77.93	67.08	56.22	74.69	63.62
SimpleCIL [64]	87.57	81.26	61.26	54.55	59.77	48.91	65.45	53.59
APER + Adapter [64]	90.65	85.15	75.82	67.95	60.47	49.37	67.18	55.24
RanPAC [37]	94.00	90.62	82.98	77.94	69.32	61.82	72.76	62.02
EASE [65]	91.51	85.80	81.74	76.17	65.34	55.04	70.84	57.86
TUNA (Ours)	94.44	90.74	84.22	79.42	73.78	64.78	76.46	66.32

(a) CIFAR B0 Inc5

(b) ImageNet-R B0 Inc20

(c) ImageNet-A B0 Inc20

(d) ObjectNet B0 Inc20

Figure 3: Performance curve of different methods under different settings. All methods are initialized with ViT-B/16-IN1K. The relative improvement over the second-best method is annotated with numerical values above the curves at the final incremental stage.

Summary of TUNA: As illustrated in Figure 1, we initialize and train an adapter for each incremental task to encode the task-specific information, and then we compute the class-wise mean and variance upon completing the training of each task-specific adapter. These statistical features are subsequently replayed during future incremental learning tasks to alleviate catastrophic forgetting in the classification head. Finally, we fuse these task-specific adapters into a universal adapter, which amalgamates cross-task knowledge while preserving domain-invariant representations. During the inference phase, we employs an entropy-guided adapter selection mechanism that combines the most confident task-specific adapter with the universal adapter to generate more accurate predictions.

5 Experiments

In this section, we conduct a thorough evaluation of our proposed method using four benchmark datasets, comparing its performance against state-of-the-art methods to demonstrate its advantages. Additionally, we provide an ablation study and further analysis to validate the robustness and effectiveness of our approach.

5.1 Implementation Details

Dataset: Given that pre-trained models encapsulate extensive knowledge from upstream tasks, we adopt the evaluation framework proposed in [64] to assess the performance on various benchmark datasets, including CIFAR100 [28], ImageNet-R [18], ImageNet-A [19], and ObjectNet [4]. These datasets represent typical CIL benchmarks and include out-of-distribution datasets that exhibit a significant domain gap relative to ImageNet. Specifically, there are 100 classes in CIFAR100, 200 classes in ImageNet-R, ImageNet-A and ObjectNet.

Dataset split: In accordance with the benchmark protocols established in [40], we employ the notation ‘B-$m$ Inc-$n$’ to represent class splits, where $m$ indicates the number of classes in the initial task, and $n$ denotes the number of classes in each subsequent incremental task. To ensure a fair and consistent comparison, we follow [40] and randomly shuffle class orders using a random seed of 1993 before splitting the data. We ensure consistency in the training and testing sets across all methods, following  [64, 45, 65]..

Comparison methods: We compare our approach with state-of-the-art PTM-based CIL methods, including prompt-based techniques (L2P [50], DualPrompt [49], and CODA-Prompt [43]), full-model fine-tuning approaches like SLCA [58], and adapter-based methods such as SSIAT [45], EASE [65], and MOS [44]. We also consider prototype-based SimpleCIL [64] and first-session adaptation approaches including RanPAC [37] and APER [64]. All comparative methods employ identical pre-trained models and experimental setups to guarantee fair comparison.

Training details: We use PyTorch [38] to implement all models on NVIDIA RTX 4090 with the same network backbone. Since the wide range of PTMs are publicly accessible [51], we choose two representative models following [64], denoted as ViT-B/16-IN1K and ViT-B/16-IN21K. They are both initially pre-trained on ImageNet21K, while the former is further finetuned on ImageNet1K. In our method, we set the batch size to 48 and train for 20 epochs using the SGD optimizer with momentum. The learning rate is initially set to 0.01 and follows a cosine annealing decay pattern. The projection dimension $r$ in the adapter is set to 16, the weight $\lambda$ in Eq. 7 is initialized at 1e-3 and follows an exponential decay schedule.

Evaluation protocol: Following the benchmark established by [40], we denote the Top-1 accuracy after the $b$-th stage as $\mathcal{A}_{b}$. Moreover, we use $\mathcal{A}_{B}$ (the performance after the last stage) and $\bar{\mathcal{A}}=\frac{1}{B}\sum_{b=1}^{B}\mathcal{A}_{b}$ (average performance along incremental stages) as measurements.

5.2 Benchmark Comparison

In this section, we conduct a comprehensive comparison of our proposed method, TUNA, against state-of-the-art approaches on four benchmark datasets and different backbone weights. Table 1 reports the comparison of different methods with ViT-B/16-IN21K. We can infer that our method achieves the best performance among all four benchmarks, substantially outperforming the current SOTA methods. We also report the incremental performance trend of different methods in Figure 3 with ViT-B/16-IN1K. As annotated at the end of each image, we find our method consistently outperforms the runner-up method, further underscoring its effectiveness.

To further validate the robustness of our approach, we extend our evaluation beyond the standard B0 benchmark (presented in Table 1 and Figure 3) to a large-base setting. In Figure 4, we compare our method with several SOTA methods with vast base classes. As shown in Figure 4, TUNA still outperforms other methods. Additionally, we also compare TUNA to traditional CIL methods such as iCaRL [40], DER [53], FOSTER [46], MEMO [63], TagFex [62] by implementing them with the same pre-trained weight in Table 2. Notably, TUNA maintains its leading performance, achieving a higher average accuracy than the closest competitor while remaining exemplar-free—a key advantage in memory-constrained scenarios.

Table 2: In contrast to conventional exemplar-based continual learning approaches, TUNA operates without storing any exemplars. All compared methods utilize the identical pre-trained backbone architecture (ViT-B/16-IN21K) for fair evaluation.

Method	Exemplars	ImageNet-R B0 Inc20		CIFAR B0 Inc10
		$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$
iCaRL [40]	20 / class	72.42	60.67	82.46	73.87
DER [53]	20 / class	80.48	74.32	86.04	77.93
FOSTER [46]	20 / class	81.34	74.48	89.87	84.91
MEMO [63]	20 / class	74.80	66.62	84.08	75.79
TagFex [62]	20 / class	83.23	78.45	92.17	89.26
TUNA	0	85.90	80.95	95.05	92.15

It is important to highlight that traditional CIL methods rely on storing exemplars to retain previously learned knowledge, whereas our approach eliminates this requirement. We follow [40] to set the exemplar number to 20 per class for these methods. TUNA still works competitively in comparison to these exemplar-based methods.

(a) ImageNet-R B100 Inc20

(b) ImageNet-A B100 Inc20

Figure 4: Experimental results on ImageNet-R and ImageNet-A with large base classes. All methods are based on the same PTM.

(a) Ablation study

(b) Inference ablation

Figure 5: Left: Ablation study of different components in TUNA. We find each component contributes to enhancing the performance. Right: Experimental results on ImageNet-A B0 inc20 with different inference strategies.

5.3 Ablation Study

In this section, we perform an ablation study to evaluate the contribution of each component in TUNA. Specifically, we present the incremental performance of various configurations on ImageNet-A B0 Inc20 in Figure 5(a). In the figure, ‘Baseline’ denotes training task-specific adapters for each task and predicting using all adapters during inference, selecting the maximum logit as the final prediction. ‘w/ entropy-based adapter selection’ means selecting the task-specific adapter based on entropy and using its output for prediction, which proves to be an effective strategy for choosing the appropriate adapter. Furthermore, ‘w/ orth loss’ introduces an orthogonality loss during training to enhance task-specific knowledge learning, and the results show that this addition improves performance. Finally, ‘w/ universal adapter’ ensembles the outputs from a universal adapter, which captures general knowledge shared across tasks, enabling the model to better handle all encountered tasks. The ablation study confirms that each component in TUNA contributes to improving CIL performance.

5.4 Further Analysis

Different inference strategies: To validate our proposed inference strategy, we conduct experiments on ImageNet-A B0 Inc20 using three inference strategies: Variation-1 (our strategy), Variation-2 (task-specific adapter selection based on entropy), and Variation-3 (sole reliance on the universal adapter). As shown in Figure 5(b), Variation-1 consistently outperforms the others across all tasks. Variation-2 fails to leverage shared knowledge between tasks, while Variation-3 lacks the granularity to capture task-specific nuances, resulting in suboptimal performance.

Parameter robustness: TUNA involves two hyperparameters, the projection dim $r$ in the adapter and the trade-off parameter $\lambda$ in Eq. 7. To evaluate their robustness, we conduct experiments on ImageNet-A B0 Inc20 by varying these parameters. Specifically, we choose $r$ among $\{8,16,32,64,128\}$, and $\lambda$ among $\{0.001,0.005,0.01,0.05,0.1\}$. We report the average performance in Figure 6(a). The results demonstrate that the performance remains stable across different parameter values.

(a) Hyperparameters robustness

(b) Variations of Eq. 6

Figure 6: Further analysis on parameter robustness and orthogonal loss implementation.

Different orthogonal loss: In Eq. 6, we force the current adapter’s up projection weight to be orthogonal to previous adapters’ up projection weights, we call it Variation-1. Additionally, we can extend this orthogonality constraint to the current adapter’s down projection weight relative to previous adapters’ down projection weights, termed Variation-2, or apply it to both the up projection and down projection weights simultaneously, denoted as Variation-3. We conduct experiments on ObjectNet B0 Inc20 setting to compare different losses. As we can see from Figure 6(b), with other settings the same, we find Variation-1 performs the best among these variations. This is likely because the up projection weight plays a more critical role in capturing task-specific features, and enforcing orthogonality on it alone is sufficient to reduce task interference. In contrast, the down projection weight primarily projects input features into a lower-dimensional space, and overly restricting it may hinder the model’s ability to encode task-specific information. Additionally, applying constraints to both weights simultaneously may introduce excessive rigidity, reducing flexibility and risking underfitting. Thus, focusing orthogonality constraints solely on the up projection weight offers a more balanced and efficient approach for continual learning.

Visualizations: To explore why combining task-specific and universal adapters boosts performance, we visualize predictions from each adapter separately using ImageNet-R images and the model trained under the B0 Inc20 setting. Figure 7 shows the original images in the first column, top-5 predictions from task-specific adapters in the second column, and top-5 predictions from the universal adapter in the third column. Task-specific adapters, focusing on limited information, often misclassify similar classes, such as predicting a golden retriever as a lion or a peacock as an ostrich. In contrast, the universal adapter, which integrates cross-task knowledge, captures shared features and refines predictions, increasing the chances of correct classification. This synergy enhances overall performance.

Figure 7: Visualizations of the predictions on ImageNet-R. The original images are depicted in the first column, followed by the top-5 prediction probability produced by task-specific adapter, and the probabilities generated by the universal adapter in the last column. The ground-truth class is highlighted with red boxes.

6 Conclusion

Incremental learning is crucial for practical systems. This paper introduces a novel method that integrates Task-Specific and Universal Adapters(TUNA) for pre-trained model-based CIL. Specifically, we train task-specific adapters to capture distinct features for their tasks. We also introduce an adapter fusion mechanism to create a universal adapter that encapsulates shared knowledge across tasks. During inference, we employ an entropy-based selection to choose the most suitable task-specific adapter and then ensemble its predictions with those from the universal adapter. Extensive experiments verify TUNA’s effectiveness.

Limitations and future works: The process of selecting the optimal task-specific adapter requires multiple forward passes through the model, resulting in increased computational time. Future works include designing methods to speed up the algorithm.