Continual learning is referred to as machine learning model’s ability to learn a sequence of tasks or data over time without forgetting previously learned knowledge. In particular, we focus on domain incremental learning where the model trained on one domain, e.g., real-life photos, has to be augmented to accommodate data from new domains, e.g., cartoon images. Typical incremental learning relies on rehearsal-based methods, which store trained samples in a buffer and replay them during training alongside new data. This results in significant memory overhead and raise concerns about data privacy. Recently, prompt-based methods address these challenges and outperform them by utilizing the pre-trained Vision Transformer (ViT) and replacing the replay buffer with a prompt pool. However, existing prompt-based models fail to capture domain-specific knowledge and perform poorly in domain incremental learning. In this paper, we propose “domain prompt incremental learning via dynamic neural network”, which combines the advantages of architecture-based and prompt-based methods. Specifically, our framework maintains two types of prompt: a instance-level prompt that improves the model’s generalization ability is shared across all input samples; and a domain prompt that encodes domain-specific knowledge is assigned for each task. Furthermore, a separated classification head is trained for each domain so that the model has a pre-trained ViT and an ensemble of classification layers, one for each domain. The experimental results shows that our approach outperforms state-of-the-art methods by 2.3% in average on two domain incremental learning (DIL) benchmarks.