[JENKINS-76294] java.util.ConcurrentModificationException during massive node disconnection #1513
Description
Jenkins and plugins versions report
Environment
Jenkins: 2.528.1
OS: SunOS - 5.11
Java: 21.0.5 - Oracle Corporation (OpenJDK 64-Bit Server VM)
---
PrioritySorter:905.v35c21f62f980
analysis-model-api:13.15.0-928.v63f6dc44cfd2
ansicolor:1.0.6
ant:520.vd082ecfb_16a_9
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.5-170.v023de017ccd7
asm-api:9.9-185.va_6c6b_3348b_c3
audit-trail:436.vc0d1e79fc5a_3
authentication-tokens:1.144.v5ff4a_5ec5c33
authorize-project:2.0.0
badge:2.12
blueocean:1.27.25
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.25
blueocean-commons:1.27.25
blueocean-config:1.27.25
blueocean-core-js:1.27.25
blueocean-dashboard:1.27.25
blueocean-display-url:2.4.4
blueocean-events:1.27.25
blueocean-git-pipeline:1.27.25
blueocean-github-pipeline:1.27.25
blueocean-i18n:1.27.25
blueocean-jwt:1.27.25
blueocean-personalization:1.27.25
blueocean-pipeline-api-impl:1.27.25
blueocean-pipeline-editor:1.27.25
blueocean-pipeline-scm-api:1.27.25
blueocean-rest:1.27.25
blueocean-rest-impl:1.27.25
blueocean-web:1.27.25
bootstrap5-api:5.3.8-895.v4d0d8e47fea_d
bouncycastle-api:2.30.1.82-277.v70ca_0b_877184
branch-api:2.1268.v044a_87612da_8
build-failure-analyzer:2.6.1
build-timeout:1.38
build-with-parameters:81.ve4a_9c2499d9a
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:373.vfe7645102093
cloud-stats:377.vd8a_6c953e98e
cloudbees-bitbucket-branch-source:937.2.1
cloudbees-folder:6.1073.va_7888eb_dd514
cobertura:1.17
code-coverage-api:4.99.0
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.28.0-2
commons-httpclient3-api:3.1-3
commons-lang3-api:3.19.0-104.v12125f33a_255
commons-text-api:1.14.0-194.v804a_dc3a_1b_d8
config-file-provider:1006.vc7366c201f57
conflict-aware-ondemand-strategy:2.190.vd08f48950f3e
coverage:2.3054.ve1ff7b_a_a_123b_
credentials:1453.v9b_a_29777a_b_fd
credentials-binding:702.vfe613e537e88
customizable-header:247.vb_371b_ea_cc8d7
customized-build-message:1.1
dark-theme:574.va_19f05d54df5
dashboard-view:2.543.vca_9da_3cb_9c60
data-tables-api:2.3.5-1482.vb_47a_3c19f78c
digitalocean-plugin:230.v2246586df170
display-url-api:2.217.va_6b_de84cc74b_
durable-task:635.v3733cef34b_5e
echarts-api:6.0.0-1165.vd1283a_3e37d4
eddsa-api:0.3.0.1-19.vc432d923e5ee
email-ext:1933.v45cec755423f
embeddable-build-status:627.v0249270b_488e
emoji-symbols-api:17.0-57.v8d44b_9a_b_d5ea_
environment-dashboard:1.1.10
external-monitor-job:223.vb_fddcf42c9b_3
fast-track:1.0.0
favorite:2.253.v9b_413168133b_
flatpickr-api:4.6.13-18.vcf5f6a_5b_8468
folder-auth:217.vd5b_18537403e
font-awesome-api:7.1.0-882.v1dfb_771e3278
forensics-api:3.1772.v99ca_3d83b_9fa_
generic-webhook-trigger:2.4.1
ghprb:1.42.2
git:5.8.1
git-client:6.4.1-rc3954.37196d9c1974
git-forensics:3.2138.vf25ea_d549e33
git-server:137.ve0060b_432302
github:1.45.0
github-api:1.330-492.v3941a_032db_2a_
github-autostatus:3.6.2
github-branch-source:1934.v33068c462b_7a_
github-checks:634.v371dc6d978a_3
github-label-filter:1.0.0
github-oauth:651.v135e939e8b_60
github-pr-comment-build:134.ve7ff0b_719821
github-pr-coverage-status:2.2.0
github-pullrequest:0.7.2
github-scm-trait-commit-skip:0.4.0
github-scm-trait-notification-context:45.v8ef831829589
google-login:109.v022b_cf87b_e5b_
google-oauth-plugin:1.335.ve6de40e2db_18
gradle:2.16.1149.v711b_998b_0532
gravatar:210.201.vf3640e8d8447
groovy-postbuild:272.v52a_03efb_8653
gson-api:2.13.2-173.va_a_092315913c
handy-uri-templates-2-api:2.1.8-38.vcea_5d521d5f3
htmlpublisher:427
http_request:1.23
instance-identity:203.v15e81a_1b_7a_38
instant-messaging:2.875.v954b_d3f33534
ionicons-api:94.vcc3065403257
ircbot:3.1020.v1b_420db_08ce4
jackson2-api:2.20.1-423.v13951f6b_6532
jacoco:3.3.7
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.6-10.v9b_7e1d1fc40b_
javadoc:354.vee1a_660b_4990
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.25
jjwt-api:0.11.5-120.v0268cf544b_89
jnr-posix-api:3.1.21-186.vb_7ec9b_23ce83
job-dsl:1.93
jobConfigHistory:1356.ve360da_6c523a_
joda-time-api:2.14.0-149.v1c3ce991d1b_9
jqs-monitoring:37.vf50a_82a_0b_f32
jquery:1.12.4-3
jquery3-api:3.7.1-619.vdb_10e002501a_
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250517-173.v596efb_962a_31
json-path-api:2.10.0-202.va_9cc16c1e476
jsoup:1.21.2-66.v6ea_38164b_8a_2
junit:1369.v15da_00283f06
ldap:793.v754d6b_41b_ea_4
libvirt-slave:315.v5d33b_333a_551
lockable-resources:1463.vd8708a_0a_1760
login-theme:284.v95044396a_484
mailer:525.v2458b_d8a_1a_71
matrix-auth:3.2.9
matrix-combinations-parameter:1.3.3
matrix-project:870.v9db_fcfc2f45b_
maven-plugin:3.27
metrics:4.2.37-489.vb_6db_69b_ce753
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
monitoring:2.5.0
oauth-credentials:0.657.v7d8dd90b_0382
okhttp-api:4.12.0-195.vc02552c04ffd
openid:218.vefb_d6f146e42
openid4java:0.9.8.0
oss-symbols-api:424.ved751e062911
pam-auth:1.12
people-view:1.16.v774d3b_4b_3a_f3
pipeline-aggregator-view:119.v4b_ec11953552
pipeline-build-step:571.v08a_fffd4b_0ce
pipeline-github:2.8-162.382498405fdc
pipeline-github-lib:65.v203688e7727e
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-gitstatuswrapper:1.2.0
pipeline-graph-analysis:245.v88f03631a_b_21
pipeline-graph-view:661.v6003f4542123
pipeline-groovy-lib:857.v085904b_c3fa_e
pipeline-input-step:540.v14b_100d754dd
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-multibranch-defaults:2.1
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.38
pipeline-utility-steps:2.20.0
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1192.v30fe6e2837ff
prism-api:1.30.0-630.va_e19d17f83b_0
project-build-times:1.2.1
project-stats-plugin:23.v47fee1f77b_84
pubsub-light:1.19
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
role-strategy:848.va_a_ea_673cf0b_c
scm-api:724.v7d839074eb_5c
script-security:1385.v7d2d9ec4d909
simple-queue:1.4.8-SNAPSHOT (private-258e29d8-klimov)
simple-theme-plugin:211.v5424a_5510e47
snakeyaml-api:2.5-143.v93b_c004f89de
solarized-theme:28.vfe25223f14fe
sse-gateway:1.28
ssh:158.ve2a_e90fb_7319
ssh-agent:386.v36cc0c7582f0
ssh-credentials:361.vb_f6760818e8c
ssh-slaves:3.1085.vc64d040efa_85
sshd:3.374.v19b_d59ce6610
structs:362.va_b_695ef4fdf9
support-core:1763.ve4ce0d10a_3f0
swarm:3.51
theme-manager:327.v780d7096ec29
thinBackup:2.1.3
throttle-concurrents:2.18
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.209.v0e69b_c43c245
variant:70.va_d9f17f859e0
view-job-filters:405.v9a_5fddc9230d
warnings-ng:12.9848.v5603977006f2
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4265.vef6200336336
workflow-durable-task-step:1464.v2d3f5c68f84c
workflow-job:1559.va_a_533730b_ea_d
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:710.v3e456cc85233
workflow-support:1004.veee3a_d67cdb_9
ws-cleanup:0.49
What Operating System are you using (both controller, and any agents involved in the problem)?
OpenIndiana on controller
Swarm Agents and SSH Agents, dozens of platforms on agents across the internet
Reproduction steps
- Set up a controller with dozens of agents
- Disconnect a lot of them at once (IRL this was part of Jenkins shutdown processing)
Expected Results
Everything goes down cleanly, with no errors in log.
Actual Results
Sometimes the agent removals overlap(?) causing a ConcurrentModificationException:
2025-12-04 07:53:29.539+0000 [id=127621] INFO jenkins.model.Jenkins$17#onAttained: Completed termination
2025-12-04 07:53:29.539+0000 [id=127621] INFO jenkins.model.Jenkins#_cleanUpDisconnectComputers: Starting node disconnection
2025-12-04 07:53:29.540+0000 [id=301] WARNING j.u.ErrorLoggingExecutorService#lambda$wrap$0
java.util.ConcurrentModificationException
at java.base/java.util.HashMap$HashIterator.remove(HashMap.java:1619)
at org.jenkinsci.remoting.protocol.IOHub.run(IOHub.java:497)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
2025-12-04 07:53:29.541+0000 [id=127626] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/192.168.251.101:2203 remote=212.20.115.91/212.20.115.91:4148]] / Computer.threadPoolForRemoting [#17474] for nutci-oo-lts terminated: java.nio.channels.ClosedChannelException
2025-12-04 07:53:29.542+0000 [id=127626] INFO h.plugins.swarm.SwarmLauncher#afterDisconnect: Removing Swarm Node for computer [nutci-oo-lts]
2025-12-04 07:53:29.542+0000 [id=301] WARNING h.u.ExceptionCatchingThreadFactory#uncaughtException: Thread Computer.threadPoolForRemoting [#17] terminated unexpectedly
java.util.ConcurrentModificationException
at java.base/java.util.HashMap$HashIterator.remove(HashMap.java:1619)
at org.jenkinsci.remoting.protocol.IOHub.run(IOHub.java:497)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
2025-12-04 07:53:29.551+0000 [id=127621] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: safe-restart thread for nutci-centos-7-amd64 terminated: java.nio.channels.ClosedChannelException
2025-12-04 07:53:29.552+0000 [id=127621] INFO h.plugins.swarm.SwarmLauncher#afterDisconnect: Removing Swarm Node for computer [nutci-centos-7-amd64]
2025-12-04 07:53:29.558+0000 [id=127621] INFO o.j.p.cloudstats.CloudStatistics#getIdFor: No support for cloud-stats-plugin by class hudson.plugins.swarm.SwarmSlave
2025-12-04 07:53:29.609+0000 [id=127621] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: safe-restart thread for nutci-ubuntu-mantic2310-amd64 terminated: java.nio.channels.ClosedChannelException
2025-12-04 07:53:29.609+0000 [id=127621] INFO h.plugins.swarm.SwarmLauncher#afterDisconnect: Removing Swarm Node for computer [nutci-ubuntu-mantic2310-amd64]
2025-12-04 07:53:29.612+0000 [id=127621] INFO o.j.p.cloudstats.CloudStatistics#getIdFor: No support for cloud-stats-plugin by class hudson.plugins.swarm.SwarmSlave
2025-12-04 07:53:29.616+0000 [id=127621] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: safe-restart thread for nutci-debian-11-amd64 terminated: java.nio.channels.ClosedChannelException
Anything else?
Tentatively considering this to be related to epic https://issues.jenkins.io/browse/JENKINS-76294 - although the failure does not seem related to (XStream) serialization which was the common theme in other cases linked to the epic.
The CME was seen during controller shutdown, but I suppose it could be reproduced by disconnecting a server with many swarm agents from the controller, and having the latter process numerous losses at once. In any case, the data structure seems to be not safe from races like this.
Are you interested in contributing a fix?
I've hunted down a few issues like this in other plugins, so feel free to pick inspiration from PRs like jenkinsci/lockable-resources-plugin#825 but I can't commit to dedicating time to this case here.
Probably a test-case with numerous parallel Threads causing agent removals can be conjured up to confirm the issue (and eventually the solution)...