[JENKINS-72226] Remote Class Loading slows down step initialization

[jenkins-infra-bot]

The remote Class Loading of a Jenkins agent may slow down step initialization and cause some delays depending on the environment (network latency).

Generally, this impact a particular step / feature the first time it is used. For example, the first execution of a checkout step using GitSCM on a newly started agent would need to load Git plugin and Git client plugin related classes. Then next execution of that step would not load those classes anymore, and that step executioon would generally be a few seconds faster (in some environment, I have seen a delay of actually almost 30s).

*BUT, if the agent process is restarted, then again the next execution will need to go through the class loading of that step. And that is regardless of whether the **jarCache* is pre-populated. So this is really about the performance of remoting Class loading.

This delay seems negligible, especially when using permanent agent. But for ephemeral agents (such as when using the Kubernetes plugin) the impact of the performance of the class loading is not negligible anymore as it happens almost all the time.
Some steps - like the git / checkout steps that probably have more class to load - causes a bigger delay than other.

Data

Thread dump on the agent side would show that we spend this time in:

"pool-1-thread-7 for test-agent id=7994 / waiting for test-agent id=382" #39 daemon prio=5 os_prio=31 cpu=692.03ms elapsed=95.84s tid=0x00007f91aaa8e800 nid=0x14003 in Object.wait()  [0x000070001269f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@​11.0.16.1/Native Method)
	- waiting on 
	at hudson.remoting.Request.call(Request.java:177)
	- waiting to re-lock in wait() <0x000000061bac6e08> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
	at com.sun.proxy.$Proxy7.fetch3(Unknown Source)
	at hudson.remoting.RemoteClassLoader.prefetchClassReference(RemoteClassLoader.java:354)
	at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:259)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:229)
	at java.lang.ClassLoader.loadClass(java.base@​11.0.16.1/ClassLoader.java:589)
	- locked <0x000000061da00000> (a hudson.remoting.RemoteClassLoader)
	at java.lang.ClassLoader.loadClass(java.base@​11.0.16.1/ClassLoader.java:522)
	at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:936)
	at hudson.util.XStream2.setupConverters(XStream2.java:286)
	at com.thoughtworks.xstream.XStream.(XStream.java:548)
	at com.thoughtworks.xstream.XStream.(XStream.java:476)
	at com.thoughtworks.xstream.XStream.(XStream.java:450)
	at com.thoughtworks.xstream.XStream.(XStream.java:403)
	at com.thoughtworks.xstream.XStream.(XStream.java:377)
	at hudson.util.XStream2.(XStream2.java:171)
	at hudson.ProxyConfiguration.(ProxyConfiguration.java:506)
	at java.lang.Class.forName0(java.base@​11.0.16.1/Native Method)
	at java.lang.Class.forName(java.base@​11.0.16.1/Class.java:315)
	at com.sun.proxy.$Proxy10.(Unknown Source)
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@​11.0.16.1/Native Method)
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@​11.0.16.1/NativeConstructorAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@​11.0.16.1/DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(java.base@​11.0.16.1/Constructor.java:490)
	at java.lang.reflect.Proxy.newProxyInstance(java.base@​11.0.16.1/Proxy.java:1022)
	at java.lang.reflect.Proxy.newProxyInstance(java.base@​11.0.16.1/Proxy.java:1008)
	at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:168)
	at hudson.remoting.Channel.export(Channel.java:812)
	at hudson.remoting.Channel.export(Channel.java:775)
	at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:232)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@​11.0.16.1/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@​11.0.16.1/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@​11.0.16.1/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@​11.0.16.1/Method.java:566)
	at java.io.ObjectStreamClass.invokeWriteReplace(java.base@​11.0.16.1/ObjectStreamClass.java:1106)
	at java.io.ObjectOutputStream.writeObject0(java.base@​11.0.16.1/ObjectOutputStream.java:1127)
	at java.io.ObjectOutputStream.writeObject(java.base@​11.0.16.1/ObjectOutputStream.java:345)
	at hudson.remoting.UserRequest._serialize(UserRequest.java:263)
	at hudson.remoting.UserRequest.serialize(UserRequest.java:272)
	at hudson.remoting.UserRequest.perform(UserRequest.java:222)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at hudson.remoting.InterceptingExecutorService$$Lambda$95/0x000000080025fc40.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(java.base@​11.0.16.1/FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@​11.0.16.1/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@​11.0.16.1/ThreadPoolExecutor.java:628)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
	at hudson.remoting.Engine$1$$Lambda$96/0x000000080025f040.run(Unknown Source)
	at java.lang.Thread.run(java.base@​11.0.16.1/Thread.java:829)

I collected FINE logs of the hudson.remoting.RemoteInvocationHandler and hudson.remoting.RemoteClassLoader when the agent perform the first checkout after being started. And also after being restarted.

• 00-first-checkout.log.1
• 01-after-restart.log.0

Note: FINE logs for those classes don't show anything between checkout steps when the agent is not restarted. So no point collecting something in that case.

How to Reproduce

• Spin up Jenkins
• Connect a permanent Agent
• Create a simple pipeline like the following

pipeline {
    
    agent {
label "test-agent"
    }

    options {
skipDefaultCheckout()
timestamps()
    }
    
    stages {
stage('Test') {
    steps {
checkout(
    changelog: false, 
    poll: false, 
    scm: [$class: 'GitSCM', 
branches: [[name: '*/master']],
extensions: [cloneOption(honorRefspec: true, noTags: true, reference: '', shallow: false)],
userRemoteConfigs: [[
    refspec: '+refs/heads/master:refs/remotes/origin/master',
    credentialsId: 'test-githubapp',
    url: 'https://github.com/jenkinsci/support-core-plugin'
    ]]
    ]
)
    }
}
    }
}

• Build the job
• Notice the delay when initializing the client:

17:15:48  The recommended git tool is: NONE
17:15:52  using credential test-githubapp

• Build again
• Notice that there is no delay anymore:

17:17:46  The recommended git tool is: NONE
17:17:46  using credential test-githubapp

• Restart the agent (kill the process, start it again
• Notice that the delay is back:

17:19:15  The recommended git tool is: NONE
17:19:19  using credential test-githubapp

---

Originally reported by allan_burdajewicz, imported from: Remote Class Loading slows down step initialization

• status: Open
• priority: Major
• component(s): remoting
• resolution: Unresolved
• votes: 0
• watchers: 2
• imported: 2025-11-25

Raw content of original issue

The remote Class Loading of a Jenkins agent may slow down step initialization and cause some delays depending on the environment (network latency).

Generally, this impact a particular step / feature the first time it is used. For example, the first execution of a checkout step using GitSCM on a newly started agent would need to load Git plugin and Git client plugin related classes. Then next execution of that step would not load those classes anymore, and that step executioon would generally be a few seconds faster (in some environment, I have seen a delay of actually almost 30s).

*BUT, if the agent process is restarted, then again the next execution will need to go through the class loading of that step. And that is regardless of whether the **jarCache* is pre-populated. So this is really about the performance of remoting Class loading.

This delay seems negligible, especially when using permanent agent. But for ephemeral agents (such as when using the Kubernetes plugin) the impact of the performance of the class loading is not negligible anymore as it happens almost all the time.
Some steps - like the git / checkout steps that probably have more class to load - causes a bigger delay than other.

Data

Thread dump on the agent side would show that we spend this time in:


"pool-1-thread-7 for test-agent id=7994 / waiting for test-agent id=382" #39 daemon prio=5 os_prio=31 cpu=692.03ms elapsed=95.84s tid=0x00007f91aaa8e800 nid=0x14003 in Object.wait()  [0x000070001269f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.16.1/Native Method)
	- waiting on <no object reference available>
	at hudson.remoting.Request.call(Request.java:177)
	- waiting to re-lock in wait() <0x000000061bac6e08> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
	at com.sun.proxy.$Proxy7.fetch3(Unknown Source)
	at hudson.remoting.RemoteClassLoader.prefetchClassReference(RemoteClassLoader.java:354)
	at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:259)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:229)
	at java.lang.ClassLoader.loadClass(java.base@11.0.16.1/ClassLoader.java:589)
	- locked <0x000000061da00000> (a hudson.remoting.RemoteClassLoader)
	at java.lang.ClassLoader.loadClass(java.base@11.0.16.1/ClassLoader.java:522)
	at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:936)
	at hudson.util.XStream2.setupConverters(XStream2.java:286)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:548)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:476)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:450)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:403)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:377)
	at hudson.util.XStream2.<init>(XStream2.java:171)
	at hudson.ProxyConfiguration.<clinit>(ProxyConfiguration.java:506)
	at java.lang.Class.forName0(java.base@11.0.16.1/Native Method)
	at java.lang.Class.forName(java.base@11.0.16.1/Class.java:315)
	at com.sun.proxy.$Proxy10.<clinit>(Unknown Source)
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@11.0.16.1/Native Method)
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@11.0.16.1/NativeConstructorAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@11.0.16.1/DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(java.base@11.0.16.1/Constructor.java:490)
	at java.lang.reflect.Proxy.newProxyInstance(java.base@11.0.16.1/Proxy.java:1022)
	at java.lang.reflect.Proxy.newProxyInstance(java.base@11.0.16.1/Proxy.java:1008)
	at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:168)
	at hudson.remoting.Channel.export(Channel.java:812)
	at hudson.remoting.Channel.export(Channel.java:775)
	at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:232)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.16.1/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.16.1/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.16.1/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.16.1/Method.java:566)
	at java.io.ObjectStreamClass.invokeWriteReplace(java.base@11.0.16.1/ObjectStreamClass.java:1106)
	at java.io.ObjectOutputStream.writeObject0(java.base@11.0.16.1/ObjectOutputStream.java:1127)
	at java.io.ObjectOutputStream.writeObject(java.base@11.0.16.1/ObjectOutputStream.java:345)
	at hudson.remoting.UserRequest._serialize(UserRequest.java:263)
	at hudson.remoting.UserRequest.serialize(UserRequest.java:272)
	at hudson.remoting.UserRequest.perform(UserRequest.java:222)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at hudson.remoting.InterceptingExecutorService$$Lambda$95/0x000000080025fc40.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(java.base@11.0.16.1/FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16.1/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16.1/ThreadPoolExecutor.java:628)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
	at hudson.remoting.Engine$1$$Lambda$96/0x000000080025f040.run(Unknown Source)
	at java.lang.Thread.run(java.base@11.0.16.1/Thread.java:829)


I collected FINE logs of the hudson.remoting.RemoteInvocationHandler and hudson.remoting.RemoteClassLoader when the agent perform the first checkout after being started. And also after being restarted. 

	
00-first-checkout.log.1
	
01-after-restart.log.0

Note: FINE logs for those classes don't show anything between checkout steps when the agent is not restarted. So no point collecting something in that case.
How to Reproduce

	
Spin up Jenkins
	
Connect a permanent Agent
	
Create a simple pipeline like the following


pipeline {
agent {
    label <span class="code-quote">"test-agent"</span>
}

options {
    skipDefaultCheckout()
    timestamps()
}

stages {
    stage(<span class="code-quote">'Test'</span>) {
        steps {
            checkout(
                changelog: <span class="code-keyword">false</span>, 
                poll: <span class="code-keyword">false</span>, 
                scm: [$class: <span class="code-quote">'GitSCM'</span>, 
                    branches: [[name: <span class="code-quote">'*/master'</span>]],
                    extensions: [cloneOption(honorRefspec: <span class="code-keyword">true</span>, noTags: <span class="code-keyword">true</span>, reference: '', shallow: <span class="code-keyword">false</span>)],
                    userRemoteConfigs: [[
                        refspec: <span class="code-quote">'+refs/heads/master:refs/remotes/origin/master'</span>,
                        credentialsId: <span class="code-quote">'test-githubapp'</span>,
                        url: <span class="code-quote">'https:<span class="code-comment">//github.com/jenkinsci/support-core-plugin'</span>

                    ]]

]

)

}

}

}

}




	
Build the job
	
Notice the delay when initializing the client:


17:15:48  The recommended git tool is: NONE
17:15:52  using credential test-githubapp



	
Build again
	
Notice that there is no delay anymore:


17:17:46  The recommended git tool is: NONE
17:17:46  using credential test-githubapp



	
Restart the agent (kill the process, start it again
	
Notice that the delay is back:


17:19:15  The recommended git tool is: NONE
17:19:19  using credential test-githubapp

2 attachments

• 00-first-checkout.log.1
• 01-after-restart.log.0

[jenkins-infra-bot]

• Remote link associated with this issue:

  • remoting #​721

[jenkins-infra-bot]

jglick:

• Original comment link
• Raw content of original comment:

  JarCache may be working as designed, but RemoteClassLoader is still making far too many RPC calls: there ought to be at most one per JAR, simply to establish the class loader hierarchy, but instead it is making dozens of calls per loaded class (despite some being skipped by the prefetch optimization). On a high-latency channel (e.g., WebSocket from a distant network) this could be extremely slow, at least when many classes in fact need to be loaded.
  
  The workaround in this case is probably to select the CLI Git tool explicitly, which should require far less class loading.
  

JarCache may be working as designed, but RemoteClassLoader is still making far too many RPC calls: there ought to be at most one per JAR, simply to establish the class loader hierarchy, but instead it is making dozens of calls per loaded class (despite some being skipped by the prefetch optimization). On a high-latency channel (e.g., WebSocket from a distant network) this could be extremely slow, at least when many classes in fact need to be loaded.

The workaround in this case is probably to select the CLI Git tool explicitly, which should require far less class loading.

[jenkins-infra-bot]

allan_burdajewicz:

• Original comment link
• Raw content of original comment:

  Interestingly, I quickly try setting an explicit gitTool but that made no difference.
  

Interestingly, I quickly try setting an explicit gitTool but that made no difference.

[jenkins-infra-bot]

jglick:

• Original comment link
• Raw content of original comment:

  FTR recent releases of CloudBees CI on K8s now include a warmed-up JAR cache (based on commonly used steps) in the custom agent image used by default for K8s builds. The OSS images could perhaps be improved to do the same, though it would be trickier since the caches are specific to a given JAR version; in the CloudBees CI case, the agent image tag is tied to the product release, which uses a known version of Jenkins core and all plugins. An OSS equivalent would need to include caches applicable to all recent core & plugin releases, and/or have much more frequent image updates than is the case today.
  
  At any rate, my understanding of the issue is that having a warm JAR cache is not sufficient, because the code to even look up the JAR / cache key is inefficient? Perhaps we could deprecate the current scheme by which the agent asks the controller for JAR keys; instead the controller side when sending a UserRequest should be proactively including some information about all the types being serialized in it for the first time in this Channel. That would avoid any additional network round-trips for remote class loading when the JAR cache is already complete.
  

FTR recent releases of CloudBees CI on K8s now include a warmed-up JAR cache (based on commonly used steps) in the custom agent image used by default for K8s builds. The OSS images could perhaps be improved to do the same, though it would be trickier since the caches are specific to a given JAR version; in the CloudBees CI case, the agent image tag is tied to the product release, which uses a known version of Jenkins core and all plugins. An OSS equivalent would need to include caches applicable to all recent core & plugin releases, and/or have much more frequent image updates than is the case today.

At any rate, my understanding of the issue is that having a warm JAR cache is not sufficient, because the code to even look up the JAR / cache key is inefficient? Perhaps we could deprecate the current scheme by which the agent asks the controller for JAR keys; instead the controller side when sending a UserRequest should be proactively including some information about all the types being serialized in it for the first time in this Channel. That would avoid any additional network round-trips for remote class loading when the JAR cache is already complete.

[jenkins-infra-bot]

jglick:

• Original comment link
• Raw content of original comment:

  From the doc linked in the PR
  
  find an encoding of the class loader hierarchy that could be efficiently transmitted to the receiving side in order for the receiving side to be able to load classes from the JAR cache without constant RPCs to the sending side
  
  Perhaps. But it would probably also suffice to deprecate the entire remote class loader hierarchy system. So far as I know, it only serves one (theoretical) purpose: mirroring the controller-side behavior in controller-to-agent callables in cases where the same FQN is defined in multiple plugins, for example where one plugin loads org.whatever:somelib:1.1 and another loads org.whatever:somelib:2.3 (without https://www.jenkins.io/doc/developer/plugin-development/dependencies-and-class-loading/#shading that is!) and both of these third-party libraries are transferred to the agent (and its JAR cache) and used. This is I think a very unlikely scenario, and one we would rather address by just shipping one version of the lib, or at worst shading the older ones. Without that constraint, we could simplify Remoting class loading drastically to just use a single URLClassLoader; whenever it fails to find a class in its current URL set, it could ask the controller for a JAR cache key to use for that class name, which would then add the whole JAR to its classpath, or the controller could probably be made to push this information as part of user requests.
  

From the doc linked in the PR

find an encoding of the class loader hierarchy that could be efficiently transmitted to the receiving side in order for the receiving side to be able to load classes from the JAR cache without constant RPCs to the sending side

Perhaps. But it would probably also suffice to deprecate the entire remote class loader hierarchy system. So far as I know, it only serves one (theoretical) purpose: mirroring the controller-side behavior in controller-to-agent callables in cases where the same FQN is defined in multiple plugins, for example where one plugin loads org.whatever:somelib:1.1 and another loads org.whatever:somelib:2.3 (without https://www.jenkins.io/doc/developer/plugin-development/dependencies-and-class-loading/#shading that is!) and both of these third-party libraries are transferred to the agent (and its JAR cache) and used. This is I think a very unlikely scenario, and one we would rather address by just shipping one version of the lib, or at worst shading the older ones. Without that constraint, we could simplify Remoting class loading drastically to just use a single URLClassLoader; whenever it fails to find a class in its current URL set, it could ask the controller for a JAR cache key to use for that class name, which would then add the whole JAR to its classpath, or the controller could probably be made to push this information as part of user requests.

[jenkins-infra-bot]

[Original relates_to from Jira: JENKINS-67829]