Eidola: Modeling Multi-GPU Network Communication Traffic in Distributed AI Workloads

The gem5-gpu simulator (Power et al., 2015) originally extended the gem5 simulation infrastructure (Binkert et al., 2011) to model heterogeneous systems that combine CPUs and GPUs. Built by coupling gem5 with GPGPU-Sim (Bakhoda et al., 2009), the tool enables full-system simulation of tightly integrated CPU-GPU architectures, supporting detailed modeling of memory hierarchies, coherence protocols, and system-level interactions. Over the past decade, gem5 has become a foundational platform for architectural research across domains including cache coherence (Ferrer et al., 2025), memory systems (Christ et al., 2024), interconnects (Liu et al., 2025; Philomina et al., 2026), and emerging accelerators (Chen et al., 2014; Jin et al., 2025), due to its cycle-level accuracy, modularity, and community support.

Upstream gem5 provides high-fidelity models of modern AMD GPU architectures, incorporating detailed execution pipelines, SIMT scheduling, memory subsystem behavior, and interconnect integration. These models have been rigorously validated and corroborated against empirical performance measurements from real AMD hardware, giving researchers confidence in its accuracy (Ramadas et al., 2023, 2024; Roarty and Sinclair, 2020). As such, gem5 enables researchers to evaluate architectural modifications in a realistic simulation context, making it a powerful tool for studying the performance implications of both low-level microarchitectural features and system-level interactions in heterogeneous systems. We build on gem5 to extend its capabilities for multi-GPU simulation and to better model the complex communication patterns observed in modern distributed machine learning workloads.

To mitigate the performance bottlenecks arising from kernel launch overheads and inter-GPU synchronization in distributed machine learning workloads, recent work has explored the fusion of computation and communication primitives. Notably, (Punniyamurthy et al., 2024) proposes a novel fused kernel that combines a general matrix–vector multiplication (GEMV) operation with an AllReduce collective communication step. The GEMV kernel performs a dense linear algebra operation that multiplies a matrix by a vector and accumulates the result. The AllReduce operation, in contrast, is a distributed communication primitive that aggregates values across multiple GPUs and then broadcasts the aggregated result back to all participants.

These kernels are instrumental in auto-regressive transformer models such as ChatGPT, LLaMA2 (Touvron et al., 2023), and Megatron-LM (Shoeybi et al., 2020), which rely heavily on repeated matrix operations and global synchronization of model parameters. The proposed GEMV+AllReduce fusion reduces the scheduling and synchronization delays typically encountered when these operations are launched as separate kernels. The fused kernel is especially beneficial in deep learning workloads such as transformer training, where fine-grained operations and frequent synchronizations across GPUs are common.

A key architectural insight from the paper is the use of AMD’s external global memory interconnect (xGMI) to implement peer-to-peer communication between GPUs. In this fused kernel design, inter-GPU synchronization is managed through xGMI writes, where one GPU directly writes to the memory space of a peer GPU. These writes target specific flag variables, which are then polled in a lightweight control loop to gate the control flow of subsequent operations. This eliminates the need for costly round-trip synchronization protocols by enabling fine-grained, low-latency signaling between GPUs. As a result, GPUs can coordinate progress through the fused kernel with minimal interruption, enabling better overlap of computation and communication and reducing idle time across the cluster. The key steps in the fused GEMV+AllReduce kernel are shown in Figure 3. Lines 2-5 note that the kernel first computes partial sums for output tiles needed by remote GPU. Once complete, each workgroup writes a flag update to peer GPUs to indicate completion, noted by line 7. Then, each workgroup proceeds to compute partial sums that will be reduced by the executing GPU. The results of these computation are stored locally on the executing GPU. The local GPU must now wait for all remote GPUs to compute the partial tiles needed by the local GPU, and for the flag updates to indicate so, noted by lines 3. Once signaled, the local GPU can perform a reduction to determine the final result for the output tile (line 17) and broadcasts the results to all GPUs (line 18).

Inter-GPU communication through xGMI operates within the GPU memory hierarchy, which is organized into several caching layers and implements directory-based cache coherence. Each compute unit (CU) contains private L1 caches that service local memory accesses and instructions. These L1 caches are connected to a shared L2 cache, which may be partially or fully shared among multiple CUs depending on the GPU architecture. The L2 cache maintains coherence among the L1 caches and interfaces with the directory, which serves as the global coherence manager for the device. The directory tracks ownership and sharing states for cache lines, coordinating memory requests between the GPU’s internal caches, device memory, and external agents such as peer GPUs or CPUs.

In (Punniyamurthy et al., 2024), the synchronization elements, or flag variables used for inter-GPU coordination, are designated as non-cacheable memory locations. Treating these regions as non-cacheable ensures that updates are observed immediately across devices without requiring explicit invalidation or coherence traffic through the L1 or L2 caches. When a peer GPU issues an xGMI write to a polling GPU’s memory space, the request is directed through the xGMI fabric to the target GPU’s cache directory. The directory records the update atomically with respect to any pending polling reads from the local GPU, ensuring that subsequent accesses observe the most recent flag value.

This behavior is facilitated by rocSHMEM, a GPU-resident communication runtime developed by AMD and AMD Research to provide GPU-centric networking through an OpenSHMEM-like interface (Devices, ). rocSHMEM exposes one-sided communication primitives such as put, get, and atomic operations directly from GPU kernels, allowing threads to perform fine-grained data movement and synchronization without host intervention. It implements these primitives atop the xGMI interconnect, enabling intra-kernel communication and synchronization across GPUs at device speed. The library allocates a single symmetric heap across all participating GPUs, residing in each device’s memory space, which ensures a uniform address layout and simplifies pointer arithmetic in distributed kernels. By integrating communication into the GPU execution model, rocSHMEM reduces code complexity and allows for tighter coupling of computation and communication phases, where progress signaling and data exchange occur continuously within a single launch.

This mechanism provides an efficient and coherent signaling pathway between GPUs. By avoiding round-trip synchronization through the CPU or host memory subsystem, xGMI writes enable fine-grained coordination at device speed. The write completes transparently at the cache directory level, simplifying coherence management in the polling GPU’s memory hierarchy while maintaining the atomicity and ordering guarantees required for synchronization. As a result, GPUs can exchange progress signals through low-latency peer-to-peer writes, supporting tightly coupled fused kernels that overlap computation and communication effectively.

While this mechanism provides programmer-transparent communication between multiple GPUs, it also introduces new architectural complexity that is difficult to analyze using existing tools. The precise timing of xGMI transactions, cache directory updates, and polling behaviors directly influence GPU utilization, synchronization efficiency, and the ability to hide communication latency. Small variations in these interactions can lead to measurable differences in overall throughput and scaling behavior, especially in tightly coupled workloads such as fused GEMV+AllReduce (Punniyamurthy et al., 2024). Capturing these nuances requires simulation frameworks that model both intra-GPU microarchitectural events and inter-GPU communication traffic at fine granularity.

This tightly integrated approach to fused computation and synchronization highlights the importance of modeling both microarchitectural behavior and interconnect traffic in modern multi-GPU systems. However, cycle-level simulators have yet to fully support such fused communication patterns, motivating our extension of gem5 to capture these interactions in a realistic and configurable simulation environment.

Training modern machine learning and artificial intelligence models at scale follows an iterative life-cycle that combines model design, large-scale deployment, and continuous performance tuning. A typical workflow begins with model construction and data preparation, followed by distributed training across multi-GPU or multi-node clusters. Each training iteration executes a forward pass to compute activations and a backward pass to propagate gradients, both of which are highly parallelized across many GPUs. Distributed data-parallel training frameworks such as PyTorch Distributed (Li et al., 2020), Horovod (Sergeev and Balso, 2018), and DeepSpeed (Rasley et al., 2020) manage the synchronization of gradients through collective communication operations, which aggregate partial results across devices.

Large-scale systems such as NVIDIA DGX SuperPODs (24) or AMD Instinct™ MI300 clusters (1) rely on high-bandwidth, low-latency interconnects to mitigate the inter-GPU communication as model sizes and datasets complexities grow. However, scaling inefficiencies persist due to factors such as kernel launch latency, load imbalance across devices, and variable communication delays introduced by the underlying network topology (Tang et al., 2022). Even in seemingly homogeneous environments, contention within the interconnect fabric and subtle runtime differences among GPUs can lead to performance variability across nodes (Schieffer et al., 2024; Yuan et al., 2024). These sources of non-determinism affect convergence time, reduce throughput, and complicate scheduling and performance prediction.

Profiling is an essential step in addressing these challenges (Ren et al., 2010; Anderson et al., 1997). During distributed training, runtime profiling tools such as NVIDIA Nsight™ Systems (25), ROCm Profiler (36), and PyTorch Profiler (16) are used to measure GPU utilization, communication latency, and kernel execution timing (Leinhauser et al., 2021; Fiorini and Dagenais, 2022; McDonough et al., 2026; Yousefzadeh-Asl-Miandoab et al., 2023). These measurements help developers identify inefficiencies such as unbalanced workloads, excessive synchronization waits, or communication bottlenecks. The collected data also inform strategies for hyperparameter tuning, data partitioning, and kernel fusion, which can significantly improve hardware utilization and reduce total training time. Profiling further provides the empirical foundation for simulation-based performance modeling, allowing researchers to explore new architectural optimizations before hardware implementation. This can be in the form of tuning a synthetic traffic generator during simulation (Badr and Jerger, 2014; Panda et al., 2017; Wang et al., 2015; Eeckhout et al., 2000), or, as done in this work, replaying real-life traffic patterns (Badr et al., 2020; Lin et al., 2024).

Figure 4 illustrates this iterative optimization cycle. After an initial training deployment, developers first perform measurement and profiling to identify performance bottlenecks at both the kernel and system levels. These observations motivate targeted instrumentation of characteristic workloads, where specific communication or synchronization behaviors are isolated and, in some cases, deliberately perturbed to study their impact. The resulting execution traces are then collected and analyzed to understand how low-level effects—such as inter-GPU communication latency or synchronization delay—propagate to overall training performance. Once validated, optimizations derived from this process are deployed across the full compute cluster, completing the cycle.

Eidola is designed to facilitate the first three stages of this workflow. By leveraging lightweight timing profiles and enabling controlled replay of inter-GPU communication events, Eidola allows researchers to systematically study the impact of communication timing and synchronization behavior without requiring repeated execution on large-scale hardware. In particular, the ability to perturb communication timing in a controlled, cycle-level simulation environment enables fine-grained analysis that complements traditional profiling tools. This makes Eidola a practical bridge between empirical measurement and architectural exploration, supporting rapid iteration on communication-centric optimizations before deployment at scale.

By accurately capturing the timing and synchronization patterns observed in real multi-GPU workloads, profiling enables researchers to construct realistic models of distributed training behavior. These models are critical for evaluating architectural proposals that target inter-GPU communication, memory hierarchy design, and collective operation efficiency. As GPU clusters continue to scale toward exascale performance, profiling and modeling remain central to understanding and optimizing the interaction between computation and communication at every layer of the AI training stack.

In the typical training life-cycle of large-scale GPU workloads, runtime profiling is an integral part of performance tuning and optimization. The timing profiles gathered during this process, or synthetically generated patterns, can be directly leveraged to inform simulation studies. Specifically, these profiles capture inter-GPU communication behavior, including peer-to-peer writes that occur during fused computation–communication kernels. By annotating each write operation with its corresponding timestamp, the collected traces provide detailed insight into when and how GPUs exchange data.

For the fused GEMV+AllReduce kernel examined in this work, only the timestamps of peer-to-peer write operations are required to reconstruct realistic inter-GPU synchronization behavior. For other distributed workloads, users can choose to collect timing information with greater granularity, including additional communication events or memory operations, depending on the desired level of modeling fidelity. This flexibility allows the simulator to accommodate a broad range of use cases, from fast, approximate modeling to highly detailed, cycle-level analysis of multi-GPU communication dynamics.

To inject these communication events into the simulation, we introduce the new GPU pseudo op, register_write, shown in Figure 5. This function was inserted into the application before the launch of the main computation kernel and recognized by the simulator as part of a setup phase. During simulation, this setup kernel executes in functional mode, not in detailed timing mode. The sole purpose of the setup kernel (or kernels if multiple writes to register) is to preload the simulator with information about upcoming inter-GPU writes that will be enacted during detailed simulation of the main kernel.

The register_write function takes four parameters: addr, data, size, and wakeupTime. The addr parameter specifies the destination address for the emulated write, data holds the value to be written, and size indicates the width of the write in bytes (ranging from one to eight bytes). The wakeupTime parameter defines the time offset, in nanoseconds, after kernel launch at which the write should be issued. At runtime, these parameters are read by the simulator and stored in a queue, sorted by wakeupTime. These timestamps are converted into cycles based on the device clock frequency defined in the gem5 configuration.

We extend the simulator with a data structure to maintain all pending writes in a write tracking table (WTT), implemented as a priority queue sorted by wakeupTime. When a write is registered, it is inserted into the WTT at a position determined by its wakeupTime, ensuring that the earliest writes are processed first. During simulation of the main kernel, the head of the WTT is polled on every simulated cycle tick. If the current simulated time is less than the wakeupTime of the head entry, the simulator simply advances to the next cycle, incurring negligible runtime overhead (only a single comparison per cycle). When the simulated time meets or exceeds the wakeupTime, all write entries at the head of the queue with the same timestamp are popped and enacted as xGMI writes. These events are processed in constant time, as the number of simultaneous writes is typically small and bounded by the number of participating GPUs. This priority queue-based design ensures both temporal accuracy and efficiency, allowing users to register writes in arbitrary order. The means sequential calls to register_write() need not correspond to the chronological order of their execution. By decoupling registration from enactment, this mechanism allows flexible modeling of inter-GPU communication patterns with minimal simulation overhead.

As mentioned above, when the main kernel enters detailed timing mode, the simulator begins polling the WTT on every simulated cycle. Once the current simulation time reaches the wakeupTime associated with the next pending event, the simulator issues the corresponding xGMI write on behalf of the emulated GPU. The write transaction completes at the cache directory level, accurately reflecting the hardware behavior described in Section 2.2, where peer-to-peer writes update non-cacheable memory regions atomically and without round-trip acknowledgment. Upon completion, the memory state of the receiving GPU is updated to reflect the new flag value. When the receiving GPU subsequently polls this address the updated data is detected and propagated to the requesting CU.

This implementation provides a faithful simulation of the low-latency, one-sided communication semantics employed in rocSHMEM (Devices, ), where intra-kernel synchronization primitives rely on direct xGMI writes rather than host-mediated messaging. By modeling these events at cycle granularity and maintaining an efficient WTT structure, the simulator achieves both temporal fidelity and scalability, enabling the study of tightly coupled multi-GPU workloads that overlap computation and communication within a single fused kernel execution. The flexibility of Eidola’s write scheduling methodology allows this framework to be useful for a variety of memory traffic generation schemes, including synthetic, (Badr and Jerger, 2014; Panda et al., 2017; Wang et al., 2015; Eeckhout et al., 2000), as well as replicatory (Badr et al., 2020; Lin et al., 2024).

We evaluate this extension in our multi-GPU simulation environment under the same test configuration described in 4. As shown in Figure 9, once SyncMon support is enabled, the number of memory reads issued by the simulated GPU no longer scales linearly with the modeled wait time. This contrasts sharply with the baseline spin-wait behavior shown in Figure 6, where increased wait times led to a proportional increase in polling reads due to continuous flag checking.

With SyncMon-enabled execution, the number of reads associated with polling on flag variables remains effectively constant across all configurations, ranging between 728 and 788 reads. This is a significant departure from the baseline, where polling traffic grows with the duration of the wait. The bounded number of flag-related reads reflects the transition from an active spin-wait to a spin-yield synchronization model, where waiting wavefronts are descheduled and only resume execution upon receiving a notification event.

Non-flag-related memory reads remain largely unchanged, staying consistent with the values observed in Figure 6 at approximately 66K reads. This indicates that the introduction of SyncMon does not perturb the underlying computational behavior or memory access patterns of the kernel. Instead, it selectively reduces only the unnecessary memory traffic generated by synchronization, isolating the benefit of the proposed mechanism. These results validate the expected behavior described in the original SyncMon proposal and demonstrate that Eidola can accurately capture the performance implications of advanced synchronization mechanisms.

To evaluate how Eidola behaves under increasing computational workload, we study the relationship between application input size and overall simulation time. In particular, we vary the input matrix dimension M, which corresponds to the number of output rows computed by the fused GEMV+AllReduce kernel. Increasing M proportionally increases the amount of arithmetic work performed by each GPU, as more multiply-accumulate operations must be completed per kernel invocation. This experiment allows us to verify that Eidola preserves expected scaling trends as application complexity grows.

Figure 10 compares total gem5 simulation time as the input dimension M is varied. Across all configurations, simulation time increases approximately linearly with M, as indicated by $r^{2}$ values ranging from 0.76 to 0.98 for the fitted trendlines. This behavior is consistent with expectations: as M grows, each GPU executes a larger number of compute operations, leading to a proportional increase in detailed timing simulation.

Importantly, both the baseline configuration (solid lines) and the mwait-instrumented configuration (dashed lines) exhibit similar scaling trends. This indicates that the additional synchronization modeling introduced by Eidola, including emulated xGMI writes and optional mwait-based behavior, does not fundamentally alter the relationship between workload size and simulation cost. Instead, the dominant factor remains the execution time of the simulated GPU pipeline. These results confirm that Eidola preserves the expected computational scaling characteristics of the underlying application while introducing only modest overhead for communication modeling.

The fused GEMV+AllReduce kernel inherently requires multiple GPUs to execute, as its computation depends on distributed data exchange and synchronization across devices. In practice, the kernel requires at least two GPUs to run and achieves its highest efficiency when executed on a power-of-two number of GPUs. As a result, directly measuring the simulation time of a single GPU execution ($t_{1GPU_{M}}$) is not possible for this workload. This presents a challenge when attempting to quantify the overhead introduced by Eidola relative to a baseline single-GPU detailed simulation.

To address this, we vary the number of emulated GPUs (eGPUs) from 3 to 255 and measure the total wall-clock simulation time across configurations. We then use these measurements to extrapolate the effective cost of simulating a single GPU and the incremental overhead associated with each additional eGPU. Specifically, we group trials by input dimension $M$, fit a linear model across configurations, and extrapolate to $eGPU=1$ to estimate both $t_{1GPU_{M}}$ and $t_{eGPU_{M}}$. This approach allows us to isolate the contribution of Eidola’s communication modeling infrastructure to overall simulation time, even in the absence of a directly measurable single-GPU baseline.

Formally, we model total simulation time $t_{M}$ as:

where $t_{1GPU_{M}}$ represents the estimated time to simulate a single GPU in detailed timing mode, and $t_{eGPU_{M}}$ captures the per-eGPU overhead associated with maintaining additional communication state and replaying inter-GPU events. Here, $t_{M}$ is the measured simulation time for a given configuration, and $eGPUs$ is the number of emulated GPUs.

Because simulation time is also a function of input size, all parameters are derived per input dimension M, as discussed in Section 5.2. After fitting the linear model in Equation 1, we normalize the measured simulation time by the estimated $t_{1GPU_{M}}$ and plot the results in Figure 11. This normalization enables direct comparison against an idealized baseline in which only a single GPU is simulated in detail.

Figures 11a) and 11b) show that simulation time grows sub-linearly with the number of eGPUs across all configurations. For small numbers of eGPUs, normalized execution time approaches 1.0, indicating that the simulation cost is dominated by the detailed modeling of the primary GPU. As the number of eGPUs increases, total simulation time increases, but at a much slower rate than linear scaling. Even in the largest configuration with 255 eGPUs and mwait instrumentation enabled, normalized execution time ranges between 7.3$\times$ and 35.9$\times$, far below the 256$\times$ cost that would be expected if all GPUs were simulated in full detail.

This behavior confirms that the overhead introduced by Eidola is modest relative to the cost of detailed GPU simulation. The estimated $t_{eGPU_{M}}$ values indicate that the additional work required to maintain the Write Tracking Table (WTT) and replay communication events is small and scales sub-linearly with the number of eGPUs. While an event-driven implementation using gem5’s native event queues could further reduce this overhead by eliminating per-cycle polling, the current design already provides a favorable trade-off between efficiency, transparency, and ease of debugging.

Overall, these results demonstrate that Eidola enables scalable multi-GPU simulation by avoiding the need to fully model every GPU in the system, while still capturing the essential communication behavior required for architectural analysis.

The SyncMon study exemplifies how our proposed methodology can be directly applied to support GPU architectural research. Across the preceding experiments, we demonstrate that Eidola enables both accurate modeling of communication-driven behavior and scalable simulation of large multi-GPU systems. In Section 5.1, we validated Eidola’s ability to model architectural mechanisms through the SyncMon case study. By replacing spin-wait polling with a notification-based synchronization primitive, we observed that flag-related memory reads remain bounded (728–788 reads) rather than scaling with wait time, while non-flag memory traffic remains stable at approximately 66K reads. This demonstrates that Eidola can isolate and quantify the impact of synchronization mechanisms on memory behavior without perturbing the underlying computation.

In Section 5.2, we showed that simulation time scales linearly with input size, confirming that Eidola preserves the expected relationship between application workload and execution cost. This ensures that increasing computational intensity does not introduce non-physical artifacts in simulation, allowing researchers to confidently study workload-dependent trends.

Finally, in Section 5.3, we further demonstrated that simulation time grows sub-linearly with the number of emulated GPUs. By extrapolating a single-GPU baseline and isolating the per-eGPU overhead, we showed that Eidola avoids the prohibitive cost of fully simulating all devices while still capturing inter-GPU communication behavior. Even at large scales, the additional overhead introduced by communication replay and Write Tracking Table (WTT) management remains modest relative to detailed GPU pipeline simulation. This scalability is critical for studying modern distributed workloads, which routinely span tens to hundreds of GPUs.

Taken together, these results highlight Eidola’s effectiveness as a research platform. By providing cycle-level, configurable modeling of multi-GPU systems, Eidola opens the door to exploring emerging hardware features in realistic, large-scale environments. As multi-GPU systems continue to underpin large-scale model training, such capabilities will be increasingly important for guiding both academic research and industrial design.