ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories

The inference memory system is architected as a multi-tier hierarchy to balance the conflicting requirements of high-bandwidth access and high-density storage. Figure 1 (a) illustrates this hierarchy, which organizes data based on its latency sensitivity and capacity footprint. GPU memory (T1) serves as the highest tier, utilizing HBM to provide the peak memory bandwidth necessary for real-time token generation. Directly supporting this is host memory (T2), which functions as a byte-addressable overflow tier for KV blocks evicted from T1. Despite their performance, the physical scaling constraints of these memory-centric tiers inevitably lead to a capacity wall as model parameters and context lengths increase. To provide deeper capacity, the hierarchy incorporates storage-based layers: single-server NVMe SSDs (T3) and cluster-wide shared storage (T4), which provide individual node capacity and massive cross-node storage, respectively.

The emergence of agentic AI and multi-turn workloads has exposed a critical capacity and accessibility gap between server-internal NVMe SSDs (T3) and remote shared storage (T4). While local storage (T3) provides fast local access, its limited physical capacity forces the system to frequently evict inference states, losing the opportunity for context reuse in subsequent turns. Conversely, while remote storage (T4) offers the necessary scale, it acts more as a cold archive rather than an active memory tier. The high latency overhead of retrieving remote states from remote storage (T4) makes it impractical to support the frequent, real-time context retrieval required by long-context inference workloads. To bridge this gap, recent industry efforts have introduced NVIDIA CMX context memory storage (NVIDIA, 2026b), a technology designed to accelerate remote access and establish a specialized shared context tier (T3.5). By providing a disaggregated, shared context layer, CMX context memory storag enables cumulative inference states, such as shared KV caches, to remain reusable and warm across the cluster. Currently, most of these implementations are DPU-centric, utilizing specialized processors like NVIDIA BlueField to offload networking and storage stacks (NVIDIA, 2026b). While DPU-based approaches improve remote-path efficiency, they require expensive, high-compute hardware and complex software stacks to manage the networking and storage layers. Instead of providing direct, low-overhead hardware access, these solutions impose a heavy management burden, demanding significant engineering effort to optimize the underlying remote storage tiers.

Our proposed ITME realizes this shared context tier (T3.5) by transforming SSD-backed capacity into a direct-access memory expansion. As illustrated in Figure 1 (b), ITME adopts a more efficient approach by presenting itself as a remote memory server via a CXL-hybrid-memory, breaking the dependency on expensive DPUs and utilizing commodity, low-cost RNICs. The internal hardware controller directly issues NVMe requests, bypassing the traditional software-defined storage stack. To efficiently manage remote transfers, ITME employs a DMA-based pipeline that orchestrates data movement, effectively masking network latency and ensuring continuous high-throughput data delivery. By transforming backing SSDs into an active, byte-addressable memory pool, ITME can host TB-scale inference states and efficiently serve highly predictable long-context data to maximize CXL-hybrid-memory performance for subsequent inference turns.

The architectural efficiency of ITME stems from the high predictability of specific LLM data types (Table 1). By exploiting these deterministic access patterns, ITME proactively manages data to effectively mask the latency of the remote CXL-hybrid memory.

• •

  Model Weights: Weights are consumed in a strict layer-by-layer sequence during the forward pass. This predictability enables ITME to utilize layer-wise prefetching, successfully overlapping I/O with computation. Offloading massive weight footprints (e.g., 325 GB for OPT-175B in FP16) to the CXL-hybrid memory (T3.5) reclaims valuable T1/T2 capacity for latency-sensitive active states.

• •

  Long-context Prefix KV Cache: Multi-turn and agentic workloads exhibit an append-write, sequential-restore access pattern. As shown in Figure 2 (a), ITME leverages scheduler-aware fetching to restore shared-prefix KV chunks from the CXL-hybrid memorys ahead of execution. Rather than eagerly reconstructing the entire prefix on the GPU, chunks are staged and replayed layer-wise. This allows prefix reuse to overlap with attention execution while efficiently exploiting the large sequential chunk transfers of the hybrid tier.

• •

  Activations and Working KV: Due to their extreme latency sensitivity, activations and the incrementally growing working KV cache are strictly prioritized for GPU memory or host memory. Any access delay during the decoding phase causes severe pipeline stalls. Because their total memory footprint remains relatively small, GPU and host memory can comfortably satisfy their capacity demands without requiring offloading to ITME.

By targeting the deterministic access patterns of weights and long-context KV caches, ITME establishes an efficient prefetching pipeline from the CXL-hybrid memory to GPU memory, using host memory as an intermediate staging layer (Figure 2 (b)).

The key insight is that layer execution exposes prefetch opportunities that can overlap data movement with GPU computation. Rather than waiting until data is demanded, ITME issues prefetches for upcoming blocks during the execution of earlier layers, allowing CXL-hybrid memory (T3.5)-to-GPU memory (T1) transfers to progress ahead of use. This pipelined execution model helps hide a significant fraction of transfer latency behind useful computation. Furthermore, DMA-based hardware prefetching removes the host CPU from the critical path, reducing software overhead and effectively masking the overall system latency

As shown in Figure 3, the effective throughput across the entire data path from the CXL-hybrid memory to host memory and subsequently to GPU memory increases with transfer size and approaches the hardware limit for sufficiently large transfers. Notably, measurements from our FPGA prototype demonstrate that when data is effectively staged into the internal DRAM cache via prefetching, the CXL-hybrid memory sustains a near-peak throughput of approximately 18 GB/s across all evaluated transfer sizes. For model weights, the transfer granularity is naturally large, for example, 1344 MB for Llama-3.1 70B. This allows ITME to utilize the available PCIe bandwidth efficiently. Combined with the deterministic layer-by-layer access pattern, this property enables a DMA-based pipeline in which the next layer’s weights are prefetched while the current layer is still under execution. For prefix KV cache, individual blocks can be relatively small, such as 16 MB at a block size of 128. To improve transfer efficiency, ITME groups multiple KV blocks into a larger aggregated transfer unit, for example, 32 blocks forming a 512MB request. Although the transfer efficiency from host to GPU memory can drop for small, non-aggregated transfers, the use of the CXL-hybrid memory together with 200Gbps RDMA enables these larger units to be streamed effectively through the pipeline. This hardware-driven data movement allows the CXL-hybrid memory to sustain high effective throughput and remain ahead of GPU demand.

Figure 4 illustrates the proposed CXL-hybrid memory architecture, which integrates a high-capacity NVMe SSD and a transparent, hardware-managed DRAM cache into a unified, byte-addressable memory space. A primary design objective is to fully saturate the host PCIe bandwidth to support high-speed DMA-based pipelining among GPUs, the host CPU, and RDMA NICs. Accordingly, the CXL-hybrid memory in our design can be utilized as a general-purpose memory expansion, while its performance is further maximized through specialized high-speed access optimized for the predictable data patterns targeted in this work. To achieve this, our design focuses on the following key points:

• •

  Hit/Miss Checker: Upon receiving a read or write request from the host, the checker evaluates the 16-way set-associative metadata to determine a hit or miss.

  • –

    Metadata Read: It first accesses the metadata buffer to verify the valid and tag fields.

  • –

    Metadata Write (Cache Hit): In the event of a hit, it updates the age information to maintain the replacement policy.

  • –

    Dirty Update: If the transaction is a write request, the dirty bit is additionally updated to 1.

  • –

    Parallel Processing: The system includes two hit/miss Checkers, with one dedicated to each CXL slice (data path) to ensure high-throughput request handling.

• •

  Miss Line Handler: This unit coordinates the metadata updates required during cache misses and evictions.

  • –

    Cache Miss Handling: The handler facilitates the transfer of the requested page from the SSD to the DRAM cache and performs a metadata write to update the corresponding entry (e.g., setting the valid bit from 0 to 1).

  • –

    Cache Eviction: When the cache is full, the handler identifies a victim line based on the LRU policy. It then executes a metadata read and Write to invalidate the evicted line (setting the valid bit from 1 to 0) before staging the new data.

To ensure data consistency among these concurrent units, the metadata interface controller implements a hardware-level locking mechanism. Every metadata update is executed as a three-stage atomic operation: (1) metadata read and lock, (2) metadata Write, and (3) metadata Unlock. This sequential process requires three clock cycles per update, which can limit the metadata throughput during intensive processing. This internal logic and the resulting metadata footprint are the primary factors in determining the optimal DRAM and SRAM capacity for the system.

• •

  SSD Capacity Scaling: Doubling the SSD capacity from 1 TB to 2 TB (2-ch) while keeping the DRAM at 32 GB requires only one additional tag bit (10-bit tag). This results in a marginal 6.7% increase in the SRAM footprint, bringing it to 16 MB.

• •

  DRAM Capacity Scaling: Conversely, doubling the DRAM cache size to 64 GB for a 1 TB SSD doubles the number of cache sets to 1 M, requiring an expanded 20-bit index. This leads to a disproportionate 86.7% surge in SRAM requirements, pushing the footprint up to 28 MB.

Considering the significant SRAM penalty, expanding the DRAM to 64 GB is not resource-efficient. Thus, we selected 32 GB as the baseline capacity, which strikes a practical balance for our hardware-managed cache.

As summarized in Table 3, our empirical evaluation demonstrates the effectiveness of the proposed memory configuration. A baseline setup utilizing 2-channel DRAM and a 1-channel SSD achieves only 10 GB/s for Host-to-DRAM transfers, falling short of the peak interface potential. Upgrading the memory to a 4-channel DRAM configuration (our 32 GB setup) improves the Host-to-DRAM throughput to an optimal 18 GB/s. However, if this is paired with only a 1-channel SSD, the backend SSD-to-DRAM throughput remains severely constrained at 9 GB/s. This bandwidth mismatch is critical, as it starves the hardware prefetcher and limits its ability to supply data to the cache at a rate sufficient to saturate the host-side bandwidth during cache misses.

By finalizing the topology with both 4-channel DRAM and a 2-channel SSD, we effectively resolve this bandwidth mismatch. This balanced configuration enables the system to refill the DRAM cache at the same rate the host accesses it, consequently enabling highly effective hardware prefetching. Note that the marginal gap between the theoretical ideal bandwidth and our measured throughput is primarily attributed to the 3-stage metadata updating overhead discussed earlier. Ultimately, this design not only satisfies the metadata footprint constraints but also aligns the internal memory performance closely with the peak capabilities of the host interface (e.g., PCIe x8), facilitating consistent, high-speed data delivery.

While the previously described topology aligns internal data paths for consistent throughput, providing a specialized prefetch API at the user level enables direct, high-performance data paths that bypass hardware-level LRU management and associated tracking overhead. To leverage this, we provide a user-level library that exposes explicit software control over the internal prefetch engine. This interface allows applications to proactively stage data into the DRAM cache, effectively masking SSD latency by exploiting the deterministic access patterns of LLM workloads.

The library provides three primary APIs tailored to different data granularities. chm_prefetch(addr) targets a single 4 KB page and can optionally apply an alignment check to skip unaligned addresses and avoid redundant hardware commands. For structured data, chm_prefetch_object(ptr, size) automatically determines and prefetches the page range covering a specific memory object, such as a tensor. Finally, chm_prefetch_size(addr, count) reduces software overhead for large deterministic streams by fetching $N$ contiguous pages from a base address.

The total capacity of these staging buffers is configurable as a proportion of available host memory, offering a flexible trade-off between I/O resilience and memory footprint. While a larger staging buffer can better absorb transient burst traffic and protect the read-priority pipeline (Section 4.2), allocating extensive pinned memory naturally increases host-side resource overhead. This tunability allows ITME to adapt to diverse hardware environments, balancing robust I/O performance against specific system memory constraints.

This data organization is designed to exploit the high-bandwidth sequential write capabilities of RDMA and remote CXL-hybrid memory. Consequently, when these request-related blocks are later prefetched, they can be read as a large, contiguous stream, enabling peak sequential throughput from remote CXL-hybrid memory. While decode-phase blocks are naturally interleaved across concurrent requests, ITME uses a configurable block size ranging from 64 to 256 tokens. As shown in Figure 3, blocks at these scales (e.g., up to 32 MB (Cheng et al., 2025)) are sufficiently large to achieve reasonable I/O performance even when processed individually.

The primary role of ITME is to efficiently orchestrate KV cache movement across the multi-tier hierarchy, leveraging specific access patterns and device characteristics. It manages this entire lifecycle, from migrating blocks immediately upon GPU eviction to proactively prefetching them based on upcoming execution demands.

However, because the remote CXL-hybrid memory relies on NVMe SSDs, unregulated asynchronous writes can severely degrade prefetch performance. A critical bottleneck arises when background SSD writes, which flush these large sealed chunks, contend for internal resources with time-critical reads. Due to the internal architecture of NVMe SSDs, if the controller’s write buffer is saturated by bulk background flushes, it creates a massive I/O bottleneck that stalls the read queue (Kim et al., 2019), significantly increasing retrieval latency. This read/write contention throttles the effective read bandwidth, leading to unpredictable latency excursions as the NAND interface is saturated by ongoing write traffic.

To mitigate this bottleneck, ITME implements a read-priority I/O scheduling policy that heavily regulates background writes. The scheduler enforces absolute priority for read requests. Whenever a slot in the load ring buffer becomes available, triggering an immediate data fetch, the scheduler suppresses any pending writes. The I/O slot is yielded to the read operation, and the write is re-enqueued to the CPU staging buffer, which safely absorbs the I/O backpressure. Since evicted data can be reconstructed, writes do not require immediate persistence. This flexibility allows the system to defer or drop non-critical writes during I/O contention, providing subsequent opportunities to persist the blocks in following inference turns. Furthermore, while eviction data is managed collectively in large chunks, the scheduler issues the actual writes to the remote SSD in smaller, regulated units to prevent long-tail blocking.

Consequently, ITME strategically schedules these asynchronous writes during the decode phase. Although layer-wise weight reads and next-turn KV prefetching still occur during this time, the decode phase inherently provides valuable opportunistic I/O windows with relatively lower storage utilization. By interleaving the regulated write units into these available intervals, the SSD returns to a clean state before the next compute-intensive prefill phase begins. This ensures that eviction maintenance does not interfere with critical prefetching, allowing the storage tier to consistently deliver peak read bandwidth.

ITME implements differentiated fallback policies for cache misses. While a model weight miss inevitably triggers a pipeline stall, a KV cache miss is resolved through dynamic recomputation on the GPU. This design choice avoids the inefficiency of retrieving isolated missing blocks from the remote tier, which would necessitate either suboptimal chunk-sized transfers or complex retrieval logic. Consequently, recomputing the missing KV segments directly on the GPU is more performance-efficient than waiting for high-latency remote retrieval.

Once chunks are staged in the CPU staging buffer, the system balances a fundamental trade-off between block-wise and layer-wise transfer granularities. While a block-wise approach enables rapid slot reclamation through a single DMA operation, it serializes execution and leaves the GPU idle until the entire block transfer completes. To align with model weight streaming and maximize pipeline efficiency, ITME adopts a layer-wise prefetching strategy for the KV cache. This strategy enables a fine-grained pipeline where the DMA transfer of layer $L+1$ overlaps with the GPU computation of layer $L$, effectively minimizing GPU idle time. Although layer-wise access extends buffer occupancy, ITME mitigates this by decoupling the staging cache from the CXL-hybrid memory ring buffer, sustaining high-bandwidth sequential transfers while maximizing GPU utilization through seamless computation-communication overlap.

The ITME remote manager operates as the central orchestration layer on the CXL-hybrid memory server, specifically tasked with managing the CXL-hybrid memory device and coordinating the prefetching of KV cache chunks, model weights, and prefix caches to decouple SSD access latency from the inference critical path. When the host GPU server offloads data in coarse-grained chunks, the manager treats these transfers as structured memory objects and assigns sequential identifiers to each, establishing a direct mapping between the sequential execution of inference turns and the physical storage layout. For model weights, which remain static and are accessed repeatedly, the manager persists them in the remote tier during the initial setup phase to enable permanent reuse across multiple inference sessions without redundant transfers.

A key feature of the remote manager is its ability to trigger the device’s internal hardware prefetch engine via user-level prefetch APIs for CXL-hybrid memory (see Section 3.2). This proactive management is synchronized with the LLM inference cycle. During the decode phase of turn $N$, the manager identifies the data requirements for turn $N+1$. Since the prompt prefix for the subsequent turn is derived from the tokens generated in preceding turns, the manager retrieves the required KV cache chunks and cached prefix segments in the exact sequential order they were previously stored. In contrast, model weights are of uniform size and are managed through a circular buffer strategy. Once an RDMA read for a specific weight layer completes, the Remote Manager immediately overwrites that location by prefetching the next layers.

The manager issues asynchronous prefetch commands to the storage controller. An internal prefetch command queue enables a continuous stream of data migration from NAND flash to the CXL-DRAM cache. This pipelined approach stages all required data in DRAM before the host issues an RDMA Read request, effectively masking SSD access latencies.

Figure 7 illustrates the end-to-end data movement across the GPU, CPU, and CXL-hybrid memory hierarchy. The pipeline initiates as the GPU local KV cache reaches capacity, triggering an eviction of blocks such as $R5$ and $R0$ to the CPU store buffer via asynchronous DMA (➊). When these blocks are not immediately required, they are staged in the CPU staging buffer and aggregated into coarse-grained chunks (e.g., 512,MB) to facilitate high-bandwidth sequential writes. If the GPU requests recently evicted data while it still resides in this staging area, a store buffer hit occurs (➋), allowing ITME to function similarly to a conventional CPU-offload system by providing low-latency retrieval.

The read-priority I/O scheduling orchestrates the transfer of these chunks to the remote CXL-hybrid memory (➌). To prevent performance interference with latency-sensitive operations, the scheduler strategically throttles this write path during high-priority read phases. Instead, it utilizes the idle intervals of the decode phase to perform append-only updates (➍) to the CXL-hybrid memory. This prioritized approach allows the system to maximize read throughput for the inference critical path by deferring background data persistence to periods of low I/O contention.

As the next inference turn approaches, the multi-tier DMA prefetching initiates the read path. During the decode phase of the current turn, the remote manager identifies the required sequence of chunks generated in previous turns. The manager then invokes the device internal engine to perform a uhardware-level prefetch (➎), promoting the associated chunks into the internal DRAM cache. Simultaneously, the scheduler orchestrates a multi-tier transfer by initiating chunk-wise RDMA read operations (➏) to move the aggregated data from the CXL-hybrid memory to the CPU load ring buffer. Finally, ITME operates a pipelined transfer that proactively pushes required data to the GPU memory ahead of each execution step. Specifically, ITME employs layer-wise prefetching for both model weights and the prefix KV cache (➐, ➑). By overlapping these retrievals with ongoing GPU computation, ITME effectively masks the latency overhead associated with accessing the CXL-hybrid memory.

To demonstrate the advantages of ITME, we implement and evaluate it within a highly optimized version of vLLM (v0.17.0) (Kwon et al., 2023).

• •

  NVMe-oF Integration: We integrated an NVMe-oF (NVMe over Fabrics) stack to enable storage access. This allows the baseline to offload and retrieve KV cache blocks from a storage node, rather than relying solely on local CPU memory or costly recomputation.

• •

  Weight Prefetching: To ensure a fair comparison with our hardware-assisted prefetching, this baseline is configured to overlap weight transfers from host memory to GPU memory with active computation. This represents an optimized software-based offloading system that minimizes the performance impact of model weights.

The performance of ITME is evaluated using the ShareGPT dataset with a configuration of 128 concurrent conversations, each spanning up to 5 turns with a minimum of 2000 tokens per conversation. As memory pressure intensifies in later stages, we focus on Turns 3–5, where the total KV cache footprint reaches 40 GB.

Figure 9 illustrates the time to first token (TTFT) speedup for the Llama-3.1 8B and 70B models, normalized against a GPU memory (T1) baseline utilizing recomputation. To ensure a fair evaluation, both the baselines and ITME are configured with weight offloading and prefetching enabled. To establish the upper performance bound of our architecture, we include an ideal GPU memory configuration with 80 GB, where the entire KV cache footprint resides within the local high-bandwidth memory.

The experimental results characterize the performance positioning of ITME between high-cost local GPU memory and high-overhead recomputation. While the Ideal GPU memory configuration achieves a maximum speedup of $3.02\times$ due to its inherent bandwidth advantage, ITME reaches a $1.81\times$ speedup by turn 5. Although ITME is constrained by the latency of remote CXL-hybrid memory compared to local GPU memory, it offers a substantial performance gain over recomputation-based baselines. This improvement is realized whenever the system can fetch KV blocks from remote CXL-hybrid memory faster than the time required for recomputation. By successfully identifying access patterns and staging KV blocks in advance, the system minimizes the overhead of remote CXL-hybrid memory and maximizes inference efficiency.

Figure 10 compares the multi-turn inference performance of ITME against the CPU-offload and recompute-only baselines. The figure evaluates inference efficiency using 256 concurrent conversations from the ShareGPT dataset. To stress-test KV cache capacity, we designed a 35-turn benchmark that generates a massive total footprint. In this setup, the CPU-offload baseline allocates a large 128 GB host memory (T2) for caching. In contrast, ITME leverages the CXL-hybrid memory (T3.5) tier while utilizing only a fraction of host memory (T2) as a staging buffer. All results are normalized against a recomputation baseline to isolate the speedup enabled by ITME’s efficient multi-tier memory management.

During the initial turns (turns 1–9), ITME exhibits comparable performance to the CPU-offload baseline. This is because the active working set still fits within the host memory tier; thus, evicted KV blocks are served directly from the CPU staging buffer without incurring the additional latency of the remote CXL-hybrid memory. Regardless of the staging buffer size (30 GB vs. 128 GB), ITME’s performance closely aligns with the CPU-offload baseline. This indicates that their underlying memory access characteristics are fundamentally identical as long as the working set is served within the host tier, before remote eviction begins. However, as the conversation extends (turns 10–20), a performance gap emerges. While the CPU-offload baseline maintains a high prefix hit rate near 90%, ITME experiences a gradual decline. As the KV cache volume expands with each turn, the increasing read/write traffic triggers I/O contention within the CXL-hybrid memory. This prevents some data chunks from reaching the CPU staging buffer in time, forcing recomputation for the delayed blocks, in contrast to the stable CPU-offload baseline.

Beyond turn 21, the CPU-offload baseline completely exhausts its 128 GB memory. With no space left to store new KV blocks, its caching system collapses, forcing the hit rate to 0% and making it as slow as the recompute-only baseline. In contrast, ITME continues to function by leveraging its massive CXL-hybrid memory. While the ITME exhibits some performance fluctuations due to unpredictable I/O stalls during intense contention, our read-priority scheduling effectively mitigates these bottlenecks. Even under such pressure, retrieving data from the remote tier remains far more efficient than full recomputation. Consequently, ITME achieves up to a 35.7% throughput improvement over the CPU-offload baseline in these extended turns.

Figure 11 compares ITME against a local NVMe-oF baseline. While real-world NVMe-oF typically operates on DPU-based JBOF systems where optimization is challenging, we utilize a local configuration to eliminate network overhead. This setup represents an ideal NVMe-oF scenario, providing a high-performance upper-bound for comparison. Both systems utilize the same CPU staging buffer and perform all data transfers in fixed-size chunks to their specific backend (either local NVMe-oF or ITME). Consistent with our host memory scaling benchmarks, we utilize the same workload configuration for this comparison. To ensure a fair evaluation, both systems follow an identical caching policy: they serve KV blocks from the CPU staging buffer upon a hit, falling back to recomputation if the required data is not present in the staging buffer.

Initially, both systems exhibit identical performance as long as the KV cache remains within the 30 GB CPU staging buffer. Since requests are served directly via buffer hits, the underlying storage latency is effectively masked. However, starting from Turn 5, the performance of the Local NVMe-oF baseline degrades significantly, eventually converging with the recomputation baseline. The performance drop occurs because heavy write traffic blocks the necessary read requests. Without a way to effectively manage or delay these writes, the system cannot retrieve data while busy with incoming traffic. Furthermore, the absence of a dedicated prefetching mechanism prevents the system from hiding this delay by moving data in advance. As a result, the required KV blocks fail to reach the CPU in time, forcing the system to perform slow recomputation. While NVMe-oF could be further enhanced through specialized optimizations, our evaluation focuses on the intrinsic I/O efficiency of each backend using the same chunk-level data management. Within this scope, ITME demonstrates enhanced efficiency by effectively addressing I/O bottlenecks and latency through its integrated prefetching and CXL-hybrid memory architecture.

To isolate the impact of weight prefetching from prefix KV cache prefetching, we evaluate the performance of ITME by prefetching model weights only. We employ the Mooncake workload with a batch size of 16 and a 4K input sequence length. The primary objective of this evaluation is to demonstrate how closely ITME matches the host memory baseline, where all weights reside in host DRAM. Since the baseline represents the theoretical performance upper bound, this analysis focuses on the efficiency of ITME in hiding network and storage overheads through its pipelined prefetching.

Figure 12 (a) compares end-to-end performance results, where the gaps between baseline, local, and remote configurations quantify storage and network overheads, respectively. While the 10.5 GB footprint of the 8B model fits within the CXL-hybrid memory DRAM cache, the 105 GB footprint of the 70B model exceeds it, yet both maintain near-baseline performance with only 1–5% deltas. These results demonstrate that ITME effectively hides storage and network latencies. Figure 12 (b) evaluates the 70B model’s throughput while scaling the CXL-hybrid memory DRAM capacity. Our results show that a 4GB DRAM prefetch buffer, supporting a three-depth pipeline, incurs only an 8% performance degradation compared to the host memory baseline. While a 4 GB allocation offers a practical balance between speed and memory efficiency, it is not a complete solution. In practice, frequent model weight fetching also consumes significant read bandwidth, creating resource contention with KV cache retrieval. Therefore, achieving optimal performance requires not only proper buffer sizing but also effective scheduling to mitigate performance degradation during these simultaneous data transfers.

Figure 13 demonstrates the hardware feasibility of the ITME architecture through an FPGA-based prototype. While the CMM-based platform provides an evaluation of peak performance potential, this implementation serves to validate the hardware-level control logic and the effectiveness of the pattern-aware prefetcher in a real-world system environment.

Figure 14 compares the bandwidth of the ITME CMM configuration with its FPGA prototype implementation. The FPGA prototype achieves an average of 18 GB/s for reads and 12 GB/s for writes during DRAM cache hits, representing a 20–25% performance gap compared to the CMM-based evaluation due to hardware-level overheads. High-variance weight bandwidth is omitted for clarity, as it maintains a highly periodic and stable pattern. These results demonstrate the efficacy of our read-priority scheduling; since read operations are critical to inference latency, writes are dispatched exclusively during idle periods to minimize interference. Even if a write operation is dropped due to contention, the system ensures architectural reliability by recomputing and restoring the context in the subsequent turn. While the CMM setup remains resilient—sustaining 4–5 GB/s even under extreme contention—the current FPGA implementation’s bandwidth degrades below 1 GB/s upon cache misses. We are actively optimizing the hardware logic to close this gap and reach the theoretical maximum of 23.3 GB/s.