TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs

LLM inference proceeds in two phases: prefilling and token generation, which impose different computational workloads and hardware bottlenecks. In prefilling, the full input sequence is processed in parallel, so execution is dominated by matrix multiplication, or GEMM. $C=A\times B$, with $A\in\mathbb{R}^{M\times K}$, $B\in\mathbb{R}^{K\times N}$, and $C\in\mathbb{R}^{M\times N}$. Here, $M$ varies with sequence length, while $N$ and $K$ span a wide range across layers in edge LLMs. We measure prefilling performance using time to first token (TTFT).

In token generation, the model produces one token at a time while reusing the key/value (KV) cache from previous tokens. Because edge inference commonly runs at batch size 1 ($B=1$), many linear layers reduce to GEMV. We measure generation performance using time per output token (TPOT), or equivalently its inverse, tokens per second (tok/s).

To reduce model size and relieve memory bottlenecks, on-device LLMs commonly rely on quantization, typically denoted as wXaY, where X is the weight precision and Y is the activation precision. In this paper, we consider two widely used formats, $W4A16$ and $W8A16$, with primary emphasis on $W4A16$ because of its popularity in local LLM deployment.

Quantization maps high-precision values to low-bit integers using scale factors and, depending on the quantization scheme, zero-points. A key distinction is between symmetric and asymmetric quantization. In symmetric quantization, real values are mapped around zero, and the integer zero directly represents the real value zero. Therefore, dequantization only requires a scale factor:

where $\hat{w}$ is the dequantized weight, $q$ is the quantized value, and $s_{g}$ is the scale for group or channel $g$.

In asymmetric quantization, the quantized integer range is shifted by a zero-point, allowing the representable range to better match non-zero-centered weight distributions. Dequantization is given by

where $z_{g}$ is the zero-point. Asymmetric quantization can improve accuracy for low-bit formats, but it also introduces additional metadata and arithmetic during dequantization.

In the $W4A16$ weight-only format used by AWQ (Lin et al., 2023), activations are represented in FP16/BF16, while model weights are stored as packed 4-bit integers with per-group scales and zero-points. Accumulation is typically performed in FP32 to preserve accuracy. Weights are quantized group-wise, so all weights in a group share the same scale and zero-point, which are stored alongside the weights. In our setup, we use AWQ-style $W4A16$ per-group asymmetric quantization with group size 128.

The $W8A16$ format is similar: activations remain in FP16/BF16, weights are stored in INT8, and accumulation is typically performed in FP32. Because INT8 offers higher precision than INT4, $W8A16$ is often deployed in symmetric quantization form. Its quantization parameters are typically applied per channel, so each output channel of the weight matrix shares the same metadata. In our setup, we use BF16 activations with INT8 weights and per-channel symmetric quantization without zero-points.

To support local LLM inference under tight power and thermal budgets, edge SoCs increasingly integrate neural processing units (NPUs) alongside CPUs and iGPUs (AMD, ; Intel, ; Qualcomm, ; Apple, ). In principle, NPUs are intended to provide competitive inference performance at higher energy efficiency, but in practice their usefulness for modern LLMs is shaped less by raw compute alone than by their programming model. Across current vendors, NPUs are commonly exposed through proprietary or compiler-driven deployment stacks, such as Apple Core ML (Apple, 2026), Qualcomm QNN (Qualcomm Technologies, Inc., 2025), Intel OpenVINO (Intel Corporation, 2025), and vendor-managed runtime flows, which simplify deployment but expose limited low-level control over kernels, data movement, and execution policies. As a result, many of these stacks do not natively support practical open-source quantization formats such as AWQ (Lin et al., 2023). Instead, they often support only restricted subsets of models and quantization schemes, such as per-tensor, per-channel, or symmetric quantization (Yang et al., 2025), making it difficult to efficiently deploy the asymmetric per-group formats common in modern edge LLMs. Therefore, although NPUs are increasingly present in edge devices, many remain difficult to use as first-class LLM accelerators.

AMD integrates its NPU (AMD, ; Rico et al., 2024) into Ryzen AI SoCs for laptops, including Strix Point (AMD, 2025b) and Krackan Point (AMD, 2025a). XDNA2 is organized as a tiled spatial array with explicit dataflow and local memory. A typical XDNA2-class NPU consists of 32 AIE compute cores arranged in a $4\times 8$ layout, along with 8 memory cores for temporary storage and 8 shim cores that use buffer descriptors (BDs) for efficient data movement (Figure 1).

Unlike the more closed software stacks above, AMD also provides MLIR-AIE (IRON) (Hunhoff et al., 2025), an open-source, close-to-metal programming flow that exposes both microkernel design within each compute tile and explicit data movement across the array. This combination of a flexible spatial architecture and an open low-level software stack makes XDNA2 a particularly strong platform for kernel and dataflow research.

Although AMD’s XDNA2 NPU provides a programmable spatial architecture, efficient execution of quantized LLM workloads on this platform remains difficult. The challenge is not only to run low-bit kernels, but to do so in a way that matches practical LLM quantization formats and co-optimizes data movement and computation. In our target setting, these difficulties arise most prominently in transformer linear layers, which dominate the prefilling phase through large GEMMs and the token generation phase through GEMV-like operations. We identify four main design challenges.

Because model weights are known at deployment time, we perform offline pre-tiling on quantized weights with no runtime overhead. This pre-tiling step serves two purposes: it packs each weight tile with the metadata required for dequantization, and it reorganizes the weight layout to improve streaming efficiency under DMA BD constraints. As a result, pre-tiling enables both metadata-aware mixed-precision execution and efficient support for large MLP layers. The pre-tiling layout is illustrated in Figure 2 with three levels: Level 1 shows the layout within each $8\times 8$ weight block, Level 2 shows how blocks and metadata are packed within each $k\times n$ tile, and Level 3 shows how tiles are arranged across the full $K\times N$ weight matrix. Arrows indicate memory continuity between tiles/data, with the purple arrow representing continuity within the same row/column and blue arrows representing continuity when switching row/column.

Packing weights with quantization metadata. In a regular GEMM kernel, the input matrices are divided into small tiles for each compute core to process. Therefore, we pre-tile the weight data and pack metadata directly into each pre-tiled weight block in memory.

For $W4A16$, as shown in Figure 2b, each quantized weight tile is stored in a way such that all packed INT4 values of the tile are stored contiguously first, followed by the BF16 scales and INT8 zero-points for the quantization groups in that tile. This enables the compute core to fetch all required information from the weight stream. For example, a $128\times 64$ (128 rows and 64 columns) weight tile needs 64 scales and 64 zero-points for dequantization because with a group size of 128, every 128 weights in the same column share the same scale and zero-point. Therefore, we pack them together such that the $128\times 64$ INT4 weights are followed by $64$ BF16 scales, which are again followed by $128$ INT8 zero-points (duplicated 64 zero-points)²²2Because the buffer descriptors (BD) transfer data through DMA engines, we duplicate the zero-point array so that each tile payload size is divisible by 128 bytes, as required by the DMA hardware. The additional memory footprint is negligible, taking only 1.5% of extra memory.. In $W8A16$, symmetric INT8 weights only require scales to dequantize, which are appended in the same manner during pre-tiling. While some metadata may be replicated during pre-tiling, the extra metadata is negligible compared to the weight traffic saved by INT4/INT8 streaming.

Interleaved pre-tiling for large-matrix support. For large LLM layers, especially MLP projections, the weight matrix can exceed the stride limits supported by the runtime BD programming model. Because the AIE array has 8 compute-core columns, the output-tile columns are distributed across these 8 columns in a round-robin manner. For example, AIE column 0 processes output-tile columns $0,8,16,\ldots$, AIE column 1 processes output-tile columns $1,9,17,\ldots$, and so on. Therefore, for a given AIE column, consecutive tiles in its execution order are separated by 8 tile columns in the original row-major weight layout. This creates a large memory stride when the weight matrix is stored in the default layout. To eliminate this large-stride access, we introduce an interleaved column-major pre-tiling layout that places tiles assigned to the same AIE column consecutively in memory, as illustrated in Figures 2c and 2d. As a result, the BD can access data contiguously without large strides in memory, enabling the kernel to support matrices with dimensions $K$ and $N$ up to 32k without losing performance.

Tiled layout for bandwidth-efficient streaming. Beyond metadata packing and large-matrix support, pre-tiling alleviates the memory bandwidth bottleneck by converting the original row-major weight matrix into a tile-major layout, matching how the hardware kernel consumes the input. This enables DMA engines to issue long contiguous reads and stream weights into the NPU, which better utilizes DRAM burst transfers and reduces wasted bandwidth. Avoiding tiling during runtime also helps maintain consistent performance across different GEMM shapes, as shown in Section 5.

Given the pre-tiled weight layout, we implement mixed-precision microkernels that fuse unpacking, dequantization, and GEMM/GEMV inside each compute core. The goal is to perform on-the-fly dequantization using registers and local memory while preserving throughput.

GEMM microkernel. Figure 3 illustrates the fused mixed-precision microkernel flow. AIE matrix-multiply primitives are optimized for BF16 $8\times 8$ blocks, so we divide the $m\times k$ activation tile and the $k\times n$ weight tile into $8\times 8$ sub-tiles (Figure 2a) and accumulate partial results into accumulators. For $W4A16$, the input weight stream contains packed INT4 values, with two 4-bit weights stored in each byte. As shown in Figure 3, the microkernel first performs unpacking: it extracts the low and high 4-bit fields using bit masking and shifting, and widens them into INT8 lanes so that each quantized weight can be processed as an individual integer value. This unpacking step does not yet recover the numerical BF16 weight; it only converts the packed storage format into an integer vector format suitable for arithmetic.

After unpacking, the widened INT8-represented quantized values are converted to BF16 and dequantized using the corresponding scale and zero-point metadata following Eq. (2). For symmetric $W8A16$, the INT8 weight value is directly converted to BF16 and multiplied by the per-channel BF16 scale, without zero-point subtraction, following Eq. (1). The resulting BF16 weights are stored in a compute-core local buffer and then consumed by the AIE matrix-multiply primitives for GEMM/GEMV operations.

A naive fusion approach would be to apply unpacking and dequantization every time an $8\times 8$ weight block is loaded from the input. However, the same weight block is reused across multiple activation slices, so dequantizing on every load is redundant and significantly reduces microkernel throughput. To avoid redundant dequantization, we leverage the local memory within each compute core: the kernel unpacks and dequantizes each quantized weight block once, stores the resulting BF16 weights into the local buffer, and reuses them for subsequent GEMM computation. Although this method introduces extra load and store operations because of the use of a local memory buffer, the VLIW instructions can hide most of the memory overhead by overlapping memory operations with dequantization and GEMM computation (Advanced Micro Devices, Inc., 2025b).

GEMV microkernel. We apply the same fusion strategy to GEMV and implement mixed-precision GEMV microkernels that directly consume quantized weights. However, GEMV requires additional optimization to improve compute utilization. Compared with full-precision GEMV, transmitting quantized weights leads to less bandwidth demand, while on-the-fly dequantization increases compute work. This combination shifts GEMV from being primarily memory-bound to being more compute-bound, making arithmetic throughput within the microkernel critical.

The baseline GEMV microkernel does not fully utilize the compute core’s arithmetic resources: it repeatedly loads $16\times 8$ weight blocks (two 8x8 weight blocks), and uses vector operations with activation fragments to accumulate only $16$ output elements per iteration, which underutilizes the vector hardware for GEMV. Since a vector register can hold up to $64$ BF16 values (Advanced Micro Devices, Inc., 2025a), we load a $64\times 8$ weight block (eight 8x8 weight blocks) into these registers per iteration from local buffer to perform GEMV with the $1\times 8$ activations, accumulating $64$ output elements per iteration. The $1\times 8$ activation fragment corresponds to one slice of the input vector along the $K$ dimension. This same activation fragment is reused across $64$ different weight rows, each of which corresponds to a different output element. Therefore, multiplying a $64\times 8$ weight block with a shared $1\times 8$ activation fragment produces $64$ partial output accumulations. As the microkernel iterates over the $K$ dimension, these partial sums are accumulated into a $64\times 1$ output vector. By widening the inner microkernel to increase work per instruction and reuse each activation fragment across more output rows, we improve per-core utilization and throughput for token generation workloads.

As discussed in Section 4.3, switching from full-precision to mixed-precision reduces weight bandwidth demand but adds unpacking and dequantization work, making GEMV more compute-bound rather than memory-bound. However, the baseline data movement pattern still feeds only one row of compute cores, leaving most of the $4\times 8$ AIE array idle. We therefore redesign the GEMV dataflow, optimizing the spatial movement of input data through shim cores, memory cores, and compute cores.

In the baseline GEMV mapping shown in Figure 4a, each shim column streams weights directly to the compute cores in one compute row. This direct streaming path is simple, but it only uses $8$ of the $32$ compute cores. Extending this direct path to all four compute rows is difficult because GEMV requires each compute core to consume a distinct weight tile. The same activation vector can be broadcast to all compute cores, but each compute core must receive a different $n\times k$ weight tile to compute distinct output elements. Since each AIE column has only one shim core connected to external memory, the shim core cannot naturally provide four independent weight streams to the four compute rows in the same column.

Our optimized mapping in Figure 4b uses the memory core as an intermediate distribution layer. The weight-delivery dataflow has two stages. In the first stage, each shim core streams a contiguous bundle of four weight tiles from external memory into its associated memory core, where the four tiles correspond to the four compute rows in that AIE column. In the second stage, the memory core distributes these four tiles to the four compute cores in the column. The activation vector is shared across the array, while each compute core receives a distinct weight tile and computes a distinct portion of the output vector.

Importantly, intermediate GEMV accumulation results do not flow through these stages. Each compute core performs unpacking, dequantization, multiplication, and accumulation locally for its assigned output elements. The memory core is used only for input weight distribution, not for reducing partial sums across compute cores. Therefore, by using memory cores as an intermediate stage for weight distribution, this new design increases spatial utilization: instead of processing $8$ output-tile groups at a time using one compute row, the optimized scheme processes $32$ output-tile groups at a time using all four compute rows. Although the memory-core distribution stage adds local data movement, this overhead is offset by the increased compute parallelism and is evaluated in the GEMV ablation study in Section 5.5.

A central reason TileFuse can support practical quantized LLM workloads is that XDNA2 exposes configurable control over both compute and data movement. In this work, reconfigurability does not mean changing the hardware itself or dynamically changing the routing fabric while a kernel is running. Instead, it refers to configuring the AIE array with workload-specific compute-tile programs, memory layouts, DMA buffer-descriptor access patterns, synchronization objects, and inter-core data-movement paths generated by the IRON/MLIR-AIE flow. TileFuse leverages this flexibility at three levels.

First, at the compute-core level, we replace a fixed full-precision GEMM/GEMV datapath with mixed-precision microkernels that fuse unpacking, dequantization, and matrix computation. This compute-tile programmability allows the core to directly consume compressed INT4/INT8 weights, convert them to BF16 locally, and reuse the dequantized weights through local memory. Without the ability to customize the microkernel, the implementation would have to materialize BF16 weights before invoking a standard GEMM/GEMV kernel.

Second, at the memory-layout and DMA level, we use offline pre-tiling to match the runtime access pattern of the AIE array. The interleaved column-major layout changes the physical order of weight tiles so that tiles assigned to the same AIE column become contiguous in memory. This allows the BD-based DMA path to stream large GEMM weights efficiently without large strides or repeated kernel invocations. In this sense, TileFuse uses XDNA2 configurability not only for arithmetic, but also for adapting memory movement to the shape of LLM weight matrices.

Third, at the array level, we configure a GEMV-specific weight-distribution path through memory cores. The baseline direct streaming pattern uses only one compute row, but token-generation GEMV requires each compute core to consume distinct weight tiles. By routing bundled weight tiles from shim cores through memory cores to multiple compute rows, TileFuse maps GEMV across the full $4\times 8$ compute array and improves spatial utilization. This is the main optimization in TileFuse that relies on array-level data-movement configuration.

These benefits come with an important limitation. NPU kernel dispatch and AIE configuration introduce runtime overhead, including the cost of launching a precompiled kernel and activating its compute programs, DMA descriptors, synchronization objects, and data-movement connections. This overhead is amortized well by large prefilling GEMMs but becomes costly for short GEMV-like operators during token generation, as discussed in the evaluation and discussion sections. Therefore, TileFuse uses XDNA2 reconfigurability where it provides clear benefits—custom fused kernels, large-matrix streaming, and full-array GEMV mapping—while the evaluation also shows that this flexibility is less favorable for latency-sensitive small operators.

To evaluate our kernels in real LLMs for both correctness and performance, we built a custom framework in C++ with Python bindings. The Python frontend loads model weights from Hugging Face safetensors, applies our offline weight pre-tiling and metadata packing, and handles tokenization and output decoding. The libtorch-based C++ backend implements the transformer execution loop and dispatches operators to the appropriate device.

In each transformer block, linear layers such as attention projections and MLP projections are dispatched to the NPU when using the TileFuse path. These linear operators call our precompiled XDNA2 kernels with pre-tiled quantized weights and BF16 activations. Nonlinear and attention-related operators, including FlashAttention-2 (Dao, 2023), softmax, residual addition, normalization, and other lightweight element-wise operations, remain on the iGPU using optimized HIP/libtorch kernels. This division follows the compute characteristics of the devices: large GEMM operators can amortize NPU dispatch and reconfiguration overhead, while latency-sensitive and irregular operators are better handled by the iGPU software stack. We adopt this hybrid design based on the compute characteristics of XDNA2 (Rico et al., 2024) and our profiling of transformer attention blocks.

Our current implementation does not execute the NPU and iGPU concurrently on different operators. Instead, execution proceeds sequentially according to the transformer data dependencies: the host dispatches an operator to either the NPU or the iGPU, waits for its result, and then launches the next dependent operator. Therefore, the hybrid setup should be interpreted as operator-level device selection rather than parallel NPU–iGPU co-execution. We adopt this design to isolate the benefit of TileFuse for quantized linear layers while relying on mature GPU kernels for the remaining transformer operations.

To evaluate kernel performance and shape sensitivity, we profile our $W4A16$ AWQ (Lin et al., 2023) and $W8A16$ LLM.int8 (Dettmers et al., 2022)-style kernels, using group size 128 and per-channel quantization, respectively. Using standard GEMM notation with $A\in\mathbb{R}^{M\times K}$ and $B\in\mathbb{R}^{K\times N}$, we evaluate two configurations with transposed $K$ and $N$ dimensions while sweeping $M$ in steps of 256.

We evaluate TileFuse on two Ryzen AI platforms: Ryzen AI 7 350 and Ryzen AI 9 HX 370. Both platforms include XDNA2-class NPUs and have similar nominal peak memory bandwidth, up to approximately 128 GB/s under comparable high-speed memory configurations (AMD, 2025a, b). However, their iGPU configurations differ substantially: the Ryzen AI 9 HX 370 integrates Radeon 890M graphics with 16 graphics cores, while the Ryzen AI 7 350-class platform uses Radeon 860M graphics with 8 graphics cores (AMD, 2025b, a). Therefore, the HX 370 provides a stronger iGPU baseline primarily because of its larger iGPU, while the AI 7 350 offers a more informative comparison for isolating the benefit of NPU offloading against a smaller integrated GPU.

As a state-of-the-art baseline, we use the fused quantized HIP kernels from llama.cpp (Gerganov and contributors, 2023) on the iGPU. These were the highest-performing quantized GEMM kernels we found for AMD Ryzen iGPUs as of January 2026, especially after a recent update that improved performance by up to 1.5$\times$ over 2025 releases. We compare against the Q4_K and Q8_0 formats.³³3Q4_K and Q8_0 are native llama.cpp block-quantized weight formats: Q4_K stores weights at 4 bits per value using the K-quant layout with additional shared scale/offset metadata, while Q8_0 stores weights at 8 bits per value with block-level scaling. In both cases, llama.cpp fuses dequantization with the HIP matrix kernels. On RDNA iGPUs, these kernels can exploit $dp4a$ instructions (41), which are GPU vector dot-product instructions that accumulate multiple low-precision integer products per instruction. This makes the quantized path more compute- and bandwidth-efficient than executing the same operation through unfused floating-point kernels.

Figure 5 shows that our NPU kernels produce stable performance across shapes on XDNA2. Some accelerators exhibit strong shape sensitivity for GEMM because different matrix dimensions can map unevenly onto the available execution resources. For example, when the number of tiles or thread blocks does not align well with the number of compute units, some resources may remain underutilized, leading to zig-zag throughput as the matrix dimension changes. In contrast to zig-zag patterns such as the untuned iGPU example shown in Figure 6 and the more severe TOPS degradation reported for Snapdragon Gen 3 (Chen et al., 2025),⁴⁴4In the cited Snapdragon Gen 3 study, GEMMs with orientation $([4096,14336]\times[14336,K])^{T}$ fall to roughly the 1-TOPS range, while the flipped orientation reaches roughly the 6-TOPS range despite comparable arithmetic work. we observe smooth scaling, which we attribute to our custom weight layout that maintains efficient data streaming under different sizes. Krackan Point and Strix Point reach peak throughput of 12 TOPs and 9 TOPs, respectively, although Strix Point shows lower throughput for the less compact $W8A16$ kernels. Given the shared XDNA2 architecture, this gap likely reflects platform-level factors such as silicon quality, thermal design, or effective NPU-side memory behavior rather than compute scaling alone.

For comparison, iGPU throughput scales roughly with the number of compute units (CUs), the primary parallel execution blocks in AMD RDNA GPUs. The 8-CU Krackan Point iGPU reaches about 5 TOPS, while the 16-CU Strix Point iGPU reaches about 7 TOPS; in contrast, CPU throughput remains marginal at roughly 200 GOPS. We also observe periodic throughput spikes for some GEMM shapes on the iGPU, which likely stem from inefficient thread-to-CU mapping and shape-dependent occupancy effects.

Finally, results for General Matrix-Vector Multiplication (GEMV) (Table 1) show consistent performance across various CPU and iGPU configurations, with the CPU being more performant and close to the results derived by the theoretical DRAM bandwidth limit. In contrast, the NPU is optimized for compute rather than memory bandwidth, so its GEMV performance is lower; Krackan Point only slightly outperforms Strix Point on these tasks for the NPU.

With our custom end-to-end framework setup introduced in Section 4.6, we evaluate the end-to-end performance. Note that the Ryzen AI 7 350 provides a more informative comparison for isolating NPU-side benefits because its iGPU baseline is substantially smaller, while the HX 370 has a stronger iGPU that can partially mask NPU-side gains in end-to-end comparisons. The end-to-end performance of the evaluated LLMs is shown in Figure 7 and Figure 8. Full request latency can be estimated as $L=l_{p}+n\cdot l_{g}$, where $l_{p}$ is prefilling latency, $l_{g}$ is per-token generation latency (the inverse of tok/s), and $n$ is the number of generated output tokens. Figure 7 reports prefilling latency across prompt lengths, while Figure 8 reports generation throughput. We further break down prefilling latency for both $W4A16$ and $W8A16$ in Table 2.

A clear pattern emerges for $W4A16$. On the Ryzen AI 7 350, our NPU implementation matches or outperforms the llama.cpp iGPU baseline for medium and long prompts, reaching up to 2.0$\times$ speedup for Llama3-8B and 1.7$\times$ for Gemma-2B and Qwen2.5-3B. For short prompts, the iGPU generally delivers lower latency because fixed NPU overheads from kernel dispatch and fabric reconfiguration dominate small matrix shapes. As prompt length increases, these costs are amortized and the NPU advantage becomes more visible.

The $W8A16$ results show a weaker trend. On the Ryzen AI 7 350, $W8A16$ still provides speedups over the iGPU for many prompt lengths, but the gains are smaller than $W4A16$; for example, Llama3-8B reaches 1.82$\times$ at 8192 tokens compared with 1.90$\times$ for $W4A16$, while Gemma-2B and Qwen2.5-3B show more modest speedups. On the Ryzen AI 9 HX 370, however, the $W8A16$ NPU path is slower than the iGPU baseline across the measured models and prompt lengths, consistent with the kernel results in the previous section, where the less compact $W8A16$ kernels have lower NPU throughput on the Ryzen AI 9 HX 370. At the same time, the llama.cpp iGPU baseline does not slow down proportionally when moving from Q4_K to Q8_0; in several cases, Q8_0 is similar to or slightly faster than Q4_K. We attribute this to llama.cpp’s block-wise fused dequantization: although Q8_0 moves more weight data, its dequantization path is simpler than the more complex 4-bit K-quant layout, reducing some of the per-block unpacking and scaling work inside the fused GPU kernel.

Comparing the Ryzen AI 7 350 and Ryzen AI 9 HX 370 reveals an important system-level tradeoff. Although the 350 shows stronger standalone $W4A16$ NPU kernel performance, its end-to-end prefilling latency remains close to the HX 370. We attribute this to the 350’s weaker iGPU, which bottlenecks FlashAttention-2 and offsets its NPU advantage. By contrast, the HX 370 benefits from a 16-CU iGPU that compensates for its weaker NPU kernel performance, leading to similar final prefilling latency across the two devices for $W4A16$.

Token generation shows the opposite trend in Figure 8. Here, the NPU underperforms the iGPU because generation is fundamentally latency-bound: GEMV operators execute in hundreds of microseconds, but NPU driver and reconfiguration overheads remain in the millisecond range. We also observe mapping inefficiencies for models such as Qwen2.5-3B, whose MLP hidden dimension is not a multiple of our minimum tile-size. The resulting padding increases over-computation and causes the NPU to fall behind the iGPU in latency.

We measure platform current during LLM inference using a lab bench power supply, and use the resulting electrical traces to compare system energy and instantaneous current draw across execution modes (OWON Technology, 2026). The resulting traces are shown in Figure 9. The figure reports Llama3-8B prefilling with a 4096-token prompt; the same qualitative trend also holds across the other evaluated models and prompt lengths, which are omitted here for brevity. Although the NPU path offloads only GEMMs to the NPU while leaving the remaining nonlinear operations on the GPU, it still substantially reduces total energy. On the Ryzen AI 7 350, the NPU path lowers energy consumption by 64.6%, largely because it completes the workload both faster and more efficiently. On the Ryzen AI 9 HX 370, where execution times are closer, the NPU path still reduces total energy by 52.8% due to its higher efficiency. In both cases, moving prefilling to the NPU also lowers the instantaneous current draw throughout most of execution.

In Table 3, we compare our results against recent NPU-driven inference frameworks. Direct apples-to-apples comparison is difficult because platforms differ in hardware, power limits, and quantization schemes. To reduce this mismatch, we choose baselines aligned with our target model families and 4-bit setting, and report results on the Ryzen AI 7 350, which performed best in our evaluation. The results show that our fused mixed-precision XDNA2 framework is highly competitive within the current edge-AI landscape.

Figure 10 and Figure 11 isolate the contribution of each optimization. We also use this ablation to quantify the benefit of leveraging XDNA2 reconfigurability. Since we cannot physically disable the reconfigurability of the AIE array, we define the baseline as the closest fixed-kernel counterpart: standard full-precision (BF16) GEMM/GEMV kernels with the original weight layout and data movement pattern, without the workload-specific optimizations introduced in this paper. Each added optimization represents one way TileFuse exploits reconfigurability: pre-tiling adapts the memory layout and DMA access pattern, mixed-precision kernels customize the compute-core datapath for fused unpacking and dequantization, and the GEMV microkernel/dataflow optimizations change the per-core work granularity and array-level weight distribution.

For GEMM, the fixed baseline does not support output dimensions above 8K, whereas our mixed-precision kernels with the interleaved column-major layout scale to 32K without a noticeable performance drop. The baseline also degrades slightly as GEMM size increases, consistent with inefficient weight streaming from DRAM under large strides. Applying pre-tiling alone improves the full-precision baseline by up to 50%, showing the benefit of adapting the memory layout and DMA streaming pattern to the AIE array. Building on this layout, mixed-precision kernels outperform the fixed baseline by roughly 50% even at smaller sizes and by up to 121.6% at larger sizes, demonstrating the additional gain from customizing the compute-core datapath to directly consume quantized weights.

Between the two mixed-precision formats, the faster choice depends on the balance between weight-streaming bandwidth and dequantization overhead. $W4A16$ transfers fewer weight bytes, but it requires packed INT4 unpacking and asymmetric zero-point handling. In contrast, $W8A16$ transfers more data, but uses symmetric INT8 weights with a simpler dequantization path. For GEMMs with smaller $M$, each streamed weight tile is reused across fewer activation rows, resulting in lower arithmetic intensity and making weight-traffic reduction more important. As $M$ increases, weight reuse and arithmetic intensity improve, so the bandwidth advantage of $W4A16$ becomes less dominant and dequantization overhead can have a larger impact on the relative performance of $W4A16$ and $W8A16$. Platform-specific memory behavior can also affect the relative performance, as discussed in Section 6.3.

For GEMV, pre-tiling alone improves the full-precision baseline by more than 20%, while fused $W4A16$ further reduces memory pressure by transmitting INT4 rather than BF16. However, its benefit remains modest at roughly 20%–40% without further optimization because unpacking and dequantization introduce extra compute. The microkernel and dataflow optimizations primarily target computation throughput, so they offer little benefit on the original memory-bound baseline. Once mixed precision shifts GEMV toward a more compute-bound regime, however, they become effective. Combining mixed precision with both the optimized microkernel and the 2D weight-distribution dataflow yields up to 281% improvement over the fixed baseline, showing the benefit of using XDNA2 reconfigurability not only for the compute-core datapath, but also for array-level data movement, as shown in Figure 11a.

TileFuse generalizes in two important ways. First, on AMD Ryzen AI systems, it enables widely used AWQ-style models to run natively on the NPU, rather than requiring the model to be reshaped around an NPU-specific quantization format. This makes XDNA2 a more practical target for off-the-shelf edge LLM deployment and, because XDNA2 is deployed across current-generation Ryzen AI SoCs, gives the broader Ryzen AI family a reusable path toward more accessible NPU acceleration. Second, while our specific microkernels and dataflow optimizations depend on XDNA2’s reconfigurable spatial array and close-to-metal control, the underlying pre-tiling idea is more portable. Offline reorganization of AWQ weights to match hardware consumption patterns can benefit other GPUs and NPUs as well, although the exact layout must be adapted to the architecture and programming model of the target platform.

XDNA2 offers substantial reconfigurability, enabling kernel- and dataflow-level optimization across diverse GEMM shapes, but this flexibility comes with non-trivial runtime overhead from kernel dispatch and fabric reconfiguration. During prefilling, where GEMM operators often run for tens of milliseconds, this overhead is largely amortized; during token generation, however, batch-1 GEMV operators often execute in only hundreds of microseconds, making runtime overhead a dominant cost and explaining why the NPU underperforms the iGPU for token generation despite TileFuse’s kernel-level optimizations.

These results point to a practical hybrid deployment strategy for interactive edge inference: the NPU is best suited for long-context prefilling, where large GEMM workloads can exploit its reconfigurable spatial array efficiently, while the iGPU remains a better target for latency-sensitive token generation. Our current implementation already moves in this direction by offloading attention and miscellaneous operations to the iGPU, and because AMD Ryzen SoCs use unified memory across the CPU, iGPU, and NPU, such hybrid execution does not introduce significant coherence or data-movement overhead. Extending this approach, a system that performs prefilling on the NPU and token generation on the iGPU could combine the prefilling gains in Figure 7 with the higher generation throughput in Figure 8.

Krackan Point machines (Ryzen AI 7 350) are positioned lower than Strix Point machines (Ryzen AI 9 HX 370). However, our kernel, end-to-end, and ablation results show that the Ryzen AI 7 350 achieves higher NPU performance than the Ryzen AI 9 HX 370 in several cases, especially after pre-tiling is applied, as shown in Figure 5, Figure 10 and Figure 11.

This behavior is not fully explained by public platform specifications. Both processors support similar memory configurations, so the difference is unlikely to come from nominal peak DRAM bandwidth alone. Instead, the results suggest that the two platforms may differ in effective NPU-side streaming bandwidth, memory-path overhead, power/thermal behavior, or runtime scheduling overhead. AMD does not publicly document the detailed NPU memory-path differences needed to isolate the exact cause, so we treat this as an empirical platform-level observation.

The format-dependent behavior in Figure 5 is consistent with this interpretation. Although $W4A16$ transfers fewer weight bytes than $W8A16$, it also has a more complex dequantization path because packed INT4 weights require unpacking and asymmetric zero-point handling. By contrast, $W8A16$ transfers more data but uses symmetric INT8 weights with simpler dequantization. Therefore, $W4A16$ is not guaranteed to always outperform $W8A16$: when the platform can stream the larger INT8 weight tiles efficiently, the lower dequantization overhead of $W8A16$ can make it competitive or even faster. This appears in several Ryzen AI 7 350 results. On the Ryzen AI 9 HX 370, however, the larger $W8A16$ data movement appears to be more costly, causing $W8A16$ to fall behind $W4A16$ despite its simpler arithmetic.

Overall, these results suggest that the observed performance difference between Strix Point and Krackan Point comes from the interaction between data layout, effective NPU-side streaming behavior, dequantization overhead, and platform-specific runtime behavior, rather than from nominal DRAM bandwidth alone. This also explains why pre-tiling is particularly important: by improving memory contiguity and reducing inefficient DMA access patterns, it allows platforms with better effective streaming behavior to benefit more from the optimized layout.