On the Limits of Performance Portability in Directive-Based GPU Programming

High-performance computing systems have become increasingly heterogeneous, with GPUs driving progress toward exascale computing ($10^{18}$ calculations per second): over 50% of TOP500 systems now integrate accelerators to maximize throughput and energy efficiency ¹¹1https://www.top500.org/lists/top500/2025/11/. While GPUs have opened new frontiers in scientific simulations, legacy applications face significant portability challenges, reflecting the long-standing trade-off between performance, portability, and productivity (Owens and others, 2008; Edwards and Trott, 2013; Heroux and Willenbring, 2009). Several EuroHPC Centers of Excellence across scientific domains - including SPACE²²2https://www.space-coe.eu/ (Shukla and others, 2026, 2025), Plasma-PEPSC³³3https://plasma-pepsc.eu/ (Williams and others, 2024), MaX⁴⁴4https://www.max-centre.eu/ (Garcia and others, 2025), and ChEESE-2P⁵⁵5https://www.cheese2.eu/ (Folch and others, 2023) - support re-engineering flagship codes for exascale performance and portable HPC best practices. Transitioning large legacy scientific codes to GPUs raises fundamental questions: (i) Can they be ported with minimal effort? (ii) How can reproducibility be preserved across evolving hardware? (iii) How can performance portability be achieved without sacrificing efficiency?

Exploiting modern GPU memory bandwidths (2-3 TB/s) requires careful parallelism exposure and memory-access management. Low-level APIs (CUDA, HIP) promise peak performance but lack portability. C++ abstraction libraries (SYCL, RAJA, Kokkos) provide portability through templates but require extensive code modifications (Davis and others, 2025). Directive-based models (OpenACC (OpenACC-Standard.org, 2023), OpenMP (OpenMP Architecture Review Board, 2021)) improve portability by abstracting device details, though often with performance penalties (Memeti et al., 2017; Khalilov and Timoveev, 2021; Krishnasamy and others, 2026): OpenACC benefits from a mature NVIDIA CUDA backend, while OpenMP offloading shows uneven optimization across compilers and architectures (Antao and others, 2016; Deakin and others, 2020). While quantitative portability metrics exist (Pennycook et al., 2016; Sewall et al., 2020; Holmen et al., 2019; Marowka, 2025), applying them to multi-kernel production scientific applications remains challenging.

This work focuses on directive-based models due to their widespread adoption in legacy MHD-Godunov (Stone et al., 2020; Grete et al., 2021) and particle-in-cell (Sishtla and others, 2019; Myers and others, 2021) codes. Large-scale astrophysical and plasma codes exemplify how memory latency, compiler behavior, and data layout interact to determine achievable performance on modern accelerators. In particular, this paper examines performance portability limitations of directive-based GPU codes by analyzing the porting of gPLUTO ⁶⁶6https://plutocode.ph.unito.it/, a production-grade MHD code for astrophysical plasma simulations, from OpenACC to OpenMP. We hypothesize that such limitations arise primarily from compiler implementation differences rather than programming model semantics, though architectural characteristics (memory hierarchy organization and sensitivity to strided memory accesses) play a crucial role. To test this, we perform kernel-level analysis of throughput, arithmetic intensity, instruction-level parallelism, register usage, and cache efficiency across GPU architectures and compiler toolchains, exploring performance portability challenges and trade-offs for a real application memory-latency dominated (Wienke et al., 2012; Deakin et al., 2018; Tandon and others, 2024; Dubey and others, 2021; Mehta et al., 2021).

The remainder of this paper is organized as follows. Section 2 describes the methodology and performance metrics used throughout the study, as well as the practical challenges encountered during the porting process. Section 3 presents application-level and kernel-level results for gPLUTO on NVIDIA A100 and AMD MI250X GPUs. Section 4 analyzes cross-architecture portability. Finally, Section 5 summarizes the main findings and discusses their implications for future GPU software development. This study operates under unified memory on both platforms (NVIDIA Managed Memory on A100, AMD Unified Memory on MI250X), as required for stable execution of gPLUTO’s pointer-intensive C++ codebase. While this introduces vendor-specific runtime behaviors (page migration, fault handling), our analysis focuses on programming model semantics and compiler-generated code; disentangling memory management from compiler and architectural effects remains future work.

This section describes the experimental methodology adopted in this work. We present the targeted HPC clusters, as well as the production application gPLUTO, the adopted strategies implemented to optimize its OpenMP translation from OpenACC, the profiling tools, and the performance and execution metrics employed throughout the analysis.

In the following sections OpenACC on NVHPC represents the most mature implementation and establishes performance upper bounds. Kernel metrics represent median or average values from steady-state time steps, excluding initialization overheads. Cross-architecture comparisons use the same OpenMP implementation to isolate architectural effects.

Cross-architecture analysis reveals substantial MI250X performance deterioration despite identical OpenMP code, as shown by total execution times in Figure 1. While both architectures sustain non-negligible HBM traffic, MI250X converts it into lower FP64 throughput due to poor memory coalescing under strided access. The much smaller L1/L2 cache (16 KB vector L1 per CU, 8 MB L2 vs. 192 KB L1 and 40 MB L2 on A100) amplifies non-unit-stride penalties, consistently with the low hit rates and VMEM under-utilization in Table 5. The HLLD() 3.7$\times$ directional anisotropy on MI250X (vs. $\approx$20% on A100) exemplifies this: strided $j,k$ sweeps bypass L1 with hit rate drops, collapsing VALU utilization and increasing VMEM stalls. Application-level performance is dictated by these latency-sensitive kernels: large FLOP/s gains ($16.5\times$ vs. A100) and high VALU utilization in Reconstruct() do not compensate previous latencies, resulting in a 3$\times$ slowdown shown in Figure 1, illustrating why aggregate throughput metrics obscure real bottlenecks.¹²¹²12Kernel-level profiling (Tables 2 and 5) reports single-invocation timings under isolated conditions, while Figure 1 aggregates several kernel launches per time step across spatial directions, RK stages, and auxiliary routines. Absolute times are not directly comparable.

Other metrics correlate with this behavior. Occupancy alone cannot hide latency without coalescing: enabled by its larger CU-level register file, low VMEM utilization and modest L2 hit rates indicate frequent wavefront stalls rather than instruction bottlenecks. Although IPC is not directly comparable across architectures, it provides a qualitative indicator that confirms this behavior. High branch efficiency on both platforms rules out divergence as a cause, confirming that memory coalescing and latency hiding drive the MI250X slowdown. Table 6 summarizes how C++ wrapper abstractions (Ary1D/Ary2D) affect performance across regimes. In 2D kernels, OpenMP’s conservative pointer handling triggers massive spilling and suboptimal grid mapping (47$\times$ slowdown). In 3D, higher parallelism hides latency, recovering near-parity (6% gap) with OpenACC through the unified CUDA backend. On MI250X, the larger CU-level register file mitigates spilling, shifting the bottleneck to memory coalescing, further amplified by the relative immaturity of OpenMP offloading on ROCm (AOCC/Clang) backend for allocation/alias analysis. Recovering A100-level performance therefore requires loop reordering, data-layout transformations, and direction-specific specialization for register management. Such interventions are well beyond straightforward pragma translation.

These effects are particularly visible in smaller kernels on A100 as well (Section 3.1.1), where OpenMP overheads (mapping, team setup, privatization) combine with complex C++ wrappers to trigger excessive register pressure and spilling. In the 3D Orszag–Tang problem, higher parallelism hides spilling latency, achieving near-parity despite moderate register pressure in individual kernels. Notably, the observed occupancy range reflects architectural and compiler differences rather than insufficient parallelism. An alternative approach to mitigate register pressure would involve explicitly constraining register usage through compiler flags (e.g., -maxrregcount on NVHPC) to increase occupancy at the cost of additional spilling. However, optimal limits are architecture- and kernel-specific, undermining portability. We therefore prefer algorithmic transformations that reduce intrinsic register pressure. Finally, an additional factor is unified memory on MI250X, whose automatic page migration may amplify strided-access overhead. Isolating this contribution for a pointer-intensive C++ code like gPLUTO requires explicit allocation strategies that underscore practical portability challenges.

This work assessed performance portability of directive-based GPU programming by porting the production astrophysical code gPLUTO to NVIDIA A100 and AMD MI250X GPUs using OpenACC and OpenMP. Directive-based models ease incremental porting of legacy codes, but require architecture-aware tuning, detailed profiling, and data layout transformations to achieve competitive performance.

A key outcome is that performance portability is governed more by compiler maturity and backend behavior than by the programming model alone. The NVHPC toolchain benefits from over a decade of CUDA-targeted optimization heuristics, while the ROCm OpenMP implementation, though based on the same LLVM infrastructure, represents a more recent engineering effort with GPU-specific passes still under active development. On NVIDIA A100, the shared CUDA backend enables near-parity between OpenACC and OpenMP. In contrast, the same OpenMP implementation and workload on AMD MI250X exhibits a slowdown of $\sim 3\times$. These are not explained by insufficient parallelism, but arise from strong sensitivity to strided memory-access patterns, limited latency hiding, and variable compiler optimization across kernels. We observe that architectural bottlenecks manifest differently across vendors: on MI250X, high sustained HBM bandwidth does not translate into throughput for irregular or directionally dependent kernels, while on NVIDIA GPUs C++ abstractions primarily manifest through register pressure and spilling. We also show that application-level performance is dictated by a small number of latency-sensitive kernels: large speedups in secondary kernels (e.g., Reconstruct()) do not compensate for slowdowns along the critical path.

Overall, porting from OpenACC on NVIDIA to OpenMP on AMD is not a ”search and replace” process: recovering performance requires intrusive transformations such as loop reorderings, data-layout changes, software transpositions, and direction-specific specializations. Future work should also isolate the impact of unified versus managed memory on AMD platforms and explore explicit data-movement strategies that avoid page-migration overhead while preserving maintainability. Furthermore, extending the analysis to Intel GPUs would provide a third vendor perspective, though system-dependent unified_shared_memory (USM) behavior and compiler maturity introduce non-trivial engineering challenges (Elwasif, 2023) (mappers, explicit allocation APIs, compiler-specific paths). Moreover, inspecting OpenMP implementations with alternative compilers (e.g., upstream Clang/LLVM) on NVIDIA hardware to isolate the impact of NVHPC-specific optimizations from intrinsic OpenMP model characteristics would be a valuable insight. Quantifying porting effort alongside performance gain, e.g., refactoring complexity per kernel, would further make these findings more actionable for the HPC community. While such work increases development cost, multi-vendor support remains essential for broad access to EuroHPC systems, long-term sustainability of scientific codes, and exploitation of architecture-specific strengths. Combining metric-based approaches with detailed profiling therefore represents an important direction for future research. In summary, correctness portability across modern GPUs is readily achievable, whereas performance portability remains a significant challenge.