Beyond Per-Token Pricing:
A Concurrency-Aware Methodology for
LLM Infrastructure Cost Estimation

The majority of LLM cost estimation tools and academic analyses employ a token-volume model: $C=\frac{T}{\Theta_{\max}}\times P_{\text{GPU}}$, where $T$ is token count, $\Theta_{\max}$ is theoretical maximum throughput, and $P_{\text{GPU}}$ is the GPU hourly price.

Pan et al. [1] present a break-even framework comparing on-premise TCO against commercial API subscription fees. Their model assumes full utilization during 160 operational hours per month, and their accompanying calculator accepts model selection, GPU selection, and GPU count—but no utilization, concurrency, or workload parameters. The SLaM framework [2] assumes 80% utilization on a single T4 GPU and computes a direct cost-per-token comparison against GPT-4 API pricing.

A survey of 15+ public calculators (Helicone, LiteLLM, YourGPT, llm-prices.com, Mem0.ai, DocsBot, and others) confirms that over 90% accept only model name and token counts as inputs. The LLM Inference TCO Calculator [12] is the most comprehensive existing web-based tool, accepting concurrent users, deployment shape, and latency factors as inputs; however, its throughput values are heuristic presets rather than empirically measured, and it explicitly disclaims queueing theory modeling.

Several systems papers model concurrency and deployment shape as optimization variables. Vidur [3] (MLSys 2024) searches over tensor parallelism, pipeline parallelism, GPU SKU, and scheduling policy to optimize QPS-per-dollar subject to latency SLO constraints—demonstrating that using the wrong workload’s optimal configuration results in up to 2$\times$ cost overhead. Mélange [4] formulates GPU allocation as a cost-aware bin-packing ILP across heterogeneous GPU types, achieving up to 77% cost reduction. SageServe [5] handles diurnal traffic patterns with ARIMA-based capacity forecasting and differentiated SLAs; Llumnix [33] complements this with runtime rescheduling of in-flight requests across replicas for tail-latency and utilization control. Building on the continuous-batching foundations established by Orca [30], Splitwise [25], DistServe [26], and Sarathi-Serve [27] attack the throughput–latency tradeoff from the scheduler/engine side: by disaggregating prefill and decode across GPUs, by disaggregating the KV cache across a dedicated pool (Mooncake [31]), or by chunking prefill, they raise the achievable $\Theta_{\max}$ at a given SLO. These are complementary to our contribution: they optimize the engine ceiling at a fixed operating point, while $C_{\mathrm{eff}}=f(H,M,Q,\lambda,L)$ characterizes how cost per token traverses the entire $\lambda$ axis below that ceiling. §5.9 provides measured cross-hardware and TP-scaling evidence that complements Vidur’s simulated cost-overhead predictions and Mélange’s allocation-objective premises.

BentoML’s llm-optimizer [13] supports concurrency sweeps with Pareto frontier visualization but does not output cost-per-token. SemiAnalysis InferenceX (formerly InferenceMAX) [14] provides nightly cross-hardware benchmarks with cost attribution at a market-aggregate level but focuses on hardware-vendor reference configurations rather than same-GPU load/cost economics; vllm-cost-meter measures the operator’s own server under their own traffic. These are complementary, not competing. GuideLLM [15] from the vLLM project automates concurrency sweeps but outputs only latency and throughput metrics without cost computation. NVIDIA’s TCO methodology [10] outlines a clean benchmarking-and-sizing workflow but does not release a standalone open-source calculator.

These are systems optimization or benchmarking tools—their contribution is finding cheaper configurations or measuring performance, not providing a cost estimation methodology that connects offered load to dollars for practitioners.

Erdil [6] builds a theoretical roofline model for LLM inference economics, producing Pareto frontiers of serial speed versus cost-per-token, explicitly modeling how tensor parallelism trades utilization for speed. WiNGPT (Zhuang et al.) [7] is the closest prior empirical work: they demonstrate a 2.6$\times$ cost reduction from concurrency 8 to 48 on A800 GPUs using vLLM, and articulate three empirical principles—diminishing marginal cost, diminishing returns to scale, and an optimal cost-effectiveness zone—that are qualitatively consistent with the $C_{\mathrm{eff}}(\lambda)$ curves reported here. However, their sweep starts at concurrency 8 (already partially batched, missing the steep cold-start regime below $\lambda{=}5$ that drives our headline spread), covers only medical-domain models on a single hardware family, and releases neither code nor data. We generalize across three architectural classes, two hardware families (H100 NVL, A100 PCIe), provide a parameterized five-variable cost function, and release an open operational meter. Concurrent request scheduling in LLM serving has also been studied with analytical cost models [8], but with a focus on scheduler-level prefill/decode cost rather than per-deployment operational cost measurement.

Prior work splits cleanly into four camps, none of which closes the loop for a practitioner. Token-volume calculators and break-even analyses [1, 2] treat utilization as a user-supplied input. Systems-optimization frameworks [3, 4, 5] find cheaper configurations but do not emit $/M-tokens for a given deployment. Benchmark harnesses and industry dashboards [13, 15, 14, 10] measure throughput or publish market-aggregate TCO but do not instrument an operator’s own server. Prior work has modeled inference cost theoretically [6], observed offered-load–cost behavior empirically in narrow domains [7], benchmarked engines under SLO constraints without emitting cost-per-token [13, 15], and proposed $/M-tokens as a benchmark metric.²²2kubernetes-sigs/inference-perf Issue #139 (July 2025) scopes exactly this feature; vllm-cost-meter can be read as a standalone realization of that RFE. No prior work combines all four: a parameterized cost function calibrated across architectures and hardware families, a live-server instrument that reads the operator’s own Prometheus metrics, and open data and tool. This paper does.

We define the effective cost-per-million-tokens $C_{\mathrm{eff}}$ as a function of four swept variables with one held implicit:

The semicolon separates the four axes we empirically sweep ($H$, $M$, $Q$, $\lambda$) from the latency-SLO parameter $L$, which we hold implicit in the sweeps reported in this paper (all configurations are measured without an SLO cutoff, and $L$ would enter as an admission-control filter on the measured distribution). Incorporating $L$ as an active sweep dimension is future work; §6.9 discusses the scope implications.

The core insight is that GPU utilization $U$ is not an independent parameter to be assumed (as in prior work) but is itself a dependent variable determined by the interaction of $H$, $M$, $Q$, $\lambda$, and $L$:

At low offered load, GPU compute units are underutilized—the GPU waits for memory transfers and cannot fill its execution pipeline. As offered load increases, realized in-flight concurrency improves utilization, but latency also increases due to queuing and batch contention. The latency SLO $L$ acts as a constraint that caps the achievable utilization.

The effective cost then becomes:

This differs fundamentally from the token-volume model, which assumes:

The ratio $C_{\mathrm{eff}}/C_{\text{naive}}=\Theta_{\max}/\Theta_{\text{achieved}}=1/U$ captures the underutilization penalty—the multiplicative factor by which naive estimates understate true costs.

We hypothesize—and demonstrate experimentally in Section˜5—that the shape of the utilization function $U(\lambda,L)$ differs systematically between dense and mixture-of-experts (MoE) architectures. Dense models exhibit more predictable scaling with realized in-flight batch size because all parameters are activated for every token. MoE models activate only a subset of parameters per token, leading to different memory access patterns and batch efficiency characteristics.

This means a single cost formula calibrated on one architecture will produce incorrect estimates for another, even on the same hardware. Model architecture must be a first-class variable in any cost estimation framework.

Given the utilization-adjusted cost $C_{\mathrm{eff}}$, we define the self-host vs. API crossover as the offered arrival-rate threshold $\lambda^{*}$ where:

This crossover point shifts based on all five input variables. Prior work reports a single break-even (e.g., “self-hosting is cheaper above 50M tokens/month”), but the actual crossover is a surface in the $(\lambda,L,H)$ space, not a single point.

All experiments are conducted on a single compute node equipped with two NVIDIA H100 NVL 96GB GPUs (Standard_NC80adis_H100_v5, Azure ML). This configuration was chosen deliberately to demonstrate that the framework can be validated with modest, widely-available infrastructure—a single cloud node at $13.96/hr on-demand—without requiring multi-node clusters or specialized hardware. Dense models (Llama 3.1 8B) and ultra-sparse MoE models (Qwen3-30B-A3B) each use a single H100 (TP=1); the sparse MoE model (Mixtral 8x7B) uses both H100s with tensor parallelism (TP=2). Single-GPU configurations (C1–C4) are costed at $6.98/hr per GPU, which is $13.96/hr/node amortized across the two H100s under the assumption that the second GPU is independently utilized by a co-tenant; a dedicated-node operator should double C1–C4 numbers. This denominator choice is made explicit in §5.

We select three models representing a gradient of architectural sparsity, all well-supported by the vLLM inference engine:

The headline numerical results in this paper come from vLLM [9], which fixes the pricing denominator and eliminates cross-engine variance from the reported figures.³³3SGLang [28] runs share the same apparatus (see §4.1 for the two pinned venvs, vllm-bench and llm-bench) and the S1–S6 rows in the companion repository reproduce our methodology on that engine; we exclude them from headline numbers to keep the pricing denominator single-engine. TensorRT-LLM is out of scope. Cross-engine benchmarks [3, 14] report that engine choice shifts $\Theta_{\max}$ but not the shape of $U(\lambda,L)$, so a single engine is sufficient to characterize the underutilization-penalty structure we report. We sweep across seven offered arrival-rate levels:

• •

  TTFT (Time to First Token)—P50, P90, P99

• •

  TPS (Tokens Per Second)—aggregate output throughput

• •

  E2E Latency—total request completion time (P50, P90, P99)

• •

  GPU occupancy—nvidia-smi utilization.gpu, the wall-clock fraction of time a kernel is resident. Reported for completeness only; it is not the economic utilization $U{=}\Theta_{\text{achieved}}/\Theta_{\max}$ defined in §3 and can read high even at $\lambda{=}1$ (a single resident decode kernel keeps the GPU nominally busy while it is arrival-limited at batch size 1), so we never treat it as a proxy for $U$

• •

  GPU Memory Utilization—VRAM occupancy

Each configuration runs 100 warmup requests (discarded) followed by 500 measured requests. We use synthetic random-token prompts with fixed input length 512 tokens and output length 256 tokens, with per-run random seeds to prevent cross-run prefix cache contamination. Prefix caching is explicitly disabled on all servers (--no-enable-prefix-caching) to isolate offered-load effects from cache artifacts. Arrivals follow a Poisson process (--burstiness 1.0).

From raw measurements, we compute:

• •

  Effective cost-per-million-tokens: $C_{\mathrm{eff}}=\frac{P_{\text{GPU}}\times 10^{6}}{\text{TPS}\times 3600}$  ($/M output tokens). Throughout, TPS is aggregate output-token throughput, so every $C_{\mathrm{eff}}$ figure is dollars per million output tokens; this matches the output-token basis of the API list prices we compare against (§5.6). Input (prefill) tokens consume compute but are not in the denominator.

• •

  Utilization rate: $U{=}\Theta_{\text{achieved}}(\lambda)/\Theta_{\max}$, the throughput ratio defined in §3 (distinct from the nvidia-smi GPU-occupancy metric)

• •

  Underutilization penalty: $C_{\mathrm{eff}}/C_{\text{naive}}$ at each operating point

• •

  Crossover arrival rate: the $\lambda$ where $C_{\mathrm{eff}}$ equals API pricing for GPT-5.5, Gemini 3.1 Pro, and Claude Sonnet 4.6

Figure˜1 shows $C_{\mathrm{eff}}$ across all six configurations. At $\lambda{=}1$ rps, effective costs range from $7.60/MTok (Llama FP16/FP8) to $15.25/MTok (Mixtral FP16)—the near-idle GPU cost amortized over a trickle of tokens. As offered load increases, $C_{\mathrm{eff}}$ falls precipitously: by $\lambda{=}25$ rps all single-H100 models are within 20% of peak-throughput cost.

The naive cost model underestimates self-hosted costs by 17.5–36.3$\times$. That is the $1/U$ gap between a utilization-naive estimate (any calculator that assumes the GPU runs at its $\Theta_{\max}$) and what you actually pay when the engine is arrival-limited at 1 rps. Prior informal estimates put the error at 3–10$\times$; ours shows they were themselves off by another 3–10$\times$. Even at 10 rps—an arrival rate most practitioners would not call “low”—the naive model is still wrong by 2.5$\times$ on dense Llama and 3.8$\times$ on ultra-sparse Qwen. The naive model is not merely optimistic; it omits the dominant cost term.

The three FP16 solid lines in Figure˜1 isolate the architectural families. At peak throughput, Llama 3.1 8B achieves 6,238 tok/s ($0.311/MTok), Qwen3-30B-A3B achieves 5,319 tok/s ($0.364/MTok), and Mixtral 8x7B on TP=2 achieves 4,454 tok/s at $0.871/MTok; the three curves converge to within $3\times$ at saturation.

Qwen’s lower throughput than Llama—despite activating only 3B parameters—reflects two structural costs: (1) MoE routing dispatch overhead per token, and (2) the full 30B parameter set occupying VRAM, constraining memory bandwidth. The shape of the scaling curve is, however, comparable between the two: the ultra-sparse Qwen model’s cost cliff from $\lambda{=}1$ to $\lambda{=}10$ rps (9.4$\times$ drop) is marginally shallower than dense Llama’s (9.7$\times$ drop), indicating that routing overhead does not meaningfully shift the per-request amortization curve at this scale. The cost-structure differences between dense and ultra-sparse MoE therefore manifest primarily as level shifts (higher idle cost, different saturation floor) rather than slope shifts, but are large enough in magnitude that a single cost formula calibrated on dense models will systematically misestimate MoE costs.

This active-parameter dominance holds in the batched decode regime measured here ($\lambda\geq 1$ rps, batch sizes reaching hundreds), where per-token arithmetic scales with active parameters while weight-memory movement is amortized across the batch. At small batch sizes or in prefill-dominated regimes, total parameters drive memory bandwidth cost and the active-vs-total distinction narrows [18]—a caveat that applies to the $\lambda{=}1$ edge of our sweep and to any deployment with long prompts and short outputs.

FP8 quantization reduces $C_{\mathrm{eff}}$ at every arrival-rate level (Figure˜2). Peak throughput gains are: +31% for Llama 3.1 8B (6,238$\to$8,155 tok/s), +74% for Qwen3-30B-A3B (5,319$\to$9,271 tok/s), and +69% for Mixtral 8x7B (4,454$\to$7,524 tok/s). The larger FP8 gain for MoE models is consistent with their memory-bandwidth-bound profile: sparse activation requires many small, scattered parameter reads, and halving parameter precision directly doubles effective bandwidth.

A notable emergent result: at peak offered load, Qwen3-30B-A3B FP8 ($0.209/MTok) is cheaper than Llama 3.1 8B FP8 ($0.238/MTok) despite having 30B total parameters. On this dense/ultra-sparse pair, the ordering inverts the common intuition that smaller models are cheaper to serve: active parameter count and memory-access pattern, rather than total model size, appear to govern the saturation cost. We treat this as suggestive—it rests on a single architecture pair with a $\sim$12% margin—and defer the larger-dense (e.g. Llama 3.1 70B) plus third-sparsity-ratio validation that would license a general claim to §7.

Table˜3 quantifies the penalty for our dense reference model. At $\lambda{=}1$ rps, effective cost is 24.4$\times$ the naive estimate. The penalty declines non-linearly: by $\lambda{=}25$ rps it falls to 1.18$\times$, and beyond $\lambda{=}50$ rps throughput approaches the hardware ceiling. Importantly, at $\lambda\geq 100$ rps, throughput plateaus at $\sim$6,230 tok/s while TTFT P50 rises to 2,561–3,748 ms, indicating queue saturation. This defines the model’s saturation threshold—the arrival rate above which additional load degrades latency without improving cost. The latency tails confirm that this cost floor is a no-SLO floor: at the $\lambda{=}50$ minimum-cost point TTFT_p99 has already reached 994.8 ms and TPOT_p99 71.3 ms—both well beyond the example production contract of §6.4 (TTFT${}_{p99}\leq 300$ ms, TPOT${}_{p99}\leq 50$ ms)—and the C2 repeat campaign shows TTFT_p50 run-to-run CV rising to $8.73\%$ at the same load (§5.8). An operator bound by such an SLO must back off to a lower arrival rate, where $C_{\mathrm{eff}}$ is strictly higher; the raw-saturation $C_{\min}$ reported here is therefore a lower bound on goodput-constrained cost—equivalently, an upper bound on the tokens-per-dollar an SLO-bound operator can achieve—exactly as the goodput-vs-saturation distinction in §4 and the operational workflow in §6.5 prescribe.⁴⁴4The $\lambda{=}5$ row’s tails (TTFT_p99 713.3 ms, TPOT_p99 16.4 ms) are inflated by a handful of outlier requests on this single, unrepeated run, with the bulk of the distribution unaffected (TTFT P50/P90 of 44.1/56.8 ms); monotone tail growth with load resumes by $\lambda{=}10$ (TTFT_p99 84.5 ms, TPOT_p99 12.3 ms). The P50, throughput, and cost columns are unaffected.

Figure˜3 extends this to all configurations. The maximum penalty is 36.3$\times$ for Qwen3-30B-A3B FP8 at $\lambda{=}1$ rps. Penalties at $\lambda{=}1$ rps range from 17.5$\times$ (Mixtral FP16) to 36.3$\times$ (Qwen FP8), converging to $\approx$1.0$\times$ by $\lambda{=}50$ rps across all configurations. The consistent pattern across architectures and precisions confirms that the underutilization trap is a structural property of discrete GPU rental cost, not an artifact of any specific model.

The cost floors quoted so far are unconstrained saturation values: they are the cheapest $C_{\mathrm{eff}}$ a config reaches at any $\lambda$. In production they are typically unreachable, because the latency-$L$ argument of $C_{\mathrm{eff}}=f(H,M,Q,\lambda,L)$ is not a free variable—it is a commitment. An operator who signs a service-level agreement (a B2B latency SLA, an internal interactive-product target) fixes $L$, and a fixed $L$ caps the offered load the deployment may sustain before its tail breaches, which in turn pins the operating point on the $C_{\mathrm{eff}}(\lambda)$ curve. The SLA is therefore a price: tightening it forces a lower $\lambda$ and a higher cost per token.

To make this concrete on the existing corpus we evaluate one fixed SLA used as the running example in §6.4: TTFT p99 $\leq 300$ ms and TPOT p99 $\leq 50$ ms. For each config we take the highest $\lambda$ in the 7-point sweep that satisfies both bounds and report its $C_{\mathrm{eff}}$ (Table˜4, Figure˜4).

Two readings follow. First, the advertised floor is a fiction for any latency-bound deployment: Llama 8B FP16’s $0.311/MTok floor lives at $\lambda{=}200$, where TTFT p99 is 7.4 seconds—no interactive product ships that. Its cheapest shippable cost under the SLA is $0.368/MTok, at $\lambda{=}25$. Second, the SLA premium over the (already-unreachable) floor ranges from 1.13$\times$ to 1.91$\times$, largest on Mixtral 8x7B FP16, whose strict per-GPU memory budget forces the SLA-feasible load down to $\lambda{=}10$. This is a deliberately conservative slice of $L$’s effect: it compares two points on the same curve. Loosening the SLA toward batch-grade latency walks the operating point rightward toward $C_{\mathrm{sat}}$, and tightening it past our interactive example walks it leftward into the steep part of Figure˜1, where the penalty is the 17.5–36.3$\times$ already reported. The practical statement is that the SLA an operator commits to in a contract is, through $\lambda$, a directly priced quantity—and one no token-volume calculator can express.

Figure˜5 plots $C_{\mathrm{eff}}(\lambda)$ against commercial API reference prices. Under the naive model, all configurations appear permanently cheaper than API alternatives—Llama FP8 at $0.238/MTok is over 50$\times$ cheaper than Gemini 3.1 Pro ($12.00/M output tokens [21]). Under the utilization-adjusted model, the picture differs.

At $\lambda{=}1$ rps, Mixtral FP16 costs $15.25/MTok—more expensive than Claude Sonnet 4.6 ($15.00/M output tokens). At $\lambda{=}5$ rps it falls to $3.16/MTok, below all API tiers. The naive analysis concludes Mixtral self-hosting is economical at any traffic level; the adjusted analysis reveals a crossover threshold near 1.5–2 rps below which the lower-priced reference tiers (Gemini 3.1 Pro, Claude Sonnet 4.6) are actually cheaper.

For Llama and Qwen models, the lower single-GPU cost means the crossover is below 1 rps—self-hosting is cost-effective even at minimal traffic, if the GPU is already provisioned—but only under the $6.98/hr co-tenant denominator of §4. Under the dedicated-node $13.96/hr rate the entire single-GPU $C_{\mathrm{eff}}(\lambda)$ curve doubles, roughly doubling the low-$\lambda$ cost (e.g. Llama/Qwen idle rises from $\approx$$7.60 to $\approx$$15.20/MTok) and pushing the API crossover to a higher arrival rate. Operators on a dedicated node should read the C1–C4 numbers at the doubled rate. (The $\lambda{<}1$ region is extrapolated outside our 7-point sweep; the exact crossover location below 1 rps is a modeled continuation of the measured $C_{\mathrm{eff}}(\lambda)$ curve, not a directly observed operating point.) However, the effective cost at 1 rps ($7.60/MTok) is 24$\times$ the naive estimate, compressing the apparent cost advantage over the lowest-priced reference tier (Gemini 3.1 Pro, $12.00/M output) from 98% (naive) to roughly a third at $\lambda{=}1$ rps.⁵⁵5At the specific $\lambda{=}1$ operating point: $1-7.60/12.00\approx 37\%$. The exact figure moves with the tier’s list price, which changes on vendor schedule—and under the dedicated-node denominator ($\approx$$15.20/MTok idle) the advantage inverts against the $12 tier and vanishes against the $15 tier. The qualitative point—expected savings compress sharply at low-traffic operating points—is what matters for infrastructure planning. An organization planning infrastructure on the naive model will be surprised to find that at realistic low-traffic operating points, most of the expected advantage evaporates.

§3 enumerated five factors held constant in the main sweep: engine, pricing mode, input:output ratio, parallelism, and arrival distribution. Two of these—the I/O ratio and the arrival distribution—admit direct empirical probes on the same hardware, as does a third held-constant protocol choice: prefix caching (disabled in the main sweep). We ran a targeted supplementary sweep across these three factors to quantify how far the headline underutilization penalty generalizes when those knobs are unfrozen. We report the results here rather than relegating them to an appendix because they materially bound the scope of our headline claim.

A natural reviewer question for any single-run benchmarking study is whether the reported spread is a structural property of the system or an artifact of measurement noise. We address this with a focused repeat-run campaign on the C2 configuration (Llama 3.1 8B FP8 on a single H100 NVL, 1xGPU deployment). We re-ran the random-token workload at four offered-load points $\lambda\in\{1,10,50,100\}$ with three independent repeats each (12 runs total, distinct seeds per repeat), holding all other knobs fixed. The driver targets a $\geq 60$ s steady-state measurement window: prompt count is $60\cdot\lambda$ clamped to $[500,6000]$ and warmup is $30\cdot\lambda$ floored at 100, producing $\{500,600,3000,6000\}$ prompts and $\{100,300,1500,3000\}$ warmups for $\lambda\in\{1,10,50,100\}$ respectively. At $\lambda{=}50$ and $\lambda{=}100$ the server is queue-limited and only $\sim$2,100 and $\sim$2,020 of the prompted requests complete within the measurement window; the CV computation uses the completed-request statistics reported by the harness. The vLLM server process stays resident across all 12 runs—a single server instance, so allocator and scheduler state are reused—which is consistent with how an operator would run back-to-back measurements and is the regime we are claiming stability for. Table 5 reports the coefficient of variation (CV) across repeats on throughput, effective cost, and TTFT_p50.

A second obvious reviewer question is whether the load-driven spread we report is a property of H100 NVL specifically—its HBM3 bandwidth, its native FP8 tensor cores, its NVLink fabric—or a structural property of the offered-load $\lambda$ axis that would reproduce on a different GPU family. We repeat the core sweep on a second, architecturally distinct platform: NVIDIA A100 80GB PCIe (no native FP8, Gen4 PCIe in place of NVLink, 1.94 TB/s HBM2e vs. H100’s 3.9 TB/s HBM3), at a list-price rate of $3.67/hr/GPU on Azure’s NC_A100_v4 family (roughly $1.90\times$ cheaper than H100 NVL per GPU). We run the identical measurement harness—same prompt corpus, same arrival-rate ladder $\lambda\in\{1,5,10,25,50,100,200\}$, same two-gate health check, same steady-state window—across six paired configurations: Llama 3.1 8B FP16 and FP8 (vLLM, TP=1), Qwen3-30B-A3B FP16 and FP8 (vLLM, TP=1), Mixtral 8x7B FP16 (vLLM, TP=2), and Llama 3.1 8B FP16 (SGLang, TP=1). A third Mixtral condition—TP=4—was added on a 4$\times$A100 node to exercise a tensor-parallel dimension the 2$\times$H100 apparatus could not. This TP=4 run (M3) is reported separately in Result 4 and is excluded from the six-configuration 7.0–11.4$\times$ spread quoted above; its own spread of 12.3$\times$ is discussed in context. An eighth A100 configuration—Mixtral 8x7B FP16 on SGLang TP=2 (M2)—is present in the public 56-run corpus but excluded from Table 6 because the SGLang engine did not scale on this hardware: throughput flatlined near 647 tok/s across every $\lambda$, consistent with a PCIe-interconnect bottleneck on TP=2 Mixtral that did not manifest on the NVLink-connected H100. The vLLM TP=2 row (M1) is reported in its place.

Every calculator we surveyed—15+ of them—makes the same mistake: it treats GPU utilization as an input parameter the user is trusted to guess (or quietly assumes 100%). Utilization is not an input. It is the output of the interaction between arrival rate, model architecture, quantization, and SLO. A calculator that accepts utilization as a field is asking its users to solve the paper they came to the calculator to avoid solving. That is why the headline error is an order of magnitude and not, say, 2$\times$: the entire class of tools has the causal direction of its own input backwards.

Beyond the calculators themselves, the offered-load axis is largely absent from how self-hosted inference is budgeted in practice: capacity plans and infrastructure reviews reason about GPU-hour and per-token rates as fixed inputs, leaving the variable that empirically moves realized cost the most—offered load—out of the cost model entirely. Restoring it as a first-class input, the framework enables three practical decisions:

1. 1.

   Given a target workload profile, what is the realistic cost of self-hosting?

2. 2.

   At what traffic volume does self-hosting become cheaper than a specific API tier?

3. 3.

   For a hybrid strategy, where should the routing boundary be set between self-hosted and API-served traffic?

A structural advantage of self-hosting that existing analyses consistently overlook: commercial API providers employ asymmetric token pricing, charging substantially more for output tokens than input tokens. GPT-5.5 charges $5.00/M input vs. $30.00/M output [19]; Claude Sonnet 4.6 charges $3.00/M input vs. $15.00/M output [20]; Gemini 3.1 Pro charges $2.00/M input vs. $12.00/M output [21]. Output tokens are priced 5–6$\times$ higher because generation is compute-intensive and sequential—each token requires a full forward pass.

Self-hosted models incur identical GPU infrastructure cost regardless of whether tokens are input (parallel prefill) or output (sequential decode). This asymmetry has a concrete implication: the effective API cost for generation-heavy workloads is far higher than aggregate per-token pricing suggests. Self-hosting, by contrast, imposes no separate input-token and output-token tariff—though the input/output mix still moves the realized cost per output token through the prefill/decode balance (§5.7), so a like-for-like comparison prices the API at its asymmetric per-token rates and the self-hosted deployment at its measured $C_{\mathrm{eff}}$ under the same workload shape.

As an illustrative back-of-the-envelope (not a measured benchmark), a code-generation pipeline with 100-token inputs and 500-token outputs ($\frac{5}{6}$ of tokens are output) pays approximately $(100\times\mathdollar 5+500\times\mathdollar 30)/600\approx\mathdollar 25.80$/MTok aggregate at GPT-5.5 list pricing—well above the nominal $5.00/M input rate typically quoted. The same tokens served on self-hosted Qwen3-30B-A3B FP8 carry no such output-token premium: its $0.209/MTok saturation cost (measured at the 512:256 shape) bills input and output tokens at the same GPU-time rate, though the realized figure still shifts with workload shape (§5.7). The asymmetric pricing effect amplifies the self-hosting advantage precisely for the workloads—code generation, long-form writing, agentic tool use—where enterprises are most likely to consider infrastructure investment.

Our framework currently models aggregate token throughput; a natural extension is to separate input and output token economics and incorporate the asymmetric pricing structure into crossover analysis.

A recurring question from infrastructure decision-makers is: “Why not just use a serverless API at $X per million tokens?” On its face, this appears to be a straightforward cost comparison. It is not. Published serverless per-token prices are unconditioned on any production service-level objective. Commercial providers do not publish enforceable per-request guarantees for time-to-first-token or inter-token latency at the p99 tail; best-effort availability is the operative contract. A dedicated self-hosted deployment that must meet a production SLO (for example, TTFT p99 $\leq$ 300 ms, TPOT p99 $\leq$ 50 ms) operates in a different cost regime entirely, because provisioning headroom to absorb arrival-rate variance without breaching the tail is precisely what drives $C_{\mathrm{eff}}$ away from $C_{\mathrm{sat}}$.

Concretely: under our six-phase live workload (1–50 rps), the best minute-level effective cost a Mixtral-8x7B FP16 deployment on 2$\times$H100 achieves is $8.23/MTok (Table 7, “Best” column, observed at the 50 rps saturation minute)—still roughly 9.5$\times$ the theoretical saturation floor ($C_{\mathrm{sat}}{=}\mathdollar 0.87$) because the averaging window covers the full arrival-rate traversal including idle sub-intervals. An operator pinned below saturation to preserve TTFT p99 headroom would sit strictly worse than this figure. The published commercial API price for a comparable class of model is often quoted at $12–$15 per million output tokens (the lower reference tiers of §5.6). The two numbers are not comparable: one reflects a best-effort stateless endpoint with no latency SLA, the other reflects a dedicated deployment that has committed to a p99 latency contract. Choosing between them based on headline per-token price alone is a category error, analogous to comparing spot cloud instance pricing against reserved-instance pricing without acknowledging availability and preemption differences.

The framework presented in this paper makes this mismatch visible. An organization with a production SLO should compute $C_{\mathrm{eff}}$ at the arrival rate its SLO permits (Table˜4 works this through for a fixed example SLA), then compare to serverless pricing while explicitly acknowledging that the serverless path surrenders tail-latency control. Our companion tool, vllm-cost-meter, gates its API-comparison feature behind an --accept-slo-mismatch flag for exactly this reason: the comparison is meaningful only when the user has consciously accepted that serverless offers no SLO counterpart.

The framework is designed to be actionable by a single infrastructure owner in a three-step workflow.

Step 1 — Deploy your chosen configuration. Select the model, quantization, and hardware that match your deployment constraints. The benchmark corpus released with this paper (the six H100 configurations C1–C6, plus the A100 cross-hardware and sensitivity sweeps; §7.1) provides a starting reference, but is not a substitute for on-hardware measurement under your own workload shape.

Step 2 — Load-test under your own workload and SLO. Run a sweep (for example, with vllm bench serve, inference-perf, LLMPerf, or GenAI-Perf[22, 23, 24]) using your real prompt-length and output-length distribution, your expected arrival pattern, and your declared TTFT/TPOT/E2EL SLO. Record the highest arrival rate $\lambda$ at which your SLO is preserved; this is the goodput of your deployment, distinct from the raw saturation $\Theta_{\max}$ reported here.

Step 3 — Plug observed goodput into the calculator. Feed the goodput-at-SLO into the $C_{\mathrm{eff}}$ formula (Eq. 3) to obtain the effective cost of your deployment under your actual operating contract. Compare that number—not $C_{\mathrm{sat}}$, and not the raw $\Theta_{\max}$ from a no-SLO benchmark—to any alternative pricing.

This workflow inverts the usual procurement order. Rather than picking a price point and then discovering latency consequences, the operator declares the SLO first and derives the cost second. The vllm-cost-meter tool supports the third step directly: its live scrape of vLLM’s Prometheus endpoint surfaces observed goodput in real time against user-declared SLO budgets, and computes $C_{\mathrm{eff}}$ continuously as arrival rate varies.

vllm-cost-meter is the concurrency-aware replacement for every calculator critiqued in Section 2.

To validate the framework beyond controlled benchmarks, we deployed vllm-cost-meter on the same H100 NVL hardware alongside live vLLM inference servers running all six benchmark configurations (C1–C6). The tool is an objective live instrument: on every tick it scrapes vLLM’s Prometheus /metrics endpoint, computes the live $C_{\mathrm{eff}}$ from observed generation tokens per second and declared hardware cost, and surfaces the instantaneous offered-load/cost relationship. A six-phase enterprise workload simulation ramped request rates from 1 to 50 rps and back, exercising the full range of arrival rates that a production deployment would traverse across a day.

Table 7 summarizes the results. In one hour of real traffic the same deployment traversed a 653$\times$ cost swing between its cheapest and most expensive minutes—which we present as an illustration of minute-window dispersion, not as a reproducible validation datum (per the caveat below). This is an extreme ratio: it contrasts the idle-valley instantaneous minute against the peak-load minute, not a time-weighted mean. Mixtral FP16 peaked at $5,379 per million tokens during the idle valley (when only a handful of tokens were billed against the full GPU-hour) and dropped to $8.23 at 50 rps. Averaged over the hour the effective cost is dominated by the high-$\lambda$ minutes; the 653$\times$ figure is best read as a stress-test artifact of minute-window cost attribution near idle—a near-singular ratio that drops as the window lengthens—rather than as a monthly billing multiplier. We quote it to make the minute-level dispersion concrete; the underlying per-minute Prometheus ticks are not checked into the public corpus (Table 7 records the aggregated Best/Worst columns), so the 653$\times$ should be read as a directional signal rather than a reproducible data point. Any cost number quoted without a $\lambda$ attached—even averaged—is meaningless to a degree most practitioners badly underestimate.

The live validation demonstrates two things. First, the concurrency-aware framework produces actionable real-time estimates: the same deployment traces a recognizable $C_{\mathrm{eff}}$ curve as $\lambda$ varies, matching the shape predicted by the benchmark sweep. Second, the underutilization penalty is not merely a theoretical artifact of offline benchmarking—it manifests immediately in live operation whenever $\lambda$ falls below the arrival rate at which the engine saturates. The curve the tool surfaces is the same curve an operator would trace by re-running the benchmark sweep on their own workload; the tool makes this visible without a separate measurement pass. The validation we do claim is one of agreement: the live, Prometheus-derived $C_{\mathrm{eff}}$ traces the same offered-load curve, at the same operating points, as the offline benchmark sweep on identical hardware—confirming that the meter reports the cost structure an operator would otherwise have to reconstruct by re-running the sweep.

We deliberately scope the headline sweep to synthetic fixed-length workloads on a single H100 NVL node—with a targeted A100 80GB PCIe validation sweep (§5.9)—for reproducibility and to avoid exposing proprietary production data. The main benchmark sweep is run with prefix caching and speculative decoding disabled, establishing a conservative performance floor and cost upper bound; our sensitivity probes (§5.7) partially relax these assumptions, enabling real prefix-cache hits and varying the arrival process and I/O shape. Production deployments with prefix caching, speculative decoding, and workload-specific tuning would achieve lower effective costs, making the absolute costs reported here upper bounds rather than typical-case estimates. Several extensions are natural: validating with variable-length conversational traces (e.g., ShareGPT, LMSYS-Chat-1M [29]), incorporating SLO constraints as a first-class optimization target rather than an observed output, modeling diurnal traffic patterns and autoscaling economics, and extending to additional hardware generations (B200, MI300X) and inference engines (SGLang, TensorRT-LLM). The framework’s structure is designed to accommodate these extensions, though each would require its own empirical validation to confirm the shape of $U(\lambda,L)$ under the new conditions.

Our benchmarks use synthetic fixed-length workloads with uniform 512-token inputs and 256-token outputs; production workloads with variable-length conversational distributions will exhibit different utilization curves. We test three model architectures (dense, ultra-sparse MoE, sparse MoE); additional architectures (e.g., state-space models, encoder-decoder) may show different cost behaviors. The headline vLLM numbers are conducted on H100 NVL; we add a cross-hardware validation sweep on A100 80GB PCIe (§5.9) to rule out single-hardware confounding, and the load-driven spread reproduces there, though still-older hardware (V100) or newer generations (B200, MI300X) would shift the curves further and are left as future work. The headline numbers use a single engine (vLLM); engine choice preserves the qualitative cliff but shifts its magnitude—the same Llama 3.1 8B FP16 configuration spreads 24.4$\times$ on vLLM versus 12.0$\times$ on SGLang (Table 6), as a lower $\Theta_{\max}$ compresses the idle-to-saturation ratio—so the absolute multiplier should be read as engine-conditional. We vary offered request rate while observing latency as an output rather than constraining it as an SLO input, leaving SLO-constrained optimization for future work. We do not model multi-tenant GPU sharing, which can improve utilization for providers serving multiple customers. GPU pricing changes frequently; the framework’s value is in the methodology, not specific dollar amounts.

The companion vllm-cost-meter tool and the raw per-run CSVs are released under an MIT license at https://github.com/pChitral/vllm-cost-meter: the 140-run core corpus—the 42-run H100 vLLM headline sweep, the 42-run H100 SGLang companion sweep, and the 56-run A100 80GB PCIe cross-hardware sweep—together with the I/O-shape, prefix-caching, and burstiness sensitivity sweep (§5.7) and the measurement-stability campaign (§5.8). All saturation points, $C_{\mathrm{eff}}$ spreads, underutilization penalties, and stability CVs in §5 are re-derivable directly from these CSVs; the variable-length and real-prefix-hit probes of §5.7 are summarized in-text. A version-tagged Zenodo DOI will be minted at camera-ready. The repository is intended as a community substrate for extending the $\lambda$-axis sweep to additional hardware, model architectures, and SLO-bounded regimes.