The Question
Your infrastructure team is planning the next phase of AI compute capacity — whether that means on-premises GPU clusters, dedicated cloud instances, or a hybrid model. The NVIDIA H100 has been the default answer for the past two years, but the market has shifted. AMD's MI300X is deployed at scale by Microsoft and Meta. Intel's Gaudi 3 is available on-premises and through Intel Cloud. NVIDIA's Blackwell architecture (B200, GB200) is ramping through 2025 and into 2026, promising a significant generational leap in performance.
The hardware benchmarks are everywhere. They are also largely useless for enterprise procurement decisions without understanding what the benchmarks are measuring, which workload types they reflect, and what software dependencies constrain your options. A GPU that delivers 2.5x the training throughput of its predecessor is only relevant if you are doing the kind of training it was designed to accelerate. A GPU with 192GB of memory is only relevant if your inference workloads are actually memory-bandwidth-bound.
The CUDA ecosystem is the subtext of every AI hardware conversation that enterprises avoid having explicitly. Most organizations have AI software that was written for NVIDIA hardware, runs on CUDA, and would require meaningful engineering effort to port to AMD ROCm or Intel oneAPI. That switching cost is real — but it is not infinite, and it is changing as the non-NVIDIA ecosystems mature.
NVIDIA's CUDA ecosystem dominance is real and creates genuine switching costs — but the memory bandwidth advantage of AMD MI300X and Intel Gaudi 3 makes them worth evaluating for inference-heavy workloads where CUDA dependency is minimal.
Why This Matters Now
In January 2025, Microsoft confirmed that it had deployed AMD MI300X GPUs at scale in Azure, running production inference workloads including portions of the Azure OpenAI Service infrastructure. This was not a pilot or a technology preview — it was a capacity decision by one of the world's largest AI infrastructure operators, choosing AMD hardware for production LLM inference because the MI300X's memory bandwidth and HBM3 capacity made it economically and technically competitive with NVIDIA H100 for that specific workload profile.
Meta made a similar decision, deploying MI300X in its AI infrastructure for inference workloads. When two of the largest AI operators in the world are deploying AMD hardware at scale in production, the narrative that CUDA lock-in makes AMD irrelevant for enterprise AI collapses.
Simultaneously, NVIDIA's Blackwell architecture launched in 2024 and began ramping supply through 2025. The B200 and GB200 NVL72 systems represent a significant generational advance — roughly 2.5x H100 training throughput and 5x inference throughput for LLMs, according to NVIDIA's published figures. But Blackwell pricing carries a significant premium over H100, and supply constraints on Blackwell continued through mid-2025, meaning many enterprises are still evaluating H100-class hardware for near-term deployments even as Blackwell becomes the long-term standard.
The result is an enterprise hardware market with more genuine optionality than at any point in the AI infrastructure cycle — and more analytical complexity in making the right choice.
What the CURVE™ Data Shows
The 2026 Stackcurve AI Infrastructure CURVE™ Report evaluated AI accelerator hardware across six dimensions: raw performance for training workloads, inference throughput and efficiency, memory capacity and bandwidth, software ecosystem maturity, total cost of ownership, and supply chain reliability. Vendors covered include NVIDIA (H100, H200, B200, GB200 NVL72), AMD (MI300X, MI325X), Intel (Gaudi 3), and Google (TPU v5e, v5p — covered in the cloud section).
NVIDIA H100 and H200 remain the enterprise standard for organizations with existing CUDA dependencies and mixed training/inference workloads. The H200's primary improvement over the H100 is memory — 141GB HBM3e at 4.8 TB/s bandwidth versus H100's 80GB at 3.35 TB/s — which directly addresses the memory-bound bottleneck that limits LLM inference throughput on H100. For organizations running inference on models in the 70B–400B parameter range, H200 is a meaningful step forward.
AMD MI300X scores highest in the memory capacity and bandwidth dimensions — 192GB HBM3 at 5.3 TB/s bandwidth is the largest GPU memory footprint available, and it is a real advantage for inference workloads running large models. The ROCm software ecosystem gap versus CUDA has narrowed substantially through 2024–2025, with PyTorch, JAX, and major inference frameworks including vLLM offering MI300X support.
Intel Gaudi 3 scores competitively on cost and on-premises availability, with 128GB HBM2e and an open software stack that does not require proprietary ecosystem investment.
NVIDIA Blackwell (B200/GB200 NVL72) leads on raw performance for large-scale training and rack-scale LLM inference, with the GB200 NVL72 representing the highest-throughput inference system currently available for very large models.
The full vendor rankings are in the 2026 Stackcurve AI Infrastructure CURVE™ Report — free to download.
The Gap Most Buyers Miss
Most enterprises evaluate AI hardware on peak FLOP counts rather than the metrics that actually predict workload performance.
TFLOPS (teraflops) is the metric vendors lead with in marketing materials and the metric enterprise buyers most commonly use for comparisons. It is also the least predictive metric for most real-world AI workloads, particularly inference. Here is why.
Memory bandwidth, not compute throughput, limits LLM inference
Large language model inference is memory-bandwidth-bound, not compute-bound. When a model generates a token, it must load model weights from GPU memory into processing cores. The speed at which that can happen is determined by memory bandwidth, not raw FLOP count. An H100's 3.35 TB/s bandwidth is the constraint on how fast it can serve tokens from a 70B parameter model — not its 989 TFLOPS BF16 compute. The AMD MI300X's 5.3 TB/s bandwidth advantage over the H100 translates directly into better tokens-per-second throughput for inference workloads on large models. The H200's jump to 4.8 TB/s bandwidth was explicitly designed to address this constraint.
Enterprises evaluating hardware for inference workloads should prioritize memory bandwidth and memory capacity per dollar, not peak FLOPS per dollar.
Multi-GPU interconnect determines performance on large model training
For training workloads on models that require tensor or pipeline parallelism across multiple GPUs — which includes any model in the 30B+ parameter range with practical training batch sizes — the speed of the inter-GPU interconnect is a primary performance determinant. NVIDIA's NVLink 4.0 provides 900GB/s bidirectional bandwidth between GPUs in an H100 SXM5 system. AMD's Infinity Fabric provides comparable connectivity within MI300X configurations. Enterprises building multi-node training clusters need to evaluate not just per-card performance but the full interconnect topology of their intended configuration.
CUDA dependency is a spectrum, not a binary
Organizations often treat CUDA dependency as a yes/no question: either you have it and are locked to NVIDIA, or you don't. In practice, it is a spectrum. PyTorch code that runs on CUDA will typically run on AMD ROCm with minor modifications. Custom CUDA kernels — the kind written by research teams for specific model architectures — require genuine porting effort. Model serving frameworks including vLLM, TGI, and SGLang support both NVIDIA and AMD hardware. For enterprises that are primarily consuming pre-trained models through inference frameworks rather than writing custom training kernels, the CUDA dependency is much lower than assumed.
Total cost of ownership must include power and cooling
A rack of H100 SXM5 systems consumes approximately 10kW per GPU under training load — a full 8-GPU server draws 80kW. The GB200 NVL72 rack-scale system consumes up to 120kW total. For enterprises evaluating on-premises AI hardware, power delivery and cooling infrastructure is a material capital and operating cost that is frequently excluded from hardware TCO calculations. Datacenter capacity planning for AI GPU clusters is a meaningfully different exercise than planning for CPU servers.
Questions Your Buying Team Should Be Asking
1. What is the ratio of training to inference workloads in your AI compute plan, and how does that ratio affect which hardware specification deserves the most weight?
Training workloads are typically compute-bound and favor high FLOP count and fast multi-GPU interconnect. Inference workloads for large language models are memory-bandwidth-bound and favor high memory capacity and bandwidth per dollar. A team primarily running inference on 70B+ parameter models should weight AMD MI300X memory bandwidth more heavily than NVIDIA H100 FLOP count. A team training large models from scratch should weight NVIDIA NVLink interconnect performance. Most enterprises are running more inference than training — which means most should be weighting memory bandwidth more heavily than they are.
2. What is your actual CUDA software dependency — specifically, do you have custom CUDA kernels, or are you using standard frameworks?
Audit your AI software stack before concluding that CUDA lock-in makes AMD or Intel hardware non-viable. If your AI workloads run on PyTorch or standard inference frameworks, the porting cost to ROCm is lower than assumed. If your team has invested in custom CUDA kernel development for specific model optimizations, the switching cost is real. The answer determines whether AMD MI300X or Intel Gaudi 3 deserve serious evaluation.
3. What is your hardware procurement timeline, and does Blackwell supply availability align with your deployment schedule?
NVIDIA B200 and GB200 NVL72 systems represent genuine generational advances for large-scale training and inference. But if your deployment timeline is within the next 6–9 months and Blackwell supply is constrained for your use case and cloud provider, H100 or H200 may be the practical answer. Avoid benchmarking your procurement decision against hardware that is unavailable in your deployment window.
4. If evaluating on-premises hardware, what is your datacenter's power capacity and cooling infrastructure, and does it support AI GPU density?
Modern AI GPU servers have power and cooling requirements that frequently exceed existing datacenter infrastructure. An H100 server draws 10kW per GPU; a full-rack H100 DGX cluster requires dedicated power and liquid cooling in most modern GPU server configurations. Enterprises that have not done a datacenter infrastructure assessment before committing to on-premises AI hardware consistently find themselves constrained by facility limitations that delay deployments by months.
5. How are you planning to access hardware — on-premises purchase, colocation, or cloud instances — and how does each access model affect your hardware optionality?
Cloud instances (AWS P5, Azure ND H100 v5, Google Cloud A3) provide access to NVIDIA H100 and H200 hardware through managed infrastructure, with AMD MI300X available through Azure ND MI300 v5 instances. On-premises purchase provides hardware optionality across NVIDIA, AMD, and Intel. Colocation with GPU-specialized providers (CoreWeave, Lambda Labs, Vultr) provides access to a wider range of hardware configurations than hyperscale cloud. Your access model constrains your hardware options — understand the constraint before evaluating hardware specifications.
The Stackcurve Take
The AI hardware market in 2026 is more competitive and more analytically complex than the NVIDIA-only default suggests. NVIDIA remains the clear leader for training workloads at scale, for organizations with significant CUDA dependencies, and for the highest-performance inference on the latest model architectures. That leadership is real and should not be dismissed.
But AMD MI300X is a genuine enterprise option for inference-heavy workloads, deployed at scale by Microsoft and Meta, with a narrowing software ecosystem gap. Intel Gaudi 3 is a credible on-premises option for cost-conscious training workloads with an open software stack. NVIDIA Blackwell sets the new performance ceiling for large-scale training and rack-scale inference, but at a price premium that requires careful TCO analysis.
The enterprises that optimize their AI hardware investments are those that evaluate workload characteristics — training vs. inference, model size, CUDA dependency, deployment timeline — before applying vendor preference. Memory bandwidth matters more than most buyers think. CUDA lock-in constrains less than most buyers assume. And the total cost of operating AI hardware, including power, cooling, and software engineering, is consistently underestimated.
The 2026 Stackcurve AI Infrastructure CURVE™ Report covers AI accelerator hardware including detailed performance and TCO analysis across NVIDIA, AMD, and Intel platforms. Download it free →
Stackcurve Advisory Briefs are independent research. No vendor pays for placement, tier assignment, or editorial influence. The CURVE™ methodology is disclosed in full at stackcurve.net/research/methodology.