Choosing between a hyperscaler vs specialist GPU cloud comes down to one operating fact that’s easy to skip past: training capacity behaves like a queueable batch resource, while latency-sensitive inference cannot queue at all, so a cluster sized correctly for one workload is routinely the wrong size for the other.
That single distinction, not raw GPU count or brand, should drive the capacity, network, support, and contract terms you negotiate — a provider that fits training poorly serves inference well for a different reason, and the two sizing methods do not transfer.
Quick take
Choose a hyperscaler when you need multi-year Reserved Instance capacity guarantees, Compute Savings Plans discounts, or Spot pricing for fault-tolerant batch training.
Choose a specialist GPU cloud when EU-jurisdiction data residency, predictable egress costs, or workload-aware scheduling to fix low GPU utilization matters more than a broad service catalog.
Neither provider type guarantees your inference latency target by default — that depends on the GPU’s memory bandwidth and KV-cache capacity, not the cloud brand.
The workload split that decides the shortlist
Training capacity planning answers a throughput question: how many GPUs of a given type complete a target model and dataset within a target wall-clock time, computed from total training FLOPs divided by achievable per-GPU throughput and adjusted for parallelism overhead.
Inference capacity planning answers a different question — how many GPUs, serving how many concurrent requests, meet a latency target with enough headroom to absorb traffic peaks — and because a request that arrives during a peak cannot be queued, inference utilization is deliberately lower than training utilization by design.
Sizing a cluster for training throughput and reusing that plan for inference produces exactly the failure mode operators keep hitting: the cluster serves inference poorly because it lacks latency headroom, or it trains slowly because inference headroom sits idle instead of absorbing sustained batch load.
What a hyperscaler contract actually commits you to
A hyperscaler contract is really five separate pricing models stacked together — On-Demand, Savings Plans, Reserved Instances, Spot, and Dedicated Hosts — and mature accounts use several at once rather than choosing one, matching each workload to the model that fits its behavior.
Compute Savings Plans cut costs up to 66% and follow you across instance families, including a Graviton or generation migration, preserving flexibility while keeping the discount.
EC2 Instance Savings Plans and Reserved Instances go up to 72% but lock the discount to one instance family and region; Reserved Instances also reserve capacity in a specific Availability Zone, a guarantee Savings Plans do not offer.
Spot Instances cut costs up to 90% by using spare capacity that AWS can reclaim with a two-minute warning, which makes them fit for fault-tolerant batch training and checkpointable jobs but a poor match for a latency-sensitive inference endpoint that needs to stay up.
The dollar spread inside one hyperscaler service makes the stakes concrete: a g5.xlarge with a single NVIDIA A10G runs about $1.006 an hour, an 8-GPU p4d.24xlarge runs about $32.77 an hour, and 40% of companies now spend more than $10M a year on AI compute without a clear read on whether it is worth it.
GPU memory and bandwidth: the spec that actually moves inference cost
On the hardware side, NVIDIA lists the H100 SXM at 80 GB of GPU memory and 3.35 TB/s of memory bandwidth, while the H100 NVL variant offers 94 GB and 3.9 TB/s — the difference that matters for inference is the decode phase, which repeatedly reads model weights and KV-cache data and is bandwidth-bound rather than compute-bound.
The H200 raises the ceiling further, with 141GB of HBM3e memory and 4.8TB/s of bandwidth, roughly 1.4x the bandwidth of an H100 80GB SXM, which matters specifically for 70B-class models running long context windows where KV-cache size — not raw compute — becomes the constraint.
None of this favors a provider type on its own: the deciding metric is cost per token under realistic traffic, because a cheaper GPU that needs more replicas or breaks its latency budget under load can end up more expensive than a premium part running at higher utilization.
Where specialist providers change the calculus
Specialist providers change the calculus mainly on jurisdiction: the EU AI Act and GDPR tighten requirements around data residency, and keeping processing inside the EU removes a recurring procurement question that a hyperscaler region alone does not resolve.
Cost structure adds a second lever — egress fees accumulate on platforms like AWS or GCP as data moves out, a switching cost that specialist providers marketing EU sovereignty use as a direct point of contrast.
Industry surveys cited in the same reporting put average GPU utilization in enterprise clusters well under half of capacity, which means a large share of every dollar spent on hardware like the H100 buys nothing — a symptom of orchestration and provisioning guesswork rather than a hardware defect.
Specialist entrants position workload-aware scheduling as the fix: predicting a job’s runtime, memory footprint, and expected utilization before it runs, so teams can pick cost-effective hardware per task instead of defaulting to an oversized static instance — a pitch backed in at least one case by €10.3M in pre-seed funding for exactly that orchestration layer.
Watch out
Egress fees and data-residency review cycles are switching costs that don’t show up in a per-GPU-hour comparison, so they need to be priced separately before a migration decision.
A Reserved Instance’s capacity guarantee is tied to one Availability Zone and instance family — changing either mid-term forfeits the guarantee even though the billing discount can survive under a Convertible RI.
Capacity, performance, and contract terms compared
| Criterion | Hyperscaler (AWS EC2) | Specialist GPU cloud |
|---|---|---|
| Capacity guarantee | Reserved Instances lock a specific Availability Zone and instance family for 1 or 3 years | Workload-aware pre-run prediction of runtime and memory footprint rather than a zone lock |
| Discount ceiling | Up to 72% off with Reserved Instances or EC2 Instance Savings Plans, up to 90% on interruptible Spot | Not established in the available evidence |
| GPU memory/bandwidth transparency | Vendor spec sheet applies across regions — H100 SXM: 80 GB, 3.35 TB/s | Same underlying NVIDIA hardware; differentiation is scheduling, not silicon |
| Data residency | Region selection only; EU AI Act/GDPR compliance is the customer’s review burden | EU-jurisdiction processing built into the platform |
| Utilization risk | Billed for provisioned capacity regardless of actual use | Targets the well-under-half utilization gap directly via prediction |
The table collapses to one operating question: are you buying a capacity guarantee and a discount ladder, or a jurisdiction and a utilization fix — because AWS’s Reserved Instance and Spot tiers solve the first problem and a specialist’s scheduling layer targets the second, and few providers do both equally well.
What to verify before you commit
The available evidence does not establish a published cost-per-token benchmark comparing a named specialist GPU cloud against AWS on the same H100 or H200 configuration, so any total-cost claim in a sales conversation needs its own side-by-side test before it goes into a budget line.
Before signing, run your actual concurrency and context-length profile against the vendor’s stated memory bandwidth, confirm whether your Reserved Instance or workload-aware commitment survives a change in instance family or region, and price the egress and data-residency review cost of switching later rather than assuming it is zero.