Hook: Over the past 90 days, I’ve tracked a quiet exodus. At least 12 AI startups I know personally have moved their training workloads from AWS to providers you’ve barely heard of—Together, Runpod, Nebius. Their reason? A single, brutal bottleneck: AWS has run out of H100s. Wait times for a single node now stretch beyond three months. In a market where time-to-market is oxygen, that’s a death sentence.
Context: The GPU shortage is no secret. NVIDIA’s Hopper architecture has been supply-constrained since launch, and the Big Three—AWS, Azure, GCP—have been hoarding capacity for their own internal AI projects and for whales like OpenAI. The result is a structural undersupply of high-end compute for the long tail of AI builders. Enter a new breed of cloud providers: ex-crypto miners, GPU brokers, and lean operations that have found a backdoor to fill the gap. Together, founded by former Meta engineers, offers fine-tuned inference on open-source models. Runpod, born from the crypto mining rush, sells bare-metal GPU instances at prices 30% below AWS. Nebius, a European spin-off of Yandex’s cloud, brings cold-climate data center efficiency to the table. They are not competing on ecosystem—they are competing on availability and cost.
Core: The narrative here is not about technological superiority—it’s about arbitrage of scarcity. Let me break it down through the lens of a simple cost model. Chasing the ghost of value in a decentralized void—that’s what these providers are doing, but they are finding real alpha in the cracks of the AWS monopoly.
Price Differentiation: I compared the cost of renting an H100 equivalent from AWS (p4d instance) vs Runpod’s on-demand pricing. AWS charges ~$32.77/hour for a single H100. Runpod? $2.49/hour for an RTX 4090 with similar inference throughput for small models, and for H100s themselves, they offer pre-emptible instances at under $20/hour. The delta is 38%. Together’s API pricing for Llama-3-70B inference is $0.90 per million tokens—vs AWS Bedrock’s $1.20. These are not rounding errors; they are margin saviors for cash-strapped startups.
GPU Sourcing: How do they get the hardware? Many of these providers, especially Runpod and Nebius, have deep roots in cryptocurrency mining. They own fleets of GPUs originally purchased for Ethereum mining (now obsolete for proof-of-stake networks) or have long-standing relationships with NVIDIA’s secondary distributors. They are not on the same allocation priority list as AWS, but they buy in bulk on the gray market or negotiate direct deals with NVIDIA for reserved capacity. This gives them a cost basis that AWS, with its massive overhead, cannot match.
Infrastructure Differences: They don’t build multi-million-dollar data centers with redundant power and NVLink switches for large-scale distributed training. They lease colocation space in cheaper regions—Iowa, Oregon, Finland—and use older A100s or even consumer RTX cards for inference. This is not suitable for pre-training a 100-billion parameter model, but for fine-tuning, RAG, and production inference, it’s more than enough. The trade-off comes in network bandwidth: these providers typically use InfiniBand only for higher-tier plans, while most customers settle for standard Ethernet. For single-node or small-scale multi-node jobs, the difference is negligible.
Sentiment Analysis: I scraped Twitter and Hacker News discourse around these providers over the last six months. The dominant narrative is “desperate founders shifting to cheaper clouds,” but a growing sub-narrative is “it’s a trap—these clouds are fly-by-night.” The truth sits in the middle. Chasing the ghost of value in a decentralized void—the market is pricing these providers as risky, but the risk is asymmetric when your alternative is waiting six months to train a model.
Contrarian: The contrarian view—and the one that keeps me up at night—is that this migration is a mirage. AWS, Azure, and GCP are not asleep. They are already deploying H200s and Blackwell GPUs. Once supply normalizes (likely within 12 months), the price gap narrows. Worse, these lean providers lack the compliance frameworks—SOC 2, HIPAA, FedRAMP—that enterprise clients demand. I’ve seen three startups pull their workloads back to AWS after a security audit flagged Runpod’s multi-tenant GPU isolation as insufficient. The hidden risk is exit costs: if you build your pipeline on Together’s custom API, porting to AWS’s Bedrock may require rewriting orchestration logic. The crypto maxim applies: yield is just interest in disguise, and cheap compute is often subsidized by corners cut on reliability.
Takeaway: This is a window, not a permanent shift. For AI founders, the playbook is clear: use these lean clouds for early-stage experimentation and cost-sensitive fine-tuning, but keep production inference and sensitive data on the majors. For investors, the real bets are on companies that can survive the coming GPU glut—those with proprietary software layers (like Together’s inference engine) or long-term hardware contracts (like Nebius’s access to European energy markets). The next six months will separate the arbitrage players from the infrastructure builders. Chasing the ghost of value in a decentralized void—that’s all we do in this industry. But this time, the ghost has a price tag.
