Step 17 of 22
Deployment & hosting
Put models, agents and apps on infrastructure that scales. A model that works on a laptop still has to survive real traffic. Deployment decides cold starts, scaling and who holds your data.
How to choose
- Custom models and pipelines without running Kubernetes: a serverless GPU platform.
- Popular open models behind an API: a hosted model marketplace.
- Apps and agents on your existing cloud: the cloud provider's own tooling, driven by your agent through MCP.
What to look for
The questions that separate the options, in order of how much they matter.
01
Bring your own model, or pick from a catalogue
Decide first whether you are deploying your own weights and code or calling someone else's hosted models. GPU platforms (Modal, Runpod, Baseten) run any container or Python code you give them. Marketplaces (Replicate, fal) are strongest when the model you want is already in their catalogue. Cloud model services (Amazon Bedrock, Google's Agent Platform) are catalogue-first; Bedrock's Custom Model Import, for example, accepts only a few architectures. A fine-tuned or unusual model narrows the field quickly.
02
Serverless or dedicated GPUs
Serverless workers start when requests arrive and stop when traffic ends, so you pay nothing while idle but meet cold starts. Dedicated GPUs stay on, cost money every hour and answer immediately. Spiky or low traffic suits serverless; steady traffic is cheaper on dedicated or reserved capacity. Most platforms offer both: Runpod has flex and always-on active workers, Baseten bills dedicated GPUs by the minute with no idle charge, and the cloud services sell provisioned throughput.
03
Cold starts, measured on your model
A cold start is the time to start a container and load weights into GPU memory before the first answer. Vendors quote best cases, such as Runpod's sub-200 ms FlashBoot claim, but load time grows with model size and image size. Ask whether cold-start time is billed (on Runpod it is), how long an idle worker waits before shutting down, and whether you can keep a minimum number of warm workers.
04
Pricing model and the multipliers
Compare like with like. GPU platforms bill per second or minute of hardware; marketplaces often bill per output image or second of video; model services bill per million tokens, with batch tiers at about half price and priority tiers above list. Then read the small print: Modal charges 1.15x to 1.75x for region pinning and 3x for non-preemptible capacity, and Google's regional endpoints cost about 10% more than global ones. For a sustained workload, work out a monthly bill under each pricing model before choosing.
05
GPU availability and types
Check that the GPU you need is listed and actually available in the quantity you need. An H100 ranges from about $2.69 an hour on Runpod's Community Cloud to $6.50 on Baseten dedicated, and B200s are now common at the top end. Cheaper community or spot capacity can be interrupted. For training or very large models, check for multi-GPU nodes with fast interconnect such as InfiniBand.
06
Data residency and compliance
Ask where requests are processed, whether your data trains anyone's model, and which certifications cover the exact product tier you will use. Amazon Bedrock states prompts are not used to improve base models and is HIPAA eligible; Runpod holds SOC 2 Type II and ISO 27001 but its compliance page does not say whether Community Cloud hosts are in scope, and its own guidance sends regulated data to single-tenant Secure Cloud. Baseten Enterprise can run inside your own cloud account. If data must stay in a region, confirm region pinning is available and what it costs.
07
Autoscaling controls
Good autoscaling lets you set minimum and maximum replicas, a concurrency target per replica and a scale-down delay. Without those, you either overpay for idle GPUs or drop requests during a spike. Check the default concurrency limits on free and entry plans, which are often low.
08
Lock-in
Portable deployments are plain containers and OpenAI-compatible endpoints. Lock-in grows with platform-specific SDKs, packaging formats such as Replicate's Cog, and higher-level services such as Bedrock Agents and Knowledge Bases or Google's Agent Runtime and Memory Bank. That can be a fair trade, but price in the cost of leaving before you adopt them.
How the options differ
Serverless GPU platforms (Modal, Runpod, Baseten) run your own code or container on GPUs that scale with traffic. Modal is code-first Python, Runpod is the cheapest and closest to raw GPUs, and Baseten is built around reliable production endpoints with per-token Model APIs alongside. Model marketplaces (Replicate, fal, Hugging Face) put a catalogue of ready-made models behind one API and also let you deploy your own; Replicate and fal lean towards image, video and audio, while Hugging Face pairs the hub with dedicated Inference Endpoints and a router to partner providers.
Hyperscaler model services (Amazon Bedrock, Google's Agent Platform, formerly Vertex AI) sell frontier and open models per token inside your existing cloud account, with its identity, networking, compliance and billing. They are the easiest route through enterprise procurement and the hardest to leave. The cloud providers' MCP servers listed on this stage are not alternatives to these platforms; they let an agent operate whichever infrastructure you choose.
8 tools, compared
Ranked by our rating. Open any review for sources, prices and the full verdict.
| Tool | Rating | Our verdict | Best for | Price |
|---|---|---|---|---|
| Hugging Facetool | 4.7/5 | Indispensable as a model hub, competitive but unremarkable as a host, and soon to be owned by the largest seller of AI chips. | Engineers and research teams who find, share, test or fine-tune open models, and organisations that need a private model registry. | Free account (100GB private storage, no included inference credit); PRO $9/month ($2 compute credit); Team $20/user/month; Enterprise $50/user/month. Spaces GPUs from $0.40/hour; Inference Endpoints from $0.03/hour (CPU), H100 $4.50/hour. |
| Modaltool | 4.5/5 | Modal turns a Python function into a GPU service with less ceremony than any cloud we have read about, and charges a premium the moment you need guarantees. | Python developers and ML teams who want to serve custom models or run GPU jobs without managing servers. | Starter $0 + compute ($30/month credit, 3 seats, 10 GPUs at once); Team $250/month + compute ($100 credit, unlimited seats, 50 GPUs); Enterprise custom. Per second: H100 about $3.95/hour, A100 80GB about $2.50/hour, B200 about $6.25/hour. |
| faltool | 4.4/5 | If image or video generation sits at the centre of your product, fal is where the new models arrive first and the per-output prices are easiest to budget. | Product teams adding image or video generation to apps who want many models behind one API. | Per output, e.g. Nano Banana about $0.04/image, Nano Banana Pro $0.15/image, Kling v3 Pro $0.14/second of video, FLUX.1 [schnell] $0.003/megapixel. GPUs: H100 $4.50/hour list (from $2.49), B200 $7.99/hour (from $5.49). Purchased credits last 365 days. Enterprise by quote. |
| Runpodtool | 4.3/5 | Runpod is where cheap GPU hours live, provided you are happy to package your own containers and to check which cloud tier your compliance actually covers. | ML engineers who want cheap GPUs for training, fine-tuning or a self-managed inference endpoint and can work with Docker. | Pods per hour: H100 SXM $3.99 Secure / $2.69 Community; A100 SXM $1.79 / $1.39; B200 $7.99 / $5.98. Serverless per second: H100 about $4.79/hour, A100 80GB $2.72/hour, 4090 $1.10/hour. Storage from $0.05/GB/month. |
| Amazon Bedrocktool | 4.3/5 | For a company whose data already lives in AWS, Bedrock turns Claude and the major open models into another IAM-governed service; for everyone else it is a long pricing page and a quota system to learn. | Companies already on AWS that need Claude or open models with enterprise data controls and one bill. | Per 1M tokens on demand, e.g. DeepSeek V3.2 $0.62 in / $1.85 out (US), Qwen3 Next 80B $0.15 / $1.20 (US). Batch and Flex 50% off; Priority +75%. Provisioned throughput per model unit per hour. Custom Model Import $0.05718 per model-unit minute (US East and West). Guardrails from $0.10 per 1,000 text units. |
| Basetentool | 4.2/5 | Baseten sells production inference with the compliance paperwork already done, and charges a clear premium for GPUs you could rent cheaper elsewhere. | Companies serving custom or fine-tuned models in production that need zero data retention, HIPAA and the option to run in their own cloud. | Basic $0/month, pay as you go; Pro and Enterprise by quote. Model APIs per 1M tokens, e.g. GPT OSS 120B $0.10 in / $0.50 out, Kimi K3 $3 / $15. Dedicated GPUs per minute: T4 about $0.63/hour, A100 about $4.00/hour, H100 about $6.50/hour, B200 about $9.98/hour. |
| Replicatetool | 4.2/5 | Replicate is where a developer goes to try a new image or video model this afternoon, and where the bill starts to hurt once one model runs all day. | Developers adding image, video or audio generation to an app without running their own GPUs. | Pay per output (e.g. FLUX 1.1 Pro $0.04/image, FLUX Schnell $3 per 1,000 images) or per second of hardware: T4 $0.81/hour, L40S $3.51/hour, A100 80GB $5.04/hour, H100 $5.49/hour, 8x H100 $43.92/hour. No free tier listed; volume discounts for large spend. |
| Google Agent Platform (Vertex AI)tool | 4.2/5 | Gemini lands here first and Claude shares the invoice, but two promotional prices end within ten weeks and the rename has split the documentation in two. | Teams on Google Cloud, or anyone standardising on Gemini, who want enterprise controls and Claude under one account. | Per 1M tokens, e.g. Gemini 3.8 Flash $0.75 in / $3.75 out (to 31 Dec 2026, then $1.50 / $7.50), Gemini 4 Argon $2 / $10 (to 30 Nov 2026, then $4 / $20), Gemini 3.1 Pro Preview $2 / $12, Claude Sonnet 5.5 $2 / $10. Non-global endpoints cost 10% more; batch is half price. Provisioned throughput $2.74 to $7.14 per GSU-hour by term. |
Side by side
The specifications that matter for this step. Checked 2026-10-23; confirm on the vendor’s site before you commit.
| Tool | Type | Billing basis | Bring your own model | H100 on demand | Data residency / compliance |
|---|---|---|---|---|---|
| Hugging Face | Hub + Inference Endpoints | Per hour (endpoints), per token (router) | Yes, any Hub model | From $4.50/hour | Not published |
| Modal | Serverless GPU (Python SDK) | Per second of compute | Yes, any Python code | About $3.95/hour | Region pinning at 1.15x–1.75x |
| fal | Media model API + GPUs | Per output or per GPU-hour | Yes, on rented GPUs | $4.50/hour list, from $2.49 | Not published |
| Runpod | GPU cloud: pods + serverless | Per second (serverless), per hour (pods) | Yes, any container | $2.69–$3.99/hour (pods) | 31 regions; SOC 2, HIPAA (Secure Cloud) |
| Amazon Bedrock | Hyperscaler model service | Per token or provisioned hourly | Limited (Llama, Mistral, Flan) | Not applicable | AWS regions; HIPAA eligible |
| Baseten | Dedicated GPUs + Model APIs | Per minute GPU or per token | Yes | About $6.50/hour | Enterprise can run in your cloud |
| Replicate | Model marketplace + Cog | Per output or per second | Yes, packaged with Cog | $5.49/hour | Not published |
| Google Agent Platform (Vertex AI) | Hyperscaler model service | Per token or provisioned GSU | Tuning; open models in Model Garden | Not applicable | Regional endpoints at about +10% |
Connect it to your agent
MCP servers that let assistants like Claude, ChatGPT and Cursor use these tools directly.
Where this stage fits
- Chat with your documents: we would start with Modal.
- Customer support agent: we would start with AWS MCP Server.
- Assistant with long-term memory: we would start with Cloudflare MCP Server.
- Image generation pipeline: we would start with fal, Replicate.
- Video production: we would start with fal.
- Voice agent: we would start with Modal.
- Document processing: we would start with AWS MCP Server.
- Your own fine-tuned model: we would start with Modal, Baseten.