Skill v1.0.1

currentAutomated scan94/100

nousresearch/hermes-agent/modal

+5 new

──Details

PublishedAugust 28, 2026 at 07:01 AM

Content Hashsha256:93da9443caef0605...

Git SHA01740f352e2f

Bump Typepatch

Compare with v1.0.0

──Files

Files (1 file, 8.8 KB)

SKILL.md8.8 KBactive

SKILL.md · 352 lines · 8.8 KB

name: modal description: Serverless GPU cloud for ML jobs and model APIs. version: 1.0.1 author: Orchestra Research license: MIT dependencies: [modal>=1.0] platforms: [linux, macos, windows] metadata: hermes: tags: [Infrastructure, Serverless, GPU, Cloud, Deployment, Modal]

Modal Serverless GPU

Guide to running ML workloads on Modal's serverless GPU cloud platform.

When to use Modal

Use Modal when:

Running GPU-intensive ML workloads without managing infrastructure
Deploying ML models as auto-scaling APIs
Running batch processing jobs (training, inference, data processing)
Need pay-per-second GPU pricing without idle costs
Prototyping ML applications quickly
Running scheduled jobs (cron-like workloads)

Key features:

Serverless GPUs: T4, L4, A10G, L40S, A100, H100, H200, B200 on-demand
Python-native: Define infrastructure in Python code, no YAML
Auto-scaling: Scale to zero, scale to 100+ GPUs instantly
Sub-second cold starts: Rust-based infrastructure for fast container launches
Container caching: Image layers cached for rapid iteration
Web endpoints: Deploy functions as REST APIs with zero-downtime updates

Use alternatives instead:

RunPod: For longer-running pods with persistent state
Lambda Labs: For reserved GPU instances
SkyPilot: For multi-cloud orchestration and cost optimization
Kubernetes: For complex multi-service architectures

Quick start

Installation

bash

pip install modal
modal setup  # Opens browser for authentication

Hello World with GPU

python

import modal
app = modal.App("hello-gpu")
@app.function(gpu="T4")
def gpu_info():
    import subprocess
    return subprocess.run(["nvidia-smi"], capture_output=True, text=True).stdout
@app.local_entrypoint()
def main():
    print(gpu_info.remote())

Run: modal run hello_gpu.py

Basic inference endpoint

python

import modal
app = modal.App("text-generation")
image = modal.Image.debian_slim().pip_install("transformers", "torch", "accelerate")
@app.cls(gpu="A10G", image=image)
class TextGenerator:
    @modal.enter()
    def load_model(self):
        from transformers import pipeline
        self.pipe = pipeline("text-generation", model="gpt2", device=0)
    @modal.method()
    def generate(self, prompt: str) -> str:
        return self.pipe(prompt, max_length=100)[0]["generated_text"]
@app.local_entrypoint()
def main():
    print(TextGenerator().generate.remote("Hello, world"))

Core concepts

Key components

Component	Purpose
`App`	Container for functions and resources
`Function`	Serverless function with compute specs
`Cls`	Class-based functions with lifecycle hooks
`Image`	Container image definition
`Volume`	Persistent storage for models/data
`Secret`	Secure credential storage

Execution modes

Command	Description
`modal run script.py`	Execute and exit
`modal serve script.py`	Development with live reload
`modal deploy script.py`	Persistent cloud deployment

GPU configuration

Available GPUs

GPU	VRAM	Best For
`T4`	16GB	Budget inference, small models
`L4`	24GB	Inference, Ada Lovelace arch
`A10G`	24GB	Training/inference, 3.3x faster than T4
`L40S`	48GB	Recommended for inference (best cost/perf)
`A100-40GB`	40GB	Large model training
`A100-80GB`	80GB	Very large models
`H100`	80GB	Fastest, FP8 + Transformer Engine
`H200`	141GB	Auto-upgrade from H100, 4.8TB/s bandwidth
`B200`	Latest	Blackwell architecture

GPU specification patterns

python

# Single GPU
@app.function(gpu="A100")
# Specific memory variant
@app.function(gpu="A100-80GB")
# Multiple GPUs (up to 8)
@app.function(gpu="H100:4")
# GPU with fallbacks
@app.function(gpu=["H100", "A100", "L40S"])
# Any available GPU
@app.function(gpu="any")

Container images

python

# Basic image with pip
image = modal.Image.debian_slim(python_version="3.11").pip_install(
    "torch==2.1.0", "transformers==4.36.0", "accelerate"
)
# From CUDA base
image = modal.Image.from_registry(
    "nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04",
    add_python="3.11"
).pip_install("torch", "transformers")
# With system packages
image = modal.Image.debian_slim().apt_install("git", "ffmpeg").pip_install("whisper")

Persistent storage

python

volume = modal.Volume.from_name("model-cache", create_if_missing=True)
@app.function(gpu="A10G", volumes={"/models": volume})
def load_model():
    import os
    model_path = "/models/llama-7b"
    if not os.path.exists(model_path):
        model = download_model()
        model.save_pretrained(model_path)
        volume.commit()  # Persist changes
    return load_from_path(model_path)

Web endpoints

FastAPI endpoint decorator

python

@app.function()
@modal.fastapi_endpoint(method="POST")
def predict(text: str) -> dict:
    return {"result": model.predict(text)}

Full ASGI app

python

from fastapi import FastAPI
web_app = FastAPI()
@web_app.post("/predict")
async def predict(text: str):
    return {"result": await model.predict.remote.aio(text)}
@app.function()
@modal.asgi_app()
def fastapi_app():
    return web_app

Web endpoint types

Decorator	Use Case
`@modal.fastapi_endpoint()`	Simple function → API
`@modal.asgi_app()`	Full FastAPI/Starlette apps
`@modal.wsgi_app()`	Django/Flask apps
`@modal.web_server(port)`	Arbitrary HTTP servers

Dynamic batching

python

@app.function()
@modal.batched(max_batch_size=32, wait_ms=100)
async def batch_predict(inputs: list[str]) -> list[dict]:
    # Inputs automatically batched
    return model.batch_predict(inputs)

Secrets management

bash

# Create secret
modal secret create huggingface HF_TOKEN=hf_xxx

python

@app.function(secrets=[modal.Secret.from_name("huggingface")])
def download_model():
    import os
    token = os.environ["HF_TOKEN"]

Scheduling

python

@app.function(schedule=modal.Cron("0 0 * * *"))  # Daily midnight
def daily_job():
    pass
@app.function(schedule=modal.Period(hours=1))
def hourly_job():
    pass

Performance optimization

Cold start mitigation

python

# Modal 1.0 autoscaler params: scaledown_window (was container_idle_timeout).
# Input concurrency moved to the @modal.concurrent decorator.
@app.function(scaledown_window=300)  # Keep warm 5 min
@modal.concurrent(max_inputs=10)     # Handle concurrent requests per container
def inference():
    pass

Model loading best practices

python

@app.cls(gpu="A100")
class Model:
    @modal.enter()  # Run once at container start
    def load(self):
        self.model = load_model()  # Load during warm-up
    @modal.method()
    def predict(self, x):
        return self.model(x)

Parallel processing

python

@app.function()
def process_item(item):
    return expensive_computation(item)
@app.function()
def run_parallel():
    items = list(range(1000))
    # Fan out to parallel containers
    results = list(process_item.map(items))
    return results

Common configuration

python

@app.function(
    gpu="A100",
    memory=32768,              # 32GB RAM
    cpu=4,                     # 4 CPU cores
    timeout=3600,              # 1 hour max
    scaledown_window=120,      # Keep warm 2 min (was container_idle_timeout)
    retries=3,                 # Retry on failure
    max_containers=10,         # Max concurrent containers (was concurrency_limit)
    min_containers=1,          # Keep N containers warm (was keep_warm)
)
def my_function():
    pass

Modal 1.0 autoscaler renames (see the migration guide):
- container_idle_timeout → scaledown_window
- concurrency_limit → max_containers
- keep_warm → min_containers
- allow_concurrent_inputs=N → the @modal.concurrent(max_inputs=N) decorator

Debugging

python

# Test locally
if __name__ == "__main__":
    result = my_function.local()
# View logs
# modal app logs my-app

Common issues

Issue	Solution
Cold start latency	Increase `scaledown_window`, use `@modal.enter()`
GPU OOM	Use larger GPU (`A100-80GB`), enable gradient checkpointing
Image build fails	Pin dependency versions, check CUDA compatibility
Timeout errors	Increase `timeout`, add checkpointing

References

[Advanced Usage](references/advanced-usage.md) - Multi-GPU, distributed training, cost optimization
[Troubleshooting](references/troubleshooting.md) - Common issues and solutions

Resources

Documentation: https://modal.com/docs
Examples: https://github.com/modal-labs/modal-examples
Pricing: https://modal.com/pricing
Discord: https://discord.gg/modal

← v1.0.0 All versions