Ray Serve: Scalable ML Model Serving & Online Inference

Basic Deployment with @serve.deployment

Install with pip install "ray[serve]". A Deployment is a class bound to an HTTP route:

import ray
from ray import serve
from starlette.requests import Request
import torch

ray.init()
serve.start()

@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_cpus": 1, "num_gpus": 0.5},
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 8,
        "target_num_ongoing_requests_per_replica": 5,
    },
)
@serve.ingress(app)  # FastAPI app as HTTP ingress
class SentimentClassifier:
    def __init__(self):
        from transformers import pipeline
        self.pipe = pipeline(
            "sentiment-analysis",
            model="distilbert-base-uncased-finetuned-sst-2-english",
            device=0,  # GPU 0
        )

    async def classify(self, request: Request) -> dict:
        body = await request.json()
        result = self.pipe(body["text"])[0]
        return {"label": result["label"], "score": result["score"]}

# Deploy and get a handle
handle = serve.run(SentimentClassifier.bind())
result = ray.get(handle.classify.remote("Ray Serve is amazing!"))
print(result)  # {"label": "POSITIVE", "score": 0.9998}

Model Composition: Pipeline DAG

Chain deployments into a pipeline where each stage processes the output of the previous:

from ray import serve
import numpy as np

@serve.deployment(ray_actor_options={"num_cpus": 1})
class TextPreprocessor:
    def preprocess(self, text: str) -> list:
        # Tokenize, clean, normalize
        return text.lower().split()

@serve.deployment(ray_actor_options={"num_gpus": 1})
class EmbeddingModel:
    def __init__(self):
        from sentence_transformers import SentenceTransformer
        self.model = SentenceTransformer("all-MiniLM-L6-v2")

    def embed(self, tokens: list) -> list:
        return self.model.encode([" ".join(tokens)]).tolist()

@serve.deployment
class PipelineOrchestrator:
    def __init__(self, preprocessor, embedder):
        self.preprocessor = preprocessor
        self.embedder = embedder

    async def predict(self, text: str) -> dict:
        # Chained async calls across deployments
        tokens = await self.preprocessor.preprocess.remote(text)
        embedding = await self.embedder.embed.remote(tokens)
        return {"embedding": embedding, "dims": len(embedding)}

# Bind deployments into a graph
preprocessor = TextPreprocessor.bind()
embedder = EmbeddingModel.bind()
pipeline = PipelineOrchestrator.bind(preprocessor, embedder)
serve.run(pipeline)

Production Autoscaling Configuration

Configure autoscaling via YAML for production deployments:

# serve_config.yaml
applications:
  - name: llm-inference
    route_prefix: /v1
    import_path: app:LLMService
    deployments:
      - name: LLMService
        num_replicas: "auto"           # enable autoscaling
        max_ongoing_requests: 10
        ray_actor_options:
          num_cpus: 2
          num_gpus: 1
          memory: 8000000000           # 8 GB RAM reservation
        autoscaling_config:
          min_replicas: 1
          max_replicas: 4
          target_num_ongoing_requests_per_replica: 3
          upscale_delay_s: 5           # fast scale-up
          downscale_delay_s: 60        # slow scale-down

# Deploy with:
# serve deploy serve_config.yaml
# Monitor with:
# serve status

Ray Serve vs BentoML vs Triton vs TorchServe