BentoML: Model Serving, Runners & Bento Packaging

Defining a BentoML Service

Install with pip install bentoml. A Service is a Python class decorated with @bentoml.service:

import bentoml
import numpy as np
from pydantic import BaseModel

# Save your model to BentoML's model store first:
# bentoml.sklearn.save_model("iris_classifier", clf)

class ClassifyInput(BaseModel):
    sepal_length: float
    sepal_width: float
    petal_length: float
    petal_width: float

@bentoml.service(
    resources={"cpu": "2", "memory": "1Gi"},
    traffic={"timeout": 10},
)
class IrisClassifier:
    # Model is loaded once at startup and reused across requests
    model_ref = bentoml.models.get("iris_classifier:latest")

    def __init__(self):
        import joblib
        self.model = joblib.load(self.model_ref.path_of("model.pkl"))

    @bentoml.api(input_spec=ClassifyInput)
    def classify(self, input_data: ClassifyInput) -> dict:
        features = np.array([[
            input_data.sepal_length, input_data.sepal_width,
            input_data.petal_length, input_data.petal_width
        ]])
        pred = self.model.predict(features)[0]
        prob = self.model.predict_proba(features)[0]
        return {"species": str(pred), "confidence": float(prob.max())}

Run locally with bentoml serve service:IrisClassifier. The service exposes POST /classify and a Swagger UI at /docs.

Runner API: Scaling Inference Separately

Runners decouple inference from your HTTP server, enabling independent scaling of each model component:

import bentoml
from PIL import Image
import io

# Define runners from saved models
embedding_runner = bentoml.pytorch.get("clip-vit:latest").to_runner()
classifier_runner = bentoml.sklearn.get("image-classifier:latest").to_runner()

@bentoml.service
class ImagePipelineService:
    # Runners are injected and managed by BentoML
    runners = [embedding_runner, classifier_runner]

    @bentoml.api()
    async def predict(self, image: bytes) -> dict:
        img = Image.open(io.BytesIO(image)).convert("RGB")

        # Async calls — HTTP server is never blocked
        embedding = await embedding_runner.encode.async_run(img)
        label = await classifier_runner.predict.async_run(embedding)

        return {"label": label, "embedding_dim": len(embedding)}

# Scale runners independently in bentofile.yaml:
# runners:
#   clip-vit: {replicas: 2, resources: {gpu: 1}}
#   image-classifier: {replicas: 4, resources: {cpu: 1}}

Building & Containerizing a Bento

Package your service and all dependencies into a Bento, then convert to Docker:

# bentofile.yaml — defines what goes into the Bento
service: "service:ImagePipelineService"
include:
  - "service.py"
  - "*.py"
python:
  packages:
    - torch==2.2.0
    - torchvision
    - Pillow
    - scikit-learn
docker:
  base_image: "pytorch/pytorch:2.2.0-cuda11.8-cudnn8-runtime"
  cuda_version: "11.8"

# Build the Bento (bundles model artifacts + code + deps)
$ bentoml build

# Output: Successfully built Bento(tag="image-pipeline:a3f9b2c1")

# Convert to Docker image (optimized, layered)
$ bentoml containerize image-pipeline:a3f9b2c1

# Deploy anywhere
$ docker run --gpus all -p 3000:3000 image-pipeline:a3f9b2c1

BentoML vs Ray Serve vs TorchServe vs Triton