Supervisely Serving Apps

Overview

This section covers the deployment and inference of models using Supervisely Serving Apps. This is a simple way to deploy a model on the Supervisely Platform with a convenient web UI. It allows you to deploy both the pretrained models, such as YOLOv8, and your own models trained in Supervisely.

In this guide, you'll learn how to run a Serving App, deploy models and make predictions.

Supervisely Serving Apps

You can interact with the deployed model in different ways:

1. Apply model in platform: You can predict easily with NN Applying apps. They allows you to predict the entire projects or datasets: Apply NN to Images, Apply NN to Videos.
2. NN Image Labeling: You can run the NN Image Labeling app right in the labeling interface and annotate images with the help of a deployed model.
3. Inference via API: Each model deployed in the Supervisely Platform acts like a server and ready to process inference requests via API. You can communicate with the model using Supervisely SDK (see the Inference API Tutorial), which has convenient methods for efficient model inference on images, projects, and videos. We also support Tracking-by-detection algorithms, such as BoT-Sort.

How to Deploy a Model

1. Run Serving App

To deploy a model, you need to run a Serving App. You can search an app on the Ecosystem, or find available serving apps in the Neural Networks section of the platform.

Find your app

Neural Networks Section

Run app

Run an app by clicking the "Run App" button, and the app interface will be opened in a new tab. In the first time, the app will download a docker image, it may take some time. After the app is started, you'll see the Serving GUI.

Serving App Interface:

Serving App Interface

If the app did not open automatically, you can find it in the "Apps" section of the platform.

Open the app from the Apps section

Apps Section

2. Deploy Model

After you've opened the app, you'll see the app's interface with the following sections:

1. "Pretrained" and "Custom" Tabs: In the pretrained tab, you can find the list of available pretrained checkpoints to select one. In the custom tab, you can select your own model trained in Supervisely.

2. Device Selector: Choose the device to load the model on.

3. Serve Button: Click the "Serve" button to start the deployment process. The app will download model weights, load it on device, and start the FastAPI server for API requests.

After the model is deployed, you'll see a green checkmark with text "Model has been successfully loaded on device", and the "Full model info" section will be filled with the model's information.

Model Deployed

How to Predict

Apply Model in Platform

After the model is deployed, you can apply it using the NN Applying apps. They allow you to predict the entire projects or datasets:

• Apply NN to Images
• Apply NN to Videos
• Apply Classifier to Images
• Apply Florence-2 + SAM 2 to Images

Search "apply" to find available apps:

Apply Apps

NN Image Labeling

Run NN Image Labeling

Open Labeling Tool and run the NN Image Labeling app from the "Apps" panel. The app will be opened in the labeling interface.

NN Image Labeling

Connect to the deployed model

In the app interface, select the deployed model from the list. Click the "Connect" button. After that, you can start annotating images with the help of the model.

Apply Model in Labeling

Inference via API

You can automate the inference process by sending requests to the deployed model via API. A running Serving App acts like a server and can process inference requests.

See the Inference API Tutorial for more details.

import os
from dotenv import load_dotenv
import supervisely as sly

# Get your Serving App's task_id from the Supervisely platform
task_id = 27209

# init sly.Api
load_dotenv(os.path.expanduser("~/supervisely.env"))
api = sly.Api()

# Create Inference Session
session = sly.nn.inference.Session(api, task_id=task_id)

prediction = session.inference_image_id(image_id=123)

predictions = session.inference_project_id(project_id=123)