Supervisely
AboutAPI ReferenceSDK Reference
  • 🤖What's Supervisely
  • 🚀Ecosystem of Supervisely Apps
  • 💡FAQ
  • 📌Getting started
    • How to import
    • How to annotate
    • How to invite team members
    • How to connect agents
    • How to train models
  • 🔁Import and Export
    • Import
      • Overview
      • Import using Web UI
      • Supported annotation formats
        • Images
          • 🤖Supervisely JSON
          • 🤖Supervisely Blob
          • COCO
          • Yolo
          • Pascal VOC
          • Cityscapes
          • Images with PNG masks
          • Links from CSV, TXT and TSV
          • PDF files to images
          • Multiview images
          • Multispectral images
          • Medical 2D images
          • LabelMe
          • LabelStudio
          • Fisheye
          • High Color Depth
        • Videos
          • Supervisely
        • Pointclouds
          • Supervisely
          • .PCD, .PLY, .LAS, .LAZ pointclouds
          • Lyft
          • nuScenes
          • KITTI 3D
        • Pointcloud Episodes
          • Supervisely
          • .PCD, .PLY, .LAS, .LAZ pointclouds
          • Lyft
          • nuScenes
          • KITTI 360
        • Volumes
          • Supervisely
          • .NRRD, .DCM volumes
          • NIfTI
      • Import sample dataset
      • Import into an existing dataset
      • Import using Team Files
      • Import from Cloud
      • Import using API & SDK
      • Import using agent
    • Migrations
      • Roboflow to Supervisely
      • Labelbox to Supervisely
      • V7 to Supervisely
      • CVAT to Supervisely
    • Export
  • 📂Data Organization
    • Core concepts
    • MLOps Workflow
    • Projects
      • Datasets
      • Definitions
      • Collections
    • Team Files
    • Disk usage & Cleanup
    • Quality Assurance & Statistics
      • Practical applications of statistics
    • Operations with Data
      • Data Filtration
        • How to use advanced filters
      • Pipelines
      • Augmentations
      • Splitting data
      • Converting data
        • Convert to COCO
        • Convert to YOLO
        • Convert to Pascal VOC
    • Data Commander
      • Clone Project Meta
  • 📝Labeling
    • Labeling Toolboxes
      • Images
      • Videos 2.0
      • Videos 3.0
      • 3D Point Clouds
      • DICOM
      • Multiview images
      • Fisheye
    • Labeling Tools
      • Navigation & Selection Tools
      • Point Tool
      • Bounding Box (Rectangle) Tool
      • Polyline Tool
      • Polygon Tool
      • Brush Tool
      • Mask Pen Tool
      • Smart Tool
      • Graph (Keypoints) Tool
      • Frame-based tagging
    • Labeling Jobs
      • Labeling Queues
      • Labeling Consensus
      • Labeling Statistics
    • Labeling with AI-Assistance
  • 🤝Collaboration
    • Admin panel
      • Users management
      • Teams management
      • Server disk usage
      • Server trash bin
      • Server cleanup
      • Server stats and errors
    • Teams & workspaces
    • Members
    • Issues
    • Guides & exams
    • Activity log
    • Sharing
  • 🖥️Agents
    • Installation
      • Linux
      • Windows
      • AMI AWS
      • Kubernetes
    • How agents work
    • Restart and delete agents
    • Status and monitoring
    • Storage and cleanup
    • Integration with Docker
  • 🔮Neural Networks
    • Overview
    • Inference & Deployment
      • Overview
      • Supervisely Serving Apps
      • Deploy & Predict with Supervisely SDK
      • Using trained models outside of Supervisely
    • Model Evaluation Benchmark
      • Object Detection
      • Instance Segmentation
      • Semantic Segmentation
      • Custom Benchmark Integration
    • Custom Model Integration
      • Overview
      • Custom Inference
      • Custom Training
    • Legacy
      • Starting with Neural Networks
      • Train custom Neural Networks
      • Run pre-trained models
  • 👔Enterprise Edition
    • Get Supervisely
      • Installation
      • Post-installation
      • Upgrade
      • License Update
    • Kubernetes
      • Overview
      • Installation
      • Connect cluster
    • Advanced Tuning
      • HTTPS
      • Remote Storage
      • Single Sign-On (SSO)
      • CDN
      • Notifications
      • Moving Instance
      • Generating Troubleshoot Archive
      • Storage Cleanup
      • Private Apps
      • Data Folder
      • Firewall
      • HTTP Proxy
      • Offline usage
      • Multi-disk usage
      • Managed Postgres
      • Scalability Tuning
  • 🔧Customization and Integration
    • Supervisely .JSON Format
      • Project Structure
      • Project Meta: Classes, Tags, Settings
      • Tags
      • Objects
      • Single-Image Annotation
      • Single-Video Annotation
      • Point Cloud Episodes
      • Volumes Annotation
    • Developer Portal
    • SDK
    • API
  • 💡Resources
    • Changelog
    • GitHub
    • Blog
    • Ecosystem
Powered by GitBook
On this page
  • Using Labeling Queues: Step-By-Step Guide
  • Step 1. Create Labeling Queues
  • Step 2. Image Annotation
  • Step 3. Review and Quality Control
  • Labeling Queues Statistics
  • Video Tutorial

Was this helpful?

  1. Labeling
  2. Labeling Jobs

Labeling Queues

Discover the power of collaborative annotation with labeling queues. Learn how to harness teamwork for efficient and accurate data annotation.

PreviousLabeling JobsNextLabeling Consensus

Last updated 5 months ago

Was this helpful?

Labeling Queues is a systematic method for distributing and managing labeling process in a , where labeling tasks are grouped into queues and sequentially distributed among annotators. The essence of this system is that the data is not distributed between annotators, but is issued from a common queue (pool) of tasks, on a "whoever labels first" basis, until there is no more data to label. This allows to balance data annotation process within the labeling team independently of the speed of individual workers.

Learn more about Labeling Queues in

Using Labeling Queues: Step-By-Step Guide

Before you start

Ensure the following:

  • A dedicated team assigned with proper roles. Refer to our Labeling Jobs documentation for details on roles.

  • Necessary projects for annotation with defined classes, figures, and tags.

Suppose you want to train a model to recognize strawberries in images. You have 10,000 images to annotate. Follow these steps to efficiently manage the task with Labeling Queues:

Step 1. Create Labeling Queues

  1. The Manager or Reviewer creates a Labeling Queue with defined criteria.

  2. Assign Annotators and Reviewers.

  3. An empty Labeling Job is created for each Labeler and Reviewer.

  4. Once setup is complete, the team is redirected to the Labeling Queues page, where jobs are automatically distributed.

Step 2. Image Annotation

  1. Annotators retrieve images one at a time for labeling.

  2. After labeling, Annotators press Confirm and pull next to continue.

  3. Faster Annotators can assist slower teammates, ensuring smooth workflow.

  4. The process continues until all images are labeled or the dataset is completed.

Step 3. Review and Quality Control

  1. Reviewers sequentially verify the annotations for accuracy.

  2. Approved annotations are finalized.

  3. Incorrect annotations are sent back to the original Annotator for correction.

  4. Reviewed images do not return to the general queue but stay in the original Annotator’s job.


Labeling Queues Statistics

📊 Managers and Reviewers can view statistics for each Labeling Job in the queue by clicking the Stats button. This provides insights into team performance and helps optimize the annotation process.


Video Tutorial

Watch our concise video tutorial 🎥 to learn how to use Labeling Queues effectively in the Supervisely platform.

📝
team
Labeling Queues: Streamline Your Labeling Pipeline.