Guides & exams
Last updated
Was this helpful?
Last updated
Was this helpful?
Creating large and diverse computer vision datasets requires working with a huge labeling workforce. That, in turn, requires professional tools to educate and examine them.
Why do we need Labeling Guides and Labeling Exams? Let’s consider several important observations:
Usually, an annotator has concrete domain specialization: self-driving cars, agriculture, satellite imagery, medicine, and so on
Not every annotator properly understands labeling requirements, thus sometimes they make systematic errors during labeling
It is a good idea to choose for your custom task the most efficient and accurate labelers based on Quality Score
From labelers perspective, comprehensive annotation policy should be presented as easy to learn guides and materials
From time to time company changes annotation policy, all labelers have to be notified and re-examined
Labeling Guides and Labeling Exams in Supervisely are designed to handle all these aspects.
Labeling Guides are videos, documents and markdown posts. You can pin here your annotation policy, examples of good and bad labeling and so on. Labeling Guides can later be attached to Labeling Jobs and Exams.
We recommend using exams via the new app in Ecosystem. It has new features, improvements, and can be customized by request.
Measure how labelers understand task. A manager sees the table with exams and their general information.
The row can be expanded and the internal table with all labelers' attempts will be displayed. That table contains Quality Scores and links to Exam Reports.
This view allows to see the big picture and monitor the entire examination process. Also, it helps to dig into details and see how every labeler performs.
Detailed error analysis at your fingertips. There are several tables with geometry score, objects count score and tags score to calculate more complete Quality Score. Also, there is a report table for every image with a bunch of statistics and calculated visual differences.
Notifications allow you to don’t miss relevant information and stay informed of what’s going on.
