# Guides & Exams

Creating large and diverse computer vision datasets requires working with a huge labeling workforce. That, in turn, requires professional tools to educate and examine them.

Why do we need **Labeling Guides** and **Labeling Exams**? Let's consider several important observations:

* Usually, an annotator has concrete **domain specialization**: self-driving cars, agriculture, satellite imagery, medicine, and so on
* Not every annotator **properly understands** labeling requirements, thus sometimes they make **systematic errors** during labeling
* It is a good idea to choose for your custom task the most efficient and accurate labelers based on **Quality Score**
* From labelers perspective, comprehensive annotation policy should be presented as **easy to learn guides and materials**
* From time to time company changes annotation policy, all labelers have to be **notified and re-examined**

**Labeling Guides** and [**Labeling Exams**](https://ecosystem.supervisely.com/apps/exams) in Supervisely are designed to handle all these aspects.

## Labeling Guides

**Labeling Guides** are videos, documents and markdown posts. You can pin here your annotation policy, examples of good and bad labeling and so on. **Labeling Guides** can later be attached to Labeling Jobs and Exams.

<figure><img src="https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2FUVRRqpv3lo2bIj3zSO6O%2Fguides.png?alt=media&#x26;token=e86fc69a-50d2-45b0-832f-70018431cd09" alt=""><figcaption></figcaption></figure>

<figure><img src="https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2FkQ6If3P90G3yktCGlPVY%2Fcreate-guides.png?alt=media&#x26;token=3c1c11e6-a7d5-4f92-804f-5794ceb6beaa" alt=""><figcaption></figcaption></figure>

## Labeling Exams

We recommend using exams via the new [app in Ecosystem](https://ecosystem.supervisely.com/apps/exams). It has new features, improvements, and can be customized by request.

Measure how labelers understand task. A manager sees the table with exams and their general information.

The row can be expanded and the internal table with all labelers' attempts will be displayed. That table contains **Quality Scores** and links to **Exam Reports**.

<figure><img src="https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2FE3lpPEfGtw9MdN3xTTQM%2Fcreate-exams.png?alt=media&#x26;token=bc41193d-e2dc-447f-8825-056df5ffc172" alt=""><figcaption></figcaption></figure>

This view allows to see the big picture and monitor the entire examination process. Also, it helps to dig into details and see how every labeler performs.

### Exam Report

Detailed error analysis at your fingertips. There are several tables with **geometry score**, **objects count score** and **tags score** to calculate more complete **Quality Score**. Also, there is a report table for every image with a bunch of statistics and calculated visual differences.

<figure><img src="https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2FRsE0BxkYJ4btZBdsSwua%2Fmanage-exams.png?alt=media&#x26;token=f49dc9ab-106d-4edd-95cd-3de93c5f4254" alt=""><figcaption></figcaption></figure>

### Notifications for managers and reviewers

Notifications allow you to don't miss relevant information and stay informed of what's going on.

<figure><img src="https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2FEwRoJN5aGG5Se5Gsh0Bc%2Fnotification.png?alt=media&#x26;token=9a359a21-1f30-4490-8e82-5e9a92b04d08" alt=""><figcaption></figcaption></figure>
