Image Consensus
Have several annotators label each image independently in a Labeling Queue, measure how well they agree, and let reviewers settle every difference in the consensus view.
Image Consensus is a mode of Labeling Queues for image projects. It works like Video Consensus: each image is labeled independently by several annotators. When the last of them submits it, the platform compares their versions, scores the agreement and creates a pre-filled Result. A reviewer then settles every difference in the consensus view and accepts the image. The final labels go to a separate dataset, and the queue statistics show how well each annotator agrees with the others.
Consensus is built into the platform, so you do not need to install or launch an app.
How it works
A manager creates a Labeling Queue for an image project, turns on Enable consensus and chooses how many annotators label each image (2 to 4).
The queue gives every image to that many different annotators. Each annotator sees only their own labels and tags.
When the last annotator submits an image, the platform matches their labels, lists the differences (discrepancies) and scores the agreement. It copies the image to the final dataset
<dataset> (consensus)with a pre-filled Result: the labels most of the annotators agree on.A reviewer opens the image in the consensus view, settles each discrepancy and clicks Accept.
The final dataset contains only the reviewed labels. The annotators' original labels remain in the source dataset.
Step 1. Create a consensus queue
Create a Labeling Queue as usual (see Labeling Queues), select an image project and dataset, and turn on Enable consensus in the Consensus section.

Annotators per image
2
How many different annotators label each image, from 2 to 4.
IoU threshold
0.5
How much two annotators' labels must overlap to count as the same object, from 0.1 to 0.95. It is under Advanced settings.
Images have no Frame tolerance. The other rules are the same as for videos: Assign to needs at least as many users as Annotators per image, reviewers should not be the queue's annotators, and Quality Check, re-annotation and multi-view are not available with consensus.
Step 2. Annotators label
Annotators label and submit images in their jobs as in any other queue. They see only their own labels and image tags, even if the project allows several tags of one kind per image. When the last annotator submits an image, the platform scores it in the background, and the image appears in the queue statistics as Ready for review.
Step 3. Review the discrepancies
A reviewer opens their job in the queue. Images from consensus queues open in the consensus view:

Annotator views show each annotator's labels and image tags, read-only, in their own colour. The chip shows the annotator's agreement on this image; hover over it to see the components.
Result is the final version, and you can edit it like any annotation. The Overlay button draws the annotators you pick on top of each other in one view.
Discrepancies lists everything the annotators disagree on. Filter it by status or type, and move through it with J / K.
Compare shows each annotator's version of the selected discrepancy and the actions:
Take <annotator> copies that annotator's version into the Result and marks the discrepancy resolved: their label (or none, if they have none), their class, or their values of the tag.
Ignore leaves the Result as it is and marks the discrepancy ignored.
Reopen sets a resolved or ignored discrepancy back to open without changing the Result.
When every discrepancy is settled, click Accept to finish the image and move to the next one.
Discrepancy types
Shape
With three or more annotators, the object's labels match pairwise but some pair overlaps less than the IoU threshold.
Missing
Some annotators labeled the object and others did not.
Only one
Only one annotator labeled the object.
Class
Annotators gave the object different classes.
Object tag
Annotators gave the object different tags or values.
Image tag
Annotators gave the image different tags or values.
Step 4. Monitor agreement
Click Stats on the queue to see the Consensus card. It is the same as for videos, without the Frames component:

How agreement is calculated
Agreement uses the same rule as Video Consensus, with an image treated as a single frame:
Matching: for each pair of annotators, labels are matched one to one by IoU, ignoring the class, and only pairs with IoU at or above the IoU threshold are matched. Matched labels form a group: one real object with each annotator's version of it. Each annotator has at most one label per group, so a second, overlapping label of the same annotator is a separate Only one object.
Objects is the F1 of the two annotators' objects: shared objects over the objects of both.
Geometry is the mean IoU of the shared objects' labels, Classes the share of shared objects with the same class, and Tags the share of label tags (on shared objects) and image tags that match.
Counts are added up over pairs of annotators, images and the queue before they are divided, exactly as for videos.
In the statistics above, for example, the queue's agreement is .
Points and polylines are compared as strokes 2% of the image diagonal wide (at least 9 pixels), and alpha masks by the pixels they cover.
How the Result is pre-filled
With annotators, an object gets a label in the Result when at least annotators labeled it (with two annotators, when either did). The label is the one that overlaps the others best, copied as drawn. It gets the class most annotators chose and the label tags of the annotator who agrees best with the others. An image tag is added when at least annotators used it, with the values most of them chose. Ties go to the annotator who submitted first.
Limitations
2 to 4 annotators per image.
Multi-view (grouped images) queues and Quality Check cannot be combined with consensus.
An image is scored once, when the last annotator submits it.
Agreement is measured at one IoU threshold: two labels that overlap just below it are two Only one objects.
Last updated