Quality control

CVAT has the following features for automated quality control of annotations:

• Validation set configuration for a task
• Job validation on job finish ({{< ilink "/docs/qa-analytics/immediate-feedback" "Immediate feedback" >}})
• Review mode for problems found
• Quality analytics

In this section, we highlight only the key steps in quality estimation. Consult the detailed guide on quality estimation in CVAT in the {{< ilink "/docs/qa-analytics/auto-qa" "Advanced section" >}}.

How to enable quality control

{{< tabpane text=true >}}

{{%tab header="In a new task" %}}

1. Go to task creation

2. Select the source media, configure other task parameters

3. Scroll down to the Quality Control section

4. Select one of the {{< ilink "/docs/qa-analytics/auto-qa#validation-modes" "validation modes" >}} available

5. Create the task

6. Upload or create Ground Truth annotations in the Ground Truth job in the task

7. Switch the Ground Truth job into the acceptance stage and completed state

{{% /tab %}}

{{%tab header="In an existing task" %}}

{{% alert title="Note" color="primary" %}} For already existing tasks only the Ground Truth validation mode is available. If you want to use Honeypots for your task, you will need to recreate the task. {{% /alert %}}

1. Open the task page

2. Select the + button next to the job list

3. Select Job Type Ground truth and configure the job parameters

4. Upload or create Ground Truth annotations in the Ground Truth job in the task

5. Switch the Ground Truth job into the acceptancestage and completed state

{{% /tab %}}

{{< /tabpane >}}

How to enable immediate job feedback

{{% alert title="Note" color="primary" %}} This feature requires a configured validation set in the task. Learn more in How to enable quality control and in the {{< ilink "/docs/qa-analytics/auto-qa#configuring-quality-estimation" "full guide" >}}. {{% /alert %}}

1. Open the task Actions menu > Quality control > Settings

2. Set Max validations per job to above zero. 3 is a good starting number

3. Save the updated settings

4. Assign an annotator to an annotation job

5. Annotate the job

6. Mark the job finished using the corresponding button in the menu

7. Once the job is completed, you'll see the job validation dialog

Each assignee gets no more than the specified number of validation attempts.

Learn more about this functionality in the {{< ilink "/docs/qa-analytics/immediate-feedback" "Immediate Feedback" >}} section.

How to check task quality metrics

1. Open the task Actions menu > Quality control

2. (Optional) Request quality metrics computation, and wait for completion

3. Review summaries or detailed reports

Learn more about this functionality {{< ilink "/docs/qa-analytics/auto-qa#quality-analytics" "here" >}}.

How to review problems found

1. Open the task Actions menu > Quality control
2. Find an annotation job to be reviewed, it must have at least 1 validation frame
3. Select the job link
4. Switch to the Review mode
5. Enable display of Ground Truth annotations and conflicts

Learn more about this functionality {{< ilink "/docs/qa-analytics/auto-qa#reviewing-gt-conflicts" "here" >}}.