CVAT Analytics and monitoring

CVAT Analytics suite of tools is designed to track and understand users' behavior, system performance, and for identifying potential issues in your application.

You can also visualize user activity through Grafana, and aggregate user working time by the jobs.

Gathered logs can be additionally filtered for efficient debugging.

By using analytics, you'll gain valuable insights to optimize your system and enhance user satisfaction.

CVAT analytics are available from the top menu.

Superusers and users with administrator role have access to analytics. Permission to access analytics can also be granted when editing a user on admin page by Has access to analytics checkbox.

{{% alert title="Note" color="primary" %}} CVAT analytics and monitoring are available only for on-prem solution. {{% /alert %}}

See:

• High-level architecture
• CVAT Analytics
  • Ports settings
  • Events log structure
  • Types of supported events
  • Working time calculation
  • Request id for tracking
  • Fetching event data as CSV from the /api/events endpoint
• Dashboards
  • Dashboard: All Events
  • Dashboard: Management
  • Dashboard: Monitoring
  • Dashboards setup
• Example of use

High-level architecture

The CVAT analytics is based on Vector, ClickHouse, and Grafana.

CVAT Analytics

CVAT and its analytics module can be set up locally, for self-hosted solution analytics are enabled by default.

{{% alert title="Note" color="primary" %}} For detailed CVAT installation instructions, see {{< ilink "/docs/administration/community/basics/installation" "Installation Guide" >}} or refer to the CVAT Course for installation videos. {{% /alert %}}

All analytics-related features will be launched when you start CVAT containers with the following command:

docker compose up -d

Ports settings

If you cannot access analytics on development environment, see {{< ilink "/docs/contributing/development-environment#cvat-analytics-ports" "Analytics Ports" >}}

Events log structure

Relational database schema with the following fields:

Types of supported events

Supported events change the scope of information displayed in Grafana.

Server events:

• create:project, update:project, delete:project

• create:task, update:task, delete:task

• create:job, update:job, delete:job

• create:organization, update:organization, delete:organization

• create:user, update:user, delete:user

• create:cloudstorage, update:cloudstorage, delete:cloudstorage

• create:issue, update:issue, delete:issue

• create:comment, update:comment, delete:comment

• create:annotations, update:annotations, delete:annotations

• create:label, update:label, delete:label

• export:dataset, import:dataset

• call:function

• create:membership, update:membership, delete:membership

• create:webhook, update:webhook, delete:webhook

• create:invitation, delete:invitation

Client events:

• load:cvat

• load:job, save:job

• send:exception

• draw:object, paste:object, copy:object, propagate:object, drag:object, resize:object, delete:object, merge:objects, split:objects, group:objects, slice:object, join:objects

• change:frame

• zoom:image, fit:image, rotate:image

• action:undo, action:redo

• run:annotations_action

• click:element

• debug:info

Working time calculation

Here is a short overview of how CVAT deals with the user's working time:

• The user interface collects events when a user interacts with the interface (resizing canvas, drawing objects, clicking buttons, etc) The structure of one single event is described here.

• The user interface sends these events in bulk to the server. Currently, it uses the following triggers to send events:

  • Periodical timer (~90 seconds)
  • A user clicks the "Save" button on the annotation view
  • A user opens the annotation view
  • A user closes the annotation view (but not the tab/browser)
  • A user clicks Logout button

• When events reach the server, it calculates working time based on timestamps of the events.

• The working time for an event is computed as the sum of the following:

  • The difference between the start time of the event and the end time of the previous event, if it is not more than 100 seconds.
  • The duration of the event, for events of type change:frame.

• After calculation, the server generates send:working_time events with time value in payload. These events may or may not be bound to a certain job/task/project, depending on the client-side events that were used to generate them.

• CVAT saves the event in the database and later these events are used to compute metrics for analytics.

Request id for tracking

Note, that every response to an API request made to the the server includes a header named X-Request-Id, for example: X-Request-Id: 6a2b7102-c4b9-4d57-8754-5658132ba37d.

This identifier is also recorded in all server events that occur as a result of the respective request.

For example, when an operation to create a task is performed, other related entities such as labels and attributes are generated on the server in addition to the Task object.

All events associated with this operation will have the same request_id in the payload field.

Export event data

You can export the event data as a CSV file locally and to cloud storage.

To export the data locally:

1. Initiate the export process by sending a POST request to /api/events/export endpoint. The endpoint accepts several query parameters to filter events: org_id, project_id, task_id, job_id, user_id, to and from. For more details, see Swagger API Documentation. For example:

   bashCopy

   curl -X POST -u 'user:pass' https://app.cvat.ai/api/events/export?job_id=123
   

2. You can check the status of the export process by sending a GET request with the rq_id to the /api/requests/{id} endpoint:

   bashCopy

   curl -I --user 'user:pass' https://app.cvat.ai/api/requests/rq_id
   

   Once the export process finishes, the request returns an object with "status": "finished" and "result_url": "URL".

3. Download the event data file locally using result_url:

   bashCopy

   curl -u user:password -o path/to/file.csv result_url
   

   This command will download and save the CSV file to path/to/file.csv on your local machine.

To save the CSV file with the event data to cloud storage, you can use the /api/events/export endpoint with cloud_storage_id and location=cloud_storage parameters, for example:

curl -X POST -u user:password "https://app.cvat.ai/api/events/export?cloud_storage_id=your_cloud_storage_id&location=cloud_storage"

Dashboards

By default, three dashboards are available in CVAT.

To access them, click General, you will be forwarded to the page with available dashboards.

Dashboard: All Events

The dashboard shows all events, their timestamps, and their source.

There are two fields with statistics at the bottom of the dashboard, about browser and OS users use.

Click on the column name to enable a filter.

If you want to inspect the value, hover over it and click on the eye icon.

Dashboard: Management

The dashboard shows user activity.

Click on the column name to enable a filter.

If you want to inspect the value, hover over it and click on the eye icon.

Dashboard: Monitoring

The dashboard shows server logs, helps handle errors, and shows user activity.

Click on the column name to enable a filter.

If you want to inspect the value, hover over it and click on the eye icon.

Dashboards setup

You can adjust the dashboards. To do this, click on the graph or table name and from the drop-down menu select Edit.

Adjust the query in the editor.

Example of query:

SELECT
    time,
    uniqExact(user_id) Users
FROM
(
    SELECT
      user_id,
      toStartOfInterval(timestamp, INTERVAL 15 minute) as time
    FROM cvat.events
    WHERE
      user_id IS NOT NULL
    GROUP BY
      user_id,
      time
    ORDER BY time ASC WITH FILL STEP toIntervalMinute(15)
)
GROUP BY time
ORDER BY time

{{% alert title="Note" color="primary" %}} By default the updated configuration will not be saved and will be reset to the default parameters after you restart the container. {{% /alert %}}

To save the updated configuration, do the following:

1. Update Configuration: Start by making your desired changes in the query.

2. Apply Changes: Once you've made your changes, click the Apply button to ensure the changes are implemented.

3. Save Configuration: To save your applied changes, on the top of the dashboard, click the Save button.

4. Replace Configuration File: After saving, replace the existing Grafana dashboard configuration file is located at components/analytics/grafana/dashboards with the new JSON configuration file.

5. Restart Grafana Service: To ensure, that all changes take effect, restart the Grafana service. If you're using Docker Compose, execute the following command: docker compose restart cvat_grafana.

For more information, see Grafana Dashboards.

Example of use

This video demonstrates available by default CVAT analytics features.