# Multiview videos ## Overview Multiview mode allows you to work with multiple videos of the same scene captured from different cameras or angles simultaneously. All videos within one dataset form a synchronized multiview group, enabling efficient annotation of complex multi-camera setups. This is particularly useful for: * **Autonomous driving** - multiple cameras mounted on a vehicle * **Surveillance** - security cameras covering the same area from different angles * **Sports analytics** - tracking athletes from multiple viewpoints * **3D reconstruction** - multi-camera setups for depth estimation ![](https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2Fgit-blob-ddf7ade55e4e53da4e02a3500887fcca559ecd47%2Fmulti-view-results.gif?alt=media) *** ## Key Concepts ### Unified Objects If you have a common object appearing across different videos (e.g., a car visible from front, left, and right cameras), you can annotate it as a **single Supervisely object with a shared ID**. * The object maintains its identity across all videos in the multiview group * When you export and re-import the project, the object is recreated as a unified entity ### Synchronized Playback All videos in a multiview group can be played back synchronously: * Navigate through frames simultaneously across all views * Configure frame offsets if videos have different starting points * Control the order of videos in the labeling interface using metadata files with `videoStreamIndex` (see "How to Create Multiview Project" section below) * Easily compare and analyze object behavior from multiple angles in real-time *** ## How to Create Multiview Project ### Via UI (Drag & Drop) 1. Prepare your data according to the required structure (see below): * Create a folder for each multiview group (dataset) * Place all camera videos inside the corresponding folder ``` 📂 project └── 📂 ds1 ├── 🎥 aerial_scene_1.mp4 └── 🎥 ground_scene_2.mp4 ``` ![](https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2Fgit-blob-d394b91bc320142ee62b2c626c871753adf67b94%2Fdata_structure.jpg?alt=media) 1. Go to your workspace and start from creating a new project (`+ New` ⇨ `New Project Wizard`) 2. Select `Videos` ⇨ `Multi-view` labeling interface and proceed 3. Press `Import` to open the import app wizard 4. Drag and drop your prepared folder or archive 5. The import app will automatically group videos by datasets {% embed url="" %} In this example we use videos from the [MAVREC Dataset](https://mavrec.github.io) (CC BY 4.0) annotated by Supervisely team.
Citation for MAVREC Dataset ```txt InProceedings{Dutta_2024_CVPR, author = {Dutta, Aritra and Das, Srijan and Nielsen, Jacob and Chakraborty, Rajatsubhra and Shah, Mubarak}, title = {Multiview Aerial Visual RECognition (MAVREC): Can Multi-view Improve Aerial Visual Perception?}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2024}, pages = {22678-22690} } ```
For detailed import format specification, see [Multiview Import Format](https://docs.supervisely.com/import-and-export/import/supported-annotation-formats/videos/multiview). ### Via Python SDK ```python import supervisely as sly # Initialize API api = sly.Api.from_env() # Create a new project with multiview settings project = api.project.create( workspace_id=WORKSPACE_ID, name="My Multiview Project", type=sly.ProjectType.VIDEOS, change_name_if_conflict=True ) # Update project settings to enable multiview api.project.set_multiview_settings(project.id) # Create a dataset (each dataset = one multiview group) dataset = api.dataset.create(project.id, "ds1") # Upload videos to the dataset video_paths = [ "/path/to/aerial_scene_1.mp4", "/path/to/ground_scene_2.mp4" ] for video_path in video_paths: api.video.upload_path( dataset_id=dataset.id, name=sly.fs.get_file_name_with_ext(video_path), path=video_path ) ``` Open the project in the Supervisely UI to start annotating in multiview mode. ![](https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2Fgit-blob-a13ef24d673cd8a3b568e91f8aac683fb2a8a035%2Ftoolbox-interface.png?alt=media) ### Optional: Specify Video Order To control the order of videos in the multiview labeling interface, you can create a metadata JSON file for each video with the suffix `.meta.json`. This file should include the `videoStreamIndex` to define its position. For example, for a video named `aerial_scene_1.mp4`, create a file named `aerial_scene_1.mp4.meta.json` with the following content: ```json { "videoStreamIndex": 0 } ``` ``` 📂 project_name ┣ 📂 ds1 ┃ ┣ 🎥 aerial_scene_1.mp4 ┃ ┣ 🎥 aerial_scene_1.mp4.meta.json # ⬅︎ meta file with videoStreamIndex ┃ ┣ 🎥 ground_scene_2.mp4 ┃ ┗ 🎥 ground_scene_2.mp4.meta.json # ⬅︎ meta file with videoStreamIndex ┗ 📂 ds2 ┣ 🎥 camera_front.mp4 ┣ 🎥 camera_left.mp4 ┗ 🎥 camera_right.mp4 ``` *** ## Labeling in Multiview Mode ### Interface Overview The multiview labeling interface includes the following key elements: * **Video panels & Synchronized timeline** - each video is displayed in its own panel with a shared timeline for synchronized navigation: * **Objects & Tags panel** - shows objects across all videos and video-specific tags: ![](https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2Fgit-blob-2ff6605b4121ed0b35066d733fdac7ca0b3fd9b7%2Fmulti-view-objects.png?alt=media) * **Multiview settings** - configure frame offsets to align videos temporally to ensure synchronized playback. You can specify number of frames or time in milliseconds for each video: ![](https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2Fgit-blob-6d624ba2f13f7894ea59ba3214eadf706ef52eb7%2Fmulti-view-settings.jpg?alt=media) ### Annotating Objects {% hint style="info" %} Note: When annotating in multiview mode, you can create unified objects across videos, but tags remain video-specific. {% endhint %} 1. **Create an object** on any video using annotation tools (rectangle, polygon, etc.) 2. **The same object** can be annotated on other videos in the group 3. Objects with the same ID are linked across all videos {% embed url="" %} ### Video-specific Tags Unlike objects, **tags apply only within a specific video**: * When you tag a figure, frame, or video, that tag is associated only with that particular video * Tags are displayed only on the video where they were created * This allows for view-specific annotations (e.g., "occluded" tag on one camera angle) ### Auto-tracking {% hint style="success" %} To speed up the annotation process, you can use the **Auto Track** app to automatically track objects across multiple videos simultaneously. {% endhint %} To use auto-tracking in multiview mode: 1. **Open a multiview video project** and navigate to the desired frame. 2. **Configure the tracking settings** in the tracking tool: * Set number of frames to track * Choose direction (forward/backward) * Select the tracking engine ([Auto Track](https://app.supervisely.com/ecosystem/apps/supervisely-ecosystem/auto-track) app) * Enable/Disable automatic tracking ![](https://1080806899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4BHwRbuyIoH-xoF3Gv%2Fuploads%2Fgit-blob-a0e6d89d908e5a92858d71de6158227ccf05082a%2Fauto-track-settings.jpg?alt=media) 3. **Annotate the object** on one of the videos and start the tracker. 4. After tracking is complete on one video, **switch to another video** in the multiview interface. 5. **Create a new figure** for the same object and press `Alt + Space` (or `Option + Space` on Mac) to complete the figure. This will trigger the tracker to extend the annotation for the same number of frames as before. {% hint style="info" %} There are two options for using auto-tracking: enabling automatic tracking when annotating or editing an object, or manually running the tracker using the hotkey `Shift + T`. If automatic tracking is enabled, the tracker runs automatically on all figures of objects across all videos. {% endhint %} {% embed url="" %} Simple example result of auto-tracking in multiview mode: {% embed url="" %} *** ## Export Use the **Export Videos in Supervisely Format** app to export your multiview project: * All videos are exported with their annotations in Supervisely format * Object relationships across videos are preserved via shared object keys * Metadata files contain `videoStreamIndex` for maintaining video order * Project meta include settings for multiview configuration The exported structure can be re-imported to recreate the exact same multiview setup. *** ## Use Cases | Use Case | Description | | ---------------------- | ------------------------------------------------------------------------------- | | **Autonomous Driving** | Annotate objects visible from front, rear, and side cameras as unified entities | | **Surveillance** | Track people or vehicles across multiple security cameras | | **Sports Analytics** | Follow athletes from different camera angles for comprehensive analysis | | **Retail Analytics** | Monitor customer behavior from multiple store cameras | | **3D Reconstruction** | Annotate corresponding points across stereo or multi-camera setups | *** ## Useful Links * [Multiview Import Format Specification](https://docs.supervisely.com/import-and-export/import/supported-annotation-formats/videos/multiview) * [Video Annotation Format](https://docs.supervisely.com/customization-and-integration/00_ann_format_navi) * [Export Videos in Supervisely Format](https://ecosystem.supervisely.com/apps/export-videos-project-in-supervisely-format) * [Video Labeling Toolbox](https://docs.supervisely.com/labeling/labeling-toolbox/videos)