ContextQMD
Back to Open3d

Web visualizer and Jupyter

docs/tutorial/visualization/web_visualizer.rst

0.19.010.9 KB
Original Source

.. _web_visualizer:

Web visualizer and Jupyter

Introduction

Open3D web visualizer enables advanced rendering and visualization of 3D models in modern browsers. You first start the visualization server on a local or remote machine, and then the 3D models can be viewed from any device with a modern browser. The web visualizer server and client communicate via the WebRTC <https://en.wikipedia.org/wiki/WebRTC>_ (Web Real-Time Communication) protocol.

In this tutorial, we cover two common use cases:

  • Standalone mode: Run Open3D web visualizer server as a standalone application and view the 3D models in a browser. It supports both C++ and Python. You'll only need add one line to your code to enable the server.
  • Jupyter mode: Run Open3D web visualizer as embedded Jupyter cells. It supports both the Python Jupyter notebook and Jupyter lab environments. You'll need to modify your code to use the non-blocking APIs and wrappers.

.. image:: ../../_static/visualization/webrtc_visualizer.gif :width: 960px

As of the current release, Open3D web visualizer is supported on the following platforms:

.. Table generation: https://www.tablesgenerator.com/text_tables

+----------------------------------------------+--------+-------+---------+ | | Ubuntu | macOS | Windows | +==============================================+========+=======+=========+ | Standalone mode server (C++/Python) | Yes | Yes | Yes | +----------------------------------------------+--------+-------+---------+ | Jupyter mode server (Python) | Yes | No | Yes | +----------------------------------------------+--------+-------+---------+ | Standalone/Jupyter mode client (web browser) | Yes | Yes | Yes | +----------------------------------------------+--------+-------+---------+

Additional notes on compatibility:

  • On Ubuntu, web visualizer server runs on EGL. This is supported in most machines as long as the it has graphics capabilities (either via integrated graphics or discrete GPU). A discrete GPU will be required if the CPU does not come with integrated graphics. For example, AWS cloud instances typically use Intel XEON processors without integrated graphics. You'll need to add a discrete GPU to the instance and install the graphics driver to enable the web visualizer.
  • Web visualizer servers are not supported on the ARM platform yet. To get the web visualizer working on ARM, you'll need to compile the WebRTC library from source for ARM. See the 3rdparty/webrtc folder for more details.
  • Google Colab and Kaggle notebook are not supported. You'll need to run you own Jupyter or JupyterLab server.
  • If you prefer using native windows (with blocking calls) instead of embedded graphics in Jupyter cells, you can set the environment variable OPEN3D_DISABLE_WEB_VISUALIZER=true before importing Open3D.

Standalone mode

In standalone mode, Open3D web visualizer server runs as a standalone application. It supports both C++ and Python. You'll only need to make very minimal changes to your code to enable the server.

C++ server ::::::::::

Ensure -DBUILD_GUI=ON and -DBUILD_WEBRTC=ON are set in the CMake config step (they are set to ON by default). In your C++ code, enable the WebRTC server backend by calling:

.. code-block:: cpp

open3d::visualization::webrtc_server::WebRTCWindowSystem::GetInstance()->EnableWebRTC();

Example:

.. code-block:: sh

mkdir build && cd build
cmake ..
make DrawWebRTC -j$(nproc)
cd bin/example
./DrawWebRTC
google-chrome http://localhost:8888  # Or, open the address in your browser

Checkout examples/cpp/DrawWebRTC.cpp <https://github.com/isl-org/Open3D/blob/main/examples/cpp/DrawWebRTC.cpp>_ for the complete source code.

Python server :::::::::::::

In your Python code, enable the WebRTC server backend by calling:

.. code-block:: python

import open3d as o3d
o3d.visualization.webrtc_server.enable_webrtc()

Example:

.. code-block:: sh

python examples/python/visualization/draw_webrtc.py
google-chrome http://localhost:8888  # Or, open the address in your browser

Checkout examples/python/visualization/draw_webrtc.py <https://github.com/isl-org/Open3D/blob/main/examples/python/visualization/draw_webrtc.py>_ for the complete source code.

IP/port binding :::::::::::::::

Similar to other web servers, the Open3D web visualizer server allows binding to custom IP address and port. This is typically required for setting up remote servers.

To set custom IP address, set the WEBRTC_IP environment variable. To set custom port number, set the WEBRTC_PORT environment variable. For instance:

.. code-block:: sh

# Bind to localhost:8888 (default)
python examples/python/visualization/draw_webrtc.py

# Bind to 127.0.0.1:8889
WEBRTC_IP=127.0.0.1 WEBRTC_PORT=8889 python draw_webrtc.py

To enable remote visualization, we typically bind to the internal IP and internal port as seen by the server. The router may translate the internal IP and internal port to external IP and port. For instance, on Google cloud, we can get the internal and external IP <https://cloud.google.com/compute/docs/ip-addresses>_. of the instance. We then set the WEBRTC_IP and WEBRTC_PORT to the internal IP and port, enable the corresponding port in the firewall rules and enable HTTP traffic. Finally, we start the web visualizer server and visit the external IP address from the browser.

.. warning::

Although the WebRTC traffic is encrypted, the web server uses HTTP by
default which is not encrypted and is suitable only for local access. If
used for remote visualization, the web page will be visible and controllable
by any one with network access. For sensitive or confidential data, please
rebuild with `appropriate web server configuration and SSL certificates
<https://github.com/civetweb/civetweb/blob/master/docs/OpenSSL.md>`_.

Jupyter mode

Install Jupyter with:

.. code-block:: sh

pip install jupyter
jupyter notebook

or, you may also install JupyterLab instead:

.. code-block:: sh

pip install jupyterlab
jupyter-lab

Then, run the example notebook docs/jupyter/visualization/jupyter_visualization.ipynb <https://github.com/isl-org/Open3D/blob/main/docs/jupyter/visualization/jupyter_visualization.ipynb>_.

Unlike standalone mode, WEBRTC_IP and WEBRTC_PORT are not used in Jupyter mode. However, you might want to specify IP and port for Jupyter <https://stackoverflow.com/q/31270570/1255535>_.

.. warning::

For remote visualization, please use a `secure Jupyter server
<https://jupyter-notebook.readthedocs.io/en/stable/public_server.html>`_.

Non-blocking functions ::::::::::::::::::::::

In Jupyter mode, we need to avoid blocking visualization API calls. Otherwise, running visualization in one cell will block the execution of the next cell. We provide Jupyter-specific helper functions to achieve non-blocking visualization. For instance, open3d.web_visualizer.draw is used instead of the regular open3d.visualization.draw in the example notebook above.

Besides draw, you may also create your own non-blocking visualization helper functions with the _AsyncEventLoop class. See python/open3d/web_visualizer.py for the _AsyncEventLoop class and the implementation of open3d.web_visualizer.draw as an example.

Build Jupyter package from source :::::::::::::::::::::::::::::::::

Open3D official Python wheels come with Jupyter web visualizer support. To build Open3D Python package from source with Jupyter web visualizer, you'll need to :

  1. Install npm and yarn. For example, you may install npm via a package manager <https://nodejs.org/en/download/package-manager/>_, and install yarn by npm install -g yarn. Make sure they can be called from the terminal.

  2. Install Jupyter build dependencies:

    .. code-block:: sh

    pip install -r python/requirements_jupyter_build.txt

  3. Build Open3D Python package:

    .. code-block:: sh

    mkdir build && cd build cmake -DBUILD_JUPYTER_EXTENSION .. make install-pip-package -j$(nproc)

Advanced topic: local server in airplane mode

When the computer has no active network interfaces (e.g. Wi-Fi is turned off and ethernet is unplugged, the machine only has the loopback lo interface), WebRTC may fail to work. In this case, we need to create a dummy interface. The workaround is tested on Ubuntu.

.. code-block:: sh

# Setup
sudo ip link add dummy0 type dummy
sudo ip addr add 1.1.1.1/24 dev dummy0
sudo ip link set dummy0 up

# Check interfaces
ip addr

# Do WebRTC things here
python examples/python/visualization/draw_webrtc.py
google-chrome http://localhost:8888  # Or, open the address in your browser

# Clean up
sudo ip link set dummy0 down
sudo ip link delete dummy0

Advanced topic: TURN server

WebRTC's ICE <https://developer.mozilla.org/en-US/docs/Glossary/ICE>_ (Interactive Connectivity Establishment) framework resolves client-server connection via STUN <https://developer.mozilla.org/en-US/docs/Glossary/STUN>_ or TURN <https://developer.mozilla.org/en-US/docs/Glossary/TURN>_ servers.

In most scenarios, a STUN server is sufficient to figure out the traffic routing. In certain network configurations (e.g. behind a NAT or firewall), a TURN server is required to forward WebRTC traffic. You may add your custom TURN server (video traffic relay) by setting the WEBRTC_STUN_SERVER environment variable. If you have more than one TURN servers, separate them with ;. For instance:

.. code-block:: sh

# UDP only
WEBRTC_STUN_SERVER="turn:user:password@my_turn_server.com:3478"
# TCP only
WEBRTC_STUN_SERVER="turn:user:password@my_tcp_turn_server.com:3478?transport=tcp"
# UDP and TCP (more than one TURN servers)
WEBRTC_STUN_SERVER="turn:user:password@my_turn_server.com:3478;turn:user:password@my_tcp_turn_server.com:3478?transport=tcp"

Advanced topic: debugging network issues

You may face issues such as failure to connect, lag or low visualization quality in specific network configurations (e.g. NAT, network proxy, VPN configurations). Here are some ideas to debug network issues:

  • Make sure localhost works before trying to host a remote server. Make sure standalone mode works before trying the Jupyter mode.
  • Test both client and server machines for connectivity and throughput at https://test.webrtc.org/ <https://test.webrtc.org/>_.
  • If you face control lag, try switching from TCP to UDP connection or switching to a closer TURN (video traffic relay) server.
  • If the visualization video has compression artifacts, you may be using a TURN server with insufficient bandwidth. Try switching to a TURN server with higher bandwidth or closer to your location.