Spectacular AI Mapping Tools

Spectacular AI enable fast Gaussian Splatting, NeRF and other 3D reconstructions from data recorded on smartphones (and other supported devices), without uploading the data to a cloud or using slow photogrammetry tools like COLMAP.

The toolbox consists of the Spectacular Rec application, Spectacular AI SDK (for other devices), and the sai-cli command line tool. For NeRF and 3DGS, you also need a powerful GPU and Nerfstudio.

In addition to video data, the Spectacular AI tools record IMU, which is fused with camera and (if available) LiDAR/ToF data when computing camera poses. This approach, VISLAM, is more robust than purely image based methods (e.g., COLMAP) and works better and faster for difficult data (monotonic environments, fast motions, narrow FoV, etc.).

Example reconstructions created using the Spectacular Rec smartphone app and Nerfstudio. Click to view the models in your browser.

Pixel 5Pixel 5SamsungPixel 5

Supported devices

Spectacular AI tools can be used for 3D mapping and reconstruction on devices supported by our SDK or the Spectacular Rec mobile application:

  • iPhone (with or without LiDAR)
  • Android (with or without ToF)
  • Luxonis OAK devices
  • Intel RealSense D455 / D435i
  • Azure Kinect DK

Data recording

On the iPhone, first download Spectacular Rec in the App Store. Android: download Spectacular Rec from Play Store. For other devices, see this page for specific instructions.

Installation

Requirements: to use Spectacular AI with Nerfstudio, you need a rather good NVidia GPU. Windows and Linux operating systems are supported. Installation steps:

  1. Install Nerfstudio.
  2. Install FFmpeg. Linux: apt install ffmpeg. Windows: see here.
  3. Install sai-cli. (with Nerfstudio, run this in the activated Conda environment): pip install spectacularAI[full]

NeRFs and Gaussian Splatting with Nerfstudio

Quick cheat sheet (more details here):

sai-cli process {recording directory} --preview3d {nerf directory}
ns-train nerfacto --data {nerf directory}

For step-by-step instructions of the entire process for iPhone, see our tutorial video.

Other mapping demos

A floor area of approximately 1000m² was mapped in 21 minutes using Azure Kinect device.

Real-time meshing on a Huawei Android phones with a ToF sensor.

NeRF reconstructions with Intel RealSense without COLMAP.

License

The Spectacular AI Apps, SDK and command line tools (sai-cli) are free for non-commercial use. Contact us for commercial licensing options.