# TRI Stereo Depth

TRI's learned stereo depth model is an optional depth backend for ZED
cameras in `rd convert`. It produces high-quality depth maps tailored for
robot manipulation scenes and supports two model variants:


| Variant | Flag | Description |
|---|---|---|
| `c64` | `--tri-stereo-variant c64` | Higher quality (default) |
| `c32` | `--tri-stereo-variant c32` | Faster, lighter |

For each variant, Raiden automatically selects the fastest available inference
backend:

1. **TensorRT** — fastest; requires a pre-compiled `.engine` file
2. **ONNX Runtime** — no compilation needed; requires a `.onnx` file

## Prerequisites

**1. Pull models via Git LFS**

The ONNX models are tracked in this repository via Git LFS under
`weights/tri_stereo/`. If you haven't already, install Git LFS and pull the files:

```bash
git lfs install   # one-time setup per machine
git lfs pull
```

**2. Install the `tri-stereo` extra**

```bash
uv tool install --reinstall -e ".[zed,tri-stereo]"
```

This installs `onnxruntime-gpu` for the ONNX backend.

## Usage

```bash
rd convert --stereo-method tri_stereo
rd convert --stereo-method tri_stereo --tri-stereo-variant c32
```

Raiden uses the ONNX backend by default. If a TensorRT engine is present it
is used automatically instead.

## TensorRT compilation

Compile ONNX models to TensorRT FP16 engines for fastest inference. This step
runs once per machine and takes a few minutes.

**1. Install TensorRT**

Add the TensorRT apt repository following the
[local repo installation guide](https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/installing.html#installation-steps-local-repo-method),
then install:

```bash
sudo apt-get install --no-install-recommends libnvinfer-bin libnvinfer-dev libnvinfer-headers-dev
uv tool install --reinstall -e ".[zed,tri-stereo,tri-stereo-trt-cu12]"    # CUDA 12
uv tool install --reinstall -e ".[zed,tri-stereo,tri-stereo-trt-cu13]"    # CUDA 13
```

!!! note
    `sudo apt-get install tensorrt` fails on Python 3.11+ systems due to a
    dependency conflict. See [FAQ](faq.md#sudo-apt-get-install-tensorrt-fails-with-python-version-conflict)
    for details.

Verify both are available:

```bash
trtexec --help
uv run python -c "import tensorrt; print(tensorrt.__version__)"
```

**2. Compile engines:**

```bash
rd make_tri_stereo_engine                    # c64 (default)
rd make_tri_stereo_engine --variant c32
```

Once the engine files are present, `rd convert --stereo-method tri_stereo`
automatically uses TensorRT. You will see `[TRIStereo-C64] Loaded TRT engine` in
the output when TensorRT is active.

!!! note "Engines are machine-specific"
    TensorRT engines are compiled for the specific GPU and driver version on
    the build machine. Recompile after upgrading drivers, CUDA, or TensorRT,
    or when moving to a different GPU.

## Weights directory layout

```
weights/tri_stereo/       # Git LFS — all model files live here
    stereo_c64.onnx       # ONNX model (c64)       — distributed via Git LFS
    stereo_c64.onnx.data  # ONNX external data (c64)
    stereo_c32.onnx       # ONNX model (c32)
    stereo_c32.onnx.data  # ONNX external data (c32)
    stereo_c64.engine     # TensorRT engine (c64)  — generated by rd make_tri_stereo_engine
    stereo_c32.engine     # TensorRT engine (c32)
```

## Acknowledgments

TRI Stereo Depth is developed at Toyota Research Institute. For more details,
see the [project page](https://sites.google.com/view/stereoformobilemanipulation).

## Citation

If you use TRI Stereo Depth, please cite:

```bibtex
@article{shankar2022learned,
  title={A learned stereo depth system for robotic manipulation in homes},
  author={Shankar, Krishna and Tjersland, Mark and Ma, Jeremy and Stone, Kevin and Bajracharya, Max},
  journal={IEEE Robotics and Automation Letters},
  volume={7},
  number={2},
  pages={2305--2312},
  year={2022},
  publisher={IEEE}
}
```