/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/sympy/external/gmpy.py:139: UserWarning: gmpy2 version is too old to use (2.0.0 or newer required) gmpy = import_module('gmpy2', min_module_version=_GMPY2_MIN_VERSION, Device: cuda dataset /data/cameron/mac_robot_datasets/dataset_20260506_124503: 11 episodes dataset /data/cameron/mac_robot_datasets/dataset_20260506_151912: 11 episodes Stats: height=[0.0321, 0.1866], gripper=recorded range [-0.074, +1.450] (861 samples) Loading MuJoCo model: /home/cameronsmith/mnt/mac/smith300_para_stuff/example_twolink.xml Pre-computing per-frame data for 302 frames (running arm FK) Smith300Dataset: 11 episodes, 247 samples, stride=1 Loading MuJoCo model: /home/cameronsmith/mnt/mac/smith300_para_stuff/example_twolink.xml Pre-computing per-frame data for 302 frames (running arm FK) Smith300Dataset: 11 episodes, 247 samples, stride=1 dataset_20260506_124503: train=210, val=37, total=247 Loading MuJoCo model: /home/cameronsmith/mnt/mac/smith300_para_stuff/example_twolink.xml Pre-computing per-frame data for 559 frames (using saved EEF poses (UMI)) Smith300Dataset: 11 episodes, 504 samples, stride=1 Loading MuJoCo model: /home/cameronsmith/mnt/mac/smith300_para_stuff/example_twolink.xml Pre-computing per-frame data for 559 frames (using saved EEF poses (UMI)) Smith300Dataset: 11 episodes, 504 samples, stride=1 dataset_20260506_151912: train=429, val=75, total=504 Combined: train=639, val=112, color_aug=ON Loading DINOv2 model... ✓ DINOv2 backbone is trainable ✓ DINO embedding dim: 384 ✓ Learnable start keypoint embedding (dim=384) ✓ Feature convs: 3× Conv2d(3×3) at pred_size=64 ✓ Volume head → (B, 6, 32, 64, 64) ✓ Gripper MLP → (B, 6, 10) [CE, indexed at query pixel] ✓ Rotation MLP → (B, 6, 3, 32) [CE, indexed at query pixel] wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information. wandb: Currently logged in as: cameronsmithbusiness. Use `wandb login --relogin` to force relogin wandb: Tracking run with wandb version 0.18.3 wandb: Run data is saved locally in /data/cameron/para/para_mac/wandb/run-20260506_200301-z7zo07hj wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run towel_umi_combined_v1 wandb: ⭐️ View project at https://wandb.ai/cameronsmithbusiness/para_smith300 wandb: 🚀 View run at https://wandb.ai/cameronsmithbusiness/para_smith300/runs/z7zo07hj Epoch 0 | train=20.2980 | val=17.7081 | px_err=6.5px Epoch 10 | train=12.0209 | val=13.9856 | px_err=2.7px Epoch 20 | train=9.5724 | val=13.7383 | px_err=2.6px Epoch 30 | train=8.0439 | val=14.7395 | px_err=2.5px Epoch 40 | train=7.0626 | val=16.6554 | px_err=2.6px Epoch 50 | train=6.4551 | val=17.4968 | px_err=2.7px Epoch 60 | train=5.7833 | val=18.0060 | px_err=2.8px Epoch 70 | train=5.2628 | val=19.1100 | px_err=2.7px Epoch 80 | train=4.8783 | val=19.3097 | px_err=2.6px Epoch 90 | train=4.4971 | val=20.6551 | px_err=3.0px Epoch 100 | train=4.2122 | val=19.9880 | px_err=2.8px Epoch 110 | train=3.9102 | val=20.3009 | px_err=2.9px Epoch 120 | train=3.6620 | val=20.7103 | px_err=2.9px Epoch 130 | train=3.4735 | val=19.7652 | px_err=2.9px Epoch 140 | train=3.1961 | val=20.7606 | px_err=2.8px Traceback (most recent call last): File "/data/cameron/para/para_mac/train_smith300_para.py", line 656, in "train/grip_loss": np.mean(train_grip_losses), File "/data/cameron/para/para_mac/train_smith300_para.py", line 538, in main "VOLUME_LOSS_WEIGHT": VOLUME_LOSS_WEIGHT, File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__ data = self._next_data() File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 302, in __getitem__ return self.datasets[dataset_idx][sample_idx] File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 356, in __getitem__ return self.dataset[self.indices[idx]] File "/data/cameron/para/para_mac/data_smith300_para.py", line 309, in __getitem__ rgb_for_model = _augment_color(rgb_ref.copy(), self._rng) File "/data/cameron/para/para_mac/data_smith300_para.py", line 73, in _augment_color if do_sat: KeyboardInterrupt Traceback (most recent call last): File "/data/cameron/para/para_mac/train_smith300_para.py", line 656, in "train/grip_loss": np.mean(train_grip_losses), File "/data/cameron/para/para_mac/train_smith300_para.py", line 538, in main "VOLUME_LOSS_WEIGHT": VOLUME_LOSS_WEIGHT, File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__ data = self._next_data() File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 302, in __getitem__ return self.datasets[dataset_idx][sample_idx] File "/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 356, in __getitem__ return self.dataset[self.indices[idx]] File "/data/cameron/para/para_mac/data_smith300_para.py", line 309, in __getitem__ rgb_for_model = _augment_color(rgb_ref.copy(), self._rng) File "/data/cameron/para/para_mac/data_smith300_para.py", line 73, in _augment_color if do_sat: KeyboardInterrupt wandb: 🚀 View run towel_umi_combined_v1 at: https://wandb.ai/cameronsmithbusiness/para_smith300/runs/z7zo07hj wandb: Find logs at: wandb/run-20260506_200301-z7zo07hj/logs