Loading dataset... CachedTrajectoryDataset: 128 demos, 4096 samples Source: {'cache_root': '/data/libero/ood_objpos_splits/exp3_left_train', 'benchmark': 'libero_spatial', 'task_ids': [0], 'frame_stride': 3} Total: 4096 samples ✓ Train: 3892 samples ✓ Val: 204 samples Initializing model (type=para)... Loading DINOv2 model... ✓ DINOv2 backbone is trainable ✓ DINO embedding dim: 384 ✓ Learnable start keypoint embedding (dim=384) ✓ Feature convs: 3× Conv2d(3×3) at pred_size=64 ✓ Volume head → (B, 4, 32, 64, 64) ✓ Gripper head → (B, 4, 2, 64, 64) [1×1 conv, CE 2-class] ✓ Rotation head → (B, 4, 3, 32, 64, 64) [1×1 conv, CE] Trainable parameters: 32,880,520 / 32,880,520 (100.00%) Computing dataset stats from random subset: 500/4096 samples (seed=42) Epochs: 0%| | 2/9999 [02:25<202:23:52, 72.89s/it] ✓ Saved stats cache: /data/cameron/para_normalized_losses/libero/checkpoints/para_exp3_left_60min/dataset_stats.json ✓ Height range from dataset: [0.918925, 1.177174] m ✓ Gripper range from dataset: [-1.000000, 1.000000] ✓ Rotation range (delta rotvec): ['-0.003', '-0.049', '-0.002'] .. ['0.014', '0.008', '0.003'] ✓ Reference rotation: ['0.9996', '0.0003', '-0.0287', '-0.0004'] ✓ Position range from dataset: ['-0.211', '-0.165', '0.919'] .. ['0.070', '0.069', '1.177'] Starting training for 9999 epochs... ✓ Rotation loss SKIPPED ✓ EMA loss weights initialized (vol=11.8, rot=3.5, grip=0.69) ============================================================ Epoch 0/9999 ============================================================ Train: 10%|█ | 51/487 [00:08<00:55, 7.89it/s, grip=0.0049, loss=0.0288, rot=0.0000, vol=0.0239] Train Loss: 0.5218 (Volume: 0.2802, Gripper: 0.2416, Rotation: 0.0000) Val - Loss: 0.3329, Volume: 4.6812, Pixel Error: 13.31px, Height Error: 25.884mm, Gripper: 0.2426 ✓ Saved best model (val_loss=0.3329) ============================================================ Epoch 1/9999 ============================================================ Train Loss: 0.1702 (Volume: 0.0853, Gripper: 0.0849, Rotation: 0.0000) Val - Loss: 0.2491, Volume: 3.2329, Pixel Error: 9.80px, Height Error: 19.492mm, Gripper: 0.1225 ✓ Saved best model (val_loss=0.2491) ============================================================ Epoch 2/9999 ============================================================ Saved step checkpoint: step_1000.pth Train Loss: 0.1190 (Volume: 0.0595, Gripper: 0.0595, Rotation: 0.0000) Val - Loss: 0.2379, Volume: 2.9213, Pixel Error: 10.11px, Height Error: 16.754mm, Gripper: 0.1078 ✓ Saved best model (val_loss=0.2379) ============================================================ Epoch 3/9999 ============================================================ Train Loss: 0.0724 (Volume: 0.0371, Gripper: 0.0353, Rotation: 0.0000) Val - Loss: 0.1695, Volume: 2.3470, Pixel Error: 7.87px, Height Error: 13.389mm, Gripper: 0.0882 ✓ Saved best model (val_loss=0.1695) ============================================================ Epoch 4/9999 ============================================================ Saved step checkpoint: step_2000.pth Train Loss: 0.0586 (Volume: 0.0288, Gripper: 0.0298, Rotation: 0.0000) Val - Loss: 0.3333, Volume: 2.6822, Pixel Error: 9.08px, Height Error: 16.219mm, Gripper: 0.0931 ============================================================ Epoch 5/9999 ============================================================ Train Loss: 0.0668 (Volume: 0.0341, Gripper: 0.0327, Rotation: 0.0000) Val - Loss: 0.3500, Volume: 3.0733, Pixel Error: 10.88px, Height Error: 17.376mm, Gripper: 0.0760 ============================================================ Epoch 6/9999 ============================================================ Saved step checkpoint: step_3000.pth Train Loss: 0.0421 (Volume: 0.0209, Gripper: 0.0211, Rotation: 0.0000) Val - Loss: 0.2545, Volume: 2.3957, Pixel Error: 8.62px, Height Error: 14.658mm, Gripper: 0.0588 ============================================================ Epoch 7/9999 ============================================================ Train Loss: 0.0692 (Volume: 0.0354, Gripper: 0.0338, Rotation: 0.0000) Val - Loss: 0.3349, Volume: 2.8043, Pixel Error: 9.75px, Height Error: 15.727mm, Gripper: 0.1127 ============================================================ Epoch 8/9999 ============================================================ Saved step checkpoint: step_4000.pth Train Loss: 0.0314 (Volume: 0.0157, Gripper: 0.0157, Rotation: 0.0000) Val - Loss: 0.1632, Volume: 2.4173, Pixel Error: 8.68px, Height Error: 12.932mm, Gripper: 0.0931 ✓ Saved best model (val_loss=0.1632) ============================================================ Epoch 9/9999 ============================================================ Train Loss: 0.0366 (Volume: 0.0186, Gripper: 0.0180, Rotation: 0.0000) Val - Loss: 0.1798, Volume: 2.3857, Pixel Error: 8.42px, Height Error: 12.300mm, Gripper: 0.0711 ============================================================ Epoch 10/9999 ============================================================ Saved step checkpoint: step_5000.pth Train Loss: 0.0864 (Volume: 0.0442, Gripper: 0.0421, Rotation: 0.0000) Val - Loss: 0.1659, Volume: 2.6471, Pixel Error: 8.38px, Height Error: 13.973mm, Gripper: 0.0907 ============================================================ Epoch 11/9999 ============================================================ Train Loss: 0.0329 (Volume: 0.0165, Gripper: 0.0164, Rotation: 0.0000) Val - Loss: 0.1842, Volume: 2.2394, Pixel Error: 7.53px, Height Error: 11.949mm, Gripper: 0.0662 ============================================================ Epoch 12/9999 ============================================================ Saved step checkpoint: step_6000.pth Train Loss: 0.0156 (Volume: 0.0077, Gripper: 0.0079, Rotation: 0.0000) Val - Loss: 0.2097, Volume: 2.0529, Pixel Error: 7.36px, Height Error: 10.545mm, Gripper: 0.0564 ============================================================ Epoch 13/9999 ============================================================ Train Loss: 0.0110 (Volume: 0.0057, Gripper: 0.0053, Rotation: 0.0000) Val - Loss: 0.2147, Volume: 2.0768, Pixel Error: 7.37px, Height Error: 10.834mm, Gripper: 0.0319 ============================================================ Epoch 14/9999 ============================================================ Saved step checkpoint: step_7000.pth Train Loss: 0.0109 (Volume: 0.0054, Gripper: 0.0054, Rotation: 0.0000) Val - Loss: 0.1879, Volume: 2.0164, Pixel Error: 6.86px, Height Error: 10.434mm, Gripper: 0.0392 ============================================================ Epoch 15/9999 ============================================================ Train Loss: 0.0040 (Volume: 0.0021, Gripper: 0.0019, Rotation: 0.0000) Val - Loss: 0.2486, Volume: 1.8517, Pixel Error: 7.14px, Height Error: 9.647mm, Gripper: 0.0319 ============================================================ Epoch 16/9999 ============================================================ Saved step checkpoint: step_8000.pth Train Loss: 0.0019 (Volume: 0.0010, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.2929, Volume: 1.8616, Pixel Error: 7.10px, Height Error: 9.446mm, Gripper: 0.0368 ============================================================ Epoch 17/9999 ============================================================ Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.3778, Volume: 1.8885, Pixel Error: 6.49px, Height Error: 9.335mm, Gripper: 0.0319 ============================================================ Epoch 18/9999 ============================================================ Saved step checkpoint: step_9000.pth Train Loss: 0.0342 (Volume: 0.0164, Gripper: 0.0178, Rotation: 0.0000) Val - Loss: 0.5521, Volume: 5.2032, Pixel Error: 22.57px, Height Error: 29.391mm, Gripper: 0.3725 ============================================================ Epoch 19/9999 ============================================================ Train Loss: 0.0910 (Volume: 0.0465, Gripper: 0.0445, Rotation: 0.0000) Val - Loss: 0.1228, Volume: 2.1499, Pixel Error: 7.87px, Height Error: 10.308mm, Gripper: 0.0515 ✓ Saved best model (val_loss=0.1228) ============================================================ Epoch 20/9999 ============================================================ Saved step checkpoint: step_10000.pth Train Loss: 0.0318 (Volume: 0.0159, Gripper: 0.0159, Rotation: 0.0000) Val - Loss: 0.2164, Volume: 2.0894, Pixel Error: 7.73px, Height Error: 10.259mm, Gripper: 0.0539 ============================================================ Epoch 21/9999 ============================================================ Train Loss: 0.0112 (Volume: 0.0058, Gripper: 0.0054, Rotation: 0.0000) Val - Loss: 0.2189, Volume: 1.9513, Pixel Error: 7.16px, Height Error: 9.192mm, Gripper: 0.0417 ============================================================ Epoch 22/9999 ============================================================ Saved step checkpoint: step_11000.pth Train Loss: 0.0100 (Volume: 0.0050, Gripper: 0.0050, Rotation: 0.0000) Val - Loss: 0.2126, Volume: 1.8906, Pixel Error: 7.18px, Height Error: 9.481mm, Gripper: 0.0662 ============================================================ Epoch 23/9999 ============================================================ Train Loss: 0.0368 (Volume: 0.0168, Gripper: 0.0200, Rotation: 0.0000) Val - Loss: 0.4919, Volume: 3.1009, Pixel Error: 12.59px, Height Error: 18.499mm, Gripper: 0.1373 ============================================================ Epoch 24/9999 ============================================================ Saved step checkpoint: step_12000.pth Train Loss: 0.0385 (Volume: 0.0213, Gripper: 0.0172, Rotation: 0.0000) Val - Loss: 0.2452, Volume: 2.2833, Pixel Error: 8.94px, Height Error: 13.309mm, Gripper: 0.0515 ============================================================ Epoch 25/9999 ============================================================ Train Loss: 0.0191 (Volume: 0.0099, Gripper: 0.0092, Rotation: 0.0000) Val - Loss: 0.1331, Volume: 1.8580, Pixel Error: 7.48px, Height Error: 9.502mm, Gripper: 0.0294 ============================================================ Epoch 26/9999 ============================================================ Saved step checkpoint: step_13000.pth Train Loss: 0.0059 (Volume: 0.0029, Gripper: 0.0030, Rotation: 0.0000) Val - Loss: 0.2022, Volume: 1.8580, Pixel Error: 7.71px, Height Error: 10.329mm, Gripper: 0.0515 ============================================================ Epoch 27/9999 ============================================================ Train Loss: 0.0020 (Volume: 0.0011, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.2097, Volume: 1.7230, Pixel Error: 7.21px, Height Error: 9.065mm, Gripper: 0.0417 ============================================================ Epoch 28/9999 ============================================================ Saved step checkpoint: step_14000.pth Train Loss: 0.0012 (Volume: 0.0006, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.2642, Volume: 1.8255, Pixel Error: 7.90px, Height Error: 10.787mm, Gripper: 0.0343 ============================================================ Epoch 29/9999 ============================================================ Train Loss: 0.0008 (Volume: 0.0004, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.2288, Volume: 1.7238, Pixel Error: 7.11px, Height Error: 8.673mm, Gripper: 0.0221 ============================================================ Epoch 30/9999 ============================================================ Saved step checkpoint: step_15000.pth Train Loss: 0.0007 (Volume: 0.0003, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.2426, Volume: 1.8603, Pixel Error: 6.78px, Height Error: 8.887mm, Gripper: 0.0270 ============================================================ Epoch 31/9999 ============================================================ Train Loss: 0.0937 (Volume: 0.0471, Gripper: 0.0466, Rotation: 0.0000) Val - Loss: 0.2408, Volume: 2.2665, Pixel Error: 8.50px, Height Error: 11.661mm, Gripper: 0.0539 ============================================================ Epoch 32/9999 ============================================================ Saved step checkpoint: step_16000.pth Train Loss: 0.0251 (Volume: 0.0129, Gripper: 0.0122, Rotation: 0.0000) Val - Loss: 0.1573, Volume: 1.8499, Pixel Error: 7.34px, Height Error: 10.077mm, Gripper: 0.0392 ============================================================ Epoch 33/9999 ============================================================ Train Loss: 0.0140 (Volume: 0.0069, Gripper: 0.0072, Rotation: 0.0000) Val - Loss: 0.3169, Volume: 2.3814, Pixel Error: 8.46px, Height Error: 11.716mm, Gripper: 0.0490 ============================================================ Epoch 34/9999 ============================================================ Saved step checkpoint: step_17000.pth Train Loss: 0.0291 (Volume: 0.0147, Gripper: 0.0144, Rotation: 0.0000) Val - Loss: 0.1248, Volume: 1.7853, Pixel Error: 7.08px, Height Error: 9.592mm, Gripper: 0.0417 ============================================================ Epoch 35/9999 ============================================================ Train Loss: 0.0136 (Volume: 0.0065, Gripper: 0.0071, Rotation: 0.0000) Val - Loss: 0.1637, Volume: 1.8936, Pixel Error: 8.09px, Height Error: 10.532mm, Gripper: 0.0368 ============================================================ Epoch 36/9999 ============================================================ Saved step checkpoint: step_18000.pth Train Loss: 0.0087 (Volume: 0.0047, Gripper: 0.0040, Rotation: 0.0000) Val - Loss: 0.1753, Volume: 1.7749, Pixel Error: 6.77px, Height Error: 8.315mm, Gripper: 0.0343 ============================================================ Epoch 37/9999 ============================================================ Train Loss: 0.0027 (Volume: 0.0013, Gripper: 0.0014, Rotation: 0.0000) Val - Loss: 0.1713, Volume: 1.7190, Pixel Error: 6.99px, Height Error: 8.691mm, Gripper: 0.0417 ============================================================ Epoch 38/9999 ============================================================ Train Loss: 0.0071 (Volume: 0.0037, Gripper: 0.0033, Rotation: 0.0000) Val - Loss: 0.4028, Volume: 1.9036, Pixel Error: 7.49px, Height Error: 9.033mm, Gripper: 0.0417 ============================================================ Epoch 39/9999 ============================================================ Saved step checkpoint: step_19000.pth Train Loss: 0.0574 (Volume: 0.0289, Gripper: 0.0285, Rotation: 0.0000) Val - Loss: 0.2012, Volume: 2.6124, Pixel Error: 9.07px, Height Error: 14.697mm, Gripper: 0.0466 ============================================================ Epoch 40/9999 ============================================================ Train Loss: 0.0253 (Volume: 0.0127, Gripper: 0.0126, Rotation: 0.0000) Val - Loss: 0.1206, Volume: 1.7253, Pixel Error: 6.50px, Height Error: 8.653mm, Gripper: 0.0270 ✓ Saved best model (val_loss=0.1206) ============================================================ Epoch 41/9999 ============================================================ Saved step checkpoint: step_20000.pth Train Loss: 0.0019 (Volume: 0.0009, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1391, Volume: 1.7839, Pixel Error: 7.12px, Height Error: 8.980mm, Gripper: 0.0441 ============================================================ Epoch 42/9999 ============================================================ Train Loss: 0.0183 (Volume: 0.0092, Gripper: 0.0091, Rotation: 0.0000) Val - Loss: 0.1714, Volume: 1.9141, Pixel Error: 7.40px, Height Error: 9.364mm, Gripper: 0.0441 ============================================================ Epoch 43/9999 ============================================================ Saved step checkpoint: step_21000.pth Train Loss: 0.0168 (Volume: 0.0077, Gripper: 0.0090, Rotation: 0.0000) Val - Loss: 0.2598, Volume: 2.9966, Pixel Error: 11.51px, Height Error: 16.870mm, Gripper: 0.1569 ============================================================ Epoch 44/9999 ============================================================ Train Loss: 0.0361 (Volume: 0.0191, Gripper: 0.0170, Rotation: 0.0000) Val - Loss: 0.1700, Volume: 1.7037, Pixel Error: 7.50px, Height Error: 9.593mm, Gripper: 0.0368 ============================================================ Epoch 45/9999 ============================================================ Saved step checkpoint: step_22000.pth Train Loss: 0.0025 (Volume: 0.0013, Gripper: 0.0012, Rotation: 0.0000) Val - Loss: 0.1613, Volume: 1.6452, Pixel Error: 6.92px, Height Error: 8.864mm, Gripper: 0.0466 ============================================================ Epoch 46/9999 ============================================================ Train Loss: 0.0018 (Volume: 0.0009, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1891, Volume: 1.8508, Pixel Error: 7.30px, Height Error: 9.067mm, Gripper: 0.0441 ⏱ Time limit reached (60.5 / 60 min). Stopping.