Loading dataset... CachedTrajectoryDataset: 10 demos, 320 samples Source: {'cache_root': '/data/libero/ood_objpos_v3_splits/exp_10demo_finetune_train', 'benchmark': 'libero_spatial', 'task_ids': [0], 'frame_stride': 3} Total: 320 samples ✓ Train: 304 samples ✓ Val: 16 samples Initializing model (type=para)... Loading DINOv2 model... ✓ DINOv2 backbone is trainable ✓ DINO embedding dim: 384 ✓ Learnable start keypoint embedding (dim=384) ✓ Feature convs: 3× Conv2d(3×3) at pred_size=64 ✓ Volume head → (B, 4, 32, 64, 64) ✓ Gripper head → (B, 4, 2, 64, 64) [1×1 conv, CE 2-class] ✓ Rotation head → (B, 4, 3, 32, 64, 64) [1×1 conv, CE] Trainable parameters: 32,880,520 / 32,880,520 (100.00%) Loading checkpoint: checkpoints/para_finetune_init_arm_deleted/best.pth ⚠ Could not load optimizer state: 'optimizer_state_dict' ✓ Resumed from epoch 1 Computing dataset stats from random subset: 320/320 samples (seed=42) Epochs: 0%| | 2/9998 [00:24<34:02:07, 12.26s/it] ✓ Saved stats cache: /data/cameron/para_normalized_losses/libero/checkpoints/para_finetune_arm_deleted/dataset_stats.json ✓ Height range from dataset: [0.918027, 1.177156] m ✓ Gripper range from dataset: [-1.000000, 1.000000] ✓ Rotation range (delta rotvec): ['-0.019', '-0.040', '-0.003'] .. ['0.033', '0.105', '0.010'] ✓ Reference rotation: ['0.9996', '0.0005', '-0.0281', '-0.0001'] ✓ Position range from dataset: ['-0.406', '-0.364', '0.918'] .. ['0.112', '0.269', '1.177'] Starting training for 9999 epochs... ✓ Rotation loss SKIPPED ✓ EMA loss weights initialized (vol=11.8, rot=3.5, grip=0.69) ============================================================ Epoch 1/9999 ============================================================ Train Loss: 2.8984 (Volume: 1.5143, Gripper: 1.3841, Rotation: 0.0000) Val - Loss: 1.9518, Volume: 11.1728, Pixel Error: 107.54px, Height Error: 50.942mm, Gripper: 0.9688 ✓ Saved best model (val_loss=1.9518) ============================================================ Epoch 2/9999 ============================================================ Train Loss: 0.9561 (Volume: 0.6284, Gripper: 0.3278, Rotation: 0.0000) Val - Loss: 0.8085, Volume: 9.9716, Pixel Error: 63.81px, Height Error: 43.059mm, Gripper: 0.5938 ✓ Saved best model (val_loss=0.8085) ============================================================ Epoch 3/9999 ============================================================ Train Loss: 0.4099 (Volume: 0.2546, Gripper: 0.1553, Rotation: 0.0000) Val - Loss: 0.5109, Volume: 8.7345, Pixel Error: 34.69px, Height Error: 36.842mm, Gripper: 0.5312 ✓ Saved best model (val_loss=0.5109) ============================================================ Epoch 4/9999 ============================================================ Train Loss: 0.2580 (Volume: 0.1380, Gripper: 0.1200, Rotation: 0.0000) Val - Loss: 0.5400, Volume: 8.1404, Pixel Error: 25.97px, Height Error: 36.281mm, Gripper: 0.3750 ============================================================ Epoch 5/9999 ============================================================ Train Loss: 0.1784 (Volume: 0.1022, Gripper: 0.0763, Rotation: 0.0000) Val - Loss: 0.2352, Volume: 7.1758, Pixel Error: 23.11px, Height Error: 36.800mm, Gripper: 0.2500 ✓ Saved best model (val_loss=0.2352) ============================================================ Epoch 6/9999 ============================================================ Train Loss: 0.1065 (Volume: 0.0594, Gripper: 0.0471, Rotation: 0.0000) Val - Loss: 0.6454, Volume: 7.0141, Pixel Error: 23.76px, Height Error: 34.178mm, Gripper: 0.5000 ============================================================ Epoch 7/9999 ============================================================ Train Loss: 0.0553 (Volume: 0.0323, Gripper: 0.0230, Rotation: 0.0000) Val - Loss: 0.2471, Volume: 6.6602, Pixel Error: 23.93px, Height Error: 41.176mm, Gripper: 0.2812 ============================================================ Epoch 8/9999 ============================================================ Train Loss: 0.0242 (Volume: 0.0172, Gripper: 0.0069, Rotation: 0.0000) Val - Loss: 0.4626, Volume: 6.6215, Pixel Error: 22.52px, Height Error: 39.452mm, Gripper: 0.2500 ============================================================ Epoch 9/9999 ============================================================ Train Loss: 0.0098 (Volume: 0.0061, Gripper: 0.0037, Rotation: 0.0000) Val - Loss: 0.2964, Volume: 6.6980, Pixel Error: 23.14px, Height Error: 37.161mm, Gripper: 0.2500 ============================================================ Epoch 10/9999 ============================================================ Train Loss: 0.0527 (Volume: 0.0190, Gripper: 0.0337, Rotation: 0.0000) Val - Loss: 1.2722, Volume: 7.7184, Pixel Error: 29.02px, Height Error: 52.557mm, Gripper: 0.5000 ============================================================ Epoch 11/9999 ============================================================ Train Loss: 0.1610 (Volume: 0.0718, Gripper: 0.0892, Rotation: 0.0000) Val - Loss: 0.5248, Volume: 7.3726, Pixel Error: 22.98px, Height Error: 35.275mm, Gripper: 0.2500 ============================================================ Epoch 12/9999 ============================================================ Train Loss: 0.1176 (Volume: 0.0629, Gripper: 0.0546, Rotation: 0.0000) Val - Loss: 0.5217, Volume: 6.6248, Pixel Error: 22.64px, Height Error: 37.887mm, Gripper: 0.3125 ============================================================ Epoch 13/9999 ============================================================ Train Loss: 0.0757 (Volume: 0.0411, Gripper: 0.0345, Rotation: 0.0000) Val - Loss: 0.9050, Volume: 6.4744, Pixel Error: 23.54px, Height Error: 35.903mm, Gripper: 0.2500 ============================================================ Epoch 14/9999 ============================================================ Train Loss: 0.0634 (Volume: 0.0331, Gripper: 0.0304, Rotation: 0.0000) Val - Loss: 0.4046, Volume: 6.3789, Pixel Error: 21.59px, Height Error: 34.723mm, Gripper: 0.1875 ============================================================ Epoch 15/9999 ============================================================ Train Loss: 0.0588 (Volume: 0.0281, Gripper: 0.0307, Rotation: 0.0000) Val - Loss: 0.9975, Volume: 6.4204, Pixel Error: 19.98px, Height Error: 31.536mm, Gripper: 0.4375 ============================================================ Epoch 16/9999 ============================================================ Train Loss: 0.0566 (Volume: 0.0325, Gripper: 0.0241, Rotation: 0.0000) Val - Loss: 0.3599, Volume: 5.9877, Pixel Error: 22.07px, Height Error: 37.302mm, Gripper: 0.1875 ============================================================ Epoch 17/9999 ============================================================ Train Loss: 0.0110 (Volume: 0.0087, Gripper: 0.0023, Rotation: 0.0000) Val - Loss: 0.4552, Volume: 5.9123, Pixel Error: 21.32px, Height Error: 35.418mm, Gripper: 0.2812 ============================================================ Epoch 18/9999 ============================================================ Train Loss: 0.0036 (Volume: 0.0026, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.4869, Volume: 5.8928, Pixel Error: 21.05px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 19/9999 ============================================================ Train Loss: 0.0010 (Volume: 0.0007, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.4952, Volume: 5.8852, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 20/9999 ============================================================ Train Loss: 0.0005 (Volume: 0.0003, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.5021, Volume: 5.8852, Pixel Error: 20.94px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 21/9999 ============================================================ Train Loss: 0.0003 (Volume: 0.0002, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.5171, Volume: 5.8804, Pixel Error: 20.91px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 22/9999 ============================================================ Train Loss: 0.0003 (Volume: 0.0002, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5314, Volume: 5.8785, Pixel Error: 21.02px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 23/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5356, Volume: 5.8795, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 24/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5442, Volume: 5.8799, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 25/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5495, Volume: 5.8798, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 26/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5516, Volume: 5.8797, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 27/9999 ============================================================ Saved step checkpoint: step_1000.pth Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5538, Volume: 5.8802, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 28/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5586, Volume: 5.8807, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 29/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5641, Volume: 5.8813, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 30/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5696, Volume: 5.8820, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 31/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5734, Volume: 5.8826, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 32/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5787, Volume: 5.8834, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 33/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5847, Volume: 5.8841, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 34/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5902, Volume: 5.8846, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 35/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5935, Volume: 5.8848, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 36/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6004, Volume: 5.8854, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 37/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6036, Volume: 5.8856, Pixel Error: 20.96px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 38/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6067, Volume: 5.8870, Pixel Error: 20.59px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 39/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6101, Volume: 5.8871, Pixel Error: 20.59px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 40/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6149, Volume: 5.8871, Pixel Error: 20.59px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 41/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6176, Volume: 5.8877, Pixel Error: 20.59px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 42/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6226, Volume: 5.8883, Pixel Error: 20.59px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 43/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6275, Volume: 5.8891, Pixel Error: 20.59px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 44/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6308, Volume: 5.8899, Pixel Error: 20.59px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 45/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6339, Volume: 5.8908, Pixel Error: 20.59px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 46/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6366, Volume: 5.8913, Pixel Error: 20.50px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 47/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6444, Volume: 5.8917, Pixel Error: 20.50px, Height Error: 34.582mm, Gripper: 0.2812 ============================================================ Epoch 48/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6477, Volume: 5.8924, Pixel Error: 20.50px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 49/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6490, Volume: 5.8925, Pixel Error: 20.50px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 50/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6508, Volume: 5.8930, Pixel Error: 20.50px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 51/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6546, Volume: 5.8935, Pixel Error: 20.51px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 52/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6583, Volume: 5.8937, Pixel Error: 20.51px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 53/9999 ============================================================ Saved step checkpoint: step_2000.pth Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6609, Volume: 5.8939, Pixel Error: 20.50px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 54/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6643, Volume: 5.8939, Pixel Error: 20.51px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 55/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6658, Volume: 5.8948, Pixel Error: 20.51px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 56/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6674, Volume: 5.8954, Pixel Error: 20.51px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 57/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6701, Volume: 5.8961, Pixel Error: 20.51px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 58/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6721, Volume: 5.8968, Pixel Error: 20.51px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 59/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6751, Volume: 5.8979, Pixel Error: 20.51px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 60/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6785, Volume: 5.8985, Pixel Error: 20.51px, Height Error: 34.320mm, Gripper: 0.2812 ============================================================ Epoch 61/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6812, Volume: 5.8993, Pixel Error: 20.51px, Height Error: 34.059mm, Gripper: 0.2812 ============================================================ Epoch 62/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6829, Volume: 5.9002, Pixel Error: 20.51px, Height Error: 34.059mm, Gripper: 0.2812 ============================================================ Epoch 63/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.6858, Volume: 5.9007, Pixel Error: 20.51px, Height Error: 34.059mm, Gripper: 0.2812 ⏱ Time limit reached (10.1 / 10 min). Stopping.