Loading dataset... CachedTrajectoryDataset: 10 demos, 320 samples Source: {'cache_root': '/data/libero/ood_objpos_v3_splits/exp_10demo_finetune_train', 'benchmark': 'libero_spatial', 'task_ids': [0], 'frame_stride': 3} Total: 320 samples ✓ Train: 304 samples ✓ Val: 16 samples Initializing model (type=para)... Loading DINOv2 model... ✓ DINOv2 backbone is trainable ✓ DINO embedding dim: 384 ✓ Learnable start keypoint embedding (dim=384) ✓ Feature convs: 3× Conv2d(3×3) at pred_size=64 ✓ Volume head → (B, 4, 32, 64, 64) ✓ Gripper head → (B, 4, 2, 64, 64) [1×1 conv, CE 2-class] ✓ Rotation head → (B, 4, 3, 32, 64, 64) [1×1 conv, CE] Trainable parameters: 32,880,520 / 32,880,520 (100.00%) Loading checkpoint: checkpoints/para_finetune_init_circle/best.pth ⚠ Could not load optimizer state: 'optimizer_state_dict' ✓ Resumed from epoch 1 Computing dataset stats from random subset: 320/320 samples (seed=42) Epochs: 0%| | 3/9998 [01:01<57:05:55, 20.57s/it] ✓ Saved stats cache: /data/cameron/para_normalized_losses/libero/checkpoints/para_finetune_circle/dataset_stats.json ✓ Height range from dataset: [0.918027, 1.177156] m ✓ Gripper range from dataset: [-1.000000, 1.000000] ✓ Rotation range (delta rotvec): ['-0.019', '-0.040', '-0.003'] .. ['0.033', '0.105', '0.010'] ✓ Reference rotation: ['0.9996', '0.0005', '-0.0281', '-0.0001'] ✓ Position range from dataset: ['-0.406', '-0.364', '0.918'] .. ['0.112', '0.269', '1.177'] Starting training for 9999 epochs... ✓ Rotation loss SKIPPED ✓ EMA loss weights initialized (vol=11.8, rot=3.5, grip=0.69) ============================================================ Epoch 1/9999 ============================================================ Train Loss: 1.8523 (Volume: 1.0727, Gripper: 0.7796, Rotation: 0.0000) Val - Loss: 1.2350, Volume: 10.0011, Pixel Error: 38.62px, Height Error: 60.211mm, Gripper: 0.3438 ✓ Saved best model (val_loss=1.2350) ============================================================ Epoch 2/9999 ============================================================ Train Loss: 0.7033 (Volume: 0.4287, Gripper: 0.2746, Rotation: 0.0000) Val - Loss: 0.6905, Volume: 8.0613, Pixel Error: 27.61px, Height Error: 45.321mm, Gripper: 0.2188 ✓ Saved best model (val_loss=0.6905) ============================================================ Epoch 3/9999 ============================================================ Train Loss: 0.3594 (Volume: 0.1946, Gripper: 0.1648, Rotation: 0.0000) Val - Loss: 0.7287, Volume: 7.6530, Pixel Error: 26.71px, Height Error: 40.223mm, Gripper: 0.3438 ============================================================ Epoch 4/9999 ============================================================ Train Loss: 0.4050 (Volume: 0.2099, Gripper: 0.1951, Rotation: 0.0000) Val - Loss: 0.3856, Volume: 7.0771, Pixel Error: 19.70px, Height Error: 38.010mm, Gripper: 0.1250 ✓ Saved best model (val_loss=0.3856) ============================================================ Epoch 5/9999 ============================================================ Train Loss: 0.1764 (Volume: 0.1042, Gripper: 0.0722, Rotation: 0.0000) Val - Loss: 0.5161, Volume: 6.6356, Pixel Error: 22.76px, Height Error: 34.663mm, Gripper: 0.2188 ============================================================ Epoch 6/9999 ============================================================ Train Loss: 0.1698 (Volume: 0.0875, Gripper: 0.0823, Rotation: 0.0000) Val - Loss: 0.4719, Volume: 7.0029, Pixel Error: 21.20px, Height Error: 31.356mm, Gripper: 0.2188 ============================================================ Epoch 7/9999 ============================================================ Train Loss: 0.0686 (Volume: 0.0467, Gripper: 0.0219, Rotation: 0.0000) Val - Loss: 0.5767, Volume: 6.4864, Pixel Error: 20.48px, Height Error: 31.559mm, Gripper: 0.2188 ============================================================ Epoch 8/9999 ============================================================ Train Loss: 0.0328 (Volume: 0.0202, Gripper: 0.0127, Rotation: 0.0000) Val - Loss: 0.4225, Volume: 6.3718, Pixel Error: 20.57px, Height Error: 32.996mm, Gripper: 0.1875 ============================================================ Epoch 9/9999 ============================================================ Train Loss: 0.0569 (Volume: 0.0232, Gripper: 0.0337, Rotation: 0.0000) Val - Loss: 0.5184, Volume: 6.4615, Pixel Error: 19.55px, Height Error: 33.091mm, Gripper: 0.1875 ============================================================ Epoch 10/9999 ============================================================ Train Loss: 0.0712 (Volume: 0.0355, Gripper: 0.0357, Rotation: 0.0000) Val - Loss: 0.6010, Volume: 6.0574, Pixel Error: 18.32px, Height Error: 29.836mm, Gripper: 0.2188 ============================================================ Epoch 11/9999 ============================================================ Train Loss: 0.0234 (Volume: 0.0174, Gripper: 0.0060, Rotation: 0.0000) Val - Loss: 0.4025, Volume: 5.9859, Pixel Error: 18.13px, Height Error: 32.602mm, Gripper: 0.2812 ============================================================ Epoch 12/9999 ============================================================ Train Loss: 0.0075 (Volume: 0.0057, Gripper: 0.0018, Rotation: 0.0000) Val - Loss: 0.4344, Volume: 5.8681, Pixel Error: 19.00px, Height Error: 31.392mm, Gripper: 0.2812 ============================================================ Epoch 13/9999 ============================================================ Train Loss: 0.0027 (Volume: 0.0018, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.4477, Volume: 5.8546, Pixel Error: 17.84px, Height Error: 31.523mm, Gripper: 0.2500 ============================================================ Epoch 14/9999 ============================================================ Train Loss: 0.0032 (Volume: 0.0011, Gripper: 0.0022, Rotation: 0.0000) Val - Loss: 0.4618, Volume: 5.8570, Pixel Error: 17.83px, Height Error: 31.065mm, Gripper: 0.2500 ============================================================ Epoch 15/9999 ============================================================ Train Loss: 0.0182 (Volume: 0.0087, Gripper: 0.0095, Rotation: 0.0000) Val - Loss: 0.4284, Volume: 5.8159, Pixel Error: 17.15px, Height Error: 31.036mm, Gripper: 0.2500 ============================================================ Epoch 16/9999 ============================================================ Train Loss: 0.0051 (Volume: 0.0033, Gripper: 0.0019, Rotation: 0.0000) Val - Loss: 0.4534, Volume: 5.7964, Pixel Error: 17.26px, Height Error: 31.368mm, Gripper: 0.2188 ============================================================ Epoch 17/9999 ============================================================ Train Loss: 0.0025 (Volume: 0.0011, Gripper: 0.0015, Rotation: 0.0000) Val - Loss: 0.4462, Volume: 5.7968, Pixel Error: 17.26px, Height Error: 31.696mm, Gripper: 0.2188 ============================================================ Epoch 18/9999 ============================================================ Train Loss: 0.0016 (Volume: 0.0014, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.4265, Volume: 5.7902, Pixel Error: 16.61px, Height Error: 31.653mm, Gripper: 0.2812 ============================================================ Epoch 19/9999 ============================================================ Train Loss: 0.0013 (Volume: 0.0006, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.4852, Volume: 5.8248, Pixel Error: 18.09px, Height Error: 31.435mm, Gripper: 0.2188 ============================================================ Epoch 20/9999 ============================================================ Train Loss: 0.0004 (Volume: 0.0003, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.4980, Volume: 5.8291, Pixel Error: 17.00px, Height Error: 32.568mm, Gripper: 0.2188 ============================================================ Epoch 21/9999 ============================================================ Train Loss: 0.0003 (Volume: 0.0002, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5047, Volume: 5.8287, Pixel Error: 17.06px, Height Error: 32.568mm, Gripper: 0.2188 ============================================================ Epoch 22/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5119, Volume: 5.8293, Pixel Error: 17.06px, Height Error: 32.501mm, Gripper: 0.2188 ============================================================ Epoch 23/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5070, Volume: 5.8243, Pixel Error: 17.06px, Height Error: 30.542mm, Gripper: 0.2188 ============================================================ Epoch 24/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5120, Volume: 5.8258, Pixel Error: 17.06px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 25/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5159, Volume: 5.8284, Pixel Error: 17.06px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 26/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.5186, Volume: 5.8289, Pixel Error: 17.06px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 27/9999 ============================================================ Saved step checkpoint: step_1000.pth Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5222, Volume: 5.8309, Pixel Error: 17.06px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 28/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5274, Volume: 5.8337, Pixel Error: 16.96px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 29/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5322, Volume: 5.8371, Pixel Error: 16.96px, Height Error: 32.297mm, Gripper: 0.2188 ============================================================ Epoch 30/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5360, Volume: 5.8378, Pixel Error: 16.96px, Height Error: 32.297mm, Gripper: 0.2188 ============================================================ Epoch 31/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5380, Volume: 5.8401, Pixel Error: 16.96px, Height Error: 32.297mm, Gripper: 0.2188 ============================================================ Epoch 32/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5391, Volume: 5.8423, Pixel Error: 16.96px, Height Error: 32.297mm, Gripper: 0.2188 ============================================================ Epoch 33/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5436, Volume: 5.8446, Pixel Error: 16.96px, Height Error: 32.297mm, Gripper: 0.2188 ============================================================ Epoch 34/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5478, Volume: 5.8459, Pixel Error: 16.96px, Height Error: 32.297mm, Gripper: 0.2188 ============================================================ Epoch 35/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5518, Volume: 5.8495, Pixel Error: 16.96px, Height Error: 32.297mm, Gripper: 0.2188 ============================================================ Epoch 36/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5560, Volume: 5.8513, Pixel Error: 16.96px, Height Error: 32.297mm, Gripper: 0.2188 ============================================================ Epoch 37/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5581, Volume: 5.8534, Pixel Error: 16.96px, Height Error: 32.297mm, Gripper: 0.2188 ============================================================ Epoch 38/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5644, Volume: 5.8501, Pixel Error: 16.96px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 39/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5679, Volume: 5.8508, Pixel Error: 17.04px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 40/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5696, Volume: 5.8521, Pixel Error: 17.04px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 41/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5704, Volume: 5.8541, Pixel Error: 16.96px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 42/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5735, Volume: 5.8560, Pixel Error: 16.96px, Height Error: 30.599mm, Gripper: 0.2188 ============================================================ Epoch 43/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.5765, Volume: 5.8594, Pixel Error: 16.96px, Height Error: 30.599mm, Gripper: 0.2188 ⏱ Time limit reached (10.1 / 10 min). Stopping.