Loading dataset... CachedTrajectoryDataset: 10 demos, 320 samples Source: {'cache_root': '/data/libero/ood_objpos_v3_splits/exp_10demo_finetune_train', 'benchmark': 'libero_spatial', 'task_ids': [0], 'frame_stride': 3} Total: 320 samples ✓ Train: 304 samples ✓ Val: 16 samples Initializing model (type=para)... Loading DINOv2 model... ✓ DINOv2 backbone is trainable ✓ DINO embedding dim: 384 ✓ Learnable start keypoint embedding (dim=384) ✓ Feature convs: 3× Conv2d(3×3) at pred_size=64 ✓ Volume head → (B, 4, 32, 64, 64) ✓ Gripper head → (B, 4, 2, 64, 64) [1×1 conv, CE 2-class] ✓ Rotation head → (B, 4, 3, 32, 64, 64) [1×1 conv, CE] Trainable parameters: 32,880,520 / 32,880,520 (100.00%) Computing dataset stats from random subset: 320/320 samples (seed=42) Epochs: 0%| | 3/9999 [01:02<57:00:39, 20.53s/it] ✓ Saved stats cache: /data/cameron/para_normalized_losses/libero/checkpoints/para_scratch_10demo/dataset_stats.json ✓ Height range from dataset: [0.918027, 1.177156] m ✓ Gripper range from dataset: [-1.000000, 1.000000] ✓ Rotation range (delta rotvec): ['-0.019', '-0.040', '-0.003'] .. ['0.033', '0.105', '0.010'] ✓ Reference rotation: ['0.9996', '0.0005', '-0.0281', '-0.0001'] ✓ Position range from dataset: ['-0.406', '-0.364', '0.918'] .. ['0.112', '0.269', '1.177'] Starting training for 9999 epochs... ✓ Rotation loss SKIPPED ✓ EMA loss weights initialized (vol=11.8, rot=3.5, grip=0.69) ============================================================ Epoch 0/9999 ============================================================ Train Loss: 1.9503 (Volume: 1.1054, Gripper: 0.8449, Rotation: 0.0000) Val - Loss: 1.5596, Volume: 11.2388, Pixel Error: 26.87px, Height Error: 68.489mm, Gripper: 0.3438 ✓ Saved best model (val_loss=1.5596) ============================================================ Epoch 1/9999 ============================================================ Train Loss: 0.9502 (Volume: 0.5585, Gripper: 0.3917, Rotation: 0.0000) Val - Loss: 0.9040, Volume: 10.2701, Pixel Error: 69.66px, Height Error: 56.739mm, Gripper: 0.6250 ✓ Saved best model (val_loss=0.9040) ============================================================ Epoch 2/9999 ============================================================ Train Loss: 0.7397 (Volume: 0.3297, Gripper: 0.4100, Rotation: 0.0000) Val - Loss: 3.5271, Volume: 11.8046, Pixel Error: 278.33px, Height Error: 55.066mm, Gripper: 1.1875 ============================================================ Epoch 3/9999 ============================================================ Train Loss: 2.0336 (Volume: 0.9329, Gripper: 1.1007, Rotation: 0.0000) Val - Loss: 2.3633, Volume: 10.4702, Pixel Error: 29.86px, Height Error: 48.594mm, Gripper: 0.5625 ============================================================ Epoch 4/9999 ============================================================ Train Loss: 1.3732 (Volume: 0.7433, Gripper: 0.6298, Rotation: 0.0000) Val - Loss: 1.1282, Volume: 7.8929, Pixel Error: 26.34px, Height Error: 50.121mm, Gripper: 0.3125 ============================================================ Epoch 5/9999 ============================================================ Train Loss: 0.8457 (Volume: 0.4505, Gripper: 0.3952, Rotation: 0.0000) Val - Loss: 1.1834, Volume: 7.3224, Pixel Error: 25.96px, Height Error: 50.761mm, Gripper: 0.2188 ============================================================ Epoch 6/9999 ============================================================ Train Loss: 0.6764 (Volume: 0.3569, Gripper: 0.3195, Rotation: 0.0000) Val - Loss: 0.6837, Volume: 6.9663, Pixel Error: 24.28px, Height Error: 47.418mm, Gripper: 0.3125 ✓ Saved best model (val_loss=0.6837) ============================================================ Epoch 7/9999 ============================================================ Train Loss: 0.5768 (Volume: 0.2923, Gripper: 0.2846, Rotation: 0.0000) Val - Loss: 0.5931, Volume: 6.2940, Pixel Error: 24.48px, Height Error: 49.353mm, Gripper: 0.3125 ✓ Saved best model (val_loss=0.5931) ============================================================ Epoch 8/9999 ============================================================ Train Loss: 0.3666 (Volume: 0.2084, Gripper: 0.1583, Rotation: 0.0000) Val - Loss: 0.6000, Volume: 5.9811, Pixel Error: 23.17px, Height Error: 47.674mm, Gripper: 0.4062 ============================================================ Epoch 9/9999 ============================================================ Train Loss: 0.3129 (Volume: 0.1583, Gripper: 0.1546, Rotation: 0.0000) Val - Loss: 0.4181, Volume: 5.8629, Pixel Error: 21.60px, Height Error: 40.815mm, Gripper: 0.2812 ✓ Saved best model (val_loss=0.4181) ============================================================ Epoch 10/9999 ============================================================ Train Loss: 0.2671 (Volume: 0.1400, Gripper: 0.1270, Rotation: 0.0000) Val - Loss: 0.4748, Volume: 5.8607, Pixel Error: 28.06px, Height Error: 47.198mm, Gripper: 0.5312 ============================================================ Epoch 11/9999 ============================================================ Train Loss: 0.1923 (Volume: 0.1061, Gripper: 0.0862, Rotation: 0.0000) Val - Loss: 0.4110, Volume: 5.6224, Pixel Error: 22.48px, Height Error: 44.946mm, Gripper: 0.3750 ✓ Saved best model (val_loss=0.4110) ============================================================ Epoch 12/9999 ============================================================ Train Loss: 0.1313 (Volume: 0.0691, Gripper: 0.0622, Rotation: 0.0000) Val - Loss: 0.3159, Volume: 5.7715, Pixel Error: 23.08px, Height Error: 45.368mm, Gripper: 0.2500 ✓ Saved best model (val_loss=0.3159) ============================================================ Epoch 13/9999 ============================================================ Train Loss: 0.2047 (Volume: 0.0944, Gripper: 0.1103, Rotation: 0.0000) Val - Loss: 0.3201, Volume: 5.6831, Pixel Error: 22.97px, Height Error: 32.761mm, Gripper: 0.3125 ============================================================ Epoch 14/9999 ============================================================ Train Loss: 0.3501 (Volume: 0.1604, Gripper: 0.1897, Rotation: 0.0000) Val - Loss: 0.4972, Volume: 6.2949, Pixel Error: 25.72px, Height Error: 48.063mm, Gripper: 0.4062 ============================================================ Epoch 15/9999 ============================================================ Train Loss: 0.1622 (Volume: 0.1072, Gripper: 0.0550, Rotation: 0.0000) Val - Loss: 0.4309, Volume: 5.6990, Pixel Error: 23.38px, Height Error: 43.062mm, Gripper: 0.3125 ============================================================ Epoch 16/9999 ============================================================ Train Loss: 0.0921 (Volume: 0.0488, Gripper: 0.0434, Rotation: 0.0000) Val - Loss: 0.6335, Volume: 5.3574, Pixel Error: 21.54px, Height Error: 42.506mm, Gripper: 0.4062 ============================================================ Epoch 17/9999 ============================================================ Train Loss: 0.0498 (Volume: 0.0317, Gripper: 0.0181, Rotation: 0.0000) Val - Loss: 0.3955, Volume: 5.2838, Pixel Error: 19.46px, Height Error: 42.382mm, Gripper: 0.3125 ============================================================ Epoch 18/9999 ============================================================ Train Loss: 0.0428 (Volume: 0.0216, Gripper: 0.0213, Rotation: 0.0000) Val - Loss: 0.4193, Volume: 5.2340, Pixel Error: 23.32px, Height Error: 43.673mm, Gripper: 0.3750 ============================================================ Epoch 19/9999 ============================================================ Train Loss: 0.0127 (Volume: 0.0095, Gripper: 0.0032, Rotation: 0.0000) Val - Loss: 0.5283, Volume: 5.2334, Pixel Error: 20.14px, Height Error: 39.809mm, Gripper: 0.3125 ============================================================ Epoch 20/9999 ============================================================ Train Loss: 0.0040 (Volume: 0.0028, Gripper: 0.0012, Rotation: 0.0000) Val - Loss: 0.5444, Volume: 5.1773, Pixel Error: 19.94px, Height Error: 40.201mm, Gripper: 0.3125 ============================================================ Epoch 21/9999 ============================================================ Train Loss: 0.0019 (Volume: 0.0012, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.5550, Volume: 5.1737, Pixel Error: 19.94px, Height Error: 40.201mm, Gripper: 0.3125 ============================================================ Epoch 22/9999 ============================================================ Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.5694, Volume: 5.1824, Pixel Error: 20.14px, Height Error: 40.201mm, Gripper: 0.3125 ============================================================ Epoch 23/9999 ============================================================ Train Loss: 0.0008 (Volume: 0.0004, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.5764, Volume: 5.1863, Pixel Error: 20.04px, Height Error: 40.201mm, Gripper: 0.3125 ============================================================ Epoch 24/9999 ============================================================ Train Loss: 0.0006 (Volume: 0.0003, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.5859, Volume: 5.1875, Pixel Error: 20.04px, Height Error: 40.201mm, Gripper: 0.3125 ============================================================ Epoch 25/9999 ============================================================ Train Loss: 0.0005 (Volume: 0.0003, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.5941, Volume: 5.1870, Pixel Error: 20.04px, Height Error: 40.201mm, Gripper: 0.3125 ============================================================ Epoch 26/9999 ============================================================ Saved step checkpoint: step_1000.pth Train Loss: 0.0004 (Volume: 0.0002, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.5992, Volume: 5.1885, Pixel Error: 20.12px, Height Error: 40.201mm, Gripper: 0.3125 ============================================================ Epoch 27/9999 ============================================================ Train Loss: 0.0004 (Volume: 0.0002, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.6084, Volume: 5.1891, Pixel Error: 20.12px, Height Error: 40.201mm, Gripper: 0.3125 ============================================================ Epoch 28/9999 ============================================================ Train Loss: 0.0003 (Volume: 0.0002, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.6202, Volume: 5.1919, Pixel Error: 20.12px, Height Error: 40.201mm, Gripper: 0.3125 ============================================================ Epoch 29/9999 ============================================================ Train Loss: 0.0003 (Volume: 0.0002, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6277, Volume: 5.1930, Pixel Error: 19.38px, Height Error: 39.939mm, Gripper: 0.2812 ============================================================ Epoch 30/9999 ============================================================ Train Loss: 0.0003 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6370, Volume: 5.1949, Pixel Error: 19.38px, Height Error: 39.939mm, Gripper: 0.2812 ============================================================ Epoch 31/9999 ============================================================ Train Loss: 0.0003 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6413, Volume: 5.1972, Pixel Error: 21.95px, Height Error: 41.246mm, Gripper: 0.3125 ============================================================ Epoch 32/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6473, Volume: 5.1980, Pixel Error: 21.95px, Height Error: 41.246mm, Gripper: 0.3125 ============================================================ Epoch 33/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6556, Volume: 5.2009, Pixel Error: 21.84px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 34/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6625, Volume: 5.2035, Pixel Error: 21.84px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 35/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6683, Volume: 5.2071, Pixel Error: 21.84px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 36/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6734, Volume: 5.2085, Pixel Error: 21.84px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 37/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6774, Volume: 5.2106, Pixel Error: 21.84px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 38/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6810, Volume: 5.2117, Pixel Error: 21.84px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 39/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6880, Volume: 5.2141, Pixel Error: 21.84px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 40/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6942, Volume: 5.2160, Pixel Error: 21.83px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 41/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6992, Volume: 5.2173, Pixel Error: 21.83px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 42/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.7042, Volume: 5.2186, Pixel Error: 21.67px, Height Error: 40.984mm, Gripper: 0.3125 ============================================================ Epoch 43/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7054, Volume: 5.2199, Pixel Error: 21.83px, Height Error: 40.984mm, Gripper: 0.3125 ⏱ Time limit reached (10.0 / 10 min). Stopping.