Loading dataset... CachedTrajectoryDataset: 128 demos, 4096 samples Source: {'cache_root': '/data/libero/ood_objpos_splits_2view/exp3_left_train', 'benchmark': 'libero_spatial', 'task_ids': [0], 'frame_stride': 3} Total: 4096 samples ✓ Train: 3892 samples ✓ Val: 204 samples Initializing model (type=dual_para)... Loading DINOv2 model... ✓ DINOv2 backbone is trainable ✓ Shared feature convs: 3× Conv2d(3×3) at pred_size=64 ✓ Per-view heads: volume(4×32), gripper(4×2), rotation(4×3×32) ✓ DualPara: 33,080,720 / 33,080,720 trainable params Trainable parameters: 33,080,720 / 33,080,720 (100.00%) Computing dataset stats from random subset: 500/4096 samples (seed=42) Epochs: 0%| | 2/9999 [05:02<423:44:56, 152.60s/it] ✓ Saved stats cache: /data/cameron/para_normalized_losses/libero/checkpoints/dual_para_exp3_left_2v/dataset_stats.json ✓ Height range from dataset: [0.919047, 1.177162] m ✓ Gripper range from dataset: [-1.000000, 1.000000] ✓ Rotation range (delta rotvec): ['-0.020', '-0.117', '-0.001'] .. ['0.023', '0.088', '0.004'] ✓ Reference rotation: ['0.9996', '0.0007', '-0.0289', '-0.0005'] ✓ Position range from dataset: ['-0.211', '-0.165', '0.919'] .. ['0.133', '-0.001', '1.177'] Starting training for 9999 epochs... ✓ Rotation loss SKIPPED ✓ EMA loss weights initialized (vol=11.8, rot=3.5, grip=0.69) ============================================================ Epoch 0/9999 ============================================================ Train Loss: 1.2123 (Volume: 7.7749, Gripper: 0.0938, Rotation: 2.0051) Val - Loss: 28.0977, Volume: 28.0977, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ✓ Saved best model (val_loss=28.0977) ============================================================ Epoch 1/9999 ============================================================ Train Loss: 0.2561 (Volume: 5.5791, Gripper: 0.0421, Rotation: 1.7678) Val - Loss: 27.1346, Volume: 27.1346, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ✓ Saved best model (val_loss=27.1346) ============================================================ Epoch 2/9999 ============================================================ Saved step checkpoint: step_1000.pth Train Loss: 0.2716 (Volume: 4.2338, Gripper: 0.0299, Rotation: 1.6876) Val - Loss: 27.2207, Volume: 27.2207, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 3/9999 ============================================================ Train Loss: 0.3164 (Volume: 4.0873, Gripper: 0.0331, Rotation: 1.6929) Val - Loss: 26.8475, Volume: 26.8475, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ✓ Saved best model (val_loss=26.8475) ============================================================ Epoch 4/9999 ============================================================ Saved step checkpoint: step_2000.pth Train Loss: 0.1465 (Volume: 3.5372, Gripper: 0.0202, Rotation: 1.6358) Val - Loss: 26.9005, Volume: 26.9005, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 5/9999 ============================================================ Train Loss: 0.0806 (Volume: 3.1221, Gripper: 0.0126, Rotation: 1.5901) Val - Loss: 29.4099, Volume: 29.4099, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 6/9999 ============================================================ Saved step checkpoint: step_3000.pth Train Loss: 0.2370 (Volume: 3.3664, Gripper: 0.0210, Rotation: 1.6445) Val - Loss: 26.5415, Volume: 26.5415, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ✓ Saved best model (val_loss=26.5415) ============================================================ Epoch 7/9999 ============================================================ Train Loss: 0.1281 (Volume: 3.0742, Gripper: 0.0175, Rotation: 1.5996) Val - Loss: 29.5486, Volume: 29.5486, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 8/9999 ============================================================ Saved step checkpoint: step_4000.pth Train Loss: 0.0980 (Volume: 2.8790, Gripper: 0.0114, Rotation: 1.5844) Val - Loss: 29.8650, Volume: 29.8650, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 9/9999 ============================================================ Train Loss: 0.0756 (Volume: 2.7016, Gripper: 0.0102, Rotation: 1.5543) Val - Loss: 29.5305, Volume: 29.5305, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 10/9999 ============================================================ Saved step checkpoint: step_5000.pth Train Loss: 0.0094 (Volume: 2.4836, Gripper: 0.0033, Rotation: 1.5251) Val - Loss: 31.2730, Volume: 31.2730, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 11/9999 ============================================================ Train Loss: 0.0377 (Volume: 2.6092, Gripper: 0.0113, Rotation: 1.5527) Val - Loss: 29.7055, Volume: 29.7055, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 12/9999 ============================================================ Saved step checkpoint: step_6000.pth Train Loss: 0.1306 (Volume: 3.1298, Gripper: 0.0273, Rotation: 1.6243) Val - Loss: 29.1413, Volume: 29.1413, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 13/9999 ============================================================ Train Loss: 0.0932 (Volume: 2.6463, Gripper: 0.0115, Rotation: 1.5589) Val - Loss: 29.4136, Volume: 29.4136, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 14/9999 ============================================================ Saved step checkpoint: step_7000.pth Train Loss: 0.0636 (Volume: 2.5743, Gripper: 0.0111, Rotation: 1.5552) Val - Loss: 31.3618, Volume: 31.3618, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 15/9999 ============================================================ Train Loss: 0.0257 (Volume: 2.4673, Gripper: 0.0062, Rotation: 1.5293) Val - Loss: 31.0744, Volume: 31.0744, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 16/9999 ============================================================ Saved step checkpoint: step_8000.pth Train Loss: 0.0222 (Volume: 2.4742, Gripper: 0.0089, Rotation: 1.5133) Val - Loss: 32.3676, Volume: 32.3676, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 17/9999 ============================================================ Train Loss: 0.1355 (Volume: 2.7040, Gripper: 0.0192, Rotation: 1.5779) Val - Loss: 28.9276, Volume: 28.9276, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 18/9999 ============================================================ Saved step checkpoint: step_9000.pth Train Loss: 0.0337 (Volume: 2.4272, Gripper: 0.0100, Rotation: 1.5248) Val - Loss: 29.7031, Volume: 29.7031, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 19/9999 ============================================================ Train Loss: 0.1000 (Volume: 2.3855, Gripper: 0.0126, Rotation: 1.5202) Val - Loss: 32.6397, Volume: 32.6397, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 20/9999 ============================================================ Saved step checkpoint: step_10000.pth Train Loss: 0.0393 (Volume: 2.1465, Gripper: 0.0091, Rotation: 1.4699) Val - Loss: 30.5776, Volume: 30.5776, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 21/9999 ============================================================ Train Loss: 0.0408 (Volume: 2.1786, Gripper: 0.0105, Rotation: 1.4781) Val - Loss: 32.4085, Volume: 32.4085, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 22/9999 ============================================================ Saved step checkpoint: step_11000.pth Train Loss: 0.0446 (Volume: 2.2451, Gripper: 0.0114, Rotation: 1.4748) Val - Loss: 33.0286, Volume: 33.0286, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 23/9999 ============================================================ Train Loss: 0.0314 (Volume: 2.1550, Gripper: 0.0082, Rotation: 1.4647) Val - Loss: 34.9595, Volume: 34.9595, Pixel Error: 0.00px, Height Error: 0.000mm, Gripper: 0.0000 ⏱ Time limit reached (61.0 / 60 min). Stopping.