Loading dataset... CachedTrajectoryDataset: 10 demos, 320 samples Source: {'cache_root': '/data/libero/ood_objpos_v3_splits/exp_10demo_finetune_train', 'benchmark': 'libero_spatial', 'task_ids': [0], 'frame_stride': 3} Total: 320 samples ✓ Train: 304 samples ✓ Val: 16 samples Initializing model (type=para)... Loading DINOv2 model... ✓ DINOv2 backbone is trainable ✓ DINO embedding dim: 384 ✓ Learnable start keypoint embedding (dim=384) ✓ Feature convs: 3× Conv2d(3×3) at pred_size=64 ✓ Volume head → (B, 4, 32, 64, 64) ✓ Gripper head → (B, 4, 2, 64, 64) [1×1 conv, CE 2-class] ✓ Rotation head → (B, 4, 3, 32, 64, 64) [1×1 conv, CE] Trainable parameters: 32,880,520 / 32,880,520 (100.00%) Loading checkpoint: checkpoints/para_finetune_init_circle_v3/best.pth ⚠ Could not load optimizer state: 'optimizer_state_dict' ✓ Resumed from epoch 1 Computing dataset stats from random subset: 320/320 samples (seed=42) /data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/wandb/sdk/data_types/image.py:324: DeprecationWarning: 'mode' parameter is deprecated and will be removed in Pillow 13 (2026-10-15) ✓ Saved stats cache: /data/cameron/para_normalized_losses/libero/checkpoints/para_finetune_circle_v3/dataset_stats.json ✓ Height range from dataset: [0.918027, 1.177156] m ✓ Gripper range from dataset: [-1.000000, 1.000000] ✓ Rotation range (delta rotvec): ['-0.019', '-0.040', '-0.003'] .. ['0.033', '0.105', '0.010'] ✓ Reference rotation: ['0.9996', '0.0005', '-0.0281', '-0.0001'] ✓ Position range from dataset: ['-0.406', '-0.364', '0.918'] .. ['0.112', '0.269', '1.177'] Starting training for 9999 epochs... ✓ Rotation loss SKIPPED ✓ EMA loss weights initialized (vol=11.8, rot=3.5, grip=0.69) self._image = pil_image.fromarray( Epochs: 0%| | 4/9998 [01:12<52:31:11, 18.92s/it] ============================================================ Epoch 1/9999 ============================================================ Train Loss: 2.1001 (Volume: 1.1878, Gripper: 0.9123, Rotation: 0.0000) Val - Loss: 1.5775, Volume: 10.7967, Pixel Error: 108.16px, Height Error: 75.043mm, Gripper: 0.8750 ✓ Saved best model (val_loss=1.5775) ============================================================ Epoch 2/9999 ============================================================ Train Loss: 0.7643 (Volume: 0.4662, Gripper: 0.2982, Rotation: 0.0000) Val - Loss: 0.7868, Volume: 9.1666, Pixel Error: 36.54px, Height Error: 46.507mm, Gripper: 0.2500 ✓ Saved best model (val_loss=0.7868) ============================================================ Epoch 3/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.4490 (Volume: 0.2560, Gripper: 0.1930, Rotation: 0.0000) Val - Loss: 0.6336, Volume: 7.7443, Pixel Error: 28.34px, Height Error: 37.617mm, Gripper: 0.2188 ✓ Saved best model (val_loss=0.6336) ============================================================ Epoch 4/9999 ============================================================ Train Loss: 0.2859 (Volume: 0.1516, Gripper: 0.1343, Rotation: 0.0000) Val - Loss: 0.5836, Volume: 7.0201, Pixel Error: 17.91px, Height Error: 32.737mm, Gripper: 0.2188 ✓ Saved best model (val_loss=0.5836) ============================================================ Epoch 5/9999 ============================================================ Train Loss: 0.1963 (Volume: 0.1023, Gripper: 0.0941, Rotation: 0.0000) Val - Loss: 0.6406, Volume: 7.0502, Pixel Error: 21.34px, Height Error: 37.014mm, Gripper: 0.2812 ============================================================ Epoch 6/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.1313 (Volume: 0.0797, Gripper: 0.0516, Rotation: 0.0000) Val - Loss: 0.7247, Volume: 6.2236, Pixel Error: 16.72px, Height Error: 29.907mm, Gripper: 0.2500 ============================================================ Epoch 7/9999 ============================================================ Train Loss: 0.0955 (Volume: 0.0492, Gripper: 0.0462, Rotation: 0.0000) Val - Loss: 0.5932, Volume: 6.1529, Pixel Error: 19.98px, Height Error: 34.895mm, Gripper: 0.1562 ============================================================ Epoch 8/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0773 (Volume: 0.0387, Gripper: 0.0386, Rotation: 0.0000) Val - Loss: 0.5937, Volume: 5.7106, Pixel Error: 15.67px, Height Error: 29.508mm, Gripper: 0.2188 ============================================================ Epoch 9/9999 ============================================================ Train Loss: 0.0678 (Volume: 0.0371, Gripper: 0.0307, Rotation: 0.0000) Val - Loss: 0.5398, Volume: 5.9570, Pixel Error: 14.05px, Height Error: 23.694mm, Gripper: 0.2500 ✓ Saved best model (val_loss=0.5398) ============================================================ Epoch 10/9999 ============================================================ Train Loss: 0.0585 (Volume: 0.0296, Gripper: 0.0289, Rotation: 0.0000) Val - Loss: 0.6828, Volume: 5.6091, Pixel Error: 16.41px, Height Error: 34.379mm, Gripper: 0.0938 ============================================================ Epoch 11/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0396 (Volume: 0.0226, Gripper: 0.0171, Rotation: 0.0000) Val - Loss: 0.8649, Volume: 5.5274, Pixel Error: 17.25px, Height Error: 27.424mm, Gripper: 0.1875 ============================================================ Epoch 12/9999 ============================================================ Train Loss: 0.0187 (Volume: 0.0106, Gripper: 0.0081, Rotation: 0.0000) Val - Loss: 1.1273, Volume: 5.4380, Pixel Error: 17.67px, Height Error: 31.488mm, Gripper: 0.2188 ============================================================ Epoch 13/9999 ============================================================ Train Loss: 0.0359 (Volume: 0.0172, Gripper: 0.0187, Rotation: 0.0000) Val - Loss: 0.5880, Volume: 5.3050, Pixel Error: 15.26px, Height Error: 24.530mm, Gripper: 0.0938 ============================================================ Epoch 14/9999 ============================================================ [async eval] launched at step 500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0299 (Volume: 0.0124, Gripper: 0.0175, Rotation: 0.0000) Val - Loss: 0.7188, Volume: 5.1512, Pixel Error: 14.56px, Height Error: 28.124mm, Gripper: 0.1875 ============================================================ Epoch 15/9999 ============================================================ Train Loss: 0.1038 (Volume: 0.0443, Gripper: 0.0595, Rotation: 0.0000) Val - Loss: 0.7150, Volume: 5.6575, Pixel Error: 18.69px, Height Error: 22.172mm, Gripper: 0.2500 ============================================================ Epoch 16/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0423 (Volume: 0.0315, Gripper: 0.0108, Rotation: 0.0000) Val - Loss: 0.7679, Volume: 5.0550, Pixel Error: 13.98px, Height Error: 24.546mm, Gripper: 0.1250 ============================================================ Epoch 17/9999 ============================================================ Train Loss: 0.0142 (Volume: 0.0077, Gripper: 0.0066, Rotation: 0.0000) Val - Loss: 0.6005, Volume: 5.0017, Pixel Error: 13.75px, Height Error: 24.752mm, Gripper: 0.1250 ============================================================ Epoch 18/9999 ============================================================ Train Loss: 0.0198 (Volume: 0.0102, Gripper: 0.0095, Rotation: 0.0000) Val - Loss: 0.7010, Volume: 4.8655, Pixel Error: 12.77px, Height Error: 23.570mm, Gripper: 0.1875 ============================================================ Epoch 19/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0691 (Volume: 0.0254, Gripper: 0.0437, Rotation: 0.0000) Val - Loss: 0.6571, Volume: 5.0555, Pixel Error: 13.24px, Height Error: 17.872mm, Gripper: 0.1562 ============================================================ Epoch 20/9999 ============================================================ Train Loss: 0.0650 (Volume: 0.0381, Gripper: 0.0268, Rotation: 0.0000) Val - Loss: 0.6206, Volume: 5.0172, Pixel Error: 14.23px, Height Error: 27.600mm, Gripper: 0.1875 ============================================================ Epoch 21/9999 ============================================================ Train Loss: 0.2413 (Volume: 0.0804, Gripper: 0.1609, Rotation: 0.0000) Val - Loss: 1.4235, Volume: 6.1410, Pixel Error: 21.98px, Height Error: 32.179mm, Gripper: 0.3438 ============================================================ Epoch 22/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 Train Loss: 0.2433 (Volume: 0.1475, Gripper: 0.0958, Rotation: 0.0000) Val - Loss: 0.6424, Volume: 5.3632, Pixel Error: 15.48px, Height Error: 26.956mm, Gripper: 0.1250 ============================================================ Epoch 23/9999 ============================================================ Train Loss: 0.0572 (Volume: 0.0445, Gripper: 0.0127, Rotation: 0.0000) Val - Loss: 0.2869, Volume: 4.7694, Pixel Error: 13.87px, Height Error: 23.320mm, Gripper: 0.1562 ✓ Saved best model (val_loss=0.2869) ============================================================ Epoch 24/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0553 (Volume: 0.0276, Gripper: 0.0277, Rotation: 0.0000) Val - Loss: 0.7067, Volume: 4.8627, Pixel Error: 14.21px, Height Error: 24.447mm, Gripper: 0.1875 ============================================================ Epoch 25/9999 ============================================================ Train Loss: 0.0152 (Volume: 0.0116, Gripper: 0.0035, Rotation: 0.0000) Val - Loss: 0.7592, Volume: 4.6174, Pixel Error: 12.96px, Height Error: 20.272mm, Gripper: 0.1875 ============================================================ Epoch 26/9999 ============================================================ Train Loss: 0.0676 (Volume: 0.0229, Gripper: 0.0447, Rotation: 0.0000) Val - Loss: 0.3991, Volume: 4.7968, Pixel Error: 10.84px, Height Error: 21.240mm, Gripper: 0.0625 ============================================================ Epoch 27/9999 ============================================================ [async eval] launched at step 1000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_1000.pth Train Loss: 0.0630 (Volume: 0.0380, Gripper: 0.0251, Rotation: 0.0000) Val - Loss: 0.6687, Volume: 4.5822, Pixel Error: 12.01px, Height Error: 20.253mm, Gripper: 0.0938 ============================================================ Epoch 28/9999 ============================================================ Train Loss: 0.0185 (Volume: 0.0138, Gripper: 0.0046, Rotation: 0.0000) Val - Loss: 0.4226, Volume: 4.5056, Pixel Error: 12.02px, Height Error: 23.152mm, Gripper: 0.1562 ============================================================ Epoch 29/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0062 (Volume: 0.0041, Gripper: 0.0021, Rotation: 0.0000) Val - Loss: 0.8488, Volume: 4.3804, Pixel Error: 10.92px, Height Error: 19.543mm, Gripper: 0.1250 ============================================================ Epoch 30/9999 ============================================================ Train Loss: 0.0266 (Volume: 0.0094, Gripper: 0.0172, Rotation: 0.0000) Val - Loss: 0.9559, Volume: 4.7484, Pixel Error: 14.80px, Height Error: 25.744mm, Gripper: 0.2188 ============================================================ Epoch 31/9999 ============================================================ Train Loss: 0.0392 (Volume: 0.0195, Gripper: 0.0196, Rotation: 0.0000) Val - Loss: 0.7192, Volume: 4.5371, Pixel Error: 13.30px, Height Error: 23.345mm, Gripper: 0.1562 ============================================================ Epoch 32/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0121 (Volume: 0.0099, Gripper: 0.0022, Rotation: 0.0000) Val - Loss: 0.5957, Volume: 4.3701, Pixel Error: 13.49px, Height Error: 24.198mm, Gripper: 0.1250 ============================================================ Epoch 33/9999 ============================================================ Train Loss: 0.0021 (Volume: 0.0018, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.6074, Volume: 4.3282, Pixel Error: 14.03px, Height Error: 21.628mm, Gripper: 0.0938 ============================================================ Epoch 34/9999 ============================================================ Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.6236, Volume: 4.3210, Pixel Error: 13.12px, Height Error: 21.586mm, Gripper: 0.0938 ============================================================ Epoch 35/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 Train Loss: 0.0004 (Volume: 0.0002, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6444, Volume: 4.3154, Pixel Error: 13.05px, Height Error: 21.486mm, Gripper: 0.0938 ============================================================ Epoch 36/9999 ============================================================ Train Loss: 0.0003 (Volume: 0.0002, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.6516, Volume: 4.3150, Pixel Error: 13.05px, Height Error: 21.486mm, Gripper: 0.0938 ============================================================ Epoch 37/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6573, Volume: 4.3149, Pixel Error: 13.05px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 38/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6676, Volume: 4.3146, Pixel Error: 13.05px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 39/9999 ============================================================ Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6768, Volume: 4.3142, Pixel Error: 13.05px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 40/9999 ============================================================ [async eval] launched at step 1500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0002 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6849, Volume: 4.3154, Pixel Error: 13.05px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 41/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.6938, Volume: 4.3147, Pixel Error: 13.05px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 42/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.7036, Volume: 4.3152, Pixel Error: 13.05px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 43/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7089, Volume: 4.3148, Pixel Error: 13.05px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 44/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.7182, Volume: 4.3158, Pixel Error: 12.98px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 45/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.7227, Volume: 4.3164, Pixel Error: 12.98px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 46/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0001, Gripper: 0.0001, Rotation: 0.0000) Val - Loss: 0.7303, Volume: 4.3172, Pixel Error: 12.98px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 47/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7360, Volume: 4.3183, Pixel Error: 12.98px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 48/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7399, Volume: 4.3187, Pixel Error: 12.98px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 49/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7445, Volume: 4.3156, Pixel Error: 12.98px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 50/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7468, Volume: 4.3164, Pixel Error: 12.98px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 51/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7508, Volume: 4.3170, Pixel Error: 12.98px, Height Error: 20.833mm, Gripper: 0.0938 ============================================================ Epoch 52/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7559, Volume: 4.3178, Pixel Error: 12.90px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 53/9999 ============================================================ [async eval] launched at step 2000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_2000.pth Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7620, Volume: 4.3184, Pixel Error: 12.90px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 54/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7644, Volume: 4.3187, Pixel Error: 12.90px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 55/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7679, Volume: 4.3193, Pixel Error: 12.90px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 56/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7713, Volume: 4.3201, Pixel Error: 12.90px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 57/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7745, Volume: 4.3204, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 58/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7806, Volume: 4.3210, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 59/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7821, Volume: 4.3222, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 60/9999 ============================================================ Train Loss: 0.0001 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7859, Volume: 4.3230, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 61/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7893, Volume: 4.3234, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 62/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7910, Volume: 4.3240, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 63/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7926, Volume: 4.3237, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 64/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7941, Volume: 4.3242, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 65/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.7983, Volume: 4.3252, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 66/9999 ============================================================ [async eval] launched at step 2500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8023, Volume: 4.3261, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 67/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8041, Volume: 4.3270, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 68/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8057, Volume: 4.3281, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 69/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8077, Volume: 4.3291, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 70/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8105, Volume: 4.3299, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 71/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8126, Volume: 4.3307, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 72/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8135, Volume: 4.3312, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 73/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8141, Volume: 4.3318, Pixel Error: 12.83px, Height Error: 20.964mm, Gripper: 0.0938 ============================================================ Epoch 74/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8168, Volume: 4.3327, Pixel Error: 13.59px, Height Error: 22.140mm, Gripper: 0.0938 ============================================================ Epoch 75/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8184, Volume: 4.3334, Pixel Error: 13.59px, Height Error: 22.140mm, Gripper: 0.0938 ============================================================ Epoch 76/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8209, Volume: 4.3209, Pixel Error: 13.59px, Height Error: 22.140mm, Gripper: 0.0938 ============================================================ Epoch 77/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8234, Volume: 4.3208, Pixel Error: 13.59px, Height Error: 22.140mm, Gripper: 0.0938 ============================================================ Epoch 78/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8248, Volume: 4.3215, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 79/9999 ============================================================ [async eval] launched at step 3000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Saved step checkpoint: step_3000.pth Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8283, Volume: 4.3226, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 80/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8302, Volume: 4.3230, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 81/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8336, Volume: 4.3244, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 82/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8359, Volume: 4.3252, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 83/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8388, Volume: 4.3261, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 84/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8399, Volume: 4.3272, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 85/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8414, Volume: 4.3282, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 86/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8412, Volume: 4.3307, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 87/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8440, Volume: 4.3315, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 88/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8460, Volume: 4.3326, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 89/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8474, Volume: 4.3330, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 90/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8535, Volume: 4.3329, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 91/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8567, Volume: 4.3339, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 92/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8584, Volume: 4.3341, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 93/9999 ============================================================ [async eval] launched at step 3500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8590, Volume: 4.3350, Pixel Error: 13.59px, Height Error: 22.401mm, Gripper: 0.0938 ============================================================ Epoch 94/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8610, Volume: 4.3357, Pixel Error: 13.48px, Height Error: 22.140mm, Gripper: 0.0938 ============================================================ Epoch 95/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8625, Volume: 4.3368, Pixel Error: 13.48px, Height Error: 22.140mm, Gripper: 0.0938 ============================================================ Epoch 96/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8637, Volume: 4.3377, Pixel Error: 13.48px, Height Error: 22.140mm, Gripper: 0.0938 ============================================================ Epoch 97/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8652, Volume: 4.3382, Pixel Error: 13.48px, Height Error: 22.140mm, Gripper: 0.0938 ============================================================ Epoch 98/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8668, Volume: 4.3391, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 99/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8667, Volume: 4.3395, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 100/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8683, Volume: 4.3406, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 101/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8700, Volume: 4.3410, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 102/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8723, Volume: 4.3419, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 103/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8727, Volume: 4.3422, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 104/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8738, Volume: 4.3434, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 105/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8771, Volume: 4.3442, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 106/9999 ============================================================ [async eval] launched at step 4000 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 Saved step checkpoint: step_4000.pth Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8791, Volume: 4.3449, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 107/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8812, Volume: 4.3458, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 108/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8839, Volume: 4.3465, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 109/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8870, Volume: 4.3474, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 110/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8901, Volume: 4.3481, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 111/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8917, Volume: 4.3484, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 112/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8932, Volume: 4.3492, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 113/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8958, Volume: 4.3499, Pixel Error: 13.48px, Height Error: 22.270mm, Gripper: 0.0938 ============================================================ Epoch 114/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8972, Volume: 4.3507, Pixel Error: 13.59px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 115/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8975, Volume: 4.3514, Pixel Error: 13.59px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 116/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8991, Volume: 4.3523, Pixel Error: 13.59px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 117/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.8994, Volume: 4.3526, Pixel Error: 13.59px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 118/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9017, Volume: 4.3536, Pixel Error: 13.59px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 119/9999 ============================================================ [async eval] launched at step 4500 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9034, Volume: 4.3544, Pixel Error: 13.59px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 120/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9018, Volume: 4.3551, Pixel Error: 13.59px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 121/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9031, Volume: 4.3560, Pixel Error: 13.56px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 122/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9040, Volume: 4.3568, Pixel Error: 13.56px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 123/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9059, Volume: 4.3575, Pixel Error: 13.56px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 124/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9072, Volume: 4.3579, Pixel Error: 13.65px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 125/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9087, Volume: 4.3585, Pixel Error: 13.65px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 126/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9104, Volume: 4.3596, Pixel Error: 13.56px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 127/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9124, Volume: 4.3600, Pixel Error: 13.65px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 128/9999 ============================================================ Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9139, Volume: 4.3610, Pixel Error: 13.65px, Height Error: 23.707mm, Gripper: 0.0938 ============================================================ Epoch 129/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0000 (Volume: 0.0000, Gripper: 0.0000, Rotation: 0.0000) Val - Loss: 0.9165, Volume: 4.3618, Pixel Error: 13.65px, Height Error: 23.707mm, Gripper: 0.0938 ⏱ Time limit reached (30.0 / 30 min). Stopping.