Loading dataset... CachedTrajectoryDataset: 64 demos, 2048 samples Source: {'cache_root': '/data/libero/ood_objpos_v3_splits/exp4_n64_train', 'benchmark': 'libero_spatial', 'task_ids': [0], 'frame_stride': 3} Total: 2048 samples ✓ Train: 1946 samples ✓ Val: 102 samples Initializing model (type=act)... Loading DINOv2 model... ✓ DINOv2 backbone is trainable ✓ CLIP projection: 512 → 384 ✓ ACT model: 30,394,048 / 30,394,048 trainable params Input: CLS(384) + start_kp(2) + eef_pos(3) + gripper(1) = 774 pos_mlp: → (B, 4, 3) [sigmoid, normalized] rot_mlp: → (B, 4, 3) [sigmoid, normalized] gripper_mlp: → (B, 4) [sigmoid, normalized] Trainable parameters: 30,394,048 / 30,394,048 (100.00%) Computing dataset stats from random subset: 500/2048 samples (seed=42) /data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/wandb/sdk/data_types/image.py:324: DeprecationWarning: 'mode' parameter is deprecated and will be removed in Pillow 13 (2026-10-15) ✓ Saved stats cache: /data/cameron/567_augmentation_viewpoint_project/checkpoints/act_persp_long/dataset_stats.json ✓ Height range from dataset: [0.917341, 1.177162] m ✓ Gripper range from dataset: [-1.000000, 1.000000] ✓ Rotation range (delta rotvec): ['-0.107', '-0.076', '-0.020'] .. ['0.040', '0.249', '0.020'] ✓ Reference rotation: ['0.9994', '-0.0011', '-0.0338', '-0.0038'] ✓ Position range from dataset: ['-0.408', '-0.366', '0.917'] .. ['0.132', '0.273', '1.177'] Starting training for 9999 epochs... ✓ Rotation loss SKIPPED ✓ EMA loss weights initialized (vol=11.8, rot=3.5, grip=0.69) self._image = pil_image.fromarray( Epochs: 0%| | 1/9999 [01:11<199:53:18, 71.97s/it] ============================================================ Epoch 0/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.2967 (Volume: 0.0341, Gripper: 0.2626, Rotation: 0.0000) Val - Loss: 0.3192, Volume: 0.0078, Pixel Error: 15.54px, Height Error: 0.000mm, Gripper: 0.1863 ✓ Saved best model (val_loss=0.3192) ============================================================ Epoch 1/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0182 (Volume: 0.0090, Gripper: 0.0092, Rotation: 0.0000) Val - Loss: 0.2284, Volume: 0.0037, Pixel Error: 10.53px, Height Error: 0.000mm, Gripper: 0.1225 ✓ Saved best model (val_loss=0.2284) ============================================================ Epoch 2/9999 ============================================================ [async eval] launched at step 500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0120 (Volume: 0.0060, Gripper: 0.0060, Rotation: 0.0000) Val - Loss: 0.2081, Volume: 0.0021, Pixel Error: 7.82px, Height Error: 0.000mm, Gripper: 0.1029 ✓ Saved best model (val_loss=0.2081) ============================================================ Epoch 3/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0096 (Volume: 0.0048, Gripper: 0.0048, Rotation: 0.0000) Val - Loss: 0.1982, Volume: 0.0026, Pixel Error: 8.88px, Height Error: 0.000mm, Gripper: 0.0833 ✓ Saved best model (val_loss=0.1982) ============================================================ Epoch 4/9999 ============================================================ [async eval] launched at step 1000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_1000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0081 (Volume: 0.0040, Gripper: 0.0040, Rotation: 0.0000) Val - Loss: 0.1771, Volume: 0.0025, Pixel Error: 7.81px, Height Error: 0.000mm, Gripper: 0.0784 ✓ Saved best model (val_loss=0.1771) ============================================================ Epoch 5/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0070 (Volume: 0.0035, Gripper: 0.0035, Rotation: 0.0000) Val - Loss: 0.1768, Volume: 0.0017, Pixel Error: 6.71px, Height Error: 0.000mm, Gripper: 0.0735 ✓ Saved best model (val_loss=0.1768) ============================================================ Epoch 6/9999 ============================================================ [async eval] launched at step 1500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0066 (Volume: 0.0033, Gripper: 0.0032, Rotation: 0.0000) Val - Loss: 0.2082, Volume: 0.0025, Pixel Error: 7.40px, Height Error: 0.000mm, Gripper: 0.1127 ============================================================ Epoch 7/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0060 (Volume: 0.0031, Gripper: 0.0030, Rotation: 0.0000) Val - Loss: 0.1521, Volume: 0.0011, Pixel Error: 5.90px, Height Error: 0.000mm, Gripper: 0.0490 ✓ Saved best model (val_loss=0.1521) ============================================================ Epoch 8/9999 ============================================================ [async eval] launched at step 2000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_2000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0053 (Volume: 0.0027, Gripper: 0.0026, Rotation: 0.0000) Val - Loss: 0.1878, Volume: 0.0019, Pixel Error: 7.13px, Height Error: 0.000mm, Gripper: 0.0735 ============================================================ Epoch 9/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0049 (Volume: 0.0025, Gripper: 0.0024, Rotation: 0.0000) Val - Loss: 0.1497, Volume: 0.0027, Pixel Error: 6.94px, Height Error: 0.000mm, Gripper: 0.0490 ✓ Saved best model (val_loss=0.1497) ============================================================ Epoch 10/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 2500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0055 (Volume: 0.0028, Gripper: 0.0027, Rotation: 0.0000) Val - Loss: 0.1680, Volume: 0.0016, Pixel Error: 6.84px, Height Error: 0.000mm, Gripper: 0.0588 ============================================================ Epoch 11/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0053 (Volume: 0.0026, Gripper: 0.0027, Rotation: 0.0000) Val - Loss: 0.1505, Volume: 0.0012, Pixel Error: 6.15px, Height Error: 0.000mm, Gripper: 0.0441 ============================================================ Epoch 12/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 3000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_3000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0046 (Volume: 0.0023, Gripper: 0.0023, Rotation: 0.0000) Val - Loss: 0.1680, Volume: 0.0023, Pixel Error: 7.28px, Height Error: 0.000mm, Gripper: 0.0637 ============================================================ Epoch 13/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0054 (Volume: 0.0027, Gripper: 0.0027, Rotation: 0.0000) Val - Loss: 0.1648, Volume: 0.0013, Pixel Error: 5.96px, Height Error: 0.000mm, Gripper: 0.0392 ============================================================ Epoch 14/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 3500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0043 (Volume: 0.0022, Gripper: 0.0021, Rotation: 0.0000) Val - Loss: 0.1323, Volume: 0.0011, Pixel Error: 5.47px, Height Error: 0.000mm, Gripper: 0.0343 ✓ Saved best model (val_loss=0.1323) ============================================================ Epoch 15/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0062 (Volume: 0.0031, Gripper: 0.0031, Rotation: 0.0000) Val - Loss: 0.1412, Volume: 0.0016, Pixel Error: 6.44px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 16/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 4000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_4000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0044 (Volume: 0.0022, Gripper: 0.0022, Rotation: 0.0000) Val - Loss: 0.1211, Volume: 0.0011, Pixel Error: 5.21px, Height Error: 0.000mm, Gripper: 0.0196 ✓ Saved best model (val_loss=0.1211) ============================================================ Epoch 17/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0042 (Volume: 0.0021, Gripper: 0.0021, Rotation: 0.0000) Val - Loss: 0.1324, Volume: 0.0012, Pixel Error: 5.29px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 18/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 4500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0050 (Volume: 0.0026, Gripper: 0.0024, Rotation: 0.0000) Val - Loss: 0.1186, Volume: 0.0011, Pixel Error: 4.84px, Height Error: 0.000mm, Gripper: 0.0147 ✓ Saved best model (val_loss=0.1186) ============================================================ Epoch 19/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0038 (Volume: 0.0019, Gripper: 0.0019, Rotation: 0.0000) Val - Loss: 0.1322, Volume: 0.0011, Pixel Error: 5.50px, Height Error: 0.000mm, Gripper: 0.0392 ============================================================ Epoch 20/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 5000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_5000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0041 (Volume: 0.0021, Gripper: 0.0020, Rotation: 0.0000) Val - Loss: 0.1658, Volume: 0.0016, Pixel Error: 6.03px, Height Error: 0.000mm, Gripper: 0.0539 ============================================================ Epoch 21/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0044 (Volume: 0.0022, Gripper: 0.0022, Rotation: 0.0000) Val - Loss: 0.1180, Volume: 0.0011, Pixel Error: 5.02px, Height Error: 0.000mm, Gripper: 0.0196 ✓ Saved best model (val_loss=0.1180) ============================================================ Epoch 22/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 5500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0059 (Volume: 0.0030, Gripper: 0.0028, Rotation: 0.0000) Val - Loss: 0.1471, Volume: 0.0016, Pixel Error: 5.81px, Height Error: 0.000mm, Gripper: 0.0490 ============================================================ Epoch 23/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0047 (Volume: 0.0023, Gripper: 0.0023, Rotation: 0.0000) Val - Loss: 0.1520, Volume: 0.0015, Pixel Error: 7.04px, Height Error: 0.000mm, Gripper: 0.0490 ============================================================ Epoch 24/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 6000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_6000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0038 (Volume: 0.0020, Gripper: 0.0018, Rotation: 0.0000) Val - Loss: 0.1318, Volume: 0.0012, Pixel Error: 4.71px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 25/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0038 (Volume: 0.0019, Gripper: 0.0019, Rotation: 0.0000) Val - Loss: 0.1578, Volume: 0.0011, Pixel Error: 5.29px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 26/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 6500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0037 (Volume: 0.0019, Gripper: 0.0018, Rotation: 0.0000) Val - Loss: 0.2017, Volume: 0.0015, Pixel Error: 5.55px, Height Error: 0.000mm, Gripper: 0.0637 ============================================================ Epoch 27/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0037 (Volume: 0.0019, Gripper: 0.0019, Rotation: 0.0000) Val - Loss: 0.1162, Volume: 0.0009, Pixel Error: 4.22px, Height Error: 0.000mm, Gripper: 0.0294 ✓ Saved best model (val_loss=0.1162) ============================================================ Epoch 28/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 7000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_7000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0034 (Volume: 0.0017, Gripper: 0.0016, Rotation: 0.0000) Val - Loss: 0.1554, Volume: 0.0013, Pixel Error: 5.86px, Height Error: 0.000mm, Gripper: 0.0392 ============================================================ Epoch 29/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0044 (Volume: 0.0023, Gripper: 0.0022, Rotation: 0.0000) Val - Loss: 0.1329, Volume: 0.0009, Pixel Error: 4.66px, Height Error: 0.000mm, Gripper: 0.0392 ============================================================ Epoch 30/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 7500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0033 (Volume: 0.0017, Gripper: 0.0016, Rotation: 0.0000) Val - Loss: 0.1192, Volume: 0.0008, Pixel Error: 4.36px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 31/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0037 (Volume: 0.0019, Gripper: 0.0018, Rotation: 0.0000) Val - Loss: 0.1220, Volume: 0.0009, Pixel Error: 4.28px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 32/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 8000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_8000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0043 (Volume: 0.0022, Gripper: 0.0022, Rotation: 0.0000) Val - Loss: 0.1219, Volume: 0.0010, Pixel Error: 4.50px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 33/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0033 (Volume: 0.0017, Gripper: 0.0016, Rotation: 0.0000) Val - Loss: 0.1356, Volume: 0.0011, Pixel Error: 4.73px, Height Error: 0.000mm, Gripper: 0.0392 ============================================================ Epoch 34/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 8500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0046 (Volume: 0.0023, Gripper: 0.0022, Rotation: 0.0000) Val - Loss: 0.1311, Volume: 0.0011, Pixel Error: 5.19px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 35/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0038 (Volume: 0.0020, Gripper: 0.0019, Rotation: 0.0000) Val - Loss: 0.1299, Volume: 0.0010, Pixel Error: 4.87px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 36/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 9000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_9000.pth Train Loss: 0.0029 (Volume: 0.0015, Gripper: 0.0014, Rotation: 0.0000) Val - Loss: 0.1178, Volume: 0.0011, Pixel Error: 4.53px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 37/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0035 (Volume: 0.0018, Gripper: 0.0018, Rotation: 0.0000) Val - Loss: 0.1232, Volume: 0.0010, Pixel Error: 4.60px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 38/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 9500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0030 (Volume: 0.0015, Gripper: 0.0015, Rotation: 0.0000) Val - Loss: 0.1272, Volume: 0.0008, Pixel Error: 4.08px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 39/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 Train Loss: 0.0028 (Volume: 0.0014, Gripper: 0.0014, Rotation: 0.0000) Val - Loss: 0.1023, Volume: 0.0009, Pixel Error: 4.33px, Height Error: 0.000mm, Gripper: 0.0049 ✓ Saved best model (val_loss=0.1023) ============================================================ Epoch 40/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 10000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_10000.pth Train Loss: 0.0027 (Volume: 0.0014, Gripper: 0.0013, Rotation: 0.0000) Val - Loss: 0.1104, Volume: 0.0007, Pixel Error: 3.81px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 41/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 Train Loss: 0.0027 (Volume: 0.0014, Gripper: 0.0013, Rotation: 0.0000) Val - Loss: 0.1156, Volume: 0.0010, Pixel Error: 4.98px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 42/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0029 (Volume: 0.0015, Gripper: 0.0015, Rotation: 0.0000) Val - Loss: 0.1335, Volume: 0.0018, Pixel Error: 5.45px, Height Error: 0.000mm, Gripper: 0.0392 ============================================================ Epoch 43/9999 ============================================================ [async eval] launched at step 10500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0032 (Volume: 0.0016, Gripper: 0.0016, Rotation: 0.0000) Val - Loss: 0.1555, Volume: 0.0019, Pixel Error: 7.67px, Height Error: 0.000mm, Gripper: 0.0490 ============================================================ Epoch 44/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0033 (Volume: 0.0018, Gripper: 0.0015, Rotation: 0.0000) Val - Loss: 0.1245, Volume: 0.0007, Pixel Error: 3.69px, Height Error: 0.000mm, Gripper: 0.0441 ============================================================ Epoch 45/9999 ============================================================ [async eval] launched at step 11000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_11000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0029 (Volume: 0.0015, Gripper: 0.0014, Rotation: 0.0000) Val - Loss: 0.1334, Volume: 0.0009, Pixel Error: 4.41px, Height Error: 0.000mm, Gripper: 0.0490 ============================================================ Epoch 46/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0032 (Volume: 0.0016, Gripper: 0.0015, Rotation: 0.0000) Val - Loss: 0.1285, Volume: 0.0009, Pixel Error: 4.67px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 47/9999 ============================================================ [async eval] launched at step 11500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0028 (Volume: 0.0014, Gripper: 0.0014, Rotation: 0.0000) Val - Loss: 0.1220, Volume: 0.0008, Pixel Error: 4.10px, Height Error: 0.000mm, Gripper: 0.0392 ============================================================ Epoch 48/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0032 (Volume: 0.0017, Gripper: 0.0016, Rotation: 0.0000) Val - Loss: 0.1140, Volume: 0.0007, Pixel Error: 3.72px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 49/9999 ============================================================ [async eval] launched at step 12000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_12000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0027 (Volume: 0.0014, Gripper: 0.0013, Rotation: 0.0000) Val - Loss: 0.1527, Volume: 0.0012, Pixel Error: 4.76px, Height Error: 0.000mm, Gripper: 0.0539 ============================================================ Epoch 50/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0024 (Volume: 0.0013, Gripper: 0.0012, Rotation: 0.0000) Val - Loss: 0.1060, Volume: 0.0009, Pixel Error: 3.89px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 51/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 12500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0025 (Volume: 0.0013, Gripper: 0.0012, Rotation: 0.0000) Val - Loss: 0.1194, Volume: 0.0007, Pixel Error: 3.68px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 52/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0021 (Volume: 0.0011, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1106, Volume: 0.0008, Pixel Error: 3.83px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 53/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 13000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_13000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0026 (Volume: 0.0013, Gripper: 0.0013, Rotation: 0.0000) Val - Loss: 0.1120, Volume: 0.0008, Pixel Error: 3.99px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 54/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0024 (Volume: 0.0013, Gripper: 0.0012, Rotation: 0.0000) Val - Loss: 0.1618, Volume: 0.0010, Pixel Error: 4.45px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 55/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 13500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0033 (Volume: 0.0017, Gripper: 0.0016, Rotation: 0.0000) Val - Loss: 0.1088, Volume: 0.0007, Pixel Error: 3.71px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 56/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0024 (Volume: 0.0012, Gripper: 0.0012, Rotation: 0.0000) Val - Loss: 0.1101, Volume: 0.0006, Pixel Error: 3.78px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 57/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 14000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_14000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0027 (Volume: 0.0014, Gripper: 0.0013, Rotation: 0.0000) Val - Loss: 0.1124, Volume: 0.0007, Pixel Error: 4.04px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 58/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0022 (Volume: 0.0011, Gripper: 0.0011, Rotation: 0.0000) Val - Loss: 0.1153, Volume: 0.0008, Pixel Error: 3.81px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 59/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 14500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0023 (Volume: 0.0012, Gripper: 0.0011, Rotation: 0.0000) Val - Loss: 0.1086, Volume: 0.0005, Pixel Error: 3.20px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 60/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0020 (Volume: 0.0010, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1171, Volume: 0.0008, Pixel Error: 3.96px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 61/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 15000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_15000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0022 (Volume: 0.0011, Gripper: 0.0011, Rotation: 0.0000) Val - Loss: 0.1145, Volume: 0.0010, Pixel Error: 4.45px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 62/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0027 (Volume: 0.0014, Gripper: 0.0013, Rotation: 0.0000) Val - Loss: 0.1921, Volume: 0.0023, Pixel Error: 7.93px, Height Error: 0.000mm, Gripper: 0.0735 ============================================================ Epoch 63/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 15500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0032 (Volume: 0.0017, Gripper: 0.0015, Rotation: 0.0000) Val - Loss: 0.1398, Volume: 0.0006, Pixel Error: 3.63px, Height Error: 0.000mm, Gripper: 0.0392 ============================================================ Epoch 64/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0024 (Volume: 0.0012, Gripper: 0.0012, Rotation: 0.0000) Val - Loss: 0.1305, Volume: 0.0008, Pixel Error: 3.54px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 65/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 16000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_16000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0020 (Volume: 0.0010, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1349, Volume: 0.0008, Pixel Error: 3.54px, Height Error: 0.000mm, Gripper: 0.0392 ============================================================ Epoch 66/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0018 (Volume: 0.0009, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1179, Volume: 0.0006, Pixel Error: 3.49px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 67/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 16500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0020 (Volume: 0.0010, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1100, Volume: 0.0006, Pixel Error: 3.57px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 68/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0018 (Volume: 0.0010, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1259, Volume: 0.0007, Pixel Error: 3.63px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 69/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 17000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_17000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0026 (Volume: 0.0013, Gripper: 0.0013, Rotation: 0.0000) Val - Loss: 0.1129, Volume: 0.0007, Pixel Error: 3.62px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 70/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0020 (Volume: 0.0010, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1042, Volume: 0.0005, Pixel Error: 3.32px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 71/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 17500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0021 (Volume: 0.0011, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1225, Volume: 0.0007, Pixel Error: 4.08px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 72/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0027 (Volume: 0.0014, Gripper: 0.0013, Rotation: 0.0000) Val - Loss: 0.1092, Volume: 0.0006, Pixel Error: 3.55px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 73/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 18000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_18000.pth [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0022 (Volume: 0.0011, Gripper: 0.0011, Rotation: 0.0000) Val - Loss: 0.1052, Volume: 0.0007, Pixel Error: 3.73px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 74/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0021 (Volume: 0.0011, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1140, Volume: 0.0006, Pixel Error: 3.35px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 75/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 18500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0019 (Volume: 0.0010, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1092, Volume: 0.0005, Pixel Error: 3.45px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 76/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0020 (Volume: 0.0010, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1120, Volume: 0.0007, Pixel Error: 3.39px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 77/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 19000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_19000.pth Train Loss: 0.0018 (Volume: 0.0009, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1155, Volume: 0.0006, Pixel Error: 3.46px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 78/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0018 (Volume: 0.0009, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1151, Volume: 0.0005, Pixel Error: 3.41px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 79/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 19500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0021 (Volume: 0.0011, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.0970, Volume: 0.0006, Pixel Error: 3.55px, Height Error: 0.000mm, Gripper: 0.0000 ✓ Saved best model (val_loss=0.0970) ============================================================ Epoch 80/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0020 (Volume: 0.0010, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1177, Volume: 0.0008, Pixel Error: 4.16px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 81/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 20000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_20000.pth Train Loss: 0.0022 (Volume: 0.0012, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1318, Volume: 0.0008, Pixel Error: 3.89px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 82/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0019 (Volume: 0.0010, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1198, Volume: 0.0004, Pixel Error: 3.28px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 83/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0020 (Volume: 0.0010, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1169, Volume: 0.0005, Pixel Error: 3.03px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 84/9999 ============================================================ [async eval] launched at step 20500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0024 (Volume: 0.0012, Gripper: 0.0012, Rotation: 0.0000) Val - Loss: 0.1404, Volume: 0.0010, Pixel Error: 4.71px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 85/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0021 (Volume: 0.0011, Gripper: 0.0010, Rotation: 0.0000) Val - Loss: 0.1237, Volume: 0.0007, Pixel Error: 3.44px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 86/9999 ============================================================ [async eval] launched at step 21000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_21000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0014 (Volume: 0.0007, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1044, Volume: 0.0004, Pixel Error: 2.81px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 87/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0020 (Volume: 0.0010, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1051, Volume: 0.0007, Pixel Error: 3.50px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 88/9999 ============================================================ [async eval] launched at step 21500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0019 (Volume: 0.0010, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1324, Volume: 0.0006, Pixel Error: 3.62px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 89/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1256, Volume: 0.0005, Pixel Error: 3.12px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 90/9999 ============================================================ [async eval] launched at step 22000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_22000.pth [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1119, Volume: 0.0005, Pixel Error: 3.04px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 91/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0016 (Volume: 0.0008, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1107, Volume: 0.0006, Pixel Error: 3.12px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 92/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 22500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0016 (Volume: 0.0008, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1441, Volume: 0.0005, Pixel Error: 3.09px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 93/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0022 (Volume: 0.0011, Gripper: 0.0011, Rotation: 0.0000) Val - Loss: 0.1290, Volume: 0.0008, Pixel Error: 3.37px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 94/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 23000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_23000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0017 (Volume: 0.0009, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1081, Volume: 0.0004, Pixel Error: 2.86px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 95/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0019 (Volume: 0.0010, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1104, Volume: 0.0008, Pixel Error: 3.67px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 96/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 23500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0017 (Volume: 0.0009, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1071, Volume: 0.0005, Pixel Error: 2.92px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 97/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0007, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1459, Volume: 0.0006, Pixel Error: 3.39px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 98/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 24000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_24000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0016 (Volume: 0.0008, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1168, Volume: 0.0004, Pixel Error: 2.79px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 99/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1286, Volume: 0.0005, Pixel Error: 3.08px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 100/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 24500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0019 (Volume: 0.0011, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1911, Volume: 0.0015, Pixel Error: 6.15px, Height Error: 0.000mm, Gripper: 0.0441 ============================================================ Epoch 101/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0030 (Volume: 0.0015, Gripper: 0.0015, Rotation: 0.0000) Val - Loss: 0.1226, Volume: 0.0008, Pixel Error: 3.82px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 102/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 25000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_25000.pth [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0018 (Volume: 0.0010, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1079, Volume: 0.0008, Pixel Error: 3.66px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 103/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0017 (Volume: 0.0010, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1081, Volume: 0.0006, Pixel Error: 3.34px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 104/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 25500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0014 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1053, Volume: 0.0004, Pixel Error: 2.61px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 105/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.0977, Volume: 0.0004, Pixel Error: 2.80px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 106/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 26000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_26000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1124, Volume: 0.0009, Pixel Error: 3.83px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 107/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0018 (Volume: 0.0009, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1485, Volume: 0.0004, Pixel Error: 2.82px, Height Error: 0.000mm, Gripper: 0.0441 ============================================================ Epoch 108/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 26500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1212, Volume: 0.0006, Pixel Error: 2.96px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 109/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0007, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1343, Volume: 0.0006, Pixel Error: 3.33px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 110/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 27000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_27000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0007, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1158, Volume: 0.0005, Pixel Error: 3.05px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 111/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1068, Volume: 0.0006, Pixel Error: 2.98px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 112/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 27500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1129, Volume: 0.0007, Pixel Error: 3.20px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 113/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0023 (Volume: 0.0011, Gripper: 0.0011, Rotation: 0.0000) Val - Loss: 0.1260, Volume: 0.0006, Pixel Error: 3.00px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 114/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 28000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_28000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1143, Volume: 0.0005, Pixel Error: 2.62px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 115/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1114, Volume: 0.0004, Pixel Error: 2.76px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 116/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 28500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1121, Volume: 0.0005, Pixel Error: 2.94px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 117/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0012 (Volume: 0.0006, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1181, Volume: 0.0006, Pixel Error: 2.87px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 118/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 29000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_29000.pth Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1439, Volume: 0.0007, Pixel Error: 3.48px, Height Error: 0.000mm, Gripper: 0.0441 ============================================================ Epoch 119/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1178, Volume: 0.0005, Pixel Error: 2.79px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 120/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 29500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0016 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1470, Volume: 0.0007, Pixel Error: 3.51px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 121/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0016 (Volume: 0.0008, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1042, Volume: 0.0004, Pixel Error: 2.78px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 122/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 30000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_30000.pth Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1153, Volume: 0.0005, Pixel Error: 2.90px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 123/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0017 (Volume: 0.0009, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.0986, Volume: 0.0009, Pixel Error: 4.91px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 124/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 30500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0017 (Volume: 0.0009, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1248, Volume: 0.0006, Pixel Error: 3.50px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 125/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0989, Volume: 0.0004, Pixel Error: 2.53px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 126/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1058, Volume: 0.0004, Pixel Error: 2.43px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 127/9999 ============================================================ [async eval] launched at step 31000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_31000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0012 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1153, Volume: 0.0006, Pixel Error: 3.80px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 128/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0016 (Volume: 0.0008, Gripper: 0.0008, Rotation: 0.0000) Val - Loss: 0.1067, Volume: 0.0004, Pixel Error: 2.94px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 129/9999 ============================================================ [async eval] launched at step 31500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1209, Volume: 0.0006, Pixel Error: 2.94px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 130/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1047, Volume: 0.0004, Pixel Error: 2.77px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 131/9999 ============================================================ [async eval] launched at step 32000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_32000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0018 (Volume: 0.0009, Gripper: 0.0009, Rotation: 0.0000) Val - Loss: 0.1007, Volume: 0.0004, Pixel Error: 3.08px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 132/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0967, Volume: 0.0005, Pixel Error: 2.81px, Height Error: 0.000mm, Gripper: 0.0049 ✓ Saved best model (val_loss=0.0967) ============================================================ Epoch 133/9999 ============================================================ [async eval] launched at step 32500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0012 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0955, Volume: 0.0004, Pixel Error: 2.41px, Height Error: 0.000mm, Gripper: 0.0049 ✓ Saved best model (val_loss=0.0955) ============================================================ Epoch 134/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0008, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1347, Volume: 0.0005, Pixel Error: 2.62px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 135/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 33000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_33000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0014 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1057, Volume: 0.0005, Pixel Error: 3.01px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 136/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0007, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1154, Volume: 0.0006, Pixel Error: 3.02px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 137/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 33500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0991, Volume: 0.0005, Pixel Error: 2.58px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 138/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.0964, Volume: 0.0007, Pixel Error: 3.59px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 139/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 34000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_34000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0012 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1104, Volume: 0.0005, Pixel Error: 2.83px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 140/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1260, Volume: 0.0004, Pixel Error: 2.65px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 141/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 34500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1016, Volume: 0.0004, Pixel Error: 2.41px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 142/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0977, Volume: 0.0004, Pixel Error: 2.40px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 143/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 35000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_35000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1002, Volume: 0.0004, Pixel Error: 2.54px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 144/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.1311, Volume: 0.0004, Pixel Error: 2.46px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 145/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 35500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1225, Volume: 0.0005, Pixel Error: 3.03px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 146/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1190, Volume: 0.0005, Pixel Error: 2.99px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 147/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 36000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_36000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0012 (Volume: 0.0007, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1018, Volume: 0.0005, Pixel Error: 2.88px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 148/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0012 (Volume: 0.0006, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0949, Volume: 0.0004, Pixel Error: 2.28px, Height Error: 0.000mm, Gripper: 0.0049 ✓ Saved best model (val_loss=0.0949) ============================================================ Epoch 149/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 36500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1100, Volume: 0.0005, Pixel Error: 2.98px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 150/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1070, Volume: 0.0004, Pixel Error: 2.49px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 151/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 37000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_37000.pth [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1199, Volume: 0.0004, Pixel Error: 2.43px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 152/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1005, Volume: 0.0003, Pixel Error: 2.70px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 153/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 37500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0983, Volume: 0.0004, Pixel Error: 2.38px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 154/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0980, Volume: 0.0005, Pixel Error: 2.74px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 155/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 38000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_38000.pth [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1062, Volume: 0.0006, Pixel Error: 2.76px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 156/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0948, Volume: 0.0004, Pixel Error: 2.26px, Height Error: 0.000mm, Gripper: 0.0049 ✓ Saved best model (val_loss=0.0948) ============================================================ Epoch 157/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 38500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1058, Volume: 0.0004, Pixel Error: 2.52px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 158/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0012 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1224, Volume: 0.0005, Pixel Error: 3.16px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 159/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 39000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_39000.pth Train Loss: 0.0012 (Volume: 0.0006, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0964, Volume: 0.0003, Pixel Error: 2.18px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 160/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1120, Volume: 0.0005, Pixel Error: 2.90px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 161/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 39500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.0980, Volume: 0.0004, Pixel Error: 2.66px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 162/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0962, Volume: 0.0004, Pixel Error: 2.42px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 163/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 40000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_40000.pth Train Loss: 0.0008 (Volume: 0.0004, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1032, Volume: 0.0004, Pixel Error: 2.11px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 164/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0015 (Volume: 0.0008, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.1097, Volume: 0.0009, Pixel Error: 3.44px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 165/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 40500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0993, Volume: 0.0005, Pixel Error: 2.59px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 166/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1078, Volume: 0.0004, Pixel Error: 2.65px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 167/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1378, Volume: 0.0004, Pixel Error: 2.84px, Height Error: 0.000mm, Gripper: 0.0245 ============================================================ Epoch 168/9999 ============================================================ [async eval] launched at step 41000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_41000.pth [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1086, Volume: 0.0005, Pixel Error: 2.69px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 169/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1009, Volume: 0.0005, Pixel Error: 3.18px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 170/9999 ============================================================ [async eval] launched at step 41500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1014, Volume: 0.0004, Pixel Error: 2.51px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 171/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1385, Volume: 0.0005, Pixel Error: 3.24px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 172/9999 ============================================================ [async eval] launched at step 42000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_42000.pth [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0964, Volume: 0.0004, Pixel Error: 2.77px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 173/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0962, Volume: 0.0004, Pixel Error: 2.33px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 174/9999 ============================================================ [async eval] launched at step 42500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1080, Volume: 0.0004, Pixel Error: 2.75px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 175/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1038, Volume: 0.0005, Pixel Error: 2.59px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 176/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 43000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_43000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0989, Volume: 0.0003, Pixel Error: 2.37px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 177/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0969, Volume: 0.0005, Pixel Error: 2.47px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 178/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 43500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0007, Rotation: 0.0000) Val - Loss: 0.0967, Volume: 0.0005, Pixel Error: 2.71px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 179/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1118, Volume: 0.0004, Pixel Error: 2.30px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 180/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 44000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_44000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0012 (Volume: 0.0006, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0959, Volume: 0.0005, Pixel Error: 2.81px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 181/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0004, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0986, Volume: 0.0004, Pixel Error: 2.61px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 182/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 44500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1006, Volume: 0.0003, Pixel Error: 2.73px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 183/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1018, Volume: 0.0003, Pixel Error: 2.25px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 184/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 45000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_45000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0008 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0961, Volume: 0.0003, Pixel Error: 2.21px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 185/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0992, Volume: 0.0006, Pixel Error: 2.37px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 186/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 45500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0961, Volume: 0.0004, Pixel Error: 2.26px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 187/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1262, Volume: 0.0004, Pixel Error: 3.06px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 188/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 46000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_46000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0008, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0984, Volume: 0.0004, Pixel Error: 3.21px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 189/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0930, Volume: 0.0003, Pixel Error: 2.49px, Height Error: 0.000mm, Gripper: 0.0000 ✓ Saved best model (val_loss=0.0930) ============================================================ Epoch 190/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 46500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0992, Volume: 0.0005, Pixel Error: 2.21px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 191/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0925, Volume: 0.0003, Pixel Error: 2.31px, Height Error: 0.000mm, Gripper: 0.0049 ✓ Saved best model (val_loss=0.0925) ============================================================ Epoch 192/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 47000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_47000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.1076, Volume: 0.0004, Pixel Error: 2.20px, Height Error: 0.000mm, Gripper: 0.0147 ============================================================ Epoch 193/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0940, Volume: 0.0003, Pixel Error: 2.12px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 194/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 47500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0005, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.1015, Volume: 0.0003, Pixel Error: 2.27px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 195/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0971, Volume: 0.0005, Pixel Error: 2.38px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 196/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 48000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_48000.pth [async eval] logged video to wandb: ep000_success.mp4 Train Loss: 0.0008 (Volume: 0.0005, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0970, Volume: 0.0003, Pixel Error: 2.30px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 197/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0990, Volume: 0.0003, Pixel Error: 2.12px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 198/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 48500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0006, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1001, Volume: 0.0006, Pixel Error: 2.91px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 199/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1431, Volume: 0.0012, Pixel Error: 4.36px, Height Error: 0.000mm, Gripper: 0.0294 ============================================================ Epoch 200/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 49000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_49000.pth Train Loss: 0.0014 (Volume: 0.0008, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0962, Volume: 0.0005, Pixel Error: 2.45px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 201/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1000, Volume: 0.0004, Pixel Error: 2.19px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 202/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 49500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.1080, Volume: 0.0003, Pixel Error: 2.17px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 203/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0939, Volume: 0.0004, Pixel Error: 2.58px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 204/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 50000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_50000.pth Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.1002, Volume: 0.0003, Pixel Error: 1.96px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 205/9999 ============================================================ [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0007, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0988, Volume: 0.0004, Pixel Error: 2.61px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 206/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 50500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1058, Volume: 0.0005, Pixel Error: 2.48px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 207/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0004, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0951, Volume: 0.0004, Pixel Error: 2.21px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 208/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0937, Volume: 0.0004, Pixel Error: 2.36px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 209/9999 ============================================================ [async eval] launched at step 51000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_51000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0984, Volume: 0.0005, Pixel Error: 2.48px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 210/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0958, Volume: 0.0004, Pixel Error: 2.25px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 211/9999 ============================================================ [async eval] launched at step 51500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0966, Volume: 0.0003, Pixel Error: 2.31px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 212/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0005, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0968, Volume: 0.0005, Pixel Error: 3.19px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 213/9999 ============================================================ [async eval] launched at step 52000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_52000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0984, Volume: 0.0004, Pixel Error: 2.82px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 214/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0938, Volume: 0.0003, Pixel Error: 1.93px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 215/9999 ============================================================ [async eval] launched at step 52500 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0970, Volume: 0.0003, Pixel Error: 2.04px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 216/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0008 (Volume: 0.0005, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0956, Volume: 0.0005, Pixel Error: 2.42px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 217/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 53000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_53000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0014 (Volume: 0.0008, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1828, Volume: 0.0005, Pixel Error: 3.24px, Height Error: 0.000mm, Gripper: 0.0343 ============================================================ Epoch 218/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0953, Volume: 0.0008, Pixel Error: 2.84px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 219/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 53500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.1406, Volume: 0.0006, Pixel Error: 2.68px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 220/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0976, Volume: 0.0004, Pixel Error: 2.28px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 221/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 54000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_54000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0010 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1348, Volume: 0.0005, Pixel Error: 2.47px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 222/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.1153, Volume: 0.0004, Pixel Error: 2.38px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 223/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 54500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0938, Volume: 0.0003, Pixel Error: 2.11px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 224/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0943, Volume: 0.0003, Pixel Error: 1.82px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 225/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 55000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_55000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0931, Volume: 0.0003, Pixel Error: 1.93px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 226/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0946, Volume: 0.0004, Pixel Error: 2.18px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 227/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 55500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0914, Volume: 0.0003, Pixel Error: 2.02px, Height Error: 0.000mm, Gripper: 0.0000 ✓ Saved best model (val_loss=0.0914) ============================================================ Epoch 228/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0943, Volume: 0.0005, Pixel Error: 2.23px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 229/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 56000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_56000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0006 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.1015, Volume: 0.0004, Pixel Error: 1.97px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 230/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.1007, Volume: 0.0004, Pixel Error: 2.45px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 231/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 56500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.mp4 Train Loss: 0.0008 (Volume: 0.0004, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0996, Volume: 0.0004, Pixel Error: 2.23px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 232/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0986, Volume: 0.0007, Pixel Error: 3.76px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 233/9999 ============================================================ [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 57000 [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_57000.pth [async eval] logged video to wandb: ep000_success.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0014 (Volume: 0.0008, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0965, Volume: 0.0004, Pixel Error: 2.43px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 234/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0955, Volume: 0.0004, Pixel Error: 2.20px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 235/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 57500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0010 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.0938, Volume: 0.0004, Pixel Error: 2.53px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 236/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0975, Volume: 0.0010, Pixel Error: 4.65px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 237/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 58000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_58000.pth [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0013 (Volume: 0.0007, Gripper: 0.0006, Rotation: 0.0000) Val - Loss: 0.0951, Volume: 0.0006, Pixel Error: 2.29px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 238/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0009 (Volume: 0.0005, Gripper: 0.0004, Rotation: 0.0000) Val - Loss: 0.0928, Volume: 0.0003, Pixel Error: 2.01px, Height Error: 0.000mm, Gripper: 0.0000 ============================================================ Epoch 239/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 58500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0946, Volume: 0.0004, Pixel Error: 2.01px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 240/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 Train Loss: 0.0006 (Volume: 0.0003, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.1171, Volume: 0.0005, Pixel Error: 3.09px, Height Error: 0.000mm, Gripper: 0.0196 ============================================================ Epoch 241/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 59000 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Saved step checkpoint: step_59000.pth Train Loss: 0.0011 (Volume: 0.0006, Gripper: 0.0005, Rotation: 0.0000) Val - Loss: 0.1005, Volume: 0.0004, Pixel Error: 2.45px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 242/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.0969, Volume: 0.0003, Pixel Error: 1.98px, Height Error: 0.000mm, Gripper: 0.0098 ============================================================ Epoch 243/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] launched at step 59500 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0003, Rotation: 0.0000) Val - Loss: 0.1012, Volume: 0.0004, Pixel Error: 2.19px, Height Error: 0.000mm, Gripper: 0.0049 ============================================================ Epoch 244/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.mp4 [async eval] logged video to wandb: ep000_fail.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.mp4 Train Loss: 0.0007 (Volume: 0.0004, Gripper: 0.0002, Rotation: 0.0000) Val - Loss: 0.0905, Volume: 0.0003, Pixel Error: 2.16px, Height Error: 0.000mm, Gripper: 0.0000 ✓ Saved best model (val_loss=0.0905) ============================================================ Epoch 245/9999 ============================================================ [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.mp4 [async eval] logged video to wandb: ep000_fail.h264.h264.h264.h264.h264.mp4