/data2/cameron/miniconda3/envs/uva/lib/python3.10/site-packages/sympy/external/gmpy.py:139: UserWarning: gmpy2 version is too old to use (2.0.0 or newer required)
  gmpy = import_module('gmpy2', min_module_version=_GMPY2_MIN_VERSION,
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: cameronsmithbusiness. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.18.3
wandb: Run data is saved locally in /data/cameron/scratch_files/hand_vids/wandb/run-20260418_104947-w6m9ncp8
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run hand_wrist_tracker
wandb: ⭐️ View project at https://wandb.ai/cameronsmithbusiness/para_hand_tracking
wandb: 🚀 View run at https://wandb.ai/cameronsmithbusiness/para_hand_tracking/runs/w6m9ncp8
HandKeypointDataset: 264 samples from 2 videos
HandKeypointDataset: 4 samples from 1 videos
Loading DINOv2 model...
✓ DINOv2 backbone is trainable
✓ DINO embedding dim: 384
✓ Learnable start keypoint embedding (dim=384)
✓ Feature convs: 3× Conv2d(3×3) at pred_size=64
✓ Volume   head → (B, 4, 1, 64, 64)
✓ Gripper  head → (B, 4, 2, 64, 64)  [1×1 conv, CE 2-class]
✓ Rotation head → (B, 4, 3, 32, 64, 64)  [1×1 conv, CE]
Model params: 32,832,780
Epoch 1/999: train_loss=6.7884 train_px=86.2px | test_loss=6.3503 test_px=70.1px
  -> Saved best (test_loss=6.3503, test_px=70.1px)
Epoch 2/999: train_loss=6.2469 train_px=77.7px | test_loss=6.2121 test_px=71.6px
  -> Saved best (test_loss=6.2121, test_px=71.6px)
Epoch 3/999: train_loss=6.0584 train_px=78.0px | test_loss=6.4053 test_px=71.4px
Epoch 4/999: train_loss=5.7308 train_px=74.2px | test_loss=6.2765 test_px=76.9px
Epoch 5/999: train_loss=5.4083 train_px=72.9px | test_loss=6.8261 test_px=68.4px
Epoch 6/999: train_loss=5.1078 train_px=74.6px | test_loss=6.8070 test_px=90.3px
Epoch 7/999: train_loss=4.6975 train_px=72.7px | test_loss=7.5561 test_px=85.3px
Epoch 8/999: train_loss=4.3307 train_px=66.2px | test_loss=8.2769 test_px=95.9px
Epoch 9/999: train_loss=3.8633 train_px=61.4px | test_loss=9.0564 test_px=71.5px
Epoch 10/999: train_loss=3.3320 train_px=53.7px | test_loss=10.3153 test_px=97.4px
Epoch 11/999: train_loss=3.0263 train_px=47.6px | test_loss=10.5408 test_px=68.5px
Epoch 12/999: train_loss=2.6464 train_px=41.0px | test_loss=9.0965 test_px=90.9px
Epoch 13/999: train_loss=2.3081 train_px=33.0px | test_loss=10.9003 test_px=84.2px
Epoch 14/999: train_loss=2.0309 train_px=26.8px | test_loss=10.6461 test_px=81.5px
Epoch 15/999: train_loss=1.8902 train_px=27.1px | test_loss=11.0242 test_px=97.0px
Epoch 16/999: train_loss=1.5986 train_px=20.9px | test_loss=11.2355 test_px=90.0px
Epoch 17/999: train_loss=1.4856 train_px=18.3px | test_loss=12.1444 test_px=87.3px
Epoch 18/999: train_loss=1.3468 train_px=16.5px | test_loss=11.1057 test_px=98.9px
Epoch 19/999: train_loss=1.2330 train_px=16.4px | test_loss=11.4065 test_px=78.7px
Epoch 20/999: train_loss=1.1087 train_px=12.2px | test_loss=12.1427 test_px=105.1px
Epoch 21/999: train_loss=1.0505 train_px=11.1px | test_loss=12.2545 test_px=62.4px
Epoch 22/999: train_loss=1.1071 train_px=12.1px | test_loss=11.2457 test_px=92.2px
Epoch 23/999: train_loss=0.8737 train_px=12.1px | test_loss=14.5934 test_px=77.5px
Epoch 24/999: train_loss=0.9426 train_px=9.3px | test_loss=11.9153 test_px=73.4px
Epoch 25/999: train_loss=0.7386 train_px=9.0px | test_loss=12.3639 test_px=84.6px
Epoch 26/999: train_loss=0.8164 train_px=10.4px | test_loss=13.5773 test_px=87.0px
Epoch 27/999: train_loss=0.7837 train_px=9.7px | test_loss=12.6988 test_px=82.7px
Epoch 28/999: train_loss=0.6958 train_px=9.1px | test_loss=10.9619 test_px=77.2px
Epoch 29/999: train_loss=0.6384 train_px=8.8px | test_loss=13.4189 test_px=73.8px
Epoch 30/999: train_loss=0.6322 train_px=7.4px | test_loss=11.3877 test_px=94.1px
Epoch 31/999: train_loss=0.5946 train_px=8.6px | test_loss=14.2104 test_px=87.4px
Epoch 32/999: train_loss=0.5625 train_px=7.5px | test_loss=13.7682 test_px=73.6px
Epoch 33/999: train_loss=0.6194 train_px=8.1px | test_loss=11.8669 test_px=73.8px
Epoch 34/999: train_loss=0.5261 train_px=9.2px | test_loss=11.7334 test_px=90.2px
Epoch 35/999: train_loss=0.4869 train_px=8.3px | test_loss=11.3042 test_px=87.2px
Epoch 36/999: train_loss=0.5475 train_px=7.4px | test_loss=11.5026 test_px=64.4px
Epoch 37/999: train_loss=0.4807 train_px=7.4px | test_loss=11.0874 test_px=84.1px
Epoch 38/999: train_loss=0.4968 train_px=8.3px | test_loss=13.6664 test_px=89.6px
Epoch 39/999: train_loss=0.4608 train_px=7.4px | test_loss=12.6604 test_px=92.8px
Epoch 40/999: train_loss=0.4755 train_px=8.0px | test_loss=13.1866 test_px=87.1px
Epoch 41/999: train_loss=0.4459 train_px=9.2px | test_loss=12.5666 test_px=80.9px
Epoch 42/999: train_loss=0.3758 train_px=7.0px | test_loss=13.2033 test_px=68.7px
Epoch 43/999: train_loss=0.4412 train_px=7.0px | test_loss=12.0786 test_px=88.4px
Epoch 44/999: train_loss=0.4122 train_px=7.5px | test_loss=11.3260 test_px=83.7px
Epoch 45/999: train_loss=0.3579 train_px=6.9px | test_loss=15.0025 test_px=91.4px
Epoch 46/999: train_loss=0.3411 train_px=6.7px | test_loss=14.7236 test_px=89.8px
Epoch 47/999: train_loss=0.3402 train_px=6.8px | test_loss=12.8556 test_px=84.2px
Epoch 48/999: train_loss=0.3349 train_px=8.1px | test_loss=11.5525 test_px=76.9px
Epoch 49/999: train_loss=0.3606 train_px=7.3px | test_loss=13.4160 test_px=93.8px
Epoch 50/999: train_loss=0.3387 train_px=7.4px | test_loss=12.9841 test_px=81.7px
Epoch 51/999: train_loss=0.3489 train_px=7.3px | test_loss=12.9204 test_px=84.7px
Epoch 52/999: train_loss=0.3848 train_px=7.1px | test_loss=10.1612 test_px=88.2px
Epoch 53/999: train_loss=0.2490 train_px=6.7px | test_loss=12.8648 test_px=66.1px
Epoch 54/999: train_loss=0.3403 train_px=6.2px | test_loss=12.9649 test_px=91.5px
Epoch 55/999: train_loss=0.3178 train_px=6.8px | test_loss=10.9146 test_px=91.1px
Epoch 56/999: train_loss=0.2862 train_px=7.3px | test_loss=14.1848 test_px=75.4px
Epoch 57/999: train_loss=0.2457 train_px=6.7px | test_loss=12.3579 test_px=95.0px
Epoch 58/999: train_loss=0.2743 train_px=6.2px | test_loss=14.3928 test_px=76.4px
Epoch 59/999: train_loss=0.3618 train_px=6.6px | test_loss=12.1864 test_px=89.5px
Epoch 60/999: train_loss=0.3148 train_px=6.6px | test_loss=11.8457 test_px=79.8px
Epoch 61/999: train_loss=0.2881 train_px=5.9px | test_loss=12.0129 test_px=87.8px
Epoch 62/999: train_loss=0.3219 train_px=6.4px | test_loss=13.1921 test_px=90.0px
Epoch 63/999: train_loss=0.3179 train_px=7.1px | test_loss=17.5416 test_px=91.1px
Epoch 64/999: train_loss=0.2508 train_px=6.4px | test_loss=15.6465 test_px=85.1px
Epoch 65/999: train_loss=0.3100 train_px=7.4px | test_loss=14.7359 test_px=101.6px
Epoch 66/999: train_loss=0.2318 train_px=6.5px | test_loss=16.1739 test_px=86.4px
Epoch 67/999: train_loss=0.2247 train_px=6.2px | test_loss=14.1891 test_px=86.8px
Epoch 68/999: train_loss=0.3504 train_px=7.5px | test_loss=13.3411 test_px=84.6px
Epoch 69/999: train_loss=0.3442 train_px=7.9px | test_loss=13.3203 test_px=101.4px
Epoch 70/999: train_loss=0.2376 train_px=6.9px | test_loss=9.9739 test_px=95.0px
Epoch 71/999: train_loss=0.1525 train_px=6.4px | test_loss=17.5290 test_px=86.0px
Epoch 72/999: train_loss=0.2566 train_px=6.6px | test_loss=10.7832 test_px=104.2px
Epoch 73/999: train_loss=0.2137 train_px=6.5px | test_loss=14.3564 test_px=93.5px
Epoch 74/999: train_loss=0.2741 train_px=7.0px | test_loss=15.4639 test_px=107.8px
Epoch 75/999: train_loss=0.1845 train_px=6.7px | test_loss=18.7238 test_px=93.5px
Epoch 76/999: train_loss=0.2932 train_px=7.3px | test_loss=15.9373 test_px=84.2px
Epoch 77/999: train_loss=0.2022 train_px=6.6px | test_loss=19.4237 test_px=80.5px
Epoch 78/999: train_loss=0.2606 train_px=7.0px | test_loss=13.7844 test_px=110.0px
Epoch 79/999: train_loss=0.2111 train_px=6.8px | test_loss=14.9847 test_px=92.7px
Epoch 80/999: train_loss=0.1635 train_px=6.6px | test_loss=15.2411 test_px=91.8px
Epoch 81/999: train_loss=0.1833 train_px=6.1px | test_loss=16.5557 test_px=103.0px
Epoch 82/999: train_loss=0.1701 train_px=6.2px | test_loss=15.7083 test_px=94.8px
Epoch 83/999: train_loss=0.2644 train_px=6.5px | test_loss=13.8315 test_px=102.3px
Epoch 84/999: train_loss=0.1897 train_px=6.3px | test_loss=14.7150 test_px=82.4px
Epoch 85/999: train_loss=0.1641 train_px=6.2px | test_loss=13.7845 test_px=73.9px
Epoch 86/999: train_loss=0.1036 train_px=5.4px | test_loss=15.0742 test_px=82.2px
Epoch 87/999: train_loss=0.2144 train_px=7.1px | test_loss=14.0012 test_px=102.4px
Epoch 88/999: train_loss=0.1977 train_px=6.9px | test_loss=16.4546 test_px=92.3px
Epoch 89/999: train_loss=0.2579 train_px=7.1px | test_loss=14.0574 test_px=87.0px
Epoch 90/999: train_loss=0.1780 train_px=6.5px | test_loss=14.8892 test_px=96.5px
Epoch 91/999: train_loss=0.1400 train_px=6.1px | test_loss=13.8420 test_px=81.0px
Epoch 92/999: train_loss=0.1726 train_px=6.0px | test_loss=13.3813 test_px=96.3px
Epoch 93/999: train_loss=0.2252 train_px=6.4px | test_loss=15.6321 test_px=81.9px
Epoch 94/999: train_loss=0.1254 train_px=6.1px | test_loss=14.2880 test_px=79.5px
Epoch 95/999: train_loss=0.2003 train_px=6.3px | test_loss=14.8188 test_px=120.5px
Epoch 96/999: train_loss=0.0924 train_px=5.8px | test_loss=16.0550 test_px=84.4px
Epoch 97/999: train_loss=0.1426 train_px=5.7px | test_loss=15.6877 test_px=99.2px
Epoch 98/999: train_loss=0.1694 train_px=6.2px | test_loss=16.0775 test_px=98.0px
Epoch 99/999: train_loss=0.2674 train_px=6.0px | test_loss=11.6570 test_px=134.7px
Epoch 100/999: train_loss=0.1928 train_px=6.5px | test_loss=14.3310 test_px=93.3px
Epoch 101/999: train_loss=0.1026 train_px=5.8px | test_loss=14.2382 test_px=71.5px
Epoch 102/999: train_loss=0.1444 train_px=6.5px | test_loss=14.5491 test_px=80.8px
Epoch 103/999: train_loss=0.1748 train_px=6.3px | test_loss=15.4621 test_px=105.2px
Epoch 104/999: train_loss=0.1715 train_px=6.2px | test_loss=16.6211 test_px=110.6px
Epoch 105/999: train_loss=0.1171 train_px=5.7px | test_loss=18.2899 test_px=126.8px
Epoch 106/999: train_loss=0.1010 train_px=5.9px | test_loss=17.7213 test_px=110.0px
Epoch 107/999: train_loss=0.1700 train_px=6.1px | test_loss=14.3741 test_px=90.0px
Epoch 108/999: train_loss=0.1588 train_px=5.6px | test_loss=18.5305 test_px=97.1px
Epoch 109/999: train_loss=0.1822 train_px=6.7px | test_loss=21.1828 test_px=124.3px
Epoch 110/999: train_loss=0.1733 train_px=6.0px | test_loss=16.4841 test_px=102.5px
Epoch 111/999: train_loss=0.1141 train_px=6.3px | test_loss=16.3488 test_px=112.0px
Epoch 112/999: train_loss=0.1696 train_px=6.0px | test_loss=15.8242 test_px=86.0px
Epoch 113/999: train_loss=0.1426 train_px=5.9px | test_loss=17.3176 test_px=80.9px
Epoch 114/999: train_loss=0.1596 train_px=6.4px | test_loss=17.1927 test_px=95.9px
Epoch 115/999: train_loss=0.1570 train_px=6.1px | test_loss=15.0444 test_px=76.9px
Epoch 116/999: train_loss=0.1140 train_px=5.8px | test_loss=19.4128 test_px=89.9px
Epoch 117/999: train_loss=0.1439 train_px=5.5px | test_loss=17.5493 test_px=92.2px
Epoch 118/999: train_loss=0.1935 train_px=5.9px | test_loss=13.4806 test_px=94.6px
Epoch 119/999: train_loss=0.2079 train_px=6.6px | test_loss=14.3291 test_px=111.2px
Epoch 120/999: train_loss=0.1660 train_px=6.8px | test_loss=12.9856 test_px=93.9px
Epoch 121/999: train_loss=0.1382 train_px=6.5px | test_loss=14.5261 test_px=99.5px
Epoch 122/999: train_loss=0.1261 train_px=6.2px | test_loss=15.9079 test_px=104.7px
Epoch 123/999: train_loss=0.0862 train_px=6.1px | test_loss=13.4783 test_px=118.5px
Epoch 124/999: train_loss=0.0718 train_px=6.0px | test_loss=13.2908 test_px=109.0px
Epoch 125/999: train_loss=0.1199 train_px=6.2px | test_loss=14.1485 test_px=116.9px
Epoch 126/999: train_loss=0.0628 train_px=6.3px | test_loss=16.1234 test_px=114.2px
Epoch 127/999: train_loss=0.1138 train_px=5.6px | test_loss=17.6378 test_px=110.3px
Epoch 128/999: train_loss=0.1168 train_px=5.6px | test_loss=18.7202 test_px=93.8px
Epoch 129/999: train_loss=0.1463 train_px=6.2px | test_loss=18.7086 test_px=107.2px
Epoch 130/999: train_loss=0.1336 train_px=5.7px | test_loss=15.6106 test_px=79.4px
Epoch 131/999: train_loss=0.0625 train_px=5.9px | test_loss=17.5315 test_px=98.6px
Epoch 132/999: train_loss=0.1105 train_px=6.1px | test_loss=19.0694 test_px=99.2px
Epoch 133/999: train_loss=0.0806 train_px=5.7px | test_loss=16.7603 test_px=91.3px
Epoch 134/999: train_loss=0.0764 train_px=5.7px | test_loss=17.6887 test_px=89.6px
Epoch 135/999: train_loss=0.1686 train_px=5.6px | test_loss=16.1502 test_px=98.2px
Epoch 136/999: train_loss=0.1491 train_px=5.8px | test_loss=17.4260 test_px=118.4px
Epoch 137/999: train_loss=0.1987 train_px=6.5px | test_loss=16.0105 test_px=106.4px
Epoch 138/999: train_loss=0.1393 train_px=6.3px | test_loss=14.2685 test_px=103.8px
Epoch 139/999: train_loss=0.1242 train_px=6.0px | test_loss=18.2357 test_px=108.4px
Epoch 140/999: train_loss=0.1309 train_px=6.4px | test_loss=18.3707 test_px=87.9px
Epoch 141/999: train_loss=0.0717 train_px=5.7px | test_loss=20.9040 test_px=81.7px
Epoch 142/999: train_loss=0.1976 train_px=5.7px | test_loss=20.0986 test_px=92.9px
Epoch 143/999: train_loss=0.1119 train_px=5.8px | test_loss=15.6399 test_px=76.3px
Epoch 144/999: train_loss=0.0656 train_px=5.5px | test_loss=19.8604 test_px=84.0px
Epoch 145/999: train_loss=0.1160 train_px=5.8px | test_loss=19.1603 test_px=104.8px
Epoch 146/999: train_loss=0.1825 train_px=5.7px | test_loss=16.5732 test_px=112.7px
Epoch 147/999: train_loss=0.1167 train_px=6.6px | test_loss=20.0054 test_px=116.6px
Epoch 148/999: train_loss=0.1085 train_px=5.8px | test_loss=16.7502 test_px=125.2px
Epoch 149/999: train_loss=0.1440 train_px=6.3px | test_loss=17.3821 test_px=115.8px
Epoch 150/999: train_loss=0.0780 train_px=5.9px | test_loss=17.4180 test_px=87.5px
Epoch 151/999: train_loss=0.1562 train_px=6.2px | test_loss=13.1427 test_px=126.9px
Epoch 152/999: train_loss=0.1055 train_px=6.2px | test_loss=15.1571 test_px=95.2px
Epoch 153/999: train_loss=0.0778 train_px=5.7px | test_loss=15.4410 test_px=90.8px
Epoch 154/999: train_loss=0.0786 train_px=5.6px | test_loss=16.1669 test_px=95.4px
Epoch 155/999: train_loss=0.0823 train_px=6.0px | test_loss=14.3959 test_px=83.3px
Epoch 156/999: train_loss=0.0875 train_px=6.0px | test_loss=15.0977 test_px=100.6px
Epoch 157/999: train_loss=0.1796 train_px=6.5px | test_loss=18.1307 test_px=106.2px
Epoch 158/999: train_loss=0.0710 train_px=6.0px | test_loss=14.1747 test_px=99.0px
Epoch 159/999: train_loss=0.0690 train_px=5.8px | test_loss=16.2486 test_px=90.0px
Epoch 160/999: train_loss=0.0786 train_px=5.7px | test_loss=13.7304 test_px=101.4px
Epoch 161/999: train_loss=0.0681 train_px=5.6px | test_loss=14.8482 test_px=95.3px
Epoch 162/999: train_loss=0.0542 train_px=5.6px | test_loss=14.4630 test_px=100.2px
Epoch 163/999: train_loss=0.1398 train_px=6.0px | test_loss=14.7136 test_px=96.3px
Epoch 164/999: train_loss=0.0843 train_px=5.9px | test_loss=19.6456 test_px=84.1px
Epoch 165/999: train_loss=0.1087 train_px=6.3px | test_loss=17.1136 test_px=82.0px
Epoch 166/999: train_loss=0.1427 train_px=6.0px | test_loss=16.1779 test_px=84.7px
Epoch 167/999: train_loss=0.0986 train_px=6.3px | test_loss=18.5724 test_px=76.5px
Epoch 168/999: train_loss=0.1296 train_px=5.6px | test_loss=16.7215 test_px=87.7px
Epoch 169/999: train_loss=0.1427 train_px=5.7px | test_loss=15.1844 test_px=102.7px
Epoch 170/999: train_loss=0.1016 train_px=6.1px | test_loss=16.8996 test_px=80.0px
Epoch 171/999: train_loss=0.1228 train_px=6.0px | test_loss=12.6510 test_px=64.6px
Epoch 172/999: train_loss=0.1211 train_px=5.6px | test_loss=16.7532 test_px=116.6px
Epoch 173/999: train_loss=0.0728 train_px=6.0px | test_loss=18.5888 test_px=88.8px
Epoch 174/999: train_loss=0.0470 train_px=5.7px | test_loss=13.0759 test_px=107.7px
Epoch 175/999: train_loss=0.1234 train_px=5.9px | test_loss=18.7926 test_px=100.4px
Epoch 176/999: train_loss=0.0853 train_px=6.1px | test_loss=13.8793 test_px=80.3px
Epoch 177/999: train_loss=0.1253 train_px=6.1px | test_loss=13.8501 test_px=83.4px
Epoch 178/999: train_loss=0.1567 train_px=6.2px | test_loss=13.0529 test_px=102.7px
Epoch 179/999: train_loss=0.1097 train_px=6.1px | test_loss=16.3000 test_px=88.8px
Epoch 180/999: train_loss=0.0668 train_px=5.8px | test_loss=14.1902 test_px=108.6px
Epoch 181/999: train_loss=0.0897 train_px=5.8px | test_loss=13.9588 test_px=102.9px
Epoch 182/999: train_loss=0.1164 train_px=5.9px | test_loss=12.9191 test_px=91.6px
Epoch 183/999: train_loss=0.1213 train_px=5.9px | test_loss=12.0318 test_px=99.0px
Epoch 184/999: train_loss=0.0790 train_px=5.6px | test_loss=12.1267 test_px=89.5px
Epoch 185/999: train_loss=0.0701 train_px=5.8px | test_loss=10.3919 test_px=106.2px
Epoch 186/999: train_loss=0.0951 train_px=5.9px | test_loss=12.8004 test_px=81.4px
Epoch 187/999: train_loss=0.0583 train_px=5.7px | test_loss=14.6550 test_px=106.7px
Epoch 188/999: train_loss=0.1130 train_px=5.7px | test_loss=16.3907 test_px=100.1px
Epoch 189/999: train_loss=0.0939 train_px=5.7px | test_loss=12.7831 test_px=87.3px
Epoch 190/999: train_loss=0.0682 train_px=5.5px | test_loss=13.8465 test_px=102.0px
Epoch 191/999: train_loss=0.1906 train_px=6.1px | test_loss=12.3461 test_px=80.2px
Epoch 192/999: train_loss=0.0947 train_px=6.0px | test_loss=16.2193 test_px=97.4px
Epoch 193/999: train_loss=0.0850 train_px=6.0px | test_loss=16.0435 test_px=85.7px
Epoch 194/999: train_loss=0.0776 train_px=5.3px | test_loss=16.2741 test_px=106.4px
Epoch 195/999: train_loss=0.0648 train_px=5.7px | test_loss=12.8417 test_px=84.8px
Epoch 196/999: train_loss=0.0800 train_px=5.5px | test_loss=16.8100 test_px=94.3px
Epoch 197/999: train_loss=0.0903 train_px=5.8px | test_loss=17.0422 test_px=101.2px
Epoch 198/999: train_loss=0.0593 train_px=5.7px | test_loss=15.7212 test_px=109.4px
Epoch 199/999: train_loss=0.0737 train_px=5.9px | test_loss=14.4009 test_px=94.5px
Epoch 200/999: train_loss=0.0534 train_px=5.8px | test_loss=15.9399 test_px=83.3px
Epoch 201/999: train_loss=0.1050 train_px=5.9px | test_loss=13.4944 test_px=82.1px
Epoch 202/999: train_loss=0.1743 train_px=5.7px | test_loss=11.4775 test_px=88.5px
Epoch 203/999: train_loss=0.0740 train_px=5.7px | test_loss=12.3229 test_px=106.9px
Epoch 204/999: train_loss=0.0984 train_px=6.0px | test_loss=12.8955 test_px=108.3px
Epoch 205/999: train_loss=0.1041 train_px=5.9px | test_loss=14.5177 test_px=98.6px
Epoch 206/999: train_loss=0.0761 train_px=5.6px | test_loss=12.3026 test_px=116.3px
Epoch 207/999: train_loss=0.0767 train_px=5.8px | test_loss=15.1324 test_px=99.5px
Epoch 208/999: train_loss=0.0502 train_px=5.8px | test_loss=16.9896 test_px=79.1px
Epoch 209/999: train_loss=0.0427 train_px=6.0px | test_loss=14.0754 test_px=94.8px
Epoch 210/999: train_loss=0.1022 train_px=5.5px | test_loss=13.6236 test_px=96.5px
Epoch 211/999: train_loss=0.0671 train_px=6.5px | test_loss=15.6266 test_px=84.3px
Epoch 212/999: train_loss=0.0707 train_px=6.0px | test_loss=19.0073 test_px=103.7px
Epoch 213/999: train_loss=0.0606 train_px=5.8px | test_loss=17.4720 test_px=110.4px
Epoch 214/999: train_loss=0.0677 train_px=6.2px | test_loss=13.0895 test_px=94.4px
Epoch 215/999: train_loss=0.0972 train_px=5.6px | test_loss=17.0242 test_px=93.6px
Epoch 216/999: train_loss=0.0973 train_px=6.1px | test_loss=16.9947 test_px=98.9px
Epoch 217/999: train_loss=0.0762 train_px=5.8px | test_loss=16.4990 test_px=109.2px
Epoch 218/999: train_loss=0.0702 train_px=5.7px | test_loss=15.7040 test_px=100.3px
Epoch 219/999: train_loss=0.0476 train_px=5.5px | test_loss=14.8273 test_px=123.3px
Epoch 220/999: train_loss=0.1104 train_px=5.8px | test_loss=15.7272 test_px=102.0px
Epoch 221/999: train_loss=0.1105 train_px=6.0px | test_loss=11.2678 test_px=95.0px
Epoch 222/999: train_loss=0.0600 train_px=5.6px | test_loss=15.8576 test_px=114.8px
Epoch 223/999: train_loss=0.0225 train_px=5.4px | test_loss=15.0556 test_px=93.4px
Epoch 224/999: train_loss=0.0116 train_px=5.5px | test_loss=16.5242 test_px=120.6px
Epoch 225/999: train_loss=0.0544 train_px=5.6px | test_loss=14.8031 test_px=75.0px
Epoch 226/999: train_loss=0.0741 train_px=5.8px | test_loss=17.1016 test_px=101.4px
Epoch 227/999: train_loss=0.1832 train_px=5.7px | test_loss=12.0560 test_px=94.8px
Epoch 228/999: train_loss=0.0986 train_px=6.0px | test_loss=12.0872 test_px=121.6px
Epoch 229/999: train_loss=0.1084 train_px=5.6px | test_loss=10.7584 test_px=97.4px
Epoch 230/999: train_loss=0.0303 train_px=5.7px | test_loss=15.0559 test_px=109.5px
Epoch 231/999: train_loss=0.1335 train_px=6.5px | test_loss=12.7373 test_px=117.8px
Epoch 232/999: train_loss=0.0620 train_px=5.7px | test_loss=11.4937 test_px=122.6px
Epoch 233/999: train_loss=0.0233 train_px=5.4px | test_loss=13.4566 test_px=105.0px
Epoch 234/999: train_loss=0.0080 train_px=5.4px | test_loss=12.9151 test_px=98.3px
Epoch 235/999: train_loss=0.0614 train_px=5.4px | test_loss=13.3998 test_px=110.1px
Epoch 236/999: train_loss=0.0070 train_px=5.5px | test_loss=12.2138 test_px=111.1px
Epoch 237/999: train_loss=0.0606 train_px=5.7px | test_loss=15.1005 test_px=121.8px
Epoch 238/999: train_loss=0.0530 train_px=5.8px | test_loss=16.7099 test_px=104.4px
Epoch 239/999: train_loss=0.0982 train_px=5.5px | test_loss=16.2852 test_px=113.5px
Epoch 240/999: train_loss=0.0608 train_px=5.4px | test_loss=14.9256 test_px=104.7px
Epoch 241/999: train_loss=0.0680 train_px=5.6px | test_loss=14.1574 test_px=95.8px
Epoch 242/999: train_loss=0.0681 train_px=5.7px | test_loss=14.0144 test_px=110.9px
Epoch 243/999: train_loss=0.0487 train_px=5.9px | test_loss=16.4889 test_px=105.4px
Epoch 244/999: train_loss=0.1081 train_px=5.7px | test_loss=16.9648 test_px=128.1px
Epoch 245/999: train_loss=0.1596 train_px=6.3px | test_loss=12.2359 test_px=111.9px
Epoch 246/999: train_loss=0.1358 train_px=6.2px | test_loss=12.4852 test_px=109.2px
Epoch 247/999: train_loss=0.0873 train_px=6.0px | test_loss=15.2066 test_px=93.7px
Epoch 248/999: train_loss=0.0744 train_px=6.1px | test_loss=15.5077 test_px=104.8px
Epoch 249/999: train_loss=0.1186 train_px=6.2px | test_loss=14.5419 test_px=111.6px
Epoch 250/999: train_loss=0.0714 train_px=5.6px | test_loss=12.6506 test_px=114.9px
Epoch 251/999: train_loss=0.0304 train_px=5.6px | test_loss=14.2460 test_px=81.6px
Epoch 252/999: train_loss=0.0453 train_px=5.4px | test_loss=13.4557 test_px=77.6px
Epoch 253/999: train_loss=0.0136 train_px=5.5px | test_loss=13.2637 test_px=124.3px
Epoch 254/999: train_loss=0.0720 train_px=5.6px | test_loss=11.9801 test_px=115.5px
Epoch 255/999: train_loss=0.0949 train_px=5.7px | test_loss=14.4254 test_px=114.8px
Epoch 256/999: train_loss=0.0748 train_px=5.9px | test_loss=12.4717 test_px=109.7px
Epoch 257/999: train_loss=0.0874 train_px=5.6px | test_loss=14.4767 test_px=98.6px
Epoch 258/999: train_loss=0.0398 train_px=5.5px | test_loss=12.5651 test_px=95.2px
Epoch 259/999: train_loss=0.0358 train_px=5.7px | test_loss=13.0187 test_px=80.5px
Epoch 260/999: train_loss=0.0312 train_px=5.4px | test_loss=17.4428 test_px=91.9px
Epoch 261/999: train_loss=0.0281 train_px=5.5px | test_loss=15.8195 test_px=93.5px
Epoch 262/999: train_loss=0.0464 train_px=5.5px | test_loss=12.0268 test_px=73.5px
Epoch 263/999: train_loss=0.0294 train_px=5.6px | test_loss=12.2606 test_px=76.5px
Epoch 264/999: train_loss=0.0451 train_px=5.6px | test_loss=16.3115 test_px=118.3px
Epoch 265/999: train_loss=0.0284 train_px=5.6px | test_loss=13.9906 test_px=128.0px
Epoch 266/999: train_loss=0.0635 train_px=5.5px | test_loss=15.2522 test_px=94.0px
Epoch 267/999: train_loss=0.0750 train_px=5.9px | test_loss=18.0730 test_px=107.0px
Epoch 268/999: train_loss=0.0564 train_px=5.6px | test_loss=14.7821 test_px=109.1px
Epoch 269/999: train_loss=0.0938 train_px=5.9px | test_loss=12.4347 test_px=105.5px
Epoch 270/999: train_loss=0.1361 train_px=6.1px | test_loss=10.2383 test_px=111.0px
Epoch 271/999: train_loss=0.1052 train_px=5.8px | test_loss=11.2153 test_px=89.2px
Epoch 272/999: train_loss=0.0916 train_px=6.1px | test_loss=13.3041 test_px=100.1px
Epoch 273/999: train_loss=0.1671 train_px=6.4px | test_loss=12.6708 test_px=97.7px
Epoch 274/999: train_loss=0.0631 train_px=5.8px | test_loss=17.8995 test_px=63.0px
Epoch 275/999: train_loss=0.0703 train_px=5.7px | test_loss=14.1854 test_px=87.7px
Epoch 276/999: train_loss=0.0800 train_px=5.6px | test_loss=13.7782 test_px=97.8px
Epoch 277/999: train_loss=0.0925 train_px=5.9px | test_loss=13.4459 test_px=70.7px
Epoch 278/999: train_loss=0.0802 train_px=5.5px | test_loss=11.7008 test_px=80.3px
Epoch 279/999: train_loss=0.0641 train_px=5.6px | test_loss=11.7928 test_px=75.6px
Epoch 280/999: train_loss=0.0127 train_px=5.5px | test_loss=12.4194 test_px=68.5px
Epoch 281/999: train_loss=0.0353 train_px=5.4px | test_loss=12.4925 test_px=91.2px
Epoch 282/999: train_loss=0.0102 train_px=5.4px | test_loss=14.5124 test_px=88.1px
Epoch 283/999: train_loss=0.0351 train_px=5.3px | test_loss=14.9493 test_px=93.9px
Epoch 284/999: train_loss=0.0369 train_px=5.6px | test_loss=16.6347 test_px=94.6px
Epoch 285/999: train_loss=0.0584 train_px=5.7px | test_loss=16.9022 test_px=85.5px
Epoch 286/999: train_loss=0.0552 train_px=5.8px | test_loss=16.8338 test_px=84.8px
Epoch 287/999: train_loss=0.0395 train_px=5.4px | test_loss=13.4157 test_px=89.5px
Epoch 288/999: train_loss=0.0600 train_px=5.6px | test_loss=15.6952 test_px=82.4px
wandb: - 187.465 MB of 187.465 MB uploaded
wandb:                                                                                
wandb: 
wandb: Run history:
wandb:             epoch ▁▁▁▁▂▂▃▃▃▃▄▄▄▄▄▄▄▄▅▅▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇█████
wandb:         test/loss ▁▂▅▄▃▄▄▃▄▄▇▅▄▅▄▆█▄▄▅▆▅▅▄▄▄▄▅▄▃▅▄▆▃▄▆▅▅▄▂
wandb:  test/pixel_error ▃▁▄▃▄▄▄▄▃▃▅▆▅▆▆█▅▃▅▁▄▃▃▄▆▃▆▅▆▄▄▂▄▄▄▄▃▄▆▅
wandb:        train/loss █▆▄▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: train/pixel_error █▆▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: 
wandb: Run summary:
wandb:             epoch 336
wandb:         test/loss 12.42045
wandb:  test/pixel_error 95.29614
wandb:        train/loss 0.06708
wandb: train/pixel_error 5.70054
wandb: 
wandb: 🚀 View run hand_wrist_tracker at: https://wandb.ai/cameronsmithbusiness/para_hand_tracking/runs/w6m9ncp8
wandb: ⭐️ View project at: https://wandb.ai/cameronsmithbusiness/para_hand_tracking
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 272 other file(s)
wandb: Find logs at: ./wandb/run-20260418_104947-w6m9ncp8/logs
Epoch 289/999: train_loss=0.1447 train_px=6.1px | test_loss=12.8320 test_px=76.4px
Epoch 290/999: train_loss=0.0423 train_px=5.9px | test_loss=12.5480 test_px=68.6px
Epoch 291/999: train_loss=0.0745 train_px=5.8px | test_loss=13.5310 test_px=66.7px
Epoch 292/999: train_loss=0.0733 train_px=6.2px | test_loss=13.8925 test_px=88.3px
Epoch 293/999: train_loss=0.0695 train_px=5.7px | test_loss=13.2819 test_px=92.0px
Epoch 294/999: train_loss=0.0479 train_px=5.4px | test_loss=13.8248 test_px=95.6px
Epoch 295/999: train_loss=0.0619 train_px=5.8px | test_loss=12.0612 test_px=98.3px
Epoch 296/999: train_loss=0.0418 train_px=5.5px | test_loss=12.5803 test_px=69.2px
Epoch 297/999: train_loss=0.0419 train_px=5.8px | test_loss=13.9862 test_px=75.5px
Epoch 298/999: train_loss=0.0805 train_px=5.8px | test_loss=15.0915 test_px=100.2px
Epoch 299/999: train_loss=0.0793 train_px=5.8px | test_loss=12.9965 test_px=89.3px
Epoch 300/999: train_loss=0.0255 train_px=5.5px | test_loss=14.4051 test_px=81.2px
Epoch 301/999: train_loss=0.0145 train_px=5.5px | test_loss=17.2681 test_px=82.6px
Epoch 302/999: train_loss=0.0342 train_px=5.6px | test_loss=14.3967 test_px=85.9px
Epoch 303/999: train_loss=0.0253 train_px=5.8px | test_loss=15.4658 test_px=97.9px
Epoch 304/999: train_loss=0.0656 train_px=5.5px | test_loss=15.3076 test_px=83.8px
Epoch 305/999: train_loss=0.0324 train_px=5.6px | test_loss=16.4385 test_px=105.5px
Epoch 306/999: train_loss=0.0875 train_px=6.5px | test_loss=21.2232 test_px=113.5px
Epoch 307/999: train_loss=0.0363 train_px=5.7px | test_loss=15.1330 test_px=92.9px
Epoch 308/999: train_loss=0.0291 train_px=5.5px | test_loss=18.8229 test_px=103.5px
Epoch 309/999: train_loss=0.1029 train_px=5.8px | test_loss=17.0031 test_px=97.5px
Epoch 310/999: train_loss=0.1022 train_px=5.7px | test_loss=16.7045 test_px=97.4px
Epoch 311/999: train_loss=0.0961 train_px=5.7px | test_loss=20.7790 test_px=103.0px
Epoch 312/999: train_loss=0.0544 train_px=5.8px | test_loss=16.2286 test_px=99.6px
Epoch 313/999: train_loss=0.1337 train_px=5.6px | test_loss=18.3502 test_px=123.0px
Epoch 314/999: train_loss=0.0504 train_px=5.8px | test_loss=19.3064 test_px=123.4px
Epoch 315/999: train_loss=0.0630 train_px=5.9px | test_loss=14.5007 test_px=105.9px
Epoch 316/999: train_loss=0.1052 train_px=6.2px | test_loss=15.4237 test_px=89.2px
Epoch 317/999: train_loss=0.0664 train_px=5.5px | test_loss=13.7617 test_px=103.9px
Epoch 318/999: train_loss=0.0143 train_px=5.4px | test_loss=15.2836 test_px=117.9px
Epoch 319/999: train_loss=0.0089 train_px=5.4px | test_loss=16.0660 test_px=122.0px
Epoch 320/999: train_loss=0.0220 train_px=5.4px | test_loss=15.5432 test_px=112.0px
Epoch 321/999: train_loss=0.0310 train_px=5.5px | test_loss=17.5510 test_px=117.8px
Epoch 322/999: train_loss=0.0140 train_px=5.3px | test_loss=15.7476 test_px=104.0px
Epoch 323/999: train_loss=0.0815 train_px=5.4px | test_loss=15.1072 test_px=117.1px
Epoch 324/999: train_loss=0.0638 train_px=5.5px | test_loss=12.9471 test_px=88.9px
Epoch 325/999: train_loss=0.0140 train_px=5.7px | test_loss=13.9290 test_px=87.9px
Epoch 326/999: train_loss=0.0205 train_px=5.5px | test_loss=12.0591 test_px=102.0px
Epoch 327/999: train_loss=0.1161 train_px=5.9px | test_loss=13.3339 test_px=95.7px
Epoch 328/999: train_loss=0.0920 train_px=5.5px | test_loss=14.1819 test_px=77.4px
Epoch 329/999: train_loss=0.0851 train_px=5.6px | test_loss=12.3013 test_px=93.1px
Epoch 330/999: train_loss=0.0931 train_px=6.2px | test_loss=11.7164 test_px=112.6px
Epoch 331/999: train_loss=0.0774 train_px=6.0px | test_loss=11.3372 test_px=123.2px
Epoch 332/999: train_loss=0.0603 train_px=6.0px | test_loss=12.6443 test_px=138.2px
Epoch 333/999: train_loss=0.0263 train_px=5.4px | test_loss=13.1726 test_px=100.1px
Epoch 334/999: train_loss=0.0649 train_px=5.6px | test_loss=10.2201 test_px=103.5px
Epoch 335/999: train_loss=0.0302 train_px=5.5px | test_loss=12.1658 test_px=86.0px
Epoch 336/999: train_loss=0.0671 train_px=5.7px | test_loss=12.4205 test_px=95.3px
Time limit reached (60 min). Stopping.

Done! Best test loss: 6.2121
Checkpoints: /data/cameron/scratch_files/hand_vids/checkpoints
wandb run: https://wandb.ai/cameronsmithbusiness/para_hand_tracking/runs/w6m9ncp8