o Gi, @sdZddlZddlZddlmZddlZddlZddlm Z edZ defddZ d ed e dBfd d Z d&d ed edBfddZdejde de fddZd'dejdedefddZ  d(dejdejdejded ef d!d"Zd#d$Zed%kr{edSdS))aV Run CoTracker 3 on RTX GIFs: forward point tracking from the first frame. Uses a grid of points on frame 0 and tracks them to the end. Grid density and video FPS are parameters you can inspect. Requires: pip install opencv-python (and PyTorch with CUDA for best speed). CoTracker3 is loaded via torch.hub from facebookresearch/co-tracker. N)Path)Imagez/data/RTX/RTX_VideorootcCsNt|}g}t|D]}|sqt|dD]}||qq|S)z#Return sorted list of root/*/*.gif.z*.gif)rresolvesortediterdiris_dirglobappend)routZsubdirfr D/data/cameron/vidgen/unified_video_action/scripts/point_track_rtx.py find_gifss  rpathreturncCst|/}t|dd}d}t|D]}|||jdd}||dur)t|nd7}qWdn1s7wY|dkrBdS||dS)zJReturn effective FPS from GIF duration metadata, or None if not available.n_framesrdurationNg@@)ropengetattrrangeseekinfogetint)rimrZtotal_msidr r rget_gif_effective_fps#s     r max_framesc Csg}t|.}t|dd}|durt||}t|D]}|||t| dqWdn1s8wYtj |ddS)z5Load all frames of a GIF as numpy (T, H, W, 3) uint8.rrNZRGBraxis) rrrminrrr nparrayconvertstack)rr framesrnrr r rload_gif_frames1s     r*r( source_fps target_fpscCs`|dks |dks ||kr|S||}ttdt||t}t|dt|d}||S)zFSubsample frames to target_fps. frames: (T,H,W,C). Returns (T',H,W,C).rr)r$roundarangelenastyperclip)r(r+r,stepindicesr r rsubsample_to_fps>s r4cudavideo grid_sizedevicecCsjtjjdddd|}||}t|||d\}}Wdn1s(wY||fS)z Run CoTracker3 offline on video. Forward tracking only (grid on first frame). video: (1, T, C, H, W) float in [0, 255] Returns: pred_tracks (1, T, N, 2), pred_visibility (1, T, N, 1) zfacebookresearch/co-trackerZcotracker3_offlineT) trust_repo)r7N)torchhubloadtono_gradcpu)r6r7r8Z cotracker pred_trackspred_visibilityr r r run_cotrackerHs  rBr@rA point_radiusline_thicknessc Cszddl}Wn tytdw|j\}}}} |} |j\} } } |d} | j|| kr8| dk|| }n|ddk}|jdkrK||| }|j|| fksaJd|jd |d | d t | tj }t |dd|d|d<t |d d|d|d <t |D]}t | D])}|||fsqt|||dft|||df}}|| |||f|d dq|dkr t | D]J}||d|fr|||fsqt||d|dft||d|df}}t|||dft|||df}}|| |||f||fd|qq| S)z Overlay tracks on frames. pred_tracks (T, N, 2), pred_visibility (T, N, 1). Uses rounded coordinates for stable drawing. Returns (T, H, W, 3) uint8. rNzGopencv-python is required for drawing tracks. pip install opencv-pythong?).rrzvisibility shape z vs (T=z, N=)).r)rr)rr)cv2 ImportErrorshapecopyreshapesizesqueezendimr$r-r0int32r1rrcircleline)r(r@rArErFrKTHWCr _NvZvisZ tracks_inttr)xyx0y0x1y1r r r draw_tracksZsB      ,   &  .& rdc# Cstjdd}|jdtttdd|jdtddd|jdtd d d|jd td d d|jdtddd|jdtddd|jdtddd|jdtddd|}|jrht |j}| sgt d|n*t |j  }|szt d|t|}|st d||d}td|t|}|dus|dkr|j}td |d!ntd"|d#|j}|dur|dkr|j|krt|||j}td$|d%|d&t||d'}|j\}} } } td(|d)| d*| |dkr(|j|kr(t|||j}|jdurt||jkr|d|j}t|}td+|jd,|d-t|dd.d/d0d} | jd/|krP| jd.| krP| jd1| ksRJ|d0} |d/| }|dksf| dkrtd2|jd3t | |j|j!d4\}}|"d}|"d}|}nt#t$||| d}| dd|f}t#t$||| d/}| dd|f}td5|d6|jd7| d8| d9 t ||j|j!d4\}}t ||j|j!d4\}}|"d}|"d}|"d}|"d}t%j&||d/dgdd:}t%j&||d/dgdd:}|j'd0kr|dddddf}||d/d}||}td;|jtd<t(|||}t |j)j*rFt |j)+d=nt |j)}|j,|j-d>}|j,j.d?d?d@dA}zddl/m0}|j1t||dBdC|jdDd?}Wn t2y~Ynw|sddl3}|j4dE} |5t|| |j| | f}!|!6s|j4dF} |5t|| |j| | f}!|D] }"|!7|8|"|j9q|!:tdG|dH|jd&tdIdS)JNzBRun CoTracker 3 on one RTX GIF (forward tracking from first frame)) descriptionz --data-rootz9Root with subdirs of .gif files (used if --video not set))typedefaulthelpz--videozBPath to a single GIF. If not set, use first GIF under --data-root.z --grid-size zGGrid density: grid_size x grid_size points on first frame (default 32).z--fpsg@z2Load GIF and output video at this FPS (default 2).z --source-fpsg4@z[Fallback FPS when GIF has no duration metadata; used for subsampling to --fps (default 20).z --max-frames<zqMax frames to process (after subsampling to --fps). When subsampling, loads more from GIF as needed (default 60).z--outz out/rtx_trackzEOutput path prefix: writes {out}_tracks.mp4 (default: out/rtx_track).z--devicer5z%Device for CoTracker (default: cuda).zVideo not found: zData root not found: zNo *.gif under rGzUsing first GIF: rz Using --source-fps z (no duration metadata in GIF)z GIF effective FPS: z.1fz Loading GIF: z (max_frames=rH)r z Loaded z frames, r^z Subsampled to z fps: z framesrCrrDzRunning CoTracker 3 (grid_size=z, forward only) ...)r8z&Running CoTracker 3 from center frame z (grid_size=z): z back + z forward ...r!z Tracks shape: zDrawing tracks ...z _tracks.mp4T)parentsexist_okFpyavlibx264)plugincodecfpsavc1mp4vzSaved: z (fps=zDone.);argparseArgumentParser add_argumentstrDEFAULT_DATA_ROOTrfloat parse_argsr6ris_fileFileNotFoundError data_rootrrrprintrr+r rsmathceilr*rMr4r/r: from_numpypermute unsqueezer7rBr8numpylistrr$ concatenaterRrdr suffix with_suffixparentnamemkdir imageio.v3v3imwrite ExceptionrKVideoWriter_fourcc VideoWriterisOpenedwritecvtColor COLOR_RGB2BGRrelease)#pargsgif_pathrgifsr+ load_limitr(rVrWrXrYr6kmiddler@rAout_frames_arr indices_back clip_back indices_fwdclip_fwd tracks_backvis_back tracks_fwdvis_fwd out_indices out_framesout_pathwritteniiorKfourccwriterr r r rmains   4   $      (     r__main__)N)r5)rCrD)__doc__rvrpathlibrrr$r:PILrrzrr{rrr*ndarrayr4TensorryrBrdr__name__r r r rs>       0+