o iBF@sdZddlmZddlZddlmZmZddlZddl Z ddl m Z ddl m Z ddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlm Z ddl!m"Z"de j#j$_%dZ&dZ'Gddde j(e Z)dS)z Depth Anything 3 API module. This module provides the main API for Depth Anything 3, including model loading, inference, and export capabilities. It supports both single and nested model architectures. ) annotationsN)OptionalSequence)PyTorchModelHubMixin)Image) create_object load_config)MODEL_REGISTRY) Prediction)export)affine_inverse)InputProcessor)OutputProcessor)logger)align_poses_umeyamaFzmodel.safetensorsz config.jsoncseZdZUdZdZded<dVdWfdd Ze    dXdYddZ dddd d d dddddddddd dd!ifdZd7d8Z    d[d\d:d;Z d]d=d>Z d^d@dAZ  Bd_d`dEdFZ  dadbdIdJZdcdLdMZdddNdOZdedQdRZdfdTdUZZS)gDepthAnything3a\ Depth Anything 3 main API class. This class provides a high-level interface for depth estimation using Depth Anything 3. It supports both single and nested model architectures with metric scaling capabilities. Features: - Hugging Face Hub integration via PyTorchModelHubMixin - Support for multiple model presets (vitb, vitg, nested variants) - Automatic mixed precision inference - Export capabilities for various formats (GLB, PLY, NPZ, etc.) - Camera pose estimation and metric depth scaling Usage: # Load from Hugging Face Hub model = DepthAnything3.from_pretrained("huggingface/model-name") # Or create with specific preset model = DepthAnything3(preset="vitg") # Run inference prediction = model.inference(images, export_dir="output", export_format="glb") N str | None _commit_hash da3-large model_namestrc sPt||_tt|j|_t|j|_|jt |_ t |_ d|_ dS)a1 Initialize DepthAnything3 with specified preset. Args: model_name: The name of the model preset to use. Examples: 'da3-giant', 'da3-large', 'da3metric-large', 'da3nested-giant-large'. **kwargs: Additional keyword arguments (currently unused). N)super__init__rrr Zconfigrmodelevalr input_processorroutput_processordevice)selfrkwargs __class__2/data/cameron/da3_repo/src/depth_anything_3/api.pyrKs    zDepthAnything3.__init__Fsaddle_balancedimage torch.Tensor extrinsicstorch.Tensor | None intrinsicsexport_feat_layerslist[int] | Noneinfer_gsbool use_ray_poseref_view_strategyreturndict[str, torch.Tensor]c Cstjrtjntj}t4tj|jj|d| |||||||WdWdS1s6wYWddS1sFwYdS)a Forward pass through the model. Args: image: Input batch with shape ``(B, N, 3, H, W)`` on the model device. extrinsics: Optional camera extrinsics with shape ``(B, N, 4, 4)``. intrinsics: Optional camera intrinsics with shape ``(B, N, 3, 3)``. export_feat_layers: Layer indices to return intermediate features for. infer_gs: Enable Gaussian Splatting branch. use_ray_pose: Use ray-based pose estimation instead of camera decoder. ref_view_strategy: Strategy for selecting reference view from multiple views. Returns: Dictionary containing model predictions )Z device_typeZdtypeN) torchcudaZis_bf16_supportedZbfloat16Zfloat16Zno_gradZautocastrtyper) rr%r'r)r*r,r.r/Zautocast_dtyper"r"r#forwardcs "zDepthAnything3.forwardTupper_bound_resizeZmini_npzgD@i@B$list[np.ndarray | Image.Image | str]np.ndarray | Nonealign_to_input_ext_scale render_exts render_ixts render_hwtuple[int, int] | None process_resintprocess_res_method export_dir export_formatSequence[int] | Noneconf_thresh_percentilefloatnum_max_points show_cameras feat_vis_fps export_kwargsOptional[dict]r c Csd|vr |s Jdd|vrt|dtsJd||||| | \}}}||||\}}}||dur9|nd}|durDt|ng}||||||||}||}| ||||}| ||}| durd|vr|rwd|vrw|d}d|vrd|vri|d<|d || | d d |vrd |vri|d <|d  |||d d |vrd |vri|d <|d  d |id|vrd|vri|d<|d ||| d|j ||| fi||S)u Run inference on input images. Args: image: List of input images (numpy arrays, PIL Images, or file paths) extrinsics: Camera extrinsics (N, 4, 4) intrinsics: Camera intrinsics (N, 3, 3) align_to_input_ext_scale: whether to align the input pose scale to the prediction infer_gs: Enable the 3D Gaussian branch (needed for `gs_ply`/`gs_video` exports) use_ray_pose: Use ray-based pose estimation instead of camera decoder (default: False) ref_view_strategy: Strategy for selecting reference view from multiple views. Options: "first", "middle", "saddle_balanced", "saddle_sim_range". Default: "saddle_balanced". For single view input (S ≤ 2), no reordering is performed. render_exts: Optional render extrinsics for Gaussian video export render_ixts: Optional render intrinsics for Gaussian video export render_hw: Optional render resolution for Gaussian video export process_res: Processing resolution process_res_method: Resize method for processing export_dir: Directory to export results export_format: Export format (mini_npz, npz, glb, ply, gs, gs_video) export_feat_layers: Layer indices to export intermediate features from conf_thresh_percentile: [GLB] Lower percentile for adaptive confidence threshold (default: 40.0) # noqa: E501 num_max_points: [GLB] Maximum number of points in the point cloud (default: 1,000,000) show_cameras: [GLB] Show camera wireframes in the exported scene (default: True) feat_vis_fps: [FEAT_VIS] Frame rate for output video (default: 15) export_kwargs: additional arguments to export functions. Returns: Prediction object containing depth maps and camera parameters Zgsz6must set `infer_gs=True` to perform gs-related export.Zcolmaprz.`image` must be image paths for COLMAP export.NZgs_videoz -gs_video)r'r)Z out_image_hwZglb)rFrHrIZfeat_visZfps)Z image_pathsrFrB) isinstancer_preprocess_inputs_prepare_model_inputs_normalize_extrinsicsZclonelist_run_model_forward_convert_to_prediction%_align_to_input_extrinsics_intrinsics_add_processed_imagesupdate_export_results)rr%r'r)r;r,r.r/r<r=r>r@rBrCrDr*rFrHrIrJrKimgs_cpuimgsex_tin_t ex_t_norm raw_output predictionr"r"r# inferencest8       zDepthAnything3.inference=tuple[torch.Tensor, torch.Tensor | None, torch.Tensor | None]c Csht}|||dur|nd|dur|nd||\}}}t}td||d|j|||fS)z.Preprocess input images using input processor.NzProcessed Images Done takingzseconds. Shape: )timercopyrinfoshape) rr%r'r)r@rB start_timerXend_timer"r"r#rNs    z!DepthAnything3._preprocess_inputsrXcCsl|}|j|ddd}|dur|j|dddnd}|dur/|j|dddnd}|||fS)z Prepare tensors for model input.T)Z non_blockingN)_get_model_devicetorG)rrXr'r)rrYrZr[r"r"r#rO-s z$DepthAnything3._prepare_model_inputsrZcCs|durdSt|ddddf}||}t|}|ddddf}|jdd}t|}tj|dd}|ddddf||ddddf<|S) zNormalize extrinsicsN.)Zdimg?)min)r Znormr2ZmedianZclamp)rrZZ transformr\Zc2wsZ translationsZdistsZ median_distr"r"r#rPGs  $z$DepthAnything3._normalize_extrinsics r^ransac_view_threshc Csz|dur|S||_t|j|t||kddd\}}}}|r8|dddddf|_|j|_|S||_|S)z#Align depth map to input extrinsicsNT*)ZransacZreturn_alignedZ random_state.rj)numpyr)rr'lenZdepth) rr'r)r^r;rn_ZscaleZaligned_extrinsicsr"r"r#rTUs    z4DepthAnything3._align_to_input_extrinsics_intrinsicsrYr[c Cs|j}|jdk} | rtj|t} |durt|nd} ||||| |||} | r1tj|t} t d| | d| S)zRun model forward pass.r3NzModel Forward Pass Done. Time:  seconds) rr4r2r3Z synchronizerarQr5rrc)rrYrZr[r*r,r.r/rZ need_syncreZ feat_layersoutputrfr"r"r#rRos   z!DepthAnything3._run_model_forwardr]cCs4t}||}t}td||d|S)z.Convert raw model output to Prediction object.z%Conversion to Prediction Done. Time: rs)rarrrc)rr]rertrfr"r"r#rSs  z%DepthAnything3._convert_to_predictioncCsh|dddd}tgd}tgd}|||}t|dd}|dtj}||_|S)z5Add processed images to prediction for visualization.rrjri)g ףp= ?gv/?gCl?)gZd;O?gy&1?g?) ZpermuteZcpurpnpZarrayZclipZastypeZuint8Zprocessed_images)rr^rXZprocessed_imgsZmeanZstdr"r"r#rUs z$DepthAnything3._add_processed_imagesNonecKs>t}t|||fi|t}td||ddS)z1Export results to specified format and directory.zExport Results Done. Time: rsN)rar rrc)rr^rDrCrrerfr"r"r#rWszDepthAnything3._export_results torch.devicecCsT|jdur|jS|D] }|j|_|jS|D] }|j|_|jStd)z Get the device where the model is located. Returns: Device where the model parameters are located Raises: ValueError: If no tensors are found in the model NzNo tensor found in model)r parametersZbuffers ValueError)rZparambufferr"r"r#rgs     z DepthAnything3._get_model_device)r)rr)NNNFFr$)r%r&r'r(r)r(r*r+r,r-r.r-r/rr0r1)*r%r9r'r:r)r:r;r-r,r-r.r-r/rr<r:r=r:r>r?r@rArBrrCrrDrr*rErFrGrHrArIr-rJrArKrLr0r )NNr6r7) r%r9r'r:r)r:r@rArBrr0r`)rXr&r'r(r)r(r0r`)rZr(r0r()Trm) r'r(r)r(r^r r;r-rnrAr0r )NFFr$)rYr&rZr(r[r(r*rEr,r-r.r-r/rr0r1)r]r1r0r )r^r rXr&r0r )r^r rDrrCrr0rx)r0ry)__name__ __module__ __qualname____doc__r__annotations__rr2Zinference_moder5r_rNrOrPrTrRrSrUrWrg __classcell__r"r"r r#r0sh  $        r)*r __future__rratypingrrrprwr2Ztorch.nnZnnZhuggingface_hubrZPILrZdepth_anything_3.cfgrrZdepth_anything_3.registryr Zdepth_anything_3.specsr Zdepth_anything_3.utils.exportr Zdepth_anything_3.utils.geometryr Z)depth_anything_3.utils.io.input_processorr Z*depth_anything_3.utils.io.output_processorrZdepth_anything_3.utils.loggerrZ!depth_anything_3.utils.pose_alignrZbackendsZcudnnZ benchmarkZSAFETENSORS_NAMEZ CONFIG_NAMEZModulerr"r"r"r#s,