o ?߱i>@sVdZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z ddl mZmZdd lmZmZdd lmZ d"d ejd ed eejdeejdBdef ddZ d#dedededededededBfddZ d#dedededede dededefddZd#dedededededBf ddZ d#dededededef d d!ZdS)$zt Imaginaire4 Attention Subpackage: Unified implementation for all Attention implementations. NATTEN backend checks )partialN)Tensor)attention_param_checksattention_tensor_checks!multi_dim_attention_tensor_checks) CausalType)NATTEN_SUPPORTED)get_bwd_dtypesget_fwd_dtypes) get_arch_taglog_or_raise_error)safe_logdtype is_training dtypes_fwd dtypes_bwdreturncCs|r |dur ||vS||vS)an Helper determining whether dtype is supported with different sets of supported dtypes for training and inference (forward+backward and forward). Parameters: dtype (torch.dtype): tensor element type. is_training (bool): whether use case can be used to backpropagate (tensor.requires_grad). dtypes_fwd (list[torch.dtype]): list of dtypes allowed for inference only (when not tensor.requires_grad). dtypes_bwd (list[torch.dtype] | None): Optional list of dtypes allowed for training only (when tensor.requires_grad), if different from dtypes_fwd. Nrrrrrra/data/cameron/vidgen/cosmos-predict2.5/cosmos_predict2/_src/imaginaire/attention/natten/checks.pydtype_supported(s rFquerykeyvalue is_causal is_varlen raise_errorc Cstt|d}t|j}|j}|j} |jd|jdk} t|jd|jd} |dkr1t ddSt j t j t j t jg} t j t j g} t|| | | d}| dv}|dvrY| sY|rY|rYd Sd }|dvrg|d |d 7}| rm|d 7}|s| r}|d|d| d7}n |d|d| d7}|s|d| d7}t d|t j t j g}t|| |d}| dvr| p| dv}|dkr|s|s| s|r|rdSd }|dkr|d |d7}|r|d7}|r|d7}| r|d 7}|s|d|d|d7}|s|d| d| d7}t d|t jt j t j g}t|| |d}| dd k}|r!|r!d!Sd }|s1|d|d|d7}|s<|d| d7}t d"||d#|d$|d%|d&| d' td(dS))a? Chooses an FMHA backend in NATTEN (cutlass-fmha, hopper-fmha, blackwell-fmha) for the current use case based on features needed and current GPU architecture. Using tensor shapes, it infers whether MLA (head_dim_value != head_dim_qk) or GQA/MQA (heads_kv != heads_q) are required. Using tensor device, it infers GPU architecture and compatible backends. Using arguments is_causal and is_varlen, and other inferred features, it picks the best available backend. It is possible for no backend to be selected, if the combination of features is not available in any one of the NATTEN backends, in which case it will return None. Parameters: query (Tensor): 4-D query tensor, with the heads-last contiguous layout (`[batch, seqlen, heads, head_dim]`). key (Tensor): 4-D key tensor, with the heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim]`). value (Tensor): 4-D value tensor, with heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim_v]`). is_causal (bool): whether or not causal masking is enabled. is_varlen (bool): whether or not a variable length (varlen) use case. Must be inferred beforehand based on arguments such as seqlens_{Q,KV} or cumulative_seqlen_{Q,KV} being passed. raise_error (bool): whether to raise an error if no backend is selected, instead of just returning None. Default is False. Returns: backend (str | None): selected NATTEN backend, if any compatible. rKzNNATTEN is not supported because compute capability is below the minimum (7.5).Nr) @)dgblackwell-fmhazIncompatible architecture (z, expected 100 or 103). z1Use case is MLA (head_dim_qk != head_dim_value). z Data type z2 is not in list of supported dtypes for training: z. z3 is not in list of supported dtypes for inference: z head_dim=z is not supported. z9NATTEN backend blackwell-fmha is not compatible. Reason: )rrr)r r!r"Z hopper-fmhaz, expected 90). zUse case is causal. zUse case is varlen. z% is not in list of supported dtypes: z with is_training=z6NATTEN backend hopper-fmha is not compatible. Reason: r cutlass-fmhaz7NATTEN backend cutlass-fmha is not compatible. Reason: z=Could not find a compatible NATTEN FMHA backend for arch_tag=z , is_causal=z , is_varlen=z , is_mla=. exception)rr r devicer requires_gradshapemaxlogdebugtorchfloat16bfloat16 float8_e5m2 float8_e4m3fnrfloat32 RuntimeError)rrrrrr target_fnarch_tagrrZis_mlahead_dimZblackwell_fmha_fwd_dtypesZblackwell_fmha_bwd_dtypesZdtype_supported_blackwellZhead_dim_supported_blackwellreasonZhopper_fmha_dtypesZdtype_supported_hopperZhead_dim_supported_hopperZcutlass_fmha_dtypesZdtype_supported_cutlassZhead_dim_supported_cutlassrrrchoose_natten_backend@s '      r@ causal_typec Cstt|d}ts|dtddSt|j}t|} t|} t|||| | dd|dd s2|dtddSt |||||d |rM|t j t j fvrM|d tddSt ||||||d } | d ur]dSdS) a> Input validation function for the NATTEN backend. Runs the common checks in addition to trying to find a compatible NATTEN backend. If any checks fail, or no compatible backend is found in NATTEN, returns False. Parameters: query (Tensor): 4-D query tensor, with the heads-last contiguous layout (`[batch, seqlen, heads, head_dim]`). key (Tensor): 4-D key tensor, with the heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim]`). value (Tensor): 4-D value tensor, with heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim_v]`). is_causal (bool): whether or not causal masking is enabled. causal_type (CausalType): causal masking mode. Choices: `CausalType.TopLeft`, `CausalType.BottomRight`. Required when `is_causal = True`. is_varlen (bool): whether or not a variable length (varlen) use case. Must be inferred beforehand based on arguments such as seqlens_{Q,KV} or cumulative_seqlen_{Q,KV} being passed. raise_error (bool): whether to raise an error if any checks fail or no backend is selected, instead of just returning False. Default is False. Returns: success (bool): whether use case is compatible with NATTEN backend. rlNATTEN is not supported in this environment. Run with debug logs to find out why, or choose another backend.r-FTzNATTEN Attention rrrsupported_dtypes_forwardsupported_dtypes_backward supports_mlasupports_gqa_mqar backend_name)NATTEN does not support the given inputs.)rrrrrAz?NATTEN Attention only supports top-left causal masking for now.)rrrN)rr rr;r r/r r rrrTopLeftDontCarer@) rrrrrArrr<r= fwd_dtypes bwd_dtypesnatten_backendrrrnatten_attention_checksN (    rOcCs4t|||dd|d}dddd}||vsJ||S)ag Chooses an FNA backend in NATTEN (cutlass-fna, hopper-fna, blackwell-fna) for the current use case based on features needed and current GPU architecture. Using tensor shapes, it infers whether MLA (head_dim_value != head_dim_qk) or GQA/MQA (heads_kv != heads_q) are required. Using tensor device, it infers GPU architecture and compatible backends. Using arguments is_causal and is_varlen, and other inferred features, it picks the best available backend. It is possible for no backend to be selected, if the combination of features is not available in any one of the NATTEN backends, in which case it will return None. Parameters: query (Tensor): 4-D, 5-D, or 6-D query tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads, head_dim]`). key (Tensor): 4-D, 5-D, or 6-D key tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim]`). value (Tensor): 4-D, 5-D, or 6-D value tensor, with heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim_v]`). raise_error (bool): whether to raise an error if no backend is selected, instead of just returning None. Default is False. Returns: backend (str | None): selected NATTEN backend, if any compatible. F)rrrrrrz cutlass-fnaz hopper-fnaz blackwell-fna)r+r)r%)r@)rrrrZ fmha_backendZ"natten_fmha_backend_to_fna_backendrrrchoose_natten_multi_dim_backend!s#  rPc Cstt|d}ts|dtddSt|j}t|}t|}t|||||dd|dd s2|dtddSt ||||d}|d ur@dSdS) a Input validation function for the NATTEN multi-dimensional backend. Runs the common checks in addition to trying to find a compatible NATTEN backend. If any checks fail, or no compatible backend is found in NATTEN, returns False. Parameters: query (Tensor): 4-D, 5-D, or 6-D query tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads, head_dim]`). key (Tensor): 4-D, 5-D, or 6-D key tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim]`). value (Tensor): 4-D, 5-D, or 6-D value tensor, with heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim_v]`). raise_error (bool): whether to raise an error if any checks fail or no backend is selected, instead of just returning False. Default is False. Returns: success (bool): whether use case is compatible with NATTEN backend. rrBr-FTz"NATTEN Multi-Dimensional AttentionrCrIN) rr rr;r r/r r rrP) rrrrr<r=rLrMrNrrr natten_multi_dim_attention_checkWs6   rQ)N)F)__doc__ functoolsrr5r0cosmos_predict2._src.imaginaire.attention.checksrrr/cosmos_predict2._src.imaginaire.attention.masksr0cosmos_predict2._src.imaginaire.attention.nattenrZ5cosmos_predict2._src.imaginaire.attention.natten.metar r /cosmos_predict2._src.imaginaire.attention.utilsr r r r3rboollistrstrr@rOrPrQrrrrs         $Z: