o ?߱i'@sdZddlmZddlmZddlmZmZddlm Z ddl m Z m Z ddl mZddlmZmZ dd ed ed ed ededededefddZdS)z Imaginaire4 Attention Subpackage: Unified implementation for all Attention implementations. Flash Attention v3 (flash3) backend checks )partial)Tensor)attention_param_checksattention_tensor_checks)FLASH3_SUPPORTED)get_bwd_dtypesget_fwd_dtypes) CausalType) get_arch_taglog_or_raise_errorFquerykeyvalue is_causal causal_type is_varlen raise_errorreturnc Cs"tt|d}ts|dtddSt|j}t|} t|} t|||| | dd|dd s2|dtddS|j d |j d krt|j d } |j d } | d krN| d kstd | krXd krenn d| krdd kstn|d| d| dt ddSt |||||d|r|t j t jfvr|dt ddSdS)a Input validation function for the flash3 backend. Parameters: query (Tensor): 4-D query tensor, with the heads-last contiguous layout (`[batch, seqlen, heads, head_dim]`). key (Tensor): 4-D key tensor, with the heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim]`). value (Tensor): 4-D value tensor, with heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim_v]`). is_causal (bool): whether or not causal masking is enabled. causal_type (CausalType): causal masking mode. Choices: `CausalType.TopLeft`, `CausalType.BottomRight`. Required when `is_causal = True`. is_varlen (bool): whether or not a variable length (varlen) use case. Must be inferred beforehand based on arguments such as seqlens_{Q,KV} or cumulative_seqlen_{Q,KV} being passed. raise_error (bool): whether to raise an error if any checks fail or no backend is selected, instead of just returning False. Default is False. Returns: success (bool): whether use case is compatible with flash3 backend. )rzFlash Attention v3 (flash3) is not supported in this environment. Run with debug logs to find out why, or choose another backend.) exceptionFTzFlash Attention v3 (flash3)) r r rsupported_dtypes_forwardsupported_dtypes_backward supports_mlasupports_gqa_mqar backend_namez>Flash Attention v3 (flash3) does not support the given inputs.@i`zFlash Attention v3 (flash3) does not support this head dim combination. Expected either head_dim_qk <= 64 and head_dim_v <= 512, or 128 <= head_dim_qk <= 192 and 96 <= head_dim_v <= 128, got head_dim_q=z , head_dim_v=.)r r rrrz=Flash Attention v3 only supports bottom-right causal masking.)rr r RuntimeErrorr devicerrrshape ValueErrorrr BottomRightDontCare) r r rrrrr target_fnarch_tag fwd_dtypes bwd_dtypesZ head_dim_q head_dim_vr+a/data/cameron/vidgen/cosmos-predict2.5/cosmos_predict2/_src/imaginaire/attention/flash3/checks.pyflash3_attention_check"s^ &    > r-N)F)__doc__ functoolsrtorchr0cosmos_predict2._src.imaginaire.attention.checksrr0cosmos_predict2._src.imaginaire.attention.flash3rZ5cosmos_predict2._src.imaginaire.attention.flash3.metarr/cosmos_predict2._src.imaginaire.attention.masksr /cosmos_predict2._src.imaginaire.attention.utilsr r boolr-r+r+r+r,s4