o ?߱it@sdZddlmZmZddlmZddlmZddlm Z          dded ed ed e d e dBd e dBdedBdedBde dBde dBde de dBdeeeefBfddZdS)z Imaginaire4 Attention Subpackage: Unified implementation for all Attention implementations. Flash Attention v2 (flash2) Backend: intermediate APIs Only safe to import when FLASH2_SUPPORTED is True. )flash_attn_funcflash_attn_varlen_func)Tensor)flash2_attention_check) CausalTypeFNquerykeyvalue is_causal causal_typescalecumulative_seqlen_Qcumulative_seqlen_KV max_seqlen_Q max_seqlen_KV return_lsebackend_kwargsreturnc  Cs|du} t|||||| ddsJ|dur|n|jdd}| dur$| ni} | r|jd|jdkrA|jdkrAdksDJJ|d} |d}|d}| |kri|kridkslJJtd |d|d|d|||| ||dd | \}}}|dksJ|d ksJ|d}|d}ntd |||||dd | \}}}t|tsJt|tsJ|d ksJ|dksJ| dd d}| r||fS|S)a Runs Flash Attention v2 on given operands (Q, K, V) with the heads-last contiguous layout (`[batch, seqlen, heads, head_dim]`). Parameters: query (Tensor): 4-D query tensor, with the heads-last contiguous layout (`[batch, seqlen, heads, head_dim]`) key (Tensor): 4-D key tensor, with the heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim]`) value (Tensor): 4-D value tensor, with heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim_v]`) is_causal (bool): whether or not causal masking is enabled. Default is False. causal_type (CausalType): causal masking mode. Choices: `CausalType.TopLeft`, `CausalType.BottomRight`. Required when `is_causal = True`. scale (float | None): Dot product scale (attention scale). Defaults to head_dim ** -0.5. cumulative_seqlen_Q (Tensor | None): (varlen) Optional 1-D tensor with size `batch + 1` indicating the cumulative sum of number of query tokens in each batch, with an additional 0 element in the beginning. Must be passed together with `cumulative_seqlen_KV` and `max_seqlen_{Q,KV}`. cumulative_seqlen_KV (Tensor | None): (varlen) Optional 1-D tensor with size `batch + 1` indicating the cumulative sum of number of key/value tokens in each batch, with an additional 0 element in the beginning. Must be passed together with `cumulative_seqlen_Q` and `max_seqlen_{Q,KV}`. max_seqlen_Q (int | None): (varlen) Optional integer indicating the maximum query sequence length in all batches. Must be passed together with `cumulative_seqlen_{Q,KV}` and `max_seqlen_KV`. max_seqlen_KV (int | None): (varlen) Optional integer indicating the maximum key/value sequence length in all batches. Must be passed together with `cumulative_seqlen_{Q,KV}` and `max_seqlen_Q`. Other Parameters: return_lse (bool): Whether to return the logsumexp values. Default is False. backend_kwargs (dict | None): Key-value pair for passing arguments specific to Flash's attention operator, if any. Returns: output (Tensor): 4-D output tensor, with the heads-last contiguous layout (`[batch, seqlen, heads, head_dim_v]`). logsumexp (Tensor): logsumexp tensor, with the heads-last contiguous layout (`[batch, seqlen, heads, 1]`). Only returned when return_lse is True. NOTE: this tensor is not contiguous in this backend (Flash2) and it should not be made contiguous unless we can guarantee its results aren't merged via `merge_attentions`. NT)rrr r r is_varlen raise_errorgr) qkv cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_k softmax_scalecausalreturn_attn_probs)rrrr r!r") rshapesqueezedimr unsqueezer isinstancerpermute)rrr r r r r rrrrrrrrroutZlse__outputlser%r%d/data/cameron/vidgen/cosmos-predict2.5/cosmos_predict2/_src/imaginaire/attention/flash2/functions.pyflash2_attentionslE  8   2    r1) FNNNNNNFN)__doc__flash_attn.flash_attn_interfacerrtorchr7cosmos_predict2._src.imaginaire.attention.flash2.checksr/cosmos_predict2._src.imaginaire.attention.masksrboolfloatintdicttupler1r%r%r%r0sR