o vic$@s$dZddlZddlmZddlmZmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZdd lmZdd lmZmZdd lmZeeeed Zd eiZ            d*dedededededBdedBdedBdedBdedBdedBdedBdedBde dBdede!dBdee"eefBf dd Z# ! " "     d+dededed#e"eBd$e"eBd%e"eBde"eBdedBde dBdede!dBdee"eefBfd&d'Z$ ! " "    d,dededed#e"eBd$e"eBd%e"eBdedBde dBdede!dBdee"eefBfd(d)Z%dS)-zl Imaginaire4 Attention Subpackage: Unified implementation for all Attention implementations. Frontend APIs N)Tensor)choose_backendchoose_multi_dim_backend)attention_param_checksattention_tensor_checks multi_dim_attention_param_checks multi_dim_attention_param_filter!multi_dim_attention_tensor_checksvarlen_tensor_checks)cudnn_attention)flash2_attention)flash3_attention) CausalType)natten_attentionnatten_multi_dim_attention)safe_log)cudnnnattenZflash2Zflash3rFquerykeyvalue is_causal causal_typescale seqlens_Q seqlens_KVcumulative_seqlen_Qcumulative_seqlen_KV max_seqlen_Q max_seqlen_KVbackend return_lsebackend_kwargsreturncCst|||dds Jt|||||dt||||||| | | d \}} } } |du}|dur.|n|jdd}| durD|durDd}td| durY| tvrYtd | d td t ||||||| d d }|dur|jd|jdko||jd|jdk}|r|jd}|jd}||dksJ||}|}t j ||d|d}t j ||d|d}tdt |||||||| | | | | |d S| durtdtd| d|tvsJt||||||||| | | | |d S)a Runs Attention on given operands (Q, K, V) with the heads-last contiguous layout (`[batch, seqlen, heads, head_dim]`). Varlen Attention is only supported for the sequence-packed layout: QKV tensors have batch size 1, and tokens from different batches are concatenated without any padding along the sequence dimension. Sequence lengths for different batches can be provided in two ways: 1. `seqlens_Q` and `seqlens_KV` (less efficient): only provide the sequence lengths as integer tensors (must be on the same device as QKV), and cumulative and maximum sequence lengths are recomputed on each call. 2. `cumulative_seqlen_{Q,KV}` and `max_seqlen_{Q,KV}` (more efficient): compute cumulative and maximum sequence lengths. `cumulative_seqlen_{Q,KV}` are integer tensors on the same device as QKV containing the cumulative sum of `seqlens_{Q,KV}`, with an additional `0` element in the beginning, therefore sized `batch+1`. `max_seqlen_{Q,KV}` are integers (not Tensors) that represent the maximum sequence lengths for Q and KV among all sequence batches. You can use `generate_varlen_parameters` to generate these parameters: ```python3 from cosmos_policy._src.imaginaire.attention.varlen import generate_varlen_parameters ( cumulative_seqlen_Q, cumulative_seqlen_KV, max_seqlen_Q, max_seqlen_KV, ) = generate_varlen_parameters(q, k, v, seqlens_Q, seqlens_KV) ``` Parameters: query (Tensor): 4-D query tensor, with the heads-last contiguous layout (`[batch, seqlen_q, heads, head_dim]`) key (Tensor): 4-D key tensor, with the heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim]`) value (Tensor): 4-D value tensor, with heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim_v]`) is_causal (bool): whether or not causal masking is enabled. Default is False. causal_type (CausalType): causal masking mode. Choices: `CausalType.TopLeft`, `CausalType.BottomRight`, `CausalType.DontCare` (only valid when seqlen_q == seqlen_kv). Required when `is_causal = True`. scale (float | None): Dot product scale (attention scale). Defaults to head_dim ** -0.5. seqlens_Q (Tensor | None): (varlen) Optional 1-D tensor with size `batch` indicating the number of query tokens in each batch. Must be passed together with `seqlens_KV`. seqlens_KV (Tensor | None): (varlen) Optional 1-D tensor with size `batch` indicating the number of key/value tokens in each batch. Must be passed together with `seqlens_Q`. cumulative_seqlen_Q (Tensor | None): (varlen) Optional 1-D tensor with size `batch + 1` indicating the cumulative sum of number of query tokens in each batch, with an additional 0 element in the beginning. Must be passed together with `cumulative_seqlen_KV` and `max_seqlen_{Q,KV}`. cumulative_seqlen_KV (Tensor | None): (varlen) Optional 1-D tensor with size `batch + 1` indicating the cumulative sum of number of key/value tokens in each batch, with an additional 0 element in the beginning. Must be passed together with `cumulative_seqlen_Q` and `max_seqlen_{Q,KV}`. max_seqlen_Q (int | None): (varlen) Optional integer indicating the maximum query sequence length in all batches. Must be passed together with `cumulative_seqlen_{Q,KV}` and `max_seqlen_KV`. max_seqlen_KV (int | None): (varlen) Optional integer indicating the maximum key/value sequence length in all batches. Must be passed together with `cumulative_seqlen_{Q,KV}` and `max_seqlen_Q`. Other Parameters: backend (str | None): Backend to run with. If unspecified (default), it will try to select the best available. return_lse (bool): Whether to return the logsumexp values. Default is False. backend_kwargs (dict | None): Key-value pair for passing arguments specific to the backend's attention operator, if any. Only valid when a specific backend is selected (backend is not None). Returns: output (Tensor): 4-D output tensor, with the heads-last contiguous layout (`[batch, seqlen_q, heads, head_dim_v]`). logsumexp (Tensor): logsumexp tensor, with the heads-last contiguous layout (`[batch, seqlen_q, heads, 1]`). Only returned when return_lse is True. Trrr raise_error)rrrrr) rrrrrrrrrNAA backend was not specified, but got backend_kwargs. Ignoring... Selected backend=, but available choices are . F)rrrrr is_varlenr r%r)repeatsdim output_sizezVBackend incompatible with GQA/MQA use case. Trying again with graph transformation... ) rrrrrrrrrrr!r r"zvCould not find a compatible Attention backend for this use case / device. Try running with debug logs to find out why.zSelected Attention backend zZ is incompatible with this use case / device. Try running with debug logs to find out why.) rrrrrrrrrrr!r") rrr shapelogdebug BACKEND_MAP ValueErrorkeysrtorchrepeat_interleave attention)rrrrrrrrrrrrr r!r"r,Zcompatible_backendZ is_gqa_mqaheadsZheads_kvZh_kquery_tkey_tvalue_tr>V/data/cameron/vidgen/cosmos-policy/cosmos_policy/_src/imaginaire/attention/frontend.pyr97sm   (     r9r& window_sizestridedilationc sLt|||dds Jt|||||d\}}}}t} fddt| DtdkrtfddtD} tfd dt|D} tfd dt|D}tfd dt|D}tfd dt|D}td d| DsyJtdd| DsJ|j|jdg| |jd|jdR}|j|jdg| |jd|jdR}|j|jdg| |jd|jdR}t dd|d|d|d|d| d| d|d|d|dt |||| |||||| | d St |||||dtddt |Dr]tdd|Dr| dkr]t d d|d|d!|d"ur;t d#|d$|d| }|d| }|d| }|d}t|||||tj| d%S|d"urd|n|jdd&}|d"ur|| d"ur|d"} t d't||||d(}|tvrtd)|d*td+t|||||||||| | d, S)-a Runs Multi-Dimensional Attention on given operands (Q, K, V) with the heads-last contiguous layout (`[batch, *, heads, head_dim]`). Supports up to and including 3 dimensions: * 1-D: `[batch, X, heads, head_dim]`, with masking arguments expecting tuples of size 1. * 2-D: `[batch, X, Y, heads, head_dim]`, with masking arguments expecting tuples of size 2. * 3-D: `[batch, X, Y, Z, heads, head_dim]`, with masking arguments expecting tuples of size 3. The dimensions here refer to the layout of tokens; that is the arrangement of tokens for each batch/head, or the `[X]`, `[X, Y]`, `[X, Y, Z]` part of the input shape. We refer to these as the "token layout shape". For now, it is always expected that Q, K, and V match in the sizes of those dimensions. Masking arguments, all of which can be set uniformly across all dimensions or per dimension, are: * `window_size`: determines the sliding window size. -1 is interpreted as the maximum window size. Must be either -1 or at least 2 and at most the token layout shape. For example, if inputs are `[batch, X, Y, Z, heads_{q,kv}, head_dim_{qk,v}]`, `window_size` must be either an integer == -1 or an integer <= `min(X, Y, Z)`, or a tuple of size 3 corresponding to the three dimensions / axes, where: * `window_size[0] == -1 or 2 <= window_size[0] <= X` * `window_size[1] == -1 or 2 <= window_size[1] <= Y` * `window_size[2] == -1 or 2 <= window_size[2] <= Z` When `window_size` is set to the maximum for any dimension, we're effectively performing self attention (no sparsity) along that dimension. Default is -1 (self attention). * `stride`: determines the step size of the sliding window. Only matters when the corresponding `window_size` is not -1 / maximum (self attention). Default is 1, indicating the smallest sliding window delay. Larger values trade off translational equivariance for potentially improved efficiency. Maximum value for `stride` along each dimension is the corresponding `window_size`. If `stride == window_size` along any dimension, it is equivalent to blocked / windowed attention (from works such as Swin Transformer, SAM, ViTDet, etc) along that dimension, meaning no overlap between windows. For more details, please refer to the GNA paper: https://arxiv.org/abs/2504.16922 * `dilation`: introduces gaps between tokens in a sliding window, similarly to dilated convolution. Default is 1, indicating no gaps. Maximum value is the largest positive integer that satisfies `window_size * dilation <= token_layout_shape` along that dimension. Higher dilation means more sparse and global context. Lower dilation means more locality. For more details, please refer to the DiNAT paper: https://arxiv.org/abs/2209.15001 * `is_causal`: per-dimension causal mask. Parameters: query (Tensor): 4-D, 5-D, or 6-D query tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads, head_dim]`) key (Tensor): 4-D, 5-D, or 6-D key tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim]`) value (Tensor): 4-D, 5-D, or 6-D value tensor, with heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim_v]`) window_size (tuple | int): Attention window (kernel) size / shape. If an integer, it will be repeated for all dimensions. For example `window_size=3`, when `len(token_layout_shape) == 3`, is interpreted as `window_size=(3, 3, 3)`. `-1`s are replaced with the corresponding `token_layout_shape`. Final window size must satisfy `2 <= window_size <= token_layout_shape`. Default is -1 (no sparsity). stride (tuple | int): Sliding window step size/shape. If an integer, it will be repeated for all dimensions. For example `stride=2`, when `len(token_layout_shape) == 3`, is interpreted as `stride=(2, 2, 2)`. Final stride must satisfy `1 <= stride <= window_size`. Default is 1. dilation (tuple | int): Dilation step size/shape. If an integer, it will be repeated for all dimensions. For example `dilation=4`, when `len(token_layout_shape) == 3`, is interpreted as `dilation=(4, 4, 4)`. Final dilation must satisfy `2 <= dilation * window_size <= token_layout_shape`. Default is 1. is_causal (tuple | bool): Toggle causal masking. If a boolean, it will be repeated for all dimensions. For example `is_causal=True`, when `len(token_layout_shape) == 3`, is interpreted as `is_causal=(True, True, True)`. Default is False. scale (float | None): Dot product scale (attention scale). Defaults to head_dim ** -0.5. Other Parameters: backend (str | None): Backend to run with. If unspecified (default), it will try to select the best available. return_lse (bool): Whether to return the logsumexp values. Default is False. backend_kwargs (dict | None): Key-value pair for passing arguments specific to the backend's multi-dim / sparse attention operator, if any. Only valid when a specific backend is selected (backend is not None). Returns: output (Tensor): 4-D, 5-D, or 6-D output tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads, head_dim_v]`). logsumexp (Tensor): logsumexp tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads, 1]`). Only returned when return_lse is True. Tr$)rArBrCrcsg|] }|dkr|qS)r@r>).0i)token_layout_shaper>r? sz/multi_dimensional_attention..rc3 |] \}}|vr|VqdSNr>rDrEstoken_layout_onesr>r? z.multi_dimensional_attention..c3rHrIr>)rDrEwrLr>r?rNrOc3rHrIr>rJrLr>r?rNrOc3rHrIr>)rDrEdrLr>r?rNrOc3rHrIr>)rDrEcrLr>r?rNrOcs|]}|dkVqdSNr>)rDxr>r>r?rNcsrSrTr>)rDrPr>r>r?rNrWr-r&zvThis Multi-Dimensional Attention problem has 1s in the token layout, which can be simplified from into .) rrrrArBrCrrr r!r"css|] \}}||kVqdSrIr>)rDrVrPr>r>r?rNscss|]}|VqdSrIr>)rDrRr>r>r?rNsr@zfThis Multi-Dimensional Attention problem is implementable with standard Attention: token_layout_shape=.NzIgnoring backend=z and backend args...)rrrr!r'r()rrrr r)r*r+ rrrrArBrCrrr!r")r rlenrangetuple enumerateallreshaper1r2r3multi_dimensional_attentionrzipanyflattenr9rZDontCarerMULTI_DIM_BACKEND_MAPr5r6)rrrrArBrCrrr r!r"num_dimsZtoken_layout_tZ window_size_tZstride_tZ dilation_tZ is_causal_tr;r<r=Zquery_1dZkey_1dZvalue_1dZ is_causal_1dr>)rMrFr?r`su ,,,         r`c Cs:|dkrtd|jdt||||||d||| d S)a Runs Spatio-Temporal Attention on unflattened QKV with the heads-last contiguous layout (`[batch, T, H, W, heads, head_dim]`). For now, it is always expected that Q, K, and V match in their shapes. Parameters: query (Tensor): 6-D query tensor, with the heads-last contiguous layout (`[batch, T, H, W, heads, head_dim]`) key (Tensor): 6-D key tensor, with the heads-last contiguous layout (`[batch, T, H, W, heads_kv, head_dim]`) value (Tensor): 6-D value tensor, with heads-last contiguous layout (`[batch, T, H, W, heads_kv, head_dim_v]`) window_size (tuple | int): Attention window (kernel) size / shape. If an integer, it will be repeated for all dimensions. For example `window_size=3` is interpreted as `window_size=(3, 3, 3)`. `-1`s are replaced with the corresponding value in `(T, H, W)`. Default is -1 (no sparsity). stride (tuple | int): Sliding window step size/shape. If an integer, it will be repeated for all dimensions. For example `stride=2` is interpreted as `stride=(2, 2, 2)`. Final stride must satisfy `1 <= stride <= window_size`. Default is 1. dilation (tuple | int): Dilation step size/shape. If an integer, it will be repeated for all dimensions. For example `dilation=4` is interpreted as `dilation=(4, 4, 4)`. Final dilation must satisfy `2 <= dilation * window_size <= (T, H, W)`. Default is 1. scale (float | None): Dot product scale (attention scale). Defaults to head_dim ** -0.5. Other Parameters: backend (str | None): Backend to run with. If unspecified (default), it will try to select the best available. return_lse (bool): Whether to return the logsumexp values. Default is False. backend_kwargs (dict | None): Key-value pair for passing arguments specific to the backend's multi-dim / sparse attention operator, if any. Only valid when a specific backend is selected (backend is not None). Returns: output (Tensor): 6-D output tensor, with the heads-last contiguous layout (`[batch, T, H, W, heads, head_dim_v]`). logsumexp (Tensor): logsumexp tensor, with the heads-last contiguous layout (`[batch, T, H, W, heads, 1]`). Only returned when return_lse is True. zjSpatio-Temporal Attention requires 6-D input tensors ([batch, T, H, W, heads, head_dim]), got query.shape=z).)TFFrY)r/r5r1r`) rrrrArBrCrr r!r"r>r>r?spatio_temporal_attentions$ ?rg) FNNNNNNNNNFN)r&r@r@FNNFN)r&r@r@NNFN)&__doc__r7rZ0cosmos_policy._src.imaginaire.attention.backendsrrZ.cosmos_policy._src.imaginaire.attention.checksrrrrr r Z-cosmos_policy._src.imaginaire.attention.cudnnr Z.cosmos_policy._src.imaginaire.attention.flash2r Z.cosmos_policy._src.imaginaire.attention.flash3r Z-cosmos_policy._src.imaginaire.attention.masksrZ.cosmos_policy._src.imaginaire.attention.nattenrrZ-cosmos_policy._src.imaginaire.attention.utilsrr2r4rdboolfloatintstrdictr\r9r`rgr>r>r>r?s              c    k