o vi+@spdZddlmZddlmZddlmZddlmZddl m Z ddl m Z m Z ddlmZdd lmZee eed Zd e iZ d%d ededededede dBdededefddZdedeefddZ  d&dedededede dBded edBdededBfddZ d%d edededededef dd Zdedeefd!d"Z d'dededed edBdef d#d$ZdS)(zl Imaginaire4 Attention Subpackage: Unified implementation for all Attention implementations. Frontend APIs )Tensor)cudnn_attention_check)flash2_attention_check)flash3_attention_check) CausalType)natten_attention_check natten_multi_dim_attention_check) get_arch_tag)safe_log)cudnnnattenflash2flash3r Fbackendquerykeyvalue is_causal causal_typeN is_varlen raise_errorreturnc CsB|durtd|tvrtd|dt||||||||dS)a Input validation function a specified backend. Runs the common and backend-specific checks. Returns False if any checks fail, otherwise True. Parameters: backend (str): selected backend. query (Tensor): 4-D query tensor, with the heads-last contiguous layout (`[batch, seqlen, heads, head_dim]`). key (Tensor): 4-D key tensor, with the heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim]`). value (Tensor): 4-D value tensor, with heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim_v]`). is_causal (bool): whether or not causal masking is enabled. causal_type (CausalType): causal masking mode. Choices: `CausalType.TopLeft`, `CausalType.BottomRight`. Required when `is_causal = True`. is_varlen (bool): whether or not a variable length (varlen) use case. Must be inferred beforehand based on arguments such as seqlens_{Q,KV} or cumulative_seqlen_{Q,KV} being passed. raise_error (bool): whether to raise an error if any checks fail or no backend is selected, instead of just returning False. Default is False. Returns: success (bool): whether use case is compatible with the backend. N2Cannot pass None backend to is_backend_compatible.Unrecognized backend name .)rrrrrrr) ValueErrorBACKEND_CHECK_MAPrrrrrrrrrV/data/cameron/vidgen/cosmos-policy/cosmos_policy/_src/imaginaire/attention/backends.pyis_backend_compatible0s*r arch_tagcCsT|dkrtd|dgS|dkrgdS|dvrgdS|dkr'gd Sd gS) a Returns list of supported backends according to arch tag (attention.utils.get_arch_tag). Backends are ordered based on their known performance levels, so that the best-performing compatible backend is selected. Parameters: arch_tag (int): Arch tag for the current CUDA device. Example: 80 for A100, 90 for H100. Returns: backend_list (list[str]): a list of backend names (string). Empty if device is not supported. KzAMinimum architecture supported for Attention is 75, got arch_tag=rZ)rr r r )dg)r r r P)r r r r logdebugr!rrrget_backend_listksr+Tc Cst|durt||||||||dr|SdSt|j}t|} | D]}t|||||||ddr1|Sq|s6dStd)a Selects a compatible backend, unless one is already selected, which runs its corresponding checks. Parameters: query (Tensor): 4-D query tensor, with the heads-last contiguous layout (`[batch, seqlen, heads, head_dim]`). key (Tensor): 4-D key tensor, with the heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim]`). value (Tensor): 4-D value tensor, with heads-last contiguous layout (`[batch, seqlen_kv, heads_kv, head_dim_v]`). is_causal (bool): whether or not causal masking is enabled. causal_type (CausalType): causal masking mode. Choices: `CausalType.TopLeft`, `CausalType.BottomRight`. Required when `is_causal = True`. is_varlen (bool): whether or not a variable length (varlen) use case. Must be inferred beforehand based on arguments such as seqlens_{Q,KV} or cumulative_seqlen_{Q,KV} being passed. backend (str | None): selected backend, if any. raise_error (bool): whether to raise an error if any checks fail or no backend is selected, instead of just returning False. Default is **True**. Returns: backend (str | None): selected backend, or None if no backends are compatible. NrFzvCould not find a compatible Attention backend for this use case / device. Try running with debug logs to find out why.)r r devicer+r) rrrrrrrrr! backend_listrrrchoose_backendsB*    r.cCs<|durtd|tvrtd|dt|||||dS)a Input validation function a specified multi-dimensional backend. Runs the common and backend-specific checks. Returns False if any checks fail, otherwise True. Parameters: backend (str): selected backend. query (Tensor): 4-D, 5-D, or 6-D query tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads, head_dim]`). key (Tensor): 4-D, 5-D, or 6-D key tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim]`). value (Tensor): 4-D, 5-D, or 6-D value tensor, with heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim_v]`). raise_error (bool): whether to raise an error if any checks fail or no backend is selected, instead of just returning False. Default is False. Returns: success (bool): whether use case is compatible with the backend. Nrrr)rrrr)rBACKEND_MULTI_DIM_CHECK_MAPrrrrrrrris_multi_dim_backend_compatiblesr1cCs$|dkrtd|dgSdgS)a Returns list of supported multi-dimensional backends according to arch tag (attention.utils.get_arch_tag). Backends are ordered based on their known performance levels, so that the best-performing compatible backend is selected. Parameters: arch_tag (int): Arch tag for the current CUDA device. Example: 80 for A100, 90 for H100. Returns: backend_list (list[str]): a list of backend names (string). Empty if device is not supported. r"zSMinimum architecture supported for Multi-Dimensional Attention is 75, got arch_tag=rr r'r*rrrget_multi_dim_backend_listsr2cCs`|durt||||ddsJ|St|j}t|}|D]}t||||ddr+|Sqtd)a Selects a compatible multi-dimensional backend, unless one is already selected, which runs its corresponding checks. Parameters: query (Tensor): 4-D, 5-D, or 6-D query tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads, head_dim]`). key (Tensor): 4-D, 5-D, or 6-D key tensor, with the heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim]`). value (Tensor): 4-D, 5-D, or 6-D value tensor, with heads-last contiguous layout (`[batch, *token_layout_shape, heads_kv, head_dim_v]`). backend (str | None): selected backend, if any. Returns: backend (str): selected backend. NTr0FzCould not find a compatible Multi-Dimensional Attention backend for this use case / device. Try running with debug logs to find out why.)r1r r,r2r)rrrrr!r-rrrchoose_multi_dim_backend)s0   r3)F)NT)N)__doc__torchrZ4cosmos_policy._src.imaginaire.attention.cudnn.checksrZ5cosmos_policy._src.imaginaire.attention.flash2.checksrZ5cosmos_policy._src.imaginaire.attention.flash3.checksr-cosmos_policy._src.imaginaire.attention.masksrZ5cosmos_policy._src.imaginaire.attention.natten.checksrr-cosmos_policy._src.imaginaire.attention.utilsr r r(rr/strboolr intlistr+r.r1r2r3rrrrs         ;3 U ,