o viH@srddlmZddlmZddlmZmZddlZddlm Z eGdddZ Gddde j Z Gd d d Z dS) ) annotations) dataclass)AnyOptionalNc@s>eZdZUdZded<dZded<dZded<dZded<d S) KVCacheConfigFbool run_with_kvstore_kvrint current_idxrecompute_cross_attn_kvN)__name__ __module__ __qualname__r__annotations__r r r rrP/data/cameron/vidgen/cosmos-policy/cosmos_policy/_src/predict2/utils/kv_cache.pyrs   rcsFeZdZdZddfdd Zddd d ZdddZddddZZS) AttentionOpWithKVCacheaA thin wrapper that adds K/V caching to an existing attention op. This wrapper expects the wrapped op to accept (q, k, v, attn_mask=None) and return attention outputs with heads already flattened on the last dim. Cache semantics: - Cache entries are stored as per-chunk tensors, where each chunk corresponds to one latent frame composed of HxW tokens (after patchify). - The `max_cache_size` capacity therefore refers to the number of latent frames (chunks), NOT the number of individual tokens. - When `max_cache_size` is None, the cache grows without an automatic rolling window; otherwise, it acts as a rolling window of at most `max_cache_size` frames. Upon overflow, the oldest frames are dropped. Nattn_opnn.Module | Anymax_cache_size Optional[int]cs,t||_|j|dd|_d|_dS)auInitialize the KV cache wrapper. Args: attn_op: The underlying attention operation (q, k, v[, attn_mask]) -> out. max_cache_size: Optional capacity measured in number of latent frames (chunks). Each chunk is a single frame worth of HxW tokens. If None, the cache does not enforce a rolling capacity. )rN)super__init__rreset_kv_cachepgstream)selfrr __class__rrr1s   zAttentionOpWithKVCache.__init__returnNonecCs0d|_dg|pd|_dg|pd|_||_dS)aReset/initialize the KV caches. Args: max_cache_size: Optional capacity measured in number of latent frames (chunks). Each chunk is a single frame worth of HxW tokens. If None, the cache does not enforce a rolling capacity. rNi) start_idxk_cachev_cacher)rrrrrr@s  z%AttentionOpWithKVCache.reset_kv_cacheq torch.Tensorkv kv_cache_cfgrc Ks|jdur |jdusJd|jr$t|j}||j|<||j|<|jri|jdkri|j|j|j}|j|j|j}tdd|DrIJtdd|DrTJt j ||gdd} t j ||gdd} n|} |} |jr|j durt dt|j|j |_|j || | fi|S)Nz9KV cache is not initialized. Call reset_kv_cache() first.rcs|]}|duVqdSNr.0xrrr ez1AttentionOpWithKVCache.forward..csr*r+rr,rrrr/fr0)dim)r#r$r r r detachrr"anytorchcatrmaxr) rr%r'r(r)kwargsindexZ history_kZ history_vZk_outZv_outrrrforwardNs&  zAttentionOpWithKVCache.forwardp2p cp_comm_typestrcCs|jj||||ddS)N)r<)rset_context_parallel_group)r process_groupranksrr<rrrr>tsz1AttentionOpWithKVCache.set_context_parallel_groupr+)rrrr)rrr r!) r%r&r'r&r(r&r)rr r&)r;)r<r=) r rr__doc__rrr:r> __classcell__rrrrr!s   &rc@s0eZdZdZddd d Zdd d ZdddZdS) VideoSeqPoszFlattened 3D grid positions for a video clip. Stores flattened t/h/w indices of length L = T*H*W to enable constructing RoPE frequencies aligned with global positions across sequential chunks. NTr HWr r!c Cs||_||_||_|dur/|dur/|dur/|jtjd|_|jtjd|_|jtjd|_dStj r=t dtj nt d}tj |j|tjd}tj |j|tjd} tj |j|tjd} tj|| | dd\}}}|d|_|d|_|d|_dS)N)dtypecudacpu)devicerGij)indexing)rDrErFtor5longpos_hpos_wpos_trH is_availablerJcurrent_devicearangemeshgridreshape) rrDrErFrPrQrRrJthwrrrrs &  zVideoSeqPos.__init__cCst|jSr+)r rPnumel)rrrrsizeszVideoSeqPos.sizet_idx 'VideoSeqPos'c Cst|}|dks|t|jkrtd|d|jdt|jt|j}||}||}tdt|jt|j|j|||j|||j||dS)zReturn a `VideoSeqPos` view for a single frame at absolute index `t_idx`. This is useful for streaming / KV-cache inference where the model is run on one frame at a time but RoPE positions must reflect global video indices. rzt_idx out of range: z (valid: [0, z))r1)rDrErFrPrQrR) r rD IndexErrorrErFrCrPrQrR)rr]Ztokens_per_framestartendrrrframes   zVideoSeqPos.frame)NNN)rDr rEr rFr r r!)r r )r]r r r^)r rrrArr\rbrrrrrCxs   rC) __future__r dataclassesrtypingrrr5torch.nnnnrModulerrCrrrrs   W