o ?߱iV@sddlZddlZddlZddlmZddlZddlmZddZ Gddde Z ddd Z d d Z d d ZGdddZddZdS)Nwraps)loggerc Cs,zzgtt}t|D]X}t|}t|}t|tr$|d}t d|d|d|vs6d|vr?WWt dSd|vrLWWt dSd|vrYWWt dSd|vrfWWt dSqWntj y}z t d |WYd }~n d }~wwWt d SWt d St w) z Retrieves the GPU architecture of the available GPUs. Returns: str: The GPU architecture, which can be "H100", "A100", or "Other". zutf-8zGPU z : Model: ZH100ZH200ZA100ZL40SB200zFailed to get GPU info: NOther) pynvmlnvmlInitnvmlDeviceGetCountrangenvmlDeviceGetHandleByIndexnvmlDeviceGetName isinstancebytesdecodeprint nvmlShutdown NVMLError) device_countihandle model_nameerrorrV/data/cameron/vidgen/cosmos-predict2.5/cosmos_predict2/_src/imaginaire/utils/device.pyget_gpu_architecturesD          rc@seZdZdZdS)GPUArchitectureNotSupportedzV Custom exception raised when the expected GPU architecture is not supported. N)__name__ __module__ __qualname____doc__rrrrr<src Csz/tttd}t|d|jddd|jddd|jdddWdStj yJ}zt d|WYd}~dSd}~ww)Nrz: /z MiB used (z MiB free)Failed to get GPU memory info: ) rrnvmlDeviceGetMemoryInfor logginginfousedtotalfreerr)strmeminforrrr print_gpu_memDs8 r+cCs.ttdtttdtdS)Nzgc()zempty cuda cache)r+rgccollectrrrrforce_gcOs  r.c Cshztttd}|jddddkWStjy3}ztd|WYd}~dSd}~ww)Nrr Pr")rrr#r r'rr)r*rrrrgpu0_has_80gb_or_lessYsr0csTeZdZeedZdeffdd Z de fddZ de efdd Z ZS) Device@ device_idxcstt||_dSN)super__init__rr r)selfr3 __class__rrr6es zDevice.__init__returncCs t|jSr4)rr r)r7rrrget_nameis zDevice.get_namecCsPd}t|jtjD] }d||}q dd|D}|ddt|DS)Nz{:064b}cSsg|]}t|qSr)int).0xrrr qsz+Device.get_cpu_affinity..cSsg|] \}}|dkr|qS)rr)r>rerrrr@ss)rnvmlDeviceGetCpuAffinityrr1_nvml_affinity_elementsformatreverse enumerate)r7Zaffinity_stringjZ affinity_listrrrget_cpu_affinityls zDevice.get_cpu_affinity)rrrmathceilos cpu_countrCr=r6r)r;listrH __classcell__rrr8rr1bs r1csddlfdd}|S)ar Decorator factory that wraps a function to execute within a specific torch device context. This decorator ensures that all tensor allocations and operations within the decorated function use the specified device by default. Args: device: The torch device to use (e.g., 'cuda', 'cuda:0', 'cpu', or torch.device object). Returns: A decorator function that wraps the target function with the specified device context. Example: @with_torch_device('cuda:0') def create_tensors(): x = torch.randn(10, 10) # Will be created on cuda:0 return x rNcstfdd}|S)Ncs<|i|WdS1swYdSr4)device)argskwargs)rOfntorchrrwrappers  $z5with_torch_device..decorator..wrapperr)rRrTrOrS)rRr decoratorsz$with_torch_device..decorator)rS)rOrVrrUrwith_torch_devicevsrWr4)r,rIrK functoolsrrlogururr$r Exceptionrr+r.r0r1rWrrrrs  #