o viO@sddlZddlmZddlmZddlmZGdddeZGdddeZgd Z gd Z ed e fd d Z ed e fddZ ed e fddZejfded eeeffddZejfded eeeffddZejfded eeeffddZdS)N)Enum) lru_cache)Dictc@eZdZdZdZdZdS) ClusterTypelocalZngcZslurmN)__name__ __module__ __qualname__LOCALZNGCZSLURMr r U/data/cameron/vidgen/cosmos-policy/cosmos_policy/_src/imaginaire/utils/cluster_env.pyrrc@r)ClusterEnvInfobasicZdetailedallN)rr r BASICDETAILEDALLr r r r rrr)Z NGC_JOB_IDNGC_ARRAY_SIZEZNGC_GPUS_PER_NODE)ZSLURM_JOB_USERZSLURM_JOB_PARTITIONZ SLURM_LOG_DIRZ SLURM_JOBIDZ SLURM_NNODESZSLURM_JOB_NAMEZSLURM_JOB_NODELISTZSLURMD_NODENAMEreturncCst ot S)z: Check if the code is running on a local machine. )is_ngcis_slurmr r r r is_local4srcC dtjvS)z. Check if the code is running on NGC. rosenvironr r r r r< rcCr)z0 Check if the code is running on SLURM. SLURM_JOB_IDrr r r r rDrrlevelcCsZ|tjkr ddtDS|tjkrddtjDS|tjkr&ddtjDStd|)a` Retrieves NVIDIA GPU Cloud (NGC) environment variables based on the specified detail level. The function filters environment variables to include only those relevant to NGC, differentiated by the detail level specified. Parameters: level (ClusterInfoLevel): The level of detail for the information returned. Defaults to ClusterInfoLevel.BASIC. Returns: dict: A dictionary containing the environment variables. If the level is BASIC, it includes only predefined key variables that are considered basic. If the level is DETAILED, it includes all environment variables that start with "NGC_". Raises: ValueError: If an unknown level is specified, an exception is raised indicating that the level is not recognized. cS"i|] }|tjvr|tj|qSr r.0kr r r a"zget_ngc_env..cS"i|] }|dr|tj|qS)ZNGC_ startswithrrr"r r r r%cr&cSi|]\}}||qSr r r#r$vr r r r%eUnknown level )rrNGC_ENV_BASIC_VARSrrrr ValueErrorr r r r get_ngc_envLs   r2cCs^|tjkr ddtDS|tjkrddtjDS|tjkr(ddtjDStd|)a' Retrieves SLURM environment variables based on the specified detail level. This function filters the environment variables related to the SLURM job scheduler environment based on the provided detail level of the cluster information. Parameters: level (ClusterEnvInfo): The detail level of the environment variables to retrieve. This can be BASIC, DETAILED, or ALL. Defaults to BASIC. Returns: Dict[str, str]: A dictionary containing the SLURM environment variables. The contents of the dictionary vary based on the level: - BASIC: Returns predefined key variables important for basic SLURM variables. - DETAILED: Includes all variables that start with "SLURM_". - ALL: Returns all environment variables available in the current session. Raises: ValueError: If an unknown level is specified, it raises an exception indicating that the level is not recognized. cSr!r rr"r r r r%r&z!get_slurm_env..cSr')ZSLURM_r(r"r r r r%r&cSr*r r r+r r r r%r-r.) rrSLURM_ENV_BASIC_VARSrrrritemsr0r1r r r get_slurm_envjs   r5cCsFdti}|tjkr|tj|S|t||t||S)an Retrieves a combination of environment variables from the cluster, merging information from both NVIDIA GPU Cloud (NGC) and SLURM environments based on the specified detail level. This function provides a unified dictionary of environment settings that are crucial for applications running in clustered computing environments. Parameters: level (ClusterEnvInfo): The level of detail for the environment variables to be retrieved. The level can be BASIC, DETAILED, or ALL. Defaults to BASIC. - BASIC: Gathers basic environment variables from both NGC and SLURM. - DETAILED: Includes more detailed information from both NGC and SLURM. - ALL: Combines all available environment variables from the system with NGC and SLURM specific ones. Returns: Dict[str, str]: A dictionary containing key-value pairs of environment variables. Initially includes the current working directory under the key 'PWD'. ZPWD)rgetcwdrrupdaterr2r5)r Zenv_infor r r get_cluster_envs  r8)renumr functoolsrtypingrrrr/r3boolrrrrstrr2r5r8r r r r s"      $