o Zi@sddlZddlZddlZddlZddlmZmZddZ Gdddej Z Gdd d ej Z Gd d d ej Z d d ZddZddZdddZGdddZddZdS)N)#discretized_gaussian_log_likelihood normal_klcCs|jttdt|jdS)z6 Take the mean over all non-batch dimensions. rdim)meanlistrangelenshape)tensorr V/data/cameron/vidgen/unified_video_action/./simple_uva/diffusion/gaussian_diffusion.py mean_flatsrc@s(eZdZdZeZeZeZdS) ModelMeanTypez2 Which type of output the model predicts. N) __name__ __module__ __qualname____doc__enumauto PREVIOUS_XSTART_XEPSILONr r r rrs  rc@s0eZdZdZeZeZeZeZ dS) ModelVarTypez What is used as the model's output variance. The LEARNED_RANGE option has been added to allow the model to predict values between FIXED_SMALL and FIXED_LARGE, making its job easier. N) rrrrrrLEARNED FIXED_SMALL FIXED_LARGE LEARNED_RANGEr r r rr!s  rc@s4eZdZeZeZeZeZddZ dS)LossTypecCs|tjkp |tjkSN)rKL RESCALED_KL)selfr r ris_vb6szLossType.is_vbN) rrrrrMSE RESCALED_MSEr!r"r$r r r rr.s rcCs@|tj|tjd}t||}tj|||tjd|d|<|S)Ndtype)nponesfloat64intlinspace) beta_startbeta_endnum_diffusion_timestepsZ warmup_fracbetasZ warmup_timer r r _warmup_beta:s   r2cCs|dkrtj|d|d|tjdd}nK|dkr$tj|||tjd}n<|dkr0t|||d}n0|dkr{s z)get_named_beta_schedule..zunknown beta schedule: )r:betas_for_alpha_barr9)Z schedule_namer0scaler r rget_named_beta_schedulefsrC+?cCsPg}t|D]}||}|d|}|td|||||qt|S)a# Create a beta schedule that discretizes the given alpha_t_bar function, which defines the cumulative product of (1-beta) over time from t = [0,1]. :param num_diffusion_timesteps: the number of betas to produce. :param alpha_bar: a lambda that takes an argument t from 0 to 1 and produces the cumulative product of (1-beta) up to that part of the diffusion process. :param max_beta: the maximum beta to use; use values lower than 1 to prevent singularities. r)r appendminr)array)r0 alpha_barZmax_betar1it1t2r r rrAs   " rAc@s.eZdZdZddZddZd/ddZd d Z d0d d ZddZ ddZ d/ddZ d/ddZ    d1ddZ       d2ddZ       d2ddZ    d3dd Z    d3d!d"Z       d4d#d$Z       d4d%d&Z d5d'd(Zd6d)d*Zd+d,Zd5d-d.ZdS)7GaussianDiffusionac Utilities for training and sampling diffusion models. Original ported from this codebase: https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/diffusion_utils_2.py#L42 :param betas: a 1-D numpy array of betas for each diffusion timestep, starting at T and going to 1. cCs||_||_||_tj|tjd}||_t|jdksJd|dk r+|dk s-Jt |jd|_ d|}tj |dd|_ td|j dd|_t|j ddd|_|jj|j fkscJt|j |_td|j |_td|j |_td|j |_td|j d|_|d|jd|j |_t|jdkrtt|jd|jddntg|_|t|jd|j |_d|jt|d|j |_dS) Nr'rzbetas must be 1-Drr8)axis)model_mean_typemodel_var_type loss_typer)rGr+r1r r allr, num_timestepscumprodalphas_cumprodrEalphas_cumprod_prevalphas_cumprod_nextsqrtsqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodloglog_one_minus_alphas_cumprodsqrt_recip_alphas_cumprodsqrt_recipm1_alphas_cumprodposterior_varianceposterior_log_variance_clippedposterior_mean_coef1posterior_mean_coef2)r#r1rPrQrRalphasr r r__init__s@"zGaussianDiffusion.__init__cCsBt|j||j|}td|j||j}t|j||j}|||fS)a Get the distribution q(x_t | x_0). :param x_start: the [N x C x ...] tensor of noiseless inputs. :param t: the number of diffusion steps (minus 1). Here, 0 means one step. :return: A tuple (mean, variance, log_variance), all of x_start's shape. r8)_extract_into_tensorrZr rVr])r#x_startr?rvariance log_variancer r rq_mean_variances  z!GaussianDiffusion.q_mean_varianceNcCsJ|dur t|}|j|jksJt|j||j|t|j||j|S)ak Diffuse the data for a given number of diffusion steps. In other words, sample from q(x_t | x_0). :param x_start: the initial data batch. :param t: the number of diffusion steps (minus 1). Here, 0 means one step. :param noise: if specified, the split-out normal noise. :return: A noisy version of x_start. N)th randn_liker rfrZr[)r#rgr?noiser r rq_samples zGaussianDiffusion.q_samplecCs|j|jksJt|j||j|t|j||j|}t|j||j}t|j||j}|jd|jdkrH|jdkrH|jdksKJJ|||fS)zm Compute the mean and variance of the diffusion posterior: q(x_{t-1} | x_t, x_0) r)r rfrbrcr`ra)r#rgx_tr?Zposterior_meanr`rar r rq_posterior_mean_variances"  z+GaussianDiffusion.q_posterior_mean_varianceTc s|duri}|jdd\}}|j|fksJ|||fi|} t| tr*| \} } nd} |jtjtjfvr{| j||dg|jddRksHJtj| |dd\} } t |j ||j} t t |j ||j} | dd}|| d|| }t|}n.process_xstart)ror?epsrgror?)rrhri pred_xstartextra)r isinstancetuplerQrrrrksplitrfrar)r\r1exprrEr`rrPrr_predict_xstart_from_epsrp)r#modelrrr?rtru model_kwargsBC model_outputrzmodel_var_valuesZmin_logZmax_logfracZmodel_log_varianceZmodel_variancervryZ model_mean_r rsrp_mean_variances^  &         .z!GaussianDiffusion.p_mean_variancecCs8|j|jksJt|j||j|t|j||j|Sr )r rfr^r_)r#ror?rwr r rr\s z*GaussianDiffusion._predict_xstart_from_epscCs(t|j||j||t|j||jSr )rfr^r r_)r#ror?ryr r r_predict_eps_from_xstartcs z*GaussianDiffusion._predict_eps_from_xstartcCs2|||fi|}|d|d|}|S)aZ Compute the mean for the previous step, given a function cond_fn that computes the gradient of a conditional log probability with respect to x. In particular, cond_fn computes grad(log(p(y|x))), and we want to condition on y. This uses the conditioning strategy from Sohl-Dickstein et al. (2015). rrh)float)r#cond_fn p_mean_varrrr?rgradientZnew_meanr r rcondition_meanisz GaussianDiffusion.condition_meanc Cst|j||j}||||d}|d||||fi|}|}|||||d<|j|d||d\|d<} } |S)a1 Compute what the p_mean_variance output would have been, should the model's score function be conditioned by cond_fn. See condition_mean() for details on cond_fn. Unlike condition_mean(), this instead uses the conditioning strategy from Song et al (2020). ryrrxr)rfrVr rrYcopyrrp) r#rrrrr?rrHrwoutrr r rcondition_scorews" z!GaussianDiffusion.condition_scorer8c Cs|j||||||d} t|} |dkjdgdgt|jdR} |dur5|j|| |||d| d<| d| td| d | |} | | d d S) a Sample x_{t-1} from the model at the given timestep. :param model: the model to sample from. :param x: the current tensor at x_{t-1}. :param t: the value of t, starting at 0 for the first diffusion step. :param clip_denoised: if True, clip the x_start prediction to [-1, 1]. :param denoised_fn: if not None, a function which applies to the x_start prediction before it is used to sample. :param cond_fn: if not None, this is a gradient function that acts similarly to the model. :param model_kwargs: if not None, a dict of extra keyword arguments to pass to the model. This can be used for conditioning. :param temperature: temperature scaling during Diff Loss sampling. :return: a dict containing the following keys: - 'sample': a random sample from the model. - 'pred_xstart': a prediction of x_0. rtrurrrNrNrrr4rirysamplery) rrkrlrviewr r rr~) r#rrrr?rtrurr temperaturerrm nonzero_maskrr r rp_samples( (  zGaussianDiffusion.p_sampleFc C4d} |j||||||||| | d D]} | } q| dS)aO Generate samples from the model. :param model: the model module. :param shape: the shape of the samples, (N, C, H, W). :param noise: if specified, the noise from the encoder to sample. Should be of the same shape as `shape`. :param clip_denoised: if True, clip x_start predictions to [-1, 1]. :param denoised_fn: if not None, a function which applies to the x_start prediction before it is used to sample. :param cond_fn: if not None, this is a gradient function that acts similarly to the model. :param model_kwargs: if not None, a dict of extra keyword arguments to pass to the model. This can be used for conditioning. :param device: if specified, the device to create the samples on. If not specified, use a model parameter's device. :param progress: if True, show a tqdm progress bar. :param temperature: temperature scaling during Diff Loss sampling. :return: a non-differentiable batch of samples. N)rmrtrurrdeviceprogressrr)p_sample_loop_progressive) r#rr rmrtrurrrrrfinalrr r r p_sample_loops   zGaussianDiffusion.p_sample_loopc  ct|ttfs J|dur|} ntj|} tt|jddd} | r0ddlm } | | } | D]6}t |g|d}t |j || |||||| d}|V|d} Wdn1scwYq2dS)a Generate samples from the model and yield intermediate samples from each timestep of diffusion. Arguments are the same as p_sample_loop(). Returns a generator over dicts, where each dict is the return value of p_sample(). NrNrtqdm)rtrurrrr) r{r|rrkrandncudar rT tqdm.autorr no_gradr)r#rr rmrtrurrrrrimgindicesrrIr?rr r rrs6   z+GaussianDiffusion.p_sample_loop_progressiverOc Cs|j||||||d} |dur|j|| |||d} |||| d} t|j||j} t|j||j} |td| d| td| | } t |}| dt| td| | d| }|dk j dgdgt |jdR}||| |}|| dd S) z] Sample x_{t-1} from the model using DDIM. Same usage as p_sample(). rNrryrr5rrNr) rrrrfrVr rWrkrYrlrrr )r#rrrr?rtrurretarrwrHZalpha_bar_prevsigmarm mean_predrrr r r ddim_samples8 (zGaussianDiffusion.ddim_samplec Cs|dksJd|j||||||d} |dur!|j|| |||d} t|j||j|| dt|j||j} t|j||j} | dt| td| | } | | ddS) zG Sample x_{t+1} from the model using DDIM reverse ODE. rOz'Reverse ODE only for deterministic pathrNrryrr) rrrfr^r r_rXrkrY) r#rrrr?rtrurrrrrwZalpha_bar_nextrr r rddim_reverse_samplePs,z%GaussianDiffusion.ddim_reverse_samplec Cr)zd Generate samples from the model using DDIM. Same usage as p_sample_loop(). N)rmrtrurrrrrr)ddim_sample_loop_progressive) r#rr rmrtrurrrrrrrr r rddim_sample_loopys  z"GaussianDiffusion.ddim_sample_loopc  cr)z Use DDIM to sample from the model and yield intermediate samples from each timestep of DDIM. Same usage as p_sample_loop_progressive(). NrNrr)rtrurrrr) r{r|rrkrrr rTrrr rr)r#rr rmrtrurrrrrrrrrIr?rr r rrs6   z.GaussianDiffusion.ddim_sample_loop_progressivecCs|j|||d\}}} |j|||||d} t|| | d| d} t| td} t|| dd| dd } | j|jks?Jt| td} t |dk| | } | | d d S) ag Get a term for the variational lower-bound. The resulting units are bits (rather than nats, as one might expect). This allows for comparison to other papers. :return: a dict with the following keys: - 'output': a shape [N] tensor of NLLs or KLs. - 'pred_xstart': the x_0 predictions. rx)rtrrri@r4)ZmeansZ log_scalesrry)outputry) rprrrr)r\rr rkwhere)r#rrgror?rtrZ true_meanrZtrue_log_variance_clippedrklZ decoder_nllrr r r _vb_terms_bpds"   zGaussianDiffusion._vb_terms_bpdcCs|duri}|durt|}|j|||d}i}|jtjks%|jtjkrE|j||||d|dd|d<|jtjkrC|d|j9<|S|jtj ksR|jtj kr|||fi|}|j t j t jfvr|jdd\} } |j| | dg|jddRksJtj|| dd \}} tj|| gdd } |j| d d d |||dd d|d<|jtj kr|d|jd9<tj|j|||ddtj|tj|i|j} |j| jkr|jksJJt| |d|d<d|vr|d|d|d<|S|d|d<|St|j)a[ Compute training losses for a single timestep. :param model: the model to evaluate loss on. :param x_start: the [N x C x ...] tensor of inputs. :param t: a batch of timestep indices. :param model_kwargs: if not None, a dict of extra keyword arguments to pass to the model. This can be used for conditioning. :param noise: if specified, the specific Gaussian noise to try to remove. :return: a dict with the key "loss" containing a tensor of shape [N]. Some mean or variance settings may also have other keys. N)rmF)rrgror?rtrrlossr5rr)rcWs|Sr r )rargsr r rr@sz3GaussianDiffusion.training_losses..)rrgror?rtvbg@@rxrmse)rkrlrnrRrr!r"rrTr%r&rQrrrr r}catdetachrrrprrrPrr9)r#rrgr?rrmrotermsrrrrZ frozen_outtargetr r rtraining_lossessv   *&  "  z!GaussianDiffusion.training_lossescCsZ|jd}tj|jdg||jd}|||\}}}t||ddd}t|t dS)a; Get the prior KL term for the variational lower-bound, measured in bits-per-dim. This term can't be optimized, as it only depends on the encoder. :param x_start: the [N x C x ...] tensor of inputs. :return: a batch of [N] KL values (in bits), one per batch element. rrrrO)Zmean1Zlogvar1Zmean2Zlogvar2r) r rkr rTrrjrrr)r\)r#rg batch_sizer?Zqt_meanrZqt_log_varianceZkl_priorr r r _prior_bpd4s zGaussianDiffusion._prior_bpdc CsJ|j}|jd}g}g}g} tt|jdddD]`} tj| g||d} t|} |j|| | d} t |j ||| | ||d}Wdn1sMwY| |d| t |d|d | | | |d}| t || d qtj|d d }tj|d d }tj| d d } ||}|jd d |}||||| d S) as Compute the entire variational lower-bound, measured in bits-per-dim, as well as other related quantities. :param model: the model to evaluate loss on. :param x_start: the [N x C x ...] tensor of inputs. :param clip_denoised: if True, clip denoised samples. :param model_kwargs: if not None, a dict of extra keyword arguments to pass to the model. This can be used for conditioning. :return: a dict containing the following keys: - total_bpd: the total variational lower-bound, per batch element. - prior_bpd: the prior term in the lower-bound. - vb: an [N x T] tensor of terms in the lower-bound. - xstart_mse: an [N x T] tensor of x_0 MSEs for each timestep. - mse: an [N x T] tensor of epsilon MSEs for each timestep. rNrNr)rgr?rm)rgror?rtrrryr5rr) total_bpd prior_bpdr xstart_mser)rr rr rTrkr rlrnrrrErrstackrsum)r#rrgrtrrrrrrr?Zt_batchrmrorrwrrr r r calc_bpd_loopDsD     zGaussianDiffusion.calc_bpd_loopr )TNN)TNNNr8)NTNNNNFr8)TNNNrO)NTNNNNFrO)TN)NN)rrrrrerjrnrprrrrrrrrrrrrrrrrr r r rrLs/  X   7 4 6 6 - % 0 !JrLcCs\t|j|jd|}t|jt|kr$|d}t|jt|ks|tj||jdS)a Extract values from a 1-D numpy array for a batch of indices. :param arr: the 1-D numpy array. :param timesteps: a tensor of indices into the array to extract. :param broadcast_shape: a larger shape of K dimensions with the batch dimension equal to the length of timesteps. :return: a tensor of shape [batch_size, 1, ...] where the shape has K dims. r).N)rk from_numpytorrr r zeros)arrZ timestepsbroadcast_shaperesr r rrf|s  rf)rD)r<numpyr)torchrkrZdiffusion_utilsrrrEnumrrrr2r:rCrArLrfr r r rs(    #  m