U yh@sHddlmZmZddlZddlmZd eeedddZdd ZdS) )ListOptionalN)map_debug_info)outputs_with_grads_idxsc s|dk r,fdd|Dfdd|Dzhggfddtjjdg}|D](}t|tjr||jqh|dqhWnPtk r}z2dtd td t|d }t ||W5d}~XYnX|S) a This is a helper function to: 1. compute the gradients for the stage inputs, and 2. accumulate gradients for the stage module's parameters. Given the input value(s) and the corresponding gradient for the output value(s), compute and accumulate gradients for all parameter values (leaves in the autograd trace) as well as return a list of the gradients for the input values Ncsg|] }|qSr.0i) stage_outputrX/var/www/html/venv/lib/python3.8/site-packages/torch/distributed/pipelining/_backward.py sz"stage_backward..csg|] }|qSrrr) output_gradsrr r scs@t|tjr\|js |jdkr dSt|tjtdfsFtdt|||nt|tt fr|dkrvdSt|tt fstdt|dt|t |t |kstt ||D]\}}||qnht|t r<|dkrdSt|t stt |t |kst|D]}||||q ndS)Nz)Expected Tensor or None gradient but got z!grad_value expected to have type z but got ) isinstancetorchTensorZ requires_gradZgrad_fntypeAssertionErrorappendtuplelistlenzipdictsetkeys)Z output_valZgrad_valovgvk)extract_tensors_with_gradsoutput_grad_tensorsstage_output_tensorsrr r%s>       z2stage_backward..extract_tensors_with_grads)Z grad_tensorsz= Failed to run stage backward: Stage output: z Output gradient: z Input: z ) rZautogradZbackwardrrrZgrad Exceptionr RuntimeError)r rZ input_valuesrZ grad_inputsvaleZexc_msgr)rr rr r!r stage_backward s8    r&cCs(|dkr |S|dkr|St||SdS)z] Coalesce two values, even if one of them is null, returning the non-null value. N)radd)lhsrhsrrr _null_coalesce_accumulatels r*)N) typingrrr_debugrintr&r*rrrr s  b