U Mh @sddlZddlmZmZmZmZmZmZmZddl m Z ddl Z ddl m Z ddl mZmZmZee j ee j fZdddgZd d Zedeeeeeee j d ddZe deddeeeeeee j d ddZedeeeeddddZdS)N)UnionIterableListDictTupleOptionalcast) deprecated)Tensor)"_group_tensors_by_device_and_dtype_has_foreach_support_device_has_foreach_supportclip_grad_norm_clip_grad_normclip_grad_value_csfdd}t||S)z This wrapper is needed to avoid a circular import when using @torch.no_grad on the exposed functions clip_grad_norm_ and clip_grad_value_ themselves. c s*t||W5QRSQRXdSN)torchZno_grad)argskwargsfuncJ/var/www/html/venv/lib/python3.8/site-packages/torch/nn/utils/clip_grad.py_no_grad_wrappers z"_no_grad.._no_grad_wrapper) functoolsupdate_wrapper)rrrrr_no_grads  r@F) parametersmax_norm norm_typeerror_if_nonfiniteforeachreturncst|tjr|g}dd|D}t|}tt|dkrFtdS|djt|g}g}|D]t\\}} \\} } |dkrt | |s|rt |r| t | qf|rt d|jdqf| fdd| Dqftjtfd d|D} |r*t| | r*t d d || d } tj| d d} |D]\\}} \\} } |dkrtt | |s|rt |rt| | |n:|rt d|jdn | |}| D]}||qqL| S)aGClip the gradient norm of an iterable of parameters. The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float): max norm of the gradients norm_type (float): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of the gradients from :attr:`parameters` is ``nan``, ``inf``, or ``-inf``. Default: False (will switch to True in the future) foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: Total norm of the parameter gradients (viewed as a single vector). cSsg|]}|jdk r|jqSrgrad.0prrr 5s z#clip_grad_norm_..rgN:foreach=True was passed, but can't use the foreach API on  tensorscsg|]}tj|qSr)rlinalg vector_norm)r'g)r rrr)Hscsg|]}|qSr)to)r'Znorm) first_devicerrr)JszThe total norm of order z for gradients from `parameters` is non-finite, so it cannot be clipped. To disable this error and scale the gradients by the non-finite norm anyway, set `error_if_nonfinite=False`gư>g?)max) isinstancerr floatlenZtensordevicer itemsr r extendZ _foreach_norm RuntimeErrortyper,r-stack logical_orisnanisinfclampZ _foreach_mul_r/Zmul_)rrr r!r"grads grouped_gradsZnormsr5_Z device_gradsZ total_normZ clip_coefZclip_coef_clampedZclip_coef_clamped_devicer.r)r0r rrs^    "   z_`torch.nn.utils.clip_grad_norm` is now deprecated in favor of `torch.nn.utils.clip_grad_norm_`.)categorycCst|||||S)zClip the gradient norm of an iterable of parameters. .. warning:: This method is now deprecated in favor of :func:`torch.nn.utils.clip_grad_norm_`. )r)rrr r!r"rrrrgs)r clip_valuer"r#cCst|tjr|g}t|}dd|D}t|g}|D]\\}}\\}}|dkrjtttt||dsv|rt |rt ttt|| t ttt||q:|rt d|j dq:|D]}tt|j| |dqq:dS)aClip the gradients of an iterable of parameters at specified value. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized clip_value (float): maximum allowed value of the gradients. The gradients are clipped in the range :math:`\left[\text{-clip\_value}, \text{clip\_value}\right]` foreach (bool): use the faster foreach-based implementation If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` cSsg|]}|jdk r|jqSrr$r&rrrr)s z$clip_grad_value_..N)r5r*r+)minr1)r2rr r3r r6r rrr Z_foreach_clamp_min_Z_foreach_clamp_max_r8r9Zclamp_)rrCr"r?r@r5rAr%rrrrxs(  )rFN)rFN)N)rtypingrrrrrrrtyping_extensionsr rr Ztorch.utils._foreach_utilsr r r Z_tensor_or_tensors__all__rr3boolr FutureWarningrrrrrrsJ$    M