U yhH-@sddlZddlmZddddddd d gZGd ddejjZGd ddejjZGd ddejjZGdddejj Z Gdddejj Z Gdddejj Z Gdd d ej jj jZGdd d ejjZdS)N)warnReLU6 HardswishELU LeakyReLUSigmoidSoftmaxMultiheadAttentionPReLUcs@eZdZdZd fdd ZddZddZed d d ZZ S) raApplies the element-wise function: :math:`\text{ReLU6}(x) = \min(\max(x_0, x), q(6))`, where :math:`x_0` is the zero_point, and :math:`q(6)` is the quantized representation of number 6. Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input .. image:: ../scripts/activation_images/ReLU6.png Examples:: >>> m = nn.quantized.ReLU6() >>> input = torch.randn(2) >>> # xdoctest: +SKIP >>> input = torch.quantize_per_tensor(input, 1.0, 0, dtype=torch.qint32) >>> output = m(input) Fcst|||_dSN)super__init__inplace)selfr __class__Z/var/www/html/venv/lib/python3.8/site-packages/torch/ao/nn/quantized/modules/activation.pyr 's zReLU6.__init__cCstjj||jSr )torchops quantizedZrelu6rrinputrrrforward+sz ReLU6.forwardcCsdS)NZQuantizedReLU6rrrrr _get_name.szReLU6._get_namecCs t|jSr )rr)moduse_precomputed_fake_quantrrr from_float1szReLU6.from_float)F)F) __name__ __module__ __qualname____doc__r rr staticmethodr __classcell__rrrrrs csLeZdZdZdfdd ZddZddZedd d Ze d d Z Z S)rzThis is the quantized version of :class:`~torch.nn.Hardswish`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor NcsD||d}t|dtj|f||dtj|f|dSN)devicedtypescale zero_pointr r Zregister_bufferrZtensor)rr(r)r&r'factory_kwargsrrrr <s  zHardswish.__init__cCstjj||j|jSr )rrrZ hardswishr(r)rrrrrBszHardswish.forwardcCsdS)NZQuantizedHardswishrrrrrrEszHardswish._get_nameFcCs |j\}}tt|t|Sr )activation_post_processcalculate_qparamsrfloatintrrr(r)rrrrHszHardswish.from_floatcCs|t|t|Sr )r.r/clsrr(r)rrrfrom_referenceMszHardswish.from_reference)NN)F rr r!r"r rrr#r classmethodr3r$rrrrr5s csLeZdZdZdfdd ZddZddZedd d Ze d d Z Z S)rzThis is the quantized equivalent of :class:`~torch.nn.ELU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor alpha: the alpha constant ?cst|||_||_dSr )r r r(r))rr(r)alpharrrr Ys z ELU.__init__cCstjjjj||j|j|jSr ) raonnr functionalZelur(r)r7rrrrr^s  z ELU.forwardcCsdS)NZ QuantizedELUrrrrrrbsz ELU._get_nameFcCs$|j\}}tt|t||jSr )r,r-rr.r/r7r0rrrreszELU.from_floatcCs|t|t||jSr )r.r/r7r1rrrr3jszELU.from_reference)r6)Fr4rrrrrQs csZeZdZdZdeeeeddfdd Zdd Zd d Z e dd d Z e ddZ Z S)raThis is the quantized equivalent of :class:`~torch.nn.LeakyReLU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor negative_slope: Controls the angle of the negative slope. Default: 1e-2 {Gz?FN)r(r)negative_sloperreturncsH||d}t|||dtj|f||dtj|f|dSr%r*)rr(r)r<rr&r'r+rrrr vs zLeakyReLU.__init__cCstjj||j|j|j|jSr )rrrZ leaky_relur<rr(r)rrrrr}szLeakyReLU.forwardcCsdS)NZQuantizedLeakyReLUrrrrrrszLeakyReLU._get_namecCs(|j\}}|t|t||j|jSr )r,r-r.r/r<r)r2rrr(r)rrrrszLeakyReLU.from_floatcCs|t|t||j|jSr )r.r/r<rr1rrrr3szLeakyReLU.from_reference)r;FNN)F)rr r!r"r.r/boolr rrr5rr3r$rrrrrns cs>eZdZdZeedfdd ZddZed dd Z Z S) rzThis is the quantized equivalent of :class:`~torch.nn.Sigmoid`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor ) output_scaleoutput_zero_pointcst||_||_dSr )r r r?r@)rr?r@rrrr s zSigmoid.__init__cCstjj||j|jSr )rrrZsigmoidr?r@rrrrrszSigmoid.forwardFcCs |j\}}|t|t|Sr )r,r-r.r/)r2rrr?r@rrrrszSigmoid.from_float)F) rr r!r"r.r/r rr5rr$rrrrrs csLeZdZdZdfdd ZddZd d Zedd d Ze ddZ Z S)ra,This is the quantized version of :class:`~torch.nn.Softmax`. Args: dim: A dimension along which Softmax will be computed (so every slice along dim will sum to 1). scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor Nr6rcs t||_||_||_dSr )r r dimr(r))rrAr(r)rrrr s zSoftmax.__init__cCs@|j}|dkr(d}tjjd||}tjj|||j|j S)Nsoftmax) rArr9r:Z_get_softmax_dimrrrCr(r))rrrA stacklevelrrrrszSoftmax.forwardcCsdS)NZQuantizedSoftmaxrrrrrrszSoftmax._get_nameFcCs$|j\}}t|jt|t|Sr )r,r-rrAr.r/r0rrrrszSoftmax.from_floatcCs||jt|t|Sr )rAr.r/r1rrrr3szSoftmax.from_reference)Nr6r)Fr4rrrrrs  c@s8eZdZejjjjZddZ e ddZ e ddZ dS)r cCsdS)NZQuantizedMultiheadAttentionrrrrrrszMultiheadAttention._get_namecCs tddS)NzpIt looks like you are trying to convert a non-observed MHA module. Please, see the examples on quantizable MHAs.)NotImplementedError)r2otherrrrrszMultiheadAttention.from_floatcCstjjj|ddddd}||_|jdk rd|jd}tj|dd\}}t |||tj }t |d||j dk r|jd}tj|dd\}}t |||tj }t |d||` |`|S)NFT)mappingrZremove_qconfigZconvert_custom_config_dictbias_k)Z reduce_rangebias_v)rr8Z quantizationconvertrrH _parameterspopZ_choose_qparams_per_tensorquantize_per_tensorquint8setattrrIZin_proj_weightZ in_proj_bias)r2rFZ convertedrHscZzprIrrr from_observeds.         z MultiheadAttention.from_observedN) rr r!rr8r9 quantizabler Z _FLOAT_MODULErr5rrQrrrrr s   csveZdZdZdeeeddfdd Zejdddd Z ejejd d d Z d dZ e dddZ e ddZZS)r a%This is the quantized equivalent of :class:`~torch.nn.PReLU`. Args: scale: quantization scale of the output tensor zero_point: quantization zero point of the output tensor num_parameters: number of parameters: 1, or the number of channels at input. Default: 1 N)r?r@num_parametersr=csNt||_||_||_tj|tjd}tj|ddtj d}| |dS)N)r'r6r)r(r)r') r r rTr(r)rZrandnr.rMrN set_weight)rr?r@rTwZqwrrrr s zPReLU.__init__)rVr=cCs ||_dSr )weight)rrVrrrrUszPReLU.set_weight)rr=cCstjj||j|j|jSr )rrrZprelurWr(r)rrrrr sz PReLU.forwardcCsdS)NZQuantizedPReLUrrrrrr szPReLU._get_nameFc Cs|j\}}|t|t||j}|j}|j}|||jtj kr\t d|j|\}} t |t|t| tj } | | |SNz9PReLU's weight observer should have dtype quint8 but got ) r,r-r.r/rTrWqconfigr'rrNrrMrU) r2rrr(r)qprelufloat_wtobserverwt_scalewt_zpqweightrrrrs$      zPReLU.from_floatc Cs|t|t||j}|j}|j}|||jtjkrNtd|j| \}}t |t|t|tj} | | |SrX) r.r/rTrWrYr'rrNrr-rMrU) r2rr(r)rZr[r\r]r^r_rrrr3!s"      zPReLU.from_reference)rS)F)rr r!r"r.r/r rZTensorrUrrr5rr3r$rrrrr s  )rwarningsr__all__r9ZReLUrrrrrrr8rRr Moduler rrrrs$  &&-