U yhD @sXddlZddlmZmZddlZddlmZddlmZm Z ddl m Z m Z e ddZ ejejejejgZejejgZdd eDZed d eDd d Ze d e e ddejeeeeejejdddZe e ddejeeeeejejdddZe de e ddejejejeeejejdddZ e e ddejejejeeejejdddZ!e de e ddejejejejejejejdddZ"e e ddejejejejejejejddd Z#e d!e e d"ddd#ejeeeeejeejejd$d%d"Z$e e d"ddd#ejejejeeejeejejd$d&d'Z%e d(e e d)ddd#ejejejeeejeejejd$d*d+Z&e e d)ddd#ejejejeeejeejejd$d,d-Z'e d.e e d/ddd#ejejejejejejeejejd$d0d1Z(e e d/ddd#eejejd2d3d4Z)e d5e e d6dejeeeejeejejfd7d8d9Z*e d:e e d;dejeeeejeejejfd7dd?d@Z,e e d;dejeeeejeejejfd>dAdBZ-dCdDZ.e dEe e dFdejejejeeeejejdGdHdFZ/e e dFdejejejeeeejejdGdIdJZ0e dKe e dLddd#ejejeejeeeejeejejdM dNdLZ1e e dLddd#ejejeejeeeejeejejdM dOdPZ2e dQe e dRdejejeejejfdSdTdRZ3e e dRdejejeejejfdSdUdVZ4e dWe e dXdYejejeejejfdSdZdXZ5e d[e e d\dejejeejejfdSd]d\Z6e e d\dejejeejejfdSd^d_Z7d`daZ8e dbe e dcdejejejeeejdddedcZ9e e dcdejejejeeejdddfdgZ:e dhe e didej;fejejejeeejejdjdkdiZe e doddejejejeeejdddrdsZ?e dte e duddpej;fejejeejeeejeejdvdwduZ@e dxGdydzdzejAjBZCe e d{d|ejejejeeeejd}d~d{ZDe e d{dejejejeeeejd}ddZEdS)N)OptionalTuple)_unsqueeze_multiple)determine_qparamsvalidate_qmin_qmax)implLibraryZquantized_decomposedZDEFcCs&i|]}|t|jt|jfqS)torchZiinfominmax.0kr r V/var/www/html/venv/lib/python3.8/site-packages/torch/ao/quantization/fx/_decomposed.py srcCs.i|]&}|tt|jtt|jfqSr )intr finfor r r r r rrscCs^|tkrtd|t|\}}||ks>td|d|||ksZtd|d|dS)NzUnsupported dtype: z9quant_min out of bound for dtype, quant_min_lower_bound: z quant_min: z9quant_max out of bound for dtype, quant_max_upper_bound: z quant_max: )_DTYPE_TO_QVALUE_BOUNDS ValueErrorAssertionError) quant_min quant_maxdtypeZquant_min_lower_boundZquant_max_upper_boundr r r_quant_min_max_bounds_checks   rzxquantize_per_tensor(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_tensorZCompositeExplicitAutograd)inputscale zero_pointrrrreturncCsp|jtjtjfkr|tj}|jtjks:td|jt|||d|}tt ||||||S)a Affine quantization for the Tensor using the same quantization parameters to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead Tensorzquantize_per_tensor.tensorcCsV|dkstd||dksExpecting zero_point tensor to be one element, but received : 9Expecting scale tensor to be one element, but received : numelrritemr*r r rquantize_per_tensor_tensorXsr3cCs|jtjtjfkr|tj}|dks Tensorzquantize_per_tensor.tensor2cCs^|dkstd||dks Tensordequantize_per_tensor out_dtype)rrrrrrr:rcCsZ|j|ks td|d|j|dkr.tj}|tkrH||||Std|dS)a Affine dequantization for the Tensor using the same quantization parameters to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per tensor quantized Tensor if combined with quantization parameters in the argument of this function (scale/zero_point) scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for input Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for input Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): dtype for input Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor Expecting input to have dtype: z , but got N,Unsupported dtype in dequantize_per_tensor: )rrr r%rr$rrrrrrrr:r r rr8s % cCs|dkrtj}tj||dSNr()r r%r)r=r r rdequantize_per_tensor_metas r?zdequantize_per_tensor.tensor(Tensor input, Tensor scale, Tensor zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensorzdequantize_per_tensor.tensorc CsZ|dkstd||dks Tensorzdequantize_per_tensor.tensor2c Csb|dkstd||dks (Tensor, Tensor)zchoose_qparams.tensor)rqminqmaxepsrrc Cs||jtjtjtjfks&td|j|tksFtdtd|t||t |\}}t |||||t |gddS)\ Given an input Tensor, derive the per tensor affine quantization parameter (scale and zero_point) for target quantized Tensor from the Tensor Args: input (torch.Tensor): floating point input Tensor quant_min (int): minimum quantized value for target quantized Tensor quant_max (int): maximum quantized value for target quantized Tensor dtype (torch.dtype): dtype for target quantized Tensor Returns: scale (float): quantization parameter for the target quantized Tensor zero_point (int): quantization parameter for the target quantized Tensor CExpecting input to have dtype torch.float32/16/b16, but got dtype: $Expecting target dtype to be one of , but got: F)has_customized_qrange) rr r%r"r#rrkeysraminmaxrTensorrrFrGrHrmin_valmax_valr r rchoose_qparams_tensor;s*    rTz|choose_qparams_symmetric.tensor(Tensor input, int quant_min, int quant_max, float eps, ScalarType dtype) -> (Tensor, Tensor)zchoose_qparams_symmetric.tensorc Cs|jtjtjtjfks&td|j|tksFtdtd|t||t |\}}t |||||t |gdtj dS)rIrJrKrLF)rMZqscheme) rr r%r"r#rrrNrrOrrPZper_tensor_symmetricrQr r rchoose_qparams_symmetric_tensorbs,    rU)rrrrHrrcCsj|jtjtjtjfks&td|j||ksBtd|d|tjdtj|jdtjdtj |jdfS)NrJzKExpecting quant_min to be smaller than quant_max but received min: z max: r-rdevice) rr r%r"r#remptydoublerWint64rrrrHrr r rchoose_qparams_tensor_metas  r\cCs(tjdtj|jdtjdtj|jdfS)Nr-rV)r rXrYrWrZr[r r r$choose_qparams_symmetric_tensor_metasr]cCs6tt|}d||<||d<|t|}||fS)Nr)listrangedimpermutetuple)xaxisZ new_axis_listyr r r_permute_to_axis_zeros rfzquantize_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_channel)rscales zero_pointsrdrrrrc Cs|jtjtjfkr|tj}|jtjks:td|j||ksXtd|t|||t ||\}}t |}t | dD]4} t t|| d|| || |||| <q|t|} | |S)au Affine per channel quantization for the Tensor using the same quantization parameters for each channel/axis to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_point (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r Expecting axis to be < rr!)rr r"r#r$r%rr`rrf zeros_liker_sizer&r'rarb) rrhrirdrrrpermute_axis_listresioutr r rrgs     cCsr|jtjtjfkr|tj}|jtjks:td|j||ksXtd|t|||tj ||dS)Nr rjr() rr r"r#r$r%rr`rr))rrhrirdrrrr r rquantize_per_channel_metas   rqzdequantize_per_channel(Tensor input, Tensor scales, Tensor? zero_points, int axis, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensordequantize_per_channel) rrhrirdrrrr:rc Cs|j|ks td|d|j|dkr.tj}||ksLtd|t|||t||\}}tj||d} t| dD]6} |dk r|| nd} ||  || || | | <q| t |} | S)a Affine per channel dequantization for the Tensor using the same quantization parameters for each channel/axis to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per channel quantized Tensor if combined with quantization parameter in the argument of this function (scales/zero_points/axis) scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_points (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for output Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): requested dtype for output Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor Expecting input to have dtype , but got dtype: Nrjr(r) rrr r%r`rrfrkr_rlr$rarb) rrhrirdrrrr:rmrnrozprpr r rrrs(   cCsf|j|ks td|d|j|dkr.tj}||ksLtd|t|||tj||dS)Nrsrtrjr()rrr r%r`rr))rrhrirdrrrr:r r rdequantize_per_channel_meta3s  rvzLchoose_qparams_per_token(Tensor input, ScalarType dtype) -> (Tensor, Tensor)choose_qparams_per_token)rrrcCsx|jddd}|jtjkr&|}|tjkrFd}d|dd}ntd||jdd  |}t |}||fS) Choose quantization parameters for per token quantization. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32/float16 Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor Returns: scales and zero_points, both float32 Tensors Tr`Zkeepdimr-z/unsupported dtype in choose_qparams_per_token: gh㈵>r ) absamaxrr r"floatint8 Exceptionr&divrk)rrrhZn_bitsrrir r rrwLs   cCs6d|df}tj|tj|jdtj|tj|jdfSNr-ryrVrlr rXrYrWrZrrrlr r rchoose_qparams_per_token_metars rz]_choose_qparams_per_token_asymmetric_impl(Tensor input, ScalarType dtype) -> (Tensor, Tensor))_choose_qparams_per_token_asymmetric_implZCompositeImplicitAutogradcCsd\}}tj|ddd}tj|ddd}t|t|}t|t|}ttjj}||t ||} | j |d} || } || } || } || } t | | dk|| || }t ||| }| tj| tjfS)rx)iryTrzr}r)r Zaminrr rkr rr%rHrr&wherer'r$)rrrFrGrRrSZ min_val_negZ max_val_posrHrZ descaled_minZ descaled_maxZzero_point_from_min_errorZzero_point_from_max_errorrr r rrs&  zWchoose_qparams_per_token_asymmetric(Tensor input, ScalarType dtype) -> (Tensor, Tensor)#choose_qparams_per_token_asymmetriccCs t||Sr6)r)rrr r rrs cCs6d|df}tj|tj|jdtj|tj|jdfSrrrr r r(choose_qparams_per_token_asymmetric_metas rcCsftt|dd}||ks>td|d|||ksbtd|d|dS)Nryz num_tokens: z scales: z zero_points: )mathprodr^rlr1r)rrhriZ num_tokensr r r!_per_token_quant_qparam_dim_checks  rz}quantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_tokenrrhrirrrcCsBt|||t||||d|||||}|S)aPer token quantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per token affine quantization zero_points (int32 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r!)rrmuladdr'r&r$rr r rrs   $cCst|||tj||dSr>rr r)rr r rquantize_per_token_meta s rzdequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensordequantize_per_tokenrrhrirrr output_dtypecCs||}|||}|S)aPer token dequantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): quantized Tensor (uint8, int8 etc.) scales (float32 torch.Tensor): quantization parameter for per token affine quantization zero_points (int32 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` )r$rr r rrscCst|||tj||dSr>rrr r rdequantize_per_token_meta=s rzquantize_per_channel_group(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size) -> Tensorquantize_per_channel_groupc Cs|dks t||jdkr2|jddkr2|jd}|jd|dksHt|dksXt|d|}t|dkszt|dd}|dd}|d|| || | |}|S)Nr-ryrr|r!) rshaper`reshaper isnansumrrr'Zclamp_r$ reshape_as) rrhrirrr group_sizeZto_quantZ input_int8r r rrSs*      cCsf|dks t||jdkr2|jddkr2|jd}|jd|dksHt|dksXttj||dS)aXGroupwise quantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r-ryrr|r()rrr`r r))rrhrirrrrr r rquantize_per_channel_group_metazs   rzdequantize_per_channel_group(Tensor input, Tensor scales, Tensor? zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size, ScalarType output_dtype) -> Tensordequantize_per_channel_group)w_int8rhrirrrrrc Cs|dks t||jdkr2|jddkr2|jd}|jd|dksHt|dksXt|d|}|dd}|dk r|dd} ntjgtj|jd} ||  | | |} | S)a!Groupwise dequantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): quantized Tensor (uint8/int8 etc.) scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` r-ryrr|NrV) rrr`rr Zzerosint32rWsubrrr$) rrhrirrrrrZw_int8_groupedruZw_dqr r rrs"    zyfake_quant_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max) -> Tensorc@s$eZdZeddZeddZdS)FakeQuantPerChannelc Cs|jtjkr|tj}|jtjkr0|tj}|jtjksLtd|j||ksjtd|ttd|tt|d|j }t ||}t ||} t |d|| } t | ||| |} t | |k| |k} || | S)Nr rjrr-r!)rr r%r$rrr`r^r_ndimrr'r& logical_andZsave_for_backward) ctxrrhrirdrrZbroadcast_dimsZunsqueeze_scalesZunsqueeze_zero_pointstemprpmaskr r rforwards    "   zFakeQuantPerChannel.forwardcCs|j\}||dddddfSr6)Z saved_tensors)rgyrr r rbackwardszFakeQuantPerChannel.backwardN)__name__ __module__ __qualname__ staticmethodrrr r r rrs rfake_quant_per_channelZAutograd)rrhrirdrrrcCst||||||Sr6)rapplyrrhrirdrrr r rrs cCs t|Sr6)r r)rr r rfake_quant_per_channel_metas r)r)r)Frtypingrrr Z torch._refsrZtorch.ao.quantization.utilsrrZ torch.libraryrrZquantized_decomposed_libZuint8rZint16rZ_INTEGER_DTYPESZ float8_e5m2Z float8_e4m3fnZ _FLOAT_DTYPESrupdaterdefinerPrrrrr+r3r4r5r7r8r?rBrCrDrErTrUr\r]rfrgrqrrrvrwrrrrrrrr%rrrrrZautogradFunctionrrrr r r rs0            /       " )    ,  9 ! +   "     $ % .