U U?hl@s~ddlmZddlZddlmZddlmZddlmZddl Z ddl Z ddl Z ddl m Z ddl mZdd lmZmZdd lmZdd lmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)Gd ddeZ*eGdddZ+GdddZ,eGdddZ-eGdddZ.eGdddZ/eGdddZ0eGdddZ1GdddeZ2dS)) annotationsN) dataclass)Enum)Any) TensorProto)onnx_pb) BaseQuantizerQuantizationParams) TensorData)DEQUANT_OP_NAME QUANT_OP_NAMEQuantizedValueQuantizedValueType __producer__ __version__add_dequant_output_suffixadd_dequant_suffixadd_quant_input_suffixadd_quant_output_suffixadd_quant_suffixcompute_scale_zpcompute_scale_zp_float8 find_by_nameget_qmin_qmax_for_qType ms_domainnormalize_axistensor_proto_to_array)CreateQDQQuantizerc@seZdZdZdZdZdS)QDQQuantTensorTyperrN)__name__ __module__ __qualname__ ACTIVATIONWEIGHTZBIASr&r&X/var/www/html/venv/lib/python3.8/site-packages/onnxruntime/quantization/qdq_quantizer.pyr,src@seZdZUded<ded<dS)QDQQuantParamProviderstr input_name node_nameNr!r"r#__annotations__r&r&r&r'r(5s r(c@s eZdZejdddfddZdS)QDQTensorQuantInfoNcCs2||_||_||_|dk |_|dk s(t||_dSN) tensor_typequant_para_provideraxis is_sharedAssertionError data_type)selfr0r1r2r5r&r&r'__init__>s   zQDQTensorQuantInfo.__init__)r!r"r#rr$r7r&r&r&r'r.=sr.c@s.eZdZUded<ded<ded<ded<dS)QDQBiasQuantInfor)r+r* weight_namefloatbetaNr,r&r&r&r'r8Hs r8c@s&eZdZUded<ded<ded<dS)QDQTensorQuantParamsr originalzQuantizationParams | None convertedset[str] | Noneconverted_recv_nodesNr,r&r&r&r'r<Ss r<c@seZdZUded<ded<dS)QDQScaleZpInitializersrscale zero_pointNr,r&r&r&r'rA[s rAc@s&eZdZUded<ded<ded<dS)QDQTensorScaleZpInitializersrAr=zQDQScaleZpInitializers | Noner>r?r@Nr,r&r&r&r'rDds rDc@s4eZdZUded<ded<ded<dddd Zd S) QDQTensorQuantizedValuerr=zQuantizedValue | Noner>r?r@returncCs6|jdkr|jS|jdkr |jS||jkr0|jS|jSr/)r>r=r@)r6Zconsumer_node_namer&r&r'get_for_consumerts   z(QDQTensorQuantizedValue.get_for_consumerN)r!r"r#r-rHr&r&r&r'rEns rEc@seZdZd`ddZddZddZdejfdd Zd d d d Z d d d dddZ d d ddZ ddZ daddZ ddZddZddZddZdbd d d d d d d!d"d#Zdcd d d d d d d$d%d&Zddd'd(Zded)d*Zdfd+d,Zd-d.Zd/d0Zd1d2Zd3d4Zd d d5d6Zdgd8d9d:d:d;d<d=d>Zdhd d?d@dAdBdCdDZdid d9d?d:d:d;dFdGdHZd dId dJdKdLZdjd dNd dOdPdQdRZ d dSdTdUdVZ!dWdXdNdYdZd[Z"d\d]d^d_Z#dS)k QDQQuantizerNc s&t|||||||||| | i|_i|_g|_| dg|_| dd|_| dd|_| dd|_ i|_ | di|_ | ddrt nd|_ | d d|_|jd krtjtjtjtjftfd d |jD} |j s|jks|jks| rtd t dt |_ ||_i|_dS)NZ"OpTypesToExcludeOutputQuantizationZAddQDQPairToWeightFZ QuantizeBiasTZDedicatedQDQPairZ QDQOpTypePerChannelSupportToAxisZUseQDQContribOpsZQDQKeepRemovableActivationsc3s|]}|jkVqdSr/)r0).0tZ opset21_typesr&r' sz(QDQQuantizer.__init__..zONNX QuantizeLinear and DequantizeLinear operators do not support 16-bit/4-bit integer quantization types prior to opset 21. The domain of QuantizeLinear and DequantizeLinear operators will be set to 'z' to enable support.)r r7tensors_to_quantizebias_to_quantizenodes_to_removegetZ'op_types_to_exclude_output_quantizationadd_qdq_pair_to_weight quantize_biasdedicated_qdq_pairtensor_to_its_receiving_nodes'qdq_op_type_per_channel_support_to_axisr qdq_op_domainZqdq_keep_removable_activations opset_versionrZUINT16ZINT16ZUINT4ZINT4anyZtensor_quant_override_qtypesactivation_qType weight_qTypeloggingwarningcalc_graph_quant_paramsquantization_paramsquantized_value_map) r6model per_channel reduce_ranger\r[ tensors_rangeZnodes_to_quantizeZnodes_to_excludeZop_types_to_quantizeZ extra_optionsZoverrides_have_opset21_typesr&rMr'r7sT      zQDQQuantizer.__init__cCsLt||j}|dk r|jS||jkrH|j|}|jdrH|jjjSdS)2 Check if tensor can be quantized Nr0) rrb initializerr5 value_infostypeHasFieldr0 elem_typer6 tensor_nameweightvir&r&r'_get_tensor_types    zQDQQuantizer._get_tensor_typecCst||j}|dk r4|jtjjtjjfkrdSnN||jkrp|j|}|j dr|j j j tjtjfkrdSnt d|ddS)rfNTr0z$failed to infer the type of tensor: z6. Skip to quantize it. Please check if it is expected.F)rrbrgr5 onnx_protorFLOATFLOAT16rhrirjr0rkr]r^rlr&r&r'_is_tensor_quantizables   z#QDQQuantizer._is_tensor_quantizablecCsv||rr|rLt|ts,tdt|d||}t|||d|j|<n&||jkrr||}t||d|j|<dS)a Adds a tensor to the list (actually a dict) of tensors to quantize. Called indirectly by op quantizers that want to quantize a tensor (i.e., "mark" a tensor for quantization). If quant_sharing_provider is not None, tensor with name tensor_name will be quantized with the same quantization parameters as the node input specified in quant_sharing_provider. Ex: A Tranpose node's output will typically use the same quantization parameter initializers used at the Transpose node's input. Args: tensor_name: name of the tensor to quantize quant_sharing_provider: name of the tensor and node that provides quantization parameter tensor_type: QDQQuantTensorType default ACTIVATION zBquant_sharing_provider must be of type QDQQuantParamProvider, not .)r0r1r5)r0r5N)rt isinstancer( TypeErrorrirpr.rO)r6rmZquant_sharing_providerr0r5r&r&r'Z__quantize_tensors     zQDQQuantizer.__quantize_tensorr))rmcCs||dtjS)z Adds a tensor to the list of tensors to quantize. Called by op quantizers that want to quantize a tensor (i.e., "mark" a tensor for quantization). Args: tensor_name: name of the tensor to quantize N)_QDQQuantizer__quantize_tensorrr$r6rmr&r&r'quantize_activation_tensorsz'QDQQuantizer.quantize_activation_tensor) output_namer*r+cCs||t||tjS)a Adds a tensor to the list of tensors to quantize. Called by op quantizers that want to quantize an output tensor using the same quantization parameters as one of the node's inputs. Ex: A Tranpose node's output will typically use the same quantization parameter initializers used at the Transpose node's input. Args: output_name: name of the node output to quantize so that it uses the same quantization params as an input. input_name: name of the node input from which the output tensor will get its quantization params. node_name: name of the node that consumes `input_name`. )rxr(rr$)r6r{r*r+r&r&r'quantize_output_same_as_input&s z*QDQQuantizer.quantize_output_same_as_inputcCs||dtjS)z Adds a tensor to the list of weight tensors to quantize. Called by op quantizers that want to quantize a weight (i.e., "mark" a weight for quantization). Args: tensor_name: name of the weight to quantize N)rxrr%ryr&r&r'quantize_weight_tensor7sz#QDQQuantizer.quantize_weight_tensorcCsZt||j}|rD|jtjjtjjfkrVtt j ||jd|j |<nt d|ddS)N)r0r2r5z9only support per-channel quantization on weight. Tensor: z is not quantized.)rrbrgr5rqrrrrsr.rr%rOr]r^)r6rmr2rnr&r&r'"quantize_weight_tensor_per_channelAsz/QDQQuantizer.quantize_weight_tensor_per_channel?c Cs|j|rPtd|d|j|dd\}}|rB|||n ||dSt||j }|dk r|j t j j t j jfkr||jkrt|||||j|<qtd|dntd|d dS) a Adds a bias tensor to the list of bias tensors to quantize. Called by op quantizers that want to quantize a bias with bias_zero_point = 0 and bias_scale = input_scale * weight_scale * beta. TODO: Explain the reasoning for using this formula. Args: node_name: name of the node that consumes the bias, input, and weight tensors. bias_name: name of the bias tensor to quantize. input_name: name of the input tensor whose scale is used to compute the bias's scale. weight_name: name of the weight tensor whose scale is used to compute the bias's scale. beta: Multiplier used to compute the bias's scale. zQuantizing bias tensor 'z=' as a weight due to the presence of user-specified overridesr) default_axisNzBias z) has already been marked for quantizationz Expected z to be a weight)tensor_quant_overridesrRr]infois_tensor_per_channelr~r}rrbrgr5rqrrrrsrPr8r^) r6r+ bias_namer*r9r;Zis_per_channelr2rnr&r&r'quantize_bias_tensorKs     z!QDQQuantizer.quantize_bias_tensorcCs|j|dSr/)rQappend)r6noder&r&r' remove_nodenszQDQQuantizer.remove_nodecCs|j|jdSr/)rb remove_nodesrQ)r6r&r&r'rqszQDQQuantizer.remove_nodescCs|jD]P}||r t||}||jD](}||jkrHg|j|<|j||q0q || |j rz| | |j s|jt|jj_t|jj_|jtkr|jtd|jjS)Nr)rbZnodesZshould_quantize_noderquantizeinputrVr_quantize_normal_tensors_quantize_sharing_param_tensorsrT_quantize_bias_tensorsrrSZclean_initializersrZ producer_namerZproducer_versionrXrZset_opset_import)r6rZ op_quantizerrmr&r&r'quantize_modelts(         zQDQQuantizer.quantize_modelcCs||jkr||j|jdkr||j|jdkr|t|j|dkr||j|s||j|s||j||||jkrx|j|=dSdS)NrTF) r`r>lenrbZinput_name_to_nodesis_graph_outputZis_graph_inputreplace_output_of_all_nodesrO)r6Zupstream_output_namer{r&r&r'try_replacing_upstream_outputs"   z*QDQQuantizer.try_replacing_upstream_outputz int | None)q_inputq_outputquant_node_name scale_namezp_namer2cCs4tjjt|||g|g|||jd}|j|gdS)zI Creates a QuantizeLinear node and adds it to the model. r2domainN)onnxhelper make_noder rXrb add_nodes)r6rrrrrr2 qlinear_noder&r&r'_create_q_nodes zQDQQuantizer._create_q_node)dq_input dq_outputdequant_node_namerrr2cCs4tjjt|||g|g|||jd}|j|gdS)zK Creates a DequantizeLinear node and adds it to the model. rN)rrrr rXrbr)r6rrrrrr2 dequant_noder&r&r'_create_dq_nodes zQDQQuantizer._create_dq_nodec CsXtjjt|||g|g|| |jd} tjjt|||g|g|| |jd} |j| | gdS)Nr)rrrr rXr rbr) r6rrrrrrrrr2rrr&r&r'_create_qdq_nodess"zQDQQuantizer._create_qdq_nodesc Cs|j}|dk rh|jdkr td|tjkr0|jn|j}|tjj j krLt j j }|j ||||jd\}}}n*|j||tjkr~|jn|j|jd\}}}t|} |j|| |jrt|} ||| t|| | t|||| n2tjjt|||g| gt|||jd} |j| dS)N zLPer-Channel support with QDQ format requires onnx opset version 13 or above.)keep_float_weightr)namerY ValueErrorrr%r\r[rrrZUINT8rqZINT8quantize_weight_per_channelrSquantize_initializerrrbreplace_input_of_all_nodesrrrrrrr rXadd_node) r6Z weight_protor0r2r9Zqtype q_weight_namerrZweight_dequant_outputZweight_quant_outputrr&r&r'_add_qdq_pair_for_initializersT    z*QDQQuantizer._add_qdq_pair_for_initializerc Cs||jr||jkrt|j|dkrt|j|}t|D]}d|d}t||}t||} t||} t||} |||| || | |||j||} |j | || |dkr8t || ||t j |d} t| dd|j|<q8n|}t|}|j |rt|}|}|j ||n|j ||||t|t|t||t|||t ||||t j |d} t| dd|j|<dS)Nr_r scale_type)rUrVrrangerrrrrrbZreplace_node_inputrrInputrErarrrr)r6rmrrr5Znum_dedicated_qdq_pairiZpostfixZ tensor_name_quant_output_postfixZ"tensor_name_dequant_output_postfixZquant_node_name_postfixZdequant_node_name_postfixrquantized_valuerrr&r&r'_add_qdq_pair_for_activationsz       z)QDQQuantizer._add_qdq_pair_for_activationcCstdd|j|gD}|jrF||jkrFt|j|dkrFtd|} |dkr^|}t} n| |} t|t|k} |j|} |} | rt|} |j || t |} | | | t |||t |}| r| s|}| r||kr|j||| || |t||||}| s8t|d}|| |t|d||t |d}| ||t |d||t |d}| r| r|}|r||kr|j||||||t|d||t||||tj|d}t||||tj|d}t||||j|<dS) a Adds Q and DQ ops to a tensor whose quantized data type is converted. That is, some consumers may use the original data type from the producer, while other consumers use the converted data type. This is generally done by adding a sequence of ops that convert from one data type (e.g., uint8) to another (e.g., uint16). T_float ---> Quant(to u8) ---> Convert(to u16) ---> Dequant(to float) ---> T_float' where Convert(to u16) is equivalent to: ---> Dequant(to float) ---> Quant(to u16) ---> This function handles the following scenarios: 1) Tensor T is not a graph output; all consumers use the converted type ---> Q1 ---> DQ1 ---> Q2 ---> DQ2 ---> 2) Tensor T is not a graph output; some consumers use the original type, others use the converted type ---> Q1 -+-> DQ1 ---> | +-> DQ1' ---> Q2 ---> DQ2 ---> 3) Tensor T is a graph output; all consumers use the converted type ---> Q1 ---> DQ1 ---> Q2 ---> DQ2 -+-> | +-> 4) Tensor T is a graph output; some consumers use the original type, others use the converted type ---> Q1 -+-> DQ1 -+-> | | | +-> | +-> DQ1' ---> Q2 ---> DQ2 ---> 5) Tensor T is a graph output that is not consumed by any other nodes. ---> Q1 ---> DQ1 ---> Q2 ---> DQ2 ---> cSsg|] }|jqSr&)r)rKrr&r&r' szFQDQQuantizer._add_qdq_ops_for_converted_activation..rz|Do not currently support converted quant_types in TensorQuantOverrides when the `dedicated_qdq_pair` extra_option is enabledN_convertZ_convert_cloner)setrVrRrUrrrbrrrrrrrZreplace_input_of_nodesrrrrrrEra)r6rmZfirst_scale_nameZ first_zp_nameZscale_data_typeZconvert_scale_nameZconvert_zp_nameZconvert_recv_nodesZtensor_recv_nodesZoriginal_recv_nodesZall_use_convertedrZ first_q_inputZfirst_q_outputZfirst_dq_outputZsecond_q_inputZsecond_q_outputZsecond_dq_outputZoriginal_quantized_valueZconverted_quantized_valuer&r&r'%_add_qdq_ops_for_converted_activationYs0          z2QDQQuantizer._add_qdq_ops_for_converted_activationc Cs|jD]\}}||jkr"q|jst||j}|rP|||j |j n| |}|snt d|d|j dkr|j||jjj|jjj|jdnF|j|jjjkst|||jjj|jjj|j|j jj|j jj|j|j|=qdS)z} Adds Q/DQ ops to tensors (activations and weights) that have been marked for quantization by op quantizers. z4Quantization parameters are not specified for param zb. In static mode quantization params for inputs and outputs of nodes to be quantized are required.N)r5)rOcopyitemsrar3rrbrgrr0r2"_make_tensor_scale_zp_initializersrr>rr=rBrrCr5r4rr@)r6rm tensor_inforgZtensor_qparam_initializersr&r&r'rs<      z%QDQQuantizer._quantize_normal_tensorsc Cs|jr|jD]\}}|j}|r|j|jkr|j|=|j|j|j}||r`t dd}d}||j kr|j |}|j r| ||j d}|j }|dkr|||j|jq|||j|j|jj|jj|jj|qqdS)a{ Adds Q/DQ ops to tensors that have been marked for quantization by op quantizers. Only operates on tensors that want to use the quantization parameter initializers from an upstream tensor. For example, a Transpose node's output tensor will typically want to use the same quantization parameter initializers as the Transpose node's input. zBQuantization parameter shared mode is not supported for weight yetNr)rOrrr1r*rarHr+Zis_input_a_initializerrr`r>_make_scale_zp_initializersr@rrrrrBr5rrC)r6rmrZquant_providerrZconverted_qparam_initsr@ tensor_paramsr&r&r'r$sH    z,QDQQuantizer._quantize_sharing_param_tensorsc Csl|jD]Z\}}||jkr q |||t||j}|j||j|j}|j dkrt |j t st dt|j d|jt|}tjjd|jg|g||j d}n|j dkrH|jtjjtjjtjjhkrtd|jd|j|j|jg}t|}|jdk r,tjjd ||g||j|jd }ntjjd ||g||jd }ntd |j d |j|q dS)zq Adds DQ ops (or Cast) for bias tensors that have been marked for quantization by op quantizers. ZCastUnexpected type z for input=)rto)NDequantizeLinearzUnexpected quantize type z for DequantizeLinear.Nrr)rzUnexpected operator type ru) rPrraquantize_bias_staticrrbrgZremove_initializerr= node_typervr5intrwrir*rrrrq_name node_qtyperrsZBFLOAT16rr RuntimeErrorrrr2rXr)r6r bias_infoinitZ quant_valuer+rinputsr&r&r'rVsZ         z#QDQQuantizer._quantize_bias_tensorscCs||jkp||jkSr/)rOrPryr&r&r'is_tensor_quantizedsz QDQQuantizer.is_tensor_quantizedFzonnx.TensorProtozonnx.TensorProto.DataTypeboolztuple[str, str, str])rnqTyperdrrGc Cst|j|jkr*|j|jj}|j|j|jfS|||||\}}}t|j|||tj d}t |dd|j|j<|||fS)a :param weight: TensorProto initializer :param qType: type to quantize to :param keep_float_weight: Whether to quantize the weight. In some cases, we only want to qunatize scale and zero point. If keep_float_weight is False, quantize the weight, or don't quantize the weight. :return: quantized weight name, zero point name, scale name N) rrar=rrrZquantize_initializer_implrr InitializerrE) r6rnrrdrrrrrr&r&r'rs,  z!QDQQuantizer.quantize_initializerrz str | Noneztuple[bool, int | None])rmrop_typerGc Cs|j|}|dkrdS|j|r(dS|j|}|jsB|sBdS|rT|j||n|}|rt|j|}|dd}t|}t |j } t || \} }| st d|d|d| dSd|fS) a Checks if a given tensor is configured to be quantized per-channel. If so, also returns the channel axis. ORT only supports per-channel quantization on static weights (i.e., ONNX initializers). If the user did not provide tensor quantization overrides for this tensor, then the value of self.per_channel determines if the weight is to be quantized per-channel. Params: tensor_name: The name of the tensor to check. default_axis: The default channel axis. This method checks if the normalized axis is within bounds. Can be overridden via the extra_options 'QDQOpTypePerChannelSupportToAxis' and 'TensorQuantOverrides'. op_type: Optional, defaults to None. The operator type that is the only consumer of this weight. Used to access the extra option 'QDQOpTypePerChannelSupportToAxis'. Returns: A tuple (is_per_channel, axis) in which the first element indicates whether the tensor is quantized per-channel and the second element is the channel axis. The returned axis is only None if the tensor is not per-channel or the axis is out of bounds. N)FNrr2zAxis z is out-of-range for weight 'z ' with rank T)Z initializersrRrZhas_per_tensor_overridesZhas_per_channel_overridesrcrWZget_per_channel_overridesrrshaperr]r^) r6rmrrweight_initializerZhas_per_chan_overridesr2Zper_chan_overridesZweight_nparrayZ weight_rankZ axis_validr&r&r'rs&       z"QDQQuantizer.is_tensor_per_channelT)r9r\ channel_axisrdrrGc Csn||jkr&|j|j}|j|j|jfS||||||\}}} t||| |tjd}t |dd|j|<||| fSr/) rar=rrrZ quantize_weight_per_channel_implrrrrE) r6r9r\rrdrrrrrr&r&r'rs.   z(QDQQuantizer.quantize_weight_per_channelr8)rrrGc Cs||jkr|j|jjS|j|jjj}t||j}t|}|j|j  |j j}t||j}t|}| ||||j \} } } } } }t|| | | tj| jdkrdnd| |d}t|dd|j|<| S)z] Quantized the bias. Zero Point == 0 and Scale == Input_Scale * Weight_Scale rrN)rr)rar=rr9rrrbrgrr*rHr+Zquantize_bias_static_implr;rrrsizerE)r6rrZweight_scale_namerZ weight_scaleZinput_scale_nameZinputscale_initializerZ input_scaleZquantized_bias_nameZquantized_bias_scale_nameZquantized_bias_zp_nameZbias_scale_datarrrr&r&r'r s:   z!QDQQuantizer.quantize_bias_staticr rA) param_nameparamsinit_name_suffixrGcCs:t|dg}t|ddr4|djtjtjfkrPtdt|dd|t|dg}|jtjkspt |j d|j }g}|d|}g} |d|} t j||||} |j| |jtjkrtjj} n,|jtjkrtjj} ntd |jd |t j| | | |d } |j| t| | S) z Creates and returns scale and zero-point initializers for the given quantization params. The initializers are named: - {param_name}_zero_point{init_name_suffix} - {param_name}_scale{init_name_suffix} rCrBdtyperz and param_name= quant_typeZ _zero_pointZ_scalezUnexpected dtype=z for param_name=))nparrayhasattrrZfloat32Zfloat16rriZfloat64r4datarRr[rrZ make_tensorZraveltolistrbZadd_initializerrqrrrrsZreshaperA)r6rrrZzero_point_valuesZ scale_valuesZzero_point_typeZzero_point_shapeZzero_point_nameZ scale_shaperZinit_zprZ init_scaler&r&r'r5s2 $         z(QDQQuantizer._make_scale_zp_initializersz#QDQTensorScaleZpInitializers | None)rmrGcCs|jdks||jkr*td|ddS|j|}t|tsXtdt|d|d|||j}|j r||||j dnd}t |||j S)a Create and returns all scale/zero_point initializers for a given tensor. If the tensor is converted to a different quantization type, this function creates two pairs of zp/scale initializers. Otherwise, only one pair of zp/scale initializers is created. Nz$Quantization parameters for tensor:"z" not specifiedr for rur) r`r]rrvr<rwrirr=r>rDr@)r6rmrZoriginal_initsZconverted_initsr&r&r'r[s  z/QDQQuantizer._make_tensor_scale_zp_initializersr zdict[str, Any]) tensor_dataquant_overridesrGc Cs|j}d|kr|dj}d|krr@r&r&r'r_s"     z$QDQQuantizer.calc_graph_quant_params)N)r)N)N)N)N)N)FF)N)TF)r)$r!r"r#r7rprtrr$rxrzr|r}r~rrrrrrrrrrrrrrrrrrrrrrr_r&r&r&r'rIsT Z    #  8 C.27+7 -&rI)3 __future__rr] dataclassesrenumrtypingrnumpyrrZonnx.numpy_helperrrrqZbase_quantizerr r Z calibrater Z quant_utilsr r rrrrrrrrrrrrrrrrregistryrrr(r.r8r<rArDrErIr&r&r&r's8       P