U U?hd@sddlZddlmZmZddlZddlZddlZzddlm Z Wne k rXdZ YnXddl m Z ddl mZddlmZmZmZmZmZmZmZmZmZmZmZddlmZGd d d ZGd d d ZdS) N)AnyDict)to_array_extended) TensorData) ONNXModel) ONNX_TYPE_TO_NP_TYPETENSOR_NAME_QUANT_SUFFIX QuantType find_by_namemodel_has_infer_metadatanormalize_axispack_bytes_to_4bit quantize_dataquantize_nparray&save_and_reload_model_with_shape_infertensor_proto_to_array)TensorQuantOverridesHelperc@s:eZdZeeefdddZddZddZdd Z d S) QuantizationParamsdatacKsi|_|D]\}}t|ts:tdt|d|dt|tttjfsftdt|d|d|dkr|j tj tj fkrt d|j d|||j|<qdS)NzKeys must be strings not z for k=.z1Values must be numpy arrays, int, float, str not scalez5scale must a float32 or float16 numpy element but is ) ritems isinstancestr TypeErrortypeintnpndarraydtypefloat32float16 ValueError)selfrkvr(Y/var/www/html/venv/lib/python3.8/site-packages/onnxruntime/quantization/base_quantizer.py__init__&s zQuantizationParams.__init__ccs|jEdHdSNrr%r(r(r)__iter__1szQuantizationParams.__iter__cCs |j|Sr+r)r%keyr(r(r) __getitem__4szQuantizationParams.__getitem__cCs t|jSr+)lenrr,r(r(r)__len__7szQuantizationParams.__len__N) __name__ __module__ __qualname__rrrr*r-r/r1r(r(r(r)r%s rc@sleZdZdddZddZddZdd Zd d Zd d ZddZ dddZ dddZ dddZ ddZ dS) BaseQuantizerNc Cst|st|}dd|jjD|_|jdd|jjD|jdd|jjDt||_ ||_ ||_ | rv| ni|_ d|j ko|j d|_ d|_d|j ko|j d|_|j d|tjtjtjfk|_|j dd |_|j d |_t|d ||_t|d ||_|dk rLttd d |rLtdtdd|Dd||_ ||_!||_"| |_#|$|_%t&|j di|_'dd|j (D|_)|j'*|j)|j+|\} } | st,| |j'-|_.dS)NcSsi|] }|j|qSr(name).0vir(r(r) Ksz*BaseQuantizer.__init__..cSsi|] }|j|qSr(r6)r8Zotr(r(r)r:LscSsi|] }|j|qSr(r6)r8itr(r(r)r:MsZEnableSubgraphZForceQuantizeNoInputCheckZWeightSymmetricZActivationSymmetricFZMinimumRealRange tensor_typecSs t|t Sr+)rr)tr(r(r)oz(BaseQuantizer.__init__..z(tensors_range contains unexpected types css|]}t|VqdSr+)r)r8r'r(r(r) qsz)BaseQuantizer.__init__..z, not TensorData.ZTensorQuantOverridescSsi|] }|j|qSr(r6)r8Zinitzerr(r(r)r:}s)/r rgraphZ value_infoZ value_infosupdateoutputinputrmodel per_channel reduce_range extra_optionsenable_subgraph_quantizationparentZforce_quantize_no_input_checkgetr ZQInt8ZQInt16Z QFLOAT8E4M3FNis_weight_symmetricis_activation_symmetricmin_real_rangegetattractivation_qType weight_qTypeanymapvaluesrset tensors_rangenodes_to_quantizenodes_to_excludeop_types_to_quantizecheck_opset_version opset_versionrtensor_quant_overrides initializerZ initializersZis_validkeysr$Zget_quant_typesZtensor_quant_override_qtypes) r%rErFrGrQrPrVrWrXrYrHZoverrides_validZ overrides_errr(r(r)r*<sR   " zBaseQuantizer.__init__cCstdSr+)NotImplementedErrorr,r(r(r)quantize_modelszBaseQuantizer.quantize_modelcCst||j}|dk Sr+)r rEr])r%Z input_namer]r(r(r)is_input_a_initializersz$BaseQuantizer.is_input_a_initializercCs|jSr+)rFr,r(r(r)is_per_channelszBaseQuantizer.is_per_channelcCsNt||j}|dk r.|jtjjtjjfkS|jr>|j dkrBdS|j |S)NF) r rEr] data_typeonnx TensorProtoFLOATFLOAT16rIrJis_valid_quantize_weight)r% weight_nameweightr(r(r)rhs z&BaseQuantizer.is_valid_quantize_weightcCsV|jdk r(t|jdkr(|j|jkr(dS|j|jkr8dS|jdk rR|j|jkrRdSdS)NrFT)rWr0r7op_typerYrX)r%noder(r(r)should_quantize_nodes   z"BaseQuantizer.should_quantize_nodecCs dd|jjjD}t|dkr(td|dj}|dkrPtd|ddS|dkrtd|d |jjj|d|jjjt j d d gd }|d kr|j t j jkrtd|d |jjj|d|jjjt j d d gd|jj_d }|S)NcSs g|]}|jr|jdkr|qS)zai.onnx)domain)r8Zopsetr(r(r) s z5BaseQuantizer.check_opset_version..rz$Failed to find proper ai.onnx domainr z$The original model opset version is ze, which does not support node fusions. Please update the model to opset >= 11 for better performance.z, which does not support quantization. Please update the model to opset >= 11. Updating the model automatically to opset 11. Please verify the quantized model. z, which does not support quantization to float 8. Please update the model to opset >= 19. Updating the model automatically to opset 19. Please verify the quantized model. )rEZ opset_importr0r$versionloggingwarningremoveextendrdhelperZ make_opsetidrQre FLOAT8E4M3FNZ ir_version)r%Zai_onnx_domainr[r(r(r)rZs6      z!BaseQuantizer.check_opset_version?cCst||j}t|}|t}|jtjjkrt |}|j t j krNtjj } n(|j t jkrdtjj} ntd|j d|t j} t jdg| j d} | d} tj| |} |j| gd}nz|||} t || t j} t j | t jd|j}tj||} |j| gt j | |j dd} d}|j} |d}tj| |}|j|g|jtjjkrv|j}ntjj}|d }|jtjjkrtj||jdgd g}nH| jdkrt j| jt jdd}tj||}ntj||gd g}|j|g|||| || fS) z] Quantized the bias. Zero Point == 0 and Scale == Input_Scale * Weight_Scale zEOnly float16 or float32 are supported with float 8 but bias dtype is rrr!ZCastZDequantizeLinear_scale _zero_pointr) r rEr]rr rQrdrer{rasarrayr!r#rgr"rfrZastypearrayreshape numpy_helper from_arrayinitializer_extendroundZint32dimsZINT32rz make_tensorsizeZzerosshape)r%Z bias_nameZ input_scaleZ weight_scalebetaZbias_initializerZ bias_dataZquantized_bias_namerZ node_qtypeZquantized_dataZ bias_scaleZbias_scale_dataZpacked_bias_initializerZ node_typeZ bias_np_dataZquantized_bias_scale_nameZpacked_bias_scale_initializerr<Zquantized_bias_zp_nameZpacked_bias_zp_initializerZ bias_zp_datar(r(r)quantize_bias_static_implsZ         z'BaseQuantizer.quantize_bias_static_implFc Cst|jt}|jd}|jd}t|}|jj|jid} d| krJ| dj}d| krd| krtj| dt|d} t| d} t || | | } t | tj st dt| | jtjkr| jtjkst d | jt | tj st dt| nt| || d |j| d |jo||j| d | d d\} } } } } t | tj sbt dt| | jtjkr~| jtjkst d | jt | tj st dt| |j}tj||g| d}tj||g| d}|j||g|sj|jtj j!krt }|j|_|j"#|j"||_| $%|_&t'dk r\t'|}|j(|j(ksz|%| %kr\t)d|j(d| %ddd|%ddd|j(dt*|ddd n|tj j+tj j,fkr0| jtj-tj.fkrt)d|dt/t0| %}tjj|||j"|dd}n,tj1| tj2|d|j"} tj34| |}|j|g|||fS)a :param weight: TensorProto initializer :param qType: type to quantize to :param keep_float_weight: Whether to quantize the weight. In some cases, we only want to qunatize scale and zero point. If keep_float_weight is False, quantize the weight, or don't quantize the weight. :return: quantized weight name, zero point name, scale name rrZ default_val quant_typer zero_pointr}Unexpected type Unexpected dtype symmetricrGrminrmaxrGrNZ rmin_overrideZ rmax_override)r~NzThe initializer of shape z! could not be created, expecting rpz, got z and shape=z raw=rQuantized weights for . must be 8-bit before packing as 4-bit values.Traw)5r7r rr\Zget_per_tensor_overridesr<rrrrflattenrr AssertionErrorrr!r"r#rrKrLrGrNrcrdrzrrtolistrErrQrer{rrycopytobytesraw_datarr RuntimeErrorrINT4UINT4int8uint8bytesrrtensor_dtype_to_np_dtyperr)r%rjZqTyperGkeep_float_weight q_weight_namezp_name scale_nameZ weight_dataZquant_overridesrrZ q_weight_data_Z scale_dtypescale_initializerzero_initializerq_weight_initializercheck packed_datar(r(r)quantize_initializer_impls                J z'BaseQuantizer.quantize_initializer_implTc( Cst||j}|dkr"td|t|}t|j}t||\} } | s`td|d|d|| }|j|} |jj |d|igd} t| } | dkr| | krtd|d | d t| d d|\}}|r||krtd |d |d| d ddd| d kr| d dj }| d  d|j p>|t jjt jjt jjfk}| d  d|joV|}g}g}g}t| D]}|||}|| kr|nd }| |}d|krrd|krrtj|dt|d}t|d}t||||}t|tjstdt||jtjkr |jtj ks0td|jt|tjsPtdt|t|tjs0tdt|nt!|||||j"| d| dd\}}}}}t|tjstdt||jtjkr|jtj kstd|jt|tjstdt|t|tjs0tdt||#||#||#|qnt$|j}t$|}d||<t%|d &|}tdt|D]*}t%||&|}t'||f|}q|t(} |d}!|d}"|j)|g}#t j*+|"|j,|#t-|.}$t j*+|!||#t-|.}%|j/|$|%g|s|t jjt jj0fkr|jtj1tj2fkrht3d| dt4t5|6}&t j*j+| |||&dd }'|j/|'gn:tj%|t j*7|d&|j)}t j89|| }'|j/|'g| |!|"fS)!Nz{} is not an initializerzWeight z# has a per-channel axis with value z that is out-of-bounds for rank Zaxisrrz.Per-channel tensor quantization overrides for z must have either 1 or z& elements in the list of dictionaries.rz"Tensor quantization overrides for z& specify an unexpected axis. Expected z , but got rrrrGrrr}rrrrrrrrrTr):r rEr]r$rr0rr r\Zget_per_channel_overridesr<rKrLrdreZINT8r{rrGrangeZtakerrrrrrr rrr!r"r#rrNappendlistrrZ concatenater rrzrrcZhstackrrrrrrrrrrrr)(r%rirQZ channel_axisrGrr]weightsZ weights_rankZ is_axis_validZ axis_normZ channel_countZquant_overrides_for_channelsZnum_channel_overridesZis_axis_override_validZ axis_overriderZzero_point_listZ scale_listZquantized_per_channel_data_listiZper_channel_dataZchannel_override_indexZchannel_quant_overridesrrZquantized_per_channel_datarZ weights_shapeZ reshape_dimsZquantized_weightsZchannel_weightsrrrZzero_scale_shaperrrrr(r(r) quantize_weight_per_channel_implls                         z.BaseQuantizer.quantize_weight_per_channel_implcCs|jdkrdS|jD]}|jdkr|jr.q||s:qt|j|jddkrXq|jd|jks|j d|jkrzq|j|j d}t |t st dt |d|j dd||j|jd<q|jdkrt tdtd d |j|j d<qdS) N)ZClipZRelurrrz for rZSoftmaxrr|)ZlowestZhighest)rVrEZnodesrkrMrmr0Zinput_name_to_nodesrDrCrrrrrr")r%rltdr(r(r)adjust_tensor_rangess$       z"BaseQuantizer.adjust_tensor_ranges)N)r|)FF)TF)r2r3r4r*r`rarbrhrmrZrrrrr(r(r(r)r5;s J# H ^ r5)rvtypingrrnumpyrrdZonnx.numpy_helperZonnx.reference.op_runr ImportErrorZ calibraterZ onnx_modelrZ quant_utilsrr r r r r rrrrrr\rrr5r(r(r(r)s   4