U U?hP@sddlZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZddlZddlZddlmZmZmZmZddlZddlmZmZmZejejejdd d Zd,ejeje eeejd d d ZGdddZ GdddZ!GdddeZ"Gdddej#dZ$GdddZ%Gddde%Z&Gddde%Z'Gddde'Z(Gdd d e'Z)Gd!d"d"e'Z*Gd#d$d$ej#dZ+Gd%d&d&e+Z,dd'e"j-d(ifee.efe e e.d)d*d+Z/dS)-N)Enum)Path)DictOptionalSequenceTupleUnion) ModelProto TensorProtohelper numpy_helper) apply_plotload_model_with_shape_infersmooth_distribution)pkqkreturncCs|tj|j|jd}|ddt|dd|dd|dd<|dk|dk@}d||<|dk|dk@}tj||<|S)z See https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.rel_entr.html#scipy.special.rel_entr. Python implementation. dtypeNr)npemptyshaperloginf)rrresc2c1rT/var/www/html/venv/lib/python3.8/site-packages/onnxruntime/quantization/calibrate.pyrel_entrs2 r )rrbaseaxisrcCs|dks|dkstd|dk s(tdt|tj}d|tj||dd}t|tj}t||\}}d|tj||dd}t||}tj||d}|dk r|t|}||j S) z Simplifeied version of entropy. Source: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html. This avoids taking a dependency on scipy just for this function. Nrz0base={base} must be a positive number or `None`.z qk is None?T)r"keepdimsr") AssertionErrorrasarrayastypefloat32sumZbroadcast_arraysr rr)rrr!r"Zvecsrrrentropy's  r,c@sTeZdZedddddddgZedddddgZdd Zed d Zed d Z dS) TensorDataavgstdlowesthighesthist hist_edgesbinscKs|D]\}}|tjkr2td|dtjd|tjkrt|ds^tdt|d||jtj tj fkrtd|jd|t |||qdS)NzUnexpected value z not in .rUnexpected type z for k=zUnexpected dtype ) itemsr-_allowed ValueError_floatshasattrtyperrZfloat16r)setattr)selfkwargskvrrr__init__Gs   zTensorData.__init__cCs4t|drt|ds(tdt|d|j|jfS)Nr0r1z0Attributes 'lowest' and/or 'highest' missing in r5)r;AttributeErrordirr0r1r>rrr range_valueRszTensorData.range_valuecCs4t|drt|ds(tdt|d|j|jfS)Nr.r/z)Attributes 'avg' and/or 'std' missing in r5)r;rCrDr.r/rErrravg_stdXszTensorData.avg_stdN) __name__ __module__ __qualname__ frozensetr8r:rBpropertyrFrGrrrrr-Cs  r-c@sZeZdZeeeeeffdddZddZ ddZ dd Z d d Z d d Z ddZdS) TensorsDatadatac Cs||_i|_|D]\}}t|ts:tdt|dt|tr|tj krvt |dkrvt |d|dd|j|<qt |dkrt |d|d|d|dd |j|<qtd |d d t |d |dt|t stdt|d||j|<qdS)NzKeys must be strings not r5rr r0r1)r0r1r2r4zUnexpected tuple for rz , it has z elements: zValues must be TensorData not ) calibration_methodrOr7 isinstancestr TypeErrorr<tupleCalibrationMethodMinMaxlenr-)r>rUrOr@rArrrrB`s    &" zTensorsData.__init__ccs|jEdHdSNrNrErrr__iter__rszTensorsData.__iter__cCs ||jkSr]rNr>keyrrr __contains__uszTensorsData.__contains__cCs |j|Sr]rNr_rrr __getitem__xszTensorsData.__getitem__cCs(||jkrtd|d||j|<dS)Nz)Only an existing tensor can be modified, z is not.)rO RuntimeError)r>r`valuerrr __setitem__{s zTensorsData.__setitem__cCs |jSr])rOvaluesrErrrrfszTensorsData.valuescCs |jSr])rOr7rErrrr7szTensorsData.itemsN)rHrIrJrrWrr-rrBr^rarbrerfr7rrrrrM_srMc@seZdZdZdZdZdZdS)rZrr rPrSN)rHrIrJr[Entropy Percentile DistributionrrrrrZsrZc@sTeZdZeddZejedddZddZ dd Z d d Z e e d d dZ dS)CalibrationDataReadercCst|drt|jptS)Nget_next)r;callablerkNotImplemented)clssubclassrrr__subclasshook__sz&CalibrationDataReader.__subclasshook__rcCstdS)z9generate the input data dict for ONNXinferenceSession runNNotImplementedErrorrErrrrkszCalibrationDataReader.get_nextcCs|Sr]rrErrrr^szCalibrationDataReader.__iter__cCs|}|dkrt|Sr])rk StopIteration)r>resultrrr__next__szCalibrationDataReader.__next__cCstdSr]rrrErrr__len__szCalibrationDataReader.__len__) start_index end_indexcCstdSr]rr)r>rxryrrr set_rangeszCalibrationDataReader.set_rangeN)rHrIrJ classmethodrpabcabstractmethoddictrkr^rvrwintrzrrrrrjs rj) metaclassc@s~eZdZdeeefeeedddZdgfdd Z d d Z e d d dZ ddZ ddZedddZedddZdS)CalibraterBaseNaugmented_model.onnxF model_pathop_types_to_calibratecCsnt|trtt||_nt|tr0t||_ntd||_||_||_||_ ||_ d|_ d|_ dg|_ dS)a :param model_path: ONNX model to calibrate. It should be a model file path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param symmetric: make range of tensor symmetric (central point is 0). :param use_external_data_format: use external data format to store model which size is >= 2Gb. :param per_channel: whether to compute ranges per each channel. z model_path should be model path.NCPUExecutionProvider)rVrWrrmodelr9raugmented_model_path symmetricuse_external_data_format per_channelZ augment_model infer_sessionexecution_providers)r>rrrrrrrrrrBs   zCalibraterBase.__init__rcCs||_|dS)zz reset the execution providers to execute the collect_data. It triggers to re-creating inference session. N)rcreate_inference_session)r>rrrrset_execution_providerssz&CalibraterBase.set_execution_providerscCs,t}tjj|_tj|j||jd|_dS)z9 create an OnnxRuntime InferenceSession. ) sess_options providersN) onnxruntimeZSessionOptionsZGraphOptimizationLevelZORT_DISABLE_ALLZgraph_optimization_levelZInferenceSessionrrr)r>rrrrrs z'CalibraterBase.create_inference_sessionrc Csdd|jjD}|dd|jjD|dd|jjDdd|jjD}t}tjtj h}|jj D]h}|j r|j |j krnt |j|jD]@}||kr||}|jdr|jjj|kr||kr||qqn||fS)z select input/output tensors of candidate nodes to calibrate. returns: tensors (set): set of tensor name. value_infos (dict): tensor name to value info. cSsi|] }|j|qSrname.0virrr sz>CalibraterBase.select_tensors_to_calibrate..cSsi|] }|j|qSrr)rZotrrrrscSsi|] }|j|qSrr)ritrrrrscSsh|] }|jqSrr)rinitrrr sz=CalibraterBase.select_tensors_to_calibrate.. tensor_type)graph value_infoupdateoutputinput initializersetr FLOATZFLOAT16noderop_type itertoolschainr<ZHasFieldr elem_typeadd) r>r value_infosrtensors_to_calibrateZtensor_type_to_calibrater tensor_namerrrrselect_tensors_to_calibrates&    z*CalibraterBase.select_tensors_to_calibratecCs|jS)zP return: augmented onnx model. Call after calling augment_graph rrErrrget_augment_modelsz CalibraterBase.get_augment_modelcCstdS)z abstract method: augment the input model to prepare for collecting data. It will: 1. augment the model to be able to collect desired statistics data 2. save augmented model to augmented_model_paths NrrrErrr augment_graphszCalibraterBase.augment_graph data_readercCstdS)z abstract method: collect the tensors that will be used for range computation. It can be called multiple times. Nrr)r>rrrr collect_data szCalibraterBase.collect_datarqcCstdS)ze abstract method: compute data based on the calibration method stored in TensorsData NrrrErrr compute_dataszCalibraterBase.compute_data)NrFFF)rHrIrJrrWrrrrBrrr rrrrjrrMrrrrrrs   " rcsjeZdZdeeefeeedfdd Zdd Z d d Z e d d dZ ddZ edddZZS)MinMaxCalibraterNrF{Gz?rc s|tj|||||| dg|_d|_t|jjj|_dd|jjjD|_ ||_ |rl|dksd|dkrlt d||_ ||_ dS)aw :param model_path: ONNX model to calibrate. It is a model path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param symmetric: make range of tensor symmetric (central point is 0). :param use_external_data_format: use external data format to store model which size is >= 2Gb :param moving_average: compute the moving average of the minimum and maximum values instead of the global minimum and maximum. :param averaging_constant: constant smoothing factor to use when computing the moving average. :param max_intermediate_outputs: maximum number of intermediate outputs before an intermediate range is computed. :param per_channel: whether to compute ranges per each channel. )rrrrrNcSsh|] }|jqSrrrrrrrr9sz,MinMaxCalibrater.__init__..rr z;Invalid averaging constant, which should not be < 0 or > 1.)superrBintermediate_outputscalibrate_tensors_ranger\rrrnum_model_outputsmodel_original_outputsmoving_averager9averaging_constantmax_intermediate_outputs) r>rrrrrrrrr __class__rrrBs"zMinMaxCalibrater.__init__csj\}}ttttjdgtj d}jj j |ddfdd}|D]}||d||dqbt jjjjd d S) z Adds ReduceMin and ReduceMax nodes to all quantization_candidates op type nodes in model and ensures their outputs are stored as part of the graph output :return: augmented ONNX model rcSs:|jD]}tj||jr|jSqtd|ddS)Nz&Model does not contain a version for 'z'.) opset_importonnxZdefshasdomainversionrc)rrrrrrget_op_versionKs  z6MinMaxCalibrater.augment_graph..get_op_versionc sd}|d|}|d}tjj||g|g||d}tjjd|g|g|d}ddjjjD}|d djjjD|d djjjD||kr||j j j }nt d |d j rNt||j j jj} dtd| } |jdkr|jtd| n@tt} ttj| tjd| } |j| jjj| jjj||gjjjt ||dgdS)Nr _Z_Reshape)r$rZReshape)inputsoutputsrcSsi|] }|j|qSrrrrrrrdszNMinMaxCalibrater.augment_graph..add_reduce_min_max..cSsi|] }|j|qSrr)rorrrrescSsi|] }|j|qSrrrirrrrfsz'Unable to guess tensor type for tensor zE, running shape inference before quantization may resolve this issue.rrPZaxesr)r)!rr Z make_noderrrrrrr<rrr9rr\rdimrange attributeappendZmake_attributerWuuiduuid4r from_arrayrarrayint64rrextendZmake_tensor_value_info) rZreduce_op_namer$Z reduce_outputintermediate_outputZ reduce_nodeZ reshape_noderZ onnx_typeZ tensor_rankZ reduced_axesZreduce_axes_nameZ reduce_axesrZreshape_shape_namer>rradd_reduce_min_maxQsF    z:MinMaxCalibrater.augment_graph..add_reduce_min_maxZ ReduceMinZ ReduceMaxZsave_as_external_dataN)rrrWrrr rrrrrrrrsaverr)r>ZtensorsrZ reshape_shapertensorrrrr@s .  zMinMaxCalibrater.augment_graphcCs g|_dSr]rrErrrclear_collected_datasz%MinMaxCalibrater.clear_collected_datarcCs|}|sqH|j|jd||jdk rt|j|jkr|qt|jdkrh|jdkrht d| }t |t st dt|d|dS)NrNo data is collected.z+compute_data must return a TensorsData not r5)rkrrrrunrr\rrr9rrVrMrXr<)r>rrtrrrrs  zMinMaxCalibrater.collect_datac Cs|s|S|D]\}}t|tr8|jd}|jd}n|\}}t||trl||jd}||jd}n ||\}}|jr||j||} ||j||} nt||} t||} t|tst||trt| | d||<q| | f||<q|S)Nrr rQ)r7rVr-rFrrminmax) r>Z old_rangeZ new_ranger`rdold_minold_maxnew_minnew_max min_value max_valuerrr merge_ranges(      zMinMaxCalibrater.merge_rangerqc stjdkrjSfddttjdDfddjD}i|D](}|D]\}}|g|q\qPjdfddtdtdD}fdd D}g}tdtdD]}jr t j ||dd } t j ||d dd } n0t j ||dd } t j ||d dd } j rxt j t | t | gdd } |t| | gq|t| | gqttjtt||} jrj| _n| _jS) z Compute the min-max range of tensor :return: dictionary mapping: {added node names: (ReduceMin, ReduceMax) pairs } rcsg|]}j|jqSrr get_outputsrrrErr sz1MinMaxCalibrater.compute_data..csg|]}tt|qSrr~ziprr output_namesrrrsNcsg|]}|ddqS)rr) rpartitionr)added_output_namesrrrsrPcs i|]}|jkr||qSr)rr)merged_output_dictr>rrrs z1MinMaxCalibrater.compute_data..r%r )r\rrrr7 setdefaultrrrrZmeanrrrabsrYrMrZr[r~rr) r>output_dicts_listdr@rAZcalibrate_tensor_namesZmerged_added_output_dictpairsrZmin_value_arrayZmax_value_arrayZmax_absolute_valueZnew_calibrate_tensors_ranger)rrrr>rrsB    zMinMaxCalibrater.compute_data)NrFFFrNF)rHrIrJrrWrrrrBrrrjrrrMr __classcell__rrrrrs"  )I!rc sbeZdZdeeefeeed fd d Zd d Z ddZ e dddZ e dddZZS)HistogramCalibraterNrF percentile-X@samerc svtj|||||dg|_d|_t|jjj|_dd|jjjD|_ d|_ ||_ ||_ ||_ | |_d|_| |_dS)a= :param model_path: ONNX model to calibrate. It is a model path. :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param use_external_data_format: use external data format to store model which size is >= 2Gb :param method: A string. One of ['entropy', 'percentile']. :param symmetric: make range of tensor symmetric (central point is 0). :param num_bins: number of bins to create a new histogram for collecting tensor values. :param num_quantized_bins: number of quantized bins. Default 128. :param percentile: A float number between [0, 100]. Default 99.99. :param scenario: see :class:`DistributionCalibrater` )rrrrNcSsh|] }|jqSrrrrrrrsz/HistogramCalibrater.__init__..)rrBrrr\rrrrr collectormethodnum_binsnum_quantized_binsrrscenario) r>rrrrrrrr rr rrrrBs$zHistogramCalibrater.__init__cCsV||j\|_}|jD]"}||jkr|jjj||qtj|j|j |j ddS)z make all quantization_candidates op type nodes as part of the graph output. :return: augmented ONNX model rN) rrrrrrrrrrr)r>rrrrrr!s  z!HistogramCalibrater.augment_graphcCs g|_dSr]rrErrrr1sz(HistogramCalibrater.clear_collected_datarcs|}|sq&jjd|qtjdkr.csg|]}tt|qSrrrrrrrBscs i|]}|jkr||qSr)rr) merged_dictr>rrrKs z4HistogramCalibrater.collect_data..)rrrr rr )rkrrrrr\r9rr7rrHistogramCollectorrrrr rr collectr)r>rrrrr@rAZclean_merged_dictr)r rr>rr4s4   z HistogramCalibrater.collect_datarqcCsh|jstdt|tr tj}n8t|tr2tj}n&t|trDtj }nt dt |dt ||j S)z Compute the min-max range of tensor :return: dictionary mapping: {tensor name: (min value, max value)} z9No collector created and can't generate calibration data.zUnknown calibrater z". This method must be overwritten.)rr9rVEntropyCalibraterrZrgPercentileCalibraterrhDistributionCalibraterrirXr<rMcompute_collection_result)r>calrrrrZs   z HistogramCalibrater.compute_data) NrFrFrrrr)rHrIrJrrWrrrrBrrrjrrMrrrrrrrs"  ,&rcs6eZdZd eeefeeedfdd ZZ S) rNrFr,rrc s tj||||||||ddS)a :param model_path: ONNX model to calibrate. It is a model path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param use_external_data_format: use external data format to store model which size is >= 2Gb :param method: A string. One of ['entropy', 'percentile', 'distribution']. :param symmetric: make range of tensor symmetric (central point is 0). :param num_bins: number of bins to create a new histogram for collecting tensor values. :param num_quantized_bins: number of quantized bins. Default 128. )rrrr NrrB) r>rrrrrrrr rrrrBnszEntropyCalibrater.__init__)NrFr,Frr rHrIrJrrWrrrrBrrrrrrms  rcs6eZdZd eeefeeedfdd ZZ S) rNrFrrrrc s tj||||||||ddS)a :param model_path: ONNX model to calibrate. It is a model path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param use_external_data_format: use external data format to store model which size is >= 2Gb :param method: A string. One of ['entropy', 'percentile', 'distribution']. :param symmetric: make range of tensor symmetric (central point is 0). :param num_quantized_bins: number of quantized bins. Default 128. :param percentile: A float number between [0, 100]. Default 99.99. )rrrrNr) r>rrrrrrrrrrrrBszPercentileCalibrater.__init__)NrFrFrrrrrrrrs  rcs6eZdZd eeefeeedfdd ZZ S) rNrF distributionrrrc stj|||||||ddS)a :param model_path: ONNX model to calibrate. It is a model path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param use_external_data_format: use external data format to store model which size is >= 2Gb :param method: A string. One of ['entropy', 'percentile', 'distribution']. :param symmetric: make range of tensor symmetric (central point is 0). :param num_bins: number of bins to create a new histogram for collecting tensor values. :param scenario: for float 8 only, if `scenario="same"`, the algorithm weights and float 8 follow the same distribution, if `scenario="p3"`, it assumes the weights follow a gaussian law and float 8 ~ X^3 where X is a gaussian law )rrr Nr)r>rrrrrrr rrrrBszDistributionCalibrater.__init__)NrFrrrrrrrrrs  rc@s,eZdZdZejddZejddZdS)CalibrationDataCollectorzL Base class for collecting data for calibration-based quantization. cCstdS)z Generate informative data based on given data. name_to_arr : dict tensor name to NDArray data Nrrr> name_to_arrrrrr sz CalibrationDataCollector.collectcCstdS)z? Get the optimal result among collection data. NrrrErrrrsz2CalibrationDataCollector.compute_collection_resultN)rHrIrJ__doc__r|r}r rrrrrrs  rc@sveZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ e dddZddZddZdS)r a` Collecting histogram for each tensor. Percentile and Entropy method are supported. ref: https://github.com//apache/incubator-mxnet/blob/master/python/mxnet/contrib/quantization.py ref: https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/_modules/ pytorch_quantization/calib/histogram.html cCs.i|_||_||_||_||_||_||_dSr])histogram_dictrrrr rr )r>rrrr rr rrrrBszHistogramCollector.__init__cCs|jSr])rrErrrget_histogram_dictsz%HistogramCollector.get_histogram_dictcCsNtd|jdkr||S|jdkrB|jr6||S||SntddS)Nz/Collecting tensor data and making histogram ...>r,rrDOnly 'entropy', 'percentile' or 'distribution' methods are supported)printr collect_valuercollect_absolute_valuer9rrrrr s     zHistogramCollector.collectcCs|D]p\}}t|tr|D](}t|tjs tdt|d|q tdd|D}t|dks|td|d|t |}n*t|tjst dt|d|n|}| }|j dkrt |}t|}n tjd|jd}tjd|jd}t|}||jkrZtj||jd \} } | |j} |jtjksFtd | | ||f|j|<q|j|} | d } | d } t| d stdt| t| d stdt| | d}| d}t|}||dkr|d|d}t|d||||}t||f}tj||d \} } | |j} | dt||7<|jtjks\td | | t | |t| |f|j|<qdS)z5 Collect histogram on absolute value r6z for tensor=css|] }|jVqdSr]r)rarrr sz.r z6The calibration expects only one element type but got rr)r4zMonly float32 or float16 is supported, every constant must be explicitly typedrPrSrz'old_min should be a numpy array but is rN)r7rVlistrndarrayr&r<rr\r'r9flattensizerrrrabsoluter histogramrr(float64r;ZarangeZhstack)r>rrdata_arrZarrZdtypesZ data_arr_nprrr2r3 old_histogramrrold_histold_hist_edgesZ temp_amaxwidthZ new_bin_edgesrrrrs` &              z)HistogramCollector.collect_absolute_valuec Cs|D]\}}t|}|}|jdkrBt|}t|}n tjd|jd}tjd|jd}tjtt |t ||jd}||j kr|j |}| ||||||j |<qtj ||j | |fd\}} || |||f|j |<qdS)z1 Collect histogram on real value rrrN)r7rr'r$r%rrrrrrmerge_histogramr'r) r>rrr)rr thresholdr*r2r3rrrrHs2       z HistogramCollector.collect_valuecCs|\}}}} } || krRtj|t|| | fd\} } | ||t||t| || fS| dkrtj|t|| |fd\} }| |7} nrt|}d| |}t|| |d}|d|}||| }tj||| |fd\} }| ||||7<| |t||t| ||fSdS)Nr.rrPr )rr'r\rrr)r>r*r)rrZ new_thresholdr+r,rrZ old_thresholdZnew_histrr2r3Z old_num_binsZ old_strideZhalf_increased_binsZ new_num_binsrrrr/hs2    z"HistogramCollector.merge_histogramcCsp|jrt|jdkrtdtd|jd|jdkr@|S|jdkrR|S|jdkrd|StddS) Nrz=Histogram has not been collected. Please run collect() first.z0Finding optimal threshold for each tensor using z algorithm ...r,rrr)rr\r9rrcompute_entropycompute_percentilecompute_distributionrErrrrs   z,HistogramCollector.compute_collection_resultcCs|jdks|jdkrtd|j}|j}i}tdt|td|jtdd|d|d |D]X\}}|d}|d }|}t ||} |j rt | |d} tj || |j d  tj || |j d f||<nTd|d } t | d | } t | | } tj || |j d tj || |j d f||<|d} |d}||d| krp| ||d f||<||d |kr||d|f||<|||dd||<tjdddkrpt||qp|S)Nrdzrrthresholds_dictrr'r2r3totalZcdfZ idx_rightZpercent_to_cut_one_sideZidx_leftrrrrrr2sD    z%HistogramCollector.compute_percentilecCs|j}|j}i}tdt|td|jdtd|j|D]T\}}|||}|||<||dd||<tj dddkrLt |d|d qL|S) Nr5r6z: (The number may increase depends on the data it collects)zNumber of quantized bins : rPr9rr:r ) rr rr\rr7get_entropy_thresholdr<r=r>r)r>rr r?rr'optimal_thresholdrrrr1s z"HistogramCollector.compute_entropyr cCs|dkrtd|d|dd|ddd}|dkr|||}||d||dd}tj||jdtj||jdfSt||krt|ddkr||||}||||d|d}tj||jdtj||jdfSt||}d|t|<d|t|<t|||}|||}||d||dd}tj||jdtj||jdfS) Nrzpower=z <= 0 is invalid.rr g?rPr) r9r*rrrrrisnanisinf)r2r3powerrfr.r/Zfactrrr_avg_stds$$  $ $zHistogramCollector._avg_stdc Cs:|jdkrtd|j}i}tdt|td|jtd|jd|D]\}}|d}|d}|jtj kst |jd kr|j ||dd \}}n(|jd kr|j ||d d \}}ntd |jtj kst |jtj kst |jtj kst t ||||| |d||<tjdddkrXt||qX|S)Niz3Invalid num_bins. Must be in range 512 <= num_bins.r5r6z Scenario : r8rr r)rEZp3gUUUUUU?z,Invalid scenario. Must be in {'same', 'p3'}.)r.r/r2r3r0r1r9r:)rr9rrr\r r7rrr(r&rFr-rrr<r=r>r) r>rr?rr'r2r3Zavg_coefZstd_coefrrrr3s<     z'HistogramCollector.compute_distributioncs|d}|d}|j}|d}|d}|djt||d}fddt|jD} t||ddD]} || } t|| d|} || || f| | |<t|| | } | }t|d| }t|| d}|d|7<|d|7<|dk tj }tj|tj d}| j|}t|D]*}||}||}t| ||||<q.|dt| ||d7<tj|jtj d}t|D]D}||}||}t|||}|dkr||||||<qt |}t |}|dks|dkrtj tj d}ntj t||d}||| |<qjt|}| |}|d}|d }|d|krn||df}|d|kr|d|f}t|dd stt|dd st|S) aFGiven a dataset, find the optimal threshold for quantizing it. The reference distribution is `q`, and the candidate distribution is `p`. `q` is a truncated version of the original distribution. Ref: http://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf rr rPcs(g|] }tjddtjddfqS)rr)rrrrrrr%sz.NrrrSr)r%rrZzerosrrcopydeepcopyr*r(rrrrr,Zargminr;r&)r>r'r r2r3rZzero_bin_indexZnum_half_quantized_binZ kl_divergenceZ thresholdsrrxryZsliced_distributionpZleft_outliers_countZright_outliers_countnonzerosZquantized_binsZnum_merged_binsindexstartendqZnormdivZmin_kl_divergence_idxrBrrrrrrAsd         z(HistogramCollector.get_entropy_thresholdN)r )rHrIrJrrBrr rrr/rr2r1 staticmethodrFr3rArrrrr s :  . (r rF)rrc CsVd}|tjkrf|dd}|dd}|dd} |dd} |dd} t||||||| | | d }n|tjkr|d d } |d d } |dd}t|||||| | d }n~|tjkr|d d } |dd}|dd}t|||||| |d}n8|tjkr*|d d } |dd}t ||||| |d}|rD| | |St d|dS)NrFrrrrr)rrrrrrrrr )rrrr rrrT)rrrrr r)rrr zUnsupported calibration method ) rZr[r>rrgrrhrrirrrr9)rrrZcalibrate_methodrZ extra_optionsZ calibratorrrrrrrr rr rrrcreate_calibratorrsx                  rQ)Nr)0r|rGrr<renumrpathlibrtypingrrrrrnumpyrrr r r r rZ quant_utilsrrrr#r floatrr,r-rMrZABCMetarjrrrrrrrr r[rWrQrrrrs^   (n_y""#