U U?h< @sdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl Z ddlZddlmZmZddlZddlmZmZddlmZdd lmZmZmZmZmZmZd ZeeZ d(e e!efe e!efe e e!e"dd d dZ#d)e!ee e e!ee!e e j$fdddZ%edZ&ee!ee!e e j$ffe!e e e j$e e e j$ddddZ'd*ee!e e j$fe ee!e e j$fee!ee!e e j$ffdddZ(e j$e j$e j$e)e e j$dddZ*e!e!ee!ee!e j$ffdddZ+e e e j$e j$fe e e j$e j$fe,dd d!Z-e-fee!ee!e j$ffee j$e j$ge,fee!e,fd"d#d$Z.e-fee!ee!e e j$ffee e j$e e j$ge,fee!ee!e,ffd%d&d'Z/dS)+aUtilities to run a given ONNX model, while saving input/output tensors of eligible operator nodes. A use case is to debug quantization induced accuracy drop. An AI engineer can run the original float32 model and the quantized model with the same inputs, then compare the corresponding activations between the two models to find where the divergence is. Example Usage: ```python class ExampleDataReader(CalibrationDataReader): def __init__(self): ... def get_next(self): ... input_data_reader = ExampleDataReader() augmented_model_path = str(Path(self._tmp_model_dir.name).joinpath("augmented_model.onnx")) modify_model_output_intermediate_tensors (path_to_onnx_model, augmented_model_path) tensor_dict = collect_activations(augmented_model_path, input_data_reader) ``` `tensor_dict` points to a dictionary where the keys are tensor names and each value is a list of tensors, one from each model run N)Path)CallableDictListOptionalSequenceUnion)helper numpy_helper)CalibraterBaseCalibrationDataReader) ONNXModel)DEQUANT_OP_NAMEDEQUANT_OUTPUT_SUFFIXQUANT_INPUT_SUFFIXTENSOR_NAME_QUANT_SUFFIX find_by_nameload_model_with_shape_inferZ_ReshapedSavedOutputF)input_model_pathoutput_model_pathop_types_for_savingsave_as_external_datareturncCs|dkr g}t||d}|j}||\}}dtt}ttjdgtj d|} |j j | |D]\} | t } tjjd| |g| g| d} |j j | t| || jjjdg} |j j | qhtj|||ddS) aAugment a given ONNX model to save node input/output tensors. Add all input/output tensors of operator nodes to model outputs so that their values can be retrieved for debugging purposes. Args: input_model: the path to load the model. op_types_for_saving: Operator types for which the input/output should be saved. By default, saving all the float32/float16 tensors. Returns: The augmented ONNX model N)Zop_types_to_calibrateZLinearReshape_ZdtypeZReshape)inputsoutputsname)r)r modelZselect_tensors_to_calibratestrtimer Z from_arraynumpyarrayZint64graph initializerappend_TENSOR_SAVE_POSTFIXonnxr Z make_nodenodeZmake_tensor_value_infotypeZ tensor_typeZ elem_typeoutputsave)rrrrZsaverZmodel_to_augmenttensorsZ value_infosZreshape_shape_nameZ reshape_shape tensor_nameZreshape_outputZ reshape_nodeZreshape_output_value_infor/Y/var/www/html/venv/lib/python3.8/site-packages/onnxruntime/quantization/qdq_loss_debug.py(modify_model_output_intermediate_tensors@s8  r1)augmented_model input_readerexecution_providersrc Cs|dkrt}tjj|_|dkr(dg}tj|||d}g}|D]}||d|q@|sdtdi}| }|D]F} t || D]6\} } | j t r| j dt } || g| qqt|S)aRun augmented model and collect activations tensors. Args: augmented_model: Path to augmented model created by modify_model_output_intermediate_tensors () input_reader: Logic for reading input for the model, augmented model have the same input with the original model. session_options: Optional OnnxRuntime session options for controlling model run. By default graph optimization is turned off execution_providers: Collection of execution providers for running the model. Only CPU EP is used by default. Returns: A dictionary where the key is tensor name and values are list of tensors from each batch NZCPUExecutionProvider)Z sess_options providersz3No data is collected while running augmented model!) onnxruntimeZSessionOptionsZGraphOptimizationLevelZORT_DISABLE_ALLZgraph_optimization_levelZInferenceSessionr&run RuntimeError get_outputsziprendswithr'_TENSOR_SAVE_POSTFIX_LEN setdefault) r2r3Zsession_optionsr4Zinference_sessionZintermediate_outputsZinput_dZ output_dictZ output_infobatchr+Z output_dataZ output_namer/r/r0collect_activationsss.  r?Z_1)qdq_cmpactivation_namepre_qdq_tensorspost_qdq_tensorsrcCs4|dk r0|dk r0i||<|||d<|||d<dS)Npre_qdqpost_qdqr/)r@rArBrCr/r/r0_add_pre_post_qdq_pairs rF)qdq_activationsfloat_activationsrc Csi}|D]\}}|trN|dtt }||}|}t||||q |tr|dtt }||}|}t||||q |tr |dtt }||}|}t||||q |s|S|D]"\}} ||} | dk r| | d<q|S)aComparing activation values to help debugging accuracy loss due to quantization. This functions takes saved activations from the QDQ model and (optionally) the float point model, and provides a data structure for comparing: * from the qdq model, activation values before and after QDQ operation * across both models, activations from the orignal model vs the corresponding activations in the QDQ model Arg: qdq_activations: Output of `collect_activations`. This must be from a quantized model with QDQ format. float_activations: Output of `collect_activations`. This must be from the float point model. Returns: Dict for comparing pre and post quantized activation tensors. E.g. ``` qdq_cmp = cmp_qdq_input_output(qdq_activations) print(qdq_cmp['activation1']['pre_qdq'][0]) print(qdq_cmp['activation1'][`post_qdq'][0]) qdq_cmp = cmp_qdq_input_output(qdq_activations, float_activations) print(qdq_cmp['activation1']['float'][0]) print(qdq_cmp['activation1']['pre_qdq'][0]) print(qdq_cmp['activation1'][`post_qdq'][0]) ``` Nfloat)itemsr;rlengetrFr_POST_QDQ_POSTFIX1) rGrHr@r.r-Zpre_namerCrBZact_nameZ act_valuesZ float_actsr/r/r0create_activation_matchings0!        rN) weight_tensor weight_scale weight_zp channel_axisrc Cs|j|jkst|jdkr&|||S|jdks4tt|j}d||<|j|}d}t|D]^}|||}|||||} |dkrt|  |}q\t|  |} t || f|}q\|dkrdS| |j|S)Nr r) shapeAssertionErrorsizendimlistrangeZtaker"Zasarrayreshape concatenate) rOrPrQrRZ reshape_dimsZ channel_countZdequantized_weightsiZper_channel_dataZdequantized_per_channel_dataZchannel_weightsr/r/r0_run_dequantize_linears&       r\)float_model_pathqdq_model_pathrcCsttt|}ttt|}i}|}|D]}|jtkrFq4|jd}t||}|s`q4| t st d|d|dq4d} |j D]} | jdkr| j} qt|} tt|jd|} t|jdkrtt|jd|} ntj| jtjd } | j| jkrdkr4nn| t} | t} | j| jkrZtd | jd | jt| | | | d }|d tt  }|d krt d|d|dq4t||}|st d|d|dq4t|}||d||<q4|S)aComparing weight values to help debugging accuracy loss due to quantization. This functions takes the float model and the qdq model, and provides a data structure for comparing their corresponding weights to locate quantization errors Arg: float_model_path: Path points to the float point model. qdq_model_path: Path points to the qdq model. Returns: Dict for comparing weight tensors. E.g. ``` qdq_weight_cmp = create_weight_matching(float_model, qdq_model) print(qdq_weight_cmp['activation1']['float']) print(qdq_weight_cmp['activation1']['dequantized']) ``` rzModel Error in 'z': Dequantized tensor name 'z' not recognized!raxisr rz2scale and zero_point must have the same shape but z != )rRNz': 'z'' per-channel quantization on 0 channelz': weight tensor 'z ' not found!)rI dequantized)rrrr%ZnodesZop_typerinputrr;rloggingerror attributerr[r Zto_arrayrKr"ZzerosrSZint32rUrYtupler8r\)r]r^Zfloat_onnx_modelZqdq_onnx_modelZmatched_weightsZ initializersr) weight_nameZ weight_valuesr_attrrOrPrQZ weight_quantZ float_valuesZ weight_floatr/r/r0create_weight_matchingsT            ri)xyrc Cst|tjr|g}n|}t|tjr,|g}n|}t|t|krHtdt|}t|}tdj}t tj ||}t tj |||}||} dt | S)Nz%Unequal number of tensors to compare!rI) isinstancer"ndarrayrKr8rZflattenZfinfoepsmaxZlinalgZnormmathlog10) rjrkZxlistZylistleftrightepsilonZ tensor_normZ diff_normresr/r/r0*compute_signal_to_quantization_noice_ratioUs   rx) weights_matcherr_funcrcCs0i}|D]\}}||d|d||<q |S)NrIrarJ)ryrzresultrgZ weight_matchr/r/r0compute_weight_errormsr})activations_matchrzrcCsZi}|D]H\}}i}||d|d|d<|d}|rL|||d|d<|||<q |S)NrDrEZqdq_errrIZ xmodel_errr{)r~rzr|rmatchZ err_resultZfloat_activationr/r/r0compute_activation_errorws r)NF)NN)N)0__doc__rcrrr!pathlibrtypingrrrrrrr"r(r r r6Z calibrater r Z onnx_modelrZ quant_utilsrrrrrrr'rKr<r boolr1rnr?rMrFrNintr\rirIrxr}rr/r/r/r0s        6  2    ? $H