U T?h0@sddlmZddlmZmZddlZddlmZmZddlm Z m Z m Z m Z ddlm ZddlmZeeZGdd d ZGd d d ZdS) ) getLogger)OptionalTupleN) array_equalndarray) NodeProto TensorProtohelper numpy_helper)onnx_pb) OnnxModelc@seZdZedddZeeeefdddZd-edd d Z d.ee e ee ed d dZ edddZ edddZeddZed/edddZed0edddZeejdddZed1eedd d!Ze d"d#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd S)2 FusionUtilsmodelcCs ||_dSNr)selfrrW/var/www/html/venv/lib/python3.8/site-packages/onnxruntime/transformers/fusion_utils.py__init__szFusionUtils.__init__) input_namereturncCsn|j|}|dk rL|jjjtjkrL||\}}t d|dd|fSt d|d|dk d|fS)NzCasted graph input z to int32TzDid not cast graph input z to int32: found F) rZfind_graph_inputtypeZ tensor_typeZ elem_typerINT32cast_input_to_int32loggerdebug)rrZ graph_inputZ cast_output cast_noderrrcast_graph_input_to_int32s z%FusionUtils.cast_graph_input_to_int32int32)rcCsf|d|}|dkr ttj}n0|dkr4ttj}n|dkrHttj}ntd||||}||fS)N_rZfloat32Zfloat16z"Invalid target_type: {target_type})intrrFLOATZFLOAT16 ValueError add_cast_node)rrZ target_type output_nameto_typerrrr cast_inputs    zFusionUtils.cast_inputN)rr%r$ graph_namec Cs|dkr|d|}|g}|dkr.|j}||krX||}|rX|jdkrX|jdg}tjd||gd}|jtd|g|jj ||d|S)NZ _cast_to_Castr)inputsoutputsto)r') routput_name_to_nodeop_typeinputr Z make_node attributeextendZmake_attributeadd_node) rrr%r$r,r'r) parent_noderrrrr#/s  zFusionUtils.add_cast_nodecCs ||dS)Nr)r&)rrrrrrJszFusionUtils.cast_input_to_int32cCs|j}||}|D]h}|jdkrd}|jD]&}|jdkr.|jttjkr.d}qVq.|r|j d}|j ||j ||qdS)Nr(Fr+Tr) rinput_name_to_nodesr-r/nameir rroutput remove_nodereplace_input_of_all_nodes)rrr3nodesnodeZis_int32Zattr$rrrremove_cast_int32Ms     zFusionUtils.remove_cast_int32cCszd}|j||krJ|||j|krJ||j||t||j|}||j|<||krl|||n |g||<|S)Nr)r.removelenappend)r:r5new_input_namer3old_input_referencerrrupdate_node_input\s   zFusionUtils.update_node_inputrc Cs<|j|}|j|}t||||}|dko6|| } | S)a Before: (input)-->parent-->node-->(output) After: (input)-->parent--> | +----->node-->(output) This function returns a flag whether the parent node can be removed. r)r.r rAZfind_graph_output) rr:r2r3Znode_input_indexZparent_input_indexZold_input_namer?r@Zparent_can_be_removedrrr skip_parentls  zFusionUtils.skip_parent)attribute_namecCsV|}|jD]}|j|kr t|}q t|trJt|ttfoHt||ddS||kSdS)aVerify that a node has expected value for an attribute. Args: node (NodeProto): a node to check attribute_name (str): name of attribute expected_value (Any): expected value of the attribute default_value (Any, optional): default value if the attribute does not exist. Defaults to None. Returns: bool: whether the check is passed or not FZ equal_nanN)r/r4r Zget_attribute_value isinstancelistrr)r:rCexpected_value default_valuevalueattrrrrcheck_node_attributes     z FusionUtils.check_node_attribute)tensorcCst|tjstdt|t|jdks:|jtjjkrBtd|j r~t t j |j dd|j}t |ddg}||_ ntd|S) zTranspose a 2-D INT8 TensorProto Args: tensor (TensorProto): tensor to be transposed Returns: tensor (TensorProto): transposed tensor z3Expected input type is an ONNX TensorProto but got z'Only INT8 2-D tensors can be transposedZint8)dtyperzonly raw buffer supported)rE onnx_protorr"rr=dims data_typeZINT8raw_datanumpyZreshapeZ frombufferZ transposetobytes)rLZ int32_dataZint32_transposed_datarrrtranspose_2d_int8_tensors  z$FusionUtils.transpose_2d_int8_tensorT)r:rcCs|jdkrtd|j||jd}|dkr8dS|jdkpX|jdkoX|jddk}|rf|sfdSt|jdkrxdS||jd}|j|jkrdS|dkrdSt |dkS) aVerify if a provided QuantizeLinear (Q) / DequantizeLinear (DQ) node is a good candidate for fusion. It is a good candidate for fusion if: (1) The Q/DQ node is for per-tensor quantization if allow_per_tensor_quantization_only is `True` (2) The Q/DQ node should have constant scale (3) The Q/DQ node should have a zero point of 0 Args: node (NodeProto): a Q/DQ node to check Returns: bool: whether the check is passed or not >DequantizeLinearQuantizeLinearz+Provided node is not a Q/DQ node. Op Type: rONFrrMT) r-rrget_constant_valuer.ndimshaper=rTall)r:rZ"allow_per_tensor_quantization_onlyscaleZscale_has_single_elementZ zero_pointrrrcheck_qdq_node_for_fusions " z%FusionUtils.check_qdq_node_for_fusion) input_indexcCsVt|j|kst|j|j|}t|trJt|ttfoHt||ddS||kSdS)a7Verify that a node has expected input value Args: node (NodeProto): a node to check input_index (int): index of its input to be verified expected_value (Any): expected value of the input Returns: bool: whether the check is passed or not FrDN) r=r.AssertionErrorrrYrErFrr)rr:r_rGrIrrrcheck_node_input_values  z"FusionUtils.check_node_input_valuecCsg}|j}|jD]@}|jdkr|jd|kr|j|jd|jd||q|r|j|t dt |ddS)z>Remove Identity nodes, except those right before graph output.ZIdentityrzRemoved z Identity nodesN) rget_graphs_output_namesr9r-r6r8r.r>Z remove_nodesrinfor=)rnodes_to_removegraph_output_namesr:rrrremove_identity_nodess    z!FusionUtils.remove_identity_nodescCs|jdSr)rremove_cascaded_cast_nodesrrrrrgsz&FusionUtils.remove_cascaded_cast_nodescCs|jdSr)rremove_useless_cast_nodesrhrrrrisz%FusionUtils.remove_useless_cast_nodescCsB|jjdd}|dkrdSg}|jD]`}|jdkr(||jd}||jd}|r(|r(||kr(td|j d|| |q(|r>t |j }t |j }|D]}tt |j|@rtt |j|@st|j|jddkr|j|jd|jdnqn|j|jd|jd|j|qdS) ziRemove reshape node that is not needed based on symbolic shape inference: input and output has same shapeT)updateNZReshaperzRemove reshape node z* since its input shape is same as output: rO)rZinfer_runtime_shaper9r-Zget_edge_shaper.r6rrcr4r>setZget_graphs_input_namesrbboolr=r3Zreplace_output_of_all_nodesr8r7)rZ shape_inferrdr:Z input_shapeZ output_shapeZgraph_input_namesrerrrremove_useless_reshape_nodess4  z(FusionUtils.remove_useless_reshape_nodes)r)NNN)rr)N)T)__name__ __module__ __qualname__r rstrrrlrr&r rr#rr; staticmethodrArBrKrPrrVrr^rarfrgrirmrrrrr s:   *r c@s$eZdZedeeedddZdS) NumpyHelperF)rL fill_zerosrcCs0|r&ddlm}t|j|j|jdSt|S)Nr)mapping)r[rN)onnxrurrQZTENSOR_TYPE_TO_NP_TYPErRr to_array)rLrtrurrrrw's  zNumpyHelper.to_arrayN)F)rnrorprrrrlrrwrrrrrs&srs)loggingrtypingrrrTrrrvrrr r r rPZ onnx_modelr rnrr rsrrrrs