U U?hd#@s6ddlZddlZddlZddlmZmZddlZddlmZ ddl Z ddl m Z m Z mZmZddlmZddlmZddlmZeeZGdd d Zd d Zed kr2eZejreejejZ ej!Z"ej#$e"re%d e"de&d e"de 'e Z(ee(ej)ej*ej+dZ,e,-e,j(.e"ddS)N)ListTuple) GraphProto ModelProto NodeProto TensorProto)quantize_matmul_bnb4) ONNXModel)attribute_to_kwargc@seZdZdZdZdZdeeedddZe e e e e e fdd d Zejejd d d Zee e edddZe e dddZddZdS)MatMulBnb4QuantizerzMPerform 4b quantization of constant MatMul weights using FP4 or NF4 data typerr N)model quant_type block_sizecCs@|pg}|tjtjfkstt||_||_||_t||_ dS)N) r FP4NF4AssertionErrorr r rrsetnodes_to_exclude)selfr rrrr`/var/www/html/venv/lib/python3.8/site-packages/onnxruntime/quantization/matmul_bnb4_quantizer.py__init__&s  zMatMulBnb4Quantizer.__init__) graph_pathreturncCsLtt|dddD]2}||}|jD]}|j|kr&||fSq&qdS)Nr )NN)rangelen initializername)rrgidgraphZtensorrrrZ__get_initializer.s   z%MatMulBnb4Quantizer.__get_initializer)fpweightrc Cst|jdkrtd|}|j\}}||}|j}||d|}|dd}tj|dd} tj||jd} t | || ||j ||| | fS)z4b quantize fp32/fp16 weightz9Current bnb4 block quantization only supports 2D tensors!r Zuint8)dtype) rshape ValueErrorZ transposecopyrnpZzerosr$rr) rr"Z fpweight_trowscolsZnumelrZ num_blocksZquantized_numelpackedabsmaxrrrbnb4_block_quant7s   z$MatMulBnb4Quantizer.bnb4_block_quant)node graph_stackrcCs|jdkr|Std|jd|j|jkrFtd|jd|S|jd}t||\}}|dkrvtd|Stj |}t |j d krtd |S| |\}}tj |} |jd | _|jD]} | j|kr|j| qqtj |} |jd | _|j| | gi} |j \} }| | d <|| d<|j| d<|j| d<tjjd|jd| j| jg|jdg|jrz|jd nddd| }td|jd|S)zdIf the node is MatMul with fp32 const weight, quantize the weight with int4, and return the new nodeZMatMulzstart to quantize z ...zexclude to quantize z$ as specified by nodes_to_exclude...r Nz2MatMul doesn't have const weight. Skip to quantizer#z)MatMul weight is not 2D. Skip to quantizeZ_Bnb4Z_absmaxKNrr MatMulBnb4r com.microsoft)inputsoutputsrdomainzcomplete quantization of )r2)op_typeloggerdebugrrinputr %_MatMulBnb4Quantizer__get_initializeronnxZ numpy_helperZto_arrayrr%r-Z from_arrayremoverextendrrhelper make_nodeoutput)rr.r/ZinputBBZBs_graphZB_arrayr+r,ZB_quantr;Z absmax_tensorkwargsr)r*Zmatmul_bnb4_noderrr_bnb4_matmul_node_weightMsV                  z,MatMulBnb4Quantizer._bnb4_matmul_node_weight)r/c Csg}|d}|jD]}dd|jD}t|ri}|jD]}|jtjjkrh||j|j | |i}nN|jtjj krg} |j D] } || | | |gq|j | i}nt|}||q8tjj|j|j|jfd|j i|}||||q|d|j |||S)NrcSs,g|]$}|jtjjks$|jtjjkr|qSr)typer=AttributeProtoGRAPHGRAPHS).0attrrrr sz9MatMulBnb4Quantizer._process_subgraph..rr.)r. attributerrFr=rGrHappendgr_process_subgraphrIZgraphsr?r updater@rAr8r;rBrEZ ClearFieldpop) rr/Z new_nodesr!r.Z graph_attrsrDrKkvvalueZsubgraphrrrrPsD         z%MatMulBnb4Quantizer._process_subgraphcCsd|jg}|j}d}|D]}|jdkrd}q|sL|tjddg|||j dS)NFr4Tr ) r r! opset_importr7r?r=r@Z make_opsetidrPZclean_initializers)rr/rUZ has_ms_domainZopsetrrrprocesss    zMatMulBnb4Quantizer.process)N)__name__ __module__ __qualname____doc__rrrintr staticmethodrrrrr<nptZ ArrayLiker(Zndarrayr-rrErPrVrrrrr s7&r cCstjdd}|jdddd|jdddd|jd d d tjtjgd d |jdd ddd|jddd dd|jd d|jddtd gdd|S)NaBlockwise FP4/NF4 quantization for MatMul 2D weight matrices. A weight matrix is partitioned into blocks, where each block is a contiguous subset inside the flattened transposed weight matrix. Each block is quantized into a set of 4b integers with an absolute value scaling factor. ) descriptionz --input_modelTzPath to the input model file)requiredhelpz--output_modelzPath to the output model filez --quant_typeFr z&Quantization data type. 0: FP4, 1: NF4)r_defaultchoicesr`z --block_size@zVBlock size for blockwise quantization. Note: bnb.nn.Linear4bit only uses block_size=64)r_rar`z-vz --verbose store_true)r_action)verbosez--nodes_to_exclude+zBSpecify the nodes to be excluded from quantization with node names)nargsrFr_rar`) argparseArgumentParser add_argumentr rr set_defaultsstr parse_args)parserrrrrns:    rn__main__zfile z already exists)rT)/riloggingostypingrrnumpyr(Z numpy.typingr]r=Z onnx.onnx_pbrrrrZonnxruntime.capi._pybind_staterZ onnx_modelr Z quant_utilsr getLoggerrWr9r rnargsrfsetLevelDEBUGZ input_modelZinput_model_pathZ output_modelZoutput_model_pathpathexistserror Exceptionloadr rrrZquantrVZsave_model_to_filerrrrs6     "'