U T?h<@sddlZddlZddlZddlZddlmZddlZddlZddlmZm Z m Z m Z m Z ddl mZmZmZmZddlmZddlmZddlmZddlmZed Zdd d Zd d ZedkreZe ej eedS)N)datetime) Precisioncreate_onnxruntime_sessionget_ort_environment_variablesprepare_environment setup_logger)DEFAULT_TOLERANCE MODEL_CLASSESPRETRAINED_GPT2_MODELS Gpt2Helper)version)QuantizeHelper) AutoConfig) __version__c Cs@t}|jdddtddtd|jddtd ttd dtd |jd dtt j d ddd|jddtt j d ddd|jdddt dd|jdddddd|jdddddd|j dd |jd!t d"dd"d#d$gd%d&|jd'ddd(d|j dd)|jd*d+t t jtt d,d-|jd.ddd/d|j dd0|jd1d2d3t d#gd4d5|jd6d3t d#gd7d5|jd8d9d3t d:d;dd?gd@d5|jdAdBdddCdD|jdEdt dFdGd|jdHdddI|j ddJ|jdKdddI|j ddL|jdMdddI|j ddN|jdOdddI|j ddP||}|S)QNz-mz--model_name_or_pathTz;Model path, or pretrained model name selected in the list: z, )requiredtypehelpz --model_classFZGPT2LMHeadModelz!Model type selected in the list: )rrdefaultchoicesrz --cache_dir.Z cache_modelsz%Directory to cache pre-trained models)rrrrz --onnx_dirZ onnx_modelszDirectory to store onnx modelsz --test_timesdz8Number of repeat times to get average inference latency.)rrrrz-vz--validate_onnx store_truezValidate ONNX model)ractionrz-oz--optimize_onnxz'Use optimizer.py to optimize onnx model) optimize_onnxz--stagera6Stage in generation: 1 (initial decoder), 2 (decoder), 0 (both). 1 - decode the first token when past_sequence_length is zero; 2 - decode the remaining tokens when past_sequence_length is not zero; 0 - one onnx model for both stages 1 and 2. Note that we will optimize 1 and 2 differently for best performance.)rrrrrz --use_gpuzuse GPU for inference)use_gpuz-pz --precisionzfPrecision of model to run. fp32 for full precision, fp16 for half precision, and int8 for quantization)rrrrz --torchscriptzuse Torchscript) torchscriptz-bz --batch_sizes+z batch size)nargsrrrz--sequence_lengthsz!sequence lengths (excluding past)z-sz--past_sequence_lengths @zpast sequence lengthsz-rz --result_csvz$CSV file for saving summary results.)rrrz --thread_numzThreads to usez--include_copy_output_latency)rr)include_copy_output_latencyz --verbose)verbosez--output_torch_latency)output_torch_latencyz--disable_io_binding)disable_io_binding)argparseArgumentParser add_argumentstrjoinr listr keysospathint set_defaultsrFLOAT32 parse_args)argvparserargsr<e/var/www/html/venv/lib/python3.8/site-packages/onnxruntime/transformers/models/gpt2/benchmark_gpt2.pyparse_arguments!s                r>c!Cstttdkrtdtd||jtjkrL|j rD|j sLt d|jtj krf|j rft d|j dkr|jdgkst dt|jdkrtjd d n|jttj|j}|j}t|||j t|jd}t}tj|j|j |d }|j|j||d }t!|j rd nd}|"||j#dk}|j$||j|jd |d} | d} t|jd} |j%||| |j&|| | d|j s|jtj'kr$| |jtj krt(|jnd} |j | d| |jtjk|j)j*|j)j+|d |j d|jtj kr$tdt,-| | d|t,.|}td| d} |j r@|j |||| | d}t/| |j d|j|j&d} | dkrfdS|0t1|j2t1|jt1|j3||j} |4| ||jtjk}|j5pd6t789d}t:|ddd}d d!d"d#d$d%d&d'd(d)d*d+d,d-g}t;j<||d.}|=|j2D]}|j3D]}|jD]t}|dkrH|dkrH|dksLt t>d/||||j?||||j*|j+|j#|j@||jtjk| | d0 }|0|||||j}z|jAs|jBr*|C|||jD\}}tE|D]V\}}tF|tGr t>d1|d2tH|d3|djInt>d1|d4|jIqnd}d}|jJrP|K| ||jD\}}n |jL| ||||jDd|jMd5\}}|jAr|}|jJsg}|D]}|N|OPq|jQ|||jtR|jtR|jd6rtd7tR|jd8td9|||||jJrd:nd|rd;nd|j|j|j tS|j |j|j |j ||||jJ|rF|d<nd=|d<d>} |T| Wn6tUk rtjVd?d d@YW5QRdSXq$qq W5QRXtdA||S)BNz3.1.0z/This tool requires transformers 3.1.0 or later.z Arguments:z'fp16 requires --optimize_onnx --use_gpuzquantization only supports CPUrrzr__name__r;r)r<r<r<r= s*       s