U zh-,@sUddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z mZmZddlmZdZe e jed<z ddlZWnek rdZYnXddlZddlmZddlmZddlmZmZmZmZdd l m!Z!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'dd l(m)Z)ej*Z*d Z+e,e-Z.d dZ/e0dddZ1ddZ2ddZ3ddZ4e0dddZ5dSe j6ddddZ7dde8e e'dd d!Z9Gd"d#d#Z:Gd$d%d%Z;Gd&d'd'Ze0dd,d-Z?dTd.d/Z@ejAd0d1d2ZBdUd5d6ZCejDd7dfee8ee8ejEee0ejFfe8d8d9d:ZGd;d<ZHdd=d>d?d@dAZIdBdCZJdDdEZKdFdGZLdHdIZMdJdKZNdLdMZOdNdOZPdVdQdRZQdS)WN)ListOptionalSequenceUnion)patchnp)fx) OutputGraph)config eval_frameoptimize_assertreset)create_instruction debug_checks is_generatortransform_code_object)CheckFunctionManager GuardedCode)samecCs |dkr dS||jSN)detachcloneZrequires_grad_ requires_gradxrG/var/www/html/venv/lib/python3.8/site-packages/torch/_dynamo/testing.pyclone_me)sr)returncCstdd|S)Nz ^_orig_mod[.])resub)namerrrremove_optimized_module_prefix/sr%cCs(g}||||t}t}|D]N\}}t|tjrHt|}|} |j} |jdkrft |} | ||d<| ||<q,||||t} | D]$\}} t|tjrt|}| | |<q|| |D]P} t| t t fr | D]}t|tj r||jqqt| tj r|| jq|S)Nz.grad)appenddictZnamed_parameters isinstancer ZOptimizedModuler%gradtorchZ zeros_likeZ named_bufferstuplelistTensor)modelZ predictionZlossexample_inputsresultsZgradsparamsr$paramZ param_copyr)buffersbufferexampleZinprrrcollect_results3s<              r6cCs^t|tjr|jSt|ttfr2tdd|DS|dkr>dSt|trLdStdt |dS)Ncss|]}t|VqdSr)requires_bwd_pass.0rrrr _sz$requires_bwd_pass..FDon't know how to reduce) r(r*r-rr,r+anyintNotImplementedErrortypeoutrrrr7[s  r7cCst|tjr||St|ttfrDtdd|Dt|St|j dkr\t |j St|j dkrt|j St|t rtdd|Dt|Stdt|dS)z/Reduce the output of a model to get scalar losscss|]}t|VqdSrreduce_to_scalar_lossr8rrrr:msz(reduce_to_scalar_loss..)ZMaskedLMOutputZSeq2SeqLMOutputZ!CausalLMOutputWithCrossAttentionsZSquashedNormalcss|]}t|VqdSrrB)r9valuerrrr:wsr;N)r(r*r-sumZnumelr,r+lenr?__name__rCZlogitsZmeanr'valueskeysr>r@rrrrCgs    rCcCs0tjtjtd}tj|s,t||S)Nz../debug)ospathjoindirname__file__existsmkdir)rKrrr debug_dir}s  rQr!)coder c CsVttjt|d6}|t|dt|d|dW5QRXdS)Nwz  ) openrJrKrLrQwritedisBytecodeinfo)r$rRextrafdrrr debug_dumps(r\)skip)r]r c Cs`dd}t|jrdSt|jt|j|}tidddddditt|jd }t|t|j S)zused to debug jump updatescSs$|dtd|dtddS)NrNOP)insertr)Z instructions code_optionsrrr insert_nopssz&debug_insert_nops..insert_nopsNFZ_idr) r`Z compiler_fnZroot_txZexportZexport_constraintsZ frame_stateZ local_scopeZ global_scopef_code) rrbrrr localsglobalsrrZcheck_fn)frameZ cache_sizehooks_r]rarRgraphrrrdebug_insert_nopss"    ric@s6eZdZddZejjeejdddZ ddZ dS) CompileCountercCsd|_d|_dSNr frame_countop_countselfrrr__init__szCompileCounter.__init__gmr/cCs:|jd7_|jjD]}d|jkr|jd7_q|jS)Nr call)rmrhnodesoprnforward)rprsr/noderrr__call__s   zCompileCounter.__call__cCsd|_d|_dSrkrlrorrrclearszCompileCounter.clearN) rG __module__ __qualname__rqr*r GraphModulerr-ryrzrrrrrjsrjc@s.eZdZddZejjeejdddZ dS)CompileCounterWithBackendcCsd|_d|_||_g|_dSrk)rmrnbackendgraphs)rprrrrrqsz"CompileCounterWithBackend.__init__rrcCs\ddlm}|jd7_|jjD]}d|jkr"|jd7_q"|j|||j ||S)Nr )lookup_backendrt) Zbackends.registryrrmrhrurvrnrr&r)rprsr/rrxrrrrys    z"CompileCounterWithBackend.__call__N rGr{r|rqr*rr}rr-ryrrrrr~sr~c@s.eZdZddZejjeejdddZ dS)EagerAndRecordGraphscCs g|_dSr)rrorrrrqszEagerAndRecordGraphs.__init__rrcCs|j||jSr)rr&rw)rprsr/rrrrys zEagerAndRecordGraphs.__call__NrrrrrrsrcCst|}tdd|S)Nz (?m)^ *#.*\n?r!)strr"r#rRrrr strip_commentsrcCsddd|dDS)NrTcSsg|] }|qSr)rstrip)r9linerrr sz)remove_trailing_space..)rLsplitrrrrremove_trailing_spacesrcCs tt|Sr)rr)Zgm_strrrr normalize_gmsrcCstjs|dk r|}t}ddt|D}ddt|D}||} ||} tt||} | |} | |} | |}| |}t|t| | |t|| |t| | |t|| ||j ||dk r||j |dS)NcSsg|]}tddqS r*randnr9rgrrrrsz!standard_test..cSsg|]}tddqSrrrrrrrs) r Zassume_static_by_defaultrjrangerr Z assertTruerZ assertEqualrmrn)rpfnnargsZ expected_opsZexpected_ops_dynamicZexpected_frame_countactualZargs1Zargs2Zcorrect1Zcorrect2Zopt_fnZval1aZval2aZval1bZval2brrr standard_tests* r)rscCs|jSr)rwrrrrrdummy_fx_compilesrT皙?cCs.|sdS||kr|ddS|dd|dS)NERRORz.3fzx SAMEzx p=z.2fr)ZspeedupZpvalueZ is_correctZpvalue_thresholdrrrformat_speedup s  rcpu)sizestridedtypedevice extra_sizecCs~tddt||Dd|}|jr^|jdkrLtj|tj|dj|d}qptj|||d}ntj|g||d}t |||S)Ncss|]\}}|d|VqdS)r Nr)r9shaperrrrr:szrand_strided..r )rr)r)rrr) rEzipZis_floating_pointitemsizer*rZfloat16toZzerosZ as_strided)rrrrrZ needed_sizer4rrr rand_strideds rcstfdd}|S)Nc sNt<}D]\}}}|t|||q||W5QRSQRXdSr) contextlib ExitStack enter_contextrobject)argskwargsstackmoduleattrvalrpatchesrr_fn0s z"_make_fn_with_patches.._fn) functoolswraps)rrrrrr_make_fn_with_patches/srcCs|Srrrrrr<r) xfail_prop decoratorc Gst||j|ji}|j|_t|D]}|drt||}t|s\t||t||q(||} t |f|} | | _|dk rt ||rt | } t|| || q(t ||s(t||t||q(|S)NZtest_) r?rG __bases__r|dir startswithgetattrcallablesetattrrhasattrunittestexpectedFailure) clsZ cls_prefixZ fn_suffixrrrZDummyTestClassr$rnew_nameZnew_fnrrrmake_test_cls_with_patches;s"      rcCstjdkr|St|S)N)r sys version_inforr]rrrrskipIfNotPy311Us rcCstjdkr|St|SN)r rrrrrskipIfNotPy312[s rcCstjdkrt|S|Sr)rrrrrrrr xfailIfPy312as  rcCstjdkrt|S|Srrrrrr skipIfPy312gs  rcCs d|_|SNT)Z_expected_failure_dynamicrrrrexpectedFailureDynamicosrcCs d|_|Sr)Z!_expected_failure_codegen_dynamicrrrrexpectedFailureCodegenDynamicusrcCs d|_|Sr)Z!_expected_failure_dynamic_wrapperrrrrexpectedFailureDynamicWrapper{srFcCsRtdtdtr$tjd|rNddlmm}|dt | dS)Ni9r) r*Z manual_seedrandomseedrZtorch_xla.core.xla_modelcoreZ xla_modelZ set_rng_staterZ xla_device)Zuse_xlaZxmrrrreset_rng_states   r)r!)NNr )Tr)F)RrrWrloggingos.pathrJrr"rtypesrtypingrrrrZ unittest.mockrr ModuleType__annotations__numpyModuleNotFoundErrorr*rZtorch._dynamo.output_graphr r!r r r rZbytecode_transformationrrrrZguardsrrutilsr unsupportedthree getLoggerrGlogrrr%r6r7rCrQCodeTyper\r=rirjr~rrrrrr}rrZfloat32rrrrrrrrrrrrrrrrrs       (     !