U Mh) @sLUddlZddlZddlmZddlmZmZmZmZm Z m Z ee e e dddZ e e e ee e ee fddd Zee ee ee d d d Ze e d ddZe e d ddZe e ee e fee e fee ee ee ffdddZee e ee e fee e fee ee ee ffdddZd6e e ee dddZe e d ddZd7e e ee fee ee e ee e fe ee dd d!Zd"Ze ed#<d$d%hZee ed&<eZee ed'<d(d(d)Zee e fed*<d+Ze ed,<d$d%hZee ed-<eZee ed.<d/d0iZ ee e fed1<dd2d3d4Z!e"d5krHe!dS)8N) defaultdict)AnyDictListSetTupleUnion)lines indentationreturncCsFd}dd|}t|D](\}}|dkr0||7}||d|7}q|S)N  r) enumeratereplace)r r outputZnew_line_with_indentilinerT/var/www/html/venv/lib/python3.8/site-packages/torch/utils/data/datapipes/gen_pyi.pymaterialize_liness rdir template_name output_name replacementsc Cs~tj||}tj||}t|}|}W5QRX|D]<\}} } t|d"}||t| | }||W5QRXq(sz"find_file_paths..csh|]}|kr|qSrrr*r(rrr-)scsh|]}tj|qSrrrrr*)dir_pathrrr-*s)setrlistdirupdate)r'r(paths all_filesZ python_filesZ filter_filesr)r0r(rfind_file_pathss r6)rr cCs\d|krd\}}n d|kr$d\}}ntd|||t|||}}|||S)zXExtract method name from decorator in the form of "@functional_datapipe({method_name})".(")r7z")(')r8z')z4Unable to find appropriate method name within line: ) RuntimeErrorfindlenrZ start_tokenZ end_tokenstartendrrrextract_method_name.s  r?cCs2d}d}||t|||}}|||S)zVExtract class name from class definition in the form of "class {CLASS_NAME}({Type}):".class ()r:r;r<rrrextract_class_name:srB) file_pathr c Csiit}}}tt}t|d}d}d\}}} d} |D]D} | dddkrZ| } | sfd| krv||| qrrrparse_datapipe_fileBsL              rU) file_pathsr c Csdiit}}}i}|D]<}t|\}}}} |||||||| q||||fS)N)r1rUr3) rVmethods_and_signaturesmethods_and_class_namesZ!methods_with_special_output_typesmethods_and_doc_stringsrrNrOZ$methods_needing_special_output_typesrPrrrparse_datapipe_filesms    rZ,)r delimiterr cCspd}d}g}|D]P}|dkr&|d7}n2|dkr8|d8}n ||krX|dkrX||d}q||7}q|||S)zRGiven a line of text, split it on comma unless the comma is within a bracket '[]'.rr [rE])rI)rr\Z bracket_countZ curr_tokenrescharrrrsplit_outside_bracket~s     racCst|}t|D]\}}|d||<|dkr8d||<q|dkrjd||dkrj||ddkrjd||<qd|kr|d d \}}|dd ||<qd d |D}d|}|S)z Clean up a given raw function signature. This includes removing the self-referential datapipe argument, default arguments of input functions, newlines, and spaces. rclsselfrrE*r z Callable ==rDz= ...cSsg|]}|dkr|qS)r r)r+trrr sz%process_signature..z, )rarrMrsplitr)rtokensrtokenheadZ default_argrrrrLs (  rLr )rCr(deprecated_filesdefault_output_typemethod_to_special_output_typerootr c s dkrtttjt|tr,|gn|}fdd|D}t|||d}t |\}}} } |D]} | | krh| | qhg} | D]h\} }|| }| | kr|| }n|}d | | }|dkrd}| d|d| d|d |d | q| jd d d | S)a #.pyi generation for functional DataPipes Process. # 1. Find files that we want to process (exclude the ones who don't) # 2. Parse method name and signature # 3. Remove first argument after self (unless it is "*datapipes"), default args, and spaces r csg|]}tj|qSrr/)r+rrorrrgsz*get_method_definitions..r.z ... z# Functional form of 'z' def rAz) -> z: cSs|ddS)Nr rE)split)srrrz(get_method_definitions..)key)strpathlibPath__file__parentresolve isinstancer6unionrZrJitemsrrIsort)rCr(rlrmrnrorVrWrXZmethods_w_special_output_typesrYfn_nameZmethod_definitionsrQ argumentsrR output_typeZ doc_stringrrprget_method_definitionss0    *riteriterDP_file_pathz __init__.pyzutils.pyiterDP_files_to_excludeiterDP_deprecated_fileszList[IterDataPipe])Zdemuxfork$iterDP_method_to_special_output_typemapmapDP_file_pathmapDP_files_to_excludemapDP_deprecated_filesshuffle IterDataPipe#mapDP_method_to_special_output_type)r cCs\ttttdt}ttttdt}t t j }d|dfd|dfg}tt|dd|dd S) a # Inject file into template datapipe.pyi.in. TODO: The current implementation of this script only generates interfaces for built-in methods. To generate interface for user-defined DataPipes, consider changing `IterDataPipe.register_datapipe_as_function`. rZ MapDataPipez${IterDataPipeMethods}z${MapDataPipeMethods}zdatapipe.pyi.inz datapipe.pyirN)rrrrrrrrrrwrxryrzr{r&rv)Ziter_method_definitionsZmap_method_definitionsrrrrrmains"r__main__)r[)r )#rrw collectionsrtypingrrrrrrrvintrr&r6r?rBrUrZrarLrr__annotations__rr1rrrrrrr__name__rrrrsB   "  <+@  -