U yh?@spUddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZmZedZejed<Gd d d eeZGd d d eeZd%e jeedddZe j e jdddZ!d&e j eee e jdddZ"e j e e jedddZ#e j e e je je e jdddZ$d'e j%edddd Z&d(eee j%feed"d#d$Z'dS))N)Enum)CallablecastDictIterableListSet)TensorMetadata)_pytree) tree_flattentree_unflattenZ graph_utilsloggerc@s$eZdZdZdZdZdZdZdZdS)OP call_functionZ call_moduleZ call_methodZget_attroutput placeholderN) __name__ __module__ __qualname__ CALL_FUNCTIONZ CALL_MODULE CALL_METHODZGET_ATTROUTPUTZ PLACEHOLDERrrU/var/www/html/venv/lib/python3.8/site-packages/torch/distributed/_spmd/graph_utils.pyrs rc@s eZdZdZdZdZdZdZdS)CommTypeZ allreduce_Z allgather_Z broadcast_Zreduce_scatter_Zscatter_N)rrrZ ALLREDUCEZ ALLGATHERZ BROADCASTZ REDUCESCATTERZSCATTERrrrrrs rT)node is_requiredreturnc CsF|jdd}|rB|dkrBtd|jd|jd|jd|j|S)NZ tensor_metaz2Callsite expects that ``tensor_meta`` exists in ``z ``, but got None instead. Node:  )metaget RuntimeErrornameoptarget)rrmetadatarrrget_node_tensor_metadata!s  "r&)graphrcCs6t|jD]}|jtjkr |Sq td|dS)zTake a graphmodule and return the graph output node. We traverse in reverse to expedite it, with the idea that last node should be output zCannot find the output node in N)reversednodesr#rrr!)r'rrrr get_output+s  r*F)r' predicate reverse_orderrcs@tttj|j}|r.tttjtt|}fdd|DS)zSTake a predicate and return all the nodes in the `graph` where the predicate holds.csg|]}|r|qSrr).0rr+rr =szfind_node..)rrfxNoder)iterr()r'r+r,r)rr.r find_node6sr3)r'subgraphrcCsPt|}t|}|D]6}|jD]*}t|tjs0q||kr||krdSqqdS)zEnsure nodes in ``subgraph`` satisfy one of the following rules. 1. The user of the node is in ``subgraph``. 2. The user of the node is output. 3. There are no users -- the node is a side-effect node. FT)setr*Zusers isinstancer0r1)r'r4 all_nodesrruserrrris_leaf_subgraph@s   r9)r'r4r$rc Cst|}t}g}|||D]}||j|j|j|j}tj |j|j}t |j|jf\} } g} t || D]B\} } t | t jr| |kr| |kst| || qp| | qpt| | \|_|_|||<||q"W5QRX|S)zClone the given subgraph and insert it before ``target``. This API currently does not support inserting after ``target``. )r5dictZinserting_beforerr$argskwargstypepytreeZarg_tree_leavesr zipr6r0r1AssertionErrorappendr )r'r4r$r7mappingZcloned_subgraphrZ cloned_nodeZoriginal_inputZ cloned_inputspecZmapped_cloned_inputZoriginal_input_nodeZcloned_input_noderrrclone_subgraphRs@      rD)gmremove_dead_codercCs$|j|r|j|dS)zRun the required steps to ensure production-ready graph. Note - per the fx docs, elimination of dead code is not very precise. Hence, the flag to make this step optional. N)r'ZlintZeliminate_dead_codeZ recompile)rErFrrr rebuild_graph{s  rG)graphsfolderrc Csb|s t}|D]<\}}ttj||dd}|t|W5QRXqt d||S)Nz.graphwzDump graphs to %s) tempfilemkdtempitemsopenospathjoinwritestrr warning)rIrJprefixrEfprrrdump_graphs_to_filess rX)T)F)T)rH)(loggingrPrLenumrtypingrrrrrrZtorch.fxr0Ztorch.fx.passes.shape_propr Z torch.utilsr r>Ztorch.utils._pytreer r getLoggerr Logger__annotations__rTrrr1boolr&ZGraphr*r3r9rDZ GraphModulerGrXrrrrs:          )