U zh^ @sNddlZddlZddlmZddlZddlZddlmZddlm Z m Z ddl m Z GdddeZ Gd d d eZejed d d Zeje dddZejedddZejedddZGdddeZGdddeZGdddeZdgdggZdgdggdgdggdgd gggZd!d!d"gd#d$d%gd#d$d%ggZejedd&d'ZdS)(N)IntEnum)ir)get_dtype_size sympy_product)Vc@seZdZdZdZdZdS) NCCL_COLLrrN)__name__ __module__ __qualname__ ALL_REDUCE ALL_GATHERREDUCE_SCATTERrrO/var/www/html/venv/lib/python3.8/site-packages/torch/_inductor/comm_analysis.pyrsrc@seZdZdZdZdZdS)NVIDIA_GPU_TYPErrr N)r r r VOLTAAMPEREHOPPERrrrrrsr)returncCsLtjjtjjjpd}d|kr&tjSd|kr4tjSd|krBtjStjSdS)NZV100ZA100ZH100) torchutilsZ collect_envZ get_gpu_inforunrrrr)Zgpu_inforrr get_gpu_typesr)nodercCsht|tjstd||j}|dk s,td|kr:tjSd|krHtjSd|krVtj Std|dS)Nz!node is not a collective kernel: Z all_reduceZ all_gatherZreduce_scatterzUnsupported collective kernel: ) isinstancer_CollectiveKernel ValueErrorZpython_kernel_nameAssertionErrorrr rr)rZ kernel_namerrrget_collective_type(s  r!cCs\d}|jD]L}t|jj}t|tjr0t|}ntj j j |dd}||t |jj 7}q |S)Nr)fallback)inputsrZlayoutsizersympyIntegerintrgraphZsizevarsZ size_hintrZdtype)rZsz_bytesZinpZnumelrrrget_collective_input_size_bytes8s    r)cCs:t|tjkr(ddlm}||jdStd|dS)Nr)_get_group_size_by_namezUnsupported collective type: )typerrZ"torch.distributed.distributed_c10dr*Z constant_args TypeError)rr*rrrget_collective_group_sizeEs r.c@seZdZdZdZdZdS)NCCL_HWrrr N)r r r NVLINKZPCINETrrrrr/Ssr/c@seZdZdZdZdS) NCCL_ALGOrrN)r r r ZTREERINGrrrrr2Ysr2c@seZdZdZdS) NCCL_PROTOrN)r r r LLrrrrr4^sr4g333333@gffffff@g333333??g@g@gC@gffffff4@gU@g6@g3@cCst|}|ddd}d}t|}t||}|}|dkrBdStj}tj}t|} t j j j } t j j j } t} |dkr|dnd} |dkr| nd}t|| }|dkr| n| }d}||}t|||dks| tjkrdnd}| tjkrd|d}n| tjtjfkr|d}d||}||}|d }tj}| tjkrZ|dkrTd|}nd}n| tjtjfkrt|d}t||}t|||}ttj||}d }|dkrd}t||}||||||7}|d }||}||S) a9 Returns estimated NCCL collective runtime in nanoseconds (ns). The following heuristics are copied from https://github.com/NVIDIA/nccl/blob/master/src/graph/tuning.cc. We aim to estimate the runtime as accurately as possible. Assumptions: - only ring algorithm (NCCL_ALGO_RING) is used - only Low-Latency protocol (NCCL_PROTO_LL) is used, i.e. Simple or LL128 is not used - 8 gpus per node # TODO: Need to find a way to get accurate "gpus per node" and "# nodes" info. - collective is one of: allreduce, reducescatter, allgather irrr g?gUUUUUU?r6geAgg@@)r)r.mathceilr2r3r4r5r!rZ _inductorconfigZ intra_node_bwZ inter_node_bwrllMaxBwsminrr rrr/r0baseLathwLatr1max)rZtensor_storage_size_bytesZtensor_storage_size_GBZnum_gpus_per_nodeZ group_sizeZnNodesZnRanksZ nccl_algoZ nccl_protoZcollZbwIntraZbwInterZ compCapIndexindex2Zindex1ZllMaxBwbwZ nChannelsZbusBwZnstepsratio bandwidthZbandwidth_GB_per_nsZintraHwZ nInterStepsZlatencyZintraLatZinterLatZ netOverheadZ latency_nsZ transport_nsrrr estimate_nccl_collective_runtimesf            rD) functoolsr8enumrr%rrrrrrZ virtualizedrrr lru_cacherZIRNoder!r'r)r.r/r2r4r=r>r;floatrDrrrrs\