U yh}@sUddlZddlZddlZddlmZmZmZmZmZm Z ddl Z ddl m Z ddl mZddgZe sddlZGdd d Zd d Zeejd _eejd _ndd lmZmZmZmZmZmZmZmZmZm Z e!e"Z#erzddl$m%Z%Wn e&k re#'dYnXGdddej(Z)e)Z*e)e+d<de,dddZ-GdddZdde,ee.dfeee,dfedddZdS)N)DictListOptionalTuple TYPE_CHECKINGUnion) is_available)not_noneinit_device_mesh DeviceMeshc@s eZdZdS)_DeviceMeshStubN)__name__ __module__ __qualname__rrO/var/www/html/venv/lib/python3.8/site-packages/torch/distributed/device_mesh.pyr sr cCsdSNrrrrr_init_device_mesh_stubsrztorch.distributed.device_mesh) _find_pg_by_ranks_and_tag_get_default_group_get_group_tagget_process_group_ranksget_rankget_world_sizeinit_process_groupis_initialized new_group ProcessGroup) ArrayLikezCDeviceMesh requires numpy >= 1.21 to be installed for type checkingc@seZdZddddZddddZdeedfdd d d Zdedd d dZ dee d ddZ e ee dddZ e ee dddZdee dddZde eeejddddZdS)_MeshEnvNreturncCsg|_i|_i|_dSr) mesh_stackchild_to_parent_mappingmesh_dim_group_optionsselfrrr__init__?sz_MeshEnv.__init__r cCs t|jdkrtd|jdS)Nrz#No device mesh is currently active!)lenr# RuntimeErrorr&rrrget_current_meshFsz_MeshEnv.get_current_mesh.) parent_meshsubmesh_dim_namesr"c sfdd|D}fdd|D}ttjj}|D]}||q8jj||jd|}}|D]"} tj | |dd} || krn| } qn| _ fdd|D| _ |j | <| S) Ncsg|]}tj|qSr)r mesh_dim_namesindex.0 mesh_dim_namer-rr Osz._MeshEnv.create_child_mesh..csg|]}j|qSr)meshsizer2mesh_dimr4rrr5Ssr)Fr/ _init_backendcsg|]}j|qSr)_dim_group_infosr8r4rrr5ns)r)) listranger6ndimremoveZpermutereshaperr device_type _parent_meshr<r$) r'r-r.Z submesh_dimsZsubmesh_dim_sizesZmesh_dims_remainedZ submesh_dimpg_ranks_by_dimZcur_rankZmesh_ndsubmeshZ res_submeshrr4rcreate_child_meshKsB     z_MeshEnv.create_child_mesh) device_meshr"cCs|j|dSr)r$get)r'rGrrrget_parent_meshusz_MeshEnv.get_parent_meshcCsD||}|j}|r@|r@t|dks,td|d}|||SdS)z Return the index of the mesh dim in the parent mesh. The device_mesh passed in needs to be sliced out from a parent mesh. z%The child mesh can only be a 1D mesh.rN)rIr/r*AssertionErrorget_mesh_dim_by_name)r'rGr-Zchild_mesh_dim_namesZchild_mesh_dim_namerrrget_parent_mesh_dimxs   z_MeshEnv.get_parent_mesh_dim)rBr"cCs t|Sr)_get_device_handle device_countrBrrrnum_devices_per_hostsz_MeshEnv.num_devices_per_hostcCstt|Sr)rr rQrPrrr num_hostssz_MeshEnv.num_hosts)rGr3r"cCsT|jdkst|jdkr td||jkrDtd|dd|jt|j|S)NrzNo `mesh_dim_names` found.zMesh dimension 'z' does not exist.z.Available mesh dimensions are: mesh_dim_names=)r/r*KeyErrorr r0)r'rGr3rrrrLs    z_MeshEnv.get_mesh_dim_by_name)dimbackend pg_optionsr"cCs||f|j|<dSr)r%)r'rTrUrVrrr_set_mesh_dim_group_optionssz$_MeshEnv._set_mesh_dim_group_options)N)rrrr(r,rstrrFrrIintrM staticmethodrQrRrLrOptionsrWrrrrr >s.  * r _mesh_resourcescudarPcCs tt|dS)a: Get the module corresponding to the device_type which is cuda or cuda-like device. For example, when the device_type is cuda, the module `torch.cuda` is returned. Return None when there is no corresponding module for device_type, otherwise return the corresponding module. N)getattrtorchrPrrrrNsrNc @seZdZUdZeed<ejed<ee edfed<dddee ejd fee edfe dd d d Z d dZ ddZddddZddddZedddZddZee dddZe ee edffddddZd5ee eefed d!d"Zeedd#d$Zed6dd%e eeefeee ejd fee edfdd&d'd(Zd7eeed d)d*Zeedd+d,Zee edfdd-d.Z edd/d0Z!d8ee eefed d1d2Z"eeedd3d4Z#dS)9r a DeviceMesh represents a mesh of devices, where layout of devices could be represented as a n-d dimension array, and each value of the n-d dimensional array is the global id of the default process group ranks. DeviceMesh could be used to describe the layout of devices across the cluster, and serves as a proxy for communication among the device lists within the cluster. DeviceMesh can be used as a context manager. .. note:: DeviceMesh follows SPMD programming model, which means the same PyTorch Python program is running on all processes/ranks in the cluster. Therefore, users need to make sure the `mesh` array (which describes the layout of devices) should be identical across all ranks. Inconsistent `mesh` will lead to silent hang. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". mesh (ndarray): A multi-dimensional array or an integer tensor describing the layout of devices, where the IDs are global IDs of the default process group. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. A reduction over the first dimension of mesh will reduce across columns (0, 4), .. and (3, 7), a reduction over the second dimension of mesh reduces across rows (0, 1, 2, 3) and (4, 5, 6, 7). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) rBr6.r/NTr:r)rBr6r/r;r"cCs||_t|tjr,|jjdkr,td|t|tjrJ|jtj dntj |dtj d|_ |rjt |nd|_ t |j |_d|_t|_|dkr|r|||j tk}|ddkst|ddkr|dnd|_dS)Ncpuz!`mesh` must be a CPU tensor, got dtypedevicerbZxlar)rrJ)rB isinstancer_Tensorrdtype ValueErrordetachtorYtensorr6tupler/flattentolist_flatten_mesh_listrC threading get_ident _thread_id_get_or_create_default_group_init_process_groupsrZnonzeror7rK_coordinate_on_dim)r'rBr6r/r;Z rank_coordsrrrr(s&  zDeviceMesh.__init__cCst}|stt}|j|kr:td|jdt|j}|s|r|}||kr||dkrtd|d|d|jd| t |t S)Nz=Mesh should not be bigger than default world size, but found z ranks!rz8DeviceMesh only support homogeneous hardware, but found z ranks and  z devices!) rrrr6numelr+rNrBrOZ set_devicerr)r'Zdefault_initializedZ world_sizeZ device_handlerQrrrrs s(  z'DeviceMesh._get_or_create_default_groupc Cs g}|jjdkrF|jtkrF|ttttttj fnt|jjD]}|j d| d|j |}|D]}| }|tjkrtj|\}}nd\}}t|||d}||krxt||krtd|jd|d|tt|||j fqxqR||_dS)NrJr))NN)ZranksrUrVzFEach device mesh dimension should get only one process group, but got z in !)r6r?rwrappendrrr=r> group_nameZswapdimsrAr7rnr\r%rrr*r+r r<) r'Zdim_group_infosrTrDZdim_meshZsubgroup_ranksrUrVZ dim_grouprrrrt&sN       zDeviceMesh._init_process_groupsr!cCstj||Sr)r\r#ryr&rrr __enter__js zDeviceMesh.__enter__cCstjdSr)r\r#pop)r'exc_type exc_value exc_tracebackrrr__exit__pszDeviceMesh.__exit__cCs6|jsd|jdnd|jd|jd}|S)Nz DeviceMesh()z, mesh_dim_names=)r/r6rn)r'Zdevice_mesh_reprrrr__repr__ts zDeviceMesh.__repr__cCs>t|dd|_|js8t|j|jj|j|j|j|j f|_|jS)N_hash) r^rhashror6shaperBr/rCrrr&rrr__hash__|s zDeviceMesh.__hash__)otherr"cCsrt|tsdSt|t|kr"dS|j|jkol|jj|jjkol|j|jkol|j|jkol|j|jkol|j |j kSdS)NFT) rer idror6rrBr/rCrr)r'rrrr__eq__s      zDeviceMesh.__eq__)r/r"csjstdt|tr|fn|}d|djd}|jkrDSt|tjksltfdd|Dsvt|nF|d}j|}t|j|t|D]\}}||krt|qt |}|S)a Slice the current DeviceMesh based on the mesh_dim_name given to create a child DeviceMesh. Args: mesh_dim_name (Union[str, Tuple[str]]): the name or the tuple of names of the mesh dimension of the parent DeviceMesh to create the child DeviceMesh for. Returns: A :class:`DeviceMesh` object The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh["tp"] on rank 0, 1, 2, 3 would return a 1D child DeviceMesh:([0, 1, 2, 3]). Calling mesh["tp"] on rank 4, 5, 6, 7 would return a 1D child DeviceMesh:([4, 5, 6, 7]). Calling mesh["dp"] on rank 0, 4 would return a 1D child DeviceMesh:([0, 4]). Calling mesh["dp"] on rank 1, 5 would return a 1D child DeviceMesh:([1, 5]). Calling mesh["dp"] on rank 2, 6 would return a 1D child DeviceMesh:([2, 6]). Calling mesh["dp"] on rank 3, 7 would return a 1D child DeviceMesh:([3, 7]). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) z1Cannot slice a DeviceMesh without mesh_dim_names!zInvalid mesh_dim_name zG specified. Valid mesh_dim_names should be a contiguous subsequence of .c3s|]}|jkVqdSrr/r1r&rr sz)DeviceMesh.__getitem__..r) r/r+rerXr*allrSr0zipr\rF)r'r/ error_msgZoutermost_dim_nameZoutermost_dim_idxijrErr&r __getitem__s,      zDeviceMesh.__getitem__)r9r"cCst|dstd|jjdkr>|dkr>td|jjddd|jjdkrX|dkrXd }nt|trnt||n|}tt |j |dd S) a Returns the single ProcessGroup specified by mesh_dim, or, if mesh_dim is not specified and the DeviceMesh is 1-dimensional, returns the only ProcessGroup in the mesh. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: A :class:`ProcessGroup` object. r<z*DeviceMesh process groups not initialized!rJNFound the DeviceMesh have dimensionsJOptional kwarg `mesh_dim` needs to be specified when device_mesh.ndim > 1.zmIf you want to get the list of all the ProcessGroups in the DeviceMesh,please use `get_all_groups()` instead.rr ) hasattrr+r6r?rerXr\rLr rr<r'r9rrr get_groups" zDeviceMesh.get_groupcsfddtjjDS)z Returns a list of ProcessGroups for all mesh dimensions. Returns: A list of :class:`ProcessGroup` object. csg|]}|qSr)r)r2rr&rrr5sz-DeviceMesh.get_all_groups..)r>r6r?r&rr&rget_all_groupsszDeviceMesh.get_all_groupsr)grouprBr6r/r"cCs<t|trt|}t|tjr*||ks:|dk rR||krRtdt|d|tj|dtj d}t |||dd}t |||j fg|_ |St|}t|dkrtd |dkrtd t|tjr|jtj dd ntj|dtj d}|jt|krtd |d t|dt |||dd}dd|D|_ |S)aN Contstructs a :class:`DeviceMesh` with ``device_type`` from an existing :class:`ProcessGroup`. The constructed device mesh has number of dimensions equal to the number of groups passed. If more than one group is passed, then the ``mesh`` argument is required. Nz Invalid mesh z for ProcessGroup with ranks r`rcFr:rz.Expects at least one ProcessGroup to be passedz0Must pass mesh if passing multiple ProcessGroups)rbrdzEExpects mesh with ndim equal to number of ProcessGroups but got mesh z and z ProcessGroupscSs g|]}t|t||jfqSr)rrrz)r2rrrrr5<s z)DeviceMesh.from_group..)rerrr_rfrnrhrXrkrYr rrzr<r=r*rirjr?)rrBr6r/Z group_ranksrGgroupsrrr from_groups\     zDeviceMesh.from_groupcCs|dkr|jS|j|Sr)r6rwr7rrrrr7FszDeviceMesh.sizecCs|jjSr)r6r?r&rrrr?IszDeviceMesh.ndimcCs t|jjSr)rlr6rr&rrrrMszDeviceMesh.shapecCstS)z: Returns the current global rank. )rr&rrrrQszDeviceMesh.get_rankcCsb|jdkr*|dkr*td|jjddn |dkr6d}t||}t|tsVtdtt|S)a{ Returns the local rank of the given mesh_dim of the DeviceMesh. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: An integer denotes the local rank. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 0, 1, 2, 3 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 4, 5, 6, 7 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 0, 4 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 1, 5 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 2, 6 would return 2. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 3, 7 would return 3. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) rJNrrrrz1We expect ProcessGroup before calling `get_rank`!) r?r+r6r rrerrKr)r'r9Zmesh_dim_grouprrrget_local_rankWszDeviceMesh.get_local_rankcCs|jr |jSdS)z Return the relative indices of this rank relative to all dimensions of the mesh. If this rank is not part of the mesh, return None. N)rur&rrrget_coordinateszDeviceMesh.get_coordinate)N)N)N)N)$rrr__doc__rX__annotations__r_rfrrrboolr(rsrtr{rrrobjectrrrYrrrrrZrr7propertyr?rrrrrrrrr sX '   (D ?$ >*r.)rB mesh_shaper/r"c Cs|dk r\tt|t|kr,tdd|t|t|kr\tddt|dt|d|rz|sztd|d d td "tjt|tj d  |}W5QRXt |||d }|S)a Initializes a `DeviceMesh` based on `device_type`, `mesh_shape`, and `mesh_dim_names` parameters. This creates a DeviceMesh with an n-dimensional array layout, where `n` is the length of `mesh_shape`. If `mesh_dim_names` is provided, each dimension is labeled as `mesh_dim_names[i]`. .. note:: `init_device_mesh` follows SPMD programming model, meaning the same PyTorch Python program runs on all processes/ranks in the cluster. Ensure `mesh_shape` (the dimensions of the nD array describing device layout) is identical across all ranks. Inconsistent `mesh_shape` may lead to hanging. .. note:: If no process group is found, init_device_mesh will initialize distributed process group/groups required for distributed communications behind the scene. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". Passing in a device type with a GPU index, such as "cuda:0", is not allowed. mesh_shape (Tuple[int]): A tuple defining the dimensions of the multi-dimensional array describing the layout of devices. mesh_dim_names (Tuple[str], optional): A tuple of mesh dimension names to assign to each dimension of the multi-dimensional array describing the layout of devices. Its length must match the length of `mesh_shape`. Each string in `mesh_dim_names` must be unique. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import init_device_mesh >>> >>> mesh_1d = init_device_mesh("cuda", mesh_shape=(8,)) >>> mesh_2d = init_device_mesh("cuda", mesh_shape=(2, 8), mesh_dim_names=("dp", "tp")) Nz"Each mesh_dim_name must be unique.z/Found repeated mesh_dim_name in mesh_dim_names z6mesh_shape and mesh_dim_names should have same length!zFound len(mesh_dim_names): z and len(mesh_shape):rz4Device type with GPU index is not supported but got z. zUIf you maintained a 'torch.device' object, it's recommended to pass in 'device.type'.r`ra)rBr6r/) r*setr+isalphar_rdZarangemathprodrYviewr )rBrr/r6rGrrrr s0)   &)r])/loggingrrptypingrrrrrrr_Ztorch.distributedrZutils._typing_utilsr __all__sysr rmodulesr r Z"torch.distributed.distributed_c10drrrrrrrrrr getLoggerrloggerZ numpy.typingr ImportErrorwarninglocalr r\rrXrNrYrrrrsP    0  l Z