U yh @sddlZddlZddlmZmZmZmZmZmZddl Z ddl m m m Zddlm m mZddlmZddlmZddlmZmZddlmZddlmZmZmZmZmZm Z ddlm!Z!m"Z"ddl#m$Z$m%Z%d d d gZ&e j'j(Z(Gd d d e j)j*Z+Gddde j)j*Z,Gdd d e j-Z.de j-ee%eeee.ddd Z/dej0ee%eee1ej0e%gdfeeej0ee%gdfeeej0ee%gdfej0ddd Z2dS)N)AnyCallablecastOptionalSequenceTuple)mesh_broadcast) Redistributeredistribute_local_tensor)compute_global_tensor_info) DTensorSpecPartial Placement ReplicateShard TensorMeta)is_rng_supported_meshOffsetBasedRNGTracker)_mesh_resources DeviceMeshDTensordistribute_tensordistribute_modulec@s<eZdZedeeedddZeej dddZ dS) _ToTorchTensorr)inputgrad_placementscCs|j|_||_|j}||SN)_spec dtensor_specr _local_tensorview_as)ctxrr local_tensorr#O/var/www/html/venv/lib/python3.8/site-packages/torch/distributed/_tensor/api.pyforward?sz_ToTorchTensor.forward grad_outputc Csl|j}|j}|j}|j}t|||j\}}t|}|p:|j}t||t|j ||j dd}t |||j ddfS)Nshapestridedtype tensor_meta requires_grad) rmeshrr-r placementstupler rr)r+rr/) r!r'rr0rZ dtensor_meta_ tensor_strideZ grad_specr#r#r$backwardNs6  z_ToTorchTensor.backwardN) __name__ __module__ __qualname__ staticmethodrrrr%torchTensorr5r#r#r#r$r>s  rc @sZeZdZed ejeeedfe e ej e ee dfddddZ edddd ZdS) _FromTorchTensorN.r)r device_meshr1 run_checkr)r*returncCs||_||_|r |r ||}}nF|sN|sNt|||\} } t| t| }}ntd|d|dd|dkr|jd|j d}n4|rt |D]&\} } | r| }t ||| dqt||t|||jd} t||| |j d}|S) Nz Found shape:z , stride:.z3Please pass both shape and stride at the same time.rr.)Zmesh_dimr,)previous_placementprevious_device_meshr r:Sizer2 RuntimeErrorZget_coordinateZ new_emptyr/ enumerate is_replicate contiguousrr rr+rr )r!rr=r1r>r)r*Z tensor_shaper4Z global_shapeZ global_strideidx placementZ dist_specZ dist_tensorr#r#r$r%osJ    z_FromTorchTensor.forwardr&cCsh|j}|j}|j|krT|j}t|||jjd}|j}t|||dd}|dddddfS|dddddfS)Nr,T)Z is_backward) rArBr1rr r-rr to_local)r!r'rArBZ current_specZ target_specr"outputr#r#r$r5s$ z_FromTorchTensor.backward)NN)r6r7r8r9r:r;rrrboolrrCintr%r5r#r#r#r$r<ns ?r<c @steZdZUejed<eed<ddgZe Z ej ed<e ej ejee ddddZddZd d Ze d d Zd dZddZeej d+ddZe d,ddddejeeeeee eejeeedfddddZddeeeejdddZd-dd eeeeee dd!d"d#Zddeeeejdd$d%Z e!ed&d'd(Z"e!eed&d)d*Z#dS).rrr_op_dispatcher)r"specr/r?c Cs^|jr|std|jdk s&tdtjj||jj|jj |j |j |j |d}||_ ||_|S)a^ Construct a DTensor from a local tensor, device mesh, and placement and other tensor properties (i.e. shape, requires_grad, strides, etc). Note: This is not a public API and it's only supposed to be used by the operator implementations and internals. If you want to construct a DTensor from a local tensor, consider using `DTensor.from_local`, if you want to construct a DTensor from a "global" tensor (where you already have tensor initialized and want to shard this tensor), consider using `distribute_tensor`. zxTo construct DTensor from torch.Tensor, it's recommended to use local_tensor.detach() and make requires_grad consistent.NzTensorMeta should not be None!)stridesr+devicelayoutr/)r/warningswarnr-AssertionErrorr:r;Z_make_wrapper_subclassr)r*r+rQrRrr)clsr"rOr/rr#r#r$__new__s"  zDTensor.__new__cCs"d|jd|jjd|jjdS)NzDTensor(local_tensor=z, device_mesh=z , placements=))rrr0r1selfr#r#r$__repr__szDTensor.__repr__cCsdg|j|jffS)ze protocol to inform how to flatten a DTensor to local tensor for PT2 tracing r)rr/rZr#r#r$__tensor_flatten__szDTensor.__tensor_flatten__c CsR|dk std|d}|\}}t|||jjd}t|j|j|d}t|||dS)NzEExpecting spec to be not None from `__tensor_flatten__` return value!rr(r,r.)rUrr-r+r r0r1r) Z inner_tensorsZ flatten_specZ outer_sizeZ outer_strider"rOr/Zunflatten_tensor_metaZunflatten_specr#r#r$__tensor_unflatten__ s*zDTensor.__tensor_unflatten__cCs8tdd|jDs|Sdd|jD}|j|j|dS)Ncss|]}t|tVqdSr) isinstancer .0pr#r#r$ #sz6DTensor.__coerce_tangent_metadata__..cSs g|]}t|trtn|qSr#)r_r rr`r#r#r$ %sz7DTensor.__coerce_tangent_metadata__..r=r1)anyr1 redistributer=)r[r1r#r#r$__coerce_tangent_metadata__"s z#DTensor.__coerce_tangent_metadata__cCs|j|j|jdS)Nre)rgr=r1)r[Zmetadata_tensorr#r#r$#__coerce_same_metadata_as_tangent__*sz+DTensor.__coerce_same_metadata_as_tangent__r#NcCstj|||piSr)rrNdispatch)rVfunctypesargskwargsr#r#r$__torch_dispatch__0s zDTensor.__torch_dispatch__T)r>r)r*.)r"r=r1r>r)r*r?c Cs|p t}|j}||jjkr.|js.||}|dkrLddt|jD}nJt |}t |D]8\}}| r\t t |}|jdkr\t |j|j||<q\t||t||||S)a Create a :class:`DTensor` from a local torch.Tensor on each rank according to the `device_mesh` and `placements` specified. Args: local_tensor (torch.Tensor): local torch.Tensor on each rank. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the tensor, if not specified, must be called under a DeviceMesh context manager, default: None placements (List[:class:`Placement`], optional): the placements that describes how to place the local torch.Tensor on DeviceMesh, must have the same number of elements as `device_mesh.ndim`. If not specified, we will by default replicate the tensor across the `device_mesh` from the first rank of each dimension of the `device_mesh`. Keyword args: run_check (bool, optional): indicate whether to run check across ranks to check meta information and data. if have :class:`Replicate` in `placements`, the data on first rank of the device mesh dimension will be broadcasted to other ranks. shape (torch.Size, optional): A List of int which specifies the size of DTensor which build on top of `local_tensor`. Note this needs to be provided if the shape of `local_tensor` are different across the ranks. If not provided, `shape` will be computed assuming the given distributed tensor is evenly sharded across ranks. stride (tuple, optional): A List of int which specifies the stride of DTensor. If not provided, `stride` will be computed assuming the given distributed tensor is evenly sharded across ranks. Returns: A :class:`DTensor` object .. note:: `from_local` is differentiable, the `requires_grad` of the created `DTensor` object will depend on if `local_tensor` requires_grad or not. NcSsg|] }tqSr#rrar3r#r#r$rdvsz&DTensor.from_local..r)rget_current_mesh device_typerQtypeis_metatorangendimlistrEis_shardrrdimr<applyr2) r"r=r1r>r)r*rsrHrIr#r#r$ from_local;s(2    zDTensor.from_local)r)rr?cCs4ts|jS|dk r(t|ts(t|}t||S)a Get the local tensor of this DTensor on its current rank. For sharding it returns a local shard of the logical tensor view, for replication it returns the replica on its current rank. Keyword args: grad_placements (List[:class:`Placement`], optional): the placements describes the future layout of any gradient layout of the Tensor returned from this function. `to_local` converts DTensor to local tensor and the returned local tensor might not be used as the original DTensor layout later in the code. This argument is the hint that user can give to autograd in case the gradient layout of the returned tensor does not match the original DTensor layout. If not specified, we will assume the gradient layout remains the same as the original DTensor and use that for gradient computation. Returns: A :class:`torch.Tensor` or `AsyncCollectiveTensor` object. it represents the local tensor on its current rank. .. note:: `to_local` is differentiable, the `requires_grad` of the local tensor returned will depend on if the `DTensor` requires_grad or not. N)r:Zis_grad_enabledrr_r2rr|)r[rr#r#r$rJszDTensor.to_localF)async_op)r=r1r~r?cCs|p|j}|dkrtdt|}t|D]B\}}|rDtdq*t|tr*|jdkr*t|j|j||<q*t |}t ||||S)aM `redistribute` performs necessary collective operations that redistribute the current DTensor from its current placements to a new placements, or from is current DeviceMesh to a new DeviceMesh. i.e. we can turn a Sharded DTensor to a Replicated DTensor by specifying a Replicate placement for each dimension of the DeviceMesh. Args: device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the DTensor, if not specified, must be called under a DeviceMesh context manager, default: None placements (List[:class:`Placement`], optional): the new placements that describes how to place the DTensor into the DeviceMesh, must have the same number of elements as `device_mesh.ndim`. Keyword args: async_op (bool, optional): whether to perform the DTensor redistribute operation asynchronously or not. Default: False Returns: A :class:`DTensor` object .. note:: `redistribute` is differentiable. Nz&placements is needed for redistribute!zTCan not redistribute to Partial, redistributing to Partial is for internal use only!r) r=rDryrEZ is_partialr_rr{rxr2r r|)r[r=r1r~irIr#r#r$rgs# zDTensor.redistributecCs&|jtg|jjdd}t||S)a Return the full tensor of this DTensor. It will perform necessary collectives to gather the local tensors from other ranks in its DeviceMesh and concatenate them together. It's a syntatic sugar of the following code: `dtensor.redistribute(placements=[Replicate()] * mesh.ndim).to_local()` Keyword args: grad_placements (List[:class:`Placement`], optional): the placements describes the future layout of any gradient layout of the full Tensor returned from this function. `full_tensor` converts DTensor to a full torch.Tensor and the returned torch.tensor might not be used as the original replicated DTensor layout later in the code. This argument is the hint that user can give to autograd in case the gradient layout of the returned tensor does not match the original replicated DTensor layout. If not specified, we will assume the gradient layout of the full tensor be replicated. Returns: A :class:`torch.Tensor` object that represents the full tensor of this DTensor. .. note:: `full_tensor` is differentiable. F)r1r~)rgrr=rxrr|)r[rZ redist_resr#r#r$ full_tensors zDTensor.full_tensor)r?cCs|jjS)z The :class:`DeviceMesh` attribute that associates with this DTensor object. .. note:: device_mesh is a read-only property, it can not be set. )rr0rZr#r#r$r=szDTensor.device_meshcCs|jjS)z The placements attribute of this DTensor that describes the layout of this DTensor on the its DeviceMesh. .. note:: placements is a read-only property, it can not be set. )rr1rZr#r#r$r1 szDTensor.placements)r#N)NN)NN)$r6r7r8r:r;__annotations__r __slots__ op_dispatchZ OpDispatcherrNr9Z_disable_dynamorLrXr\r]r^rhri classmethodrorrrrrCrrMr}rJrgrpropertyr=r1r#r#r#r$rsx  *  Q  %  7  )tensorr=r1r?c CsPtjd|pt}|j}|dkrrzddlm}||||WStk rp}zd}t||W5d}~XYnXt j st |rt |t _ |j std||jjkr|js||}|dkrdd t|jD}t||jkrtd t|d |jd t|trV|j|kr*td |jd|d |jt|krRtd|jd|d|S|}t|}t|D]\}} | rt t!| } | j"dkrt!| j"|j} | ||<| #|||}n:| $rt t%| } | &|||}ntd| d|dqnt|}|dk st'dt(||t)|*|+|j,dd} t|-|j.| |j.dS)a' Distribute a leaf torch.Tensor (i.e. nn.Parameter) to the ``device_mesh`` according to the ``placements`` specified. The rank of ``device_mesh`` and ``placements`` must be the same. If you want to construct a DTensor in the middle of the Autograd computation, please use ``DTensor.from_local`` instead. Args: tensor (torch.Tensor): torch.Tensor to be distributed. Note that if you want to shard a tensor on a dimension that is not evenly divisible by the number of devices in that mesh dimension, we use ``torch.chunk`` semantic to shard the tensor and scatter the shards. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to distribute the tensor, if not specified, must be called under a DeviceMesh context manager, default: None placements (List[:class:`Placement`], optional): the placements that describes how to place the tensor on DeviceMesh, must have the same number of elements as `device_mesh.ndim`. If not specified, we will by default replicate the tensor across the `device_mesh` from the first rank of each dimension of the `device_mesh`. Returns: A :class:`DTensor` or `XLAShardedTensor` object. Note: When initialize the DeviceMesh with the `xla` device_type, `distribute_tensor` return `XLAShardedTensor` instead. see [link](https://github.com/pytorch/pytorch/issues/92909) for more details. The XLA integration is experimental and subject to change. ztorch.dtensor.distribute_tensorxlar)xla_distribute_tensorDTo use DTensor API with xla, you must install the torch_xla package!NzY`distribute_tensor` should be used to distribute leaf tensors! but found non-leaf tensor!cSsg|] }tqSr#rprqr#r#r$rd`sz%distribute_tensor..zW`placements` must have the same length as `device_mesh.ndim`! Found placements length: z, and device_mesh.ndim: r@z-Cannot distribute a DTensor with device mesh z to a different device mesh z,Cannot distribute a DTensor with placements z to a different placements z-. do you want to call `redistribute` instead?z8Trying to distribute tensor with unsupported placements z on device mesh dimension !z(distributing a tensor should not be Noner()r0r1r-r.)/r:_C_log_api_usage_oncerrrrstorch_xla.distributed.spmdr ImportErrorrandomZ _rng_trackerrrZis_leafrDrQrtrurvrwrxlen ValueErrorr_rr=r1r2detachryrErzrrr{Z _shard_tensorrFrZ_replicate_tensorrUr rsizer*r+Zrequires_grad_r/) rr=r1rsremsgr"rHrIrOr#r#r$rs"              )moduler= partition_fninput_fn output_fnr?c stjdptj}|dkrvzddlm}|||WStk rt}zd}t||W5d}~XYnXt j t dddd } |dkr| D]\} } | | qn(| D]\} } || | | | qdk rRt tj} | d kr"tjd td d |fd dn0| dkrB|fddntd| ddk rt tj} | d krtjdtd d |fddn0| dkr|fddntd| d|S)a9 This function expose three functions to control the Tensors inside the module: 1. To perform sharding on the module before runtime execution by specifying the ``partition_fn`` (i.e. allow user to convert Module parameters to :class:`DTensor` parameters according to the `partition_fn` specified). 2. To control the inputs or outputs of the module during runtime execution by specifying the ``input_fn`` and ``output_fn``. (i.e. convert the input to :class:`DTensor`, convert the output back to torch.Tensor) Args: module (:class:`nn.Module`): user module to be partitioned. device_mesh (:class:`DeviceMesh`): the device mesh to place the module. partition_fn (Callable): the function to partition parameters (i.e. shard certain parameters across the `device_mesh`). If `partition_fn` is not specified, by default we replicate all module parameters of `module` across the mesh. input_fn (Callable): specify the input distribution, i.e. could control how the input of the module is sharded. `input_fn` will be installed as a module `forward_pre_hook` (pre forward hook). output_fn (Callable): specify the output distribution, i.e. could control how the output is sharded, or convert it back to torch.Tensor. output_fn will be installed as a module `forward_hook` (post forward hook). Returns: A module that contains parameters/buffers that are all `DTensor`s. Note: When initialize the DeviceMesh with the `xla` device_type, `distribute_module` return nn.Module with PyTorch/XLA SPMD annotated parameters. See [link](https://github.com/pytorch/pytorch/issues/92909) for more details. The XLA integration is experimental and subject to change. ztorch.dtensor.distribute_modulerr)xla_distribute_modulerN)mr0r?c Sstg|j}|jD]6\}}|dk rt|ts||tt |j ||q|j D],\}}|dk rZt|tsZt ||||j |<qZdSr) rrx _parametersitemsr_rZregister_parameternn Parameterrdata_buffers)rr0Zfull_replicatekeyparambufferr#r#r$replicate_module_params_bufferssz:distribute_module..replicate_module_params_bufferszDeprecating input_fn that takes two arguments (inputs, device_mesh), please use input_fn that takes in (module, inputs, device_mesh) instead!) stacklevelcs |Srr#)r3inputsr=rr#r$z#distribute_module..cs ||Srr#)modrrr#r$r rz-input_fn should take in 3 arguments, but got z arguments!zDeprecating output_fn that takes two arguments (inputs, device_mesh), please use output_fn that takes in (module, inputs, device_mesh) instead!cs |Srr#rroutputsr=rr#r$rrcs ||Srr#rrr#r$rrz.output_fn should take in 3 arguments, but got )r:rrrrrrsrrrrModulerZ named_modulesrinspect signature parametersrSrT FutureWarningZregister_forward_pre_hookrZregister_forward_hook) rr=rrrrsrrrrnameZsubmodnum_argsr#)r=rrr$rst&                )NN)NNNN)3rrStypingrrrrrrr:Z#torch.distributed._tensor._dispatch distributedZ_tensorZ _dispatchrZ torch.distributed._tensor.randomrZtorch.nnrZ+torch.distributed._tensor._collective_utilsrZ'torch.distributed._tensor._redistributer r Z torch.distributed._tensor._utilsr Z)torch.distributed._tensor.placement_typesr r rrrrrrZtorch.distributed.device_meshrr__all__ZopsZatenZautogradFunctionrr<r;rrrstrrr#r#r#r$sP      0]Q