U yh@sddlmZmZddlmZmZmZmZmZm Z ddl Z ddl m Z ddlmmZddlmZddlmZmZddlmZmZddlmZmZmZmZddlm Z Gd d d eZ!Gd d d e!Z"Gd dde!Z#dS))ABCabstractmethod)AnyCallableDictListOptionalTupleN)FakeTensorMode)DataParallelStylepartition_data_parallel)_convert_to_distributedSchema) DeviceMesh Placement ReplicateShard) GraphModulec @speZdZdZeeejje ej j e e efe e efeedfe e efedddZeeedddZd S) ParallelModez Basic Parallel Mode interface. Each parallelism pattern should implement this interface to describe how to partition and compile the graph in the spmd compiler. .gmmodel optimizerparams_and_buffers named_statesargskwargsreturncCstdS)z Partition a single device graph to a distributed graph. TODO(@wanchaol): some of these arguments are not necessary for partitioning, remove the unnecessary ones later. NNotImplementedError)selfrrrrrrrr!W/var/www/html/venv/lib/python3.8/site-packages/torch/distributed/_spmd/parallel_mode.py partitionszParallelMode.partitionrrcCstdS)a Transform and compile a distributed graph with a set of graph transformation and optimization passes for each parallel mode. The returned result should be a compiled executable graph in the distributed environment. Nrr rr!r!r"transform_and_compile,s z"ParallelMode.transform_and_compileN)__name__ __module__ __qualname____doc__rrtorchnnModuleroptim Optimizerrstrrr r#r&r!r!r!r"rs     rc @seZdZdZddddeeeeegefdddZ ee j j ee j jeeefeeefeed feeefed d d Zeed ddZdS) DataParallelzData Parallelism mode. replicaterN)input_batch_dim custom_passes)parallel_styler3r4cCsh|dkrtj|_n2|dkr$tj|_n |dkr6tj|_ntd|||_|dk rZ||_n dd|_dS)a DataParallel Mode that partition the model and graph to data parallel style parallelism (i.e. DDP/FSDP/ZERO-3). It currently supports three different parallel styles: "replicate", "fully_shard", and "default". See :class:`DataParallelStyle` for more details. Args: parallel_style (str): parallel style to use. Currently supports "replicate", "fully_shard", and "default". Keyword args: input_batch_dim (int): the batch dimension of the input tensor. default: 0 custom_passes (Callable[[GraphModule], GraphModule], optional): A custom callable that overrides the default graph transformation and optimization passes. r2Z fully_sharddefaultzUnknown parallel style: NcSs|SNr!rr!r!r"ez'DataParallel.__init__..)r Z REPLICATEr5Z FULLY_SHARDDEFAULT RuntimeErrorr3 _gm_passes)r r5r3r4r!r!r"__init__<s   zDataParallel.__init__.rc Cs6tdtt}t|||||||||j|j }|S)Ncuda)rr+arangedistget_world_sizer r5r3) r rrrrrrrmeshr!r!r"r#gs  zDataParallel.partitionr$cCs ||S)z>optimize a distributed graph with a set of optimization passesr=r%r!r!r"r&sz"DataParallel.transform_and_compile)r2)r'r(r)r*r0intrrrr>r+r,r-r.r/rrr r#r&r!r!r!r"r19s( -      r1c @seZdZdZd eeegefdddZeej j eej j e eefe eefeedfe eefeddd Zeed d d ZdS)DTensorExpandModez The DTensor Expand mode. It's replicating the parameters and shard the inputs to represent DDP like behavior, it's currently a transitent mode before we move to the new data parallel expansion. N)r4cCs$i|_|dk r||_n dd|_dS)NcSs|Sr7r!r8r!r!r"r9r:z,DTensorExpandMode.__init__..)_placements_overrider=)r r4r!r!r"r>szDTensorExpandMode.__init__.rc Cstj||}tdtt} t| t dgd} t| t gd} gg} } t |D]6}t |tj s|tdt|| || | qZt |D]@}t |tj r| || | q| td| | q|D]p}t |tj r8| |t||jkr,| t| |jt|dn | | q| td| | qtdddd| D}W5QRXt||| | d d dS) Nr?r)rCZ placementszexpecting Tensor but got T)Zallow_non_fake_inputscSsg|]}t|qSr!)r+Z empty_like).0Zinpr!r!r" sz/DTensorExpandMode.partition..F)Z default_meshZ_allow_partial)pytreeZarg_tree_leavesrr+r@rArBr?rrrZ tree_leaves isinstanceZTensorAssertionErrortypeappendemptyidrGr r )r rrrrrrrZ flat_argsrCZ shard_schemaZreplicate_schemaZinpsZschemaspoaZ fake_inpsr!r!r"r#sH            zDTensorExpandMode.partitionr$cCs ||S)z Transform and compile a distributed graph with a set of graph transformation and optimization passes for the dtensor fallback parallel mode. rDr%r!r!r"r&sz'DTensorExpandMode.transform_and_compile)N)r'r(r)r*rrrr>r+r,r-r.r/rr0rr r#r&r!r!r!r"rFs       9rF)$abcrrtypingrrrrrr r+Ztorch.distributed distributedrAZtorch.utils._pytreeutilsZ_pytreerJZtorch._subclassesr Z%torch.distributed._spmd.data_parallelr r Z"torch.distributed._spmd.distributer rZtorch.distributed._tensorrrrrZtorch.fxrrr1rFr!r!r!r"s    'O