U yhL @s ddlZddlZddlZddlZddlmZmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZddlZddlmZddlmmmZddlmmmZddlmmmZ ddl!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3m4Z4m5Z5m6Z6ddl7m8Z8dd l9m:Z:m;Z;mZ>m?Z?m@Z@mAZAdd lBmCZCdd lDmEZEdd lFmGZGdd lHmIZIerddlJmKZKdZLzddlMmNZNmOZOWnePk rdZLYnXeQdZRdZSeejTejTfZUeeejTeUfZVe?jWe6jWe?jXe6jXe?jYe6jYe?jZe6jZe?j[e6j[iZ\e?jZe?j[gZ]e?jYe?j[fZ^e de,eVe?eeCee'e,dddZ_e e,eVe'e,dddZ`e eeadddZbe e'eadddZce eQejTd d!d"Zde ejTeQejTd#d$d%ZeejTeQeejTejTfd#d&d'Zfe de,e"jgee ej"jgeee ej"jhee ej"jgfe,d(d)d*Zie eeadd+d,d-Zje e,e"jgee"jheeeQejkfe,d.d/d0Zle e,e"jge,d1d2d3Zme e,ee?ee>ee;eaeaeQeQe,d4 d5d6Zne e,e,d7d8d9Zoe e,e:eae,d:d;d<Zpe de,e'e,d=d>d?Zqe e,e,d7d@dAZre e,e"jgeeeQejkfeee"jggdfeae,dBdCdDZse e,e e"jhe"jgdEdFdGZte"jgee ej"jgee"jgdHdIdJZudej"jgeej"jgee ej"jheej"jhdKdLdMZvej"jgeej"jgeewdNdOdPZxe"jgeewdQdRdSZye"jgee"jheeeQejkfddTdUdVZzeeeQejkfeQeejkdWdXdYZ{e"jgee"jhee"jgeeaeafdZd[d\Z|e"jgee"jggdfee"jgdd]d^d_Z}e"jgeejkee"jgd`dadbZ~e"jgee"jge e"jgdNdcddZe"jgee"jheejeejkddedfdgZe e"jhe ejeejkddhdidjZdkdlZe"jgee"jheejkeQejkdmdndoZe"jge e"jhejTddpdqdrZe ejddsdtduZe"jgee"jhe e"jhdvdwdxZee"jhddydzd{Ze?d|d}d~ZejTe$jdddZdS)N)AnyCallableDequeDict GeneratorIterableIteratorList no_type_checkOptionalSetTuple TYPE_CHECKINGUnion) default_hooks)_mesh_resources DeviceMesh)_get_default_group)_FSDPDeviceHandle _FSDPState_get_module_fsdp_state_is_fsdp_flattened!_named_parameters_with_duplicatesclean_tensor_name TrainingState)_FSDP_USE_FULL_PREC_IN_EVAL FlatParameterFlatParamHandleHandleShardingStrategy)_FreeEventQueue)BackwardPrefetch CPUOffloadFullOptimStateDictConfigFullStateDictConfigMixedPrecisionShardingStrategyStateDictConfig StateDictType)_Policy)DTensorExtensions)_sync_params_and_buffers)is_traceable_wrapper_subclass)RemovableHandleT) deferred_initfakeFiZ _fsdp_synced)state process_groupsharding_strategypolicy device_meshreturncCs|dk r|dk rtd|tk}|r\|dkrN|dkrN|dkrNtd|dqt|||}n.|rv||_|jdd|_n|dk r|nt|_|j|_|j|_ |j }|r||j 9}t j ||_||j|_|S)NzcCannot pass both process_group and device_mesh at the same time. Please just pass only one of them.zManual wrapping with zA requires explicit specification of process group or device_mesh.rZmesh_dim) ValueErrorHYBRID_SHARDING_STRATEGIES*_init_process_group_state_for_hybrid_shard _device_mesh get_groupr0rranksize world_size_inter_node_pgr DefaultStateZ_get_gradient_predivide_factorZ_gradient_predivide_factorZ_gradient_postdivide_factor)r/r0r1r2r3Zis_hybrid_strategyZdata_parallel_world_sizer@T/var/www/html/venv/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py_init_process_group_stategs@   rB)r/r0r3r4cCs|rBt|r0||_|jdd|_|jdd|_qtd|jnX|dkrrt}t||j \}}||_||_n(t |r|\|_|_ntdt |t |jd|_|S)Nrr5z,Expected device_mesh to have ndim=2 but got zmExpected process_group to be passed in as either None or Tuple[dist.ProcessGroup, dist.ProcessGroup] but got r0)"_is_valid_hybrid_shard_device_meshr9r:r>r0r6ndimr!_init_intra_and_inter_node_groups_device_handleZ device_count_is_valid_hybrid_shard_pg_typetype_get_default_comm_hook_stateZ_inter_node_state)r/r0r3Z default_groupZintra_node_groupZinter_node_groupr@r@rAr8s2  r8)r0r4cCs(t|to&t|dko&tdd|DS)Ncss|]}t|tjVqdSN) isinstancedist ProcessGroup).0Zpgr@r@rA sz1_is_valid_hybrid_shard_pg_type..)rNtuplelenallrDr@r@rArIs   rI)r3r4cCst|to|jdkS)NrL)rNrrF)r3r@r@rArEsrE)num_devices_per_noder4cCst|\}}|S)aU Return a process group across the current node. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return an intra-node subgroup across [0, 1, ..., 7] or [8, 9, ..., 15] depending on the process's rank. For example, rank 3 would get [0, 1, ..., 7]. )rOZ new_subgroups)rVZintra_node_subgroup_r@r@rA_init_intra_node_process_groups rX)global_process_grouprVr4c sd}t|}t|}|}t|}tD]6fddt|D}tj||d}|kr6|}q6|dk st|d|S)a Return an inter-node process group where each contained rank has the same local rank. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return inter-node process group [0, 8], [1, 9], [2, 10], and so forth depending on the process's rank. For example, rank 1 would get [1, 9], rank 5 would get [5, 13]. Ncsg|]}|qSr@r@)rQiZ local_rankrVr@rA sz2_init_inter_node_process_group..)Zranksbackendz. expected to assign inter-node pg, but did not)rO get_backendZget_world_sizeZget_rankrangeZ new_groupAssertionError) rYrVZ inter_node_pgZsharding_backendr=Z num_nodesZ my_local_rankZranks_for_inter_groupgrpr@r[rA_init_inter_node_process_groups"    rbcCst|t||fS)a Initialize intra and inter-node process groups and return the ones corresponding to this process's rank. This function can be used to initialize process groups for ``HYBRID_SHARD`` or ``_HYBRID_SHARD_ZERO2`` in FSDP. This function assumes each node has an equal number of CUDA-enabled devices. Returns: Tuple[dist.ProcessGroup, dist.ProcessGroup]: Intra and inter-node process group. )rXrb)rYrVr@r@rArG srG)r/moduleignored_modulesignored_statesr4cCs|dk r|dk rtdd}|dk }|r.css|]}t|tjVqdSrMrNrhModulersr@r@rArRXscSsh|] }t|qSr@rJrsr@r@rA [sz(_check_ignored_states..)keyzUignored_states expects all nn.Parameter or all nn.Module list elements but got types css|]}t|tjVqdSrMrtrsr@r@rArRascSsh|] }t|qSr@rvrsr@r@rArwbsz>ignored_modules expects nn.Module list elements but got types )rTrUsortedreprr6)rerq all_paramsZ all_modulesZ sorted_typesr@r@rArgLs rg)r/rcignored_params device_idr4cCsd}|dk r&t|tjr|nt|}|dkrt||D]J}|jjdkrJq8|dkrZ|j}q8|jj|jkr8td|jd|jjq8|ptdtj}t ||_ |S)a Determine device handle used for initializing FSDP. If a device is specified by ``device_id``, then returns device handle corresponds to that device type. Otherwise, If the module is already on a non-CPU device, then the device type is that non-CPU device type. If the module is on CPU or meta, then the device type is the current cuda device. This method will be called once ignored paramters was determined, as the device handle maybe needed for other initialization. N>cpumetazLFSDP does not support modules with different device types but got params on z and cuda) rNtorchdevice_get_orig_paramsrJ RuntimeErrorrcurrent_devicerZ from_devicerH)r/rcr|r}Zdetermined_deviceparamr@r@rA_init_device_handleis,   r)r/rcr4cCs<t||_i}|D]\}}t|}|j||<q||_|SrM)_get_buffer_namesZ _buffer_names named_buffersrZdtype_buffer_name_to_orig_dtype)r/rcr buffer_namebufferr@r@rA_init_buffer_states  r) r/r1mixed_precision cpu_offloadlimit_all_gathersuse_orig_paramsbackward_prefetch_limitforward_prefetch_limitr4c Cs|jdkr4|tjkr,td|p"tjdtj}n|tjkrNtjdtdd|pVtj|_|pbt|_ |dk rt j dt |j tjtdd k|_|pt|_||_||_tj|_d|_t|_t|_t !|j|||_"d|_#t$}||_%d} | |_&g} | |_'|S) NrCz/FSDP is switching to use `NO_SHARD` instead of z since the world size is 1.zoThe `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.) stacklevelz'torch.distributed.fsdp.mixed_precision.1)(r=r%NO_SHARDwarningswarn FULL_SHARD FutureWarningr1r$rrZ_CZ_log_api_usage_oncestrosenvirongetrZ_use_full_prec_in_evalr!rr_use_orig_paramsrZIDLEZtraining_stateZ_is_rootrZ_free_event_queuerOZget_debug_levelZ _debug_levelexec_order_utilsZ_ExecOrderDataZ_exec_order_dataZ_unshard_eventdict_fully_sharded_module_to_handle_handleparams) r/r1rrrrrrrrrr@r@rA_init_core_statesP       r)r/r4cCs4g}||_g}||_g}||_d|_d|_d|_|S)NT)_root_pre_forward_handles_pre_forward_handles_post_forward_handlesZ_sync_gradientsZ _comm_hookZ_comm_hook_state)r/rrrr@r@rA_init_runtime_statesr)r/backward_prefetchforward_prefetchr4cCs||_||_|SrM)rr)r/rrr@r@rA_init_prefetching_statesr)r/r3r4cCs,|r"t|jdk r"t|j|_nd|_|SrM)rZget_parent_meshr9r)rH_fsdp_extension)r/r3r@r@rA_init_extensionsrcCs*tj|_t}t|_||_i}||_|SrM)r'ZFULL_STATE_DICTZ_state_dict_typer#r"Z_optim_state_dict_configZ_state_dict_configZ_unshard_params_ctx)r/Zstate_dict_configZunshard_params_ctxr@r@rA_init_state_dict_statesr)r/fully_sharded_moduler} param_init_fnsync_module_statesr4c st|j|t|j}t|jj\}}|s6|rN|dk rNt||jn.|rbt||jn|r|tj |fdddddjD}t |j||t |j|j_ t t|j} |rt|| jjtkrt|| jt| |S)zHInitialize a ``FlatParamHandle`` from a module ``fully_sharded_module``.Ncst|dko|jkSrM)rrk) submoduler/r@rA;s z0_init_param_handle_from_module..)Zcheck_fncSsh|]}|D]}|qqSr@buffers)rQZignored_modulerr@r@rArw?s z1_init_param_handle_from_module..)_check_single_device_modulerm_get_device_from_device_idr;_need_to_materialize_modulerk_materialize_with_param_init_fn_materialize_meta_moduler-Zmaterialize_module_move_module_to_device_get_compute_devicecompute_devicerfr_sync_module_params_and_buffersr0r1r7r>_init_param_handle_from_params) r/rr}rrdevice_from_device_idis_meta_moduleis_torchdistX_deferred_initignored_buffersmanaged_paramsr@rrA_init_param_handle_from_module!sl     r)r/rrc Cst|dkrdSt|||jt|j|jj|jj|jj |jj |j |j |j d }||jr^t|j|j||_||j|j<td}|jjr|jj|kr||dS)Nr)Zfsdp_extensionr~)rTrrSHARDING_STRATEGY_MAPr1rZoffload_paramsrZ param_dtypeZ reduce_dtypeZkeep_low_precision_gradsr0rrZshardrr`rappendZ flat_paramrZ_fully_sharded_modulerrZ flat_param_to)r/rrhandle cpu_devicer@r@rAr_s.     r) root_modulerkr4c Csd}z|dk rt|nt}Wn:tk rX}zt|dt||W5d}~XYnX|D]8}t|tjjst|dt|t|r^tdq^| D]}t |s| |qdd|D}||krt d|| D]0}t|}|dk rt|d s t||jq|S) ah Check that ``_ignored_modules`` is an iterable of ``nn.Module`` s without any FSDP instances. Return the modules contained in their module subtrees as a :class:`set`. Nested FSDP instances are excluded, but their already-computed ignored modules are included. ``_ignored_modules`` represents the argument passed by the user to FSDP. z>`ignored_modules` should be an iterable of `torch.nn.Module`s Nzbut got zbut got an iterable with z1`ignored_modules` should not include FSDP modulescSs*h|]"}|D]}t|tjs|qqSr@)modulesrN fsdp_fileFullyShardedDataParallel)rQrcchildr@r@rArws   z'_get_ignored_modules..zTrying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: rk)set TypeErrorrJrNrrhrurr6rtraversal_utilsZ _composableaddrrhasattrr`updaterk) rrkZ msg_prefixZignored_root_modulesercrdroptional_fsdp_stater@r@rArj~s6 *     rj)rrdrpr4cCszt}dd|D}|||dk r>dd|D}|||D].}t|}|dk rFt|dsht||jqF|S)z Return the parameters of the modules in ``ignored_modules`` and the parameters in ``ignored_parameters``. :class:`FlatParameter` s are excluded from the result. cSs&h|]}|D]}t|s|qqSr@) parametersr)rQmpr@r@rArws  z&_get_ignored_params..NcSsh|]}t|s|qSr@)rrQrr@r@rArwsrm)rrrrrr`rm)rrdrpZall_ignored_paramsZparams_in_ignored_modulesZparams_in_ignored_parametersrrr@r@rArls    rl)rrdr4cslt}dd|D|fdd|D|D].}t|}|dk r8t|dsZt||jq8|S)z6Return the cleaned buffer FQNs in ``ignored_modules``.cSsh|]}|D]}|qqSr@r)rQrrr@r@rArws z,_get_ignored_buffer_names..cs h|]\}}|krt|qSr@r)rQrrZbuffers_in_ignored_modulesr@rArwsNro)rrrrrrr`ro)rrdZall_ignored_buffer_namesrrr@rrArns  rn)rr4cCsdd|DS)zrReturn the fully prefixed names of all buffers in the module hierarchy rooted at ``root_module`` as a class:`set`.cSsh|]\}}t|qSr@r)rQrrWr@r@rArwsz$_get_buffer_names..)r)rr@r@rArsr)rcr|r}r4cCs^ddt||D}t|dkr@td|kr@|dkrZtdnt|dkrZtd|dS) z Raise an error if ``module`` has original parameters on multiple devices, ignoring the parameters in ``ignored_params``. Thus, after this method, the module must be either fully on the CPU or fully on a non-CPU device. cSsh|] }|jqSr@rrQrr@r@rArwsz._check_single_device_module..rLr~NzTTo support a module with both CPU and GPU params, please pass in device_id argument.rCz;FSDP only supports single device modules but got params on )rrTrrr)rcr|r}Zdevicesr@r@rArs  r)r}r;r4c Csn|dkr dSt|tjr|nt|}|tdkrjtd|d|dtjdtdtj}|S)z Return a ``torch.device`` for the specified ``device_id``. Processes ``device_id`` and returns either the corresponding device or ``None`` if ``device_id`` is ``None``. Nrz"FSDP got the argument `device_id` z on rank zJ, which does not have an explicit index. FSDP will use the current device z. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.)rNrrrrrr)r}r;rr@r@rArs r)rcr|rdr4cCsxtt||}tdd|D}|D]*}||kr6q(|jddD]}||jO}qBq(| ontontdd|D}||fS)z Return if ``module`` has parameters on meta device and if ``module`` is using torchdistX deferred initialization. At most of the returned bools can be ``True``. If either is ``True``, then ``module`` needs to be materialized. css|] }|jVqdSrM)is_metarr@r@rArRDsz._need_to_materialize_module..Frecursecss|]}t|VqdSrM)r.Zis_fakerr@r@rArRPs)rfranyrrr_TORCHDISTX_AVAIL)rcr|rdrrrbufrr@r@rAr7s  r)rrrdr4cCs@t|s td|dt|t||}|D] }||q.dS)Nz Expected z to be callable but got )callabler6rJ_get_modules_to_materialize)rrrdmodules_to_materializercr@r@rArUs r)rrrdc Cs|pttj}t||}zhtV|D]J}t|jdd|j dd}t t |dk}|r.|j |dd| q.W5QRXWnFtk r}z(tdt|dt|d|W5d}~XYnXdS)NFrr)rrzIUnable to call `reset_parameters()` for module on meta device with error z(. Please ensure that your module oftype z* implements a `reset_parameters()` method.)rrrrrno_grad itertoolschainrrrTrfZto_emptyZreset_parameters BaseExceptionrrrrJ) rrrdZmaterialization_devicerrcZmodule_state_iterZhas_module_statesrr@r@rArcs(   rcCspg}t|g}|h}|rl|}|||D]4}||kr4t|dkr4||kr4||||q4q|SrM) collectionsdequepopleftrchildrenrr)rrdrqueueZvisited_modulesrcZ child_moduler@r@rArs      r)rcr|rrr4c std|dk rt}||g}g}|r|}|fdd|jddD|fdd|jddD| D]}t |t j s||qq,fdd |D} fd d |D} t | | |dStt|d} | dk r| jkrtdS) a Move ``module`` depending on ``device_from_device_id`` and its current device. This includes moving ignored modules' parameters. - If ``device_from_device_id`` is not ``None``, then this moves ``module`` to the device. - If ``device_from_device_id`` is ``None``, then this does not move ``module`` but warns the user if it is on CPU. Precondition: ``_check_single_device_module()``. r~Nc3s|]}|jkr|VqdSrMrrrr@rArRs z)_move_module_to_device..Frc3s|]}|jkr|VqdSrMr)rQrrr@rArRs csg|]}|kr|qSr@r@r)r|r@rAr\sz*_move_module_to_device..csg|]}|kr|qSr@r@r)rr@rAr\s)rrrrrrextendrrrrNrr_move_states_to_devicenextr_warn_cpu_init) rcr|rrrrrZ curr_modulerZparams_to_moveZ bufs_to_moverr@)rrr|rArs0       r)rrrr4c Cst|dkrt|dkrdSt|dkr4|dj}nt|dkrJ|dj}td}|dk r|D]>}t,|||_|jdk r|j||j_W5QRXq`|D]}|||_qn||krtdS)z Move states to the specified device. Precondition: ``_check_single_device_module()`` and module's parameters and buffers have been materialized if needed. rNr~)rTrrrtodataZgradr)rrrrrrrr@r@rArs"         rcCstddS)NamThe passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.)rrr@r@r@rArsr)rcr|rr;r4cCstt||d}|dk r,|jjdkr,|j}n*|dk rD|jdkrD|}ntdtj}|dk r||krtd|d|d||S)ax Determine and return this FSDP instance's compute device. If a device is specified by ``device_id``, then returns that device. Otherwise, If the module is already on a non-CPU device, then the compute device is that non-CPU device. If the module is on CPU, then the compute device is the current device. Since this method should be called after materializing the module, any non-CPU device should not be meta device. For now, the compute device is always a CUDA GPU device with its explicit index. Precondition: ``_check_single_device_module()`` and ``_move_module_to_device()``. Nr~rz4Inconsistent compute device and `device_id` on rank z: z vs )rrrrJrrrr6)rcr|rr;rrr@r@rArsr)rcrr0r4c sg}|D]`}t|tds t|td|trb\}}fdd|D}||q |q |D]H}|tr\}}fdd|D} || qr|qrt |t ||t dddS) z Synchronize module states (i.e. parameters ``params`` and all not-yet-synced buffers) by broadcasting from rank 0 to all ranks. Precondition: ``sync_module_states == True`` and ``self.process_group`` has been set. FTcsg|]}t|qSr@getattrrQattr)detached_bufferr@rAr\7sz3_sync_module_params_and_buffers..csg|]}t|qSr@rr)detached_paramr@rAr\@sr)srcN) rr FSDP_SYNCEDsetattrdetachr+Z__tensor_flatten__rr+_check_module_states_for_sync_module_statesr*PARAM_BROADCAST_BUCKET_SIZE) rcrr0 module_statesrattrsrWZ inner_buffersrZ inner_paramsr@)rrrAr"s0          r)rr4cCs"|rtdd|DrtddS)Ncss|]}|jtdkVqdS)r~N)rr)rQZtensorr@r@rArRQsz>_check_module_states_for_sync_module_states..zThe module has CPU parameters or buffers when `sync_module_states=True`, which requires them to be on GPU. Please specify the `device_id` argument or move the module to GPU before passing it to FSDP.)rr6)rr@r@rArNs  r)rcr|r4ccsF|}z$t|}||kr t|s |Vq Wntk r@YnXdS)aD Return an iterator over the original parameters in ``module``. The iterator does not return the parameters in ``ignored_params``, any ``FlatParameter`` s (which may be present due to nested FSDP wrapping), or any original parameters already flattened (only relevant when ``use_orig_params=True``). N)rrr StopIteration)rcr|Z param_genrr@r@rAr[s  r)r|r4cCsFt|D]8\}}||krt|std|d|d|jqdS)a5 Check that original parameters in ``fsdp_module`` have been flattened. The flattened parameters are made invisible to ``named_parameters()`` for the module hierarchy rooted at ``fsdp_module``. This should be called as a sanity check after flattening the wrapped module's parameters. z Found an unflattened parameter: z;  N)rrrr< __class__)Z fsdp_moduler| param_namerr@r@rA_check_orig_params_flattenedqs rr1cCs|tjkrtjStjSrM)r%rrZallreduce_hookZreduce_scatter_hookrr@r@rA_get_default_comm_hooksrcCs tj|dS)NrD)rr?rDr@r@rArKsrK)N)N)N)N)rrrrtypingrrrrrrrr r r r r rrrZtorch.distributed distributedrOZ(torch.distributed.fsdp._exec_order_utilsZfsdpZ_exec_order_utilsrZ'torch.distributed.fsdp._traversal_utilsZ_traversal_utilsrZ2torch.distributed.fsdp.fully_sharded_data_parallelZfully_sharded_data_parallelrZtorch.nnrhZ(torch.distributed.algorithms._comm_hooksrZtorch.distributed.device_meshrrZ"torch.distributed.distributed_c10drZ$torch.distributed.fsdp._common_utilsrrrrrrrZ"torch.distributed.fsdp._flat_paramrrrrZ%torch.distributed.fsdp._limiter_utilsrZtorch.distributed.fsdp.apir r!r"r#r$r%r&r'Ztorch.distributed.fsdp.wrapr(Z&torch.distributed.tensor.parallel.fsdpr)Ztorch.distributed.utilsr*Ztorch.utils._python_dispatchr+Ztorch.utils.hooksr,rZ torchdistxr-r. ImportErrorintrrrPZHybridShardProcessGroupTypeZProcessGroupTyperrZ SHARD_GRAD_OPZ HYBRID_SHARDZ_HYBRID_SHARD_ZERO2rr7Z#NO_RESHARD_AFTER_FORWARD_STRATEGIESrBr8boolrIrErXrbrGrurirrrgrrrrrrrrrrrjrlrrnrrrrrrrZTensorrrrrrrrrrr?rKr@r@r@rAs@    $  (       3+% / ,A   = <  #       "  7   ( -