U yh @sUddlZddlZddlZddlZddlmZmZmZddlm Z ddl m Z m Z m Z mZmZmZmZmZmZmZmZmZddlZddlmZddlmZddlmZddlm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(dd l)m*Z*m+Z+m,Z-m.Z.m/Z/m0Z0m1Z1m2Z2dd l3m4Z4m5Z5dd l6m7Z7dd l8m9Z:dd l;mdZ?dZ@dZAeeBZCee&eejDeEeFeBfZGeeGeeGeeGeeBdffZHeeBeHfZIeeIZJeeBeeIeJffZKeLZMee eNd<ejOd d!ZPeGd"ddZQeGd#d$d$eQZRejSdd%diejTeBeUeUeCd'd(d)ZVGd*d+d+ZWd,d-ZXddd.ejTeejYjZd/feUeeejTeeQeRd0d1d2Z[eeBeHfeKeRdd3d4d5Z\eejTejYjZfeBe d6d7d8Z]eeBe feReeBe fd9d:d;Z^ejTeReeBeHfd<d=d>Z_ejTeeBeHfeRe7d?d@dAZ`ejYjZddBdCdDZaeKeeBeHfdEdFdGZbejYjZeeBeHfeReKdHdIdJZcejTeejYjZd/feReKdKdLdMZdejTejYjZeKeReKdNdOdPZeejTeejYjZd/feKeRddQdRdSZfddd.ejTeeejTeeQeeBeHfdTdUdZgddd.ejTeejYjZeejYjZfeeejTeeQeKdVdWdZhddd.ejTeejYjZeejYjZfeeejTeeQeeeBeHfeKfdVdXdZiejTeeejTeeBeHffeeBeHffeeBeHfdYdZd[Zjdd\ejTeeBeHfeeQe7d]d^dZkdd\ejTeejYjZeejYjZfeKeeQdd_d`dZldd\ejTeejYjZeejYjZfeeBeHfeKeeQe7dadbdZmedd\ejTeeQddcdddeZnedd\ejTeejYjZd/feeQddfdgdhZodS)jN)asdict dataclassfield)chain) AnyCallablecastDict GeneratorIterableList no_type_checkOptionalSetTupleUnion) ShardedTensor)_broadcast_state_dict_flatten_state_dict_gather_state_dict_offload_state_dict_to_cpu_unflatten_state_dict)DTensor)_CHECKPOINT_PREFIX)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)_IncompatibleKeys)DistributedDataParallel) tree_map_onlyFQNS_T PrimitiveType ValueType DictValueTypeListDictValueTypeOptimizerStateTypeStateDictOptionsget_model_state_dictget_optimizer_state_dictget_state_dictset_model_state_dictset_optimizer_state_dictset_state_dictZ _flat_param param_groupsparamsstate_patched_state_dictccs.t}tz dVW5|r(tXdSN)gc isenableddisableenable) is_enabledr>Y/var/www/html/venv/lib/python3.8/site-packages/torch/distributed/checkpoint/state_dict.py _gc_context[s  r@c@sfeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <d S) r-ap This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes`` (deprecated): when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). - ``broadcast_from_rank0``: when the option is True, rank0 should receive a full state_dict and will broadcast the tensors in the state_dict/ optim_state_dict one by one to other ranks. Other ranks will receive the tensors and shard according to the local shards in the model and optimizer. ``full_state_dict`` must be set to True when using this option. This option currently only supports DTensor, not the legacy ShardedTensor. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictbroadcast_from_rank0flatten_optimizer_state_dictN) __name__ __module__ __qualname____doc__rAbool__annotations__rBrCrDrErFrGr>r>r>r?r-fs $      c@seZdZUeedZeeee j fee e j ffe d<eedZ eeee j fee e j ffe d<eedZeee d<dZee d<dZee d<ejZee d<eedZeeje d <d S) _StateDictInfo)default_factoryfqn_param_mappingshared_params_mappingsubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)rHrIrJrdictrPr rstrtorchTensorr'rMrQsetrRrrSrLrT contextlib nullcontextrUrlistrVr nnModuler>r>r>r?rNs   rN)maxsizeT)modelnameskip_ddp_prefixskip_compiler_prefixreturnc s|td}d|kr|hS|d}g}|}t|D]8\}}t|trj|dksTt|j}|sh||q4t|t r|t |dkr||dt krd |t |t } rdfdd| jDSt |t}|tkr||t ||}q4t|tjjjr.|dkst|j}|sn||q4|||tjjjkrd|t |dkrntdq4t ||}q4d |tdhS) a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `Set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. .modulecsh|]}|qSr>r>.0fqnprefixr>r? sz_get_fqns.. _orig_modz-Expect `_extra_state` to be the last obj name)replacersplit enumerate isinstanceDDPAssertionErrorriappendFSDPlen _FLAT_PARAMjoingetattrZ_fqnsr#rYZ_dynamoZ eval_frameZOptimizedModulerqr_modules_EXTRA_STATE_KEY_SUFFIX RuntimeError) rbrcrdreZ obj_namesZ fqn_obj_namesZcurr_objiZ curr_obj_nameZ flat_paramr>rnr? _get_fqnssD                 rc@s eZdZdS) _EXTRA_STATEN)rHrIrJr>r>r>r?rsrc#s4ttjttdfdd |dEdHdS)N)ricurr_fqnrfc3s||r|dnd}|D].\}}|kr6q$||}||EdHq$t|jdd|jddD]*\}}||jkrqn||}||fVqnt|jdtj j tj j kr|tj j j }|tfVdS)NrhrgF)recurseget_extra_state)addZnamed_childrenrZ named_buffersnamed_parametersZ_non_persistent_buffers_setr} __class__r_r`rr~rirr)rirrc submodulenew_fqnobjrZvisited_modulesr>r?rs*      z+_iterate_valid_model_state..recurserg)r[r_r`rXr )rbr>rr?_iterate_valid_model_statesr) submodulesoptions.)rboptims optim_onlyrrrfc Cs6|rtdt|r |s td|p(t}i}i}t|D]\}}t|trNq:t||} | |d} | dk rt t t || | ||||<n | ||<| D]} t|ts||| <qq:t|D]"\} } | D]} t tj| || <qqt} |rVt|}|D]L\}}||krqt||} t| dks>td| dd| Dq|jrn|jsntdt|}|r|jrt|j|jd }t|j|jp|jd }tj }nt!|jd }t"|jd }tj#}t$j%d d }t&j'|||||d }nt$j(}t)ft*|||| |t t+t,j-|| t|dkdS)zW Verify the model and options passed by the user and generates _StateDictInfo. zGetting submodules only model/optim state_dict is deprecated and will be removed in 2.5. This feature can be achieved by manually filtering out the state_dict returned from get_state_dict.z;Optimizers are not passed in but optim_only is set to True.Nrjz)Submodule FQN should only have 1 instancecss|]}|dVqdS)rhNr>rkr>r>r? Asz"_verify_options..z?full_state_dict must be True when broadcast_from_rank0 is True.)offload_to_cpuZ rank0_only)rc ss<t*tj||||d dVW5QRXW5QRXdS)Nristate_dict_typestate_dict_configoptim_state_dict_config)warningscatch_warningsryrrr>r>r?$fsdp_state_dict_type_without_warning_s z=_verify_options..fsdp_state_dict_type_without_warningrr)rPrQrRrUrVrSrT).rwarn FutureWarningrr-rrurrgetrrrXupdatecopyr^itemsrYrZr[ named_modulesrzrwrFrA ValueErrorryrVrrBrr!ZFULL_STATE_DICTrrZSHARDED_STATE_DICTr\contextmanager functoolspartialr]rNrr r_r`)rbrrrrrPrQrcparamfqnsrmZparam_Zfqns_rRrirVrrrrrUr>r>r?_verify_optionss              r)model_state_dictoptim_state_dictinforfcCs|jD]}t|}|dk stdq|jrf|sf|jsf|jsf|jrF|jsf|jrf|j sft dt d|j r|s|jr||js|j st d||D] }t|krt |dtdqdS)Nz)Expected a fsdp_state with a fsdp module.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=rhzgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)rVr"rwrSrRrCrBrArErFrdistget_rankrTkeysr{)rrrriZ fsdp_statekeyr>r>r?_verify_state_dictsN   r)rapirfcCs,t||}|tkr(tjt|j||d}|S)N)self)r}r7rrr)rrcallr>r>r?_state_dict_fns r) state_dictrrfcCsF|jr0|jrtjstnd}t||j|dS|jr>t|S|SdS)N)r)rB ranks_only)rArBrY distributedZis_initializedtuplerr)rrrr>r>r?_maybe_full_or_cpu_state_dictsr)rbrrfc Cs|js iS|t|d}W5QRXt|D]t}t||}t|dks\t||ftt |}||kr6t ddd}|||st d|d|| |||<q6|j ri}|D]L}|j D]@}||sq|jr||||<q|t|d} |||| <qq|}|jrZ|D]6\}} | jr6q"t||}|D]}| |qDq"t|D](\}} t| rf| jrf| |qft||S)Nrrj)rfcSst|t|krdS|d}|d}d}t|D]P\}}|||krr|d7}|t|kr|t|dkSq4|dkr~q4q4dSq4dS)NFrhrrj)rirqT)rzrsrt)rrmZ fqn_splitZ key_splitZfqn_idxZkey_idxZkey_namer>r>r?verifys    z%_get_model_state_dict..verifyzAn unexpected key, z, exists. FQN is )rSrUrr^rrrzrwnextiterrLrpoprR startswithrDrCr requires_gradrrY is_tensoris_metar) rbrrrrrmrnew_state_dictrorrpr>r>r?_get_model_state_dictsF        r)rbrrrfc CsT|jr|s|jstiiSi}t|D]f\}}t||}t||ddd}t||D]8\}} |jrltdkr|| kr|||| <||| <qRq&|jrd} | D]<\}}t |r| dkr| dkr|j } q| |j kstq| dk stt||| |jd| D]\}} | ||<q|(ttt|d||jdW5QRSQRXdS)NF)rdrer)devicerEload_state_dict)rrE)rSrFr$rrziprrrrrYrdimrrwrrErUrr) rbrrlocal_state_dictrvaluerZfqns_with_prefixrmZfqn_with_prefixrZ local_stater>r>r?_load_model_state_dictsX       r)optimrfcCs|jr dS|jD]6}|tD](}|jdk r2td|jrt||_qqg}|jD]"}d|krR||dd|d<qR|j dd|jD]}d|kr| d|d<q|j dddS) zH Initialize optim states by calling the step() with zero grads. Na state_dict can only be used if the optimizer states are initialized (usually after one step() with gradients) or gradients are None. For the later case, state_dict will fake the gradients as zero to initialize the optimizer states. However, the gradients are not None.lrg)closurerT)Z set_to_none) r6r4_PARAMSZgradrrrYZ zeros_likerxsteprZ zero_grad)r param_grouprZlrsr>r>r?_init_optim_state<s(       r)rrfc Csdd}i}tt|tD]@\}}tt|D](\}}||||td|d|<q4qtt|tD]L}|t}ttt |D].}|D] \}}||td|d|<qqqn|S)aI This API flattens the optimizer state_dict to support optimizer resharding for MPMD, e.g., pipeline parallelism. Without the API, the original optimizer state_dict looks like: { "state": { "layer1.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, "layer2.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, }, "param_group": [ { "lr": 0.0, "betas": (0.9, 0.95), ..., "params": ["layer1.weight", "layer2.weight"] } ] } With this API, the optimizer state_dict looks like: { "state.layer1.weight.step": 10, "state.layer2.weight.step": 10, "state.layer1.weight.exp_avg": SomeTensor, "state.layer2.weight.exp_avg": SomeTensor, "state.layer1.weight.exp_avg_sq": SomeTensor, "state.layer2.weight.exp_avg_sq": SomeTensor, "param_group.layer1.weight.lr" : 0.1, "param_group.layer2.weight.lr" : 0.1, "param_group.layer1.weight.betas" : (0.9, 0.95), "param_group.layer2.weight.betas" : (0.9, 0.95), } Note that if any of the value is a container, like the betas in the example, this API won't flattent it. cSs*t|tjttfs&tdt|ddS)NzUFlattening optimizer state_dict only supports tensor, int, float states now. Type is rh)rurYrZintfloatNotImplementedErrortype)vr>r>r?_raise_if_type_not_supportedsz?_flatten_optim_state_dict.._raise_if_type_not_supportedrh) rr*_STATErr+_PGrrr rX) rrretrmr6krrrr>r>r?_flatten_optim_state_dictbs* r)rrrrfc Cs`i}g}t|t|i}|jD]>}|tgi|tD]}|j|D]r}|dt} t| tsbt| ||j stqDi||<|j | D]*} |td|d| t t ||| <qqDq6t tt|dtd} | D]|} | tkrq|td| d| } | |dkr| |d| <q|d| | krtd| d| d| d|d| d qq|S)z This API unflattens the state_dict generated by _flatten_optim_state_dict(). See the docstring of _flatten_optim_state_dict() for more detail. rhrzaAll the parameters in the same parameter group should have the same saved param_group value. But z is z while other(s) is )rrr4rxrrPrur^rwrr6rrr*r rXr)rrrr6pg_state return_osdrrrmr5Z state_nameZfirst_param_fqnrrr>r>r?_unflatten_optim_state_dicts:       $r)rb optimizersrrfc s|js iStitgi}|D]}t|t|d}|jr|t|||}W5QRX|sbqt |t D]*}d|krr|t ||t| dd<qr|tD]}dd|t D}||t <qnt tdd|jD}tt|tt|} i|D]X\} } t|| } t| d ks(ttt| } | | krBq| | }|| <| |<qt |t D]$} | } |t | |t| <qn|tD] }fd d|t D|t <q|sqtt|t|ttt|t|tq|jrtt t!|}t"||S) Nrrq _orig_mod.rgcSsg|]}|ddqS)rrgrrrlrr>r>r? sz)_get_optim_state_dict..css|]}|tVqdSr8)r)rlgr>r>r?rsz(_get_optim_state_dict..rjcsg|] }|qSr>r>)rlpidZfqn_pid_mappingr>r?rs)#rTrrrrrVrUryrr^rrrrrr from_iterabler4rWrrangerzrrrwrrrr*rr+extendrGr,rr)rbrrrrZosdrrr5Zparam_pid_mappingrrrrmrgroupr>rr?_get_optim_state_dictsX            r)rbrrrrfc Csi}g}t|t|i}i}tddtt|tDr<|S|jD]}|tgi|tD]} |j | D]} | |j krd} tt |tD]"} | tt t | tkrd} qqnd} | sql|dt} t| tst| | | jrtt|t| || <tt |tD]6} | tt t | tkr t|td|t| <q qlq^qBtt |tD]R}|t|d}|dkrxqV|D]$\}}|tkrq||||<qqV|S)a Extract the corresponding optim state_dict from ``optim_state_dict`` for ``optim`` and return the result optim state_dict. Args: model (nn.Module): the root model. optim (torch.optim.Optimizer): the optimizer. optim_state_dict (Dict[str, ValueType]): the superset optim state_dict that contains the optim state_dict of ``optim``. info (_StateDictInfo): state dict information. Returns: The optim state_dict of ``optim``. css|]}t|tVqdSr8)rurrr>r>r?r(sz*_split_optim_state_dict..FTrrj)rrallrr*rr4rxrrPrQr+r rXrur^rwrrzidrr)rbrrrr6rrZ pg_mappingrrrmZ in_paramsZloaded_param_groupr5idxrrr>r>r?_split_optim_state_dictsX      "  r)rbrrrrfc s.|js dS|D]}t||rTt|kr8t||||}qXt|ttttf||}ni}|j rZ| D]\}}t ||}t ||dd} || krqht |dkst || |tD]6} ttttf| } fdd| tD} | | t<qtt|t} t| D]&}|kr | || |<q qh|t|||}W5QRXn|jrd|_t||f|}d|_dfdd}ttj||}dk st t|\}}t|\}}t ||d |D]6}||kr||kst ||||<||||<qt!||}t"|d |d qdS) NF)rerjcsg|]}|qSr>r)rlr)rmfqn_with_compilerr>r?r|sz*_load_optim_state_dict..Tcs2|dkr.dkr|jn|jkr.td|S)NrzDevice mismatch)rrr)trr>r?_devices   z'_load_optim_state_dict.._devicerrr)#rTrrrrrr rXr)rVrrrzrwrrrrr*r^rrrrUryZoptim_state_dict_to_loadrFrArr&rYrZrrrr)rbrrrrrZ original_fqn_rZfqns_with_compilerrvalr5Z osd_staterrrZ flatten_osdZ osd_mappingZflatten_local_osdZlocal_osd_mappingZ optim_keyr>)rrmrr?_load_optim_state_dictVs             r)rbrrrfc CsLt<t|td||d}t||}t|i||W5QRSQRXdS)aH Return the model state_dict of ``model``. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. submodules (deprecated): Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``model``. :rtype: typing.Dict[str, ValueType] FrrrN)r@rrrr)rbrrrrr>r>r?r.s  )rbrrrrfc CshtXt|tjjr|fnt|}t||d||d}t|||}ti|||W5QRSQRXdS)a Return the combined state_dict for optimizers. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``optimizers``. :rtype: OptimizerStateType TrN) r@rurYr Optimizerrrrr)rbrrrrrr>r>r?r/s   c Csvtft|tjjr|fnt|}t||d||d}t||}t|||}t |||||fW5QRSQRXdS)a Return the model state_dict and optimizers state_dict. ``get_state_dict`` can process any module that is parallelized by PyTorch FSDP/fully_shard, DDP/replicate, tensor_parallel/parallelize_module, and any combination of these parallelisms. The main functions of ``get_state_dict`` are: 1.) returning a model and optimizer state_dict that can be resharded with a different number of trainers and/or different parallelisms. 2.) hiding the parallelism-specific state_dict APIs. Users don't have to call these APIs. 3.) sanity checking the result state_dict. The keys of the result state dictionary are the canonical FQNs (Fully Qualified Names). A canonical FQN refers to the FQN based on a parameter's position in an nn.Module hierarchy. More specifically, a canonical FQN to a parameter is the FQN returned by ``module.named_parameters()`` or ``module.named_buffers()`` when the module is not distributed by any parallelisms. Since the optimizer internally uses parameter IDs to represent a parameter, there will be a conversion from the parameter IDs to the canonical FQNs when calling this API. ``get_state_dict`` can also process a module that is not parallelized. In such a case, ``get_state_dict`` only performs one function -- converting the optimizer parameter IDs to the canonical FQNs. Example: >>> # xdoctest: +SKIP >>> import torch >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.nn.parallel import DistributedDataParallel as DDP >>> from torch.distributed.checkpoint.state_dict import get_state_dict >>> fsdp_model = FSDP(copy.deepcopy(model)) >>> fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_model = DDP(copy.deepcopy(model)) >>> ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) >>> fsdp_state_dict, fsdp_optim_state_dict = get_state_dict(fsdp_model, fsdp_optim) >>> # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), >>> # the asserts will fail. >>> assert ddp_state_dict == fsdp_state_dict >>> assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. :rtype: typing.Tuple[typing.Dict[str, ValueType], OptimizerStateType] FrN) r@rurYrrrrrrr)rbrrrrrrr>r>r?r0s F    )rbrrfc s|siSttt|tjrtdtt t tjt t t ff|}i}| D]p\}}|D]^\}}||krvqdt||}t|dkstdtt|d|fdd| DqdqT|St t t t f|SdS)NzPassing model_state_dict as a ``Dict[nn.Module, Dict[str, Any]]``is deprecated and will be removed in 2.5. If you need this feature, please preprocessing the model_state_dict to achieve the same functionality.rjz/FQNs for a submodule should only have 1 elementrhcsi|]\}}||qSr>r>)rlZsubfqnrrnr>r? usz/_unflatten_model_state_dict..)rurrrr_r`rrrrr rXr)rrrrzrwr) rbrZcast_state_dictrrZsub_state_dictrcmrr>rnr?_unflatten_model_state_dict[s* r)r)rbrrrfc CsRt||}t8t|td|d}t|i|t|||W5QRSQRXdS)a=Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Dict[str, ValueType]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys :type model_state_dict: typing.Dict[str, ValueType] FrrN)rr@rrrr)rbrrrr>r>r?r1|s )rbrrrrfc Cs\tLt|tjjr|fnt|}t||d|d}ti||t||||W5QRXdS)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None :type optim_state_dict: typing.OptimizerStateType TrN) r@rurYrrrrrr)rbrrrrr>r>r?r2s  )rbrrrrrfc Cs|t||}tbt|tjjr&|fnt|}t||| |d}t|||t ||||t |||W5QRSQRXdS)a4Load the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. :type model_state_dict: typing.Dict[str, ValueType] :type optim_state_dict: typing.OptimizerStateType rN) rr@rurYrrrrrrr)rbrrrrrr>r>r?r3s$*  )rbrrfcsjtjt||dfdd}||_tjt||dtttfdfdd }||_t |t |dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rbrcsSr8r>r>_state_dict_callr>r?state_dict_call)sz0_patch_model_state_dict..state_dict_callrcs|ddS)N)rr>r_load_state_dict_callr>r?load_state_dict_call4sz5_patch_model_state_dict..load_state_dict_callN) rrr.rr1r rXrrr7r)rbrrrr>rrr?_patch_model_state_dicts   r )rbrrrfcstjt|||dfdd}tjt|||dtttfdfdd }t|t|t |t j j rr|fnt |}|D]}||_||_q~dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rbrrcsSr8r>r>rr>r?resz4_patch_optimizer_state_dict..state_dict_callrcs|ddS)N)rr>rrr>r?rosz9_patch_optimizer_state_dict..load_state_dict_callN)rrr/r2r rXrr7rrurYrrrrr)rbrrrrrr>rr?_patch_optimizer_state_dict?s.    r )TT)pr\rr9r dataclassesrrr itertoolsrtypingrrrr r r r r rrrrrYZtorch.distributedrrZtorch.nnr_Z'torch.distributed._shard.sharded_tensorrZ#torch.distributed._state_dict_utilsrrrrrZtorch.distributed._tensorrZ;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrZtorch.distributed.fsdprrrryrrrr r!Z$torch.distributed.fsdp._common_utilsr"r#Ztorch.nn.modules.moduler$Ztorch.nn.parallelr%rvZtorch.utils._pytreer&__all__r{rrrrXr'rZrrr(r)r*r+r,r[r7rMrr@r-rN lru_cacher`rLrrrrrrrrrrrrrrrrrr.r/r0rr1r2r3r r r>r>r>r?sv 8     (      . >$    -    D  .&A  . @ I ]  )  1  Z$ %  , -  >6