U Mh@s4ddlZddlZddlmZmZmZmZddlmZddl Z ddl m Z m Z ddl mZmZmZdddd d gZeeed d d Zeeeeed ddZeedddZe jfeeeedddZeedddZd eeeeddd ZededeeedddZeedddZee jedddZdS)!N)AnyListOptionalSet) deprecated)_ShardingIterDataPipeSHARDING_PRIORITIES)DataPipe DataPipeGraph traverse_dpsapply_random_seedapply_shardingapply_shuffle_seedapply_shuffle_settingsget_all_graph_pipes)graphreturncCs t|tSN)_get_all_graph_pipes_helperset)rrQ/var/www/html/venv/lib/python3.8/site-packages/torch/utils/data/graph_settings.pyrs)rid_cachercCsLg}|D]:\}\}}||kr"q |||||t||q |Sr)itemsaddappendextendr)rrresultsZdp_iddatapipe sub_graphrrrrs  r)rrcCs,t|trdSt|dr(t|jr(dSdS)NTr F) isinstancerhasattrinspectismethodr rrrr_is_sharding_datapipe(s  r%)rnum_of_instances instance_idrcs(t|}dfdd ||S)z Apply dynamic sharding over the ``sharding_filter`` DataPipe that has a method ``apply_sharding``. RuntimeError will be raised when multiple ``sharding_filter`` are presented in the same branch. Ncs|D]\}}d}t|rt|dk r8td|d|t|j}t|jdkr`|n|jd|}|dkr|}||qdS)NzoSharding twice on a single pipeline is likely unintended and will cause data loss. Sharding already applied to z while trying to apply to )sharding_group)valuesr% RuntimeErrorr" signaturer len parameters)rZ prev_appliedZdprZappliedsig_helperr'r&r)rrr1;s zapply_sharding.._helper)N)r )rr&r'r)rrr0rr 0s cCs8t|drt|dsdSt|jr0t|js4dSdS)N set_shuffleset_seedFT)r!r"r#r2r3r$rrr_is_shuffle_datapipeRs r4)rshufflercCsb|dkr |St|}t|}dd|D}|sJ|rJtd|}|g}|D]}||qN|S)aE Traverse the graph of ``DataPipes`` to find and set shuffle attribute. Apply the method to each `DataPipe` that has APIs of ``set_shuffle`` and ``set_seed``. Args: datapipe: DataPipe that needs to set shuffle attribute shuffle: Shuffle option (default: ``None`` and no-op to the graph) NcSsg|]}t|r|qSr)r4).0piperrr jsz*apply_shuffle_settings..z`shuffle=True` was set, but the datapipe does not contain a `Shuffler`. Adding one at the end. Be aware that the default buffer size might not be sufficient for your task.)r rwarningswarnr5r2)rr5r all_pipesZ shufflersZshufflerrrrrZs  z`apply_shuffle_seed` is deprecated since 1.12 and will be removed in the future releases. Please use `apply_random_seed` instead.)category)rrngrcCs t||Sr)r )rr=rrrryscCst|drt|jrdSdS)Nr3TF)r!r"r#r3r$rrr_is_random_datapipesr>cCst|}t|}t}g}|D]2}t||kr0qt|r|||t|q|D].}ttj dtj dj |d }| |qV|S)a6 Traverse the graph of ``DataPipes`` to find random ``DataPipe`` with an API of ``set_seed``. Then set the random seed based on the provided RNG to those ``DataPipe``. Args: datapipe: DataPipe that needs to set randomness rng: Random number generator to generate random seeds r)Zdtype) generator)r rridr>rrinttorchemptyZint64Zrandom_itemr3)rr=rr;cacheZrandom_datapipesr7Z random_seedrrrr s     )N) r"r9typingrrrrtyping_extensionsrrBZ(torch.utils.data.datapipes.iter.shardingrrZtorch.utils.data.graphr r r __all__rrArboolr%DEFAULTr r4r FutureWarningrr> Generatorr rrrrs<     "