U h(l@s,ddlmZddlmZmZmZmZmZddlZddl m m Z ddlm Z m Z ddlmZddlmZdd lmZmZmZdd lmZdd lmZmZdd lmZmZd ddddddgZeeeeeeeeeeefdddZ ej!"dej ej eee dddZ#ej!"deeeefeeeefeeeefdddZ$ej!"de eeeefeeeefeeeefe dddZ%ej!"dd:e e e e eeeeee&e&ee ee e'e d" d#d$Z(ej!"d$Gd%d&d&e j)Z*Gd'd(d(e j)Z+Gd)d d e j)Z,eeeeeeeeee&eee'ee,d* d+d,Z-ed-d.d/Z.Gd0ddeZ/Gd1ddeZ0Gd2ddeZ1eed3e/j2fd4dd!d5ee/e'ee,d6d7dZ3eed3e0j2fd4dd!d5ee0e'ee,d6d8dZ4eed3e1j2fd4dd!d5ee1e'ee,d6d9dZ5dS);)partial)AnyCallableListOptionalTupleN)nnTensor)VideoClassification)_log_api_usage_once)register_modelWeights WeightsEnum)_KINETICS400_CATEGORIES)_ovewrite_named_paramhandle_legacy_interface) PatchMergingSwinTransformerBlockSwinTransformer3dSwin3D_T_WeightsSwin3D_S_WeightsSwin3D_B_Weightsswin3d_tswin3d_sswin3d_b) shift_sizesize_dhw window_sizereturncCs:tdD](}||||kr||||<d||<q||fS)Nr rrange)rrrir$[/var/www/html/venv/lib/python3.8/site-packages/torchvision/models/video/swin_transformer.py_get_window_and_shift_size s    r&)relative_position_bias_tablerelative_position_indexrr cCs^|d|d|d}||d|d|f}|||d}|dddd}|S)Nrr )flattenviewpermute contiguous unsqueeze)r'r(rZ window_volrelative_position_biasr$r$r%_get_relative_position_bias/sr1)r patch_sizer cs.fddtdD}|d|d|dfS)Ncs,g|]$}||||qSr$r$.0r#r2rr$r% @sz(_compute_pad_size_3d..r rr)r r!)rr2pad_sizer$r5r%_compute_pad_size_3d?sr8)xrrrr c sj|j|}|dd|dd|dd}fddtdD}d}|dD]Z}|dD]L} |dD]>} |||d|d| d| d| d| df<|d7}qvqjq^||ddd|ddd|ddd}|dddddd|ddd}|d|d}||dktd |dktd }|S) Nrr)r cs:g|]2}d| f| | f| dffqS)rNr$r3rrr$r%r6Ps   z._compute_attention_mask_3d..r gY)Z new_zerosr"r,r-reshaper/Z masked_fillfloat) r9rrr attn_mask num_windowsZslicescountdhwr$r:r%_compute_attention_mask_3dGs4 0    2$rFr=T) input qkv_weight proj_weightr0r num_headsrattention_dropoutdropoutqkv_bias proj_biastrainingr c  Cs|j\} } }}}t| ||f|d|d|df}t|ddd|dd|dd|df}|j\}}}}}|||f}t|dkrtj||d |d |d fdd}|d|d|d|d|d|d}|| |d|d|d|d|d|d|d|d|d|}|ddddddd d  | ||d|d|d|}t ||| }| | d| dd|||ddddd}|d|d|d}}}|||d }| | d d }||}t|dkrt||d|d|df|d|d|df|d|d|df}|| d|||| d| d}||dd}|d || d| d}tj|d d}tj||| d}| | dd | d| d|}t ||| }tj||| d}|| |d|d|d|d|d|d|d|d|d|}|ddddddd d  | ||||}t|dkrtj||d|d|dfdd}|ddd| d|d|ddf}|S)a Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: input (Tensor[B, T, H, W, C]): The input tensor, 5-dimensions. qkv_weight (Tensor[in_dim, out_dim]): The weight tensor of query, key, value. proj_weight (Tensor[out_dim, out_dim]): The weight tensor of projection. relative_position_bias (Tensor): The learned relative position bias added to attention. window_size (List[int]): 3-dimensions window size, T, H, W . num_heads (int): Number of attention heads. shift_size (List[int]): Shift size for shifted window attention (T, H, W). attention_dropout (float): Dropout ratio of attention weight. Default: 0.0. dropout (float): Dropout ratio of output. Default: 0.0. qkv_bias (Tensor[out_dim], optional): The bias tensor of query, key, value. Default: None. proj_bias (Tensor[out_dim], optional): The bias tensor of projection. Default: None. training (bool, optional): Training flag used by the dropout parameters. Default: True. Returns: Tensor[B, T, H, W, C]: The output tensor after shifted window attention. rr)r )r)r r )ZshiftsZdimsr r<r;gr*)dim)prON)shaper8FpadsumtorchZrollr,r-r>ZlinearsizematmulZ transposerFr/ZsoftmaxrLr.)rGrHrIr0rrJrrKrLrMrNrObtrDrEcr7r9_tphpZwpZ padded_sizerAqkvqkvZattnr@r$r$r%shifted_window_attention_3dssx!"(  (. 0(( &"*rfc seZdZdZdeeeeeeeeeedd fdd Zddd d Z ddd d Z eee j d ddZ e e dddZZS)ShiftedWindowAttention3dz2 See :func:`shifted_window_attention_3d`. Tr=N) rSrrrJrMrNrKrLr c stt|dks"t|dkr*td||_||_||_||_||_t j ||d|d|_ t j |||d|_ | |dS)Nr z.window_size and shift_size must be of length 2)bias)super__init__len ValueErrorrrrJrKrLrLinearrbproj#define_relative_position_bias_tabledefine_relative_position_index) selfrSrrrJrMrNrKrL __class__r$r%rjs z!ShiftedWindowAttention3d.__init__)r cCs^ttd|jddd|jddd|jdd|j|_tjj|jdddS)Nr rr){Gz?std) r ParameterrYZzerosrrJr'init trunc_normal_rqr$r$r%ros4z.r rr)r Zij)Zindexingr*r() r"rYstackZmeshgridr+r-r.rrXZregister_buffer)rqZ coords_dhwZcoordsZcoords_flattenZrelative_coordsr(r$rzr%rps ,(((>, z7ShiftedWindowAttention3d.define_relative_position_index)rr cCst|j|j|S)N)r1r'r()rqrr$r$r%get_relative_position_bias#sz3ShiftedWindowAttention3d.get_relative_position_biasr9r c Cs|j\}}}}}|||g}|j|j}}t|||\}}||} t||jj|j j| ||j ||j |j |jj |j j |jd S)N)rrKrLrMrNrO)rUrcopyrr&r|rfrbweightrnrJrKrLrhrO) rqr9r_r]rDrErrrr0r$r$r%forward&s&  z ShiftedWindowAttention3d.forward)TTr=r=)__name__ __module__ __qualname____doc__intrboolr?rjrorprYr r|r __classcell__r$r$rrr%rgs(  rgcsReZdZdZd eeeeeedej fddfdd Z e e d d d Z Z S) PatchEmbed3da;Video to Patch Embedding. Args: patch_size (List[int]): Patch token size. in_channels (int): Number of input channels. Default: 3 embed_dim (int): Number of linear projection output channels. Default: 96. norm_layer (nn.Module, optional): Normalization layer. Default: None r `N.)r2 in_channels embed_dim norm_layerr csdtt||d|d|df|_tj|||j|jd|_|dk rV|||_n t|_dS)Nrr)r )Z kernel_sizeZstride) rirjr tuple_patch_sizerZConv3drnnormZIdentity)rqr2rrrrrr$r%rjKs  zPatchEmbed3d.__init__r}c Cs||\}}}}}t|||f|j}t|d|dd|dd|df}||}|ddddd}|jdk rx||}|S)zForward function.rr r)r r;N)rZr8rrVrWrnr-r)rqr9r_r]rDrEr7r$r$r%ras$   zPatchEmbed3d.forward)r rN)rrrrrrrrrModulerjr rrr$r$rrr%rAs rcseZdZdZdddddddedf eeeeeeeeeeeeeeee de j fee de j fe de j fee de j fddfd d Z e e d d d ZZS)raY Implements 3D Swin Transformer from the `"Video Swin Transformer" `_ paper. Args: patch_size (List[int]): Patch size. embed_dim (int): Patch embedding dimension. depths (List(int)): Depth of each Swin Transformer layer. num_heads (List(int)): Number of attention heads in different layers. window_size (List[int]): Window size. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob (float): Stochastic depth rate. Default: 0.1. num_classes (int): Number of classes for classification head. Default: 400. norm_layer (nn.Module, optional): Normalization layer. Default: None. block (nn.Module, optional): SwinTransformer Block. Default: None. downsample_layer (nn.Module): Downsample layer (patch merging). Default: PatchMerging. patch_embed (nn.Module, optional): Patch Embedding layer. Default: None. g@r=皙?iN.)r2rdepthsrJr mlp_ratiorLrKstochastic_depth_prob num_classesrblockdownsample_layer patch_embedr cstt|| |_| dkr,tttd} | dkrBttjdd} |dkrNt }|||| d|_ tj |d|_ g}t |}d}tt|D]}g}|d|}t||D]T| t||d}|| ||||fd d |D||||| td |d7}q|tj||t|dkr|| || qtj||_|dt|d|_| |j|_td|_t|j| |_|D]@}t|tjrtjj|jd d |jdk rtj |jqdS)N) attn_layergh㈵>)eps)r2rr)rTrr r)cs$g|]}ddkrdn|dqS)r rr$)r4rEZi_layerr$r%r6sz.SwinTransformer3d.__init__..)rrrrLrKrrrrtru)!rirjr rrrrgrZ LayerNormrrZDropoutpos_droprXr"rkr?appendZ SequentialfeaturesZ num_featuresrZAdaptiveAvgPool3davgpoolrmheadmodules isinstancerxryrrhZzeros_)rqr2rrrJrrrLrKrrrrrrZlayersZtotal_stage_blocksZstage_block_idZi_stageZstagerSZsd_probmrrrr%rjs^         zSwinTransformer3d.__init__r}cCs^||}||}||}||}|ddddd}||}t|d}||}|S)Nrr;r)r r ) rrrrr-rrYr+r)rqr9r$r$r%rs       zSwinTransformer3d.forward)rrrrrrrr?rrrrrjr rrr$r$rrr%rns8 L) r2rrrJrrweightsprogresskwargsr c KsZ|dk rt|dt|jdtf||||||d|} |dk rV| |j|dd| S)Nr categories)r2rrrJrrT)rZ check_hash)rrkmetarZload_state_dictZget_state_dict) r2rrrJrrrrrmodelr$r$r%_swin_transformer3ds  r)r)r)r))rZmin_sizeZmin_temporal_sizec @sHeZdZedeedddddeddd d d d d iddddZeZdS)rz9https://download.pytorch.org/models/swin3d_t-7615ae03.pthrg ףp= ?gv/?gCl?gZd;O?gy&1?g?Z crop_sizeZ resize_sizeZmeanrvFhttps://github.com/SwinTransformer/Video-Swin-Transformer#kinetics-400The weights were ported from the paper. The accuracies are estimated on video-level with parameters `frame_rate=15`, `clips_per_video=12`, and `clip_len=32`iv Kinetics-400g(\mS@gK7aW@zacc@1zacc@5g7A`E@gnb^@ZrecipeZ_docsZ num_paramsZ_metricsZ_ops _file_sizeurlZ transformsrN rrrrrr _COMMON_METAKINETICS400_V1DEFAULTr$r$r$r%rs.c @sHeZdZedeedddddeddd d d d d iddddZeZdS)rz9https://download.pytorch.org/models/swin3d_s-da41c237.pthrrrrrrrif$rgMbXS@g'1W@rgҵT@gK7Ik@rrNrr$r$r$r%rs.c @seZdZedeedddddeddd d d d d iddddZedeedddddeddd d ddd iddddZeZ dS)rzsZZ pretrained)r)rr)rrrr c KsFt|}tfdddgdddddgddddgdd d gd ||d |S) a Constructs a swin_tiny architecture from `Video Swin Transformer `_. Args: weights (:class:`~torchvision.models.video.Swin3D_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.video.Swin3D_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.video.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.video.Swin3D_T_Weights :members: r r;rrPr rQrr2rrrJrrrr)rverifyrrrrr$r$r%rxs    c KsFt|}tfdddgdddddgddddgd d d gd ||d |S) a Constructs a swin_small architecture from `Video Swin Transformer `_. Args: weights (:class:`~torchvision.models.video.Swin3D_S_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.video.Swin3D_S_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.video.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.video.Swin3D_S_Weights :members: r r;rr rPrrrrQrr)rrrrr$r$r%rs    c KsFt|}tfdddgdddddgddddgdddgd ||d |S) a Constructs a swin_base architecture from `Video Swin Transformer `_. Args: weights (:class:`~torchvision.models.video.Swin3D_B_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.video.Swin3D_B_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.video.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.video.Swin3D_B_Weights :members: r r;rr rQrr)rrrrr$r$r%rs    )r=r=NNT)6 functoolsrtypingrrrrrrYZtorch.nn.functionalrZ functionalrVr Ztransforms._presetsr utilsr Z_apirrr_metarZ_utilsrrZswin_transformerrr__all__rr&Zfxwrapr1r8rFr?rrfrrgrrrrrrrrrrrr$r$r$r%s       0     )  p [-n :$$$$