U yâhXã@snddlZddlZddlmZddlmmZddlmZddlmZm Z ddl Z dgZ Gdd„dej ƒZ dS)éN)Únn)ÚTensor)ÚOptionalÚTupleÚMultiheadAttentionc sâeZdZejZdgZdeeee e e e ee ee ddœ ‡fdd„ Z d d „Z e d d „ƒZejjd d„ƒZe dd„ƒZdeeee ee e ee e eee efdœ dd„Zdeeee ee e ee e eee efdœ dd„Z‡ZS)rÚ batch_firstçTFN) Ú embed_dimÚ num_headsÚdropoutÚbiasÚ add_bias_kvÚ add_zero_attnÚkdimÚvdimrÚreturnc sü| | dœ} tƒj||||||||| f | Žtj|j|jfd|i| —Ž|_tj|j|jfd|i| —Ž|_tj|j|jfd|i| —Ž|_ tj|j|jfd|i| —Ž|_ t j jj  ¡|_t j j ¡|_t j j ¡|_t j j ¡|_t j j ¡|_t j j ¡|_dS)N)ÚdeviceÚdtyper )ÚsuperÚ__init__rZLinearr Úlinear_QrÚlinear_KrÚlinear_VÚout_projÚtorchÚaoZ quantizedZFloatFunctionalÚq_scaling_productÚ quantizationZ QuantStubÚquant_attn_outputÚquant_attn_output_weightsZ DeQuantStubÚ dequant_qÚ dequant_kÚ dequant_v) Úselfr r r r r rrrrrrZfactory_kwargs©Ú __class__©ú\/var/www/html/venv/lib/python3.8/site-packages/torch/ao/nn/quantizable/modules/activation.pyr@s*  þýzMultiheadAttention.__init__cCsdS)NZQuantizableMultiheadAttentionr&)r#r&r&r'Ú _get_name[szMultiheadAttention._get_namec Csˆt|ƒ|jkst‚t|dƒs$tdƒ‚||j|j|j|jdk |jdk |j |j |j |j ƒ }|j|_|j |_ |j|_|jj|j_|jj|j_|jr¼|j}d}||j}|j||…dd…f}|dk rØtj |||…|j¡}tj ||j¡|j_||j_|j}|}||j}|j||…dd…f}|dk r@tj |||…|j¡}tj ||j¡|j_||j_|j}|}|j|d…dd…f}|dk ržtj ||d…|j¡}tj ||j¡|j_||j_n®t |j¡|j_t |j¡|j_t |j¡|j_|jdkrd|j_d|j_d|j_nXt |jd|j…¡|j_t |j|j|jd…¡|j_t |j|jdd…¡|j_| ¡tjj j!|dd}|S)NÚqconfigz$The float module must have 'qconfig'réT)Zinplace)"ÚtypeÚ _FLOAT_MODULEÚAssertionErrorÚhasattrr r r Ú in_proj_biasÚbias_krrrrÚbias_vr)rÚweightr Ú_qkv_same_embed_dimÚin_proj_weightrrÚ ParameterZ requires_gradrrrÚ q_proj_weightÚ k_proj_weightÚ v_proj_weightÚevalrrÚprepare)ÚclsÚotherZobservedr Ú_startÚ_endr2r&r&r'Ú from_float^stü   ÿ  ÿ ÿ    zMultiheadAttention.from_floatc Cst| |j|j|j|j ¡ddk |jdk |j|j|j |j ¡ }|j |j ksLt ‚|jdk rht  |j ¡¡|_|jdk r„t  |j ¡¡|_|j ¡\}}t  | ¡¡|j_|dk rºt  |¡|j_|j ¡\}}| ¡}|j ¡\}}| ¡}|j ¡\}} | ¡}|j rìd} | |j} ||j| | …dd…f<|jdk rTt|dkƒsFt ‚||j| | …<| } | |j} ||j| | …dd…f<|jdk r¤t|dkƒs–t ‚||j| | …<| } ||j| d…dd…f<|jdk rpt| dkƒsÜt ‚| |j| d…<n„t  |¡|_t  |¡|_t  |¡|_|jdkr6d|j_d|j_d|j_n:||jd|j…<||j|j|jd…<| |j|jdd…<|S)zæUtility to convert the quantized MHA back to float. The motivation for this is that it is not trivial to conver the weights from the format that is used in the quantized version back to the float. éNrr*)r,r r r rZ _weight_biasr0rrrrr3r-rr5Ú dequantizer1rr2r rrr4r/Úallr6r7r8) r#ÚfpÚwÚbZwQZbQZwKZbKZwVZbVr=r>r&r&r'rAŸshý            zMultiheadAttention.dequantizecCs tdƒ‚dS)NzdIt looks like you are trying to prepare an MHA module. Please, see the examples on quantizable MHAs.)ÚNotImplementedError)r;r<r&r&r'Ú from_observedäsz MultiheadAttention.from_observed) ÚqueryÚkeyÚvalueÚkey_padding_maskÚ need_weightsÚ attn_maskÚaverage_attn_weightsÚ is_causalrc Cs| ||||||||¡S)aÆ Note:: Please, refer to :func:`~torch.nn.MultiheadAttention.forward` for more information Args: query, key, value: map a query and a set of key-value pairs to an output. See "Attention Is All You Need" for more details. key_padding_mask: if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. need_weights: output attn_output_weights. attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch. Shape: - Inputs: - query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``. - key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``. - value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``. - key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length. If a BoolTensor is provided, the positions with the value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged. - attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length. 3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a BoolTensor is provided, positions with ``True`` is not allowed to attend while ``False`` values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. - is_causal: If specified, applies a causal mask as attention mask. Mutually exclusive with providing attn_mask. Default: ``False``. - average_attn_weights: If true, indicates that the returned ``attn_weights`` should be averaged across heads. Otherwise, ``attn_weights`` are provided separately per head. Note that this flag only has an effect when ``need_weights=True.``. Default: True (i.e. average weights across heads) - Outputs: - attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``. - attn_output_weights: If ``average_attn_weights=True``, returns attention weights averaged across heads of shape :math:`(N, L, S)`, where N is the batch size, L is the target sequence length, S is the source sequence length. If ``average_attn_weights=False``, returns attention weights per head of shape :math:`(N, num_heads, L, S)`. )Ú _forward_impl) r#rHrIrJrKrLrMrNrOr&r&r'Úforwardís 7 þzMultiheadAttention.forwardc CsÔd} d} |dk r|rtdƒ‚|r(tdƒ‚|jrHdd„|||fDƒ\}}}| ¡\} } } |j| ksdt‚| d¡| d¡krŒ| d¡| d¡kst‚|j|j}||j|jks´tdƒ‚t|ƒd}| |¡}| |¡}| |¡}|j   ||¡}|dk râ|j t j krtjd d d | t j¡}| ¡sF|j t jksFtd |j ›ƒ‚| ¡d krŒ| d¡}t| ¡ƒd| d¡| d¡gkrâtdƒ‚nV| ¡d krÎt| ¡ƒ| |j| d¡| d¡gkrâtdƒ‚ntd| ¡›dƒ‚|dk r|j t j krtjdd d | t j¡}|jdk rì|jdk rì| dkrÆ| dkrÆ|j}|dk sTt‚|j}|dk sht‚t  || d| d¡g¡}t  || d| d¡g¡}|dk r®t |d¡}|dk rêt |d¡}n$| dksØtdƒ‚| dks tdƒ‚n |jdksüt‚|jdks t‚| ¡ | | |j|¡ dd¡}|dk rV| ¡ d| |j|¡ dd¡}|dk r€| ¡ d| |j|¡ dd¡}| dk r¼|  d¡| |jks¤t‚|  d ¡|ks¸t‚| }| dk rø|  d¡| |jksàt‚|  d ¡|ksôt‚| }| d¡}|dk r4| d¡| ks t‚| d¡|ks4t‚|j r |d7}t  !| d¡df| ¡d d…¡}|j"rŠt  #|| $¡| %¡|j ¡}t j||gdd}t  !| d¡df| ¡d d…¡}|j"rât  #|| $¡| %¡|j ¡}t j||gdd}|dk r t |d¡}|dk r t |d¡}| &|¡}| '|¡}| (|¡}t  )|| dd ¡¡}t| ¡ƒ| |j| |gkstt‚|dk r¦|j t jkrž| *|tdƒ¡n||7}|dk rò| | |j| |¡}| +| d¡ d ¡tdƒ¡}| | |j| |¡}tj,|dd}tj-||j-|j.d}t  )||¡}t| ¡ƒ| |j| |gksBt‚|jr\| | | |j¡}n| dd¡ ¡ | | |j¡}| /|¡}| 0|¡}| 1|¡}|rÈ| | |j| |¡}|rÀ|j2dd}||fS|dfSdS)Nz#Only allow causal mask or attn_maskz*causal mask not supported by AO MHA modulecss|]}| dd¡VqdS)rr@N)Ú transpose)Ú.0Úxr&r&r'Ú @sz3MultiheadAttention._forward_impl..rr@z(embed_dim must be divisible by num_headsgà¿z^Byte tensor for `attn_mask` in `nn.MultiheadAttention` is deprecated. Use bool tensor instead.é)Ú stacklevelz;Only float and bool types are supported for attn_mask, not r*z,The size of the 2D attn_mask is not correct.z,The size of the 3D attn_mask is not correct.zattn_mask's dimension z is not supportedzeByte tensor for `key_padding_mask` in `nn.MultiheadAttention` is deprecated. Use bool tensor instead.)rr@z#bias cannot be added to static key.z%bias cannot be added to static value.éÿÿÿÿ)Údimz-inf)ÚpÚtraining)3r-rÚsizer r ÚfloatrrrrZ mul_scalarrrZuint8ÚwarningsÚwarnÚtoÚboolZis_floating_pointrYZ unsqueezeÚlistÚ RuntimeErrorr0r1ÚcatÚrepeatÚnnFÚpadÚ contiguousÚviewrRrZzerosZ is_quantizedZquantize_per_tensorZq_scaleZ q_zero_pointr r!r"ZbmmZ masked_fill_Z masked_fillZsoftmaxr r[rrrZmean)r#rHrIrJrKrLrMrNrOZstatic_kZstatic_vZtgt_lenZbszZembed_dim_to_checkZhead_dimZscalingÚqÚkÚvr0r1Zsrc_lenZk_zerosZv_zerosZattn_output_weightsZ attn_outputr&r&r'rP(sø ,      ý  ÿ $ * ý             $$       "  þÿ "    z MultiheadAttention._forward_impl) rTFFNNFNN)NTNTF)NTNTF)Ú__name__Ú __module__Ú __qualname__rrr,Z __constants__Úintr]rarrr(Ú classmethodr?rZjitZunusedrArGrrrQrPÚ __classcell__r&r&r$r'rsx-üü @ D øø ?øø) rZ torch.jitrZtorch.nn.functionalZ functionalrfrÚtypingrrr^Ú__all__rr&r&r&r'Ús  ÿ