U Mh2@sdZddlmZmZddlmZddlmZddlZddl m Z m Z m Z ddl mZddlmZmZmZmZdd lmZd d d d gZeje eje eje Gdd d eZGdd d ejZeddd Zeddd ZdS)zCDefines bias subclasses that work with scaled_dot_product_attention)autoIntEnum)Optional)warnN)can_use_efficient_attentioncan_use_flash_attention SDPAParams)_raise_kernel_warnings)_calculate_scale_input_requires_grad_postprocess_flash_output_validate_sdpa_input)scaled_dot_product_attentioncausal_upper_leftcausal_lower_right CausalVariant CausalBiasc@seZdZdZeZeZdS)ra# Enum for causal variants used in attention mechanisms. Defines two types of causal biases: `UPPER_LEFT`: Represents upper-left triangular bias for standard causal attention. The equivalent pytorch code for constructing this bias is: .. code-block:: python torch.tril(torch.ones(size, dtype=torch.bool)) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]] `LOWER_RIGHT`: Represents lower-right triangular bias, the include values are aligned to the lower right corner of the matrix. The equivalent pytorch code for constructing this bias is: .. code-block:: python diagonal_offset = size[1] - size[0] torch.tril( torch.ones(size, dtype=torch.bool), diagonal=diagonal_offset, ) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]] Note that these variants are equivalent to each other when the sequence lengths of the query and key/value tensors are equal since the triangular matrix is square. .. warning:: This enum is a prototype and subject to change. N)__name__ __module__ __qualname____doc__r UPPER_LEFT LOWER_RIGHTrrI/var/www/html/venv/lib/python3.8/site-packages/torch/nn/attention/bias.pyrs0c @seZdZdZeeedddZejej dddZ ejej ddd Z de ejej dd d Z edej ej ej deee eej dddZedddZddZd S)ra  A bias representing causal attention patterns. For an overview of the bias structure, see the :class:`CausalVariant` enum. This class is used for defining causal (triangular) attention biases. For construing the bias, there exist two factory functions: :func:`causal_upper_left` and :func:`causal_lower_right`. Example: .. code-block:: python from torch.nn.attention.bias import causal_lower_right bsz, num_heads, seqlen_q, seqlen_kv, head_dim = 32, 8, 4, 12, 8 # Create a lower-right causal bias attn_bias = causal_lower_right(seqlen_q, seqlen_kv) q = torch.randn(bsz, num_heads, seqlen_q, head_dim, device="cuda", dtype=torch.float16) k = torch.randn(bsz, num_heads, seqlen_kv, head_dim, device="cuda", dtype=torch.float16) v = torch.randn(bsz, num_heads, seqlen_kv, head_dim, device="cuda", dtype=torch.float16) out = F.scaled_dot_product_attention(q, k, v, attn_bias) .. warning:: This class is a prototype and subject to change. )variant seq_len_q seq_len_kvcCs>t|tst||_||_||_||kr:|tjkr:tddS)a Initializes the CausalBias instance with a specified variant and sequence lengths. Args: variant (CausalVariant): The type of causal bias to use (either UPPER_LEFT or LOWER_RIGHT). seq_len_q (int): The sequence length of the query tensor. seq_len_kv (int): The sequence length of the key/value tensor. Raises a warning if the LOWER_RIGHT variant is used with seq_len_q > seq_len_kv, as it may produce NaNs. zTLower right causal bias will produce NaNs in the output when seq_len_q > seq_len_kv!N) isinstancerAssertionErrorrrrrr)selfrrrrrr__init__ns zCausalBias.__init__)devicereturncCsttj|j|j|tjdS)zUpper left causal biasr"Zdtype)torchtrilonesrrboolr r"rrr _upper_leftszCausalBias._upper_leftcCs.|j|j}tjtj|j|j|tjd|dS)zLower right causal biasr$)Zdiagonal)rrr%r&r'r()r r"Zdiagonal_offsetrrr _lower_rights zCausalBias._lower_rightNcCsB|dkrtd}|jtjkr(||S|jtjkr>||SdS)a Materializes the causal bias into a tensor form. Depending on the variant, this method generates either an upper-left or lower-right triangular matrix to represent the causal bias. Args: device (Optional[torch.device]): The device on which to create the tensor. Defaults to CPU. Returns: torch.Tensor: The materialized bias tensor. Ncpu)r%r"rrrr*rr+r)rrr _materializes     zCausalBias._materializeF)querykeyvalue attn_mask dropout_p is_causalscaler#c Cs|r td|j|jks$|jtjkr:t|||d|d|dS|jtjkrt|||d|||t |||d||}t |r4| dddk}| d} t | |} |r t jj|dd| ddf}t jj|dd| ddf}t jj|dd| ddf}t jjj||||dd| d d} t| | St|rd} t|||rTd} t jjj|d d |d d |d d ddddd|t|j| |dd dd d St|t|||||j|d|dSntd |jdS)a Handles the logic for computing attention with the specified causal bias. Args: query (Tensor): Query tensor; shape :math:`(N, ..., L, E)`. key (Tensor): Key tensor; shape :math:`(N, ..., S, E)`. value (Tensor): Value tensor; shape :math:`(N, ..., S, Ev)`. attn_mask (CausalBias): The type of causal attention to apply. A boolean mask where a value of True indicates that the element *should* take part in attention. A float mask of the same type as query, key, value that is added to the attention score. dropout_p (float): Dropout probability; if greater than 0.0, dropout is applied is_causal (bool): If true, assumes upper left causal attention masking and errors if both attn_mask and is_causal are set. scale (optional float): Scaling factor applied prior to softmax. If None, the default value is set to :math:`\frac{1}{\sqrt{E}}`. Returns: output (Tensor): Attention output; shape :math:`(N, ..., L, Ev)`. Raises: ValueError: If the causal bias variant is not a CausalVariant type. z.CausalBias should not be used with causal=TrueNT)r2r3r4r5rF)r4Zreturn_debug_maskr5) ZbiasZ cu_seqlens_qZ cu_seqlens_kZ max_seqlen_qZ max_seqlen_kr3Zcustom_mask_typecompute_log_sumexpr5Zseqlen_kzrNotImplementedErrorrB)clsfunctypesargskwargsrrr__torch_function__szCausalBias.__torch_function__cCs |S)N)r-__repr__)r rrrrJszCausalBias.__repr__)N)r.FN)rN)rrrrrr@r!r%r"Tensorr*r+rr- staticmethodfloatr(rB classmethodrIrJrrrrrSs, i )r#cGs*t|dkstd|\}}ttj||S)a& Creates an upper-left triangular causal bias. This function generates a upper-left triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the upper left corner of the matrix. This equivalent to the `is_causal=True` argument in `scaled_dot_product_attention`. The equivalent pytorch code for constructing this bias is: .. code-block:: python torch.tril(torch.ones(size, dtype=torch.bool)) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]] Args: size: The size of the bias matrix. Returns: CausalBias: The UPPER_LEFT triangular causal bias variant. r9z*causal_upper_left only supports 2D tensors)lenrrrrr<rrrrrrscGs*t|dkstd|\}}ttj||S)a: Creates a lower-right triangular causal bias. This function generates a lower-right triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the lower right corner of the matrix. The equivalent pytorch code for constructing this bias is: .. code-block:: python diagonal_offset = size[1] - size[0] torch.tril( torch.ones(size, dtype=torch.bool), diagonal=diagonal_offset, ) For instance, with `shape=(3,4)`, the materialized bias tensor will be: .. code-block:: text [[1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]] Args: size: The size of the bias matrix. Returns: CausalBias: The LOWER_RIGHT triangular causal bias variant. r9z+causal_lower_right only supports 2D tensors)rOrrrrrPrrrr@s)renumrrtypingrwarningsrr%Ztorch.backends.cudarrrZtorch.nn.attentionr Ztorch.nn.attention._utilsr r r r Ztorch.nn.functionalr__all__Z_dynamoZallow_in_graphrrKrrrrrrrs"        5M!