U Mh@s>dZddlZddlmZddlZddlmZddlm Z ddl m Z ddl m Z dd ZeejejejejejgejfZejejejejejejd d d Zefejejejeejd ddZejejejejejejd ddZejejejejejejd ddZejejejejejejd ddZedddZdS)zIThis module implements the user facing API for flex_attention in PyTorch.N)Callable)flex_attention)_set_compilation_env)-_temp_remove_pre_dispatch_torch_function_mode)_validate_sdpa_inputcGsdd}t||S)z*Compose a sequence of score_mod functions.csfdd}|S)Ncs|||||||||SN)scorebhmnfgrT/var/www/html/venv/lib/python3.8/site-packages/torch/nn/attention/_flex_attention.pyinnersz)_compose..compose2..innerr)rrrrrrcompose2sz_compose..compose2) functoolsreduce)fsrrrr_composesrr batchheadtoken_qtoken_kvreturncCs|Srrr rrrrrrr _identity sr)querykeyvalue score_modrc Cstjr>|||fD]}tj|dqt||||\}}|St||||dddkrdtdtj svt dt rtjj ZtHtjtddd ||||\}}|W5QRW5QRW5QRSQRXW5QRXW5QRXd S) a This function implements scaled dot product attention with an arbitrary attention score modification function. This function computes the scaled dot product attention between query, key, and value tensors with a user-defined attention score modification function. The attention score modification function will be applied after the attention scores have been calculated between the query and key tensors. The attention scores are calculated as follows: The ``score_mod`` function should have the following signature: .. code-block:: python def score_mod( score: torch.Tensor, batch: torch.Tensor, head: torch.Tensor, token_q: torch.Tensor, token_kv: torch.Tensor ) -> torch.Tensor: Where: - ``score``: A scalar tensor representing the attention score, with the same data type and device as the query, key, and value tensors. - ``batch``, ``head``, ``token_q``, ``token_kv``: Scalar tensors indicating the batch index, head index, query index, and key/value index, respectively. These should have the ``torch.int`` data type and be located on the same device as the score tensor. Args: query (Tensor): Query tensor; shape :math:`(B, H, L, E)`. key (Tensor): Key tensor; shape :math:`(B, H, S, E)`. value (Tensor): Value tensor; shape :math:`(B, H, S, Ev)`. score_mod (Callable): Function to modify attention scores. By default no score_mod is applied. Returns: output (Tensor): Attention output; shape :math:`(B, H, L, Ev)`. Shape legend: - :math:`N: \text{Batch size} ... : \text{Any number of other batch dimensions (optional)}` - :math:`S: \text{Source sequence length}` - :math:`L: \text{Target sequence length}` - :math:`E: \text{Embedding dimension of the query and key}` - :math:`Ev: \text{Embedding dimension of the value}` .. warning:: `torch.nn.attention.flex_attention` is a prototype feature in PyTorch. It doesn't support training currently. Please look forward to a more stable implementation in a future version of PyTorch. Read more about feature classification at: https://pytorch.org/blog/pytorch-feature-classification-changes/#prototype rz&NYI: S and L must be a multiple of 128z'flex_attention requires dynamo support.eagerT)backendZ fullgraphN)torchcompilerZis_dynamo_compilingZ_dynamoZ mark_staticflex_attention_hoprsize ValueErrorZis_dynamo_supported RuntimeErrorrutilsZdisable_cache_limitrcompile)r r!r"r#xout_rrr_flex_attention*s06   r4cCst||k|tdSNz-infr)wherefloatrrrr_causal{sr9cCs |||Srrrrrr _rel_biassr:cCst||k|||tdSr5r6rrrr _rel_causalsr; num_headscs,tjtjtjtjtjtjdfdd }|S)Nrcs(t|dd }||||S)Ng @)r)Zexp2)r rrrrscaler<rr _alibi_biassz)_generate_alibi_bias.._alibi_bias)r)Tensor)r=r@rr<r_generate_alibi_biass rB)__doc__rtypingrr)Z&torch._higher_order_ops.flex_attentionrr+Ztorch._higher_order_ops.utilsrZ"torch.fx.experimental.proxy_tensorrZtorch.nn.attention._utilsrrrAZ_score_mod_signaturerr4r9r:r;intrBrrrrs`        R