U Mh@sUdZddlZddlmZmZddlZgZeeed<ej e dddZ ej e ej dd d Z e eeed d d Zdej ej ej eej dddZdS)zCDefines utilities for interacting with scaled_dot_product_attentionN)ListOptional__all__)tensorsreturncGstdd|DS)z0Returns True if any of the tensors requires gradcss|] }|jVqdS)N)Z requires_grad).0tr K/var/www/html/venv/lib/python3.8/site-packages/torch/nn/attention/_utils.py sz'_input_requires_grad..)any)rr r r _input_requires_grad sr ) inpt_tensorog_sizercCs"|d|kr|dd|fS|S)z'Handles the unpad of the last dimension.N)size)rrr r r _postprocess_flash_outputsr) head_dim_sizescalercCs|dk r |Sdt|S)z For FlashAttention we pad the head dimension to be a multiple of 8 so we need to scale the output by the original head size and not the padded. Ng?)mathsqrt)rrr r r _calculate_scalesrF)querykeyvalue attn_maskcCs|j|jks|j|jkr:td|jd|jd|jd|j|jksR|j|jkrttd|jd|jd|jd|dks|dks|dkrtd |d |d |ddS) NzLExpected query, key, and value to have the same dtype, but got query.dtype: z , key.dtype: z, and value.dtype: z instead.zSExpected query, key, and value to have the same device type, but got query.device: z, key.device: z, and value.device: zUExpected query, key, and value to all be at least 2 dimensional, but got query.dim: z , key.dim: z and value.dim: )Zdtype ValueErrorZdevicedim)rrrrZ dropout_pZ is_causalrr r r _validate_sdpa_input!s $"r )NrFN)__doc__rtypingrrZtorchrstr__annotations__ZTensorboolr intrfloatrr r r r r s"