U T?h@s<ddlZddlmZmZddlZeeZGdddZdS)N)ListTuplec@seZdZdZededddZeddZedd Zee e e j d d d Z ede e j ed ddZ ede e e j d ddZdS)PastKeyValuesHelperzEHelper functions to process past key values for encoder-decoder modelF)presentcCsg}g}t|D]h}||r0d|d|gnd|d|g||rbd|d|gnd|d|gq||S) NZpresent_key_self_Zpresent_value_self_Zpast_key_self_Zpast_value_self_Zpresent_key_cross_Zpresent_value_cross_Zpast_key_cross_Zpast_value_cross_)rangeextend) num_layersrZpast_self_namesZpast_cross_namesir `/var/www/html/venv/lib/python3.8/site-packages/onnxruntime/transformers/models/t5/past_helper.pyget_past_namess z"PastKeyValuesHelper.get_past_namesc Cshg}g}t|D]N\}}t|dks6tdt||\}}}}|||g|||gq||fS)aSplit present state from grouped by layer to grouped by self/cross attention. Before: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), ... After: (past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ...), (past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ...) !Expected to have four items. Got  enumeratelenAssertionErrorr) present_key_values present_self present_crossZ_ipresent_layer_ipresent_key_selfpresent_value_selfpresent_key_crosspresent_value_crossr r r group_by_self_or_cross#sz*PastKeyValuesHelper.group_by_self_or_crosscs0tdksttfddtDS)aReorder past state from grouped by self/cross attention to grouped by layer. Before: past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ..., past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ... After: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), r c3sR|]J}d|d|ddd|dd|dgVqdS)Nr ).0r rpastr r ?s  z5PastKeyValuesHelper.group_by_layer..)rrtupler)r rr rr group_by_layer8sz"PastKeyValuesHelper.group_by_layer)past_key_valuescCsdd}t|d}tt|dD]>}d|}|||||d||||||dff7}q |S)aCategorize present_key_values from self and cross attention to layer by layer. Reorder past state from grouped by self/cross attention to grouped by layer. Before: past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ..., past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ... After: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), Args: present_key_values: From past_key_values of a model (group by self and cross attention) Returns: past_tuples: present key and values grouped by layer. r rr r)rr)r$Z past_tuplesZhalf_idxr idxr r r back_group_by_layerIs   z'PastKeyValuesHelper.back_group_by_layer)rconcatc Csxg}g}t|D]N\}}t|dks6tdt||\}}}} |||g||| gq|rl||S||fSdS)aCategorize present_key_values into self and cross attention. Split present state from grouped by layer to grouped by self/cross attention. Before: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), ... After: (past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ...), (past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ...) Args: present_key_values: From past_key_values of a model (group by layer) concat: If concat self attention with cross attention key/value to return Returns: present_self (Tuple[torch.Tensor]): present key and values from self attention present_cross (Tuple[torch.Tensor]): present key and values from cross attention r rNr) rr'rr_rrrrrr r r group_by_self_and_crossgs z+PastKeyValuesHelper.group_by_self_and_crossTcsg}|rt|dnt|}|s$dndt|D],}|fddd|d|fDq0t|D],}|fddd |d |fDqf|S) zProcess input names of model wrapper. Args: past_key_values: Consider `self` and `cross` past_key_values Returns: names (List[string]): input names r Zpast_Zpresent_csg|] }|qSr r rsprefixr r sz7PastKeyValuesHelper.get_input_names..Z key_self_Z value_self_csg|] }|qSr r r*r,r r r.sZ key_cross_Z value_cross_)rrr)r$encodernamesrr r r,r get_input_namess   * *z#PastKeyValuesHelper.get_input_namesN)F)F)T)__name__ __module__ __qualname____doc__ staticmethodboolr rr#rtorchZTensorr&r)r1r r r r rs  r) loggingtypingrrr8 getLoggerr2loggerrr r r r s