U Mâh‘2ã@sPddlmZddlZddlmmZddlmZGdd„deƒZGdd„dejƒZ e dœd d „Z Gd d „d ej ƒZ Gd d„dej ƒZGdd„dej ƒZGdd„dej ƒZGdd„dej ƒZGdd„dej ƒZGdd„dej ƒZGdd„dej ƒZGdd„dej ƒZGdd„dej ƒZGdd „d ej ƒZGd!d"„d"ej ƒZGd#d$„d$ej ƒZdS)%é)ÚBaseSparsifierN)Únncs$eZdZ‡fdd„Zdd„Z‡ZS)ÚImplementedSparsifierc stƒj|ddS)N)Údefaults)ÚsuperÚ__init__)ÚselfÚkwargs©Ú __class__©úX/var/www/html/venv/lib/python3.8/site-packages/torch/testing/_internal/common_pruning.pyr szImplementedSparsifier.__init__cKs4d|jjdjd<|jd}| dd¡d|d<dS)Nrzlinear1.weightZ step_counté)ZparametrizationsÚweightÚmaskÚstateÚget)rÚmoduler Z linear_stater r r Ú update_masks z!ImplementedSparsifier.update_mask)Ú__name__Ú __module__Ú __qualname__rrÚ __classcell__r r r r r s rc@seZdZdZedd„ƒZdS)ÚMockSparseLinearzË This class is a MockSparseLinear class to check convert functionality. It is the same as a normal Linear layer, except with a different type, as well as an additional from_dense method. cCs||j|jƒ}|S)z )Z in_featuresZ out_features)ÚclsÚmodÚlinearr r r Ú from_densesÿzMockSparseLinear.from_denseN)rrrÚ__doc__Ú classmethodrr r r r rsr)ÚreturncCsBd}|D]4}|t|ƒkr6t |||¡s<|d7}q qq dSqdS)zW Checks to see if all rows in subset tensor are present in the superset tensor rrFT)ÚlenÚtorchÚequal)Z subset_tensorZsuperset_tensorÚiÚrowr r r Úrows_are_subset#s  r&cs(eZdZdZ‡fdd„Zdd„Z‡ZS)Ú SimpleLinearzžModel with only Linear layers without biases, some wrapped in a Sequential, some following the Sequential. Used to test basic pruned Linear-Linear fusion.c sftƒ ¡t tjddddtjddddtjdddd¡|_tjdddd|_tjdddd|_dS)NééF©Úbiasééé )rrrÚ SequentialÚLinearÚseqÚlinear1Úlinear2©rr r r r7s ýzSimpleLinear.__init__cCs"| |¡}| |¡}| |¡}|S©N)r1r2r3©rÚxr r r ÚforwardAs   zSimpleLinear.forward©rrrrrr8rr r r r r'3s r'cs(eZdZdZ‡fdd„Zdd„Z‡ZS)Ú LinearBiaszŠModel with only Linear layers, alternating layers with biases, wrapped in a Sequential. Used to test pruned Linear-Bias-Linear fusion.c s^tƒ ¡t tjddddtjddddtjddddtjddddtjdddd¡|_dS) Nr(r)Tr*r,Fér.)rrrr/r0r1r4r r r rLs ûzLinearBias.__init__cCs| |¡}|Sr5)r1r6r r r r8Vs zLinearBias.forwardr9r r r r r:Hs r:cs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚLinearActivationzùModel with only Linear layers, some with bias, some in a Sequential and some following. Activation functions modules in between each Linear in the Sequential, and each outside layer. Used to test pruned Linear(Bias)-Activation-Linear fusion.c s†tƒ ¡t tjddddt ¡tjddddt ¡tjdddd¡|_tjdddd|_t ¡|_ tjdd dd|_ t ¡|_ dS) Nr(r)Tr*r,Fr-r;r.) rrrr/r0ÚReLUÚTanhr1r2Úact1r3Úact2r4r r r r`s û zLinearActivation.__init__cCs6| |¡}| |¡}| |¡}| |¡}| |¡}|Sr5)r1r2r?r3r@r6r r r r8ns      zLinearActivation.forwardr9r r r r r<[s r<cs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚLinearActivationFunctionala,Model with only Linear layers, some with bias, some in a Sequential and some following. Activation functions modules in between each Linear in the Sequential, and functional activationals are called in between each outside layer. Used to test pruned Linear(Bias)-Activation-Linear fusion.c sŽtƒ ¡t tjddddt ¡tjddddt ¡tjdddd¡|_tjdddd|_tjdd dd|_tjd d dd|_ t ¡|_ dS) Nr(r)Tr*r,Fr-r;ér.) rrrr/r0r=r1r2r3Úlinear3r?r4r r r r}s ûz#LinearActivationFunctional.__init__cCsJ| |¡}| |¡}t |¡}| |¡}t |¡}| |¡}t |¡}|Sr5)r1r2ÚFÚrelur3rCr6r r r r8‹s       z"LinearActivationFunctional.forwardr9r r r r rAws rAcs(eZdZdZ‡fdd„Zdd„Z‡ZS)Ú SimpleConv2dz‡Model with only Conv2d layers, all without bias, some in a Sequential and some following. Used to test pruned Conv2d-Conv2d fusion.c shtƒ ¡t tjddddddtjdddddd¡|_tjdddddd|_tjdddddd|_dS) Nré r;Fr*é@é0é4©rrrr/ÚConv2dr1Úconv2d1Úconv2d2r4r r r ršs þzSimpleConv2d.__init__cCs"| |¡}| |¡}| |¡}|Sr5©r1rMrNr6r r r r8£s   zSimpleConv2d.forwardr9r r r r rF–s rFcs(eZdZdZ‡fdd„Zdd„Z‡ZS)Ú Conv2dBiaszˆModel with only Conv2d layers, some with bias, some in a Sequential and some outside. Used to test pruned Conv2d-Bias-Conv2d fusion.c sztƒ ¡t tjddddddtjddddddtjdddddd¡|_tjdddddd|_tjdd dddd|_dS© NrrGr;Tr*rHFrIrJrKr4r r r r®s ýzConv2dBias.__init__cCs"| |¡}| |¡}| |¡}|Sr5rOr6r r r r8¸s   zConv2dBias.forwardr9r r r r rPªs rPcs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚConv2dActivationaModel with only Conv2d layers, some with bias, some in a Sequential and some following. Activation function modules in between each Sequential layer, functional activations called in-between each outside layer. Used to test pruned Conv2d-Bias-Activation-Conv2d fusion.c sŒtƒ ¡t tjddddddt ¡tjddddddt ¡tjddddddt ¡¡|_tjdddddd|_tjdd dddd|_ dSrQ) rrrr/rLr=r>r1rMrNr4r r r rÅs úzConv2dActivation.__init__cCs6| |¡}| |¡}t |¡}| |¡}t |¡}|Sr5)r1rMrDrErNZhardtanhr6r r r r8Òs      zConv2dActivation.forwardr9r r r r rR¿s rRcs(eZdZdZ‡fdd„Zdd„Z‡ZS)Ú Conv2dPadBiasaQModel with only Conv2d layers, all with bias and some with padding > 0, some in a Sequential and some following. Activation function modules in between each layer. Used to test that bias is propagated correctly in the special case of pruned Conv2d-Bias-(Activation)Conv2d fusion, when the second Conv2d layer has padding > 0.csÚtƒ ¡t tjdddddddt ¡tjddddddt ¡tjdddddddt ¡tjdddddddt ¡tjddddddt ¡¡ |_tjdd ddddd|_t ¡|_ tjd d ddddd|_ t ¡|_ dS) NrrGr;T)Úpaddingr+Fr*rHrIrJ) rrrr/rLr=r>r1rMr?rNr@r4r r r rás" ö  zConv2dPadBias.__init__cCs6| |¡}| |¡}| |¡}| |¡}| |¡}|Sr5)r1rMr?rNr@r6r r r r8ôs      zConv2dPadBias.forwardr9r r r r rSÛs rScs(eZdZdZ‡fdd„Zdd„Z‡ZS)Ú Conv2dPoolzèModel with only Conv2d layers, all with bias, some in a Sequential and some following. Activation function modules in between each layer, Pool2d modules in between each layer. Used to test pruned Conv2d-Pool2d-Conv2d fusion.c sÂtƒ ¡t tjddddddtjddddt ¡tjddddddt ¡tjdddd¡|_ tjdd dddd|_ tjdddd|_ t ¡|_ tjd d dddd|_ tjd d dddd|_dS) NrrGr;T©Ú kernel_sizerTr+é©rWZstriderTrHrIrJ)rrrr/rLÚ MaxPool2dr=r>Ú AvgPool2dr1rMÚmaxpoolÚaf1rNÚconv2d3r4r r r rs ú zConv2dPool.__init__cCs\| |¡}| |¡}| |¡}| |¡}| |¡}tj|dddd}t |¡}| |¡}|S©NrXrrY) r1rMr\r]rNrDZ avg_pool2drEr^r6r r r r8s       zConv2dPool.forwardr9r r r r rUýs rUcs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚConv2dPoolFlattenFunctionalaModel with Conv2d layers, all with bias, some in a Sequential and some following, and then a Pool2d and a functional Flatten followed by a Linear layer. Activation functions and Pool2ds in between each layer also. Used to test pruned Conv2d-Pool2d-Flatten-Linear fusion.c s¸tƒ ¡t tjddddddtjddddt ¡tjddddddt ¡tjdddd¡|_ tjdddddd|_ t ¡|_ tjdd dddd|_ t  d ¡|_tjd d dd |_dS) Nrr;TrVrXrYr)r(é )rré r*)rrrr/rLrZr=r>r[r1rMr]rNÚAdaptiveAvgPool2dÚavg_poolr0Úfcr4r r r r$s ú  z$Conv2dPoolFlattenFunctional.__init__cCs^| |¡}| |¡}tj|dddd}| |¡}| |¡}| |¡}t |d¡}|  |¡}|Sr_) r1rMrDÚ max_pool2dr]rNrdr"Úflattenrer6r r r r84s       z#Conv2dPoolFlattenFunctional.forwardr9r r r r r`s r`cs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚConv2dPoolFlattenaModel with Conv2d layers, all with bias, some in a Sequential and some following, and then a Pool2d and a Flatten module followed by a Linear layer. Activation functions and Pool2ds in between each layer also. Used to test pruned Conv2d-Pool2d-Flatten-Linear fusion.c sÂtƒ ¡t tjddddddtjddddt ¡tjddddddt ¡tjdddd¡|_ tjdddddd|_ t ¡|_ tjdd dddd|_ t  d ¡|_t ¡|_tjd d dd |_dS)Nrr;TrVrXrYr)r(ra)rXrXé,rbr*)rrrr/rLrZr=r>r[r1rMr]rNrcrdZFlattenrgr0rer4r r r rFs ú   zConv2dPoolFlatten.__init__cCs\| |¡}| |¡}tj|dddd}| |¡}| |¡}| |¡}| |¡}| |¡}|Sr_) r1rMrDrfr]rNrdrgrer6r r r r8Ws       zConv2dPoolFlatten.forwardr9r r r r rh@s rhcs4eZdZdZeeeedœ‡fdd„ Zdd„Z‡ZS)ÚLSTMLinearModelzCContainer module with an encoder, a recurrent module, and a linear.©Ú input_dimÚ hidden_dimÚ output_dimÚ num_layerscs,tƒ ¡t |||¡|_t ||¡|_dSr5)rrrÚLSTMÚlstmr0r©rrlrmrnror r r rfs zLSTMLinearModel.__init__cCs | |¡\}}| |¡}||fSr5)rqr)rÚinputÚoutputZhiddenÚdecodedr r r r8ms zLSTMLinearModel.forward©rrrrÚintrr8rr r r r rjcsÿrjcs4eZdZdZeeeedœ‡fdd„ Zdd„Z‡ZS)ÚLSTMLayerNormLinearModelz9Container module with an LSTM, a LayerNorm, and a linear.rkcs8tƒ ¡t |||¡|_t |¡|_t ||¡|_dSr5) rrrrprqZ LayerNormÚnormr0rrrr r r rvs  z!LSTMLayerNormLinearModel.__init__cCs*| |¡\}}| |¡}| |¡}||fSr5)rqryr)rr7rr r r r8~s  z LSTMLayerNormLinearModel.forwardrvr r r r rxssÿrx)Ztorch.ao.pruningrr"Ztorch.nn.functionalrZ functionalrDrr0rÚboolr&ÚModuler'r:r<rArFrPrRrSrUr`rhrjrxr r r r Ús&   "!"#