yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 3.22847 seconds 3RunComplete 14.2729 seconds 4Run 14.186 seconds 5Callbacks 674.7 microseconds, 4 calls, 168.675 microseconds average 6Spectrogram 29.6112 milliseconds, 3 calls, 9.8704 milliseconds average 7Sample 7.7473 milliseconds, 27 calls, 286.937 microseconds average 8Encode 11.8931 seconds 9Decode 2.29185 seconds 10DecodeStep 2.28406 seconds, 27 calls, 84.5949 milliseconds average 11 GPU Tasks 12LoadModel 1.9083 seconds 13Run 14.0698 seconds 14Encode 11.9404 seconds 15EncodeLayer 10.2786 seconds, 32 calls, 321.205 milliseconds average 16Decode 2.12941 seconds 17DecodeStep 2.12938 seconds, 27 calls, 78.8661 milliseconds average 18DecodeLayer 1.98655 seconds, 864 calls, 2.29924 milliseconds average 19 Compute Shaders 20mulMatTiledEx 8.49883 seconds, 320 calls, 26.5589 milliseconds average 21mulMatTiled 2.04655 seconds, 385 calls, 5.31573 milliseconds average 22mulMatByRowTiled 982.48 milliseconds, 8346 calls, 117.719 microseconds average 23mulMatByRowTiledEx 481.123 milliseconds, 1664 calls, 289.137 microseconds average 24convolutionMain2Fixed 415.244 milliseconds 25softMaxFixed 404.223 milliseconds, 896 calls, 451.142 microseconds average 26matReshapePanels 210.915 milliseconds, 193 calls, 1.09282 milliseconds average 27norm 154.589 milliseconds, 2684 calls, 57.5965 microseconds average 28convolutionMain 126.883 milliseconds 29addRepeatGelu 112.131 milliseconds, 898 calls, 124.867 microseconds average 30copyConvert 100.589 milliseconds, 1856 calls, 54.1968 microseconds average 31scaleInPlace 91.3539 milliseconds, 896 calls, 101.957 microseconds average 32fmaRepeat1 86.7731 milliseconds, 2684 calls, 32.3298 microseconds average 33copyTranspose 77.5852 milliseconds, 1792 calls, 43.2953 microseconds average 34addRepeat 76.3677 milliseconds, 960 calls, 79.5497 microseconds average 35addRepeatEx 70.8699 milliseconds, 2656 calls, 26.6829 microseconds average 36softMaxLong 39.4035 milliseconds, 27 calls, 1.45939 milliseconds average 37addRepeatScale 25.0842 milliseconds, 1728 calls, 14.5163 microseconds average 38softMax 14.0555 milliseconds, 864 calls, 16.2679 microseconds average 39convolutionPrep1 13.9331 milliseconds, 2 calls, 6.96655 milliseconds average 40diagMaskInf 8.1717 milliseconds, 864 calls, 9.45799 microseconds average 41convolutionPrep2 5.2098 milliseconds, 2 calls, 2.6049 milliseconds average 42add 2.9724 milliseconds 43addRows 91.2 microseconds, 27 calls, 3.37778 microseconds average 44 Memory Usage 45Model 892.591 KB RAM, 2.8815 GB VRAM 46Context 1.98376 MB RAM, 1.13447 GB VRAM 47Total 2.85543 MB RAM, 4.01597 GB VRAM