yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 7.92578 seconds 3RunComplete 8.33686 seconds 4Run 8.25683 seconds 5Callbacks 337.7 microseconds, 4 calls, 84.425 microseconds average 6Spectrogram 16.4214 milliseconds, 3 calls, 5.4738 milliseconds average 7Sample 3.8768 milliseconds, 27 calls, 143.585 microseconds average 8Encode 4.14309 seconds 9Decode 4.11338 seconds 10DecodeStep 4.10947 seconds, 27 calls, 152.203 milliseconds average 11 GPU Tasks 12LoadModel 7.53025 seconds 13Run 8.05464 seconds 14Encode 3.98133 seconds 15EncodeLayer 3.29696 seconds, 32 calls, 103.03 milliseconds average 16Decode 4.07331 seconds 17DecodeStep 4.07331 seconds, 27 calls, 150.863 milliseconds average 18DecodeLayer 3.81856 seconds, 864 calls, 4.41963 milliseconds average 19 Compute Shaders 20mulMatTiled 3.6307 seconds, 705 calls, 5.14993 milliseconds average 21mulMatByRowTiled 1.45034 seconds, 10010 calls, 144.889 microseconds average 22fmaRepeat1 774.011 milliseconds, 2684 calls, 288.38 microseconds average 23copyTranspose 416.996 milliseconds, 1792 calls, 232.699 microseconds average 24addRepeatScale 223.798 milliseconds, 1728 calls, 129.513 microseconds average 25norm 211.821 milliseconds, 2684 calls, 78.9197 microseconds average 26softMaxFixed 209.124 milliseconds, 896 calls, 233.398 microseconds average 27copyConvert 186.898 milliseconds, 1856 calls, 100.699 microseconds average 28addRepeatEx 176.985 milliseconds, 2656 calls, 66.6361 microseconds average 29softMaxLong 162.482 milliseconds, 27 calls, 6.01787 milliseconds average 30convolutionMain2Fixed 154.739 milliseconds 31addRepeatGelu 132.088 milliseconds, 898 calls, 147.091 microseconds average 32softMax 98.1905 milliseconds, 864 calls, 113.646 microseconds average 33scaleInPlace 57.3956 milliseconds, 896 calls, 64.0576 microseconds average 34convolutionMain 45.7933 milliseconds 35addRepeat 36.9315 milliseconds, 960 calls, 38.4703 microseconds average 36diagMaskInf 28.2987 milliseconds, 864 calls, 32.7531 microseconds average 37convolutionPrep1 9.0334 milliseconds, 2 calls, 4.5167 milliseconds average 38convolutionPrep2 1.1608 milliseconds, 2 calls, 580.4 microseconds average 39add 320 microseconds 40addRows 244 microseconds, 27 calls, 9.03704 microseconds average 41 Memory Usage 42Model 892.591 KB RAM, 2.8815 GB VRAM 43Context 1.98376 MB RAM, 1.07641 GB VRAM 44Total 2.85543 MB RAM, 3.95791 GB VRAM