yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 593.861 milliseconds 3RunComplete 1.13909 seconds 4Run 1.06578 seconds 5Callbacks 279.4 microseconds, 4 calls, 69.85 microseconds average 6Spectrogram 12.0744 milliseconds, 3 calls, 4.0248 milliseconds average 7Sample 3.0016 milliseconds, 28 calls, 107.2 microseconds average 8Encode 614.44 milliseconds 9Decode 451.048 milliseconds 10DecodeStep 448.036 milliseconds, 28 calls, 16.0013 milliseconds average 11 GPU Tasks 12LoadModel 452.128 milliseconds 13Run 1.0518 seconds 14Encode 599.964 milliseconds 15EncodeLayer 506.67 milliseconds, 24 calls, 21.1113 milliseconds average 16Decode 451.832 milliseconds 17DecodeStep 451.828 milliseconds, 28 calls, 16.1367 milliseconds average 18DecodeLayer 412.142 milliseconds, 672 calls, 613.307 microseconds average 19 Compute Shaders 20mulMatTiled 562.478 milliseconds, 529 calls, 1.06329 milliseconds average 21mulMatByRowTiled 256.062 milliseconds, 7803 calls, 32.8158 microseconds average 22softMaxFixed 27.1687 milliseconds, 696 calls, 39.0355 microseconds average 23normFixed 24.1828 milliseconds, 2093 calls, 11.5541 microseconds average 24fmaRepeat1 23.3089 milliseconds, 2093 calls, 11.1366 microseconds average 25addRepeatEx 22.3395 milliseconds, 2064 calls, 10.8234 microseconds average 26softMaxLong 19.7192 milliseconds, 28 calls, 704.257 microseconds average 27copyConvert 19.301 milliseconds, 1440 calls, 13.4035 microseconds average 28copyTranspose 15.3011 milliseconds, 1392 calls, 10.9922 microseconds average 29addRepeatScale 13.6043 milliseconds, 1344 calls, 10.1222 microseconds average 30convolutionMain2Fixed 12.1242 milliseconds 31addRepeatGelu 11.6172 milliseconds, 698 calls, 16.6436 microseconds average 32addRepeat 11.5331 milliseconds, 744 calls, 15.5015 microseconds average 33scaleInPlace 9.5743 milliseconds, 696 calls, 13.7562 microseconds average 34convolutionMain 7.0349 milliseconds 35softMax 5.8329 milliseconds, 672 calls, 8.67991 microseconds average 36diagMaskInf 4.5297 milliseconds, 672 calls, 6.74062 microseconds average 37convolutionPrep1 1.5258 milliseconds, 2 calls, 762.9 microseconds average 38convolutionPrep2 383 microseconds, 2 calls, 191.5 microseconds average 39addRows 194.6 microseconds, 28 calls, 6.95 microseconds average 40add 95.2 microseconds 41 Memory Usage 42Model 877.966 KB RAM, 1.42785 GB VRAM 43Context 1.9831 MB RAM, 723.782 MB VRAM 44Total 2.84049 MB RAM, 2.13467 GB VRAM