yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 841.605 milliseconds 3RunComplete 62.1145 seconds 4Run 62.0268 seconds 5Callbacks 10.184 milliseconds, 37 calls, 275.243 microseconds average 6Spectrogram 200.241 milliseconds, 42 calls, 4.76764 milliseconds average 7Sample 63.0473 milliseconds, 511 calls, 123.38 microseconds average 8Encode 37.2409 seconds, 10 calls, 3.72409 seconds average 9Decode 24.7715 seconds, 10 calls, 2.47715 seconds average 10DecodeStep 24.7082 seconds, 511 calls, 48.3526 milliseconds average 11 GPU Tasks 12LoadModel 410.579 milliseconds 13Run 61.8044 seconds 14Encode 37.8702 seconds, 10 calls, 3.78702 seconds average 15EncodeLayer 32.1896 seconds, 240 calls, 134.123 milliseconds average 16Decode 23.9262 seconds, 10 calls, 2.39262 seconds average 17DecodeStep 23.9233 seconds, 511 calls, 46.8167 milliseconds average 18DecodeLayer 21.6888 seconds, 12264 calls, 1.76849 milliseconds average 19 Compute Shaders 20mulMatTiledEx 27.5216 seconds, 2400 calls, 11.4673 milliseconds average 21mulMatTiled 10.2385 seconds, 2890 calls, 3.54273 milliseconds average 22mulMatByRowTiled 9.38114 seconds, 120741 calls, 77.6964 microseconds average 23mulMatByRowTiledEx 4.19991 seconds, 24048 calls, 174.647 microseconds average 24softMaxFixed 1.95105 seconds, 12504 calls, 156.034 microseconds average 25convolutionMain2Fixed 1.31354 seconds, 10 calls, 131.354 milliseconds average 26matReshapePanels 1.04699 seconds, 1450 calls, 722.064 microseconds average 27addRepeatGelu 777.683 milliseconds, 12524 calls, 62.0954 microseconds average 28scaleInPlace 750.056 milliseconds, 12504 calls, 59.9853 microseconds average 29copyConvert 701.517 milliseconds, 25488 calls, 27.5234 microseconds average 30normFixed 697.931 milliseconds, 37793 calls, 18.4672 microseconds average 31fmaRepeat1 529.007 milliseconds, 37793 calls, 13.9975 microseconds average 32addRepeatEx 511.269 milliseconds, 37272 calls, 13.7172 microseconds average 33copyTranspose 459.017 milliseconds, 25008 calls, 18.3548 microseconds average 34softMaxLong 382.205 milliseconds, 511 calls, 747.955 microseconds average 35convolutionMain 328.996 milliseconds, 10 calls, 32.8996 milliseconds average 36addRepeat 305.31 milliseconds, 12984 calls, 23.5144 microseconds average 37addRepeatScale 261.749 milliseconds, 24528 calls, 10.6715 microseconds average 38softMax 148.894 milliseconds, 12264 calls, 12.1407 microseconds average 39diagMaskInf 104.681 milliseconds, 12264 calls, 8.53562 microseconds average 40convolutionPrep2 45.8033 milliseconds, 20 calls, 2.29017 milliseconds average 41convolutionPrep1 32.3779 milliseconds, 20 calls, 1.61889 milliseconds average 42add 7.3228 milliseconds, 10 calls, 732.28 microseconds average 43addRows 1.6948 milliseconds, 511 calls, 3.31663 microseconds average 44 Memory Usage 45Model 877.966 KB RAM, 1.42785 GB VRAM 46Context 91.0716 MB RAM, 833.407 MB VRAM 47Total 91.929 MB RAM, 2.24172 GB VRAM