yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 7.95251 seconds 3RunComplete 109.423 seconds 4Run 109.351 seconds 5Callbacks 12.7226 milliseconds, 44 calls, 289.15 microseconds average 6Spectrogram 270.286 milliseconds, 41 calls, 6.59235 milliseconds average 7Sample 69.0965 milliseconds, 527 calls, 131.113 microseconds average 8Encode 35.943 seconds, 9 calls, 3.99366 seconds average 9Decode 73.3946 seconds, 9 calls, 8.15496 seconds average 10DecodeStep 73.3251 seconds, 527 calls, 139.137 milliseconds average 11 GPU Tasks 12LoadModel 7.55659 seconds 13Run 109.16 seconds 14Encode 36.3141 seconds, 9 calls, 4.0349 seconds average 15EncodeLayer 29.8405 seconds, 288 calls, 103.613 milliseconds average 16Decode 72.8459 seconds, 9 calls, 8.09398 seconds average 17DecodeStep 72.8458 seconds, 527 calls, 138.227 milliseconds average 18DecodeLayer 69.0153 seconds, 16864 calls, 4.09247 milliseconds average 19 Compute Shaders 20mulMatTiled 36.8159 seconds, 6345 calls, 5.80234 milliseconds average 21mulMatByRowTiled 28.0431 seconds, 199430 calls, 140.616 microseconds average 22copyTranspose 8.11917 seconds, 34304 calls, 236.683 microseconds average 23fmaRepeat1 7.85961 seconds, 51704 calls, 152.012 microseconds average 24addRepeatScale 4.11915 seconds, 33728 calls, 122.129 microseconds average 25softMaxFixed 3.22072 seconds, 17152 calls, 187.775 microseconds average 26copyConvert 2.8333 seconds, 34880 calls, 81.2298 microseconds average 27addRepeatEx 2.78075 seconds, 51168 calls, 54.3455 microseconds average 28norm 2.76591 seconds, 51704 calls, 53.495 microseconds average 29addRepeatGelu 2.35162 seconds, 17170 calls, 136.961 microseconds average 30softMaxLong 2.24788 seconds, 527 calls, 4.26543 milliseconds average 31softMax 2.21477 seconds, 16864 calls, 131.331 microseconds average 32convolutionMain2Fixed 1.38064 seconds, 9 calls, 153.405 milliseconds average 33addRepeat 1.30665 seconds, 17728 calls, 73.7057 microseconds average 34scaleInPlace 1.10329 seconds, 17152 calls, 64.3245 microseconds average 35diagMaskInf 937.457 milliseconds, 16864 calls, 55.5892 microseconds average 36convolutionMain 374.967 milliseconds, 9 calls, 41.663 milliseconds average 37convolutionPrep1 119.171 milliseconds, 18 calls, 6.62059 milliseconds average 38convolutionPrep2 27.8894 milliseconds, 18 calls, 1.54941 milliseconds average 39addRows 5.2536 milliseconds, 527 calls, 9.96888 microseconds average 40add 2.8285 milliseconds, 9 calls, 314.278 microseconds average 41 Memory Usage 42Model 892.591 KB RAM, 2.8815 GB VRAM 43Context 92.2612 MB RAM, 1.14026 GB VRAM 44Total 93.1329 MB RAM, 4.02176 GB VRAM