yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 600.5 milliseconds 3RunComplete 14.9475 seconds 4Run 14.8676 seconds 5Callbacks 8.0039 milliseconds, 37 calls, 216.322 microseconds average 6Spectrogram 193.196 milliseconds, 42 calls, 4.59991 milliseconds average 7Sample 52.0611 milliseconds, 511 calls, 101.881 microseconds average 8Encode 5.97889 seconds, 10 calls, 597.889 milliseconds average 9Decode 8.8778 seconds, 10 calls, 887.78 milliseconds average 10DecodeStep 8.82556 seconds, 511 calls, 17.2712 milliseconds average 11 GPU Tasks 12LoadModel 457.133 milliseconds 13Run 14.7971 seconds 14Encode 6.01034 seconds, 10 calls, 601.034 milliseconds average 15EncodeLayer 5.11447 seconds, 240 calls, 21.3103 milliseconds average 16Decode 8.78676 seconds, 10 calls, 878.676 milliseconds average 17DecodeStep 8.78674 seconds, 511 calls, 17.1952 milliseconds average 18DecodeLayer 8.10499 seconds, 12264 calls, 660.876 microseconds average 19 Compute Shaders 20mulMatTiled 6.3857 seconds, 5290 calls, 1.20713 milliseconds average 21mulMatByRowTiled 4.79001 seconds, 144789 calls, 33.0827 microseconds average 22normFixed 417.279 milliseconds, 37793 calls, 11.0412 microseconds average 23fmaRepeat1 399.385 milliseconds, 37793 calls, 10.5677 microseconds average 24softMaxFixed 382.654 milliseconds, 12504 calls, 30.6025 microseconds average 25addRepeatEx 378.135 milliseconds, 37272 calls, 10.1453 microseconds average 26copyConvert 319.573 milliseconds, 25488 calls, 12.5382 microseconds average 27copyTranspose 258.327 milliseconds, 25008 calls, 10.3298 microseconds average 28softMaxLong 244.787 milliseconds, 511 calls, 479.035 microseconds average 29addRepeatScale 223.995 milliseconds, 24528 calls, 9.13223 microseconds average 30addRepeatGelu 181.428 milliseconds, 12524 calls, 14.4864 microseconds average 31softMax 150.065 milliseconds, 12264 calls, 12.2362 microseconds average 32scaleInPlace 147.891 milliseconds, 12504 calls, 11.8275 microseconds average 33addRepeat 145.362 milliseconds, 12984 calls, 11.1955 microseconds average 34convolutionMain2Fixed 124.04 milliseconds, 10 calls, 12.404 milliseconds average 35diagMaskInf 78.8542 milliseconds, 12264 calls, 6.42973 microseconds average 36convolutionMain 66.8723 milliseconds, 10 calls, 6.68723 milliseconds average 37convolutionPrep1 15.4358 milliseconds, 20 calls, 771.79 microseconds average 38convolutionPrep2 3.8144 milliseconds, 20 calls, 190.72 microseconds average 39addRows 3.5214 milliseconds, 511 calls, 6.89119 microseconds average 40add 929.8 microseconds, 10 calls, 92.98 microseconds average 41 Memory Usage 42Model 877.966 KB RAM, 1.42785 GB VRAM 43Context 91.0716 MB RAM, 785.219 MB VRAM 44Total 91.929 MB RAM, 2.19467 GB VRAM