yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 945.134 milliseconds 3RunComplete 2.19628 seconds 4Run 2.08991 seconds 5Callbacks 762.3 microseconds, 4 calls, 190.575 microseconds average 6Spectrogram 12.2602 milliseconds, 3 calls, 4.08673 milliseconds average 7Sample 3.2495 milliseconds, 27 calls, 120.352 microseconds average 8Encode 1.31469 seconds 9Decode 774.432 milliseconds 10DecodeStep 771.17 milliseconds, 27 calls, 28.5618 milliseconds average 11 GPU Tasks 12LoadModel 803.014 milliseconds 13Run 2.07007 seconds 14Encode 1.29615 seconds 15EncodeLayer 1.11858 seconds, 32 calls, 34.9556 milliseconds average 16Decode 773.917 milliseconds 17DecodeStep 773.915 milliseconds, 27 calls, 28.6635 milliseconds average 18DecodeLayer 719.389 milliseconds, 864 calls, 832.626 microseconds average 19 Compute Shaders 20mulMatTiled 1.20543 seconds, 705 calls, 1.70983 milliseconds average 21mulMatByRowTiled 471.054 milliseconds, 10010 calls, 47.0584 microseconds average 22norm 72.3989 milliseconds, 2684 calls, 26.9743 microseconds average 23fmaRepeat1 41.7212 milliseconds, 2684 calls, 15.5444 microseconds average 24copyTranspose 41.168 milliseconds, 1792 calls, 22.9732 microseconds average 25softMaxFixed 40.8861 milliseconds, 896 calls, 45.6318 microseconds average 26addRepeatEx 30.1243 milliseconds, 2656 calls, 11.342 microseconds average 27copyConvert 28.9217 milliseconds, 1856 calls, 15.5828 microseconds average 28softMaxLong 25.3209 milliseconds, 27 calls, 937.811 microseconds average 29convolutionMain2Fixed 19.8769 milliseconds 30addRepeatScale 18.2236 milliseconds, 1728 calls, 10.5461 microseconds average 31addRepeatGelu 15.7554 milliseconds, 898 calls, 17.545 microseconds average 32addRepeat 14.2968 milliseconds, 960 calls, 14.8925 microseconds average 33scaleInPlace 13.9332 milliseconds, 896 calls, 15.5504 microseconds average 34softMax 8.5928 milliseconds, 864 calls, 9.94537 microseconds average 35convolutionMain 8.532 milliseconds 36diagMaskInf 5.6745 milliseconds, 864 calls, 6.56771 microseconds average 37convolutionPrep1 2.303 milliseconds, 2 calls, 1.1515 milliseconds average 38convolutionPrep2 422.9 microseconds, 2 calls, 211.45 microseconds average 39addRows 198.7 microseconds, 27 calls, 7.35926 microseconds average 40add 119.8 microseconds 41 Memory Usage 42Model 892.591 KB RAM, 2.8815 GB VRAM 43Context 1.98376 MB RAM, 1.07641 GB VRAM 44Total 2.85543 MB RAM, 3.95791 GB VRAM