yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 2.88964 seconds 3RunComplete 140.747 seconds 4Run 140.661 seconds 5Callbacks 20.302 milliseconds, 44 calls, 461.409 microseconds average 6Spectrogram 468.419 milliseconds, 41 calls, 11.4249 milliseconds average 7Sample 139.558 milliseconds, 527 calls, 264.815 microseconds average 8Encode 87.5396 seconds, 9 calls, 9.72662 seconds average 9Decode 53.0971 seconds, 9 calls, 5.89968 seconds average 10DecodeStep 52.9566 seconds, 527 calls, 100.487 milliseconds average 11 GPU Tasks 12LoadModel 1.86694 seconds 13Run 140.175 seconds 14Encode 88.7441 seconds, 9 calls, 9.86046 seconds average 15EncodeLayer 75.809 seconds, 288 calls, 263.226 milliseconds average 16Decode 51.4306 seconds, 9 calls, 5.71451 seconds average 17DecodeStep 51.43 seconds, 527 calls, 97.5901 milliseconds average 18DecodeLayer 48.1822 seconds, 16864 calls, 2.85711 milliseconds average 19 Compute Shaders 20mulMatTiledEx 69.1011 seconds, 2880 calls, 23.9934 milliseconds average 21mulMatTiled 21.009 seconds, 3465 calls, 6.06321 milliseconds average 22mulMatByRowTiled 20.0965 seconds, 166278 calls, 120.861 microseconds average 23mulMatByRowTiledEx 9.61326 seconds, 33152 calls, 289.975 microseconds average 24softMaxFixed 3.7631 seconds, 17152 calls, 219.397 microseconds average 25norm 2.23806 seconds, 51704 calls, 43.2859 microseconds average 26convolutionMain2Fixed 2.12825 seconds, 9 calls, 236.472 milliseconds average 27matReshapePanels 2.0333 seconds, 1737 calls, 1.17058 milliseconds average 28addRepeatGelu 1.5491 seconds, 17170 calls, 90.2211 microseconds average 29scaleInPlace 1.32928 seconds, 17152 calls, 77.5001 microseconds average 30copyConvert 1.23135 seconds, 34880 calls, 35.3026 microseconds average 31fmaRepeat1 1.10337 seconds, 51704 calls, 21.3401 microseconds average 32addRepeatEx 1.00095 seconds, 51168 calls, 19.562 microseconds average 33copyTranspose 846.807 milliseconds, 34304 calls, 24.6854 microseconds average 34addRepeat 704.028 milliseconds, 17728 calls, 39.7128 microseconds average 35softMaxLong 608.58 milliseconds, 527 calls, 1.1548 milliseconds average 36convolutionMain 522.249 milliseconds, 9 calls, 58.0277 milliseconds average 37addRepeatScale 500.937 milliseconds, 33728 calls, 14.8523 microseconds average 38softMax 236.054 milliseconds, 16864 calls, 13.9975 microseconds average 39diagMaskInf 171.964 milliseconds, 16864 calls, 10.1971 microseconds average 40convolutionPrep1 60.7331 milliseconds, 18 calls, 3.37406 milliseconds average 41convolutionPrep2 33.441 milliseconds, 18 calls, 1.85783 milliseconds average 42add 12.0883 milliseconds, 9 calls, 1.34314 milliseconds average 43addRows 1.9724 milliseconds, 527 calls, 3.74269 microseconds average 44 Memory Usage 45Model 892.591 KB RAM, 2.8815 GB VRAM 46Context 92.2612 MB RAM, 1.19934 GB VRAM 47Total 93.1329 MB RAM, 4.08084 GB VRAM