yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 1.49776 seconds 3RunComplete 110.474 seconds 4Run 110.407 seconds 5Callbacks 14.0412 milliseconds, 44 calls, 319.118 microseconds average 6Spectrogram 201.605 milliseconds, 41 calls, 4.91719 milliseconds average 7Sample 65.5117 milliseconds, 527 calls, 124.311 microseconds average 8Encode 64.8806 seconds, 9 calls, 7.20896 seconds average 9Decode 45.5097 seconds, 9 calls, 5.05663 seconds average 10DecodeStep 45.4439 seconds, 527 calls, 86.2313 milliseconds average 11 GPU Tasks 12LoadModel 951.086 milliseconds 13Run 110.123 seconds 14Encode 65.7998 seconds, 9 calls, 7.31109 seconds average 15EncodeLayer 56.2581 seconds, 288 calls, 195.341 milliseconds average 16Decode 44.3232 seconds, 9 calls, 4.9248 seconds average 17DecodeStep 44.3227 seconds, 527 calls, 84.1039 milliseconds average 18DecodeLayer 41.6477 seconds, 16864 calls, 2.46962 milliseconds average 19 Compute Shaders 20mulMatTiledEx 51.4882 seconds, 2880 calls, 17.8778 milliseconds average 21mulMatByRowTiled 17.0626 seconds, 166278 calls, 102.615 microseconds average 22mulMatTiled 15.2984 seconds, 3465 calls, 4.41513 milliseconds average 23mulMatByRowTiledEx 9.04106 seconds, 33152 calls, 272.715 microseconds average 24softMaxFixed 3.04176 seconds, 17152 calls, 177.342 microseconds average 25norm 1.98157 seconds, 51704 calls, 38.3254 microseconds average 26convolutionMain2Fixed 1.8145 seconds, 9 calls, 201.611 milliseconds average 27matReshapePanels 1.526 seconds, 1737 calls, 878.526 microseconds average 28addRepeatGelu 1.24631 seconds, 17170 calls, 72.5866 microseconds average 29scaleInPlace 1.23743 seconds, 17152 calls, 72.1448 microseconds average 30copyConvert 1.02044 seconds, 34880 calls, 29.2557 microseconds average 31fmaRepeat1 993.664 milliseconds, 51704 calls, 19.2183 microseconds average 32addRepeatEx 953.85 milliseconds, 51168 calls, 18.6415 microseconds average 33copyTranspose 705.073 milliseconds, 34304 calls, 20.5537 microseconds average 34addRepeat 581.089 milliseconds, 17728 calls, 32.778 microseconds average 35addRepeatScale 553.89 milliseconds, 33728 calls, 16.4222 microseconds average 36softMaxLong 387.949 milliseconds, 527 calls, 736.147 microseconds average 37convolutionMain 363.351 milliseconds, 9 calls, 40.3723 milliseconds average 38softMax 242.956 milliseconds, 16864 calls, 14.4068 microseconds average 39diagMaskInf 179.046 milliseconds, 16864 calls, 10.617 microseconds average 40convolutionPrep1 45.2096 milliseconds, 18 calls, 2.51164 milliseconds average 41convolutionPrep2 28.6853 milliseconds, 18 calls, 1.59363 milliseconds average 42add 8.1107 milliseconds, 9 calls, 901.189 microseconds average 43addRows 1.542 milliseconds, 527 calls, 2.926 microseconds average 44 Memory Usage 45Model 892.591 KB RAM, 2.8815 GB VRAM 46Context 92.2612 MB RAM, 1.19934 GB VRAM 47Total 93.1329 MB RAM, 4.08084 GB VRAM