yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 950.578 milliseconds 3RunComplete 27.5329 seconds 4Run 27.434 seconds 5Callbacks 10.6484 milliseconds, 44 calls, 242.009 microseconds average 6Spectrogram 199.106 milliseconds, 41 calls, 4.85624 milliseconds average 7Sample 58.7404 milliseconds, 527 calls, 111.462 microseconds average 8Encode 11.3813 seconds, 9 calls, 1.26459 seconds average 9Decode 16.0418 seconds, 9 calls, 1.78242 seconds average 10DecodeStep 15.9829 seconds, 527 calls, 30.3281 milliseconds average 11 GPU Tasks 12LoadModel 805.211 milliseconds 13Run 27.338 seconds 14Encode 11.3967 seconds, 9 calls, 1.2663 seconds average 15EncodeLayer 9.78685 seconds, 288 calls, 33.9821 milliseconds average 16Decode 15.9412 seconds, 9 calls, 1.77125 seconds average 17DecodeStep 15.9412 seconds, 527 calls, 30.249 milliseconds average 18DecodeLayer 15.0511 seconds, 16864 calls, 892.499 microseconds average 19 Compute Shaders 20mulMatTiled 12.0503 seconds, 6345 calls, 1.89919 milliseconds average 21mulMatByRowTiled 9.45404 seconds, 199430 calls, 47.4053 microseconds average 22norm 1.32432 seconds, 51704 calls, 25.6135 microseconds average 23fmaRepeat1 583.884 milliseconds, 51704 calls, 11.2928 microseconds average 24addRepeatEx 536.551 milliseconds, 51168 calls, 10.4861 microseconds average 25softMaxFixed 534.105 milliseconds, 17152 calls, 31.1395 microseconds average 26copyConvert 500.4 milliseconds, 34880 calls, 14.3463 microseconds average 27copyTranspose 377.38 milliseconds, 34304 calls, 11.001 microseconds average 28addRepeatScale 315.294 milliseconds, 33728 calls, 9.34814 microseconds average 29addRepeatGelu 283.978 milliseconds, 17170 calls, 16.5392 microseconds average 30softMaxLong 245.57 milliseconds, 527 calls, 465.976 microseconds average 31scaleInPlace 226.545 milliseconds, 17152 calls, 13.2081 microseconds average 32softMax 212.206 milliseconds, 16864 calls, 12.5834 microseconds average 33addRepeat 209.397 milliseconds, 17728 calls, 11.8117 microseconds average 34convolutionMain2Fixed 184.615 milliseconds, 9 calls, 20.5128 milliseconds average 35diagMaskInf 107.423 milliseconds, 16864 calls, 6.36998 microseconds average 36convolutionMain 74.7954 milliseconds, 9 calls, 8.3106 milliseconds average 37convolutionPrep1 20.9316 milliseconds, 18 calls, 1.16287 milliseconds average 38convolutionPrep2 3.8103 milliseconds, 18 calls, 211.683 microseconds average 39addRows 3.7939 milliseconds, 527 calls, 7.19905 microseconds average 40add 1.0895 milliseconds, 9 calls, 121.056 microseconds average 41 Memory Usage 42Model 892.591 KB RAM, 2.8815 GB VRAM 43Context 92.2612 MB RAM, 1.14026 GB VRAM 44Total 93.1329 MB RAM, 4.02176 GB VRAM