yum-archive/TaSTT-Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
git clone https://git.yummers.dev/yum-archive/TaSTT-Whisper
993cfa5
master
1 CPU Tasks 2LoadModel 1.99675 seconds 3RunComplete 81.256 seconds 4Run 81.1666 seconds 5Callbacks 17.8976 milliseconds, 37 calls, 483.719 microseconds average 6Spectrogram 483.273 milliseconds, 42 calls, 11.5065 milliseconds average 7Sample 140.511 milliseconds, 511 calls, 274.972 microseconds average 8Encode 50.3768 seconds, 10 calls, 5.03768 seconds average 9Decode 30.7646 seconds, 10 calls, 3.07646 seconds average 10DecodeStep 30.6234 seconds, 511 calls, 59.9284 milliseconds average 11 GPU Tasks 12LoadModel 976.318 milliseconds 13Run 80.8284 seconds 14Encode 51.1656 seconds, 10 calls, 5.11656 seconds average 15EncodeLayer 43.8924 seconds, 240 calls, 182.885 milliseconds average 16Decode 29.6502 seconds, 10 calls, 2.96502 seconds average 17DecodeStep 29.6441 seconds, 511 calls, 58.012 milliseconds average 18DecodeLayer 26.9439 seconds, 12264 calls, 2.19699 milliseconds average 19 Compute Shaders 20mulMatTiledEx 37.1919 seconds, 2400 calls, 15.4966 milliseconds average 21mulMatTiled 13.9953 seconds, 2890 calls, 4.84268 milliseconds average 22mulMatByRowTiled 11.8792 seconds, 120741 calls, 98.3858 microseconds average 23mulMatByRowTiledEx 4.47094 seconds, 24048 calls, 185.917 microseconds average 24softMaxFixed 2.44162 seconds, 12504 calls, 195.267 microseconds average 25convolutionMain2Fixed 1.51096 seconds, 10 calls, 151.096 milliseconds average 26matReshapePanels 1.38964 seconds, 1450 calls, 958.371 microseconds average 27addRepeatGelu 963.292 milliseconds, 12524 calls, 76.9157 microseconds average 28normFixed 925.912 milliseconds, 37793 calls, 24.4996 microseconds average 29copyConvert 875.162 milliseconds, 25488 calls, 34.3362 microseconds average 30scaleInPlace 770.121 milliseconds, 12504 calls, 61.59 microseconds average 31fmaRepeat1 696.227 milliseconds, 37793 calls, 18.4221 microseconds average 32copyTranspose 657.921 milliseconds, 25008 calls, 26.3084 microseconds average 33addRepeatEx 630.019 milliseconds, 37272 calls, 16.9033 microseconds average 34softMaxLong 623.51 milliseconds, 511 calls, 1.22018 milliseconds average 35convolutionMain 471.348 milliseconds, 10 calls, 47.1348 milliseconds average 36addRepeatScale 379.836 milliseconds, 24528 calls, 15.4858 microseconds average 37addRepeat 354.984 milliseconds, 12984 calls, 27.3401 microseconds average 38softMax 197.387 milliseconds, 12264 calls, 16.0948 microseconds average 39diagMaskInf 131.012 milliseconds, 12264 calls, 10.6827 microseconds average 40convolutionPrep2 49.7619 milliseconds, 20 calls, 2.48809 milliseconds average 41convolutionPrep1 42.2907 milliseconds, 20 calls, 2.11454 milliseconds average 42add 10.5473 milliseconds, 10 calls, 1.05473 milliseconds average 43addRows 2.1075 milliseconds, 511 calls, 4.12427 microseconds average 44 Memory Usage 45Model 877.966 KB RAM, 1.42785 GB VRAM 46Context 91.0716 MB RAM, 833.407 MB VRAM 47Total 91.929 MB RAM, 2.24172 GB VRAM