summaryrefslogtreecommitdiffstats
path: root/Scripts/transcribe.py
Commit message (Collapse)AuthorAge
* Check in vad.py and delete transcribe.pyyum2023-09-10
| | | | | | | Oops, I meant to check this in a while back. Since transcribe_v2.py now has feature parity with transcribe.py, delete the old code.
* Switch to VadCommitteryum2023-09-07
| | | | | | | | FuzzyRepeatCommitter was approximating this behavior in the best-performing configuration, so switch to it in earnest. This committer simply commits audio once we detect a long enough gap in speech. That's it!
* Fix reference to deprecated symbolv0.14.1yum2023-09-01
|
* Add Unity panel toggle for phonemes (in-game audio indicator)yum2023-09-01
| | | | If not set, the prefab will have its audio sources removed.
* transcribe.py now just reads from config fileyum2023-08-31
| | | | | Duplicating config between args and config is a huge pain in the ass to maintain. Now we just launch using the config generated by the UI. ezpz.
* Bugfixes and tweaksyum2023-08-31
| | | | | | | | | | | | | * Temporarily restore normal process priority. Working on adding a UI option to set STT prio. * Give audio indicator phonemes a 1/3 chance to do nothing. Makes result sound a little better imo. * Quiet down steamVR thread when steamVR isn't running * Fix use of `button_id` and `hand_id` in steamvr.py * Increase amount of silence allowed before transcript from 1 to 5 seconds. You want enough buffer to allow for a few full transcripts, else you risk spuriously dropping audio. * Enable background loading in audio metadata (required by vrc sdk)
* Deprecate commit similarity thresholdyum2023-08-30
| | | | | | | | This is now dynamically set inside transcribe.py. As the buffer grows long, the threshold grows exponentially, keeping the buffer short. The threshold starts small so that transcription starts strict (accurate, slow) and get looser (inaccurate, fast) as needed.
* Switch back to openvryum2023-08-28
| | | | | openxr doesn't have any notion of background process, making it unusable trash :)
* Put audio feedback into its own threadyum2023-08-25
| | | | | | | | | | | | | | | | | | I this improves the code structure of the controller input thread and leads to some deduplication, so I'm going to keep it. However, the intended purpose was to decrease lag when pressing buttons, and in that regard it failed. The lag goes all the way down to the input layer, implying that the input thread is not able to consistently run at its intended 100 Hz sample rate. I suspect that the Python global interpreter lock (GIL) is at fault. Since we can't realistically move all our functionality into one thread in a non-blocking model, I think multiprocessing is the logical choice going forward. Each thread in transcribe.py would become its own process, and pub/sub through some intermediary process sitting in the middle.
* Finish pyopenvr -> pyopenxr migrationyum2023-08-25
| | | | pyopenvr is both deprecated and buggy, so switch to pyopenxr.
* Animate pre-speech ellipsisyum2023-08-11
| | | | Text box now shows an animated ellipsis prior to first speech.
* Deprecate old parametersyum2023-08-11
| | | | | Deprecate the visual and auditory speech indicators, saving 4 bits across the board. Fixed overhead is now 21 bits.
* Fix race condition in commit logicyum2023-08-01
| | | | | | | | Transcription thread now blocks until microphone thread deletes samples as requested. (This is hacky design, it should use a work queue or something, but I don't feel like doing that right now)
* Only back off transcription loop when not transcribingyum2023-08-01
| | | | | | | | | | It's possible that the user has toggled off transcription while the algorithm is still working. In this case we should *not* begin exponential backoff since there's still work to do. Also: * Shorten the hot-path sleep from 50ms to 5ms. * Remove unused variable in SleepInterruptible
* Preserve audio chunk length when dropping samplesyum2023-07-08
| | | | | | | | | | | | | | When we commit a transcription, we drop the corresponding audio data. Audio data is represented as a list of chunks. Each chunk contains a few hundred samples of audio data, representing O(10ms) of audio. If we want to drop a few seconds of data, this means simply deleting many chunks of audio. There's usually a chunk where we want to drop some portion of audio data. Instead of slicing away that part of the chunk, which would change its length, this change zeroes it out. This preserves the assumption that each chunk has the same temporal length.
* Commit logic now drops parts of framesyum2023-07-08
| | | | | | We used to drop entire frames only, leading to situations where more audio is dropped than desired. Now we drop frames down to the precision of the individual audio sample requested.
* Update READMEyum2023-07-07
| | | | | | | | | Mostly updating roadmap stuff. Non-VRC use cases are "complete" since I was mostly targeting streaming. The ability to type into arbitrary text fields is still somewhat nascent & could be improved. Also update some other random stuff to be more up to date. KillFrenzy Avatar Text is now MIT, pog!
* Enforce a stricter avg_logbprob than defaultv0.13.1yum2023-07-07
| | | | | | | | Common hallucinations sneak in around -0.9 avg_logprob. Also: * Limit temperatures to just 0.0. Multiple values cause latency to occasionally spike.
* Filter out segments based on avg_log_prob & no_speech_probyum2023-07-07
| | | | | | | Surprisingly, these args do not cause transcribe() to omit those segments from the result, so we have to manually filter them out. Hallucinated phrases generally have one or both of these params set high.
* Use 16-bit ints with generated silenceyum2023-07-07
| | | | Each sample of audio data is a 16-bit int, not an 8-bit int.
* Fix performance regressionyum2023-07-07
| | | | | Each chunk of audio samples should be encoded as a binary string, not as a list.
* Enforce minimum 5.0 second duration on audio bufferyum2023-07-06
| | | | | | | | | | | | New commit logic would reduce buffer to a size smaller than this, causing it to hallucinate things like: * "See you next time!" * "Thanks for watching!" * "Bye!" The hope is that by keeping the buffer at least 5.0 seconds long, as described in the paper, this will cut down on these events.
* Add visual commit indicator to OBS browser sourceyum2023-06-30
| | | | | | | | Circle goes red when speaking, grey when done. Ideally it would be in the top right portion of the browser source, but this is a good start. Also, hard-cap transcripts to 4096 chars. This prevents the STT from lagging during long sessions.
* Bugfix: trailing period filter ignores ellipsesyum2023-06-30
| | | | ... also print out "Ready!" when the STT is done loading.
* fix: set gpu device index in whisper modeljsopn2023-06-30
|
* Fix race condition around audio frames droppingyum2023-06-28
| | | | | | | | | | | onAudioFramesAvailable would bail out if audio_state.audio_paused is set, preventing frames from being dropped. This would cause transcriptions to get repeated sometimes. Now that frame dropping code always runs. Also adjust the code structure of the keyboard/VR input handlers to be more similar.
* Bugfix: commit no longer wipes out audio bufferyum2023-06-28
| | | | | | | | | | | | Audio data is stored in chunks of frames, not in individual frames. When I commit a transcript, I want to get rid of the portion of the audio data responsible for that particular transcript. I have code that does this, but it was dropping a slice of the list assuming that each sample is stored individually. Extra fun: Because we have to decimate mic frames, we have to convert between whisper frames and mic frames to drop the correct amount of audio data.
* Add profanity filteryum2023-06-28
| | | | | | | Add toggle to UI to enable a profanity filter. It replaces vowels in bad words with asterisks. Bugfix: filters now apply to OBS
* Add toggle for debug modeyum2023-06-28
| | | | | | | | Most transcription output is now gone by default. Users can enable a more verbose output by toggling `Enable debug mode`. Bugfix: Toggling off transcription would reset audio state, frequently resulting in the loss of the last few words spoken.
* Add UI for fuzzy commit thresholdyum2023-06-27
| | | | | | | | | | | | | | Recap: In the STT there's an algorithm that tries to determine when a transcript is "stable" enough to commit. If that is too loose, then accuracy suffers; if too strict, then the audio buffer eventually fills. To mitigate the problem, I check whether the last N transcripts are within some edit distance (Levenshtein edit distance) of each other. The fuzzy matching lets us forgive small instabilities, like differences in uppercase/lowercase or punctuation, while rejecting large instabilities. The default value of 8 seems to be in the sweet spot of accuracy & performance, but it will likely be tuned in the future.
* Adjust commit logic to use fuzzy string match thresholdyum2023-06-27
| | | | | | | | ... instead of simple equality. TODO: add UI for threshold. Bugfix: Frame::onAppStop() joins the OBS app thread.
* Add ability to preserve transcript while using push to talkyum2023-06-27
| | | | | | | | | | | | | This is useful when streaming. Occasionally the STT can get into a bad state, and manually segmenting clears it up. However doing so would clear your accumulated transcript, which isn't always desired. Add ability to preserve the transcript. A small wrinkle: the new commit logic requires N consecutive identical windows before committing. To make this feature play nicely with it, I had to forcibly commit any preview text that hasn't yet been committed. Failing to do this would usually cause short utterances / the most recently said stuff to get wiped out.
* Add UI for browser srcyum2023-06-26
| | | | Add ability to toggle on/off browser src & configure port.
* Bugfix: Transcript no longer repeats when paused in desktopyum2023-06-26
| | | | | Hitting the desktop keybinding to stop transcription would sometimes cause the last transcript to repeeat itself.
* Add browser source, hardcoded to port 8097yum2023-06-26
| | | | | | | | | | | | | | | | | | | Transcription output now streams to localhost:8097. In OBS: * Create a browser source. * url: localhost:8097 * width: 2200 * height: 400 TODO: * Put behind toggle. * Create input field for port. Misc cleanup: * transcribe.py: Drop frames from audio capture thread instead of the transcription thread. Doing it the other way would result in occasional data loss.
* Rework transcription commit logicyum2023-06-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | At the core of the STT, there's a loop which uses Whisper to convert audio into a transcript. As you say something, whisper sees growing fragments of your sentence: t0: "Hell" t1: "Hello" t2: "Hello, world!" So we need some algorithm which takes these fragments and accumulates them into an ever-growing transcript. Previously I did this with fuzzy string matching. I'd find the region where the two transcripts overlap and edit the two together to produce a longer transcript. The big problem is that if there's no overlap, it's not clear whether whisper radically changed its mind as to what was said, or whether the user paused for a long time before saying something new. So I'd have to reset the growing transcript. Now I get the timestamps from Whisper and wait for it to give me the same 3 transcripts for the last utterance. Once the transcript stabilizes like this, I commit the text. This enables a temporally stable, ever-growing transcript that's also quite accurate. To prevent a latency regression, I also introduce the notion of "preview text", which is a preview of an utterance that has not yet stabilized. These previews do not contribute to the ever-growing transcript, but do get fed through the rest of the app, so they show up in-game / in OBS. Once they eventually stabilize, they get committed to the ever-growing transcript. This change is lightly tested!
* Finish translation for Western European language speakersv0.12.0yum2023-05-30
| | | | | | | | | | | | | | NLLB needs its input to be split up into sentences. I use the sentence_splitter Python package to do this. It supports ~20 Western European languages, but notably, no Asian languages. * Sort spoken language list. English is still at the top. * Remove 'Translation source' dropdown. Infer this from the spoken language. * Add lang_compat.py to map language codes between the various libraries (whisper, nllb, sentence_splitter). * Fix bug where old text would appear in textbox when you first bring it up.
* Add ability to translate into 200 languagesyum2023-05-25
| | | | | | | | | Use Meta's No Language Left Behind (NLLB) algorithm to provide translation capabilities into 200 languages. Obviously most are very untested. This requires either 4.1 or 7.1 GB of RAM and significiantly increases transcription latency.
* Add more text filtersyum2023-05-24
| | | | | | | | | | Add 3 filters: * Remove trailing period * Convert to uppercase * Convert to lowercase All may be composed. Upper/lower just overwrite each other so just use one.
* Add UI toggle for uwu filteryum2023-05-24
| | | | | UI now has a checkbox for the uwu filter. Does not materially affect resource usage or latency when enabled.
* Begin work on uwu filteryum2023-05-24
| | | | | | Use UwwwuPP to translate your boring old speech into uwu-ified version. Still need to add a UI toggle for this.
* Add ability to type using STTyum2023-05-23
| | | | | | | | | | To use it, do a medium hold + long hold. Keep the long hold depressed until you're done speaking. The transcription will be typed into the currently selected input field. * Add more audio feedback * Make audio feedback play asynchronously so it doesn't slow down the controller input state machine as much.
* Add ability to update textbox in placeyum2023-05-22
| | | | | | | By holding the button while talking for at least 1.5 seconds, you can update the contents of the textbox without unlocking it from worldspace. So now you can carefully position your textbox once, then continually speak into it without having to reposition it every time.
* Add keyboard togglev0.11.4yum2023-05-22
| | | | | | Users can now configure a keybind to start/stop/dismiss the STT when in desktop mode. The default keybind is ctrl+x, since by default VRC doesn't use 'x' for anything.
* Fix accidental semicolon typofaker2023-05-22
|
* Enable selecting specific GPU when transcribingyum2023-05-21
| | | | | | Useful on devices with multiple GPUs, such as gaming laptops. * Update GUI/README.md.
* Restore string matching, remove affinity maskv0.11.1yum2023-04-25
| | | | | | Affinity mask no longer affects performance. String matching is still needed for temporal stability in fast-paced long-form transcription tasks.
* ~Finish integrating faster-whisperyum2023-04-24
| | | | I'm able to use the new code to show text in game. Not yet play-tested.
* Begin integrating faster-whisperv0.11.0yum2023-04-23
| | | | | | This is a much faster, lower-VRAM reimplementation of Whisper in Python. Early testing is extremely promising: fast transcription speed, extremely low resource usage (CPU/RAM/VRAM), high accuracy.
* Set PYTHONPATH in synchronous multiprocessing layeryum2023-03-08
| | | | | | | | | A user saw an error like `ModuleNotFoundError: No module named _socket`. StackOverflow blames this on PYTHONPATH, so let's try setting it. * Fix latent bug in Scripts/transcribe.py. PyAudio.open() positional parameters must be specified in correct order, even when telling it which parameter is which. *shrug*
generated by cgit v1.2.3 (git 2.39.1) at 2026-06-01 22:30:05 +0000