summaryrefslogtreecommitdiffstats
path: root/Scripts/transcribe.py
Commit message (Collapse)AuthorAge
* Finish translation for Western European language speakersv0.12.0yum2023-05-30
| | | | | | | | | | | | | | NLLB needs its input to be split up into sentences. I use the sentence_splitter Python package to do this. It supports ~20 Western European languages, but notably, no Asian languages. * Sort spoken language list. English is still at the top. * Remove 'Translation source' dropdown. Infer this from the spoken language. * Add lang_compat.py to map language codes between the various libraries (whisper, nllb, sentence_splitter). * Fix bug where old text would appear in textbox when you first bring it up.
* Add ability to translate into 200 languagesyum2023-05-25
| | | | | | | | | Use Meta's No Language Left Behind (NLLB) algorithm to provide translation capabilities into 200 languages. Obviously most are very untested. This requires either 4.1 or 7.1 GB of RAM and significiantly increases transcription latency.
* Add more text filtersyum2023-05-24
| | | | | | | | | | Add 3 filters: * Remove trailing period * Convert to uppercase * Convert to lowercase All may be composed. Upper/lower just overwrite each other so just use one.
* Add UI toggle for uwu filteryum2023-05-24
| | | | | UI now has a checkbox for the uwu filter. Does not materially affect resource usage or latency when enabled.
* Begin work on uwu filteryum2023-05-24
| | | | | | Use UwwwuPP to translate your boring old speech into uwu-ified version. Still need to add a UI toggle for this.
* Add ability to type using STTyum2023-05-23
| | | | | | | | | | To use it, do a medium hold + long hold. Keep the long hold depressed until you're done speaking. The transcription will be typed into the currently selected input field. * Add more audio feedback * Make audio feedback play asynchronously so it doesn't slow down the controller input state machine as much.
* Add ability to update textbox in placeyum2023-05-22
| | | | | | | By holding the button while talking for at least 1.5 seconds, you can update the contents of the textbox without unlocking it from worldspace. So now you can carefully position your textbox once, then continually speak into it without having to reposition it every time.
* Add keyboard togglev0.11.4yum2023-05-22
| | | | | | Users can now configure a keybind to start/stop/dismiss the STT when in desktop mode. The default keybind is ctrl+x, since by default VRC doesn't use 'x' for anything.
* Fix accidental semicolon typofaker2023-05-22
|
* Enable selecting specific GPU when transcribingyum2023-05-21
| | | | | | Useful on devices with multiple GPUs, such as gaming laptops. * Update GUI/README.md.
* Restore string matching, remove affinity maskv0.11.1yum2023-04-25
| | | | | | Affinity mask no longer affects performance. String matching is still needed for temporal stability in fast-paced long-form transcription tasks.
* ~Finish integrating faster-whisperyum2023-04-24
| | | | I'm able to use the new code to show text in game. Not yet play-tested.
* Begin integrating faster-whisperv0.11.0yum2023-04-23
| | | | | | This is a much faster, lower-VRAM reimplementation of Whisper in Python. Early testing is extremely promising: fast transcription speed, extremely low resource usage (CPU/RAM/VRAM), high accuracy.
* Set PYTHONPATH in synchronous multiprocessing layeryum2023-03-08
| | | | | | | | | A user saw an error like `ModuleNotFoundError: No module named _socket`. StackOverflow blames this on PYTHONPATH, so let's try setting it. * Fix latent bug in Scripts/transcribe.py. PyAudio.open() positional parameters must be specified in correct order, even when telling it which parameter is which. *shrug*
* Revert "Apply previous window conditioning to decoding layer"yum2023-02-22
| | | | | | | | This reverts commit cece1ee8f1b985c2a89adb661dd02c6d44787f67. This does *not* in fact result in improved temporal stability. It makes makes things so unstable that even single-sentence messages fail to ever stabilize.
* Apply previous window conditioning to decoding layeryum2023-02-22
| | | | | Per the Whisper source code, this should result in better temporal stability.
* Remove exponential backoff capv0.7.0yum2023-02-19
| | | | | | | | | | Allows sustained exponential backoff when not transcribing. Used to cap out at 1s. * Add more items to README TODO list * Adjust emote metadata * Emotes bugfix: Non-existent emote map doesn't cause transcription engine to bail out.
* Finish emotesyum2023-02-13
| | | | | | | | | | | Emotes require 2 bytes per char. They're encoded into the region [0xE000, infinity). The texture is 4k, and uses 1k vertical pixels per emote segment, for a maximum of 32 segments. * Reduce volume of noise indicator by 90%. Quiet is probably better. Might want to add a volume slider idk. * Bugfix: emotes without a transparency channel now work * Address a couple Unity performance complaints about the shader
* Begin work adding emotesyum2023-02-13
| | | | | | | | | | | | Done: * Users can add images to Fonts/Emotes/ * The basename of that image ('clueless.png' becomes 'clueless') is the keyword to make the image show up in game. * Fix a bug in the shader where letters on the 2nd texture and later would have UV outside of [0.0, 1.0] Not yet implemented: * transcribed words are encoded using emotes mapping
* Built-in chatbox no longer shows empty messagesyum2023-02-04
| | | | * Reduce noise on/off indicator volume by 50%
* GUI: Add ability to choose buttonyum2023-01-25
| | | | | | We use a button to start/stop transcription. Previously this was hardcoded to left joystick. Now users can pick from {left, right} x {joystick, a, b}.
* Enable using built-in chatboxv0.3yum2023-01-22
| | | | | | | | | | | | | VRChat exposes a built-in chatbox which can be seen by anyone who has it enabled. This was not the case when I started this project: the chatbox would only be visible to friends. Since this is clearly useful, enabling the STT on public models, let's enable sending data to it. Caveats: * The built-in chatbox has anti-spam tech which limits us to updating about once every 2 seconds. The custom chatbox has no such limitation and is thus typically much faster.
* Bugfix: user-provided paths may now contain spacesyum2023-01-04
| | | | | | | | | | | | | | | | Previously, paths containing spaces would be interpreted by python's argument parser as multiple separate arguments, causing it to fail. Now we escape paths inside PythonWrapper using std::quoted(). * Improve PII filtering. Python output would contain multiple path separators (like C:\\Users\\foo\\), defeating the PII regex. * Silence compiler warning in PII filter. * Document usability improvements. * Transcription layer exponential backoff goes to ~infinity when paused. This is a hack, since we really don't need to transcribe at all when paused, but it lets us keep the code simple. Good enough until the next rewrite. * Shader only samples background when necessary. * Limit matchStrings() print()s to DEBUG mode
* Portability bugfixesyum2023-01-01
| | | | | * Expose option to run transcription engine on CPU instead of GPU * Use embedded git when setting up the Python virtual environment
* Bugfix: regions truncate correctly at page boundariesyum2022-12-30
| | | | | | | | Boards whose size is an even multiple of CHARS_PER_SYNC would lose the entire last region. * Attempt to fix runaway memory usage of GUI text frames, but this needs more work
* GUI: Expose transcription window durationyum2022-12-30
| | | | | Users can pick longer transcription durations for accuracy-critical tasks, or shorter durations for latency-critical tasks.
* Bugfix: regenerated FX layers now work on uploaded avatarsyum2022-12-30
| | | | | | | | | | | | | VRChat won't update the FX layer associated with an avatar unless its GUID changes. Delete the GUID file when overwriting our generated FX layer to work around this. * Change paging behavior: when a region is updated, we re-page everything that comes after it. This fixes the issue where we go back to update something, then jump back to the current screen, leaving some random chunk of text somewhere on the board. * Reduce transcription time from 28s to 10s. I'm going to expose this to the user since there's a fundamental latency/stability tradeoff here.
* Fine-tune transcriptionyum2022-12-30
| | | | | | | | Bump up recording window to 28 seconds. This helps a lot with long-form transcription tasks, s.a. transcribing an audiobook. We should expose this as a parameter, since at 10s the transcription delay is typically 300ms, while at 28s it's typically 1.1-1.2s.
* GUI: Users can now control board dimensionsyum2022-12-29
| | | | | | | | Users can now control how many letters wide and tall the board is. Tested at 4x48, 5x60, 10x120, and 20x240. At 20x240, Unity freezes and does not make forward progress. Perhaps creating 4800 float parameters isn't a truly scalable interface.
* Users can disable local beepyum2022-12-29
| | | | | | | | | | | | | | | The transcription engine beeps when you start/stop transcribing so you know that it's listening. Users can now disable this. * add help text to all input fields in GUI * make TaSTT generated file textctrls readonly, since I haven't tested them being reassigned * document idea to configure unity & transcription apps with config files * controller input thread no longer crashes if steamvr isn't running, it just slowly spins and waits * when you stop transcribing, the transcription engine re-transcribes a few times. I think this should improve end-of-transcription tail latencies * transcribe.py now prints out its args
* Encapsulate paging & text wrapping logicyum2022-12-27
| | | | | | | | Define proper interfaces for these things. Simplify osc_ctrl, temporarily dropping support for emotes (they were broken anyway). * Bugfix: Japanese no longer crashes transcribe.py, but it still doesn't show up in the wxTextCtrl
* Make transcription sleeps interruptibleyum2022-12-24
| | | | | This reduces the expected delay to wake up the board & start transcribing from 750 milliseconds to 2.5 milliseconds.
* GUI: expose chars per sync, bytes per charyum2022-12-24
| | | | | | | | | | | | Users can now control how many characters they send per sync event, as well as the number of bytes used to represent each character. This gives them the power to pick between faster paging and fewer sync params. International users must use 2 bytes per char (at least for now). * package.ps1: don't distribute the gigantic TTF files, just the bitmaps
* Quick hack: don't exponentially back off when unpausedyum2022-12-22
| | | | | This fixed some slowness I was seeing when waking up the STT. The right fix is to add interruptible sleeps. Let's fix this soon.
* Control tweak: introduce long/short hold behavioryum2022-12-20
| | | | | | | | | | | | The typical use pattern is now possible without entering radial. Leaving mounted to the world for a long time is no longer possible. Maybe I need an override param? Left joystick controls: * Short press toggle 1: show board, lock to hand, start transcribing * Short press toggle 2: lock to world, stop transcribing * Long press: hide board, stop transcribing
* GUI: Begin work generating animatoryum2022-12-20
| | | | The GUI can now generate guid.map and animations.
* Add ability to select modelyum2022-12-18
| | | | | | | * icon now works when pinned to taskbar * add model selection * add script to dump mic devices * whisper models now download into the virtual environment
* GUI: Add mic, language selectionyum2022-12-18
| | | | | | | | Users can now select their mic & spoken language in the GUI. * pyaudio now samples at the mic rate, fixing an issue where frames would drop. We downsample in the callback by dropping frames. * add Sounds folder to package
* Finish python virtual envyum2022-12-17
GUI can now download all TaSTT dependencies and install them into a virtual environment. * Add buttons to check embedded python version & install dependencies * Add class to wrap interacting with embedded Python * Put all TaSTT python scripts into a folder
generated by cgit v1.2.3 (git 2.39.1) at 2026-06-01 22:28:00 +0000