summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAge
* Put audio feedback into its own threadyum2023-08-25
| | | | | | | | | | | | | | | | | | I this improves the code structure of the controller input thread and leads to some deduplication, so I'm going to keep it. However, the intended purpose was to decrease lag when pressing buttons, and in that regard it failed. The lag goes all the way down to the input layer, implying that the input thread is not able to consistently run at its intended 100 Hz sample rate. I suspect that the Python global interpreter lock (GIL) is at fault. Since we can't realistically move all our functionality into one thread in a non-blocking model, I think multiprocessing is the logical choice going forward. Each thread in transcribe.py would become its own process, and pub/sub through some intermediary process sitting in the middle.
* Finish pyopenvr -> pyopenxr migrationyum2023-08-25
| | | | pyopenvr is both deprecated and buggy, so switch to pyopenxr.
* Various shader cleanupyum2023-08-21
| | | | | | | * remove unused variables, functions, keywords * rename fixedN to floatN * move min/max raymarch distance to top of ray_march.cginc * fix frame emission
* Workaround: use STT mesh to write depth bufferyum2023-08-12
| | | | | | There's some bug around using the raymarched world position to write the depth buffer. I wasn't able to find it quickly, so for now, use the original world position to write the depth buffer.
* Improve numerical stability in raymarcheryum2023-08-12
| | | | | | | | | | | | Increase units by a factor of 100 to avoid running into numerical instability on 32-bit floats. This comes at zero measured performance cost. This makes a visible difference in quality. Other minor changes: * Raymarching loop tries to get up to 4x closer than MINIMUM_HIT_DISTANCE before bailing out. This comes at no measured performance cost. * Convert `fixed` types to `float` in STT_text.cginc.
* Bugfix: Shader no shows text mirroredyum2023-08-12
|
* Bugfix: ellipsis waits for boardyum2023-08-12
| | | | | Regression created while optimizing shader. Performance still around 730 microseconds on my computer with this change.
* Optimize skew-pyramid frame SDFyum2023-08-12
| | | | | Use symmetry to reduce # of distance calculations by 50%. Because the pyramid can be skewed, we can't reduce this by another factor of 2.
* Small raymarching optimizationsyum2023-08-11
| | | | | | | | | | | | Using PIX to quantify changes, reduce raymarcher runtime from ~1.0 ms to ~850 us. In order of impact: * Tighten raymarch min/max distances * Make `in_mirror` check truly branchless * Gate ellipsis animation with non-divergent if statement Everything else is < 10 microseconds of improvement.
* Animate pre-speech ellipsisyum2023-08-11
| | | | Text box now shows an animated ellipsis prior to first speech.
* Deprecate old parametersyum2023-08-11
| | | | | Deprecate the visual and auditory speech indicators, saving 4 bits across the board. Fixed overhead is now 21 bits.
* Add animated ellipsis to shaderyum2023-08-11
| | | | | | | Not yet done: * Animator toggle * OSC integration
* Remove extraneous dot when chatbox spawnsyum2023-08-10
| | | | | The chatbox background would not completely disappear at _Emerge = 0, so two dots would show up when the chatbox spawns.
* Bump up ray marching stepsyum2023-08-10
| | | | Performance impact remains to be seen.
* Clean up pbr.cgincyum2023-08-10
| | | | | Remove unused code & cruft. Ray marcher now updates i.worldPos before executing PBR shading, which fixes some artifacts.
* Add basic PBR parameters to new shaderyum2023-08-10
| | | | | No UVs for raymarched geometry yet, so drop textures. Also drop most old shader settings.
* Add skew pyramid to bottom of ray-marched chatboxyum2023-08-10
| | | | Looks more like a comic-style speech bubble now.
* Bugfix: shader no longer shows up as pinkv0.14.0yum2023-08-10
| | | | | Fix up .mat to point to correct textures/shader. Also delete templates after copying shaders.
* Fix user-reported bug in generate_shader.pyyum2023-08-10
| | | | Specify file encoding when generating shaders.
* Add show/hide animation for ray-marched custom chatboxyum2023-08-10
| | | | * Fix mirror behavior for ray-marched chatbox
* Begin work on show/hide animationsyum2023-08-10
|
* Add ray-marched custom chatboxyum2023-08-09
| | | | | | | | * Refactor shader code to make development easier. Templates are now as small as possible. * Update scaling code. Use Unity scaling instead of a blendshape. * Check in a fuckton of shader FOSS. Mostly unused. * Update TaSTT.fbx. Now has 6 faces instead of 2.
* Fix issue where white boxes appear on custom chatboxv0.13.3yum2023-08-09
| | | | | | GUI was not correctly managing .meta files, causing two textures to use the same GUID. Unity would notice and regenerate GUIDs, breaking the custom chatbox material's texture references.
* Fix race condition in commit logicyum2023-08-01
| | | | | | | | Transcription thread now blocks until microphone thread deletes samples as requested. (This is hacky design, it should use a work queue or something, but I don't feel like doing that right now)
* Only back off transcription loop when not transcribingyum2023-08-01
| | | | | | | | | | It's possible that the user has toggled off transcription while the algorithm is still working. In this case we should *not* begin exponential backoff since there's still work to do. Also: * Shorten the hot-path sleep from 50ms to 5ms. * Remove unused variable in SleepInterruptible
* Add ability to auto-regen unity assetsyum2023-07-25
| | | | | | | | | | | | | | | | Add two buttons: start auto re-generation of Unity assets, and stop. These start/stop a thread which periodically (every 3 seconds) hashes the user-provided animator, menu and parameters. When any one of these change, it invokes the function to generate Unity assets. The hash is non-cryptographic, so it's light. The only hit is that we have to read the entire file contents every few seconds, and compute a sum across that entire memory region. This is extremely light unless you're on a spinning platter hard drive with a small cache. Still seeing the bug where the material drops ref to the font bitmaps. Probably need to update the .mat using the guids in the bitmap .meta files.
* Subsequent calls to `Generate unity assets` don't break texturesyum2023-07-25
| | | | | Avoid deleting bitmap .meta files so that once the user sets up their shader, it doesn't break.
* Unity assets can be generated at a configurable pathv0.13.2yum2023-07-24
| | | | Useful for projects with multiple avatars with different animators.
* Bugfix: unity panel now shows saved pathsyum2023-07-24
| | | | | | The paths you enter in the Unity panel (animator, menu, params, and assets folder) are saved in the app config, but were not populated correctly on app restart or pane redraw. Now they are.
* Preserve audio chunk length when dropping samplesyum2023-07-08
| | | | | | | | | | | | | | When we commit a transcription, we drop the corresponding audio data. Audio data is represented as a list of chunks. Each chunk contains a few hundred samples of audio data, representing O(10ms) of audio. If we want to drop a few seconds of data, this means simply deleting many chunks of audio. There's usually a chunk where we want to drop some portion of audio data. Instead of slicing away that part of the chunk, which would change its length, this change zeroes it out. This preserves the assumption that each chunk has the same temporal length.
* Commit logic now drops parts of framesyum2023-07-08
| | | | | | We used to drop entire frames only, leading to situations where more audio is dropped than desired. Now we drop frames down to the precision of the individual audio sample requested.
* Update READMEyum2023-07-07
| | | | | | | | | Mostly updating roadmap stuff. Non-VRC use cases are "complete" since I was mostly targeting streaming. The ability to type into arbitrary text fields is still somewhat nascent & could be improved. Also update some other random stuff to be more up to date. KillFrenzy Avatar Text is now MIT, pog!
* Enforce a stricter avg_logbprob than defaultv0.13.1yum2023-07-07
| | | | | | | | Common hallucinations sneak in around -0.9 avg_logprob. Also: * Limit temperatures to just 0.0. Multiple values cause latency to occasionally spike.
* Filter out segments based on avg_log_prob & no_speech_probyum2023-07-07
| | | | | | | Surprisingly, these args do not cause transcribe() to omit those segments from the result, so we have to manually filter them out. Hallucinated phrases generally have one or both of these params set high.
* Use 16-bit ints with generated silenceyum2023-07-07
| | | | Each sample of audio data is a 16-bit int, not an 8-bit int.
* Fix performance regressionyum2023-07-07
| | | | | Each chunk of audio samples should be encoded as a binary string, not as a list.
* Enforce minimum 5.0 second duration on audio bufferyum2023-07-06
| | | | | | | | | | | | New commit logic would reduce buffer to a size smaller than this, causing it to hallucinate things like: * "See you next time!" * "Thanks for watching!" * "Bye!" The hope is that by keeping the buffer at least 5.0 seconds long, as described in the paper, this will cut down on these events.
* Begin work on proxy serveryum2023-07-03
| | | | | | | | | | | | | | | | | | | | | Create a simple server with 3 endpoints: * /create_session: Create a session and return its identifier. * /set_transcript: Update a session's transcript. * /get_transcript: Fetch a session's transcript. Right now the session ID provides authentication *and* authorization. There is no public/private ID so you have to trust whoever you share your ID with. IDs are long and generated by the server, so it should be somewhat secure against low-effort hacking. Other updates: * Drop whisper_requirements.txt - no longer needed. * Vendor curl to make it easier to interact with the server. TODO: * Fuzz test the server.
* Add profanity filteryum2023-07-02
| | | | Forgot to check this in, oops!
* Add visual commit indicator to OBS browser sourceyum2023-06-30
| | | | | | | | Circle goes red when speaking, grey when done. Ideally it would be in the top right portion of the browser source, but this is a good start. Also, hard-cap transcripts to 4096 chars. This prevents the STT from lagging during long sessions.
* Bugfix: trailing period filter ignores ellipsesyum2023-06-30
| | | | ... also print out "Ready!" when the STT is done loading.
* Merge pull request #3 from jsopn/fix-gpu-device-indexv0.13.0yum-food2023-06-30
|\ | | | | Set GPU device index in whisper model
| * fix: set gpu device index in whisper modeljsopn2023-06-30
|/
* Fix race condition around audio frames droppingyum2023-06-28
| | | | | | | | | | | onAudioFramesAvailable would bail out if audio_state.audio_paused is set, preventing frames from being dropped. This would cause transcriptions to get repeated sometimes. Now that frame dropping code always runs. Also adjust the code structure of the keyboard/VR input handlers to be more similar.
* Bugfix: commit no longer wipes out audio bufferyum2023-06-28
| | | | | | | | | | | | Audio data is stored in chunks of frames, not in individual frames. When I commit a transcript, I want to get rid of the portion of the audio data responsible for that particular transcript. I have code that does this, but it was dropping a slice of the list assuming that each sample is stored individually. Extra fun: Because we have to decimate mic frames, we have to convert between whisper frames and mic frames to drop the correct amount of audio data.
* Add profanity filteryum2023-06-28
| | | | | | | Add toggle to UI to enable a profanity filter. It replaces vowels in bad words with asterisks. Bugfix: filters now apply to OBS
* Add toggle for debug modeyum2023-06-28
| | | | | | | | Most transcription output is now gone by default. Users can enable a more verbose output by toggling `Enable debug mode`. Bugfix: Toggling off transcription would reset audio state, frequently resulting in the loss of the last few words spoken.
* Add UI for fuzzy commit thresholdyum2023-06-27
| | | | | | | | | | | | | | Recap: In the STT there's an algorithm that tries to determine when a transcript is "stable" enough to commit. If that is too loose, then accuracy suffers; if too strict, then the audio buffer eventually fills. To mitigate the problem, I check whether the last N transcripts are within some edit distance (Levenshtein edit distance) of each other. The fuzzy matching lets us forgive small instabilities, like differences in uppercase/lowercase or punctuation, while rejecting large instabilities. The default value of 8 seems to be in the sweet spot of accuracy & performance, but it will likely be tuned in the future.
* Adjust commit logic to use fuzzy string match thresholdyum2023-06-27
| | | | | | | | ... instead of simple equality. TODO: add UI for threshold. Bugfix: Frame::onAppStop() joins the OBS app thread.
* Add ability to preserve transcript while using push to talkyum2023-06-27
| | | | | | | | | | | | | This is useful when streaming. Occasionally the STT can get into a bad state, and manually segmenting clears it up. However doing so would clear your accumulated transcript, which isn't always desired. Add ability to preserve the transcript. A small wrinkle: the new commit logic requires N consecutive identical windows before committing. To make this feature play nicely with it, I had to forcibly commit any preview text that hasn't yet been committed. Failing to do this would usually cause short utterances / the most recently said stuff to get wiped out.
generated by cgit v1.2.3 (git 2.39.1) at 2026-07-16 22:10:55 +0000