The Technical Architecture of a Talking Guitar

To make a guitar "speak," the project moves beyond simple audio playback by integrating a digital signal processing (DSP) pipeline within a DAW (Digital Audio Workstation) using the JUCE framework. The core workflow involves three distinct stages: text-to-speech conversion, word segmentation, and pitch-mapped synthesis.

Solving the Segmentation Problem

Converting text to speech is straightforward, but mapping that speech to a guitar requires precise word segmentation. The author initially attempted Energy Gap Segmentation, which cuts audio based on silence (decibels near zero). This failed because natural speech often lacks silence between words. The solution was a hybrid approach using a Sonority Peak Syllabifier, which identifies vowels as markers for syllable boundaries. Even with this, manual adjustment of segment boundaries remains necessary for high-quality output.

Pitch Detection and Synthesis

To allow the guitar to "sing" or match notes, the system must extract the fundamental frequency from the instrument in real-time. The author uses the YIN algorithm for pitch detection. Once the pitch is identified, the system generates a synthesized tone shaped by an ADSR envelope (Attack, Decay, Sustain, Release). This synthesized signal is then passed through a vocoder, which uses the AI-generated speech as a modulator to shape the guitar's output into recognizable words.

Scaling to Conversational AI

By integrating a local Large Language Model (LLM) and OpenAI's Whisper for speech-to-text, the guitar can function as an interactive agent. The pipeline works as follows:

  1. Input: A user speaks into a microphone.
  2. Processing: Whisper transcribes the audio, which is sent to a local LLM for a response.
  3. Output: The LLM's text response is converted to audio and routed through the guitar's DSP chain.

For singing, the author uses the VocalSet dataset—a collection of recorded vocal samples. Because real-time pitch shifting of complex vocal samples is computationally expensive, the author uses the World vocoder to pre-bake pitch-shifted samples for each vowel, mapping them to specific frets on the guitar. This allows the instrument to "sing" by triggering these pre-processed samples in response to live fret input.