Skip to content
jargo

Your first bot

The complete voice agent is about fifteen lines. This page builds it up one piece at a time, explaining each. For a complete program to run, see examples/getting-started/06-voice-agent, which builds the same agent on the dev runner.

1. The services

Section titled “1. The services”

Each service is a plain Config struct and a constructor. No environment variables are read by the library; that is your app’s job.

key := os.Getenv("OPENAI_API_KEY")


stt := chat.NewSTT(chat.STTConfig{APIKey: key, SampleRate: opus.SampleRate})
llm := chat.NewLLM(chat.LLMConfig{APIKey: key})
tts := chat.NewTTS(chat.TTSConfig{APIKey: key})

Any of the three can be swapped for a different provider independently. They are all just processors behind a small interface. See Services.

2. The transport

Section titled “2. The transport”
params := transport.DefaultParams()
params.AudioInSampleRate = opus.SampleRate
params.AudioOutSampleRate = opus.SampleRate


t := rtc.NewTransport(conn, params)

t.Input() and t.Output() are the processors that sit at the head and tail of the pipeline. conn is a *rtc.Connection you have already answered a WebRTC offer on.

3. The conversation context

Section titled “3. The conversation context”
convo := frames.NewLLMContext("You are a helpful voice assistant. Keep replies short.")
agg := aggregators.New(convo)

agg.User() collects transcriptions into a user message and triggers the LLM; agg.Assistant() records what the bot actually said. They share convo, which is how the conversation accumulates across turns. See LLM context.

Keep replies short is not decoration. A model that writes paragraphs sounds terrible out loud and costs you latency on every turn.

4. Turn-taking

Section titled “4. Turn-taking”

Optional, but it is the difference between a demo and something usable:

vd, _ := vad.NewSilero()
tr, _ := turn.NewSmartTurnV3()


vadProc := vadproc.New(vadproc.Config{VAD: vd})
turnsProc := turns.NewUserTurnProcessor(turns.Config{
    Strategies: turns.UserTurnStrategies{
        Start: turns.DefaultStartStrategies(),
        Stop: []turns.StopStrategy{
            turns.NewTurnAnalyzerStop(turns.TurnAnalyzerConfig{Analyzer: tr}),
        },
    },
})

Both constructors need the ONNX Runtime and return an error without it. Handle that by running without turn-taking rather than failing. The bot still works; it just falls back to STT endpointing and loses barge-in.

When you add these, tell the aggregator to wait for them:

agg := aggregators.New(convo, aggregators.WithTurnTaking())

5. Assemble

Section titled “5. Assemble”

Order matters. This is the reference wiring:

task := pipeline.NewWorker(pipeline.New(
    rtvi.NewProcessor(), // client events over the data channel
    t.Input(),          // audio in
    vadProc,            // is the user speaking?
    stt,                // audio → text
    turnsProc,          // has the user finished?
    agg.User(),         // text → context, triggers the LLM
    llm,                // context → response tokens
    tts,                // tokens → audio
    t.Output(),         // audio out
    agg.Assistant(),    // record what was actually said
), pipeline.WorkerConfig{
  Params: pipeline.Params{
      AudioInSampleRate:  opus.SampleRate,
      AudioOutSampleRate: opus.SampleRate,
      EnableMetrics:      true,
      EnableUsageMetrics: true,
  },
})

Two placements are easy to get wrong:

  • agg.Assistant() goes last, after the output transport. Put it earlier and the context will record sentences the user never heard, because an interruption truncates the audio but not the text.
  • turnsProc goes after stt, not next to vadProc. It needs transcriptions to decide a turn is over. It reaches the processors behind it by pushing frames upstream.

6. Run

Section titled “6. Run”
ctx, cancel := context.WithCancel(context.Background())
go func() { <-conn.Done(); cancel() }()


if err := task.Run(ctx); err != nil {
    slog.Error("task", "err", err)
}

Run blocks until the pipeline finishes. One task per connection: build the whole thing inside your /offer handler and let it die with the connection.

Make it speak first

Section titled “Make it speak first”

By default the bot waits for the user. To greet on connect, queue a run of the current context:

task.QueueFrame(frames.NewLLMRunFrame())

Or say something fixed, bypassing the LLM entirely:

task.QueueFrame(frames.NewTTSSpeakFrame("Hi, what can I help with?"))

Where to go next

Section titled “Where to go next”