Skip to content
jargo

Telephony

A phone agent is the same pipeline over a different transport. Instead of WebRTC, audio arrives as µ-law 8 kHz over a WebSocket from a phone provider.

The transport

Section titled “The transport”

transport/wsserver serves the WebSocket endpoint the provider streams to. The wire format is provider-specific and supplied as a Serializer, so the transport itself stays provider-agnostic:

ser := twilio.New(twilio.Config{ /* … */ })


params := wsserver.DefaultParams()
params.AudioInSampleRate = 16000
params.AudioOutSampleRate = 16000


t, err := wsserver.Accept(w, r, ser, params)
Provider Package
Twilio Media Streams transport/wsserver/twilio
Telnyx Media Streaming transport/wsserver/telnyx
Plivo Audio Streaming transport/wsserver/plivo
Exotel Media Streaming transport/wsserver/exotel

Each is New(Config) returning a *Serializer. One serializer serves one session: build it per call, not once at startup; it is not safe to share.

Ending the call

Section titled “Ending the call”

The Twilio, Telnyx and Plivo serializers hang the call up over the provider’s REST API when the pipeline sends an EndFrame or CancelFrame. That is the default: a pipeline that has finished otherwise leaves the caller listening to silence, and the leg billing, until something else hangs up.

Supply the credentials the REST call needs (AccountSID/AuthToken for Twilio, APIKey for Telnyx, AuthID/AuthToken for Plivo). A serializer that is to hang up with nothing to authorize it is refused at Setup, so the pipeline fails to start rather than running a whole conversation and then leaving the leg up.

Set AutoHangUp to false for a bot that is one step of a longer call the provider goes on to route elsewhere:

off := false
ser := twilio.New(twilio.Config{AutoHangUp: &off})

Twilio calls carried on a regional edge are ended there, not at the global host. Name both halves, which is what Twilio’s api.{edge}.{region}.twilio.com host format needs, or give BaseURL outright for a Twilio-compatible or self-hosted backend:

ser := twilio.New(twilio.Config{
    AccountSID: "...", AuthToken: "...",
    Region: "au1", Edge: "sydney",
})

Session length

Section titled “Session length”

Params.SessionTimeout bounds how long one session may run. When it elapses with the socket still open, wsserver.EventSessionTimeout fires. Nothing is closed by it: a call that has gone on too long usually wants to be told so before the pipeline ends, rather than being cut off mid-sentence.

params.SessionTimeout = 10 * time.Minute
t, _ := wsserver.Accept(w, r, ser, params)
events.On(t.Events(), wsserver.EventSessionTimeout, func(ctx context.Context, _ struct{}) {
    task.QueueFrame(frames.NewTTSSpeakFrame("We are out of time. Goodbye."))
    task.StopWhenDone()
})

Origins

Section titled “Origins”

Params.AllowedOrigins names the origins a browser may open the socket from. Empty, the default, allows every origin, which is what a phone provider needs: it is not a browser and sends no Origin header at all.

Set it whenever the endpoint is one a browser connects to (the rtviws serializer, say). Without it, a page on any other site can open the socket in a visitor’s browser and hold a conversation as them. Accept refuses a request whose origin is not listed, and one carrying no origin at all, with wsserver.ErrOriginNotAllowed and no reply written, leaving the endpoint to choose what to tell it:

params := wsserver.DefaultParams()
params.AllowedOrigins = []string{"https://app.example"}


t, err := wsserver.Accept(w, r, ser, params)
if errors.Is(err, wsserver.ErrOriginNotAllowed) {
    http.Error(w, "forbidden", http.StatusForbidden)
    return
}

Sample rates

Section titled “Sample rates”

Telephony audio is µ-law 8 kHz on the wire, and always will be. The pipeline does not have to run at that rate: the serializer converts at each edge, so set the pipeline rate to whatever suits its services and let it follow.

const pipelineSampleRate = 16000


params.AudioInSampleRate = pipelineSampleRate
params.AudioOutSampleRate = pipelineSampleRate


tts := elevenlabs.NewTTS(elevenlabs.Config{
    APIKey:     key,
    SampleRate: pipelineSampleRate,
})


task := pipeline.NewWorker(pipeline.New(procs...), pipeline.WorkerConfig{
  Params: pipeline.Params{
      AudioInSampleRate:  pipelineSampleRate,
      AudioOutSampleRate: pipelineSampleRate,
  },
})

Running the whole pipeline at 8 kHz works and saves two conversions, but it hands 8 kHz to the transcriber and asks the voice for 8 kHz back. Both are audibly worse than converting once on the way in and once on the way out, so 16 kHz is the better default. Ask the TTS provider for the pipeline rate directly, so its audio is not synthesized at 24 kHz and downsampled twice.

Two knobs on wsserver.AudioConfig cover the rest:

ser := twilio.New(twilio.Config{
    Audio: wsserver.AudioConfig{
        SampleRate:          24000, // override the pipeline rate for this leg
        ResamplerClearAfter: -1,    // never clear the resampler history
    },
})

A stream resampler that has sat idle starts the next chunk fresh, so the tail of one utterance is not filtered into the start of the next. Providers whose chunks arrive at irregular intervals want that off, since those gaps are gaps in delivery rather than gaps in the audio.

The pipeline

Section titled “The pipeline”

Identical to the WebRTC one, minus the RTVI processor, since there is no data channel on a phone call:

pipeline.New(
    t.Input(), vadProc, stt, turnsProc,
    agg.User(), llm, tts, t.Output(), agg.Assistant(),
)

See examples/twiliobot for the complete server.

The idle watchdog

Section titled “The idle watchdog”

Phone calls need this more than browser sessions do: a caller who wanders off leaves the line open and the meter running. Configure it on the turn processor:

turnsProc := turns.NewUserTurnProcessor(turns.Config{
    Strategies:  strategies,
    IdleTimeout: 10 * time.Second,
    OnIdle: func(ctx context.Context, c *turns.UserIdleController) error {
        return c.Push(ctx, frames.NewTTSSpeakFrame("Are you still there?"),
            processor.Downstream)
    },
})

OnIdle fires each time the timeout elapses, so escalate and eventually hang up rather than asking forever. To end the call from inside the callback, push an EndWorkerFrame, the mechanism a processor uses to reach the Task:

return c.Push(ctx, frames.NewEndWorkerFrame(), processor.Downstream)

Retune the timeout mid-call with UserIdleTimeoutUpdateFrame: shorter while waiting for a yes/no, longer while the caller reads out a number.

DTMF

Section titled “DTMF”

Keypresses arrive as InputDTMFFrame (a system frame, so they are never dropped by a barge-in). processor/dtmf aggregates digits into complete entries, so an account number typed at speed arrives as one value rather than eight frames.

Play tones outbound with OutputDTMFFrame.

For menu navigation, processor/ivr provides a navigator that takes the LLM’s place in the pipeline. It first asks the LLM whether an automated system or a person answered. On an automated system it switches the LLM to a navigation prompt built around your goal, and the voice-activity detector to a two-second stop so a pause between menu options does not end the turn; the LLM then presses keys with <dtmf> tags and reports progress with <ivr> tags, which the navigator raises as on_ivr_status_changed. On a person it raises on_conversation_detected with the conversation so far, for your bot to take over:

nav := ivr.NewNavigator(ivr.NavigatorConfig{LLM: llm, Goal: "Reach the billing department."})
events.On(nav.Processor().Events(), ivr.EventIVRStatusChanged, func(ctx context.Context, s ivr.Status) {
    // completed: hand over to the bot; stuck: hang up or retry.
})


pipe := pipeline.New(in, stt, userAgg, nav, tts, out, assistantAgg)

processor/voicemail detects an answering machine so the bot does not hold a conversation with a recording.

The voicemail detector listens to what the other side says and asks a classifier (see classifier/llm and classifier/jev) whether a person answered. It sits after the STT service, and its gate after the TTS service: the gate holds the bot’s speech back until the verdict, so a greeting is never talked over. The verdict acts once the caller has been quiet for DecisionTimeout, since “hi, this is Sam” is how a person answers and how a greeting starts. A voicemail verdict drops the held speech, interrupts the bot, keeps any further input from the conversation, and raises on_voicemail_detected once the greeting has been quiet for VoicemailResponseDelay, so the handler can leave a message:

detector, err := voicemail.New(voicemail.Config{Classifier: c})
if err != nil {
    return err
}
events.OnSignal(detector.Events(), voicemail.EventVoicemailDetected, func(ctx context.Context) {
    _ = detector.PushFrame(ctx, frames.NewTTSSpeakFrame("Please call us back."), processor.Downstream)
})


pipe := pipeline.New(in, stt, detector.Detector(), userAgg, llm, tts, detector.Gate(), out, assistantAgg)

Practical notes

Section titled “Practical notes”
  • 8 kHz µ-law hurts STT accuracy. Expect a real drop versus wideband audio and budget for it in prompts. Confirm important values back to the caller. Running the pipeline at 16 kHz does not recover what the wire never carried: it only stops the transcriber and the voice from working at telephone bandwidth on top of that.
  • Turn-taking matters more on the phone. There is no video, no visual backchannel, and callers expect the rhythm of a phone conversation. Tune it.
  • Recording is usually regulated. Consent requirements vary by jurisdiction.