I Gave My Agent a Phone Number

· 8 min read

My phone rang. Unknown number.

I picked up. “Hello?”

“Hi! This is a test call from OpenClaw.”

That voice. I’d steered it into existence. Hadn’t written a single line of the implementation myself.

Here’s how that happened.

The Idea

Late 2025. I was running OpenClaw (then called Clawdbot) as my personal AI assistant. It could message me on Telegram, Discord, WhatsApp. But I had a thought:

What if it could actually call me?

I imagined coding by voice. Calling in from a Nokia 3310 and building a website. Okay, maybe not that extreme. But a real phone call? Ring my phone, say something, have a conversation?

That seemed cool enough to try.

How We Built It

I wasn’t just giving vague directions. I did the groundwork:

  • Researched Twilio, Telnyx, and telephony basics
  • Set up the Twilio account (the phone number verification alone took an hour)
  • Found documentation on Media Streams, TwiML, and audio encoding
  • Provided all of this as context to my agent

Then I steered:

Me: “Here’s the Twilio docs for Media Streams. I want to support multiple providers. Twilio now, Telnyx later. Design an architecture that makes swapping easy.”

The agent took it from there. Researched μ-law encoding (phone audio format), proposed a four-layer design, started implementing.

The Architecture

Agent Layer— voice_call tool, session context
Call Manager— State machine, transcripts, timers
Provider Abstraction— Twilio | Telnyx | Plivo | Mock
Media Streams— Audio encoding, WebSocket streaming

The provider abstraction was my requirement. I knew I’d want Telnyx later (it’s cheaper). Building that flexibility in from the start meant we wouldn’t have to gut the code later.

Debugging Live

We debugged live.

# Terminal 1: Gateway running
pnpm gateway:dev

# Terminal 2: Watching logs
openclaw voicecall tail

# Terminal 3: The agent, editing code

When the gateway crashed, we’d fall back to Claude Code for deeper debugging. But most of the work happened through the OpenClaw agent itself. Editing files, checking logs, restarting services.

No deploy cycles. No waiting. Change the code, gateway restarts, test again.

The First Call

AgentBasic call flow implemented. Can you test? I'll call your number.
MeGo ahead.
Phone rings. I pick up. Silence.
MeI hear nothing.
Agentthinking...Checking Media Stream connection... I see 'connected' but no 'media' events. The audio stream isn't starting.
Agent checks TwiML response, finds missing Stream URL, fixes it.
AgentFixed. Try again?
Phone rings. I pick up.
Agent(through phone) Hello! This is a test call from OpenClaw.
MeI hear it! But it's choppy.
Agentthinking...Audio buffer size. Let me adjust chunk timing...

This loop repeated until it worked smoothly. Agent tries, I test, agent debugs. I never touched the code directly.

What Shipped

The PR didn’t get merged directly. Peter, the creator of OpenClaw, saw it and adapted it to fit the plugin architecture he was building:

Peter's response on the PR

The core logic, the provider abstraction, the call flow: all made it in. Since then, the community has made it robust, adding Telnyx, Plivo, OpenAI Realtime STT, and conference calls.

A Note on Safety

Giving an agent the ability to make phone calls is powerful. It’s also the kind of thing that can go wrong in ways that cost real money or annoy real people.

We added an allowList to the config. The agent can only call numbers you explicitly permit. No config, no calls. Simple boundary, big peace of mind.

This isn’t paranoia. It’s just good hygiene when you’re giving software a new way to reach the outside world.

I don’t know if this is how everyone will build software in the future. But it worked for me.

The hard part isn’t building anymore. It’s knowing what’s worth building. Learning to harness these tools, to orchestrate them well. That’s the new skill.

← Back to writings