04 Embedded · Audio · AI · Interaction

AI Study Bot

A Kirby-shaped desktop companion that listens to spoken questions, answers them with AI through a speaker, and shows a little personality on a round display — and doubles as an offline study timer when you just want to focus.

ESP32-S3 GC9A01A round TFT INMP441 I²S mic MAX98357 I²S amp TTP223 touch Speech-to-text → LLM → text-to-speech C / Arduino
Kirby lit up inside the custom milled enclosure
1.28" GC9A01A · 240×240 · Kirby expressions

The idea.

Combine the embedded and hardware skills from my earlier projects with something more interactive — a device with a personality, not just a circuit on a bench.

The goal was a small desktop companion that could listen to a spoken question, process it with AI, and answer out loud, using a display and physical touch to make it feel like a character rather than a gadget. It also had to be useful when offline: a built-in study tool with a checklist mode, a Pomodoro timer, and a plain study timer, so the device earns its desk space even with no network.

What it does.

One button flips Kirby between a focused study tool and a talking AI assistant. The voice loop runs entirely on the device, calling out to the cloud only for the heavy AI work.

The display and touch sensor layer personality and interaction on top: waking from sleep, reacting to touch, and moving Kirby's mouth in time with the reply.

Features

VOICE

AI conversation mode

Press the button → Kirby greets you → records your question → transcribes it → asks the LLM → speaks the answer, with the mouth animated to the audio. Then loops back to listening.

OFFLINE

Study tool

Checklist mode, a Pomodoro cycle with breaks, and a plain countdown timer — all working with no WiFi, so the device is useful even when the AI isn't reachable.

DISPLAY

Expressive face

Multiple Kirby expressions on the round TFT, plus an audio-reactive mouth that opens in proportion to how loud the speech is, for a natural talking look.

INPUT

Touch & button

A capacitive touch sensor wakes Kirby and drives interactions; a dedicated button toggles AI mode. Idle → sleeping face; tap → awake and smiling.

RESILIENCE

Graceful degradation

If WiFi or the AI service fails, the device reports why on screen and falls back to the offline study tool instead of hanging or looping.

AUDIO

Full I²S audio path

Digital microphone in and Class-D amplifier out over two independent I²S ports, with the amplifier hardware-muted between phrases to keep the speaker silent when idle.

Hardware.

Off-the-shelf modules chosen so each subsystem could be tested on its own before integration. Two components are wired for the next iteration and not yet in the software path.

🖥️
GC9A01A round TFT1.28", 240×240, SPI — Kirby's face and the study UI
🧠
ESP32-S3 (N16R8)Dual-core MCU with PSRAM — the extra RAM is the whole reason for moving here
🎤
INMP441I²S MEMS microphone for voice capture
🔊
MAX98357A + speakerI²S Class-D amplifier driving a small speaker
👆
TTP223 touch sensorCapacitive pad for waking and interacting with Kirby
🔘
AI-mode button PLANNEDDedicated toggle so the touch pad stays free for Kirby interactions
🔒
Fingerprint sensor PLANNEDUser authentication for per-person study profiles
🎀
Milled Kirby/cat enclosureCustom shell so it reads as a desktop companion, not a dev board
ESP32 stack and components
ESP32 stack and components

Testing setup.

Every subsystem was proven in isolation first. This turned out to be the single most useful decision of the whole build — when integration broke, I always had a known-good reference to compare against.

A. Display

Verified the GC9A01A could render the different Kirby expressions cleanly at full frame.

Round GC9A01A display driven over breadboard
Round GC9A01A display driven over breadboard

B. Touch

Confirmed the TTP223 as a reliable physical input for waking and interaction.

Touch interaction under test
Touch interaction under test

C. Audio

Ran the INMP441 mic and MAX98357 amp together in a standalone loopback sketch to establish reliable input and output on the new wiring.

Mic, amp and speaker wired on the breadboard
Mic, amp and speaker wired on the breadboard

D. AI pipeline

Verified the round trip from the ESP32 to the cloud AI service and back, end to end.

Study mode running on the round display
Study mode running on the round display

Troubleshooting.

The interesting part. Integrating the working subsystems surfaced a chain of bugs that each looked like a hardware fault but turned out to be software — and finding them is what pushed the design toward its final form.

Case 01

No sound at all — not even a test tone

Symptom

The speaker was completely silent — no AI voice, and not even a boot-time test beep. The serial log showed the I²S driver reporting a clock-pin conflict.

Root cause

The I²S pin struct's first field is the master-clock pin. I'd left it out of the initializer — but omitting a field doesn't leave it unset, it gets zero-initialized to GPIO0. Both the mic and amplifier ports silently tried to output a master clock (which neither chip even uses) on GPIO0. The second port to initialize was refused its pins entirely, so the amplifier ended up with no data line.

Fix

Set the master-clock pin explicitly to NO_CHANGE on both ports, and check the driver's return codes instead of assuming success.

Lesson

A silent default masqueraded as working code. Every hardware-init call is now checked and logged, so a failure announces itself instead of hiding.

Case 02

AI calls failing with plenty of memory free

Symptom

Every cloud request failed at the TLS handshake even though the log showed ~93 KB of heap free.

Root cause

A secure connection needs a single contiguous ~40 KB block — total free heap says nothing about the largest available block. A ~96 KB audio buffer parked in the middle of RAM had fragmented the heap into pieces too small for the handshake.

Fix

Allocate the recording buffer only while actually recording, and release it the instant the audio is uploaded — so every handshake runs against an unfragmented heap.

Trade-off

Recording length is now bounded by available RAM (I dropped it from 4 s to 3 s), and the buffer can't be held persistently. This constraint is the direct reason for moving to the ESP32-S3 with more memory.

Case 03

The whole device stopped booting

Symptom

After adding network-time sync, everything went dead — touch and button unresponsive, display frozen on the boot face.

Root cause

The time-sync step waited for a server in a loop with no timeout, inside setup(). When the server didn't answer, setup never finished, so the main loop never ran. The sync wasn't even needed, because certificate validation was disabled.

Fix

Removed the blocking step entirely.

Lesson

Never block startup on a network call without a timeout. A frozen device looks identical to a dead one.

Case 04

Rewiring seemed to do nothing

Symptom

After moving the mic and amp to new pins, the audio behaved as if the wiring hadn't changed, and two peripherals were fighting over the same GPIO.

Root cause

An old set of pin definitions still overrode the new wiring, and two of the stale pins overlapped the new ones.

Fix

Consolidated everything to a single authoritative pin map — one source of truth the whole sketch reads from.

Lesson

Duplicate configuration is a bug waiting to happen. Hardware pins get defined exactly once.

Case 05

Kirby didn't feel alive

Symptom

In AI mode the face flickered between two different Kirby bitmaps, and the talking mouth appeared as a hard rectangle stamped over the face.

Root cause

Each state redrew a full-screen bitmap, and the open mouth was a flat filled box whose single colour never matched Kirby's shading.

Fix

Keep one face on screen the whole time; change only the mouth and a small status dot. The mouth is redrawn by re-blitting the real artwork underneath and drawing the opening on top, so the motion looks natural.

Trade-off

A small status dot (listening / thinking) instead of a separate "thinking" face — chosen for a continuous, single-character feel over a two-frame slideshow.

Case 06

A failed reply retried forever

Symptom

When an AI turn failed, the device looped listening → thinking → fail → listening endlessly, flickering with no error.

Fix

Added a failure counter that stops after three attempts, shows a plain-language error on the display (which step failed, and the status code), and drops back to the offline study tool.

Lesson

Retry logic needs an exit. Failures should be legible to the user, not silent.

One more, earlier on: the voice pipeline started on a different cloud AI provider and moved to another after friction with keys, model names, and audio formats. Because the pipeline is provider-agnostic — speech-to-text → LLM → text-to-speech — swapping services was a contained change rather than a rewrite.

Compromises & trade-offs.

Every fix above bought something and cost something. Naming the costs honestly is part of the engineering.

Shorter recording window

Recording length is capped by RAM so the secure handshake has room. This is the constraint driving the ESP32-S3 upgrade — more memory buys back longer questions.

Certificate validation disabled

The device skips TLS certificate checks to keep the handshake light. Acceptable for a hobby device on a home network; flagged as future hardening.

Status dot over a dedicated animation

A listening/thinking dot instead of a distinct thinking face — a deliberate choice for a continuous, natural character and lighter redraws.

Cloud-dependent intelligence

The AI needs connectivity and adds round-trip latency. Mitigated by the fully offline study mode, and by evaluating lower-latency services (fast LLM inference, dedicated speech) for the next revision.

What I learned.

Next steps.