Ambient AI scribe for clinical questionnaires, and the UX debate about going hands-free — Oct 1, 2026
Register to get the calendar and joining links for the weekly calls
One registration covers every upcoming Agents on FHIR call.
Loading your registration…
Don't have an account? Signing in creates one in seconds.
Architecture: Pydantic AI loop, six tools, and Jev as the prompt classifier
Olim walked through the ambient scribe as a passive listener attached to a FHIR Questionnaire renderer. The spine of the system is a Pydantic AI agent loop that calls six highly specialized tools against the renderer's state: read (compact outline of the form, with an expanded mode for a single field), field (set answers), navigate (walk across multi-page questionnaires), clarify (ask the doctor for the missing bit, like the time part of a datetime), validate (run the standard SDC validation), and submit. Every tool is also published as a Web MCP endpoint, so a doctor's own external agent can connect to the same page and drive it alongside the scribe.
Transcription is OpenAI's GPT Live, running near real-time with roughly a half-second initial delay, paired with voice activity detection that chops long speech into logical units and feeds them to the agent as they land. Between transcription and the agent sits a Jev decision model that classifies each incoming utterance as either a new follow-up prompt (queued as a fresh instruction) or a steering prompt that modifies whatever the agent is currently doing. That split is what makes mid-sentence corrections like "actually, remove nausea" work: the scribe can revise a field already being filled rather than queueing a contradictory second instruction.
We are gathering medical information and those should be extremely accurate.
Provenance ships for free. On submission the renderer captures the full transcript as a DocumentReference embedded in the QuestionnaireResponse and records which AI model and which agent produced the result.
Demo: filling a multi-page encounter form by voice
The pre-recorded demo ran against two linked questionnaires — an outpatient visit form and a follow-up arrangements form. A custom cursor UI, modelled on the browser-use pattern, moves around the page to show where the agent is reading or writing; a small badge next to it shows how many prompts are queued and how many are in flight.
The demo walked through the moves that tend to break naive scribes: the doctor's first correction ("Alex is 43, not 42") was classified as a steering prompt and overwrote the field already filled rather than appending a new value; vitals that lived on page two were filled by the agent navigating there on its own without being told; "remove nausea" and "pain is four now" were applied as deltas against existing state; "exactly seven days from today" and "next week in the morning" triggered the clarify tool to ask the doctor to pin down the follow-up date; and "please check both forms for missing required answers" ran the validate tool and reported back on the gap. When the doctor started talking about transport and wheelchair access, the agent correctly jumped from the outpatient form into the follow-up questionnaire without being told to switch.
The agent knows about all the fields, but we don't want to feed them in detail. The compact state describes the outline; when the agent wants to observe a field in detail, it calls read in the expanded mode.
The UX debate: form view, note view, or no screen at all
Matt pushed hard on the UX. The transcript overlay on top of the moving cursor is doing two different jobs at once — captioning speech and showing agent actions — and it ends up hard to read. He suggested splitting them: a dedicated speech strip, like movie subtitles, and the cursor layer kept purely for what the agent is doing.
I found the voice text thing a bit distracting. I would almost put that in a separate area that's just dedicated for speech, like a cartoon bubble. Then the cursor is showing you what's relevant and it's typing in. I would separate them. I wouldn't try and combine them.
He then pressed on the form paradigm itself. For a workflow a clinician does a hundred times a day, the form is a drag — the doctor already knows what a SOAP note looks like, and shouldn't need the structure on the screen to think it through. Matt's alternative was a blank note view that fills in as the doctor speaks, with the agent prompting back only if something required is missing.
I almost want to be able to do the whole thing without looking at the screen. The doctor dictates it already. So just be like "I want to add a prescription", and boom, there's just a prescription that gets added.
Olim argued the two modes are both legitimate — the passive listener who stays out of the doctor's way during the conversation, and the active-participation mode where the doctor is driving the form directly — and the UX has to be read against which mode a clinic actually wants. The team is already building a no-UI streaming variant of the scribe for the first case.
Nikolai agreed that voice-reply is the wrong direction (too slow, too costly, and physicians can read faster than they can listen), but kept pushing for a strong visual channel: a single-screen layout that fits without scrolling, so the doctor can glance at it hands-free. The group riffed on alternatives — a wall-mounted TV in the exam room, a small clicker with a few programmable buttons to signal "stop" or "pause", or just a well-placed microphone with low enough sensitivity to pick up only the near speaker.
Pushing more work down to Jev
Nikolai argued more of the loop belongs in Jev, not the main LLM. For navigation specifically, the agent has a known finite list of pages and widgets, which is exactly the shape Jev is built for; classification should take milliseconds, not a round-trip through Pydantic AI and GPT. Olim confirmed that path was already on the roadmap — navigation commands can skip the main agent entirely — and noted that OpenAI had recently released its own decision API on top of GPT-6 Luna, so the pattern is going mainstream.
Nikolai also flagged context compaction as something worth revisiting. Both OpenAI and Anthropic now ship server-side compaction APIs that are considerably faster than the naive "re-read the context and summarize" approach, and he wasn't sure Pydantic AI's current compaction path uses them. Olim had not hit context limits yet, but took the note.
The voice stack: GPT Live now, local Whisper on the roadmap
The scribe already ships with a client-side Whisper integration that's been in the renderer for about two years, running the small model via WebGPU directly in the browser; it works offline but is slow on long audio and needs manual stitching, so the current build defaulted to GPT Live for faster feedback. Nikolai asked whether Whisper Turbo could carry it on-device for machines with a decent GPU, and Matt extended the point further: a clinic buying a good machine once beats paying five cents a minute for cloud transcription forever.
If you're a clinic, you just get a good machine versus paying five cents a minute for Whisper or ChatGPT Live. It would make sense.
Multi-speaker diarization came up on the input side — Nvidia recently published a voice model that handles sixteen simultaneous speakers, Microsoft's Vibe Voice runs locally on a Mac at roughly half real-time, and Nikolai argued the simpler answer in a two-person exam room is a dedicated microphone per speaker rather than any model trick. On the output side, both agreed the economics of text-to-speech have flipped: Gemini's latest TTS, which is an LLM rather than a specialized model, now generates ten minutes of high-quality audio for cents, cheap enough that Nikolai is reconsidering voice replies in his own harness. The team is also tracking multimodal speech-to-speech models that collapse the pipeline, though the current OpenAI offering is still pricey and harder to engineer against because there is no clean hook between stages.
Matt's Ellie: ambient listening without structured records
Matt sketched out a different ambient-listening project he's working on for global health, called Ellie. Instead of building a FHIR record out of the conversation, Ellie uses a cheap 30-dollar digital recorder — the kind with a 256 GB card that captures four weeks of continuous audio — placed in a rural clinic, transcribes everything, and runs LLM queries against the transcript to extract operational and quality-of-care metrics.
Even if you just listen to everything that happens in a doctor's office, there's so much ambient knowledge you can extract from it.
The queries are the point: how many patients came in that day, how many malaria cases, were there any stockouts, did the clinician run the malaria test before prescribing antibiotics, how long was the average visit, did the clinic even open on time. It is supportive supervision via transcript — a very different design target from Olim's scribe, but one that lives in the same ambient-listening family and would get materially simpler the moment a model like Jev could handle the classification step.
Nikolai closed by connecting the two threads: the Formbox-renderer-plus-scribe build Olim showed and the ambient quality-signal build Matt sketched are both bets that the right path for AI in the exam room is passive observation plus deterministic structure around the LLM — not an agentic black box the clinician has to trust blind. The team is iterating on the UI, the Jev wiring, and the on-device voice stack in parallel; the no-UI streaming endpoint lands next. Nikolai pointed at the current renderer work as the open-source line to track — the Formbox library Matt had been using is where the population and extraction pieces are being added.