Workout Director

Workout Director is an iPhone and iPad training app, published on the App Store by Shelgaon Ltd. I wrote the SwiftUI app and its TypeScript backend on AWS Lambda, and I ran the TestFlight and App Store releases. It has been on the App Store since October 2026.

The problem

Training on your own, you either count reps in your head or stop to tap a phone. The idea was a phone propped up across the room that counts for you and talks you through the session, without filming you. That last part set the main constraint: the camera feed never leaves the phone. Video is never recorded, saved or uploaded.

Counting reps on the device

Pose tracking runs on the phone with Google MediaPipe, and a rep counter sits on top of the pose stream. It counts squats, push-ups and more from the camera. Planks and other holds are timed sets instead.

MediaPipe's pose model comes in three sizes, so I measured them before picking one. On 156 single-person photos from the COCO validation set, Lite placed 72% of joints within 5% of the person's size, Full 79% and Heavy 84%. On 14 exercise clips, Full's joints jumped half as much from frame to frame as Lite's. Heavy took about three times as long per frame as Full and added 21 MB to the app, which wasn't worth it for live use, so the app ships Full. If MediaPipe can't start, or runs below 12 frames a second, that workout switches to Apple's Vision body pose without resetting the count.

Most of the work was not counting things that aren't reps: half reps, walking, standing up and getting into position. Around the counter, the app gets the person in frame first with a short countdown and prompts such as Move back or Step into frame, and it always leaves a manual path: minus and plus are one tap away, and you can correct the count by voice.

A voice coach that asks before it acts

The voice coach is the only feature that needs the network during a set. The phone opens a WebRTC call to a realtime voice model. The model never sees video: it gets short text notes about the workout state over the data channel. The provider key stays inside AWS; the backend checks the user's access and limits and mints a short-lived client secret for each call, and the app falls back to on-device speech recognition and a text reply if the realtime service is unavailable.

The model doesn't change anything itself. When you say "rest for 90 seconds", it replies with a tool call, and the phone runs it against its own workout state and shows a small chip with Undo. The tools are the things you would otherwise do by hand: log a set, correct the reps, start, extend or skip a rest, swap or skip an exercise, add or remove a set, pause, and end the workout. Swapping an exercise, dropping the remaining sets or changing the rest applies at once with Undo. Adding load waits for a Confirm, and ending the workout waits for a spoken yes.

Trainer chat

The trainer chat answers questions in text. Each message goes through the backend to a text model, with the context it needs: the person's profile, facts it has been asked to remember, the training log from the last seven days and the last 24 messages. Older messages are folded into a running summary, so the prompt stays short as a conversation grows.

The model has to reply in a fixed JSON shape: one to three messages, plus up to six things to remember or forget. Messages are rate limited and count against a per-user budget, and a message the phone sends twice, for example after a dropped connection, gets the stored reply instead of a second model call.

The backend

The backend is TypeScript on AWS Lambda behind API Gateway and CloudFront, defined with the AWS CDK. It is split into functions by job: profile, sessions and sync; the trainer and voice tokens; sign-in; operations; and subscription webhooks. App data is in DynamoDB with point-in-time recovery, and sign-in data is in Postgres on Neon.

Phones in a gym are often offline, so uploads take an idempotency key, which means a retried upload doesn't create a duplicate workout, and sync does a last-write-wins merge so the same log can follow you to another device.

Releasing safely

A wrong count is the first thing a user notices, so counting changes shipped behind switches. Camera counting had its own feature flag, internal builds had a pose debug overlay, and individual behaviours had kill switches served from a public config endpoint, so a behaviour could be turned off without waiting for App Review. New counting models ran in shadow mode first, alongside the live counter, without changing the count anyone saw.

Releases followed the same steps every time: a backend diff that stops if it would replace or delete a stateful resource, a health check after each deploy, and an app release that goes through TestFlight, a preflight checklist and a phased rollout. A minimum-version check let the backend ask old builds to update.

Screens