Drop a photo into a chat and ask for a 3D scene. About three minutes later it is spinning in that same conversation, 1,834,189 Gaussians of it, rendered on your Mac and lit by an environment probe the same pass produced.
That's beta 9. The conversation became the command layer, and what it commands is five generative engines we wrote ourselves.
It's the biggest release ToolPiper has shipped. Ninety-six new tools since beta 8, taking the catalog from 325 to 421. Three engine bumps carrying roughly 950 upstream commits. Every feature already in the product got a performance and effectiveness pass, and Studio and Max both got more work than in any release before this one.
Five engines, written from scratch
One MLX-Swift framework hosts five first-class media engines. Image, video, 3D object and mesh, Gaussian splat, and text. Three further cores run on the same host, including SAM 3 segmentation and on-device text to speech.
Five model architectures are implemented from scratch in Swift. FLUX.2 Klein and SD3.5 Medium for images, Wan2.1 for video, TripoSplat and AnySplat for Gaussian splats.
The shortcuts were all available and we took none of them. This isn't MetalSplatter, OpenSplat or gsplat wrapped in a Mac UI. Not a Core ML conversion. Not a Python process hiding behind a local port. Not a cloud API with a native front end. The architectures are rebuilt in MLX-Swift, running on Apple Silicon, in the same process as everything else ToolPiper does.
We ship architecture and never weights. Models come from pinned Hugging Face repositories at pinned revisions, under their own licenses, downloaded by you. Generation fails closed with a weightsUnavailable error until you bring a model, rather than us shipping someone else's checkpoint inside our binary.
Both image and video are verified end to end on real weights, by hand, on a real machine. FLUX.2 Klein produces correct images. Wan2.1 produced a coherent, temporally consistent 480 by 480 clip at 33 frames and 25 steps in about six and a half minutes, holding roughly 24 GB on a 32 GB M2 Max. Those are measured numbers from the machine this was built on.
Twenty-nine image verbs, all free in beta 9
Tonal work through curves, levels, adjustments, threshold and posterize. Geometry through crop, rotate, straighten, perspective, free transform, resize and flip. Compositing through masked blending, region erase and fill. Generative work through inpaint, outpaint and image-to-image editing. Text, gradients, import and neural upscaling on top of that.
Layers and history aren't a feature bolted on afterward. They're the asset graph. Every edit mints a new asset carrying a derivedFrom edge back to its parent and a record of what made it, so the edit history is the data model. Non-destructive by construction rather than by anyone remembering to be careful.
Seventeen more verbs give you the working document itself. Layers, masks, layer transforms, adjustment layers, flatten, merge down, reorder. One document format serves the image side and the video side, so a project isn't stuck in either one.
"Remove the background, brighten the subject, and extend the canvas to the left" is one sentence in chat, and it runs the exact same verbs the editor UI calls. The app and the agent reach the same functions.
Selection, masks and splats
Segmentation is the region authority for all of it. image_segment mints one mask asset per detected region, fully on device. image_cutout composites a mask back against its source. image_mask_refine is the classical selection toolkit, applied non-destructively so it produces a new derived mask instead of editing one in place. image_inpaint regenerates only the masked area, and video_segment does the same across frames, tracking each region as it moves.
Two backends serve it. An Apple Vision floor that ships in the app, always present and needs no setup, and SAM 3 at roughly 3.44 GB behind the SAM license and a Hugging Face token, which you can turn on. SAM 3 is strong at prompted selection, and text and box prompting are both proven. Apple Vision stays the recommended default until we run the benchmark that would justify promoting a gated 3.4 GB model over the built-in one.
On the 3D side, image_to_splat turns a single image into a navigable object. image_to_world turns one image, a set of frames, or an entire video into a pose-free scene. Both render inline in chat through a WebGL2 viewer, and in Studio.
That scene at the top of this article took about three minutes from a real video, keyframe sampled. Ask for the identical thing again and it serves cached bytes in zero seconds. The same job used to run past twelve minutes without converging, and a lighting-probe level of detail fixed it.
Both splat engines went through a full operator sign-off against real checkpoints. Real-weight binds with no missing or phantom keys, golden parity against our own CPU reference rasterizer, live single-image and live video runs, and replay. Every row green.
Everything a model runner does, free
ToolPiper embeds the upstream llama-server binary directly, build b10881, the same engine llama.cpp ships. Not a fork, not a reimplementation, not a wrapper around someone else's runner. Three bumps landed since beta 8, roughly 950 upstream commits of work carried into the product in two months.
Your models stay ordinary GGUF files you can point any tool at, not sha256-named blobs behind a manifest. Download, load and switch them from the app. No terminal, no account, no caps.
Speed is parity, and we measured it rather than asserting it. Same models, same quantizations, same Metal-accelerated GGML lineage, within single digits in both directions with the winner flipping by model. Pick a runner on storage, interface and direction, not on tokens per second.
Where it stops being a comparison is the surface. Ollama, LM Studio and Jan all run models locally for free, and Jan is Apache 2.0, which is a stronger auditability story than ToolPiper has. All three are MCP clients. They consume tools. ToolPiper serves them, 421 of them in beta 9, and 358 were free in that build, including every generative verb in this article. Since the beta 10 retier, 181 of the 440 tools are free and generation sits on Studio. They also run on Windows and Linux, and we don't.
Everything a model runner does is free here. The paid tiers buy things no model runner sells at any price.
Automation that tells you whether it worked
Twenty-three computer-use tools and twenty-five browser tools, all free in beta 9 (Pro since the beta 10 retier, with connecting and snapshots still free), and every one of them reports back.
Each action returns a verification block naming the dispatch path it used, whether it was a no-op, how many nodes changed, and how long it took. A click that did nothing reports noOp: true instead of success. Assert verbs read accessibility state and check it, and a false assertion inside a batch halts the batch rather than quietly passing.
Coordinate-tier clicks and typing now route straight to the target process. No cursor movement, no focus change, no window raised, so you can keep working in another app while it drives. When a match has a real accessibility identity it gets pressed through an element handle, and only a text-recognition-only match gets clicked at a pixel. It used to click everything at a pixel, which cost occlusion and focus failures on targets that had a better path available all along.
Safety gates sit on the acting path. Prompt-injection scanning, confirmation for risky steps, a secret-field probe that redacts credentials from the recorder, durable undo, and a cascade you can cancel mid-flight with the Escape key. Grounding never leaves the device. The cloud path isn't switched off, it's deleted, and a CI check fails the build if anyone reintroduces it.
Two other things that watch a screen
Microsoft Copilot Vision answers questions about what's on your screen. Microsoft's own support page says it "will not click, enter text, or scroll on your behalf" and that it is "there to answer questions rather than take actions." That's the product working as designed, and for someone learning an unfamiliar app by voice the guidance flow is polished and free. The Microsoft agents that do act are separate and gated, and on the Mac there's no documented Vision capability at all.
Anthropic ships computer use in two layers. Claude Cowork and Claude Code can drive your own Mac's screen, in research preview for Pro and Max subscribers since March 2026. The developer toolset on Anthropic's API drives an environment you host instead, which their reference implementation puts in a Docker container running a virtual Linux desktop. Either way, Anthropic ranks screen control last. Its Claude Code docs say "Computer use is the broadest and slowest, so Claude tries the most precise tool first," starting with an MCP server when one exists.
The mechanism underneath is screenshots in, coordinates out, and Anthropic names the cost of that plainly. Their docs list "Computer vision accuracy and reliability" as a limitation, saying Claude "might make mistakes or hallucinate when outputting specific coordinates while generating actions." A pure pixel loop holds no identity for the thing it clicked. That is why ToolPiper resolves an accessibility identity first and falls back to a pixel only when the match came from text recognition alone, and it's the same reason icon-only targets are gated here rather than guessed at.
None of which makes ToolPiper a rival to Anthropic. Claude Code and Claude Desktop are two of the clients that call these tools, and the scheduler's flagship case launches a Claude Code session on your own repo. The two fit together. When Claude needs to act on a Mac, it reaches for an MCP server before the screen, and ToolPiper is an MCP server built for the Mac, with a verification block on every action.
Browser automation matches those semantics. Recording, accessibility-native selector resolution, hierarchical selectors, local self-healing, mutation diffing, drive mode and test export, plus network logging, performance tracing, request mocking, coverage, storage management, WebAuthn, autofill, device emulation and page-to-PDF. Six browsers are supported, and automation targets the channel's real default profile, because testing against a clean profile tests something you don't actually use.
You can measure whether it's working on your machine rather than taking our word for it. A telemetry summary endpoint buckets executions by action, selector and domain and ranks your apps by accessibility miss rate. A live event stream shows every dispatch as it happens.
With a catalog this size, sending every tool to every model stopped making sense a long time ago. Which tools get sent is now decided per turn, server side, from the surface you're working in. Chat and code run in retrieval mode, where a semantic match returns the most relevant tools under a budget derived from measured per-tool token cost rather than an estimate. Two new verbs, tool_search and tool_docs, let an agent go looking for a capability instead of being handed everything up front.
Every provider implements a different subset of JSON Schema for tool calling. Gemini rejects keywords OpenAI requires. OpenAI strict mode requires keywords Gemini rejects. We write each schema once in a minimal safe subset and transform it per provider at dispatch, and if a provider still refuses, chat retries without tools and keeps working. You never see any of it.
Other MCP servers land in the same catalog
ToolPiper has served MCP tools since early beta. Beta 9 makes it a host as well, so it consumes other MCP servers and re-exposes their tools beside its own, under one permission model and one audit trail. Install a server from the registry and its verbs appear in chat as mcp__server__tool, callable exactly the way everything else is.
VisionPiper is the first one. It's a separate menu bar app, free on the Mac App Store, and it runs a control-plane MCP server of its own on 127.0.0.1:9999 with fifteen typed tools. Steer the capture region, grab a screenshot, capture only if the screen changed, start and stop a recording, list recordings, export GIF or WebP, read stream state.
Federated in, those fifteen become ToolPiper verbs. "Watch this region and tell me when the build finishes" is one sentence, and the model holds the vision tools and the 421 native ones in the same turn. Screen capture stops being a second app you alt-tab to and becomes a capability the agent already has.
The two apps compose without either importing the other. Media comes back as saved file paths rather than streamed bytes, with one size-capped inline PNG for preview, and the frame stream stays on VisionPiper's own data plane where ScreenCaptureKit and VideoToolbox do the work. Permission seeding auto-allows the two read-only status tools and pins every other one to an explicit ask.
Three invariants keep an upstream from degrading the local server. Each server gets a call timeout of its own, so a slow one can't hold a call open past its ceiling. Stale registrations are evicted on a 24-hour clock. Wire namespaces are sanitized, so one server's name can't forge another's.
Where Studio and Max went
Studio at $29 took the heaviest work in this release. The video pipeline is a real timeline now, with composition, narration, screenplay and rehearsal verbs, keyframes, render ranges and file export, alongside storyboard assembly, pose capture and upscaling.
Max at $49 gained the scheduler, the one new group in this release behind a paid tier. A schedule can fire any MCP tool with JSON arguments, a native agent prompt, an autopilot run, or an arbitrary local command. Nothing about it touches the network. Local clock, local file, local process.
The case that prompted it is specific. Claude Code ships its own scheduling, but it runs in Anthropic's cloud against an isolated checkout, which is useless for work that lives on your machine. Here, "tonight, launch a fresh Claude Code session on this repo with this task" is a first-class action against your real files.
Autopilot landed free in beta 9. Agentic loops are otherwise hand-built, where you write a continuation prompt, open a fresh chat, paste the state back in, run a step, judge the result and repeat. The context doing the work is also the context grading it, and nothing guarantees it ever stops. State a goal once instead, either a phased document or a one-off prompt, and each iteration spawns clean from a small carryover plus a queryable history. Resident context stays flat, and the check for whether the goal is met runs in a separate fresh context that sees only the goal and the artifact. It holds no model credentials of its own, and that's an invariant rather than a current limitation.
Teams at $99 brings server-enforced entitlements, an attributed audit log and marketplace allowlists, so an organization can administer ToolPiper across its people rather than each machine being an island. It's in beta.
The tool audit trail is free for everyone and always on. Every tool this machine runs is recorded locally, with who ran it and whether it was allowed, and never sent anywhere. On a team deployment it additionally attributes each record to a person and drains to the organization's own collector, which is theirs and not ours.
Two more are close. Beta 10 is benchmarking, real measured numbers for real models on real Apple Silicon, which is the one thing in this space nobody can copy by writing about it. PiperMesh lets you use compute from another ToolPiper instance or offer your own, free to contribute and paid to consume, and the transport is live inside ToolPiper today with nobody on it yet.
One download, and then never again
Beta 9 has to be installed by hand, once.
Beta 8 embeds Sparkle 2.9.1, below a cache-path fix that landed in 2.9.3. The copy of Sparkle that performs an update is the one compiled into the app doing the updating, not the one in the app being installed. Every update beta 8 attempts dies on the first try and on every retry after that. Beta 9 fixes every update from here forward and structurally cannot fix the hop onto itself.
So we spent the release making sure it's the last build that ever needs this. Updates stagger across rollout groups, so a bad build can't reach everyone inside one check window. Installs are consent gated, in code rather than by convention. The license system holds a ring of eight signing keys instead of one, because with a single key a compromise had no remedy at all. And the client decodes server responses tolerantly, so we can add a subscription status or a tier or a field and nothing in the field falls over trying to parse it.
A shipped app needs a forced reinstall when it meets something it can't understand and has no way to say so. Beta 9 is built so the next problem is a server-side change instead of a download.
What isn't done yet
Everything new here is at its first beta version.
Animating an image isn't wired yet, though the video engine is proven on real weights and the verb is registered, which makes it a wire rather than a research problem. Video generation is proven at short clip length, not full length. Moving splats are blocked on a quality bar we haven't cleared, and we're not shipping motion until it's good. No local vision model reads small on-screen text well enough to clear our bar, so that path uses a cloud model today, and we've published each candidate we tried and why it failed. Icon-only click targets stay gated until we bless a grounding model, and until then they fail with a typed error naming the reason. SAM 3 is opt-in and hasn't been benchmarked against the Apple Vision default.
Those failures all behave the same way. A click that doesn't land says so. A missing model returns a named error instead of a wrong answer. A target that can't be found fails with a reason rather than clicking somewhere hopeful. That runs through the entire release.
So tell us what breaks. Which apps miss, which model worked, which verb you reached for and couldn't find.
A note from me
One person builds this.
Five media engines written from scratch in Swift. An embedded llama.cpp tracked to the current release. 440 tools, 179 of them free since the beta 10 retier; in beta 9 every generative verb in this article shipped free. Image, video, 3D objects, 3D scenes, segmentation, all running on the machine in front of you, with no account required to try it.
I think this is the best AI tooling on the Mac, and most of it costs nothing. Download it and judge that yourself.