Beta 9 is out, and it has to be downloaded by hand. I'll explain why in a moment, because the reason turns out to be good news rather than bad.

First, the strange part. The features page lists the same 47 features it listed the day beta 8 shipped in June. Nothing added. Nothing removed. Nothing moved from beta to shipped or the other way. If you only read that page, you'd conclude two months went by and nothing happened.

Underneath it, 121 planning documents were built, shipped, and folded into the docs they belong to, and the tool catalog grew by ninety-six. Every feature in the product got a performance and effectiveness pass. That's the release. We didn't widen the shelf, we spent two months making everything already on it faster, more reliable, and honest about what it actually does.

Beta 9 has to be downloaded, and that's the good part

Automatic updates can't deliver beta 9. Here's the mechanism, because it's more interesting than the inconvenience.

Beta 8 embeds Sparkle 2.9.1, which sits below a cache-path fix that landed in 2.9.3. Every update beta 8 attempts dies on the first try, and on every retry after that, forever. The copy of Sparkle that performs an update is the one compiled into the app doing the updating, not the one in the app being installed. So beta 9 fixes every update from here forward and structurally cannot fix the hop onto itself. There's no version of this where the old build repairs itself.

What makes that worth writing about instead of apologizing for is what we did with the release it forced. If you're going to make the entire installed base download something once, that build had better be the last one that ever needs it.

So beta 9 carries the escape hatches. Updates now stagger across rollout groups, so a bad build can't reach everyone inside one check window. Installs are consent gated, and that's a gate in code rather than a convention. The license system holds a ring of eight signing keys instead of one, because with a single key a compromise had no remedy at all. And the client now decodes server responses tolerantly, so we can add a subscription status or a tier or a field and nothing in the field falls over trying to parse it.

That last one is the whole point. The reason a shipped app ever needs a forced reinstall is that it met something it couldn't understand and had no way to say so. Beta 9 is built so the next problem is a server-side change instead of a download. One manual install, and this class of problem is behind us.

Create anything by talking to it

This is the part of the release I'm most proud of, and it's the piece that took the most work.

The idea is simple to state and was not simple to build. The conversation is the command layer. You describe what you want, the model calls typed verbs against a typed artifact, and the result renders in the chat you asked from. Image, then video, then a 3D object, then a whole navigable 3D scene, each one able to become the input to the next. Same conversation, same graph, no export step in between.

We wrote our own engines

One MLX-Swift framework hosts five first-class media engines: image, video, 3D object and mesh, Gaussian splat, and text. Three further cores run on the same host, including SAM 3 segmentation and on-device text to speech.

Five model architectures are implemented from scratch in Swift: FLUX.2 Klein, SD3.5 Medium, Wan2.1, TripoSplat, and AnySplat.

It's worth being specific about what that isn't, because the shortcuts were all available and we didn't take them. It isn't MetalSplatter, OpenSplat or gsplat wrapped in a Mac UI. It isn't a Core ML conversion. It isn't a Python process hiding behind a local port. It isn't a cloud API with a native front end. It's the architectures rebuilt in MLX-Swift, running on Apple Silicon, in the same process as everything else ToolPiper does.

We ship architecture and never weights. Models come from pinned Hugging Face repositories at pinned revisions, under their own licenses, downloaded by you. That's a deliberate call and it has a visible consequence: generation fails closed with a weightsUnavailable error until you bring a model, rather than us shipping someone else's checkpoint inside our binary.

Both image and video are verified end to end on real weights, by hand, on a real machine. FLUX.2 Klein produces correct images. Wan2.1 produced a coherent, temporally consistent 480 by 480 clip at 33 frames and 25 steps in about six and a half minutes, holding roughly 24 GB on a 32 GB M2 Max. Those are measured numbers from the machine this was built on, not projections.

The Photoshop work, driven by conversation

Twenty-nine new image verbs landed, and every one of them is free. Tonal work through curves, levels, adjustments, threshold and posterize. Geometry through crop, rotate, straighten, perspective, free transform, resize and flip. Compositing through masked blending, region erase and fill. Generative work through inpaint, outpaint and image-to-image editing. Plus text, gradients, import, and neural upscaling.

Layers and history aren't a feature bolted on top. They're the asset graph. Every edit mints a new asset carrying a derivedFrom edge back to its parent and a record of what made it, which means the edit history is the data model. Non-destructive by construction rather than by anyone remembering to be careful.

Seventeen more verbs give you the working document itself: layers, masks, layer transforms, adjustment layers, flatten, merge down, reorder. The same document format serves the image side and the video side, so a project isn't stuck in one of them.

The result is that "remove the background, brighten the subject, and extend the canvas to the left" is one sentence in chat, and it runs the exact same verbs the editor UI calls. Not a scripting layer bolted onto an app. The app and the agent reach the same functions.

Selection and masks

Segmentation is the region authority for everything above. image_segment mints one mask asset per detected region, fully on device. image_cutout composites a mask back against its source to give you the subject on transparency. image_mask_refine is the classical selection toolkit, applied non-destructively so it produces a new derived mask instead of editing one in place. image_inpaint regenerates only the masked area. video_segment does the same across frames and tracks each region as it moves.

Select, refine, cut out, fill, extend. That's the layer loop you'd run in Photoshop or GIMP, expressed as verbs a model can call in order.

Two backends serve it, and the honest description matters here. There's an Apple Vision floor that's free, always present, and needs no setup. And there's SAM 3, roughly 3.44 GB, gated behind the SAM license and a Hugging Face token, which you can turn on. SAM 3 is genuinely strong at prompted selection, and text and box prompting are both proven. But Apple Vision remains our recommended default, and I want to be plain about why: we have not run the benchmark that would justify promoting a gated 3.4 GB model over a free one. When we've run it, we'll say what it showed.

3D and Gaussian splats

image_to_splat turns a single image into a navigable 3D object. image_to_world turns one image, a set of frames, or an entire video into a pose-free 3D scene. Both produce a Gaussian cloud that renders inline in chat through a WebGL2 viewer, and in Studio.

The workflow is the one the whole release is built around. Drop a file into chat, the model calls image_to_world, and the scene renders in the conversation. No pipeline to configure, no intermediate file to hand somewhere else.

A real video, keyframe sampled, becomes a scene of 1,834,189 Gaussians plus an equirectangular environment lighting probe, end to end in about three minutes. Ask for the identical thing again and it serves cached bytes in zero seconds. That used to take over twelve minutes and not converge, and a lighting-probe level of detail fixed it.

Both splat engines have been through a full operator sign-off against real checkpoints: real-weight binds with no missing or phantom keys, golden parity against our own CPU reference rasterizer, live single-image and live video runs, and replay. Every row green. In a beta, most things are taken on faith, and I'd rather point at the handful that aren't.

Computer use that acts, and tells you whether it worked

We pushed hard on this one, with Microsoft Copilot Vision as the thing to measure against, and I think we're ahead. But the difference isn't a scoreboard, it's a category, so let me describe it accurately.

Copilot Vision is guidance. Microsoft's own support page says it "will not click, enter text, or scroll on your behalf" and that it's "there to answer questions rather than take actions." That's not a gap they're hiding, it's the product working as designed, and for a person learning an unfamiliar app by voice the hand-holding flow is polished and free. Microsoft's agents that do act are separate products, and they're gated: the Windows one is an off-by-default Insider preview running sandboxed, the Edge one is browser only, the Studio one is enterprise with per-step credits.

On the Mac there's no documented Copilot Vision capability at all. The Mac Copilot app does chat, voice and image upload.

ToolPiper sees and acts, closed loop, on the machine. What that means concretely:

  • Every action tells you whether it landed. Each one returns a verification block with the dispatch path it used, whether it was a no-op, how many nodes changed, and how long it took. A click that didn't do anything reports noOp: true instead of success.
  • Assertions are real. The assert verbs read accessibility state and check it, and a false assertion inside a batch halts the batch rather than quietly passing.
  • It doesn't take over your desktop. As of this release, coordinate-tier clicks and typing route directly to the target process. No cursor movement, no focus change, no window raised. You can keep working in another app while it drives. Every result discloses how it was delivered.
  • It clicks the right way. When a match has a real accessibility identity, it's pressed through an element handle. Only a text-recognition-only match gets clicked at a pixel. It used to click everything at a pixel, which cost occlusion and focus failures on targets that had a better path available all along.
  • Safety gates sit on the acting path. Prompt-injection scanning, confirmation for risky steps, a secret-field probe that redacts credentials from the recorder, durable undo, and a cascade you can cancel mid-flight with the Escape key.
  • Grounding never leaves the device. The cloud path isn't switched off, it's deleted, and a CI check fails the build if anyone reintroduces it.

And you can measure whether it's working for you rather than taking our word for it. A telemetry summary endpoint buckets executions by action, selector and domain, and ranks your apps by accessibility miss rate. A live event stream lets you watch every dispatch as it happens. The share of actions resolving without escalating is effectively a per-app score for how well automation works on your machine.

Two things to be straight about. Reading small text on your screen with a vision model works today through a cloud model, and no local model has cleared our bar yet. We've published which ones we tried and exactly why each failed. And clicking icons that carry no text label is gated until we finish blessing a grounding model, so until then those targets fail with a typed error naming the reason rather than guessing at a location.

Browser automation

Twenty-five browser tools, and they're in good shape. Recording, accessibility-native selector resolution, hierarchical selectors, local self-healing, mutation diffing, drive mode, and test export are all live, along with network logging, performance tracing, request mocking, coverage, storage management, WebAuthn, autofill, device emulation and page-to-PDF.

Six browsers are supported: Chrome Dev, Canary and Stable, Brave, Brave Nightly, and Chromium. Automation targets the channel's real default profile, not a throwaway one, because testing against a clean profile tests something you don't actually use.

Verification semantics match the native side. No-op detection, actionability waits, honest self-healing, and dialogs that aren't silently auto-accepted. Three connection verbs collapsed into one this release.

Tools that adapt to your model in real time

With a catalog this size, sending every tool to every model stopped being sensible a long time ago. Schemas are context the model pays for before you've typed anything, decision quality degrades when there's too much to choose between, and a small local model may have less context than the full catalog would consume.

So which tools get sent is now decided per turn, server side, based on what surface you're working in. Chat and code run in retrieval mode: a semantic match returns the most relevant tools under a budget derived from measured per-tool token cost rather than an estimate, and the model can ask for more if it needs them. Two new verbs, tool_search and tool_docs, let an agent look for a capability instead of being handed everything up front.

Underneath that sits the schema layer, which has been quietly doing the unglamorous work for a while. Every provider implements a different subset of JSON Schema for tool calling. Gemini rejects keywords OpenAI requires. OpenAI strict mode requires keywords Gemini rejects. Ollama has trouble with keywords the cloud providers handle fine. We write each tool schema once in a minimal safe subset and transform it per provider at dispatch. If a provider still refuses, chat retries without tools and keeps working instead of failing. You never see any of it, which is the point.

The engine underneath

ToolPiper runs GGUF models on an embedded upstream llama-server, build b10485, upstream's current release. Not a fork, not a reimplementation, not a wrapper around someone else's runner.

Three engine bumps landed since beta 8, roughly 950 upstream commits of work carried into the product in two months.

One decision from the last bump is worth sharing, because it's the kind of thing most products don't explain. A newer build tag was available and we passed on it. The commit we pinned was simultaneously upstream's newest annotated release and carried the build identity our public pages already publish, where every page naming an engine build derives it from a single constant. Taking the newer untagged commit would have meant either breaking that derivation or letting the site quietly disagree with itself about what it runs. We took about ninety minutes of drift instead. The pin is chosen for the identity it carries downstream, not for being newest.

Autopilot

Agentic loops are otherwise hand-built. You write a continuation prompt, open a fresh chat, paste the state back in, run a step, judge the result, and repeat. The context doing the work is also the context grading it, the original goal decays as the conversation grows, and nothing guarantees it ever stops.

Autopilot turns that into one tool group. You state a goal once, either a phased document or a one-off prompt, and each iteration spawns clean from a small carryover plus a queryable history. Resident context stays flat, and the check for whether the goal is met runs in a separate fresh context that sees only the goal and the artifact.

It holds no model credentials of its own. That's an invariant, not a current limitation.

The scheduler

ToolPiper could already do a great many things on demand. The scheduler adds the missing axis, which is doing them later or on a repeating clock, with nothing in the cloud.

A schedule can fire any MCP tool with JSON arguments, a native agent prompt, an autopilot run, or an arbitrary local command. The case that prompted it is specific: Claude Code ships its own scheduling, but it runs in Anthropic's cloud against an isolated checkout, which is useless for work that lives on your machine. Here, "tonight, launch a fresh Claude Code session on this repo with this task" is a first-class action against your real files.

Nothing about it touches the network. Local clock, local file, local process. The scheduler is the one new group in this release behind a paid tier.

Security, across everything

A lot of this release went into work nobody asks for by name. The MCP security architecture, loopback trust, host safety risk gating, credential unification, a sweep of every endpoint that reads a bearer token, and keychain migration for cloud keys all landed.

The one worth calling out is the tool audit trail. Every tool this machine runs is recorded locally, with who ran it and whether it was allowed. Free for everyone, always on, and never sent anywhere. On a team deployment it additionally attributes each record to a person and drains to the organization's own collector, which is theirs and not ours.

Teams

Server-enforced entitlements, an attributed audit log, and marketplace allowlists, so an organization can administer ToolPiper instances across its people rather than each machine being an island. This is in beta and I'd rather say so than imply otherwise.

Upcoming

Here's what's next, and in each case what's already built versus what's left, because "coming soon" on its own means nothing.

Animating an image. The video engine already generates coherent video from text on device, verified on real weights. What's missing is image conditioning. The verb is registered and the engine is proven, so this is a wire rather than a research problem.

Longer video. The pipeline is proven end to end at short clip length with byte-exact weight acquisition at a pinned revision. The full-length pinned clip and parity pinning are what remain.

Moving splats. Every piece of the static 3D Gaussian stack that 4D needs is built and signed off. What's left is a quality bar we haven't cleared, and our own review of the field says so. We're not shipping motion until it's actually good, and I'd rather tell you that than put a date on it.

Reading your screen with a local model. The whole delivery chain works and a cloud model answers screen questions accurately today. No local model has cleared our bar. We've published each candidate and the specific reason it failed, because a list of failures is more useful to you than a promise.

Icon-only click targets. The grounding seam, model pin, circuit breaker and benchmark harness are complete, and both halves of the win-gate already passed on our test battery. What's left is annotating a final set of fixtures and one re-run before we bless the pin.

Beta 10 is benchmarking. Real measured numbers for real models on real Apple Silicon. It's the one thing in this space that can't be copied by writing about it, because you have to actually run the models.

PiperMesh. Use compute from another ToolPiper instance, or offer yours. Contributing is free and consuming is paid. The transport is live inside ToolPiper today with nobody on it yet, and the add-on surface is still behind a flag.

This is beta, and these are first versions

Everything new in beta 9 is at its first beta version. Not a polished third revision with the edges sanded off. The first one that works well enough to put in your hands.

Concretely, and I'd rather list these than write a generic disclaimer: icon-only click targets fail until grounding is blessed. No local vision model reads small screen text well enough yet. Animating an image isn't wired. Video generation is proven at short clip length, not full length. SAM 3 is opt-in and hasn't been benchmarked against the free default.

What I'd ask you to notice is how those failures behave. A click that doesn't land says so. A missing model returns a named error instead of a wrong answer. A target that can't be found fails with a reason rather than clicking somewhere hopeful. That's a design commitment running through the entire release, and it's the thing I'd defend hardest. A beta that fails loudly is more useful than a 1.0 that fails quietly.

So tell me what breaks. Which apps miss. Which model worked. Which verb you reached for and couldn't find. That's the whole reason this is in your hands at this stage.

A note from me

One person builds this.

Five media engines written from scratch in Swift. An embedded llama.cpp tracked to the current release. Over 420 tools, and 358 of them are free, including every generative verb in this article: image, video, 3D objects, 3D scenes, segmentation. All of it running on the machine in front of you, with no account required to try it.

I think this is the best AI tooling on the Mac, and most of it costs nothing. Download it and judge that yourself.