Building Over 450 MCP Tools in Swift: Architecture of a Unified Local AI Server

The Model Context Protocol (MCP) is becoming the standard interface between AI assistants and external tools. Most MCP servers are single-purpose: one for browser automation, one for database access, one for file management. ToolPiper takes a different approach - it ships over 450 MCP tools spanning LLM inference, text-to-speech, speech-to-text, embeddings, OCR, vision, pose estimation, browser automation, RAG, and real-time stream processing. All from a single native macOS application, written entirely in Swift.

This paper describes the architecture decisions behind building a unified MCP server at this scale, the transport design that makes it work with every current AI client, and the tool design principles we developed along the way.

Why One Server Instead of Many

The local AI MCP ecosystem is fragmented. A developer who wants local LLM inference, speech-to-text, and browser automation currently needs three separate MCP servers - typically three Node.js processes, three sets of dependencies, three configuration entries. The largest competitor in the space has roughly 139 GitHub stars. No existing server covers more than two of the capability categories ToolPiper provides.

Unification isn't just a convenience play. It enables workflows that fragmented servers can't support. A single MCP tool call can transcribe an audio file, pass the text to a local LLM for summarization, and speak the summary - all through one server process that shares model state, memory budgets, and authentication. There's no inter-process serialization, no conflicting port allocations, no version drift between components.

One Transport: Streamable HTTP

MCP defines two standard transports: stdio, where the client spawns the server as a child process, and Streamable HTTP. ToolPiper started with both. The first version shipped a small Swift command-line bridge that read JSON-RPC from stdin and forwarded it to the running app. Once every current MCP client could speak Streamable HTTP, the bridge was only an extra process with its own authentication to maintain, so it was retired. Streamable HTTP is now the only transport, and a client configures it once:

claude mcp add --transport http toolpiper http://127.0.0.1:9998/mcp

Streamable HTTP: Zero Extra Processes

ToolPiper serves the MCP protocol directly from the running app - no CLI middleman, no extra process:

AI Client
    ↓ Streamable HTTP (POST localhost:9998/mcp)
ToolPiper.app (MCP server built-in)
    ↓
Backends (llama.cpp, FluidAudio, MLX Audio, Apple Intelligence, Vision, etc.)

MCPRoutes.swift implements the JSON-RPC protocol over HTTP. Sessions are tracked via Mcp-Session-Id headers, and a known session that expired while a client sat idle is revived on its next call instead of failing. Every tools/call first enters the invocation pipeline (ToolDispatchService), which handles admission, tier, permission, firewall, consent and the call record. It then reaches its handler through ToolRegistry. No tool call reaches a REST route.

Single Source of Truth: Shared Definitions and Handlers

We keep a strict separation between tool definitions, handler logic, and transport mechanics:

• Definitions - the PiperToolDefinitions package, which compiles without the app: MCPToolDefinitions.swift plus one MCPToolDefs+<Category>.swift file per area (Core, Browser, Testing, System and so on). Each definition carries the tool name, description, typed input with its generated JSON Schema, its tier and its annotations. Alongside the tools, the server declares five resources and five prompts.
• Handlers - each tool has one registered ToolHandler, grouped by area in *ToolHandlers.swift files and registered in ToolRegistry.production. A handler decodes a typed input and calls the feature's service. A REST route, where one exists, is a sibling that calls the same service.

Adding a new tool means a definition, a handler and its registry entry. The catalog then feeds tools/list, the tier check and tool retrieval from the same source.

Tool Design: Coarse Tools, Not REST Mirrors

ToolPiper's REST API is fine-grained, and exposing it one endpoint per tool would overwhelm any AI agent. The catalog is organized into 14 categories. A representative slice:

The guiding principle: a good REST API is not a good MCP server. REST endpoints are fine-grained by design - separate calls for connect, navigate, click, type. MCP tools should be coarse-grained, wrapping entire flows that an AI agent would naturally perform as a single step.

For example, the browser_action tool accepts a selector and an action type (click, fill, select, hover, scroll, etc.) in a single call. The REST API underneath involves selector resolution, element location, input dispatch, and AX tree diffing - but the AI sees one tool with one clear purpose.

Output Formatting: Semantic Text, Not JSON

Every MCP tool returns semantic plain text, not raw JSON. This follows the pattern established by Playwright MCP and reflects a key insight: AI models process natural language better than nested JSON structures.

• Confirmations are terse: "Done.", "Enabled.", "Deleted." - not {"success": true}
• Accessibility trees render naturally: real newlines and indentation, not JSON-escaped \n
• Wrappers are unwrapped: tools that call endpoints returning {"text": "..."} extract the inner value
• Diffs use visual prefixes: + for added, - for removed, ~ for changed AX nodes
• Structured data falls back to pretty JSON: model lists and storage dumps use indented, multi-line JSON when the data is inherently structured

Why Swift-Native

Most MCP servers are written in TypeScript/Node.js. We chose Swift for several reasons:

• No runtime dependency. Users don't need Node.js, npm, or npx installed. The MCP server is part of the app they already run.
• Shared types with the host app. ToolPiper is a Swift macOS app. The MCP server shares its type definitions, JSON codable structs, and build system. When we add a field to a model config, both the app and the MCP server see it immediately - there's no TypeScript↔Swift translation layer to maintain.
• Performance. JSON-RPC parsing runs with no garbage collection pauses and predictable memory usage, and tool execution is in-process, so a call does not cross a process boundary.
• Swift has MCP support. There is an official Swift MCP SDK, so Swift is a first-class language in the MCP ecosystem.

Paid Tools Answer With a Sentence

208 of ToolPiper's 475 tools are free and need no account; the rest belong to the Pro, Studio and Max tiers. Every tool is listed on every tier, so an agent can find the right one. When an agent calls a tool above the user's tier, the call returns a short plain-text sentence naming the tier and the upgrade link instead of a bare error. MCP hosts relay that sentence to the user, and a host that supports MCP elicitation can also show it as a native prompt. The sentence is the floor on every host, because we control the tool-result bytes but not how a host renders them.

Annotations Where They Change Behavior

MCP tool annotations help AI clients decide how to treat a tool. We send the two that change what a host does, and each one only when it is true, because MCP reads a missing hint as false:

• readOnlyHint - on tools that only read state, such as system_status or browser_snapshot
• destructiveHint - on the minority of tools the catalog marks destructive, and on any tool the user has set to Ask, so the host asks for confirmation

idempotentHint and openWorldHint stay off the wire to keep the tool list small.

Resources: Read-Only State

In addition to tools, the MCP server exposes five resources for AI clients that support the MCP resources spec:

• status - server health, version, loaded models and system resources
• models - the model catalog with availability and categories
• backends - which inference engines are running and their capabilities
• tests - saved PiperTests (Max tier)
• categories - tool categories, counts and the profiles for filtering tools/list

Resources are read-only snapshots. They complement tools by giving AI clients ambient context about system state without requiring explicit tool calls.

How Is Access Secured?

With over 450 tools - including a system_* layer that can run shell commands and drive the desktop - access control is not an afterthought. The model has two halves: frictionless on-machine, scoped off-machine.

Local clients connect over loopback with zero configuration. On launch, ToolPiper mints a short-lived ambient bearer token at ~/Library/Application Support/ToolPiper/.toolpiper-token (mode 0600, rotated every launch) and trusts loopback connections at the socket level. Claude Code, Cursor and other local MCP clients just work - there's no key to copy into a config file.

Anything off-machine - a LAN device or a PiperMesh peer - is held to a higher bar: it must present a scoped Bearer tp_<64 hex> token with the correct audience. Those tokens are 64 hex characters of CSPRNG entropy, and they're never written into ~/.claude/ or any client config you don't control.

Two surfaces inside ToolPiper make this auditable. The Connected Apps pane lists every authenticated client as a row you can label and inspect for its MCP-tools access state; one click revokes the bearer immediately, and the row stays visible marked as revoked - state is never hidden. The tool governance overlay is restrict-only: deny a specific tool such as system_run_command for every client, or flip the built-in tool marketplace from open browse to an explicit allow-list of servers and tools. It can only narrow what a tier already permits, never widen it.

Governance today is a per-device, locally-edited overlay - not yet an enterprise SSO/admin-console product. The policy shape is built so an organization can push it later, at which point the controls lock with a "Managed by your organization" badge. For now: allow/block tool governance, per-device today, org-managed push coming.

What We Learned

Building this MCP tool catalog taught us several things that aren't obvious from the MCP specification alone:

1. Fewer tools are better. We started with plans for 60+ tools mirroring the REST API. Agents performed worse with more tools - they spent tokens reasoning about which tool to use instead of using the right one. Careful curation, plus profile-based filtering, improved task completion rates.
2. Descriptions matter more than schemas. The natural-language tool description drives AI tool selection more than the JSON Schema does. We write descriptions that say when to use the tool, not just what it does. "Run a prompt through a local LLM on this machine via ToolPiper" tells the AI this is for local inference, not cloud API calls.
3. Plain text output outperforms JSON. When tools returned raw JSON, AI models spent tokens parsing and explaining the structure. Plain text responses let the model incorporate results directly into its reasoning.
4. The shared-definition pattern scales. One catalog, split into per-category definition files with one registered handler per tool, has carried the server past 450 tools. Adding a tool means a definition, a handler and its registry entry.

ToolPiper's MCP server is available today. Install ToolPiper from modelpiper.com and configure your AI client with claude mcp add --transport http toolpiper http://127.0.0.1:9998/mcp.

This is a technical paper in the local-first AI on macOS series. For a deep dive into the browser automation tools, see AX-Native Browser Automation.