Chapter 3 · Part 2

Muse's hands: connectors and a browser

A brain with no hands can only talk. Everything Muse actually does, like reading your inbox, messaging a friend or paying for a ticket, happens through a tool. A tool is a named action with defined inputs that the surrounding app knows how to carry out.

A personal agent has two very different kinds of hands. Scroll to see both.

Connectors: when a service has an API, the agent calls a typed function like email.search or pay.checkout.

scroll↓

Connectors: the fast, precise hands

A connector wraps a real service (email, WhatsApp, a calendar, a payment provider) as a set of tools the model can call. At launch Meta listed WhatsApp, email and shopping integrations, with payments through Stripe's Link and more payment and password-manager options on the way.

From the model's point of view a connector is just a menu of functions like email.search(query) or calendar.add(event). It writes a request such as "call email.search with phone bill", the app performs it, and the result comes back as text. That's fast, cheap and reliable, because nothing depends on the model reading a web page correctly.

Meta's engineering write-up adds one more piece: skills. These are written instructions that sit alongside a connector and teach the agent how to use it well, covering its quirks, the right order of steps and what to avoid. Tools tell the agent what's possible, and skills tell it how to do it well.

📌The same idea, as an open standard

Many agents now plug into tools through the Model Context Protocol (MCP), an open standard that lets any app expose tools to any AI. If you'd like to write a connector yourself, see Build an MCP Server.

Computer use: the universal hands

Most of the web has no API for agents. The phone company's "change plan" page, a small shop's checkout and a government form were all built for people with a mouse. So Muse can also open a browser and work it like a person:

  1. Take a screenshot of the page.
  2. Let the multimodal model look at it and work out where the "Select plan" button is.
  3. Emit an action such as click(x, y) or type("…").
  4. Take a new screenshot and check that it worked.

This is why Chapter 2's "natively multimodal" point matters. Meta describes Muse Spark as able to operate an app "by reading the pixels", including apps it was never specifically set up for. It's slower and more fragile than an API, since a pop-up or a redesigned page can throw it off, but it works almost anywhere a human could.

Hands mean risk

Each new tool makes the agent more useful, and each one is also something that could be misused. An agent that can read your email is handy. An agent that can send email as you, or pay with your card, needs guardrails. Muse lets you choose per app whether it may only read or may also act, and it checks with you before sensitive actions like sending an email or buying something. Part 3 covers how those checks are enforced so they can't be skipped.