The Short Version
Don't Teach the Model Your Business Logic Twice
The tempting first draft of "let AI log something for the user" is usually a standalone set of functions built just for the model: a bit of validation here, a database write there, close enough to what the rest of the app already does. It's also a quiet commitment to maintaining the same rules in two places forever. Every time the real logic changes, a new required field, a fixed edge case, a tightened validation rule, both copies need the same fix, in two different code paths, and they will eventually drift.
The more durable approach is to expose the exact same service layer the rest of the product already calls, and let the model be just another caller of it. Whatever already enforces "a workout needs a duration" or "an amount can't be negative" keeps enforcing it, whether the request came from a form submission or a model deciding to call a function. The assistant doesn't get its own parallel notion of what's valid; it gets the same one everything else already has to satisfy.
What MCP Actually Solves
This is the specific problem the Model Context Protocol (MCP) is built around. Instead of every product inventing its own bespoke bridge between a model and its data, MCP standardizes the shape of that bridge: a server exposes a set of named, typed tools ("look up X," "create Y"), and any MCP-aware client can discover and call them the same way, whether that client is a general-purpose assistant like Claude Desktop or Cursor, or one built directly into your own app. The tools themselves stay exactly what they always were, thin wrappers around logic that already existed and was already tested.
Once a server like that exists for one client, an in-app assistant doesn't need its own separate integration built from scratch, it can connect to the same server the same way any other client would. The AI provider changes, the UI changes, the underlying source of truth for what's actually allowed to happen does not.
Read vs. Write Is the Split That Matters
Not every tool call carries the same risk, and treating them identically is a mistake in both directions. A lookup, checking a balance, reading recent history, has no consequence if the model misunderstands the question; forcing a confirmation step on every read makes the assistant feel unusably slow for no safety benefit. An action that changes data is a different story entirely: get it wrong, and something real was logged, updated, or deleted on the user's behalf without them actually agreeing to it.
The useful, boring pattern is to draw that line explicitly and never blur it: read-only actions execute immediately, and anything that writes pauses first, shows the user in plain language exactly what's about to happen, and only proceeds on an explicit yes. That single split does most of the safety work in a conversational interface, cheaply, without needing a bespoke review flow for every individual action type.
A System Prompt Is a Suggestion, Not a Wall
It's tempting to scope an assistant with a system prompt, "only answer questions about X," and call the scoping problem solved. It helps, but it isn't a security boundary, because there's no reliable way to make a general-purpose language model permanently forget the rest of what it knows. A prompt is an instruction the model usually follows, not a technical constraint it's incapable of violating; a sufficiently unusual or adversarial phrasing can occasionally talk its way past one.
Treat that as an honest limitation rather than a hidden one, and design around it rather than past it. Pair the prompt-level scoping with backstops that don't depend on the model behaving: per-account rate limits so the blast radius of any single account's misuse stays bounded, the read/write split above so a scope violation still can't silently change data, and enough logging to notice a pattern of someone actually trying to push past the boundary, rather than finding out from a support ticket.
Photos and Other Media Cost More Than You Think
The moment an assistant accepts a photo or file, two separate questions show up that a text-only chatbot never has to answer: whether you actually need to keep it, and what it costs to keep sending it back to the model. On storage, the honest default is to ask whether the feature genuinely needs the original file preserved, or whether it was only ever useful for that one exchange; a lot of "just in case" storage turns out to be evidence for something that already has a perfectly good text record.
On cost, a vision-capable model bills real tokens for every image in a request, and a conversation that keeps resending the same image on every later turn (a naive way to preserve "context") ends up re-billing it again and again as the exchange grows. If a later turn genuinely needs to remember that a photo was involved, a short text summary of it is almost always enough; the raw bytes rarely need to survive past the message that actually used them.
Rate Limits Are Not Optional
Whatever an assistant is capable of doing, someone will eventually do it as many times as the interface allows, whether that's a curious user hammering a feature, a script testing what sticks, or an honest bug in a client retrying too aggressively. A per-account limit, not just a global one, is what keeps a single account's behavior from becoming everyone else's problem, and it's worth building in from the start rather than adding after the first surprising bill or the first abuse report. Expect to tune the actual numbers once real usage exists; getting the mechanism in place early is the part that's hard to retrofit later.
Frequently Asked Questions
What is the Model Context Protocol (MCP)?
MCP is an open standard for connecting an AI model to a set of tools and data sources, first popularized by Anthropic. Instead of every app inventing its own bespoke way to let a model call functions, an MCP server exposes a consistent list of named tools with typed inputs, and any MCP-aware client, Claude Desktop, Cursor, or an assistant built directly into a product, can discover and call them the same way.
Should an AI assistant be allowed to change data without confirmation?
Not if the change is real and user-visible. The safest pattern is splitting tools into read-only (safe to execute immediately, since nothing is lost if the model gets one wrong) and write (pause and show the user exactly what's about to happen, requiring an explicit confirmation before it executes). This turns a wrong guess into a moment a human can catch, instead of a silent mistake in someone's data.
Can a system prompt reliably restrict what a chatbot will talk about or do?
Only partially. A system prompt is a behavioral instruction, not a technical wall, there's no way to make a general-purpose language model permanently forget its underlying knowledge, so a sufficiently determined or oddly-phrased request can sometimes get past a prompt-only restriction. Treat it as the first layer, not the only one: pair it with real backstops like rate limits, the read/write split above, and monitoring for repeated boundary-testing.
What should an app do with photos or files sent to an AI assistant?
Decide deliberately rather than by default. Every image sent to a vision-capable model costs real tokens, and if it's resent as part of a growing conversation history on every later turn, that same image gets re-billed again and again. Decide up front whether you actually need to keep the file at all, and if a conversation needs to remember that a photo existed, a short text note is usually enough, you rarely need the original bytes hanging around in every later request.
Where This Leaves You
None of this is really about picking a model or building a chat UI, both of those are the easy, well-documented part by now. The decisions that actually determine whether an AI feature is safe and sustainable are the plumbing underneath it: whether it's calling the same trusted logic the rest of the product already relies on, whether a real change always gets a human's eyes on it first, and whether the safety net holds even in the case where the model itself doesn't behave the way the prompt asked it to. Health Partner's own in-app assistant follows exactly this shape, reusing existing, already-tested logic rather than a parallel copy of it, pausing for confirmation before anything is actually logged or changed, and treating a photo as something to look at once, not something to keep. Download Health Partner to see it in practice.
References
- Anthropic. Model Context Protocol specification. modelcontextprotocol.io. modelcontextprotocol.io
- OWASP. OWASP Top 10 for Large Language Model Applications. owasp.org. owasp.org/www-project-top-10-for-large-language-model-applications
Also on the blog
Blocking an app sounds simple, but iPhone and Android solve it completely differently. Here's exactly how Health Partner's walk-to-unlock Screen Guard works on each platform.
Is Your Journal Really Private? What Real End-to-End Encryption Actually Looks Like"Your data is encrypted" often means less than it sounds. Here's how to check if it actually protects you, using real end-to-end encrypted journaling as the worked example.