Local AI vs Cloud AI for App Features

Local AI should be the default consideration when an app feature transforms data the user already has on their device. That includes summarizing an article being read, extracting action items from notes, classifying a document, rewriting selected text, and normalizing messy input. A cloud model can still be necessary for some work, but sending every AI-shaped feature through an API adds a network dependency, account and billing requirements, vendor availability concerns, rate limits, and data-handling questions to a feature that may have had none.

The case is laid out clearly in "Local AI Needs to be the Norm", using an iOS client for The Brutalist Report as the concrete example. Its optional article summaries use Apple's local model APIs. The article text is already on the phone because the user is reading it, and the summary is generated there too. No separate prompt-log policy, server request, or vendor account is required for that path.

That is a useful boundary for deciding between local AI and cloud AI: whether the job actually requires a model outside the device.

When should an app use local AI?

Use local AI when the model is performing a bounded transformation on user-owned data already present on the device, and when the output can tolerate the capability limits of the available local model.

The source article names five common transformations:

• Summarize
• Classify
• Extract
• Rewrite
• Normalize

These are narrower jobs than open-ended question answering. A model summarizing an article does not need broad world knowledge if the source text is supplied in the prompt. A model extracting action items from a note is working from the note. A document categorizer needs to assign a label or set of labels from provided content.

This distinction matters because the data path is simpler. A typical cloud-backed feature has several moving parts:

There is still engineering work in local inference: devices vary, a local model can be unavailable, long inputs need a deliberate processing strategy, and output needs validation and presentation. Those are application concerns rather than remote-service concerns, and they should be treated as such.

If a feature needs the kind of intelligence only available from a cloud-hosted model, using the cloud is a reasonable technical decision. If the feature is a local data transformation and the available model can handle it, routing the content to a server creates a dependency the product did not need.

Why does cloud AI change the product architecture?

A remote model call turns the feature into a distributed system:

user input → application backend → model API → application backend → user interface

The feature must account for network conditions, external vendor uptime, rate limits, account billing, and the application's own backend health. Each of those conditions can prevent the user-facing result from appearing.

The privacy boundary changes at the same time. Once user content leaves the device for an AI provider, the application needs answers to practical questions: what content is sent, how long it is retained, who can access it, whether it is used for training, how consent works, what audit record exists, and what happens during a breach or government request. A privacy policy can explain those decisions, but it cannot remove the underlying transfer.

For features involving email, notes, documents, or reading activity, that transfer is often the feature's most consequential design decision. The content is frequently personal or proprietary, and the user may reasonably interpret an in-app summary or extraction tool as operating on the material they selected rather than forwarding it to another service.

Local execution makes a stronger promise because the architecture supports it: the input stays where it started.

How do Apple's local model APIs support on-device summaries?

The source article describes an Apple-platform flow built with FoundationModels. It starts by importing the framework, obtaining the system model, and checking its availability before creating a session.

import FoundationModels

let model = SystemLanguageModel.default

guard model.availability == .available else {
    return
}

let session = LanguageModelSession {
    """
    Provide a brutalist, information-dense summary in Markdown format.
    - Use **bold** for key concepts.
    - Use bullet points for facts.
    - No fluff. Just facts.
    """
}

let response = try await session.respond(
    options: .init(maximumResponseTokens: 1_000)
) {
    articleText
}

let markdown = response.content

The mechanics here are straightforward. The application supplies the article text and instructions for the desired output. The model returns content that can be shown in the interface. The code checks whether the system model is available, which should be a normal branch in the product rather than an exceptional failure. If local intelligence is optional, the app can omit the control or explain that the feature is unavailable.

For a news-reading client, the summary request is narrowly scoped:

1. The text comes from the article the user opened.
2. The requested output is a concise summary.
3. The summary has an explicit token cap of 1,000.
4. The application can render the result as Markdown.

The model is operating over a supplied document, which makes on-device summarization a good local-AI candidate. The article's input and output are both bounded. The quality bar is also understandable: a reader needs a useful summary of a page rather than an answer assembled from the internet.

How should local AI handle long documents?

A long article should not be pushed through as though input length were unlimited. The source proposes a two-pass chunking flow for longer content:

1. Convert the content to plain text.
2. Split it into chunks of roughly 10,000 characters.
3. Produce concise, facts-only notes for each chunk.
4. Run a second pass that combines those notes into a final summary.

This is a practical map-reduce pattern for document processing. The first pass reduces each chunk into a smaller intermediate representation. The second pass reads those representations rather than the full original document.

document
  ↓
~10k-character chunks
  ↓
facts-only notes for each chunk
  ↓
final synthesis prompt
  ↓
summary

The intermediate notes deserve product attention. If they are vague, repetitive, or full of editorial language, the second pass has poor material to work from. A narrow first-pass instruction such as "extract facts and key claims" gives the synthesis step a cleaner input than "summarize this however you think best."

Chunking also exposes a limitation that should be explicit in the interface. A local summary of a long document is a generated aid and does not replace reading the source. The product can make that clear through its design: preserve the original document, link summary bullets back to the relevant section where possible, and avoid presenting generated text as an authoritative record.

Why are typed outputs better than asking for JSON?

Generated Markdown can work for a summary view. It becomes fragile when application logic needs individual fields, arrays, labels, or values. The common alternative has been prompting for JSON and attempting to parse it afterward. The source article argues for a stronger pattern in Apple's tooling: define the Swift type the application needs, describe the fields, and ask the model to generate that type.

import FoundationModels

@Generable
struct ArticleIntel {
    @Guide(description: "One sentence. No hype.")
    var tldr: String

    @Guide(description: "3–7 bullets. Facts only.")
    var bullets: [String]

    @Guide(description: "Comma-separated keywords.")
    var keywords: [String]
}

let session = LanguageModelSession()

let response = try await session.respond(
    to: "Extract structured notes from the article.",
    generating: ArticleIntel.self
) {
    articleText
}

let intel = response.content

This matters because the type becomes the contract between the model and the user interface. tldr, bullets, and keywords have defined places in the application. The UI does not need to scrape headings from Markdown or guess whether the model preserved a JSON schema.

The field guides also show where prompt design belongs. "One sentence. No hype." is a constraint for tldr. "3–7 bullets. Facts only." constrains the list. Those instructions are tied to the data shape the interface expects, which is easier to reason about than a large prompt that asks for a polished text block containing several hidden structures.

Typed generation does not remove the need for defensive handling. Empty fields, unexpectedly broad keywords, and low-quality summaries remain possible. It does make the failure surface more legible. The application receives a defined object and can enforce display limits, omit missing fields, or retry with a narrower instruction.

What local AI cannot replace

Local models have limits, and treating them as replacements for every cloud model produces bad product decisions. The source makes this distinction directly: some use cases demand intelligence available only from a cloud-hosted model.

That is especially relevant when a feature depends on knowledge beyond the supplied input, requires stronger reasoning across a complex task, or needs capabilities the local platform model does not provide. A local model summarizing an article is operating over the article. A feature that needs to search and reason over the broader world has a different requirement.

The right comparison is therefore task-specific:

A product should make that routing decision feature by feature. "AI everywhere" does not describe a useful architecture. An on-device model used as a typed, bounded data-transforming subsystem can keep private content local and eliminate a remote dependency. When the task exceeds that boundary, a cloud call should be a deliberate exception with clear handling for the data it receives.