Local AI or cloud: Which architecture fits?

Draw the data path first

Start with the inputs. Where do documents, search queries and retrieved passages go? Which services create indexes or numerical text representations? Where are responses, logs and backups stored? Include identity services, monitoring and connected tools. Each transfer needs a purpose and an appropriate access rule.

NIST’s zero trust architecture does not infer trust solely from network location. That principle applies here: a locally reachable service still needs authentication and authorisation. “Local” describes a deployment location. Privacy and compliance with specific requirements also depend on configuration, processes and actual use.

Sources: NIST · Zero Trust Architecture

Local operations need an owner

Own infrastructure deserves consideration when a workflow must function without an external model connection or strict transfer boundaries apply. It also calls for an operating plan: who updates the system, manages keys and handles outages? Who controls storage, backups and retention? Those responsibilities persist even when the model software is straightforward to install.

A local runtime’s name is insufficient evidence. Ollama, for example, documents a separate mode that disables cloud models and web search. For a specific installation, those settings and surrounding applications must be assessed together. Test realistic document lengths and concurrent users as well. A model fitting in memory does not establish that it can meet the required response time under load.

Sources: Ollama · FAQ

Make cloud commitments specific

For a cloud offering, the exact service, configuration and agreed terms matter. Microsoft’s Foundry documentation distinguishes processing locations across deployment types and storage introduced by stateful features. A statement about model training therefore does not settle which content is stored or processed to operate the service.

For your workflow, data owners and operations teams should agree which features may be used, who receives access and how retention and deletion work. Include troubleshooting and support: what content could become visible in those processes? Record the options actually selected. A product name or chosen region cannot replace that description. Changes to features or settings should trigger another assessment.

Sources: Microsoft · Data, privacy, and security for Foundry Models

Build hybrid systems around a testable boundary

A hybrid architecture can prepare data locally and send selected tasks to an external service. That boundary must specify content. If local search passes confidential passages to a cloud model, those passages have left the local environment. A summary may still contain sensitive information. Define which data may cross and what happens when classification is uncertain.

Then compare permitted options using the same reference workflow: task quality, response time, behaviour during failure, ongoing effort and the ability to switch providers or infrastructure. A short operating trial should also cover lost connectivity or an unavailable service. The appropriate architecture is one whose capabilities and limits your team can demonstrate and sustain in everyday use.