Local AI for Small Business: Running Your Own Models with Ollama

Most businesses use AI the way they use electricity: they rent it, per token, from somebody else's data center. It works — until the price changes, the terms change, the model they built their workflow around gets deprecated, or a client asks the uncomfortable question: where exactly does our data go?

There's a second option that's gotten genuinely good while nobody was watching: running open-weight language models on your own hardware. And the tool that made it a one-evening project instead of a research degree is Ollama.

What Ollama actually is

Ollama is a small, free, open-source program that runs large language models on your own machines — Mac, Windows, Linux, a spare server. It handles the hard parts (quantized model weights, GPU and memory management, an OpenAI-compatible API) and wraps them in two commands:

ollama pull gpt-oss:20b — download the model.
ollama run gpt-oss:20b — chat with it. It's now serving on localhost:11434, and anything that speaks the OpenAI API format can talk to it.

That's the entire on-ramp. No API keys, no usage dashboard, no billing alert. The model runs on your silicon, answers at whatever speed your hardware allows, and stops when you close the lid.

What it takes: honest hardware math

• 8 GB RAM laptop: small models (3–4B parameters, e.g. Llama 3.2 3B). Drafting emails, summarizing notes. Slower, but real.
• 16 GB Mac or desktop: the sweet spot. gpt-oss:20b runs comfortably at 128K context and is genuinely good at everyday business work.
• 24–32 GB with a decent GPU: 27–32B class models (Gemma 3 27B, Qwen3 32B) — the quality ceiling for most local setups.
• 64 GB+ Mac (unified memory) or a used datacenter GPU: 70B-class and beyond. This is where "local" starts matching rented models on substance.

The pattern: you're trading a monthly bill for a one-time hardware purchase. A $2,000 machine amortized over three years is about $55/month — and it can serve every employee in the office simultaneously without asking anyone's permission.

The pros — why this is worth doing

• Your data never leaves the building. Client contracts, financials, employee records, the draft email about the customer who's suing you — none of it crosses the internet. For a lot of small businesses this alone settles the build-vs-rent question: it's the difference between "AI we can use on anything" and "AI we can use on the safe stuff."
• No meter running. Local inference costs zero per token. Experiment freely. Let the automation hammer the model at 3am. Run the 40-step workflow that would cost real money per run on a metered API. Usage-based pricing quietly punishes exactly the experimentation that makes AI valuable; owning the model deletes that tax.
• Vendor risk disappears. No terms-of-service update repurposing your data. No model deprecation notice breaking a workflow you shipped to production. No price restructure. The model you validated this month is the model still running next year — because it's yours.
• It keeps working. Internet down, provider having an outage, region blocked, free tier throttled — your AI doesn't notice. For anything embedded in daily operations, that reliability is a feature you can't buy.
• Open-weight models got good. This is the part that changed. A 20B open model in 2026 does work that required a frontier API in 2023. The gap between "best closed model" and "best open model" on ordinary business tasks is now small enough that privacy and cost routinely win the trade.

The cons — because the honest version sells itself

• Hardware has a ceiling. The biggest open models want 100 GB+ of memory. On a laptop you're running the compact tiers, and on hard reasoning or long-document analysis you'll feel it.
• You're the ops team now. A model on your server needs updating, monitoring, and securing like everything else on your server. Nobody's on-call for you at 2am. This is the cost small businesses actually underestimate — not the hardware, the responsibility.
• Speed follows the hardware. A datacenter serving thousands of users has GPUs you don't. On a modest machine, expect a calm trickle instead of a firehose.
• Setup is easy; tuning is not. Pulling the model is one command. Picking the right model, quant, and context length for your workload — and knowing why — takes a weekend of reading or one good consultant.

Ollama Cloud: the pressure-release valve

Here's the part that makes this a complete strategy instead of a hardware lottery. Ollama now offers Ollama Cloud: the same models you can't fit locally — the 120B-class and larger — running on Ollama's rented GPUs, delivered through the exact same interface.

The command is nearly identical (ollama run gpt-oss:120b-cloud), the API your tools already speak doesn't change, and the model name gets a -cloud suffix. Locally-developed code, scripts, and automations keep working. You're renting the GPU, not the platform.

The terms matter for the same people who care about local in the first place: Ollama runs cloud requests on dedicated compute in the US and Europe with a stated policy of zero data retention — prompts aren't logged and they don't train on your data. Pricing is simple tiers: a free tier for light use, Pro at $20/month, Max at $100/month — a fraction of what equivalent frontier API usage runs, with no token-by-token surprises.

Where it earns its keep: burst capacity for the heavy job that would choke a local box, frontier-class quality when a task genuinely needs a 120B model, and the try-before-you-buy path — run the big model in the cloud for a month and let the results tell you whether the hardware upgrade is justified.

Be clear-eyed about the trade: cloud models are still somebody else's computers. Zero-retention policies are far better than consumer AI tools' terms, but if a task is truly sensitive, local is the only version with no exceptions.

The hybrid play

This isn't local versus cloud — it's a routing decision, and Ollama is the only tool that makes both sides speak the same language:

• Local by default for anything private or high-frequency: client data, internal docs, the automation that runs all day. Fixed cost, zero data egress.
• Cloud for the heavy lifting: the deep-reasoning job, the 400-page contract analysis, the month you're prototyping something new.

And because the interface is identical, the flip costs nothing. If a cloud plan changes or a bill spikes, you swap the model tag and the workload moves on-premise that afternoon. Try pulling that off with a proprietary API.

The entry point is as low as it gets: install Ollama on a machine you already own and run a 4B model tonight, or spend $20 on a month of cloud and see what a 120B model does with your actual work. Either way you learn the same lesson — AI that lives on your side of the network changes what you're willing to use it for.