Field Notes: A Week With a Laptop-Sized Open Model and No Internet Connection

The experiment had one rule. For seven days, every AI task I'd normally send to a cloud model would go to a small open-weights model running on my own laptop. I turned the Wi-Fi off whenever I used it, to make sure nothing was sneaking out.

The laptop is three years old, with 16 GB of memory. The model was a quantised open-weights model in the 7-to-8-billion-parameter range, meaning it had been compressed to fit in memory at some cost to quality. I ran it with one of the free desktop apps that make local models about as easy to install as a music player.

Here's the diary, lightly edited.

Days one and two: speed and small jobs

Day one was the speed check. It works, and it's slower than I'm used to, though not painfully so. On my machine it produced roughly 15 to 20 tokens a second, about the pace of a quick reader reading aloud. That's fine for a paragraph and tedious for a report.

The fan spun up immediately. My laptop sounded like it was considering a career in aviation.

On day two I threw my usual small jobs at it. Rewrite this email to be shorter. Turn these meeting notes into action items. Explain this error message. Write a regular expression to pull dates out of filenames.

It handled all of them well enough that I mostly forgot I was offline. The rewrite was a little stiff, the regex worked first time, and the action items were better than mine.

The week's biggest lesson arrived on day two: a surprising share of daily AI use doesn't need a frontier model. It needs a decent one that's always there.

Day three: the wall

Then I asked it to help compare two long vendor contracts. It started well, then lost track of which clause came from which document, and confidently merged them into a single imaginary contract.

In practice, small models hold less in mind at once, and their reasoning over many steps is shakier. I tried splitting the contracts into sections and comparing one pair at a time. That was better but slow, and I wouldn't trust the result without checking every line, which defeats the point.

Day four: knowing what it doesn't know

A small offline model has no idea what happened after its training data was collected, and it has no way to look anything up. Ask it about recent events and it either says so, which is good, or invents something plausible, which is not. Mine did both, depending on how I phrased the question.

I set a rule: nothing factual about the world goes to the local model unless I give it the source text myself.

Days five and six: privacy and power

Day five is where the experiment paid for itself. I had a folder of personal medical paperwork I'd been meaning to organise for months but didn't want to upload anywhere. The local model summarised each letter, pulled out appointment dates and flagged two follow-ups I'd missed.

Nothing left the laptop. For certain kinds of task, like health documents, finances, unreleased work or anything under an NDA, that isn't a nice-to-have. It's the whole reason to bother.

The cost showed up on day six. Running the model hard drained my battery about twice as fast as normal use. On a train, I rationed it like phone signal in the countryside. If you plan to work this way, plug in.

Day seven: the verdict

My scorecard after a week:

• Short rewriting, summarising and formatting: great, and indistinguishable from cloud for most of it.
• Code snippets and quick explanations: good, occasionally out of date.
• Long documents and multi-step reasoning: poor. Use a bigger model.
• Anything about current events: don't, unless you paste in the source.
• Private documents: excellent, and the main reason I'm keeping it installed.

I've settled on a split that I suspect will become normal for a lot of people: local for the private and the repetitive, cloud for the hard and the new. The small model now handles maybe a third of my daily AI tasks. It's like keeping a decent knife at home even though you sometimes eat at restaurants.

One caveat for anyone tempted to try this: results vary a lot with your hardware, the model you choose and how heavily it's compressed. Try two or three models before you judge the whole category.