Running a Small Language Model on My 2018 Laptop Was Slower and Better Than I Expected

I have a business laptop from 2018 with 16 gigabytes of RAM and no dedicated graphics card. It is my personal machine, separate from my work laptop. Over a long weekend in August I tried to see whether I could run a small open-weight language model on it entirely offline, and whether it would be good for anything.

The short answer is yes, slowly, for a narrow set of tasks.

Getting it running

I used a popular open-source tool for running quantized models locally. Quantized means the model's numbers are stored with less precision, which makes it much smaller and a bit less accurate. I tried three models in the range of 3 to 8 billion parameters.

The 8-billion-parameter model, quantized to 4 bits, took about 5 gigabytes of RAM and generated roughly 4 tokens per second. That is painfully slow for chatting, about the speed of someone typing carefully.

A 3-billion-parameter model ran at about 8 tokens per second and used under 3 gigabytes. That one became my daily driver for the experiment.

Setup took about 40 minutes, most of which was downloading.

What it was good at

I gave it the kind of boring jobs I would never send to a paid service because they felt too trivial.

• Renaming files. I had 600 scanned receipts with names like scan_0412. I fed it the OCR text of each and asked for a short filename with date and shop. It got around 90 percent right.
• Summarizing my own notes. I keep messy daily notes. It could turn a week of them into five bullet points reasonably well.
• Rewriting commit messages for a personal project into a consistent style.
• Explaining shell commands I found in old scripts.

For these, slowness did not matter. I let it run while I made dinner.

What it was bad at

It was weak at anything requiring broad knowledge or careful reasoning. Questions about specific libraries were often confidently wrong. Longer code generation drifted into nonsense after about 40 lines. It also struggled with Polish, my first language, producing grammatically awkward sentences that read like a translation of a translation.

Why I kept it

The thing I did not expect to value was privacy. Those receipts include addresses and card endings. My notes include things about my family. I would not paste them into a hosted service, but I was happy to let a model on my own disk read them with the network cable unplugged.

There is also no bill. The electricity cost for a weekend of experiments was too small to measure on my meter.

I would not recommend this setup to anyone who needs speed or accuracy. But for batch jobs on private data where good enough is fine, an old laptop and a small model are surprisingly capable. I have since set it up on my partner's old laptop too, so she can sort her own photo captions without uploading them anywhere.