The 21-Second Benchmark That Was Wrong: Ollama Reuses an Encoded Image

This note is for anyone measuring how long a local vision model takes per image, especially under Ollama. The published measurement for RPOMS AI once said a warm analysis took about 21 seconds on an Intel Iris Xe laptop, and that image processing took 264 milliseconds. Both figures were real readings. Neither described a real inspection.

Both came from sending the same image twice. The first request paid for everything. The second found that Ollama had reused the recently processed prompt, image included, so it skipped about 27 seconds of image encoding and only paid for generation. Every real inspection is a new photo, so no real inspection would ever see the 21-second figure.

A dedicated benchmark on 22 September 2026, with a new photo per request and nothing else running on the machine, gave the figures now in the documentation. The full breakdown is in Running Qwen2.5-VL 7B on an Intel Iris Xe.

The last row stays in the table on purpose, labelled as a cache hit. It is a useful number to know. If a benchmark result lands near it, the benchmark is probably measuring the cache.

The trap turned up a second time, in a less obvious place. The repeatable evaluation on 22 September ran all thirteen reference photos through the production pipeline. Seven of them had been sent to the model minutes earlier, during the performance benchmark, so they skipped most of the encoding. One took 6 seconds. On top of that, the project's typecheck and lint were running on the same machine during the evaluation and competing for the CPU.

I did not edit the report to hide this. The generated report is kept as produced, with a "Read this first" section at the top. It says the verdicts are reliable and some of the timings are not, lists the two distortions, and points to the dedicated benchmark as the authoritative source for latency. The verdicts were unaffected: the model runs at temperature 0, and they matched the earlier run exactly.

Earlier in the project the model ran on an Apple M4, and a figure of about 5 seconds per photo was reported there. It was never re-measured with new images. So the documentation now says plainly that it may carry the same effect. It is not quoted as the M4's real speed.

• Send an image the server has never seen in every timed request. Varying the prompt is not enough if the image repeats.
• Separate cold from warm. Loading the model after it was unloaded cost 6.2 s on this machine, which is a different thing from encoding cost.
• Run nothing else on the machine. Builds, linters and browser tabs all compete for the same CPU and GPU.
• Report a range, not just a mean. The 82-second worst case came from an unusually long answer, and it is the case that hits the timeout.
• Keep corrections visible. The documentation says what the old figure was and why it was wrong, so nobody rediscovers it from an old commit and trusts it.

The broader habit is the one behind the whole project, described in A Vision Model That Only Observes: record what was measured, including the mistakes, because the next person will otherwise repeat them.