What Moving Self-Hosted Inference to a Server Will and Won't Fix
A migration plan, not a result: why moving RPOMS AI's model from a laptop to a dedicated server would improve availability but not accuracy, and the gates it must pass first.
The pull toward new hardware
This is for anyone running a model on a stopgap machine and feeling the pull toward "proper" hardware. RPOMS AI runs its vision model, Qwen2.5-VL 7B under Ollama, on a Windows laptop with integrated graphics. The site on Vercel reaches it through a token gateway and a quick tunnel. The intended permanent host is a dedicated server: an 8th-generation Core i5 with 32 GB of RAM, running Ubuntu Server, used for inference only.
To be explicit: this article is a plan. The migration has not started, and by the project's own readiness gates it is not ready to start. It is worth writing down anyway, because thinking it through changed what the move is expected to achieve.
What the move would improve: availability
The laptop has real availability problems. It must be awake with the launcher running. It is also used as a workstation. Its quick tunnel gets a new address on every restart, which then has to be pasted into Vercel. A machine that is always on, does nothing else, and holds a stable named tunnel fixes all of that.
CURRENT Vercel -> Cloudflare quick tunnel -> Windows laptop -> gateway -> Ollama -> Qwen2.5-VL -> rules
FUTURE Vercel -> secure tunnel/gateway -> Ubuntu server -> gateway -> Ollama -> Qwen2.5-VL -> rulesWhat it would not improve: accuracy
The model, the prompt and the rules would be identical, so what the model sees would be identical. Accuracy is the main problem today. On the thirteen reference photos the model's raw observations give 3 correct verdicts and 6 false accepts, and only the qualification gate keeps that from reaching anyone. Moving hosts is not a fix for "too many photos come back check manually". Saying so up front stops a hardware purchase from being sold internally as a quality improvement.
Speed cannot be predicted, only measured
On the laptop, a new photo takes a median of 41 seconds, and the integrated GPU is the bottleneck. It was more than half busy in 198 of 200 sampled seconds, while the CPU averaged 48%. With the GPU disabled the same photo took 119 seconds. The details are in Running Qwen2.5-VL 7B on an Intel Iris Xe.
So the server's speed depends mostly on whether Ollama can use its graphics at all, and on how fast its CPU is if not. Neither can be read off the laptop, and the plan deliberately gives no estimate. That matters because of a hard limit. The public check is one synchronous request inside a 60-second function, with 50 seconds allowed for the model. A host that needs longer than that per new photo would answer "AI unavailable" for every real inspection, however accurate it was. The 896 px cap exists for the same reason.
Four gates before migrating
- The pipeline is correct on real photos. The 896 px cap is deployed and a full-size phone photo is analysed within the timeout in production.
- There are real, verified photos to evaluate on. The thirteen references are low-resolution extracts from a spreadsheet, not enough to judge hardware or models.
- The server is benchmarked with the same tools on the same photos and meets the latency requirement: a new photo end to end in under 50 seconds, including one with a long answer.
- The model is re-qualified on the server. The same model and prompt produced different results on an Apple M4 and on the laptop's Vulkan backend, so a qualification does not transfer between machines.
The documented status is that gates 1 and 2 are not met and gate 3 has not been attempted. No GPU is to be bought before the server has been benchmarked.
If the server turns out too slow
The options are ranked by cost, and the decision is to be made after measuring.
| Option | What it buys | What it costs |
|---|---|---|
| Make the check asynchronous | Removes the 60-second ceiling. Speed becomes waiting, not failure | A queue, a status endpoint, and a polling page |
| A smaller model | Faster inference | Full re-qualification. It may be worse, and the 3B Qwen2.5-VL is under a non-commercial licence |
| Add a GPU | Raw speed | Money, justified only by a measured requirement |
The first option is the interesting one. It is the only change that makes the host's speed irrelevant to correctness. Whether to self-host at all is a related decision, discussed for other apps in Vercel or a Self-Hosted Ubuntu Server.
The laptop stays as the fallback until the server passes all four gates. Switching back is an environment variable change and a redeploy. The budget today is zero: Ollama is MIT licensed, Qwen2.5-VL 7B and cloudflared are Apache-2.0, and a named Cloudflare tunnel is free with a domain on Cloudflare. RPOMS AI has no model active in production and has never been used on real line photos. Everything above is preparation for when it is.
The work behind this
Related
Keep reading
Running Qwen2.5-VL 7B on an Intel Iris Xe With Ollama and Vulkan: 119 s to 41 s
Setting up a 7B vision model on a Windows laptop with no discrete GPU: enabling the Iris Xe under Ollama, where the time actually goes, and what the GPU profile showed.
Why Every Real Photo Timed Out: Capping Vision Input at 896 px to Fit a 60-Second Function
Full-size phone photos never got an answer from RPOMS AI's vision model. Encoding cost grew faster than pixel count; capping at the evaluated scale took 114 s down to 38 s.
Exposing a Local Ollama to a Vercel App: A 90-Line Token Gateway and a Cloudflare Quick Tunnel
Ollama has no authentication, so its port must never face the internet. The small Node gateway RPOMS AI puts in front of it, the tunnel, and how each hop is diagnosed.