Cloud vs. Local

Layers under testCOMPUTE · MODELS

The test

Take one real workflow and run it two ways — hosted and local — with identical inputs, identical evaluation, and identical accounting.

The primary comparison will use the same open-weight model, revision, quantisation, serving runtime, prompt, context limit, and decoding settings on a local machine and on a rented cloud GPU. That isolates deployment location more cleanly than comparing unrelated models. A hosted frontier API may appear as a separate reference, but it will not be presented as the controlled half of the same experiment.

Workload lock

The workload is published before any measured run. It will contain at least 100 items drawn from one repeatable production-shaped task, divided into a visible development set and a held-out scoring set. Inputs are replayed in the same order, and outputs are stored before scoring.

Both deployments receive the same warm-up sequence. Measurements are taken at one request, at moderate concurrency, and at the highest concurrency that still meets the declared service-level target. Network time is measured from the operator’s location rather than removed from the cloud result.

What gets measured

• Cost per unit of finished work, not cost per token.

• Latency and throughput under realistic concurrency.

• Output quality, scored the same way on both sides.

• The operational overhead each option actually carries.

Accounting

Cloud cost includes instance or API charges, storage, data transfer, required managed services, and idle time during the run. Local cost includes purchase price, financing if any, measured electricity, storage, replacement allowance, and operator time. Capital cost is reported under multiple utilisation and service-life assumptions instead of being hidden inside one break-even number.

Quality is scored blind to deployment. Deterministic checks are used wherever the task permits them; human or model-assisted grading must use a published rubric and retain the underlying outputs for audit.

Reproduction and limits

The publication packet will contain the model and runtime versions, container or install manifest, hardware configuration, cloud region and instance type, request logs with sensitive content removed, raw timings, power readings, outputs, and scoring code.

The conclusion will apply only to the tested model, hardware, region, workload, and date. It will not claim that “local” or “cloud” wins in general.

Status

Protocol v0.1 · planned. The comparison boundary and accounting rules are set. The workload, local configuration, cloud instance, and measured run remain to be locked. No winner has been selected in advance.