A model fits on one GPU. How many users fit with it?

OpenAI showed that gpt-oss-120b could run on one 80 GB GPU. A burst of twenty contract reviews shows what a buyer must test before promising a two-minute turnaround.

A lawyer drops a contract into a private model on Friday afternoon and receives a useful list of renewal dates. On Monday, twenty colleagues upload their files before a meeting. The same system now takes several minutes to finish the last review. The hardware has passed Friday’s feasibility test. To understand Monday, the infrastructure team needs the arrival times, the lengths of those contracts and the deadline for the meeting.

GPU capacity Model weights occupy only part of the available memory. Active contexts, runtime buffers and headroom determine what else fits.

What the 80 GB specification buys

On 5 August 2025, OpenAI released gpt-oss-120b and gpt-oss-20b under the Apache 2.0 licence. The announcement described the larger model running on one GPU with 80 GB of memory and the smaller on a device with as little as 16 GB. For an engineering team with hardware already installed, this was enough to justify trying the weights locally before placing an order.

The precise wording matters. The 120b model contains about 117 billion parameters, with approximately 5.1 billion active per token. Its architecture selects a subset of expert networks as it processes each token. OpenAI’s model card explains that the expert weights use MXFP4, a compact numerical format. The memory result depends on the serving software being able to use that representation. A runtime that expands weights into a larger format can produce a different answer to the basic question of fit.

Even after a model loads, a running server needs memory for intermediate calculations and for the attention state of active requests. The latter is often called the key-value or KV cache. It lets the model refer back to information from earlier tokens as it generates the next one. Its allocation changes as requests enter and leave. The original PagedAttention research documents how inefficient management of that state can restrict the number of requests served together. Its experiments predate gpt-oss, so use the mechanism to design a test and measure the newer model directly.

Work backwards from the morning meeting

Suppose twenty reviews arrive at exactly 9 am, and each must finish by 9:02. Model a short contract as 4,000 input tokens and 200 generated tokens. Model a long contract as 20,000 input tokens and 800 generated tokens. Count all generated tokens that occupy the server, including any spent before the final answer becomes visible. The following mix and deadline are planning assumptions, independent of either model’s measured speed.

Scroll horizontally to compare the figures.

Illustrative contract reviews arriving together at 9 am
Document groupReviewsTotal input tokensTotal generated tokens
Short contracts1664,0003,200
Long contracts480,0003,200
Two-minute batch20144,0006,400

Over those 120 seconds the batch demands an average of 1,200 input tokens processed per second and about 53.3 generated tokens per second across the service. Those are separate requirements. A server spends work reading contracts and then generates each review a token at a time. Adding the rates into one impressive tokens-per-second number conceals how the work is divided. Read both as necessary average rates, before allowing for scheduling, uneven completions or failure. Neither figure establishes that the last user meets the deadline.

The timing of arrivals changes the purchase. Spread the same twenty cases over an hour and an overnight batch queue may be quite adequate. Submit four more long contracts at 9:01 and the machine has only a minute left to absorb them. A legal team with offices in two time zones may find that their peaks rarely coincide. Ask for the busiest real interval before multiplying a single-user benchmark by the number of employee accounts.

Run the queue the way people use it

Begin with one short and one long case on each candidate configuration. Fix the model revision, runtime, quantisation, reasoning setting and generation limits. Verify the answer quotes the correct renewal clause and identifies an amendment that overrides the original date. If either case exceeds its context budget or ends midway through a reference, capture the outcome as a failed review. Silently truncating the contract to get a faster number changes the service the legal team is buying.

Replay the burst on a warmed server, sending each request at its scheduled time even if the previous one is still running. A test client that waits for each answer before submitting the next quietly throttles the traffic. Time the request from the client side. Record when the first useful text appears and when the full answer is available. Alongside those times, collect queue wait, active requests, input and generated token counts, peak memory and any cancelled or rejected requests. An average can conceal the single long review still missing at the meeting.

Try the same replay with two smaller replicas if the smaller model meets the quality standard. That changes the number of independent places a request can land. Dividing one larger model across two GPUs is a different experiment: a single request may occupy both devices. Similarly, changing the maximum response length can admit more work while cutting short the cases that need it most. Vary one choice at a time and reread the answers from the slowest cases before accepting a throughput gain.

Turn the run into a capacity proposal

A good purchase brief includes the largest supported document, a distribution of ordinary lengths, the burst shape and the completion target. Give each candidate the same review rubric. Count the number of correct reviews completed within the deadline, including rejected jobs in the original demand. Repeat the burst enough times to observe normal variation. Run it once more after a service restart to see what Monday looks like if the weights have to reload.

Dedicated capacity also has a clock outside the meeting. A machine reserved all week accrues cost through quiet afternoons. Divide the full reservation and operating cost for the period by the accepted reviews it delivered on time. If the workload can move to a scheduled window, compare a reservation for that window with a continuously available service. Add spare capacity to each proposal only where the business actually requires continued service during a failure, then demonstrate what happens when one worker goes away.

For a Cooma TokenFactory capacity discussion, take the replay and its acceptance results along with the model name. Ask each bidder to run the same burst on the proposed configuration. Compare how many correctly cited reviews actually arrive before the meeting starts.