AI system design modeling: a tokenomics sketch
Ask what an AI system will cost and you often get a single number back: dollars per token, or a line item on a software bill. It's a tidy answer, and it hides most of what decides whether the system is worth running. It says nothing about whether the setup will stay up, how fast it answers, what it cost to tune the model, or who is on the hook when something breaks.
The short version: first check that a setup can run reliably, stay available, be serviced and stay secure. Only then look at cost, and look at it along three separate lines (speed, the cost of training or tuning a model, and the cost of serving answers) that you keep apart and never add into one figure.
This sketch comes from BridgeTek Labs, the research side of BridgeTek, and it is the frame we use when a team asks whether to rent capacity or buy it. BridgeTek is a data science practice. We don't resell the hardware, cloud or licenses we recommend.
A word on the numbers, which we'll say once. The dollar figures and multipliers here are order-of-magnitude scenarios for teaching. None of them is a quote, a bill of materials (BOM, the itemized hardware list behind a purchase) or a measured result for a named buyer, and channel SKUs and buyer-specific costings stay outside this page. Where a number comes from a third party we haven't verified, or rests on assumptions we can't publish, we call it fenced: we show it because it helps the picture, and we hold it at arm's length.
Why one number misleads
"Tokenomics", the economics of paying for AI by the token, usually gets squeezed into a single dollars-per-token figure. That squeezes together three costs that behave very differently:
- Firepower: how fast a setup produces answers, measured in tokens per second (a token is a small chunk of text, a word or part of one).
- Training and adaptation: what it costs to train a small model or tune an existing one to your work. We measure this as rental-equivalent spend, meaning what the GPU hours would cost at rental rates, whether you rented them or not.
- Serving: what it costs to answer requests day to day, which depends heavily on whether you pay a closed model's API or run open-weight models on hardware you own.
A single number also skips the production bar. A cheap option that fails on reliability, availability, serviceability or security is no win at all.
There's a related trap. A volume number, such as tickets per month or requests per day, tells you how much work there is. It says nothing about the hardness mix, the share of easy and hard tasks inside that volume, and it isn't the same as a count of scored eval tasks (an eval is a test suite with a fixed set of tasks and a scoring rule). Public benchmark boards report a cost per task: what it cost to run one model through one scored task. Where the model runs, rented or owned, is a different unit again, and we deal with it last so it never lands on the same dollar axis as the task costs. This article works at the level of a sketch. The step past it is to score models on tasks drawn from your own work with a harness, the code that runs a model through a fixed set of tasks and scores the results.
Satisfy reliability, serviceability, availability and security first. Then talk $/token. Modeling structures the tradeoffs and surfaces the gap between where you should be and where you are; a measured harness on your own workload is what settles the decision.
Start with the production bar
We call the first gate the production bar, and it has four parts:
- Reliability: it gives correct, consistent results.
- Availability: it's up when people need it.
- Serviceability: someone can monitor, fix and update it.
- Security: it protects your data and your users.
In day-to-day operation, these show up as tracing (a record of what the system did and why), guardrails, human oversight and sandboxes that contain what an AI agent can touch.
Meeting the bar takes people. Owning inference on-premises means having an AI engineering, MLOps (the practice of running models in production) and reliability team. Deciding whether to buy or build is really a question of whether you can support what you build.
The scale ladder
With the production bar in place, we can talk about size. We describe setups on a five-rung scale ladder, from a phone to a fleet of racks. Each rung has a typical job and a typical way of owning it:
| Rung | Typical job | Usual ownership |
|---|---|---|
| Phone / edge | Local assistant; low firepower | Device or personal software subscription |
| Laptop or desktop / 1 GPU | Local assistant for one person, plus small fine-tunes | Workstation, lightly owned |
| 1–few GPUs, or one to a few high-memory Macs / small server | Mid-serve: serving a mid-sized model to a team, plus adaptation runs | Owned inference starts here |
| 1 rack | Mid-to-high throughput serving | Owned inference, fuller stack |
| 10–20+ racks | Frontier-class self-hosting (teaching band) | Fuller stack; CapEx plus team at a ~$1–10M/yr scenario |
The ~$1–10M/year and 10–20 racks labels are order-of-magnitude teaching labels. No measured bill of materials or list price stands behind them. The ~$1–10M/year label is yearly for 10–20 racks, while published estimates of 2026 prices for the largest racks put a single 72-GPU rack at roughly $3–6.5 million to buy. Those are different units: a year for 10–20 racks, and the price to buy one rack, so the yearly label is not low against that price. (CapEx, short for capital expenditure, is the money that buys hardware up front.)
The ladder doesn't depend on whose accelerator you use. What places a workload on a rung is how much memory the model and its working cache need, and how many tokens a second the busiest hour demands. A mid-serve rung can be a server with two workstation GPUs or a small cluster of high-memory Macs.
Figure 1 draws the same ladder as a row of boxes, smallest on the left. Its message: mid-serve is the rung between a laptop and a rack. That highlighted rung, one to a few accelerators, is the one we care about most right now: it's where a team first runs its own inference, and where measured numbers are cheapest to get. What holds on this rung tells us nothing directly about multi-rack builds or the $1–10M band.
Three separate cost axes
Once a setup clears the production bar and has a place on the ladder, we look at cost along three axes, which we label A, B and C. We keep them apart because each answers a different question, and we never put them ahead of the production bar.
Axis A: firepower
Firepower is speed: tokens per second. Its teaching job is to make one point clear to leadership: AI comes in several performance classes.
The contrasts are easy to find in public posts and demos. Specialty stacks post very high token rates: Taalas, a specialty chip maker, reports about 17,000 tokens a second for one user on Llama 3.1 8B, a small model hard-wired into its chip (February 2026). For the mid-serve rung, Light Foundry reported about 243 tokens a second (1 Aug 2026) on two RTX PRO 6000 Blackwell cards serving DeepSeek-V4-Flash. Other public write-ups of the same two-card setup report roughly 200–400 tokens a second for a single request, and roughly 700–1,900 in total across 10 to 24 simultaneous requests, depending on software and settings. These are third-party figures, which we fence: we haven't measured them ourselves, and none of them is a service-level agreement (SLA) anyone has promised. Keep firepower separate from cloud hosting costs and from vendor performance claims.
That last pair of numbers shows the distinction that matters most on this axis. The speed one user sees is set mostly by memory bandwidth and model size, and it barely moves when you add hardware. The total across all users at once is what grows with more cards or more racks. A sizing needs both numbers, and every tokens-per-second figure should say which one it is.
Consumer-class chat at about 50–150 tokens a second is fenced: we haven't verified it; don't quote it as fact.
Figure 2 shows illustrative firepower for each rung of the ladder. Its message: firepower rises as you move from a phone to a rack. The vertical axis is logarithmic: each labelled step (1, 10, 100, 1,000, 10,000 tokens a second) is ten times the one below, so the jump from bar to bar is far bigger than it looks. The bars are rough classes and mix two measures: the lower rungs are roughly what a single user sees, and the rack bars are total output shared across many users. The highlighted bar is the few-GPU rung.
Axis B: training and adaptation cost
Axis B asks what owned or rented compute buys you when you train a small model from scratch or adapt an existing one. We measure it in rental-equivalent dollars or GPU hours.
One public example: Puro-2B, a model of about two billion parameters trained from scratch on consumer RTX 5090 GPUs. Its authors' technical report puts the rental-equivalent cost at about $4.4k for a checkpoint that matches Qwen2-1.5B on their benchmarks, and about $6.9k (22,514 GPU-hours) for the final model. That is the authors' own accounting, which we haven't reproduced, so it's fenced. None of this is a channel quote or a cost worksheet for a named buyer.
Axis C: serving cost
Axis C is the cost of serving tokens: paying for a closed model's API, or serving open-weight models yourself on owned or rented hardware. You'll often hear two rough claims here:
- About 3×: the cost reduction often claimed when a workload moves from cloud compute to owned on-premises hardware. This is the general infrastructure story.
- About 10×: the reduction often claimed when you move from paying for closed API tokens to serving open-source models on-premises yourself.
Treat both as scenario labels to test. We haven't measured them here and we attach no absolute dollars to them. Whether either shows up depends mostly on utilization and on what you would otherwise pay. Owned hardware is sized for the busiest hour and sits partly idle the rest of the day, while a metered API charges only for the tokens used. The illustrative worked examples for these labels are not shown on this page. Neither label is a validated multiplier for any named buyer, and neither is ever a reason to skip the production bar.
Artificial Analysis offers an outside check on how fast API prices move. Its cost-per-task Pareto view (the set of models where you can't get a higher score without paying more), its inference-price history and its Index-over-time charts all show the cheapest API price per million tokens falling quickly within each intelligence band. That supports a simple point: prices for cloud MaaS (model-as-a-service, hosted models you pay for per use) move quickly. It doesn't turn API prices into steps on the phone-to-rack hardware ladder, which stays on its own axis.
What each rung costs to own
Figure 3 brings the ladder and cost together. Its message: cost jumps by orders of magnitude as you climb the ladder. Each bar is the illustrative CapEx plus opex per year for one rung (opex is operating expenditure, the running costs such as power and people). The vertical axis is logarithmic, with real dollar labels from $100 to $1 million: each step is ten times the one below. The bars climb from a few hundred dollars a year for a phone to millions for a multi-rack fleet, which is the ~$1–10M/yr frontier self-hosting teaching band. The highlighted bar is the few-GPU rung.
Choosing how much to own
With the production bar and the three axes in hand, the choice becomes a menu of ownership options. Each option has a different set of things that dominate the decision:
| Ownership option | What it means | What usually dominates |
|---|---|---|
| SaaS / MaaS | Hosted models and agents; you trust the vendor's operations | Production bar (through vendor SLAs) plus Axis C |
| Owned inference | Private serving on your own or a partner's hardware | Production bar (you run operations) plus Axes A and C |
| Fuller stack | Hardware plus adaptation, harness and runtime | Production bar plus Axes B and C, plus enterprise agent frameworks |
In practice we sum each option up on a short posture card. Generic sizing (accelerator count, node count, network and storage bandwidth) maps onto a handful of illustrative classes: edge or workstation (phone to laptop, one user), mid-serve (one to a few accelerator nodes or high-memory Macs, with measured tokens per second when the evidence exists), rack or on-premises (an OEM-neutral factory shape) and cloud, metered or committed (a hyperscaler-neutral rental posture). Each card states the capacity class, the ownership posture, the inputs that would flip the card to a different answer, and a plain note that it is a teaching card with no quote or procurement list behind it. Commercial quotes are handled separately from the teaching model.
Our companion article, Cost, intelligence and walltime: one datapoint, many access paths, zooms in on individual access options (subscriptions, APIs and owned mid-serve hardware) and plots cost, intelligence and walltime for each.
The gap between should and actual
The most useful output of all this modeling is a gap. We compare two things:
- Should: a production-grade posture across all the metrics, with a sensible ownership option and ladder rung for the kind of workload.
- Actual: what we observe. Common examples are a move to the cloud made without a total cost of ownership (TCO) sheet, a vendor switch made without a cost comparison, or laptop-class hardware asked to serve a whole fleet.
The gap between them is what the model surfaces. It is input for planning; a finished audit is separate work.
We check the gap on seven things: the four production metrics, then firepower, serve cost, and TCO (whether the buyer has a quantitative picture of what good looks like). In the stylized picture we teach with, the should posture sits above the actual one on all seven. The widest gap is usually on TCO, because many buyers have no quantitative bar for what good looks like, and the narrowest is on firepower.
What to do next
How do you get from a sketch like this to a plan? We use a progressive estimate, much like a building contractor who starts with a rough sketch, moves to a representative estimate, lays out options and then draws up plans. Figure 4 shows the six steps, left to right. Its message: estimates get tighter as the work moves from idea to build. The highlighted step is the hardware estimate, where a rung on the scale ladder turns into an order-of-magnitude figure.
- Ideation: the pillars and outcomes you want, before any vendor list.
- Architecture: the harness, the runtime and the inference plane.
- Representative hardware: a rung on the scale ladder and an order-of-magnitude estimate.
- Options: on-premises or cloud, and where you sit on the ownership menu.
- Plans: a timeboxed statement of work.
- Build, only after agreement.
Three things hold at every step: production metrics come first, the should-versus-actual gap stays in view, and on-premises ownership needs an AI engineering and MLOps team. None of it is a binding quote.
The Token Economics Sprint takes these steps for a named use case. It builds a model of metered, rented and owned cost for your workloads, side by side and from your own figures, and hands it over to your team. Scoring candidate models on your own tasks with a harness is a separate measurement, and it is what replaces the public boards. Production comes first, then the three cost axes, then the gap between where you should be and where you are.
Caveats
A few terms in this article are easy to stretch, so here is exactly what each one covers:
- A public-cloud TCO worksheet takes CapEx, power, utilization and GPU-hour inputs for a buyer's purchase order. It's a different thing from an Axis A tokens-per-second demo.
- A tokens-per-second leadership demo illustrates firepower. It is neither a serving-cost multiplier nor a SKU quote.
- Training rental dollars from small-run reports teach training economics. They are no channel bill of materials.
- The 3× and 10× labels are teaching scenarios for Axis C. They are neither measured truth nor dollars ready for an account.
- $1–10M and 10–20 racks are the frontier self-hosting teaching band. There is no measured bill of materials or quote behind them.
- Should versus actual is a modeling output and a way to frame the gap. It stops short of a finished customer audit.
- Dollars per token alone is one slice of the economics and is never enough on its own to settle a production decision.
- Monthly ticket volume is only a count until the hardness mix is defined. It isn't a cost per task.
Third-party demos, reports, preprints and teaching numbers (Taalas, Light Foundry, the Puro-2B report, 3× and 10×, and $1–10M rack economics) have not been independently verified by BridgeTek. They are safe for general education. They don't go into account packs automatically, and no named buyer is the modeled customer.
All four figures are illustrative teaching charts. Their labels are scenario labels and show no vendor's SKUs.
Sources
- Artificial Analysis: model pricing, Intelligence Index and cost-per-task views, read 2026-09-01 / 2026-09-02
- Light Foundry, third-party report, 1 Aug 2026: about 243 tokens a second on two RTX PRO 6000 Blackwell cards serving DeepSeek-V4-Flash (fenced; third-party measurement)
- Ovidiu Dan, "Running DeepSeek-V4-Flash at 700 tokens/s on 2x RTX Pro 6000", 20 Jun 2026: about 210 tokens a second for one request, about 700 in total across 10 (fenced; third-party measurement)
- Local-AGI, DeepSeek V4 Flash 0731 benchmark on 2 × RTX PRO 6000: about 400 tokens a second for one request and about 1,860 in total across 24, at 2K context (fenced; third-party measurement)
- Taalas, "The path to ubiquitous AI", February 2026: about 17,000 tokens a second per user on a hard-wired Llama 3.1 8B; EE Times saw 15,000+ on the public demo (fenced; vendor figure)
- Luo et al., "Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090", August 2026: rental-equivalent cost of about $4.4k and $6.9k (fenced; the authors' own accounting)
- Tom's Hardware, Vera Rubin NVL72 rack pricing, 24 Mar 2026: about $2.8–3.4 million for a GB200 NVL72 rack and $6–6.5 million for a GB300 NVL72 rack (fenced; unconfirmed estimates, NVIDIA publishes no list price)
About the author



Jason Larkin
Principal Consultant, BridgeTek AI Services
Jason Larkin models and predicts how complex systems behave, from a back-of-the-envelope estimate to a full simulation. He applies that to the cost and design of AI systems in his writing for BridgeTek Labs, and to quantum and other emerging computing in his consulting work.
More from BridgeTek Labs
- Cost, intelligence and walltime: one datapoint, many access paths
One working dataset (Sept 2026) compares cost, intelligence and walltime across free, paid and owned access tiers.
BridgeTek Labs · AI system design ·