Price Free AI Capacity in Deadline Slack, Not in Tokens

It is 02:14. Your nightly batch window closes at 02:30. The free lane is still running, and the token meter reads zero.

Last week that felt like a win. Tonight the window is gone. Which operational action follows from that evidence?

Not delete the free lane. Price it.

Two numbers, one decision

Most teams measure the wrong number.

1. Token spend. Cheap, often zero on a free tier.
2. Deadline slack. Seconds you can spend before an SLO breaks.

Free capacity converts the first into the second. That conversion has a rate. Measure the rate, not the sticker price.

Define slack use per job class:

slack_used = free_p95_duration * (1 + retry_ratio) - paid_p50_duration

Three terms matter. The free lane p95 per attempt. The extra attempts per completed job. The paid baseline you gave up.

What the retry tax actually costs

A retry is not a retry. It is wall-clock time plus a new queue position plus a fresh attempt.

• Preemption returns the worker mid-attempt. The partial work is gone.
• Throttling returns a rate-limit signal. The partial work is gone.
• Worker loss returns nothing at all. You wait for a timeout to notice.

Each event resets the attempt clock. Each event consumes slack, not tokens. That is how a free lane stays cheap and still blows the window.

If your provider returns 429, stop and back off. Do not retry around the limit.

The artifact: a break-even router

Here is a small script. It reads measured job-class statistics from one fixed window and prints a routing decision.

It does not call a model. It prices time.

#!/usr/bin/env python3
# breakeven.py - route batch job classes by deadline slack, not list price.
import json
import sys
from dataclasses import dataclass


@dataclass
class ClassStats:
    name: str
    attempts: int          # free-lane attempt starts in the window
    completed: int         # jobs that finished inside the class deadline
    free_wall_p50: float   # seconds per attempt, free lane
    free_wall_p95: float   # seconds per attempt, free lane
    paid_wall_p50: float   # seconds per job, reserved lane
    paid_cost: float       # currency units per completed job, reserved lane
    slack_budget: float    # seconds of negative slack this class may spend
    miss_penalty: float    # currency units charged when the deadline slips


def retry_ratio(s):
    if s.completed == 0:
        return None
    return max(s.attempts - s.completed, 0) / s.completed


def free_duration_p95(s):
    r = retry_ratio(s)
    return None if r is None else s.free_wall_p95 * (1.0 + r)


def slack_used(s):
    d = free_duration_p95(s)
    return None if d is None else d - s.paid_wall_p50


def miss_proxy(s):
    # Crude upper-bound proxy, not a fitted probability model.
    used = slack_used(s)
    if used is None:
        return 1.0
    if used <= s.slack_budget:
        return 0.0
    return min(1.0, (used - s.slack_budget) / max(used, 1e-9))


def route(s):
    used = slack_used(s)
    if used is None:
        return 'hold', 'no completed jobs in window'
    if used > s.slack_budget:
        return 'reserved', 'p95 slack use exceeds budget'
    risk = miss_proxy(s) * s.miss_penalty
    if risk >= s.paid_cost:
        return 'reserved', 'risk-adjusted free cost >= paid cost'
    return 'free', 'slack within budget and risk below paid cost'


def main(path):
    with open(path) as fh:
        rows = [ClassStats(**json.loads(l)) for l in fh if l.strip()]
    print('class             slack_used  budget  miss_p  free_risk  decision')
    for s in sorted(rows, key=lambda r: r.name):
        used = slack_used(s)
        decision, why = route(s)
        risk = miss_proxy(s) * s.miss_penalty
        used_s = 'n/a' if used is None else '%ds' % used
        print('%-17s %9s %6ds %7.2f %10.2f  %s' % (
            s.name, used_s, s.slack_budget, miss_proxy(s), risk, decision))
        print('%-17s reason: %s' % ('', why))


if __name__ == '__main__':
    main(sys.argv[1] if len(sys.argv) > 1 else 'jobs.jsonl')

Declared workload for the sample fixture

State the conditions before you quote any result.

• Two job classes, 1,000 and 900 completed jobs in one 24h window.
• Free lane: one worker pool, no reserved capacity, timing measured per attempt.
• Paid baseline: p50 per job from the previous week on reserved capacity.
• No model-quality comparison. This drill prices time, not accuracy.

Run it locally in under a minute

Create the fixture. These are sample values, not published measurements.

{"name":"nightly-embed","attempts":1180,"completed":1000,"free_wall_p50":38.0,"free_wall_p95":96.0,"paid_wall_p50":41.0,"paid_cost":0.42,"slack_budget":120.0,"miss_penalty":180.0}
{"name":"ocr-backfill","attempts":960,"completed":900,"free_wall_p50":22.0,"free_wall_p95":150.0,"paid_wall_p50":24.0,"paid_cost":0.11,"slack_budget":60.0,"miss_penalty":40.0}

Run the router:

python3 breakeven.py jobs.jsonl

Expected output on the fixture above. Label it expected until your own window reproduces it.

class             slack_used  budget  miss_p  free_risk  decision
nightly-embed            72s    120s    0.00       0.00  free
                  reason: slack within budget and risk below paid cost
ocr-backfill            136s     60s    0.56      22.36  reserved
                  reason: p95 slack use exceeds budget

The second class is the interesting one. Its per-attempt p95 is slower, and its retry ratio adds about 7 percent. Slack use lands at 136 seconds against a 60 second budget.

Free is the expensive lane here, and tokens never entered the calculation.

Failure injection: prove the ledger counts

A routing rule you never broke is a guess. Break the lane first.

# worker.py is a throwaway harness, not a model client.
python3 worker.py --class ocr-backfill --concurrency 4 &
WPID=$!
for i in 1 2 3; do
  sleep 60
  pkill -f 'worker.py --class ocr-backfill' -n
  sleep 5
  python3 worker.py --class ocr-backfill --concurrency 4 &
  WPID=$!
done
kill $WPID
python3 breakeven.py jobs.jsonl

Watch one thing: does a killed attempt show up as attempt=fail or does it vanish?

If it vanishes, your attempts counter is wrong. Your retry ratio is then wrong. Your routing decision is then wrong.

A minimal harness so the smoke test runs:

# worker.py - throwaway harness for the ledger drill only.
import argparse, random, time

p = argparse.ArgumentParser()
p.add_argument('--class', dest='cls', required=True)
p.add_argument('--concurrency', type=int, default=1)
p.add_argument('--attempt-seconds', type=float, default=30.0)
p.add_argument('--fail-rate', type=float, default=0.15)
a = p.parse_args()

while True:
    time.sleep(a.attempt_seconds)
    ok = random.random() > a.fail_rate
    print('%s attempt=%s' % (a.cls, 'ok' if ok else 'fail'), flush=True)

This harness proves your ledger counts. It proves nothing about any model. Swap in your real worker for that.

Telemetry you must emit

Without these fields the ledger is unreadable a week later.

• job_class and route at admission time
• attempt_index per job, starting at 0
• attempt_started_at, attempt_ended_at
• attempt_outcome: ok, preempted, timeout, throttled, worker_lost
• deadline_at and slack_remaining_s at completion
• tokens_in, tokens_out, even on a free tier
• worker_id, so a bad host is visible

Keep tokens_in on free runs. You need it later to compute the exchange rate between free capacity and reserved capacity.

Decision table and threshold rationale

Why 50 percent as a default? Because retry ratio is usually noisier than wall time. If your retry ratio p95 moves more than your duration p95, tighten the threshold.

If the miss penalty is contractual, do not run the free lane at any threshold. Slack budgets are for engineering tradeoffs, not for contracts.

Cleanup and rollback

Run these in order. Write them down before the drill, not after.

1. Drain the class. Set free-lane route weight to 0.
2. Wait for in-flight attempts, or cancel them and record worker_lost explicitly.
3. Restore the reserved weight from your config store, never from memory.
4. Keep the window JSONL file. It is the evidence for the next review.
5. Revert alert thresholds only after two clean windows.

Step 3 matters most. A rollback typed by hand at 02:20 is a second incident.

Where free capacity fits in this drill

The drill needs a free lane before you can price one. MonkeyCode is an open-source project with free model access; the operator states 10M free tokens and a free server option.

I treat both availability claims as operator-supplied. Limits and terms change, so confirm them on the project page before you plan around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In this workflow the free tier is exactly the lane whose retry ratio you are measuring. Point the harness at one small class, run one window, read the ledger. Start with one class and one window on the free tier, then decide.

Limitations and who should not use this

Be honest about the edges.

• Not for synchronous user traffic with a hard p99. Free lanes are variable by nature.
• Not for classes where a single miss is contractually unacceptable. Keep those reserved.
• Not reliable under about 200 completed jobs per window. The p95 is noise below that; widen the window.
• miss_proxy is a proxy, not a fitted model. Replace it with your own estimator if you have one.
• Do not use this to justify free capacity against a regulated deadline without a named risk owner.

The threshold question

Every number here is a threshold with a rationale, and yours will differ. The useful question is not whether free capacity is cheap.

It is: how many seconds of slack does a retry buy, and who pays when the budget runs out?

Answer that with a measured window, and routing stops being a mood.