How do you run LLM calls on Cloudflare Workers? OpenAI–Claude fallback

Why do model calls live in a separate Worker instead of the app?

FeedHand is our own SaaS that turns a website link into a month of planned social media posts. The TanStack Start web app runs in a Worker called feedhand-web; model calls run in a second Worker, feedhand-ai-gateway. The monorepo has two apps and 14 packages, and the gateway depends only on @feedhand/ai.

The gateway’s wrangler.jsonc sets workers_dev: false, preview_urls: false and defines no routes, so it has no address on the internet. The only way in is the AI_GATEWAY service binding from the web Worker on the same account. On top of that, every request must carry GATEWAY_TOKEN, compared in constant time.

The app uses the Anthropic SDK and an OpenAI client unchanged. The token goes where the SDK puts its API key (x-api-key or Authorization: Bearer), and the gateway swaps in the real key. The SDK only gets a different baseURL and a fetch that routes through the service binding.

The binding is marked "remote": true. Even under local vite dev, requests go to the deployed gateway, so the developer machine never holds a provider key. All four secrets (Anthropic, OpenAI, the gateway token and the media API key) exist only in the gateway’s secret settings.

What does the gateway check before forwarding a request?

The gateway is a policy layer, not a pass-through proxy. Its rules live in apps/ai-gateway/src/policy.ts as pure functions, so they are tested without a Worker runtime. Some concrete limits:

• Only three allowlisted Claude models are accepted; max_tokens is capped at 64,000, and at 32,000 on the OpenAI side.
• Bodies are capped at 4 MiB, with 1 to 12 messages.
• Fields that would let a caller reach tools or code execution through our key are refused. The one allowed tool is web search, at most 8 searches per call, priced at $0.01 each.
• Every body sent to OpenAI gets store: false.
• For audio transcription the multipart form is rebuilt, so only five checked fields reach OpenAI.

The routes are fixed too: /anthropic/v1/messages, /openai/v1/chat/completions, /openai/v1/responses for web search only, /openai/v1/audio/transcriptions, and GET /usage for spend. Anything else returns 404.

The logging rule is written into the code as a comment: metadata only, never prompts, outputs or keys. A log line carries the provider, lane, model, HTTP status, dollar cost and brand id. That is enough to see which brand spent what without copying customer content into logs.

How do parallel calls avoid overshooting the budget?

The spend ledger is a single Durable Object instance: Budget, opened with idFromName("global"), holding two tables, spend and credits, in SQLite storage. With one instance, each method finishes before the next starts, across all isolates.

The flow has two steps. First a reservation: the worst-case cost is held before the provider is called. For Anthropic it counts every three input bytes as a token and adds the full output budget; real text runs 3–4 bytes per token, so it stays high. Then settlement: when the provider answers, the row is updated to the real cost. A failed request settles to 0, since it is not billed. A broken stream or unparseable answer keeps the worst case, so the caps err safe.

A reservation checks several ceilings at once: $40 a day and $600 a month globally, $3 a day for the public free tools, $25 a day for the free preview lane, and $0.6 for the whole life of one preview. On top of these, the app sends the brand’s remaining monthly and daily budget as headers on every call. The preview lane has its own pool, so unpaid use can never drain the paid lane.

For streamed Claude responses the body is split with tee(): one branch goes to the user, the other reads token counts from the SSE events inside waitUntil and settles the ledger.

OpenAI first, Claude as backup: which errors switch providers?

It is decision K42 in the decision log, dated 27 October 2026: every AI call goes to OpenAI first, and Claude takes over when OpenAI is down, rate limited or failing. The founder’s note with it asks for OpenAI first, Anthropic as backup, and no failure ever. The reason is price: per million tokens, the entry lists gpt-6.1-sol at $2/$10 and Opus 5.5 at $4/$20.

The switch rule in packages/ai/src/llm.ts is deliberately narrow. On OpenAI, a connection error or a 401, 402, 403, 408, 409, 429 or 500–504 moves to the backup. Claude adds 529, plus the 400 “credit balance is too low” that Anthropic returns instead of a 402. A bad request or a refusal does not switch: it would fail the same way elsewhere.

A conversation switches at most once (a switched flag). Two transcripts are kept: an append-only Claude one with thinking blocks, so the prompt cache stays valid, and a plain-text one any backup can read, so a switch mid-repair continues where it stopped.

Structured output has two fallbacks. A 400 in OpenAI’s strict json_schema mode repeats the turn in json_object mode and remembers the schema in a WeakSet. A “grammar is too large” from Claude moves the schema into the system prompt, and zod plus a repair turn do the checking.

Why does AI spend permission live in one module?

The gateway knows what a call can cost, not who may spend. That decision is on the web side, in apps/web/src/server/spend-access.ts. Its opening rule: a customer can never cost more than they pay, and someone without a plan costs next to nothing.

spendAccess(planId, development, purpose) is a pure function that never touches the database, with 17 unit tests. Its outcomes:

• An active paid or internal plan spends on the app lane.
• The free preview may make only the purpose: "preview" call, on the trial lane; everything else is refused with preview_only.
• A brand whose package is not bought yet gets pending_package, an expired one lapsed, an unknown one no_plan, and none of them spend.

The budget maths sits in the same file. A paid brand’s monthly model budget is 10% of its package price plus 10% of the period’s add-ons. The daily ceiling is half of that, at least $3 and never more than the monthly budget, so an afternoon of re-runs cannot burn a month. The preview’s budget is $0.6 for life and never refills.

The plan belongs to the brand, not the workspace: each brand is its own package with its own budget. assertSpendAccess in server/ai.ts runs in every server function and Workflow step that spends on AI or media, and the result travels to the gateway as four headers.

Why does FeedHand use 6 separate Cloudflare Workflows?

The web Worker’s config binds six Workflows: brand-analysis, plan-build, production, market-research, competitor-research and replacement. All are exported from src/server.ts; TanStack Start handles HTTP and background work lives in the Workflow classes.

A commit message gives the reason. The replacement for a rejected post used to be written in waitUntil, so a deploy or outage in those seconds lost it silently. ReplacementWorkflow now runs these steps:

• undo-window: a 5-second sleep, the interface’s 4.5-second undo plus a margin.
• check: is the post still rejected and still the latest version?
• charge: one edit credit is taken, with an audit_log marker written in the same transaction under a Postgres advisory lock, so a step that commits and then dies is not charged again on retry.
• write: the model call, with 2 retries, exponential backoff and a 5-minute timeout.
• refund: if the last attempt fails too, the credit goes back once.

Video production follows the same idea. Each clip is submitted in its own step (submit-clip-N, 12 retries 45 seconds apart), because the media provider runs at most 5 videos at once per account and a 429 is normal under load. A restart after clip 2 resumes at clip 3.

What traps come with durable steps?

The rule that needs the most care is written in steps.ts: code outside step.do runs again on every replay, after each sleep, retry or eviction. So progress marks only move forward. advanceStep ignores an update that would move a finished step back to “running” or drop its counter from 30 to 0. A failure, on the other hand, always lands and is never hidden behind an earlier “done”.

The second is result size. A step’s return value can hit the 1 MiB limit, so brand analysis’s collect step returns source ids, not text; the old shape is still read for runs replayed after a deploy.

The third is connections: each step opens one database client and closes it at the end (withStepDb, at most 2 connections).

Plan building balances cost against time. A month is written in batches of 10 posts, and a batch never mixes two markets. The first batch runs alone and writes the prompt cache; the rest read it, 6 steps in parallel. Each write step has 2 retries and a 10-minute timeout. If the strategy call fails the plan does not stop: a deterministic strategy derived from the brand profile takes over.

`packages/rules` and `ai-tells.ts`: how is model output made publishable?

Every post a model returns goes through the @feedhand/rules package: 27 source files, about 7,500 lines and 19 test files. The rules are deterministic; there is no model call in them.

ai-tells.ts (813 lines) looks for 16 kinds of signs that a language model wrote the text: stock openers (“Discover…”, “Unlock…”), the “not just X, it’s Y” construction, three stacked adjectives, one-word staccato sentences, a question answered right away (“The secret? Patience.”), em dashes for rhythm, emoji bullets and hashtag walls inside the text. The lists are data for English, Turkish, German, French and Spanish. Each tell weighs 1 or 2; a text goes back for a rewrite at a total of 2, so a single weak signal is not enough.

A false positive costs a repair turn, so the detector is conservative: quotes are skipped, brand and product names are blanked, and the owner’s own habits are not tells. Mechanical tells such as em dashes and hashtag walls are fixed in code without spending a turn. For the rest, only the affected posts go to a targeted repair turn, and a batch is never dropped because of a tell.

The same package holds platform limits. limits.ts caps Instagram at 5 hashtags per post: the Graph API docs still say 30, but the app now accepts 5. Links on X are off by default because a post with a URL costs $0.20 through the API against $0.015 without. Promotional Reddit posts are kept under 10% (the 9:1 rule of thumb).

A checklist if you are building something similar

FeedHand is live, with payments open. Everything here exists in code and is tested; none of it is a claim about performance under traffic. The decision log also notes that the AI-tell measurement ran on 600 posts from the deterministic draft writer, not a real model.

• Keep provider keys in a separate Worker with no public URL, reached by service binding plus a token.
• Keep the SDKs; a baseURL and a fetch are enough.
• Reserve the worst-case cost before the call and settle to the real cost after. A single Durable Object is enough for atomicity.
• Give free usage its own lane and its own pool.
• Switch providers only on errors that count as outages, at most once per conversation; do not carry a bad request over to the other provider.
• Keep the question of who may spend in one pure function that never reads the database, and test every rule.
• Move any money-spending job that starts in waitUntil into a Workflow step, and make the charging step idempotent.
• Assume code outside step.do runs again, and keep step results small.
• Check model output with deterministic rules; fix mechanical issues in code and send back only what needs rewriting.

At AMG Software Studio we build AI-powered SaaS products like this, for ourselves and for clients.