One LLM provider layer for OpenAI, Claude and Azure

Every LLM SDK looks interchangeable for about a day. Then the differences show up. Anthropic's Messages API takes the system prompt as a top-level field and requires max_tokens on every call. OpenAI's Chat Completions API puts the system prompt in the messages array and reports usage as prompt_tokens and completion_tokens, where Anthropic reports input_tokens and output_tokens. Azure OpenAI addresses a deployment you created rather than a model name. Error shapes, rate-limit headers and structured output support all differ too.

At Sendler.ai I built the AI layer: LLM-personalised messaging, AI reply classification and handling, and agentic workflows with tool calling. Under all of it sat a provider layer across OpenAI, Anthropic Claude and Azure OpenAI. This is how I'd design one again.

The interface is the contract

Feature code should never import a provider SDK. It talks to one interface, and adapters do the translation.

export type ProviderName = 'openai' | 'anthropic' | 'azure-openai';
 
export interface LlmRequest {
  task: string; // e.g. 'reply-classification': drives routing, metering and evals
  accountId: string;
  system?: string;
  messages: { role: 'user' | 'assistant'; content: string }[];
  maxOutputTokens: number; // required everywhere, because one provider requires it
  temperature?: number;
}
 
export interface LlmResponse {
  text: string;
  usage: { inputTokens: number; outputTokens: number };
  provider: ProviderName;
  model: string;
}
 
export interface LlmProvider {
  readonly name: ProviderName;
  complete(req: LlmRequest, model: string): Promise<LlmResponse>;
}
 
/** Adapters throw this for failures worth trying elsewhere: 429s, 5xx, timeouts. */
export class RetryableProviderError extends Error {}

Two fields do more work than they look like they do. task names the job rather than the model, so routing, cost reports and evals are all keyed by what the call is for. accountId makes every call attributable to a tenant, which the metering below depends on.

The adapters own the messy parts: mapping roles, moving the system prompt, normalising usage, mapping a logical model to an Azure deployment, and turning SDK errors into either RetryableProviderError or a plain error.

Tool calling is where adapters earn their keep. OpenAI and Anthropic each describe tools, and return the model's tool calls, in their own shapes. The layer should define one tool definition format and one tool-call result type, so agentic workflows only ever deal with those and never learn which provider answered.

Fallbacks, but only for the right failures

A fallback chain per task is just an ordered list of provider and model pairs. The real decision is which errors trigger it:

• Fall back on rate limits, server errors, overload responses and timeouts.
• Don't fall back on bad requests. A prompt that is too long or malformed will fail the same way elsewhere or, worse, succeed with different behaviour.
• Treat content-filter refusals as a policy question, not an outage. Routing around one provider's filter to another provider is a choice someone should make deliberately.

A fallback is also a different model with different output, so it has to pass the same evals as the primary.

The same wrapper is the natural home for cost control:

type Chain = { provider: LlmProvider; model: string }[];
 
export async function complete(req: LlmRequest, chain: Chain): Promise<LlmResponse> {
  await limits.consume(req.accountId, req.task); // per-account rate limit
  const hold = await credits.reserve(req.accountId, estimateCredits(req));
 
  let lastError: unknown;
  for (const { provider, model } of chain) {
    try {
      const res = await provider.complete(req, model);
      await credits.settle(hold, {
        task: req.task,
        provider: res.provider,
        model: res.model,
        ...res.usage, // provider-reported tokens, not a local estimate
      });
      return res;
    } catch (err) {
      lastError = err;
      if (!(err instanceof RetryableProviderError)) break;
    }
  }
  await credits.release(hold);
  throw lastError;
}

Metering lives in the call path

Sendler.ai had credit-based usage metering and per-account rate limits to control cost and protect customer accounts. In an LLM layer, the right place for both is the one function every call goes through:

• Check the account's rate limit before spending anything.
• Reserve an estimate of credits first, so concurrent calls can't overspend a balance.
• Settle with the usage the provider reports, not a client-side token count.
• Record task, provider and model on every usage row.

That last point pays off quickly. Cost per task and per account becomes a query, and so does the real cost of your fallbacks.

Validate structured output every time

JSON mode and structured outputs reduce malformed responses. They don't make validation optional. Validate at the boundary with a schema, allow one repair attempt that shows the model its validation error, and then fail into a safe default instead of guessing.

import { z } from 'zod';
 
const ReplyLabel = z.object({
  intent: z.enum(['interested', 'not_interested', 'out_of_office', 'referral', 'other']),
  confidence: z.number().min(0).max(1),
});
type ReplyLabel = z.infer<typeof ReplyLabel>;
 
function parseLabel(text: string) {
  let value: unknown;
  try {
    value = JSON.parse(text);
  } catch {
    value = undefined;
  }
  return ReplyLabel.safeParse(value);
}
 
export async function classifyReply(req: LlmRequest, chain: Chain): Promise<ReplyLabel | null> {
  const firstText = (await complete(req, chain)).text;
  const first = parseLabel(firstText);
  if (first.success) return first.data;
 
  const retry = await complete(
    {
      ...req,
      messages: [
        ...req.messages,
        { role: 'assistant', content: firstText },
        { role: 'user', content: `That was invalid: ${first.error.message}. Reply with JSON only.` },
      ],
    },
    chain,
  );
  const second = parseLabel(retry.text);
  return second.success ? second.data : null; // null means: route to a human
}

The labels here are illustrative. The shape is the point: parse, validate, repair once, then hand off.

Evals before any swap

Changing a model is a deploy, even when it's one line of config. Before swapping, I run the candidate against an eval set for that specific task:

• For classification, a labelled set of real, anonymised examples, scored per label rather than only overall.
• For generated messages, rubric checks that code can run: length limits, no unfilled template variables, the right person and company named. Then a smaller sample reviewed by people.
• For every task: schema failure rate, latency and cost per call.

The candidate ships when it matches or beats the current model on quality and the cost and latency trade-off is worth it. Because routing is keyed by task, one task can move while the rest stay where they are.

A provider layer isn't abstraction for its own sake. It's the one place where every LLM call is rate-limited, metered, validated and attributable to an account. Name calls by task, fall back only on transient errors, settle credits on reported usage, validate every structured response, and let your evals, not a launch announcement, decide when a model changes.