Multi-Provider LLM Architecture

Vendor lock-in ke satu LLM provider adalah risiko bisnis. Dengan multi-provider architecture, Anda bisa switching provider tanpa rewrite kode, optimize cost per task, dan maintain uptime lebih tinggi.

Mengapa Multi-Provider?

1. Redundansi — Provider down? Switch otomatis
2. Cost optimization — Tiap tugas pakai model termurah yang cocok
3. Feature access — Model berbeda punya keunggulan berbeda
4. Negotiation power — Tidak tergantung satu vendor

Abstraction Layer

interface LLMProvider {
  name: string;
  chat(messages: Message[], options?: ChatOptions): Promise<ChatResponse>;
  stream(messages: Message[], options?: ChatOptions): AsyncGenerator<string>;
  estimateTokens(text: string): number;
}

interface ChatOptions {
  model?: string;
  temperature?: number;
  maxTokens?: number;
  jsonMode?: boolean;
}

interface ChatResponse {
  content: string;
  inputTokens: number;
  outputTokens: number;
  model: string;
  provider: string;
}

Implementasi per Provider

class OpenAIProvider implements LLMProvider {
  name = "openai";
  private client = new OpenAI();

  async chat(messages: Message[], options?: ChatOptions): Promise<ChatResponse> {
    const response = await this.client.chat.completions.create({
      model: options?.model ?? "gpt-4o-mini",
      messages: messages.map((m) => ({ role: m.role, content: m.content })),
      temperature: options?.temperature,
      max_tokens: options?.maxTokens,
    });

    return {
      content: response.choices[0].message.content ?? "",
      inputTokens: response.usage?.prompt_tokens ?? 0,
      outputTokens: response.usage?.completion_tokens ?? 0,
      model: response.model,
      provider: this.name,
    };
  }

  async *stream(messages: Message[], options?: ChatOptions) {
    const s = await this.client.chat.completions.create({
      model: options?.model ?? "gpt-4o-mini",
      messages,
      stream: true,
    });
    for await (const chunk of s) {
      const token = chunk.choices[0]?.delta?.content;
      if (token) yield token;
    }
  }

  estimateTokens(text: string) {
    return Math.ceil(text.length / 4);
  }
}

Router: Pilih Provider Otomatis

class LLMRouter {
  private providers: LLMProvider[];

  constructor(providers: LLMProvider[]) {
    this.providers = providers;
  }

  async chat(
    messages: Message[],
    strategy: "cheapest" | "fastest" | "best" | "fallback" = "fallback"
  ): Promise<ChatResponse> {
    const ordered = this.orderByStrategy(strategy);
    const errors: Error[] = [];

    for (const provider of ordered) {
      try {
        return await provider.chat(messages);
      } catch (error) {
        errors.push(error as Error);
        console.warn("[router] " + provider.name + " failed: " + error);
      }
    }

    throw new Error("All providers failed: " + errors.map((e) => e.message));
  }

  private orderByStrategy(strategy: string): LLMProvider[] {
    switch (strategy) {
      case "cheapest":
        return [...this.providers].sort((a, b) => a.costPerToken - b.costPerToken);
      case "fastest":
        return [...this.providers].sort((a, b) => a.avgLatency - b.avgLatency);
      case "best":
        return [...this.providers].sort((a, b) => b.qualityScore - a.qualityScore);
      default:
        return this.providers;
    }
  }
}

Unified Config

# config/llm.yaml
providers:
  openai:
    api_key: "${OPENAI_API_KEY}"
    models:
      - gpt-4o (best quality)
      - gpt-4o-mini (cheapest)
    priority: 1
  anthropic:
    api_key: "${ANTHROPIC_API_KEY}"
    models:
      - claude-sonnet-4-20250514 (best reasoning)
    priority: 2
  google:
    api_key: "${GOOGLE_API_KEY}"
    models:
      - gemini-2.5-pro (best value)
    priority: 3

routing:
  default_strategy: fallback
  task_strategies:
    classify: cheapest
    code: best
    chat: fastest

Latihan

Implementasikan LLMRouter dengan 3 provider (OpenAI, Anthropic, Google). Buat test yang memverifikasi fallback bekerja ketika provider utama gagal.