Skip to content

feat(llm): LLM Provider Abstraction Layer (Issue #99 - $600 Bounty)#608

Open
sirius-016 wants to merge 11 commits into
Spectral-Finance:mainfrom
sirius-016:feat/llm-provider-abstraction
Open

feat(llm): LLM Provider Abstraction Layer (Issue #99 - $600 Bounty)#608
sirius-016 wants to merge 11 commits into
Spectral-Finance:mainfrom
sirius-016:feat/llm-provider-abstraction

Conversation

@sirius-016
Copy link
Copy Markdown

Summary

Implements a complete LLM Provider Abstraction Layer as described in Issue #99.

What's included

Core Modules (7 new files + 1 modified)

  • Lux.LLM.Provider — New behaviour with metadata/0 and health_check/0 callbacks, model info helpers
  • Lux.LLM.Registry — ETS-backed provider registry (GenServer for writes, ETS for reads)
  • Lux.LLM.Router — Intelligent routing with 5 strategies: :cost, :latency, :quality, :round_robin, :fallback
  • Lux.LLM.Fallback — Circuit breaker (closed/open/half_open states) + exponential backoff retry
  • Lux.LLM.Monitoring — Per-provider & per-model cost/latency tracking with ETS aggregation
  • Lux.LLM.Cache — ETS LRU cache with TTL, hit/miss statistics
  • Lux.LLM.Manager — High-level chat/3 API that orchestrates routing + fallback + caching + monitoring
  • Lux.LLM — Updated with chat/3, configure/1, usage_report/0, total_cost/0, cache_stats/0

Tests (7 files)

  • Unit tests for Provider, Registry, Router, Fallback, Monitoring, Cache, Manager

Documentation

  • Livebook guide (guides/llm_abstraction.livemd)
  • Benchee benchmarks (benchmarks/llm_benchmark.exs)

Acceptance Criteria

  • Universal provider interface (Lux.LLM.Provider behaviour)
  • Provider registry and management system (Lux.LLM.Registry)
  • Automatic model selection logic (Lux.LLM.Router with cost/latency/quality strategies)
  • Intelligent fallback handling (circuit breaker + exponential backoff)
  • Cost tracking and optimization (Lux.LLM.Monitoring with per-token pricing)
  • Performance monitoring and analytics (latency, request counts, usage reports)
  • Documentation and examples (Livebook guide, inline docs, benchmarks)

Total: 17 files, ~50KB

Closes #99

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM Provider Abstraction Layer $600

1 participant