Terraphim AI documentation

Handover Document - LLM Router Feature Integration

Date: 2026-01-04 Branch: feature/llmrouter-integration-research (merged to origin) Status: Complete, pushed to origin

1. Feature Summary

Implemented LLM Router with dual-mode support for intelligent LLM selection across multiple providers:

  • Library Mode: In-process routing via RoutedLlmClient wrapping static LLM client
  • Service Mode: HTTP proxy client (ProxyLlmClient) forwarding to external terraphim-llm-proxy service

Architecture Components

| Component | File | Purpose | |-----------|------|---------| | RouterMode enum | crates/terraphim_config/src/llm_router.rs | Library/Service mode selection | | RouterStrategy enum | crates/terraphim_config/src/llm_router.rs | Cost-first, Quality-first, Balanced, Static | | LlmRouterConfig | crates/terraphim_config/src/llm_router.rs | Full router configuration | | RoutedLlmClient | crates/terraphim_service/src/llm/routed_adapter.rs | Library mode wrapper | | ProxyLlmClient | crates/terraphim_service/src/llm/proxy_client.rs | Service mode HTTP client | | MergedRouterConfig | crates/terraphim_service/src/llm/router_config.rs | Role + env config merging |

Configuration Example

roles:
  - name: "Engineer"
    llm_router_enabled: true
    llm_router_config:
      mode: "library"  # or "service"
      strategy: "balanced"
      proxy_url: "http://127.0.0.1:3456"

2. Implementation Steps Completed

| Step | Description | Status | |------|-------------|--------| | Step 1 | Workspace integration (terraphim_llm_proxy dependency) | ✅ | | Step 2 | Configuration types (RouterMode, RouterStrategy, LlmRouterConfig) | ✅ | | Step 3 | Adapter layer (RoutedLlmClient for library mode) | ✅ | | Step 4 | Integration point (build_llm_from_role() modification) | ✅ | | Step 5 | Service mode (ProxyLlmClient for HTTP proxy) | ✅ | | Test Fixes | Update all Role initializations in test/example files | ✅ |

3. Files Modified

Core Implementation (5 files)

crates/terraphim_config/src/llm_router.rs      # Router config types (NEW)
crates/terraphim_service/src/llm.rs             # Integration point (MODIFIED)
crates/terraphim_service/src/llm/routed_adapter.rs  # Library mode (NEW)
crates/terraphim_service/src/llm/proxy_client.rs    # Service mode (NEW)
crates/terraphim_service/src/llm/router_config.rs   # Config merging (NEW)

Test/Example Updates (14 files)

crates/terraphim_config/examples/atomic_server_config.rs
crates/terraphim_config/examples/multi_agent_atomic_server_config.rs
crates/terraphim_mcp_server/tests/mcp_autocomplete_e2e_test.rs
crates/terraphim_mcp_server/tests/mcp_rolegraph_validation_test.rs
crates/terraphim_middleware/tests/haystack_refactor_test.rs
crates/terraphim_middleware/tests/mcp_haystack_test.rs
crates/terraphim_middleware/tests/perplexity_haystack_test.rs
crates/terraphim_multi_agent/examples/enhanced_atomic_server_example.rs
crates/terraphim_multi_agent/examples/multi_agent_coordination.rs
crates/terraphim_multi_agent/examples/workflow_patterns_working.rs
crates/terraphim_multi_agent/tests/llm_integration_test.rs
terraphim_server/src/workflows/multi_agent_handlers.rs
terraphim_server/tests/api_context_tests.rs
terraphim_server/tests/debug_rolegraph.rs

4. Test Results

cargo test --workspace --features llm_router

# terraphim_service --lib: 118 passed, 5 ignored
# All LLM router tests pass
# All proxy client tests pass

Pre-existing Failures (Unrelated)

5 failures in terraphim_agent --test integration_test:

  • API returns "success" but tests expect "Success" (case sensitivity)
  • These are pre-existing issues, not caused by LLM Router

5. Key Design Decisions

5.1 Feature-Gated Architecture

All LLM Router code is behind #[cfg(feature = "llm_router")]:

  • Production builds remain unchanged
  • No performance impact when feature disabled
  • Gradual rollout possible

5.2 Configuration Merging

MergedRouterConfig::from_role_and_env() combines:

  1. Role configuration (Role.llm_router_config)
  2. Environment variable overrides (LLM_PROXY_URL, etc.)

This allows deployment-specific overrides without modifying role files.

5.3 Error Handling

  • Use ServiceError::Config for proxy connection failures
  • Graceful degradation when proxy unavailable
  • Library mode works without external dependencies

5.4 Test Strategy

  • Unit tests in module #[cfg(test)] blocks
  • Integration tests in tests/ directory
  • All tests updated to include new Role fields

6. Known Issues / Technical Debt

| Issue | Severity | Notes | |-------|----------|-------| | Unused methods in ProxyLlmClient | Low | is_proxy_mode(), name() not called externally | | Unused log_requests field | Low | Could be implemented for debugging | | Pre-existing API case sensitivity | Low | terraphim_agent integration tests |

7. Next Steps

Immediate

  1. Review PR: Create PR for feature/llmrouter-integration-research merge
  2. Test with real proxy: Start terraphim-llm-proxy on port 3456 and verify service mode
  3. Document configuration: Add examples to role configuration documentation

Future Enhancements

  1. Dynamic routing: Update router config at runtime without restart
  2. Metrics: Add routing decision metrics (cost, latency, quality)
  3. Fallback strategies: Configure fallback when primary LLM fails
  4. Provider rotation: Round-robin or weighted random provider selection

8. Commands Reference

# Build with LLM Router
cargo build --workspace --features llm_router

# Run tests
cargo test --workspace --features llm_router

# Run only terraphim_service tests
cargo test -p terraphim_service --features llm_router --lib

# Enable in config
# roles:
#   - name: "Engineer"
#     llm_router_enabled: true
#     llm_router_config:
#       mode: "service"
#       proxy_url: "http://127.0.0.1:3456"

9. Lessons Learned

See lessons-learned.md section "LLM Router Integration - 2026-01-04" for detailed patterns:

  1. Feature-gated module organization
  2. Configuration re-export for public API
  3. Test file updates for struct schema changes
  4. ServiceError variant selection
  5. Submodule import paths in Rust
  6. JSON serialization test assertions
  7. Default trait for configuration structs
  8. Mode-based client selection

10. Verification Checklist

  • [x] Workspace builds with --features llm_router
  • [x] All terraphim_service lib tests pass (118 passed, 5 ignored)
  • [x] LLM router tests pass (llm_router_tests)
  • [x] Proxy client tests pass (proxy_client_tests)
  • [x] All test files updated with new Role fields
  • [x] Branch pushed to origin
  • [x] Documentation updated (lessons-learned.md)
  • [x] Handover document created
Handover Document: Terraphim GitHub Runner Server Integration Auto-Update Feature Implementation Handover