Terraphim AI documentation

Research Document: PR #529 Gap Analysis

Status: Review
Author: AI Agent Analysis
Date: 2026-02-20
Reviewers: Terraphim Team

Executive Summary

PR #529 implements a functional TinyClaw multi-channel AI assistant with real execution capabilities. This research document identifies gaps between the current implementation and production-ready standards. The implementation covers core functionality (shell execution, LLM integration, channel adapters) but has gaps in comprehensive testing, documentation, error handling, and integration testing.

Key Finding: While all 220+ tests pass and clippy is clean, there are 15 identified gaps across testing, documentation, configuration, and operational readiness that should be addressed before production deployment.

Essential Questions Check

| Question | Answer | Evidence | |----------|--------|----------| | Energizing? | Yes | Multi-channel AI assistant is core product feature | | Leverages strengths? | Yes | Builds on existing Terraphim knowledge graph and search | | Meets real need? | Yes | Closes #519 - implements TinyClaw agent requested by users |

Proceed: Yes - 3/3 YES

Problem Statement

Description

PR #529 replaces all placeholder implementations in TinyClaw with real, functional code including:

  • Real shell execution via tokio::process::Command
  • Tool execution via ToolRegistry
  • LLM steps via Ollama HTTP API
  • Telegram channel adapter with allowlist
  • Discord channel adapter with allowlist
  • Agent loop with compression and tool-calling

Impact

  • Users: Can now interact with Terraphim via Telegram/Discord/CLI
  • Developers: Clean architecture for extending channels and tools
  • Operations: Gateway mode enables multi-user deployments

Success Criteria

  • All tests pass (220+)
  • Clippy clean (0 warnings)
  • Pre-commit hooks pass
  • Manual testing confirms functionality

Current State Analysis

Existing Implementation

| Component | Location | Purpose | Status | |-----------|----------|---------|--------| | SkillExecutor | crates/terraphim_tinyclaw/src/skills/executor.rs | Execute skill workflows | Functional | | Agent Loop | crates/terraphim_tinyclaw/src/agent/agent_loop.rs | Tool-calling conversation | Functional | | Telegram | crates/terraphim_tinyclaw/src/channels/telegram.rs | Telegram bot adapter | Feature-gated | | Discord | crates/terraphim_tinyclaw/src/channels/discord.rs | Discord bot adapter | Feature-gated | | Onboarding | crates/terraphim_agent/src/onboarding/ | CLI setup wizard | Complete | | TUI/REPL | crates/terraphim_agent/src/main.rs | Interactive interface | Complete |

Data Flow

  1. CLI Mode: User input → SkillExecutor → Output
  2. Agent Mode: CLI channel → MessageBus → AgentLoop → Response
  3. Gateway Mode: Telegram/Discord → MessageBus → AgentLoop → Response

Integration Points

  • Ollama: HTTP API at http://127.0.0.1:11434/api/generate
  • Telegram: teloxide crate with bot token
  • Discord: serenity crate with bot token
  • Tools: ToolRegistry with filesystem, web, shell tools

Constraints

Technical Constraints

  • Rust 2021 edition, tokio async runtime
  • Feature-gated channel adapters (telegram, discord)
  • Ollama required for LLM functionality
  • ToolRegistry optional but recommended

Business Constraints

  • Must pass CI/CD with all tests
  • Pre-commit hooks required (fmt, clippy, build, test, UBS)
  • Documentation must be complete for user-facing features

Non-Functional Requirements

| Requirement | Target | Current | |-------------|--------|---------| | Shell timeout | 120s | Implemented | | Test coverage | >80% | Unknown (no coverage report) | | Clippy warnings | 0 | Achieved | | Documentation | Complete | Partial gaps |

Vital Few (Essentialism)

Essential Constraints (Max 3)

| Constraint | Why It's Vital | Evidence | |------------|----------------|----------| | Security: No secrets in logs | Prevents token leakage | Telegram token logged partially | | Reliability: Gateway outbound dispatch | Without this, responses dropped | Fixed in PR but needs testing | | Usability: Clear TUI/REPL distinction | Users confused about offline mode | Addressed but needs validation |

Eliminated from Scope

| Eliminated Item | Why Eliminated | |-----------------|----------------| | Matrix channel adapter | Disabled due to sqlite conflict | | Voice transcription | Whisper not integrated | | Advanced scheduling | Not in top 5 priorities |

Dependencies

Internal Dependencies

| Dependency | Impact | Risk | |------------|--------|------| | terraphim_config | Role/Haystack types | Low - stable | | terraphim_automata | Thesaurus loading | Low - stable | | terraphim_types | Core types | Low - stable |

External Dependencies

| Dependency | Version | Risk | Alternative | |------------|---------|------|-------------| | teloxide | 0.13 | Low | None for Telegram | | serenity | 0.12 | Low | None for Discord | | reqwest | workspace | Low | None |

Risks and Unknowns

Known Risks

| Risk | Likelihood | Impact | Mitigation | |------|------------|--------|------------| | Token leakage in logs | Medium | High | Review all log statements | | Missing error boundaries | Medium | Medium | Add channel error handling | | No rate limiting | High | Medium | Document limits, add config | | Gateway message loss | Low | High | Add outbound dispatch tests |

Open Questions

  1. Load testing: How many concurrent sessions can the gateway handle? - Needs benchmark
  2. Ollama fallback: What happens when Ollama is unreachable for extended periods? - Needs graceful degradation spec
  3. Session storage: Sessions stored to disk - what's the cleanup policy? - Needs retention config

Assumptions Explicitly Stated

| Assumption | Basis | Risk if Wrong | Verified? | |------------|-------|---------------|-----------| | Ollama runs locally on 11434 | Default config | LLM steps fail | Documented | | ToolRegistry always available | Gateway mode uses default registry | Tool steps show placeholder | Test exists | | Telegram/Discord tokens valid | Runtime check | Channel fails to start | Error logged |

GAP IDENTIFICATION

Based on comprehensive analysis, the following gaps have been identified:

GAP-001: Telegram Token Logging

Location: crates/terraphim_tinyclaw/src/channels/telegram.rs:55 Issue: Token prefix logged (token[..10]) - partial token exposure Severity: Medium Evidence:

log::info!(
    "Telegram bot starting (token: {}...)",
    &self.config.token[..self.config.token.len().min(10)]
);

GAP-002: Discord Token Logging

Location: crates/terraphim_tinyclaw/src/channels/discord.rs:76 Issue: Similar token prefix logging Severity: Medium

GAP-003: Missing Gateway Outbound Dispatch Tests

Location: crates/terraphim_tinyclaw/src/main.rs:155-169 Issue: Critical fix for gateway mode has no specific tests Severity: High Evidence: Comment says "Dispatch outbound messages to channels" but no test verifies this loop

GAP-004: No Channel Error Recovery

Location: crates/terraphim_tinyclaw/src/channels/telegram.rs:73, discord.rs:94 Issue: Channel panics/failures not handled gracefully Severity: Medium Impact: One channel failure could crash entire gateway

GAP-005: Session Storage Cleanup Policy

Location: crates/terraphim_tinyclaw/src/session.rs Issue: Sessions persist forever - no retention/expiration Severity: Medium Impact: Unbounded disk growth

GAP-006: No Rate Limiting

Location: crates/terraphim_tinyclaw/src/agent/agent_loop.rs Issue: No protection against message floods Severity: Medium Impact: Gateway could be overwhelmed

GAP-007: Matrix Adapter Disabled

Location: crates/terraphim_tinyclaw/Cargo.toml:50-52 Issue: Matrix feature commented out due to rusqlite conflict Severity: Low Impact: Third major platform unsupported

GAP-008: SkillExecutor No ToolRegistry Error

Location: crates/terraphim_tinyclaw/src/skills/executor.rs:311-317 Issue: Returns placeholder message instead of error when no registry Severity: Low Evidence: Returns descriptive string instead of failing

GAP-009: Health Check Endpoint Missing

Location: N/A - Gateway mode Issue: No health check for load balancers Severity: Medium Impact: Can't deploy with k8s or load balancers properly

GAP-010: Configuration Validation

Location: crates/terraphim_tinyclaw/src/config.rs Issue: No validation of Telegram/Discord tokens on startup Severity: Low Impact: Failures discovered only when channel starts

GAP-011: Documentation - Gateway Mode

Location: docs/ Issue: No documentation for running gateway mode Severity: Medium Impact: Users can't self-host multi-channel bot

GAP-012: Documentation - Channel Setup

Location: docs/ Issue: No guide for setting up Telegram/Discord bots Severity: Medium Impact: Barrier to using channel features

GAP-013: Load/Stress Testing

Location: Tests Issue: No tests for concurrent sessions or load Severity: Low Impact: Unknown capacity limits

GAP-014: Graceful Degradation Spec

Location: Agent design Issue: No documented behavior when Ollama/proxy unavailable Severity: Medium Impact: Inconsistent user experience

GAP-015: CLI vs Agent Feature Parity

Location: crates/terraphim_agent/src/main.rs vs terraphim_tinyclaw/src/main.rs Issue: Some commands available in one but not the other Severity: Low Impact: User confusion about which tool to use

Research Findings

Key Insights

  1. Security: Token logging needs sanitization (GAP-001, GAP-002)
  2. Reliability: Gateway dispatch is critical but untested (GAP-003)
  3. Operations: Missing observability (health checks, metrics) (GAP-009)
  4. Documentation: Gateway deployment guide needed (GAP-011, GAP-012)

Technical Spikes Needed

| Spike | Purpose | Estimated Effort | |-------|---------|------------------| | Load testing | Determine concurrent session capacity | 4 hours | | Security audit | Verify no other token leakage | 2 hours | | Recovery testing | Test channel failure scenarios | 3 hours |

Recommendations

Proceed/No-Proceed

PROCEED - Implementation is solid foundation but address HIGH/MEDIUM gaps before production.

Priority Matrix

| Priority | Gaps | Action | |----------|------|--------| | P0 (Critical) | GAP-003 | Add gateway dispatch tests before merge | | P1 (High) | GAP-001, GAP-002, GAP-004, GAP-009 | Fix in follow-up PR | | P2 (Medium) | GAP-005, GAP-006, GAP-011, GAP-012, GAP-014 | Document or defer | | P3 (Low) | GAP-007, GAP-008, GAP-010, GAP-013, GAP-015 | Backlog |

Risk Mitigation

  1. Before merge: Ensure GAP-003 tested
  2. Week 1: Address security gaps (GAP-001, GAP-002)
  3. Week 2: Add operational features (GAP-009)
  4. Month 1: Complete documentation (GAP-011, GAP-012)

Next Steps

  1. Review this research document
  2. Prioritize gaps based on deployment timeline
  3. Create issues for P0/P1 gaps
  4. Proceed to design phase for prioritized gaps

Appendix

Reference Materials

  • PR #529: https://github.com/terraphim/terraphim-ai/pull/529
  • Issue #519: https://github.com/terraphim/terraphim-ai/issues/519
  • Files changed: 22 files (+1,711 -125)

Test Summary

  • 220+ tests passing
  • 13 integration tests (tinyclaw)
  • 3 benchmark tests
  • Clippy: 0 warnings
Disciplined Research: Current PR and Issue Portfolio Implementation Plan: PR #529 Gap Coverage