Context Collections Management

Guide to organizing, managing, and switching between context collections in Terraphim for optimal AI-assisted development.

What are Context Collections?

Context collections are named sets of documents, vibe-rules, and configuration organized for specific development contexts. Think of them as workspaces or profiles that include:

Documents: Reference materials, architecture docs, API documentation
Vibe-Rules: Coding standards, patterns, and best practices
Configuration: LLM settings, haystack configurations, relevance functions
Knowledge Graph: Pre-built semantic relationships between concepts

Collections enable quick context switching without losing accumulated knowledge.

Collection Types

1. Domain Collections

Organized by technical domain or area of expertise.

Examples:

web-backend: API design, database patterns, authentication
cli-tools: Argument parsing, POSIX conventions, error messages
wasm: Memory management, JS interop, size optimization
embedded: Hardware constraints, real-time, power management
data-science: Algorithm notebooks, ML patterns, visualization

2. Project Collections

Project-specific knowledge and conventions.

Examples:

project-alpha: Alpha project architecture, API contracts, team conventions
client-beta: Client-specific patterns, business rules, integration docs
internal-tools: Company-wide standards, authentication patterns, logging

3. Language Collections

Language-specific patterns and idioms.

Examples:

rust-advanced: Async patterns, unsafe code, FFI, macros
python-data: NumPy patterns, Pandas idioms, ML pipelines
typescript-react: Component patterns, hooks, state management

4. Learning Collections

Educational materials and tutorials.

Examples:

rust-beginner: Ownership basics, borrowing, lifetimes
tokio-tutorial: Async fundamentals, runtime, channels
kubernetes-ops: Deployment patterns, monitoring, debugging

Directory Structure

docs/
├── context-collections/
│   ├── web-backend/
│   │   ├── collection.json          # Collection metadata
│   │   ├── architecture/
│   │   │   ├── api-design.md
│   │   │   ├── auth-patterns.md
│   │   │   └── database-access.md
│   │   ├── vibe-rules/
│   │   │   ├── rest-api-conventions.md
│   │   │   ├── error-responses.md
│   │   │   └── rate-limiting.md
│   │   └── references/
│   │       ├── http-status-codes.md
│   │       └── jwt-claims.md
│   │
│   ├── cli-tools/
│   │   ├── collection.json
│   │   ├── patterns/
│   │   │   ├── argument-parsing.md
│   │   │   ├── subcommands.md
│   │   │   └── configuration.md
│   │   └── vibe-rules/
│   │       ├── error-messages.md
│   │       ├── output-formatting.md
│   │       └── exit-codes.md
│   │
│   └── wasm/
│       ├── collection.json
│       ├── architecture/
│       │   ├── memory-management.md
│       │   └── js-interop.md
│       └── vibe-rules/
│           ├── wasm-bindgen-patterns.md
│           └── performance-tips.md

Collection Metadata

Each collection has a collection.json file:

{
  "name": "Web Backend Development",
  "shortname": "web-backend",
  "description": "Context for building web backend services with REST APIs, databases, and authentication",
  "version": "1.0.0",
  "tags": ["web", "backend", "api", "database"],
  "languages": ["rust", "python", "typescript"],
  "frameworks": ["axum", "salvo", "fastapi", "express"],
  "created": "2025-01-15",
  "updated": "2025-01-20",
  "author": "Engineering Team",
  "token_estimate": 15000,
  "dependencies": [
    "global/naming-conventions",
    "global/documentation-standards"
  ],
  "recommended_llm": {
    "provider": "ollama",
    "model": "llama3.2:3b",
    "temperature": 0.7,
    "system_prompt_template": "web-backend-engineer"
  },
  "haystacks": [
    {
      "location": "docs/context-collections/web-backend",
      "service": "Ripgrep",
      "priority": 1
    },
    {
      "location": "docs/vibe-rules/global",
      "service": "Ripgrep",
      "priority": 2
    }
  ]
}

Creating Collections

Method 1: Directory-Based (Simple)

# Create collection directory
mkdir -p docs/context-collections/my-collection/{architecture,patterns,vibe-rules,references}

# Add documents
cat > docs/context-collections/my-collection/patterns/example-pattern.md <<EOF
# Example Pattern
#my-tag #pattern

Pattern description here...
EOF

# Create collection metadata
cat > docs/context-collections/my-collection/collection.json <<EOF
{
  "name": "My Collection",
  "shortname": "my-collection",
  "description": "Description here",
  "tags": ["example"]
}
EOF

Method 2: Symlink-Based (Reuse Existing)

# Create collection directory
mkdir -p docs/context-collections/my-collection

# Symlink existing vibe-rules
ln -s ../../../vibe-rules/rust/async-patterns.md \
  docs/context-collections/my-collection/

ln -s ../../../vibe-rules/global/naming-conventions.md \
  docs/context-collections/my-collection/

# Create collection metadata
cat > docs/context-collections/my-collection/collection.json <<EOF
{
  "name": "My Collection",
  "shortname": "my-collection",
  "description": "Curated rules from global and rust collections",
  "tags": ["rust", "async"]
}
EOF

Method 3: CLI Tool (Planned)

# Create collection from template
terraphim collection create web-backend \
  --template=web-backend \
  --languages=rust,python \
  --frameworks=axum,fastapi

# Add documents
terraphim collection add web-backend \
  --path=docs/api-design.md

# Import vibe-rules
terraphim collection import web-backend \
  --rules=rust/async-patterns,global/naming-conventions

Switching Collections

Using Role Configuration

Create a role per collection:

{
  "roles": {
    "Web Backend Engineer": {
      "name": "Web Backend Engineer",
      "relevance_function": "terraphim-graph",
      "kg": {
        "knowledge_graph_local": {
          "input_type": "markdown",
          "path": "docs/context-collections/web-backend"
        }
      },
      "haystacks": [
        {
          "location": "docs/context-collections/web-backend",
          "service": "Ripgrep"
        },
        {
          "location": "docs/vibe-rules/global",
          "service": "Ripgrep"
        }
      ],
      "llm_system_prompt": "You are an expert web backend engineer..."
    },
    "CLI Tools Developer": {
      "name": "CLI Tools Developer",
      "relevance_function": "terraphim-graph",
      "kg": {
        "knowledge_graph_local": {
          "input_type": "markdown",
          "path": "docs/context-collections/cli-tools"
        }
      },
      "haystacks": [
        {
          "location": "docs/context-collections/cli-tools",
          "service": "Ripgrep"
        }
      ],
      "llm_system_prompt": "You are an expert CLI tool developer..."
    }
  }
}

Switch between collections by changing the selected role:

# Via API
curl -X POST http://localhost:PORT/config \
  -d '{"selected_role": "Web Backend Engineer"}'

# Via TUI
terraphim-agent roles select "Web Backend Engineer"

# Via desktop UI
# Settings → Roles → Select "Web Backend Engineer"

Dynamic Collection Loading

Load collections on-demand without restarting:

// Via Terraphim service API
let service = TerraphimService::new(config_state);

// Load new collection
service.load_collection("web-backend").await?;

// Switch active collection
service.activate_collection("web-backend").await?;

// Knowledge graph rebuilds automatically

Collection Management

Listing Collections

# List available collections
ls docs/context-collections/

# Get collection metadata
cat docs/context-collections/web-backend/collection.json | jq

# Via API
curl http://localhost:PORT/collections

Exporting Collections

Export collection as shareable archive:

# Create collection archive
tar czf web-backend-collection.tar.gz \
  docs/context-collections/web-backend/

# Include dependencies
tar czf web-backend-full.tar.gz \
  docs/context-collections/web-backend/ \
  docs/vibe-rules/global/

# Share with team
scp web-backend-collection.tar.gz team@server:/collections/

Importing Collections

Import collection from archive:

# Extract collection
tar xzf web-backend-collection.tar.gz -C docs/context-collections/

# Verify collection
ls docs/context-collections/web-backend/

# Rebuild knowledge graph
cargo run -- --config context_engineer_config.json

Versioning Collections

Use git to version collections:

# Tag collection version
git tag -a web-backend-v1.0 -m "Web Backend Collection v1.0"

# View collection history
git log docs/context-collections/web-backend/

# Rollback to previous version
git checkout web-backend-v0.9 -- docs/context-collections/web-backend/

# Create branch for experimental changes
git checkout -b web-backend-experiment

Collection Composition

Hierarchical Collections

Collections can inherit from parent collections:

{
  "name": "Project Alpha Backend",
  "shortname": "project-alpha-backend",
  "extends": "web-backend",
  "overrides": {
    "llm_system_prompt": "You are working on Project Alpha backend..."
  },
  "additional_haystacks": [
    {
      "location": "projects/alpha/docs",
      "service": "Ripgrep"
    }
  ]
}

Resolution order:

Project-specific collection
Parent collection (web-backend)
Global vibe-rules
Language-specific vibe-rules

Collection Mixing

Combine multiple collections:

{
  "name": "Full Stack Developer",
  "shortname": "fullstack",
  "combines": [
    "web-backend",
    "typescript-react",
    "database-admin"
  ],
  "haystacks": [
    // Haystacks from all three collections merged
  ]
}

Collection Overlays

Temporary modifications without changing base collection:

{
  "name": "Web Backend (Performance Focus)",
  "shortname": "web-backend-perf",
  "base": "web-backend",
  "overlay": {
    "additional_rules": [
      "performance/latency-optimization.md",
      "performance/memory-profiling.md"
    ],
    "llm_temperature": 0.5  // More focused
  }
}

Token Management

Estimating Collection Size

# Count tokens in collection
find docs/context-collections/web-backend -name "*.md" -exec wc -w {} + | tail -1

# Estimate with metadata
jq '.token_estimate' docs/context-collections/web-backend/collection.json

Optimizing Collection Size

Strategies:

Selective Loading: Only load relevant subcollections
Document Pruning: Remove obsolete or rarely-used docs
Snippet Extraction: Store only relevant snippets, not full docs
Lazy Loading: Load documents on-demand, not upfront

Example:

{
  "lazy_loading": {
    "enabled": true,
    "load_threshold": 10000,  // tokens
    "priority_tags": ["critical", "core"]
  }
}

Token Budget Per Context

Configure per-collection token budgets:

{
  "token_budget": {
    "total": 20000,
    "allocation": {
      "system_prompt": 500,
      "vibe_rules": 5000,
      "architecture": 8000,
      "code_examples": 5000,
      "references": 1500
    }
  }
}

Best Practices

1. Keep Collections Focused

Good:

web-backend/
├── api-design.md
├── auth-patterns.md
└── database-access.md

Bad:

web-backend/
├── api-design.md
├── mobile-ui-patterns.md  # Wrong domain
├── ml-algorithms.md       # Wrong domain
└── database-access.md

2. Use Consistent Structure

All collections should follow same structure:

collection-name/
├── collection.json
├── architecture/
├── patterns/
├── vibe-rules/
└── references/

3. Document Dependencies

{
  "dependencies": [
    "global/naming-conventions",
    "global/documentation-standards",
    "rust/error-handling"
  ]
}

4. Tag Aggressively

# API Design Pattern
#api #rest #http #design #web #backend

Pattern description...

More tags = better search and discovery.

5. Include Examples

Every pattern should have working examples:

# Pattern Name

## Example
\`\`\`rust
// Working code
\`\`\`

## Anti-Example
\`\`\`rust
// What not to do
\`\`\`

6. Version Control Everything

# Track all changes
git add docs/context-collections/
git commit -m "docs(web-backend): add rate limiting pattern"

# Tag stable versions
git tag web-backend-v1.1

7. Measure Usage

Track which collections are most useful:

# Search frequency per collection
curl http://localhost:PORT/stats/collections

# Most referenced documents
curl http://localhost:PORT/stats/documents?collection=web-backend

Integration with Development Tools

VS Code

Create collection-specific workspaces:

// .vscode/web-backend.code-workspace
{
  "folders": [
    {
      "path": "."
    }
  ],
  "settings": {
    "terraphim.collection": "web-backend",
    "terraphim.autoload": true
  }
}

Claude Desktop

Collection-specific MCP configurations:

{
  "mcpServers": {
    "terraphim-web": {
      "command": "/path/to/terraphim_mcp_server",
      "args": ["--config", "web_backend_config.json"]
    },
    "terraphim-cli": {
      "command": "/path/to/terraphim_mcp_server",
      "args": ["--config", "cli_tools_config.json"]
    }
  }
}

Git Hooks

Auto-rebuild knowledge graph on collection changes:

#!/bin/bash
# .git/hooks/post-commit

# Check if any collection changed
if git diff-tree -r --name-only --no-commit-id HEAD | grep "context-collections"; then
  echo "Collection changed, rebuilding knowledge graph..."
  cargo run --release -- --rebuild-kg
fi

Troubleshooting

Collection Not Loading

# Verify collection exists
ls docs/context-collections/my-collection/

# Check collection.json is valid
cat docs/context-collections/my-collection/collection.json | jq

# Verify role configuration
cat config.json | jq '.roles["My Role"].kg.knowledge_graph_local.path'

Search Returns Wrong Results

# Check active collection
curl http://localhost:PORT/config | jq '.selected_role'

# Verify haystack configuration
curl http://localhost:PORT/config | jq '.roles[] | .haystacks'

# Rebuild knowledge graph
cargo run -- --config config.json --rebuild

Token Budget Exceeded

# Check collection size
find docs/context-collections/web-backend -name "*.md" -exec wc -w {} +

# Reduce collection:
# 1. Remove rarely-used documents
# 2. Enable lazy loading
# 3. Use document summaries instead of full text

Future Enhancements

Planned features:

Collection Marketplace: Share collections with community
Auto-Generated Collections: ML-based collection creation from codebases
Collection Analytics: Track which documents help most
Smart Merging: Automatically merge overlapping collections
Collection Testing: Verify collection quality with test queries
Cloud Sync: Sync collections across devices

Examples

Web Backend Collection

See: docs/context-collections/web-backend/

Includes:

REST API design patterns
Authentication/authorization
Database access patterns
Error handling conventions
Rate limiting strategies

CLI Tools Collection

See: docs/context-collections/cli-tools/

Includes:

Argument parsing patterns
Subcommand structure
Error message formatting
Configuration file handling
POSIX conventions

WASM Collection

See: docs/context-collections/wasm/

Includes:

Memory management patterns
JS interop with wasm-bindgen
Performance optimization
Size optimization
Error handling across boundaries

Context Collections Management

What are Context Collections?

Collection Types

1. Domain Collections

2. Project Collections

3. Language Collections

4. Learning Collections

Directory Structure

Collection Metadata

Creating Collections

Method 1: Directory-Based (Simple)

Method 2: Symlink-Based (Reuse Existing)

Method 3: CLI Tool (Planned)

Switching Collections

Using Role Configuration

Dynamic Collection Loading

Collection Management

Listing Collections

Exporting Collections

Importing Collections

Versioning Collections

Collection Composition

Hierarchical Collections

Collection Mixing

Collection Overlays

Token Management

Estimating Collection Size

Optimizing Collection Size

Token Budget Per Context

Best Practices

1. Keep Collections Focused

2. Use Consistent Structure

3. Document Dependencies

4. Tag Aggressively

5. Include Examples

6. Version Control Everything

7. Measure Usage

Integration with Development Tools

VS Code

Claude Desktop

Git Hooks

Troubleshooting

Collection Not Loading

Search Returns Wrong Results

Token Budget Exceeded

Future Enhancements

Examples

Web Backend Collection

CLI Tools Collection

WASM Collection

See Also