Expert LLMFit (BETA)

This skill provides the domain expertise required to maintain the LlmFitEngine inside remote-affiliate-api/src/core/llmfit.engine.ts and keep it synchronized with the upstream logic from https://github.com/AlexsJones/llmfit.

Core Concepts & Formulas

1. Inference is Memory-Bandwidth-Bound:

• Theoretical Max TPS = Memory Bandwidth (GB/s) / Model Size (GB)
• Real TPS Estimate = Max TPS * Efficiency Factor (0.55) * RunMode Penalty.
• The efficiency factor (0.55) accounts for kernel launch overhead and KV-cache reads.

1. RunMode Penalties:

• gpu: 1.0 (Full speed)
• moe_offload: 0.8 (Expert switching latency)
• cpu_gpu: 0.5 (Partial offload penalty)
• cpu_only: 0.3 (Fallback to slow RAM)

1. Memory Requirements (VRAM/RAM):

• Base Model Size = Parameters * BytesPerParam
• FP16/BF16: 2.0 bytes
• Q8: 1.0 byte
• Q4_K_M: 0.5 bytes (Standard for local inference recommendations)
• KV Cache Overhead: Context length requires additional VRAM.
• Rule of Thumb VRAM: (Model Size in GB * 1.2) to account for context and OS overhead.

1. Mixture of Experts (MoE):

• Fit/VRAM checks use Total Parameters.
• Speed/TPS estimates use Active Parameters (e.g., Mixtral 8x7B uses ~12.9B active params per token).

Verification & Adaptation Workflow

Whenever asked to "verify LLMFit logic" or "update the hardware recommendation engine", follow these steps:

Step 1: Upstream Sync

• Clone the upstream repository into a temporary directory:

  mkdir -p /tmp/llmfit_external && git clone https://github.com/AlexsJones/llmfit /tmp/llmfit_external

• Inspect key Rust files:
• llmfit-core/src/fit.rs (for TPS and scoring formulas).
• llmfit-core/src/hardware.rs (for GPU bandwidth updates and compute capabilities).
• llmfit-core/src/models.rs (for Quantization multipliers and MoE active parameters).

Step 2: Codebase Audit (remote-affiliate-api)

• Compare the upstream formulas with our local TypeScript implementation in src/core/llmfit.engine.ts.
• Ensure the estimateTokensPerSecond method correctly applies the 0.55 efficiency factor and RunMode penalties.
• Verify calculateRequiredMemory properly uses active vs total parameters.

Step 3: Database Metadata Alignment

• Check prisma/schema.prisma and prisma/seed.ts.
• Ensure HardwareMetadata table contains updated bandwidthGBs for new GPUs (e.g., RTX 50 series, M4 Apple Silicon).
• Ensure LlmModel table correctly populates activeParametersBillions for MoE models during HuggingFace sync (src/services/huggingface.service.ts).

Step 4: Cleanup

• Remove the temporary upstream clone:

  rm -rf /tmp/llmfit_external

Critical Rules

• DO NOT replace our TypeScript implementation with Rust code. We only port the mathematical logic and hardware constants.
• ALWAYS default to 4-bit quantization (Q4_K_M) for consumer recommendation baselines.
• NEVER recommend a model if its minVramGb exceeds the user's available HardwareMetadata.vramGb.