STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming

STREAM (Smart Tiered Routing Engine for AI Models) introduces a multi-tier inference middleware that integrates local, HPC, and cloud resources for large language model (LLM) applications. Key features include a three-tier routing architecture, a dual-channel HPC streaming system achieving a median token transfer time (TTFT) of 0.54 seconds, and an HPC-as-API proxy mode that allows seamless access to HPC resources via an OpenAI-compatible endpoint. This solution addresses the challenges of model size and context limitations while providing significant performance improvements, making it easier for practitioners to utilize HPC resources without requiring extensive expertise.