#21
Strata is a free, open-source project that enables users to run a 125-billion-parameter AI model locally on a consumer gaming PC, requiring only a 12 GB+ NVIDIA or AMD GPU and 32 GB+ of RAM on Windows or Linux. The core model it runs is Qwen3.8-Flash-Next, a large language model typically requiring server infrastructure, now made accessible for local chat, code generation, image reading, and integration with coding agents and apps. Strata exposes an OpenAI-compatible API endpoint at localhost, meaning it works seamlessly with tools like Claude Code, Cursor, Codex CLI, and GitHub Copilot as a drop-in local backend. Installation is straightforward via a batch file or shell script, with an AI-assisted setup path that auto-detects hardware and selects the appropriate quantized model variant. Multiple quantization sizes are offered — Q2_0, IQ2_XS, IQ3_XXS, IQ3_S, and others from Unsloth — balancing speed and quality depending on available RAM (32 GB to 96 GB+). Benchmarks show impressive throughput: up to 94 tokens/second generation and 2,650 tokens/second prompt ingestion on an RTX 5070, and 60 tokens/second on an AMD RX 9070 XT. The project supports multi-GPU setups, experimental Apple Silicon (M1–M5), Intel Arc, AMD Ryzen AI Max, and older GPUs, with community-contributed benchmarks and documentation. Advanced tuning options like async expert caching and RAM pinning offer additional speed gains. The project is at an early but active release stage (v0.1.10), with a browser-based chat UI, live GPU/CPU/RAM monitor, and MCP server support. Its appeal for GitHub trending lies in democratizing frontier-scale AI inference on commodity hardware, with broad hardware compatibility and a polished, beginner-friendly setup experience.
ServingAppsMCPlocal-firston-devicemultimodal
★ 27,591 total+5,757 this week
#31
FreeLLMAPI is an open-source aggregation and routing layer that consolidates the free tiers of 34 AI inference providers into a single OpenAI-compatible API endpoint, delivering approximately 7.4 billion tokens per month of free inference capacity across 474 model families and 635 provider endpoints. The core premise is straightforward but powerful: every major AI lab now offers a free tier, but each one comes with its own SDK, rate limits, and failure modes. Managing them individually is impractical, so FreeLLMAPI collapses all of them behind a single `/v1` endpoint that any OpenAI-compatible client can point to without modification. The project is available as a desktop application for macOS and Windows, as a Docker-deployable self-hosted server, and as mobile apps on both Google Play and the Apple App Store, making it accessible across virtually every platform a developer might use.
The supported provider list is extensive and includes major names such as Google, Groq, Cerebras, Mistral, OpenRouter, Cloudflare, Cohere, NVIDIA, HuggingFace, Z.ai (Zhipu), ModelScope, and OpenCode Zen, along with 22 additional free providers. Beyond these, the project supports a "custom" provider slot that allows users to point the router at any OpenAI-compatible endpoint, including local inference servers like llama.cpp, LM Studio, vLLM, and Ollama, or remote gateways. This makes FreeLLMAPI useful not just for aggregating cloud free tiers but also for unifying local and remote inference under one interface.
One of the most technically notable aspects of FreeLLMAPI is its self-updating model catalog. The router pulls a signed catalog from freellmapi.co twice a day, meaning new models, quota changes, and compatibility fixes propagate automatically without requiring users to update their installation. Free installs receive the monthly snapshot, so new models reach them approximately 30 days after joining the live feed, while premium subscribers ($19/year) receive catalog updates the same day they are published. This live-catalog approach directly addresses one of the most painful aspects of working with free tiers: the free-tier landscape shifts weekly as providers launch models, retire them, and change quotas without notice.
The routing engine is sophisticated, implementing six distinct routing strategies that rank available models using live per-model speed, capability, and reliability scores. When a provider returns a 429 rate-limit error or a 5xx server error, the router automatically falls over to the next available model in the chain, applying cooldowns and rotating keys as needed. Per-key rate tracking maintains RPM, RPD, TPM, and TPD counters per platform-model-key combination, learning providers' reported ceilings so that routing always stays within every free-tier cap. API keys are stored encrypted on the local installation.
The API surface coverage is comprehensive. FreeLLMAPI exposes every major OpenAI-style endpoint: `/v1/chat/completions`, `/v1/responses` (required by Codex CLI), `/v1/completions` for editor ghost-text autocomplete, `/v1/images/generations`, `/v1/videos/generations`, `/v1/audio/speech`, `/v1/audio/transcriptions`, `/v1/embeddings`, and `/v1/models`, all supporting both streaming and non-streaming modes. Beyond OpenAI compatibility, the project also implements the Anthropic Messages API at `/v1/messages`, enabling Claude Code and official Anthropic SDKs to run against the free pool. Native Gemini surface support via `/v1beta` endpoints covers `generateContent`, streaming, token counting, and model listing, while an opt-in Ollama emulation layer serves NDJSON chat, generate, tags, metadata, and embeddings for clients like Zed and JetBrains AI.
A particularly interesting feature is "Fusion," a multi-model synthesis mode accessible by requesting the virtual `fusion` model. When invoked, the router fans a prompt out to a panel of diverse free models in parallel, then uses a judge model to synthesize a single answer from the collected drafts. This ensemble approach can produce higher-quality outputs than any single free model alone. Tool calling and structured outputs are also fully supported, with OpenAI-style `tools` round-tripping across providers, including rescue logic that converts plain-text tool calls into proper `tool_calls` objects. Standard sampling parameters like `response_format`, `seed`, `logprobs`, and penalties are passed through on a per-provider basis.
The project supports a wide ecosystem of coding agents and CLI tools, including Claude Code, Codex CLI, Gemini CLI, Cursor, Cline, Roo Code, OpenCode, Aider, Continue, Goose, Qwen Code, Kilo Code, Zed, JetBrains AI, and many others. Most agents can be configured with a single `npx freellmapi setup-<agent>` command that reads the live catalog, backs up existing configuration, and merges the new settings automatically. Unified model profiles and named fallback chains (such as a coding chain or a vision chain) can be switched from the dashboard or specified per request using the `auto:<profile>` syntax, and custom chains can be renamed in place from the chain manager.
In terms of maturity and positioning, FreeLLMAPI presents itself as a production-ready tool with CI badges, a Docker image hosted on GitHub Container Registry, MIT licensing, and multilingual documentation in English, Simplified Chinese, and Russian. It compares itself directly against OpenRouter, LiteLLM, and Portkey in a feature comparison table. The project is trending on GitHub likely because it addresses a genuinely widespread developer pain point — the desire to use powerful LLMs for free at meaningful scale — while offering a polished, multi-platform experience with a self-sustaining update mechanism. The combination of 7.4 billion free tokens per month, broad agent compatibility, and a low-friction setup makes it a compelling tool for developers, researchers, and hobbyists who want to experiment with
ServingAgentsMCPlocal-firstmultimodalworkflow-automation
★ 37,172 total+3,760 this week
#75
Colibri is an open-source, pure C inference engine designed to run very large mixture-of-experts (MoE) language models on ordinary consumer hardware — no GPU required, though one is supported and exploited when available. The project's tagline, "tiny engine, immense model," captures its core philosophy: keep the codebase minimal and auditable while enabling models with hundreds of billions of parameters to run on machines people already own. The key technical insight driving colibri is that MoE models, despite their enormous total parameter counts, only activate a small fraction of their weights per token — the so-called routed experts. Colibri exploits this sparsity by keeping the dense components (attention layers, shared experts, embeddings) resident in RAM while streaming the routed experts from disk on demand, through a cache that learns which experts a given workload tends to use. A GPU, when present, holds the hottest experts and dense layers in VRAM, further accelerating inference. This architecture means the model does not need to fit entirely in fast memory; it needs to be intelligently placed across the memory hierarchy.
The project currently ships thirteen engines. Ten target language models: GLM-5.2 and GLM-5.3 (a 744B MoE), GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, MiMo-V2.6 Flash (and Pro), Qwen3.8-Flash-Next, Qwen3.6 (which also covers Qwen3-Coder and the dense Qwen3.8-27B), and OLMoE. One engine handles image generation via Qwen-Image-2.1. Two engines handle decision tasks — Laya and GLiNER2.5-Decide — with a third decision model, Clef, running on the Qwen3.6 engine. This breadth of supported model families across language, vision, and decision tasks is notable for a project of this scope and minimalism.
Getting started is designed to be a single-step process. On Windows, users unzip the repository and double-click START-HERE.bat; on Linux and macOS, a shell script handles everything. The setup script automatically inspects the machine's RAM, free disk space, CPU, and available GPUs, then recommends the most capable model that fits the hardware profile. It builds the engine from source when a compiler is available, targeting CUDA for NVIDIA GPUs on Linux when the CUDA toolkit is present, Vulkan for broader GPU support on Linux and Windows, and falling back gracefully to a CPU-only prebuilt binary otherwise. Integrated GPUs are also supported for select engines where benchmarks showed them to be faster than the CPU alone. Model downloads are resumable, so interrupted downloads can be continued without restarting. After setup, colibri launches a local web dashboard and exposes both OpenAI-compatible and Anthropic-compatible API endpoints, making it a drop-in local backend for existing AI tooling and coding assistants.
The minimum hardware requirement is 8 GB of RAM and 22 GB of free disk space for the smallest model, with 16 GB or more recommended. Larger models like GLM-5.2 require 429 GB of download and at least 16–24 GB of RAM, with disk streaming making up the difference. Measured decode speeds range from around 0.05–0.1 tokens per second cold on a 25 GB laptop for GLM-5.2, up to 30 tokens per second with CUDA for the smaller Qwen3.6-35B-A3B model. The project also supports the Model Context Protocol (MCP), allowing AI coding assistants to autonomously detect hardware, recommend models, install, start, stop, and monitor colibri through a dedicated MCP server.
Colibri is explicitly positioned as an open research platform. Its design philosophy mandates that any optimization must be validated with a reproducible end-to-end benchmark, and the default policy never silently alters model precision or router semantics. The dashboard visualizes every expert activation in real time, making the inference process transparent and inspectable. The codebase is structured as one C file per model family, keeping it compact and auditable. The project is free and open source, with the author accepting voluntary coffee donations to fund continued development. README translations are available in Simplified Chinese, Traditional Chinese, Italian, Japanese, and Indonesian, suggesting an intentionally international audience. The combination of extreme hardware accessibility, broad model support, API compatibility, MCP integration, and research-grade transparency makes colibri a compelling and distinctive entry in the local LLM inference space, explaining its presence on GitHub Trending.
ServingModelsMCPlocal-firston-devicemultimodal
★ 42,773 total+1,467 this week
#129NEWPartial week
Project Maya is an open-source inference engine that enables running GLM-5.3-Flash, a massive 321-billion-parameter Mixture-of-Experts language model, entirely on consumer or prosumer hardware without sending data to any external server. The project's core innovation is a purpose-built engine that intelligently tiers model experts across GPU VRAM, system RAM, and NVMe SSD, dynamically moving experts as a conversation progresses and auto-calibrating to the specific hardware profile of each machine. It supports one to sixteen NVIDIA GPUs (compute capability 7.0+, i.e., V100 and RTX 20 series onward), with experimental AMD and Windows support also available. The project ships four quantization variants — Maya-S, Maya-S24, Maya-M, and Maya-L — ranging from roughly 95 GB to 156 GB, each retaining between 97.7% and 99.2% of the original FP8 model's zero-shot accuracy, with the setup wizard recommending the best fit for a user's hardware. Performance benchmarks contributed by real users show decode speeds from around 14 tokens/second on a laptop RTX 5090 up to 118 tokens/second on four RTX 4090s, with prefill rates reaching over 4,000 tokens/second on high-end multi-GPU rigs. The engine supports up to 1 million token context windows and handles up to eight concurrent requests, batching them efficiently so dense weights are read once for all parallel sessions. It exposes both OpenAI-compatible and Anthropic-compatible REST APIs, making it a drop-in backend for coding agents like Claude Code, and includes streamed tool calls, thinking-level controls, and CORS configuration. A browser-based chat interface and a live hardware monitor are bundled out of the box, and setup is a single shell script that builds everything locally without system-wide installation. The project also supports a community-contributed "derisked" variant (Maya-M-Derisked) that reduces refusals via a weight edit. Led by Peasant Smith and co-maintained actively, with a growing community of benchmark contributors, Project Maya is notable for democratizing frontier-scale LLM inference on private, owned hardware — a capability previously confined to datacenters.
ServingModelsAppslocal-firston-devicemultimodal
★ 1,042 total+595 this week