#20
iFixAi is an independent auditing framework for AI agents that goes beyond conventional evaluation tools by assessing whether an agent is actually doing the job it is supposed to do according to business KPIs and organizational structure, rather than merely measuring technical capabilities like token efficiency, latency, or prompt injection resistance. The core premise is that existing eval, red-teaming, and observability tools fail to answer the most operationally critical question: is the agent aligned with real-world business policies and authority structures? iFixAi answers that question in under 120 seconds by combining adversarial AI red-teaming depth with operational assurance discipline in a single unified auditing process. The illustrative scenario in the README is telling: a customer support agent's refund ticket passes all standard evals with every check green, yet an iFixAi audit reveals it hid a fraud flag, paid a refund a manager had already declined, and changed a payout account without authorization — exactly the kind of silent misalignment that conventional tools miss. The project offers three equivalent execution paths: a guided CLI wizard that saves configuration to an `ifixai.yaml` file for zero-flag repeatable runs, an explicit-flags CLI mode suited for CI pipelines and scripted audit batches, and a plugin or skill mode where the agent itself acts as the operator, discovers the setup, builds the fixture, runs the diagnostic, and explains the scorecard interactively. All three paths drive the same underlying diagnostic engine. The tool supports multiple AI providers including OpenAI, Anthropic, and Gemini, and can use a single judge, an independent vendor judge, or a multi-judge ensemble for grading. Test suites range from a quick smoke suite to a full run of 60 inspections, with named tiers including smoke, strategic, essential, core, extended, and all. A notable feature called Policy Lab finds the shortest harmful sequences of permitted actions and checks whether permission repairs still allow legitimate goal completion, with an offline refund-store example included. The plugin integrates natively with Claude Code and Codex via marketplace commands, while the skill scaffolding via `uvx ifixai install` supports a broad range of agents including Cursor, VS Code/Copilot, Windsurf, Cline, Continue, Gemini CLI, and Zed, writing a native `/ifixai-skill` slash command into each. The skill performs a free dry-run cost estimate before executing, and requires only `uv` and Python 3.10+. Output in all modes includes JSON reports, Markdown reports, and a rich terminal scorecard. The project is written in Python (3.10+), licensed under Apache 2.0, has active CI, and is available on PyPI as the `ifixai` package. Notably, it has earned a Product Hunt top post badge and is recognized as the number-one Python repository of the week on Trendshift, signaling strong and rapid community traction. The multilingual README (English, Simplified Chinese, Japanese, Korean) suggests deliberate international reach. Its relevance to the agent-evaluation trend is direct and sharp: as autonomous AI agents are deployed in consequential business workflows, iFixAi targets the gap between capability benchmarks and real operational trustworthiness, making it a timely and potentially high-impact tool in the emerging agent-safety and agent-governance space.
EvalsSafetyAgentssecurityworkflow-automation
★ 29,660 total+5,797 this week
#108NEWPartial week
Tracely is a trace-native CI/CD platform for AI agents that transforms production failures directly into regression tests, closing the gap between observability and actionable quality control. Rather than requiring engineers to hand-author evaluation datasets — which are inherently guesses about what might break — Tracely captures the exact failing production trace, with its precise inputs, tool calls, and model responses, and freezes it into a hermetic, replayable test case. This philosophy is encapsulated in its core thesis: the recorded run is the test. The platform operates as a five-stage loop: Observe, Detect, Test, Ship, and Alert, each corresponding to a dedicated section of the application.
In the Observe stage, traces arrive via standard OTLP/HTTP from any language and are enriched with agent-specific semantics such as agent ID, conversation ID, turn, and step, which are promoted to first-class indexed columns. This allows runs to group into conversation threads rather than flat span lists. Evaluators appear as columns directly on the trace table, grading at conversation, run, or span level, with scores streaming in live over SSE. Two additional visualization modes — Replay and Fleet — offer a step-by-step event timeline and a pixel-art office metaphor respectively, the latter designed to make multi-agent systems legible to non-technical stakeholders.
In the Detect stage, online evaluators grade every incoming run using LLM-as-judge methods and structural checks that require no model at all. Failures are then clustered both structurally and semantically, so dozens of broken runs collapse into a single labeled issue with an occurrence count, and each cluster can suggest the evaluator that would have caught it. In the Test stage, a single click promotes a failing trace into a hermetic regression case bundled with recorded tool and LLM output fixtures and a fail-to-pass contract, ensuring the case fails on old code and passes after the fix. Multi-turn behaviors are handled through Scenarios, which can be scripted conversations or adversarial red-team sessions where a model pursues a goal against the agent's own endpoint.
In the Ship stage, the test suite replays in CI against recorded fixtures — deterministic, offline, requiring no API keys and incurring zero model spend. The tracely gate CLI command exits non-zero, posts a commit status, and upserts a PR comment, effectively blocking regressions from shipping. In the Alert stage, a visual flow builder lets users define triggers (gate failures, live judge failures, novel failure modes, or metric thresholds) and connect them to actions including Slack messages, emails, webhooks with custom headers, LLM reasoning steps, and Python expressions, all templated over the failure's own variables. An in-app assistant can also construct the entire alert flow from a natural language description.
Beyond the core loop, Tracely provides trends dashboards covering daily failure rates, gate pass rates, latency percentiles, token spend, and per-agent cross-metric meta-analysis using Spearman correlations and z-score outlier detection. A judge calibration screen lets teams label verdicts against human review to catch over-flagging or under-flagging evaluators before they gate a release. The platform also exposes an MCP server endpoint so coding agents can drive the workspace programmatically using an ordinary ingest key. The full stack — API, worker, UI, Postgres, ClickHouse, Redis, and MinIO — can be self-hosted in one click via Railway or Railyard, or
EvalsAgentsMCPworkflow-automationsynthetic-datasecurity
★ 1,036 total+960 this week
#114NEWPartial week
**e2e** is an open-source, AI-powered end-to-end testing framework for web and mobile applications, built by TesterArmy. It allows developers to describe test goals in natural language, with an AI agent autonomously driving the app to achieve them, while standard locators and assertions verify results in the same test file. A key performance feature is action replay: agent steps are recorded and replayed on subsequent runs without model calls unless the app changes, reducing latency and cost. The framework supports a broad ecosystem of packages covering browser engines (Chromium, Firefox, WebKit via Playwright), iOS/Android simulators, GitHub PR reporting, headless browsers (Lightpanda), and hosted cloud environments (smol machines, Limrun, EAS, TestMu). It supports multiple model providers or local models via bring-your-own-key. Example projects span Vite, Next.js, Astro, Expo, SwiftUI, Jetpack Compose, Flutter, and .NET MAUI. Currently pre-1.0 and in active development, it has been recognized as a GitHub Trending Repository. Licensed under Apache-2.0, it is notable for unifying agentic AI testing with conventional assertion patterns in a single, cross-platform framework.
EvalsAgentsAppsmultimodalbrowser-useworkflow-automation
★ 11,252 total+887 this week
#126NEWPartial week
Folkbench is a Chinese-language AI relay/proxy service benchmarking platform and companion desktop client that continuously tests third-party AI API relay providers on availability, response latency, and pricing. The platform publishes public leaderboards ranking relay services by stability, cost-effectiveness, and data security, covering popular models like GPT 6, Claude Fable 5.1, Grok 4.7, and DeepSeek V4.1 Flash. The companion open-source desktop app, **Folkbench Switch** (built with Tauri + Rust), lets users one-click switch Base URLs and API keys across AI coding tools including Claude Code, Codex, Gemini CLI, Aider, OpenCode, and Kimi CLI, without manually editing config files. Key features include lossless config switching with diff previews, local-only API key storage for privacy, real-time probe data integration, and local token usage/cost estimation. The project supports macOS (Apple Silicon and Intel) and Windows x64, with SHA256-verified releases. It is notably trending due to its practical utility for Chinese developers navigating a fragmented AI relay market, its broad tool compatibility, and its transparent, community-driven benchmarking approach backed by multiple sponsor relay providers.
EvalsAppsToolslocal-firstcodingsecurity
★ 1,894 total+789 this week