Vals Smith

Vals Smith is a developer tool that turns a GitHub repo into a custom benchmark for AI models. Vals Smith generates a custom evaluation suite from a public or private GitHub repository, then runs chosen AI models and coding agents against tasks derived from merged pull requests.

Vals Smith builds a private evaluation suite out of a GitHub repository, so teams can see how different AI models and coding agents perform on their own code instead of relying on generic leaderboards. It accepts public or private repositories and is aimed at engineering teams weighing which model or agent to adopt for a specific project. The workflow has three stages. A user signs in with GitHub and selects a repository, and the tool reads the codebase to map its structure, languages and test surface. Next, the user chooses which models and coding agents to include, along with the harness that governs how they operate against the repo. Frontier models and agents can be mixed within one suite. Finally, results arrive as pass rates with per-task breakdowns, letting teams compare candidates side by side and see which handles their project best. The methodology borrows from established software-engineering benchmarks. Vals Smith looks for merged pull requests that represent clear, testable work and converts each into a self-contained task made of an issue statement, hidden tests and a reproducible setup. Candidates that rely on brittle infrastructure or ambiguous behavior are rejected. Each task is validated by confirming that its hidden tests fail before the original fix and pass after it. During evaluation, a model sees only the issue statement and the repository at its starting state, with no access to the real fix. A task counts as resolved only if the hidden tests pass and existing behavior does not regress, and the final score is the share of tasks completed. The site also publishes benchmarks for public repositories, such as pallets-eco/blinker, psf/requests and SWE-bench, each showing task counts, language mix and top models. The company states that user data stays private and points to a data policy and terms of use. Compared with general coding leaderboards, the value here is relevance to a team's own codebase, though early public examples show small model counts, and access appears to run through booking a call or applying as a design partner.