Language Model Evaluation Harness
EleutherAI tooling for running language-model tasks with shared evaluation infrastructure.
Documentation and sourceEvidence
A score means more when you can see the task, model revision and configuration behind it. Record a run, compare saved outputs, or choose an evaluation framework for your workload.
The recorder works locally in your browser and exports a portable JSON file. It does not call a model or publish your data. OMHQ has not published its own benchmark runs yet; the tools below are external projects, not our test results.
EleutherAI tooling for running language-model tasks with shared evaluation infrastructure.
Documentation and sourceCode-generation evaluation tooling with expanded tests for supported programming benchmarks.
Documentation and sourceAn evaluation framework with tasks, solvers, scorers and logs for language-model and agent evaluations.
Documentation and sourceA coding benchmark with time-aware problem collection and published evaluation resources.
Documentation and sourceKeep the original prompt and output, model identifier and revision, runtime or harness, generation settings and run time. Add observed latency and cost only when measured. A single successful example is not a benchmark, and an exact-text match is not a general quality score.
The harness affects the result. For code or tool-using evaluations, isolate execution and preserve permissions and traces with the result.