Agent Infra Index
An independent infrastructure comparison site and Agent Infra Weekly newsletter, backed by repeatable browser and sandbox benchmarks, source-linked provider claims, freshness review, and explicit cost and publication gates.
Infrastructure Decisions Need Evidence
Agent Infra Index is an independent comparison database for the infrastructure agents depend on: code sandboxes, browser infrastructure, inboxes and email, memory, and coordination, hosting, and payments. Agent Infra Weekly is the companion newsletter.
Explore: agentinfraindex.com · Benchmark methodology and results · Agent Infra Weekly
Why Build It?
Choosing infrastructure from a collection of pricing pages is difficult. A free plan is not the same as introductory credit. A fast successful sample does not tell you how often a workload fails. A source checked last month does not establish today's eligibility.
The index makes those distinctions part of the product. The aim is not to declare one universal winner, but to give builders inspectable evidence and enough context to make their own trade-offs.
Claims Are Independently Reviewable
Provider records separate pricing, free access, and agent instructions into independent claims. Each can carry its own source, scope, substantive verification date, review deadline, and review state.
- A successful source fetch is not confirmation of a claim.
- A changed page creates a review candidate; it does not automatically replace an accepted price.
- Ongoing allowances, free plans, trials, and introductory credits remain distinct.
- Blocked sources preserve the last valid evidence and expose the need for review.
- Stable provider IDs protect links when display names change.
This avoids the misleading simplicity of one provider-wide “verified” badge. Evidence can be strong for one attribute and unresolved for another.
Repeatable Benchmarks, Visible Failures
The live benchmark harness covers sandbox providers including E2B, Daytona, and Modal, and browser providers including Browserbase, Browserless, Steel, Hyperbrowser, and Kernel. Coverage depends on credentials, allowances, and a safe run admission decision; an adapter existing does not mean every scheduled batch measured it.
Sandbox workloads create an environment, install a fixed dependency, execute hashing and disk I/O tasks, then clean up. Browser workloads create a session, connect through CDP, extract known items from a deterministic fixture and a reference page, capture a screenshot, and release the session.
Public results retain raw successful and failed samples, completion rates, error categories, and median/p90 timings for successful samples. A low completion rate remains visible beside a fast timing. Version changes, regions, small samples, reference-page failures, and cleanup differences are disclosed rather than hidden in a single score.
These are not a ranked leaderboard. Actual provider costs are not measured yet.
Operational Gates Are Part of the System
Live benchmarking can create billable resources. An allowance ledger and bounded reservations gate provider calls before execution. Unknown or stale allowance information pauses work; authentication, quota, and cleanup failures can stop a batch. The system does not automatically upgrade to a paid plan or retry until a result looks better.
Publication distinguishes newly measured data from old results retained during a paused week. A diagnostic cannot stand in for a measurement, and a coverage update does not silently renew the date of an older benchmark.
Architecture and Publishing
| Component | Role |
|---|---|
| Astro and TypeScript | Static directory, provider/category pages, benchmark results, methodology, and discovery files |
| Python harness | Sandbox workloads and sample collection |
| Node and Playwright harness | Remote browser workloads, validation, and cleanup |
| GitHub Actions | Scheduled measurements, source collection, and claim-freshness checks |
| JSON evidence model | Versioned provider claims, samples, review states, and publication inputs |
| Cloudflare Pages and KV | Public site plus validated provider-submission and sponsor-inquiry forms |
| Beehiiv | Newsletter signup and editorial publication |
The build allowlists public data and redacts sensitive fields before export. Visitors can inspect raw benchmark JSON and download CSV data; private review queues, account allowances, source snapshots, and operational records are not part of those exports.
Human Editorial Control
Source monitoring and generated newsletter drafts help an editor focus on meaningful changes. They do not automatically accept pricing claims or send the newsletter. A private typed-model triage foundation exists, but live model calls are not enabled in the reviewed state; offline synthetic evaluation is not a live model accuracy result.
Sponsorship is structurally separate from evidence. Paid placements must be labeled, and cannot change organic directory order, benchmark conditions, or published results.
Current State and Limits
The directory, public benchmark results, downloadable data, newsletter signup, and inquiry forms are live. Some providers remain unmeasured or paused, and unresolved claims require substantive source review. Inbox benchmarking currently has offline synthetic conformance tests only: no live inbox-provider performance measurements are claimed.
What This Work Demonstrates
Agent Infra Index combines product engineering with evidence operations: reproducible testing, data provenance, freshness, publication safety, and cost-aware automation. The central design choice is to keep “observed,” “verified,” “paused,” and “unknown” distinct, even when a simpler marketing table would be easier to build.