Evals & Supply-Chain Hardening

Lockstep versions; never hand-edit models.generated.ts; lgtm gate

Learning Objectives

After this chapter you will be able to:

• Explain how the behavioral eval harness bridges AgentSession events into vitest-evals
• Describe the supply chain strategy: exact pin + shrinkwrap whitelist + OIDC publish
• Explain how lockstep versioning keeps 7 packages released in sync
• Understand why the lgtm gate and AGENTS.md serve both human and AI audiences

Key Source Files

File	Lines	Responsibility
packages/evals/src/pi-harness.ts	205	Eval harness: AgentSession → TranscriptEvent mapping
packages/evals/scripts/run-evals.mjs	78	Eval runner script: vitest-evals invocation
scripts/release.mjs	~150	Release: commit + tag + push automation
scripts/generate-coding-agent-shrinkwrap.mjs	~200	Shrinkwrap whitelist generation
AGENTS.md	—	Project rules (dual audience: humans + AI agents)

Design Motivation: Why Behavioral Evals?

Problem: Traditional tests assert "input → exact output." But agent behavior is probabilistic — the same prompt may produce different tool-call sequences. You cannot assert(result === expected).

Analogy: Like exam rubric scoring — you cannot judge only "right/wrong"; you need a scoring rubric. Agent evals use a judge model to score against criteria, not exact matching.

Solution: vitest-evals framework + custom harness — transform AgentSession's event stream into standard TranscriptEvents, then evaluate with a judge model against rubrics.

Behavioral Eval Harness

// pi-harness.ts
function createPiCodingAgentHarness(opts: HarnessOptions) {
  return {
    async run(prompt: string): Promise<TranscriptEvent[]> {
      // ① Create isolated environment
      const tmpDir = mkdtempSync(join(tmpdir(), "pi-eval-"))
      const session = createAgentSession({
        cwd: tmpDir,
        model: opts.model,
        tools: opts.tools,
      })

      // ② Run prompt, collect events via subscription
      const events: TranscriptEvent[] = []
      session.agent.subscribe((e) => {
        events.push(toTranscriptEvent(e))  // AgentEvent → TranscriptEvent
      })
      await session.prompt(prompt)

      // ③ Cleanup
      rmSync(tmpDir, { recursive: true })
      return events
    }
  }
}

// Event mapping: agent domain → eval domain
function toTranscriptEvent(e: AgentEvent): TranscriptEvent {
  switch (e.type) {
    case "text_delta":  return { type: "message", content: e.text }
    case "tool_use":    return { type: "tool_call", name: e.name, args: e.args }
    case "tool_result": return { type: "tool_result", output: e.output }
  }
}

Isolation: Each test case gets its own mkdtemp directory — evals never interfere with each other, and cleanup is immediate.

Supply Chain Hardening

Dependency management:
  ├── save-exact=true          → exact versions in package.json (no ^ or ~)
  ├── min-release-age=2        → only install packages published >2 days ago (→ Ch15)
  ├── npm ci --ignore-scripts  → never run postinstall scripts (→ Ch15)
  ├── shrinkwrap whitelist     → new deps with lifecycle scripts require approval
  └── npm run check            → CI gate: verifies pin/imports/shrinkwrap consistency

Publishing:
  ├── lockstep: scripts/sync-versions.js keeps 7 packages at same version
  ├── release: scripts/release.mjs → commit + tag vX.Y.Z + push
  └── publish: OIDC trusted publish (idempotent, skips if already published)

Shrinkwrap whitelist: npm-shrinkwrap.json locks all transitive dependencies. If a dependency has lifecycle scripts (postinstall), it must appear on the whitelist — unlisted script-bearing dependencies are rejected by CI.

OIDC publish: No long-lived npm token — GitHub Actions' OIDC token performs trusted publish. The token exists only during CI execution, minimizing leak risk.

Lockstep Versioning

// scripts/sync-versions.js
// 7 packages always share one version:
// pi-ai, pi-agent-core, pi-tui, pi-coding-agent, pi-server, pi-storage, pi-evals
const VERSION = readRootPackageJson().version
for (const pkg of PACKAGES) {
  pkg.version = VERSION
  pkg.dependencies["pi-ai"] = VERSION  // internal refs also pinned
}

Why lockstep? When users run npm install pi, all internal packages are version-consistent — no compatibility hell from pi-ai@1.2 paired with pi-agent-core@1.1.

Contribution Gate & lgtm

CONTRIBUTING.md policy:
  ① New contributors' issues/PRs auto-close by default
     └── Prevents spam PRs / AI-generated low-quality contributions
  ② Maintainer reviews then unlocks via lgtm/lgtmi
     └── lgtm = approve PR
     └── lgtmi = approve issue
  ③ AGENTS.md is a shared rulebook for humans AND AI agents
     └── Minimal-kernel philosophy: non-core goes into extensions

Why auto-close? Open-source projects face floods of spam PRs (SEO links, meaningless AI-generated changes). Default-close + maintainer-unlock shifts review cost from "reject each one" to "approve the worthy few."

Engineering Insights

1. pi-harness.ts is only 205 lines: Complexity lives in vitest-evals and AgentSession. The harness is pure glue — mapping AgentEvent to TranscriptEvent and managing temp directories.

2. npm run check is the CI gate: It verifies three things — all dependencies are exact-pinned, no unauthorized ts-imports exist, shrinkwrap matches package.json. Any failure = red CI.

3. AGENTS.md serves two readers: It is both a human contributor guide and an AI agent system prompt (→ Chapter 12). One file, two audiences — brevity is mandatory.

Cross-Chapter Links

Direction	Chapter	Relationship
Prerequisite	Ch11 AgentSession	Harness drives AgentSession to collect events
Prerequisite	Ch15 Package Manager	min-release-age, --ignore-scripts policies
Prerequisite	Ch04 Session Tree	Evals can replay recorded sessions
Related	Ch13 Extensions	"Minimal kernel: non-core goes into extensions"
Related	Ch12 System Prompt	AGENTS.md injected into system prompt

Check Your Understanding

1. Why can't agent evals use traditional assert(result === expected)? What are the limitations of judge-model scoring?
2. What security advantages does OIDC trusted publish have over a long-lived npm token?
3. If lockstep were removed and the 7 packages versioned independently, what problems would users encounter?

Takeaways

• Behavioral evals: AgentSession → TranscriptEvent → judge scoring, isolated temp environments
• Supply chain: exact pin + shrinkwrap whitelist + OIDC publish (no long-lived tokens)
• Lockstep: 7 packages share one version, eliminating compatibility hell
• lgtm gate: new contributors auto-closed, maintainer unlocks; AGENTS.md serves two audiences

---

This chapter is based on the main-branch source of earendil-works/pi.