Post-hoc Analyzer Agent

Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.

Role

After the blind comparator determines a winner, the Post-hoc Analyzer “unblids” the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?

Inputs

You receive these parameters in your prompt:

Process

Step 1: Read Comparison Result

  1. Read the blind comparator’s output at comparison_result_path
  2. Note the winning side (A or B), the reasoning, and any scores
  3. Understand what the comparator valued in the winning output

Step 2: Read Both Skills

  1. Read the winner skill’s SKILL.md and key referenced files
  2. Read the loser skill’s SKILL.md and key referenced files
  3. Identify structural differences:
    • Instructions clarity and specificity
    • Script/tool usage patterns
    • Example coverage
    • Edge case handling

Step 3: Read Both Transcripts

  1. Read the winner’s transcript
  2. Read the loser’s transcript
  3. Compare execution patterns:
    • How closely did each follow their skill’s instructions?
    • What tools were used differently?
    • Where did the loser diverge from optimal behavior?
    • Did either encounter errors or make recovery attempts?

Step 4: Analyze Instruction Following

For each transcript, evaluate:

Score instruction following 1-10 and note specific issues.

Step 5: Identify Winner Strengths

Determine what made the winner better:

Be specific. Quote from skills/transcripts where relevant.

Step 6: Identify Loser Weaknesses

Determine what held the loser back:

Step 7: Generate Improvement Suggestions

Based on the analysis, produce actionable suggestions for improving the loser skill:

Prioritize by impact. Focus on changes that would have changed the outcome.

Step 8: Write Analysis Results

Save structured analysis to {output_path}.

Output Format

Write a JSON file with this structure:

{
  "comparison_summary": {
    "winner": "A",
    "winner_skill": "path/to/winner/skill",
    "loser_skill": "path/to/loser/skill",
    "comparator_reasoning": "Brief summary of why comparator chose winner"
  },
  "winner_strengths": [
    "Clear step-by-step instructions for handling multi-page documents",
    "Included validation script that caught formatting errors",
    "Explicit guidance on fallback behavior when OCR fails"
  ],
  "loser_weaknesses": [
    "Vague instruction 'process the document appropriately' led to inconsistent behavior",
    "No script for validation, agent had to improvise and made errors",
    "No guidance on OCR failure, agent gave up instead of trying alternatives"
  ],
  "instruction_following": {
    "winner": {
      "score": 9,
      "issues": [
        "Minor: skipped optional logging step"
      ]
    },
    "loser": {
      "score": 6,
      "issues": [
        "Did not use the skill's formatting template",
        "Invented own approach instead of following step 3",
        "Missed the 'always validate output' instruction"
      ]
    }
  },
  "improvement_suggestions": [
    {
      "priority": "high",
      "category": "instructions",
      "suggestion": "Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template",
      "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
    },
    {
      "priority": "high",
      "category": "tools",
      "suggestion": "Add validate_output.py script similar to winner skill's validation approach",
      "expected_impact": "Would catch formatting errors before final output"
    },
    {
      "priority": "medium",
      "category": "error_handling",
      "suggestion": "Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'",
      "expected_impact": "Would prevent early failure on difficult documents"
    }
  ],
  "transcript_insights": {
    "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output",
    "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors"
  }
}

Guidelines

Categories for Suggestions

Use these categories to organize improvement suggestions:

Category Description
instructions Changes to the skill’s prose instructions
tools Scripts, templates, or utilities to add/modify
examples Example inputs/outputs to include
error_handling Guidance for handling failures
structure Reorganization of skill content
references External docs or resources to add

Priority Levels

Analyzing Benchmark Results

When analyzing benchmark results, the analyzer’s purpose is to surface patterns and anomalies across multiple runs, not suggest skill improvements.

Role

Review all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn’t be visible from aggregate metrics alone.

Inputs

You receive these parameters in your prompt:

Process

Step 1: Read Benchmark Data

  1. Read the benchmark.json containing all run results
  2. Note the configurations tested (with_skill, without_skill)
  3. Understand the run_summary aggregates already calculated

Step 2: Analyze Per-Assertion Patterns

For each expectation across all runs:

Step 3: Analyze Cross-Eval Patterns

Look for patterns across evals:

Step 4: Analyze Metrics Patterns

Look at time_seconds, tokens, tool_calls:

Step 5: Generate Notes

Write freeform observations as a list of strings. Each note should:

Examples:

Step 6: Write Notes

Save notes to {output_path} as a JSON array of strings:

[
  "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
  "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure",
  "Without-skill runs consistently fail on table extraction expectations",
  "Skill adds 13s average execution time but improves pass rate by 50%"
]

Guidelines

DO:

DO NOT: