Session Token Efficiency: The Metric That Shows If Your AI Agent Is Actually Working

Everyone tracks tokens. Input tokens, output tokens, total tokens, cost per token. But here's the thing: token counts don't tell you if a session was productive.

The Problem With Token Counts

Consider two sessions:

Session A: 50K input, 2K output, $0.15 cost
Session B: 50K input, 2K output, $0.15 cost

Same numbers. Same cost. But Session A might have accomplished nothing while Session B shipped a feature.

We needed a better metric.

Introducing Session Token Efficiency

After running 29 AI agents in production, we developed a new metric:

Session Token Efficiency = max(0, final_context - initial_context) / max_tokens × 100

What Each Part Means

initial_context: The first turn's input tokens. This includes:
- System prompt
- Injected files (SOUL.md, AGENTS.md, etc.)
- Any auto-loaded context
final_context: The last turn's cumulative input tokens. This is how much the context window grew during the conversation.
max_tokens: The model's context window (200K for Claude 3.5/4, 128K for GPT-4)

The Interpretation Scale

Efficiency	Meaning	Action
0%	Session loaded context but did nothing	Investigate - wasted tokens
10-40%	Light use - short conversation	Normal for quick Q&A
40-70%	Moderate use	Good for most tasks
70%+	Excellent window utilization	Ideal for deep work sessions

Why This Metric Matters

1. It Separates Setup From Work

A 200K token session isn't impressive if 195K of that was just loading the system prompt. Session token efficiency strips out the "sunk cost" of initial context and measures actual work done.

2. It Catches Wasted Sessions

We found sessions with:

150K input tokens
3K output tokens
2% session token efficiency

These agents were loading massive context windows, producing almost nothing, and burning money. Traditional metrics made them look busy.

3. It Guides Model Selection

Our cron jobs were accidentally falling back to Claude Opus (expensive) instead of Haiku (cheap). Session token efficiency helped us spot this—cron jobs had high token counts but near-zero efficiency because they're designed for quick checks, not deep work.

New rule: Cron jobs should use Haiku. Only main sessions need expensive models.

Real Numbers From Our Production System

After implementing this metric, here's what we found across 17 active sessions over 60 minutes:

Total: 4.9M input │ 1.1M output │ 40% avg session token eff │ 17 sessions

Warnings:
⚠️  4 sessions with <10% window fill (low session token efficiency)

The 4 flagged sessions? All cron jobs that should have been on cheaper models.

How We Calculate It

For each session, we parse the JSONL transcript:

// First turn captures initial context
if (turnCount === 0) {
  initialContext = turnInput;  // system prompt + injected files
}

// Last turn's input = total context used
cumulativeContext = turnInput;

// Session token efficiency
const sessionTokenEfficiency = 
  Math.max(0, cumulativeContext - initialContext) / maxTokens * 100;

The metric is now available in our CLI:

orch spend --minutes 60
orch spend --group agent --json

Practical Takeaways

For Individual Sessions

Aim for 70%+ efficiency on deep work sessions
Don't sweat low efficiency on quick Q&A (that's expected)
Investigate 0% sessions - something's wrong if an agent loads context and does nothing

For System Design

Match model cost to expected efficiency: Cheap models for low-efficiency tasks (cron, monitoring), expensive models for high-efficiency tasks (coding, analysis)
Monitor aggregate efficiency: If your average is below 30%, your agents might be starting too many sessions instead of continuing existing ones
Track trends over time: Declining efficiency often indicates context bloat or prompt problems

For Cost Optimization

Session token efficiency directly impacts cost per useful output:

Effective cost = raw_cost / (session_token_efficiency / 100)

A $1 session at 10% efficiency costs you $10 of "useful work." A $1 session at 70% efficiency costs you $1.43 of "useful work."

What's Next

We're adding session token efficiency to our Mission Control dashboard with:

Real-time per-agent breakdown
Historical trends
Automatic alerts for consistently low-efficiency agents
Model recommendation engine (suggesting cheaper models for low-efficiency patterns)

Try It Yourself

If you're running OpenClaw agents:

# Install/update orch CLI
npm install -g @supernalintelligence/agent-tools

# Check your session token efficiency
orch spend --minutes 60

# Group by agent
orch spend --group agent

# Export for analysis
orch spend --json > spend-report.json

The bottom line: Token counts tell you what you spent. Session token efficiency tells you if it was worth it.