Plus: Anthropic signs $35B compute deal with Lambda; Study tests small models as rubric-based RL judges
Welcome to a Tuesday where the industry is rethinking how we measure value, from outcome-based billing to the very rubrics we use to judge intelligence.
PRICING
💰 OpenAI tests outcome-based billing
OpenAI has begun letting some of its largest customers pay only when its AI actually completes the job. The arrangement is limited to select major accounts rather than offered generally.
The details:
Intercom charges $0.99 for each conversation its Fin agent resolves and nothing for ones it does not.
Zendesk restricted billing to Verified Resolutions, confirmed by an LLM evaluation within 72 hours of the conversation.
Salesforce launched Agentforce at $2 per conversation, charged for every 24-hour session whether or not anything was resolved.
⚖️ Study tests small models as rubric-based RL judges
Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human preferences. Traditional RLHF relies on scalar reward signals that lack interpretability and fail to capture the multifaceted nature of response quality.
The details:
Rubric-guided reinforcement learning addresses these limitations by introducing structured, interpretable evaluation criteria, or rubrics, as the backbone.
Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria.
Training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more.
