Summary

Adds cost tracking infrastructure for model and embedding providers. Cost is computed at inference time from raw provider responses using user-configured JSON Pointer mappings, then stored in both ClickHouse and Postgres.

New module: cost.rs

• Cost type alias for rust_decimal::Decimal (stored as Decimal(18, 9) / NUMERIC(18, 9))
• Uninitialized → Normalized config pattern:
  • UninitializedCostConfigEntry / UninitializedCostRate — user-facing types with cost_per_million and cost_per_unit, deserialized from TOML/JSON
  • CostConfigEntry / CostRate — normalized runtime types (always per-unit), converted at config load time via TryFrom / From impls
• CostPointerConfig (unified or split streaming/non-streaming pointers) — stored as String in user-facing config, validated and parsed into json_pointer::JsonPointer at config load time
• ResponseMode enum (NonStreaming / Streaming) instead of boolean flag
• compute_cost_from_response() / compute_cost_from_json() — extracts values via JSON Pointers, applies rates, sums contributions
• CostConfigEntry::compute_contribution() — per-entry cost computation, easily extensible for future expression-based cost models
• Custom decimal_from_value serde visitor — formats f64 to shortest decimal string (Ryu algorithm) then parses to Decimal, avoiding binary float precision loss (e.g. 0.1 in TOML → exactly Decimal(0.1))
• Negative rates are allowed (e.g., caching discounts); negative totals log a warning and return None

Configuration

• New optional cost: Option<CostConfig> field on model providers and embedding providers
• Pointers validated at config load time using json_pointer crate
• Example:

  [models."gpt-4o-mini-2024-07-18".providers.openai]
  type = "openai"
  model_name = "gpt-4o-mini-2024-07-18"
  cost = [
    { pointer = "/usage/prompt_tokens", cost_per_million = 0.15, required = true },
    { pointer = "/usage/completion_tokens", cost_per_million = 0.60, required = true },
  ]

Database

• ClickHouse migration 0047: ALTER TABLE ModelInference ADD COLUMN cost Nullable(Decimal(18, 9))
• Postgres migration: ALTER TABLE tensorzero.model_inferences ADD COLUMN cost NUMERIC(18, 9)
• All SELECT/INSERT queries for model_inferences updated to include cost

Inference pipeline

• Non-streaming: cost computed immediately after provider response from raw_response
• Streaming: cost computed per-chunk inside stream processing loop, enriching chunks before yielding to downstream consumers
• Embeddings: cost computed after embedding response (always non-streaming)
• Usage struct gains cost: Option<Cost> field (derives Copy)
• Streaming aggregation: uses streaming_max for cost (same strategy as tokens)
• Cross-model aggregation: uses sum_or_poison — sums cost when all inferences have cost; any None → total None (poison semantics)

API / wire types

• ModelInference (internal API): new cost: Option<Cost> (omitted when None)
• StoredModelInference: new cost: Option<Cost>
• OpenAICompatibleUsage: new tensorzero_cost: Option<Decimal> extension field

Provider changes

All 16 providers updated to initialize cost: None in Usage construction (cost is computed later from raw_response): Anthropic, AWS Bedrock, Dummy, Fireworks, GCP Vertex Anthropic, GCP Vertex Gemini, Google AI Studio Gemini, Groq, Mistral, OpenAI, OpenAI Responses, OpenRouter, TGI, Together, XAI, relay.

Variant changes

cost field threaded through best_of_n_sampling, chat_completion, mixture_of_n, and variant mod.rs.

Optimizer changes

All 4 optimizers updated with cost: None: DICL, Fireworks SFT, OpenAI, Together SFT.

Dependencies

• rust_decimal = { version = "1.37", features = ["serde-float"] } (new)
• json_pointer (new, for pointer validation)
• sqlx feature "rust_decimal" added

Tests

69 cost-related tests passing (unit + E2E):

Unit tests in cost.rs (45 tests)

• Rate conversion: per-million → per-unit normalization, per-unit passthrough
• Cost computation: basic token pricing, per-unit pricing (web searches), required/optional missing fields, negative totals, empty config, split pointers (streaming vs non-streaming), unparseable values, string-encoded numbers, booleans, zero tokens, float values, mixed positive/negative (caching discount), discount exceeding base cost
• Response parsing: valid JSON, invalid JSON
• TOML deserialization: unified pointers, per-unit rates, split pointers, invalid pointers, invalid split pointers, missing rates, missing pointers, invalid cost values, exact decimal precision (0.1, 0.3), string-quoted decimals, negative costs, both rates present, empty pointer string
• Config normalization: per-million → per-unit, per-unit passthrough, load_cost_config with valid config
• JSON Pointer validation: valid, invalid (missing leading slash)
• Full pipeline (TOML → normalize → provider JSON → cost): OpenAI-style, Anthropic-style with cache, required field missing, per-unit web search, invalid JSON, zero tokens, large token counts (GPT-4), Gemini-style nested usage, direct cost passthrough
• Warning assertion: test_compute_cost_required_missing_logs_warning uses capture_logs() to verify warning is logged when required pointer is missing

Unit tests in usage.rs (20 tests)

• Streaming aggregation: empty, single chunk, cumulative (Anthropic-style), final-only (OpenAI-style), all None, mixed None/Some, non-cumulative detection (debug_assert), Anthropic real-world pattern
• Cross-model aggregation: empty, single, multiple all-Some, None propagation (poison semantics), both None propagate, all None
• Cost aggregation: both present (sum), one missing poisons total, missing first poisons total, all None, zero is present (not missing), empty returns zero, three-way mix poisons (mixture-of-N scenario)

E2E tests (16 tests)

• Simple cost (non-streaming + streaming): cost present in API response, cost stored correctly in ClickHouse
• DICL poison semantics (non-streaming + streaming): model has cost config, embedding model doesn't → total cost is None
• Best-of-N poison semantics (non-streaming + streaming): candidates have cost, evaluator doesn't → total cost is None
• Best-of-N all-cost (non-streaming + streaming): all providers (candidates + evaluator) have cost config → total cost is sum
• Mixture-of-N poison semantics (non-streaming + streaming): one candidate model lacks cost → total cost is None
• Mixture-of-N all-cost (non-streaming + streaming): all providers (candidates + fuser) have cost config → total cost is sum

Test infrastructure additions

• best_of_n_0_with_usage dummy provider: returns evaluator content with standard usage fields for cost tracking tests
• dummy_dicl variant: uses dummy-embedding-model (no cost config) for DICL poison semantics tests
• cost_test_best_of_n, cost_test_best_of_n_all_cost, cost_test_mixture_of_n, cost_test_mixture_of_n_all_cost functions in E2E config

Test plan

• cargo check --all-targets --all-features
• cargo clippy --all-targets --all-features -- -D warnings
• cargo fmt --check
• Unit tests: 65 tests passing (cost.rs + usage.rs)
• E2E tests: 69 tests run, 69 passed (including 16 cost-specific E2E tests)
• All existing tests continue to pass (6387 skipped = non-cost tests filtered out)

Closes #6260, closes #6261