Skip to content

feat(memory): surface actual LLM usage/cost from inference API in sync audit #3110

@senamakel

Description

@senamakel

Summary

The sync audit log currently estimates token usage and cost using a character-length heuristic (body.len() / 4) and hardcoded DeepSeek v4 flash pricing ($0.07/M in, $0.28/M out). The backend inference API already returns actual token counts and charged_amount_usd in the response (openhuman.usage.input_tokens, openhuman.usage.output_tokens, openhuman.billing.charged_amount_usd), but this data is lost because the memory ChatProvider trait's chat_for_text() only returns the text content.

What needs to change

  1. Thread usage through the memory ChatProvider — extend ChatProvider::chat_for_text() (or add a parallel method) to return (String, Option<UsageInfo>) so callers get the actual token counts and cost.

  2. Accumulate usage in summarise() — the SummaryOutput should include input_tokens, output_tokens, charged_amount_usd from the provider response.

  3. Pass real usage to the audit logingest_summary and run_github_sync use the provider-reported values instead of estimates.

Where the data lives

  • UsageInfo (with charged_amount_usd) is already parsed in src/openhuman/inference/provider/compatible.rs:707 (extract_usage)
  • The ChatResponse struct at traits.rs:85 carries usage: Option<UsageInfo>
  • The memory layer calls chat_for_text() which calls chat_with_history() which only returns String
  • The audit log is in src/openhuman/memory_sync/sources/audit.rs

Acceptance criteria

  • Sync audit entries show provider-reported token counts (not len/4)
  • Sync audit entries show charged_amount_usd from the backend (not hardcoded pricing)
  • Old entries (before this change) still render correctly with estimated values

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions