Model Invocation Audit Schema — v1.1¶

Purpose¶

Every time any eco|monetize™ agent spawns a connection to a local or remote model — Ollama, Claude API, OpenAI, Gemini, HuggingFace, or any future provider — it MUST log one record to the central audit log. This creates a platform-reportable history of all model activity across all agents.

Why this matters: Without this, there is no way to answer "how many model calls did we make, to what models, with what cost and latency, and did any fail?" as the agent fleet scales. This schema is the foundation for cost reporting, reliability monitoring, and governance audits.

Central audit log location¶

/Claude/operations/logs/model-invocations/model-invocations.jsonl

Format: JSON Lines — one JSON object per line, newline-delimited
Append-only — never overwrite, never truncate
Concurrent-write safe — each agent opens the file in append mode
Queryable with any JSON Lines reader, jq, or Python

Log rotation policy¶

To prevent unbounded file growth, mining.mind archives the log monthly:

Active log: model-invocations.jsonl (current month, all writes go here)
Archive: archive/YYYY-MM.jsonl (prior months, read-only)
Rotation runs on the first session of each new calendar month
No record is ever deleted — archiving is a move, not a truncation

Schema — one record per model invocation¶

{
  "ts_start":      "2026-04-21T10:32:00.000Z",
  "ts":            "2026-04-21T10:32:47.200Z",
  "session_id":    "20260421-103200-a3f9b1",
  "agent":         "mining.mind",
  "script":        "council-review-launcher.py",
  "host":          "ricks-macbook-pro",
  "model_id":      "llama3.3-70b",
  "model_name":    "llama3.3:70b",
  "provider":      "Meta (Llama)",
  "provider_type": "local",
  "purpose":       "council-review",
  "topic":         "cdo-prompt-engineering-retrofit",
  "mission_id":    "M-2026-0418-cdo-prompt-engineering-retrofit",
  "tokens_in":     1840,
  "tokens_out":    612,
  "latency_s":     47.2,
  "status":        "success",
  "error_msg":     null,
  "output_file":   "/Claude/operations/reports/council-reviews/2026-04-21-cdo-prompt/llama3.3-70b-review-2026-04-21.md"
}

Field definitions¶

Field	Type	Required	Description
`ts_start`	ISO 8601 UTC	yes	Timestamp when the API call was initiated (before network/inference time)
`ts`	ISO 8601 UTC	yes	Timestamp when the response was fully received (call completion)
`session_id`	string	yes	Unique ID for the script execution. All calls from one run share this ID. Format: `YYYYMMDD-HHMMSS-{6-char hex}`
`agent`	string	yes	Agent ID (`mining.mind`, `code.platform`, etc.)
`script`	string	yes	Script or tool name that made the call
`host`	string	yes	Machine hostname (`socket.gethostname()`). Distinguishes travel laptop from Mac Studio for latency analysis
`model_id`	string	yes	Short canonical model identifier used as a key (e.g. `llama3.3-70b`, `gpt-4o`)
`model_name`	string	yes	Exact model name as passed to the provider (e.g. `llama3.3:70b`, `gpt-4o`)
`provider`	string	yes	Human-readable provider name (`Meta (Llama)`, `OpenAI`, `Anthropic`, etc.)
`provider_type`	enum	yes	`local` (Ollama / on-device) or `external` (API call to remote provider)
`purpose`	string	yes	What the call was for — must be an approved slug from the Purpose Slugs table below
`topic`	string	yes	Kebab-case slug identifying the artifact or task. No proper nouns, no customer names, no deal identifiers — see PII policy below
`mission_id`	string	no	Mission ID if this call is part of a Mission (`M-YYYY-MMDD-slug`). Null if not Mission-bound
`tokens_in`	int	no	Input tokens (prompt). Null if provider does not report or if streaming (see Streaming section)
`tokens_out`	int	no	Output tokens (completion). Null if provider does not report or if streaming
`latency_s`	float	yes	Wall-clock seconds from `ts_start` to `ts`. Always present, even on error
`status`	enum	yes	`success`, `error`, `skipped`
`error_msg`	string	no	Error detail if `status` is `error` or `skipped`. Null on success
`output_file`	string	no	Absolute path to the primary output file produced by this call. Null if no file output

PII policy — `topic` field¶

The topic field is operational metadata stored in an unencrypted log. It must never contain:

Customer names, company names, or contact information
Deal names or opportunity identifiers
Any string that could identify a specific individual or organization

Use: kebab-case slugs describing the type of artifact or task — cdo-prompt-engineering-retrofit, weekly-positioning-review, eri-framework-v2, meeting-signal-extraction.

Do not use: ron-davis-hpe-meeting, acme-corp-deal-review, john-smith-objections.

If a call is for a specific meeting or customer artifact, encode the reference in mission_id or output_file — those paths already have access controls — not in topic.

Provider token reporting — by provider¶

Provider	`tokens_in` source	`tokens_out` source
Ollama	`prompt_eval_count` in response body	`eval_count` in response body
OpenAI (GPT-4o)	`usage.prompt_tokens`	`usage.completion_tokens`
Google (Gemini)	`usage_metadata.prompt_token_count`	`usage_metadata.candidates_token_count`
Anthropic (Claude API)	`usage.input_tokens`	`usage.output_tokens`
HuggingFace Inference	provider-dependent — log `null` if unavailable	same

If a provider does not return token counts, log null. Do not estimate or infer.

Streaming calls¶

When using streaming APIs (stream: true in Ollama, stream=True in OpenAI SDK):

latency_s = time-to-complete (time from call start to last chunk received, not time-to-first-token)
tokens_in / tokens_out = accumulate from streaming chunks if the provider emits usage per chunk. If not available from the stream, log null — do not estimate
ts_start and ts follow the same convention as non-streaming calls

The reference implementation (council-review-launcher.py) uses non-streaming mode and is the simpler pattern. Prefer non-streaming for batch/governance workloads where latency-to-first-token doesn't matter.

Approved `purpose` slugs¶

Use these consistently so queries work across agents:

Slug	Use for
`council-review`	Multi-model council review of an artifact
`meeting-transcription`	Whisper or model-based audio transcription
`signal-extraction`	Extracting structured signals from meeting transcripts or docs
`summarization`	Condensing documents, transcripts, or long-form content
`qlora-training`	Fine-tuning / QLoRA training runs
`rag-query`	RAG retrieval + synthesis queries
`eval`	Evaluation harness runs
`embedding`	Embedding generation (nomic-embed-text, etc.)
`general`	Any call not covered by the above

Add new slugs by updating this table and bumping the schema minor version. No ADR required for slug additions; ADR required for field additions or type changes.

Implementation guide for agent developers¶

Python snippet — copy verbatim¶

import json, socket, time, uuid
from datetime import datetime, timezone
from pathlib import Path
import os

# Resolve vault root from env var — update ECO_VAULT_ROOT if your home dir differs
VAULT_ROOT = Path(os.environ.get("ECO_VAULT_ROOT", "/Users/rhartley/Claude"))
AUDIT_LOG = VAULT_ROOT / "operations/logs/model-invocations/model-invocations.jsonl"

def log_invocation(
    session_id, agent, script, model_id, model_name, provider,
    provider_type, purpose, topic, mission_id,
    tokens_in, tokens_out, ts_start, ts_end, status, error_msg, output_file
):
    latency_s = round((ts_end - ts_start), 2)
    record = {
        "ts_start":     datetime.fromtimestamp(ts_start, tz=timezone.utc).isoformat(),
        "ts":           datetime.fromtimestamp(ts_end, tz=timezone.utc).isoformat(),
        "session_id":   session_id,
        "agent":        agent,
        "script":       script,
        "host":         socket.gethostname(),
        "model_id":     model_id,
        "model_name":   model_name,
        "provider":     provider,
        "provider_type": provider_type,
        "purpose":      purpose,
        "topic":        topic,
        "mission_id":   mission_id,
        "tokens_in":    tokens_in,
        "tokens_out":   tokens_out,
        "latency_s":    latency_s,
        "status":       status,
        "error_msg":    error_msg,
        "output_file":  output_file,
    }
    AUDIT_LOG.parent.mkdir(parents=True, exist_ok=True)
    with AUDIT_LOG.open("a", encoding="utf-8") as f:
        f.write(json.dumps(record) + "\n")

Do not rename fields. Field naming must be consistent across all agents for the audit log to be queryable. If a field doesn't apply to your use case, pass None — do not omit it.

Timing pattern¶

t_start = time.time()   # wall-clock float, before API call
# ... make your API call ...
t_end = time.time()     # after response received (or error caught)

log_invocation(..., ts_start=t_start, ts_end=t_end, ...)

Use time.time() (not time.monotonic()) so timestamps align with the ISO UTC fields. Always capture both timestamps even on error — latency_s on a failed call is diagnostic.

Environment variable¶

Set ECO_VAULT_ROOT in your shell profile or .env if your vault root differs from /Users/rhartley/Claude:

export ECO_VAULT_ROOT="/path/to/your/Claude"

Querying the audit log¶

Count calls by model¶

jq -r '.model_id' model-invocations.jsonl | sort | uniq -c | sort -rn

Total tokens by provider type (success only)¶

jq 'select(.status == "success") | {provider_type, tokens_in, tokens_out}' model-invocations.jsonl \
  | jq -s 'group_by(.provider_type) | map({provider_type: .[0].provider_type, total_in: map(.tokens_in // 0) | add, total_out: map(.tokens_out // 0) | add})'

All calls for a Mission¶

jq 'select(.mission_id == "M-2026-0418-cdo-prompt-engineering-retrofit")' model-invocations.jsonl

Error rate by agent¶

jq -r '[.agent, .status] | @tsv' model-invocations.jsonl \
  | awk '{counts[$1"_"$2]++} END {for (k in counts) print counts[k], k}' | sort -rn

Average latency by model and host¶

jq -r '[.model_id, .host, (.latency_s | tostring)] | @tsv' model-invocations.jsonl \
  | awk '{key=$1"/"$2; sum[key]+=$3; count[key]++} END {for (k in sum) printf "%.1fs avg  %s  (%d calls)\n", sum[k]/count[k], k, count[k]}' | sort

Local vs external call split¶

jq -r '.provider_type' model-invocations.jsonl | sort | uniq -c

Compliance requirement¶

Any agent or script that calls a model MUST log to this file. This is not optional. Failure to log is a Section 6E SOP breach (SEV3).

Enforcement: - The CDO (Dave) owns compliance enforcement - mining.mind produces a weekly model-call digest from the JSONL as part of Friday operations reporting — CDO receives a pre-digested summary (calls by agent, errors, total tokens, latency p50/p95) rather than raw file access - The CDO's Friday Close (per CLAUDE.md Section 12) includes a standing review of the digest

Reference implementation¶

/Claude/scripts/agents/council-review-launcher.py — mining.mind's council review launcher. First agent to implement this schema (2026-04-21). Use it as the reference implementation.

Changelog¶

Version	Date	Changes
v1.1	2026-04-21	Peer review pass: added `ts_start` + `host` fields; added PII policy for `topic`; added log rotation policy; added streaming guidance; added `summarization` purpose slug; moved snippet to `time.time()` + `ECO_VAULT_ROOT` env var; added weekly digest to compliance enforcement
v1.0	2026-04-21	Initial schema (mining.mind)

Owner: mining.mind Executive sponsor: cdo (Dave) Approved by: ceo (Rick Hartley) Effective: 2026-04-21 Version: 1.1