0.8.6: auto-skills (API.md 3.2/3.3/6, ORACLE.md Auto-skills, Exploration)

API.md

@@ -359,6 +359,56 @@ matter to the oracle alone). With L = the option ids of a choice question:
 | `superset` | one skill's labels plus labels it does not know | the oracle only; its answer teaches that skill only for a key with `learning_allowed` |
 | `untrained` | anything else, and every `score` / `noul` question | the oracle only; without it the request is 422 |
 
+**Auto-skills (0.8.6).** A choice question no skill fits (`untrained` because
+of its options, not an ambiguous one and not one forced with `cmf.skill`) is
+learned from the oracle's answers when the caller's key has `learning_allowed`
+and `learning.auto_skills` is on (the default). Its *contract* is the whole
+question — `instructions` and `criteria` with their descriptions, not the id
+set: the key is `auto-<12 hex of the sha256 of the canonical JSON of {type,
+instructions, criteria}>` with the criteria keys sorted (so the criteria in
+any order are one contract, while other instructions, a changed description,
+a dropped or an added option are each another contract — `{yes, no}` under
+"Is this spam?" and under "Is this urgent?" are two skills that never answer
+each other's question, and positional ids `{A, B, C, D}` whose descriptions
+change with every request are never learned as one). The answers accumulate
+as examples of that skill, and once enough evidence is there (2 labels with ≥
+`auto_min_rows` fit rows, the active labels holding ≥ `auto_min_coverage` of
+the examples, a macro agreement with the oracle on a held-out fifth of the
+rows ≥ `auto_min_agreement`) the skill is fitted, written as a generation and
+served: the same contract is then `exact`, `action: local`, `skill: auto-…`,
+always `certified: false` (`decision_path` `router:uncertified`). An
+auto-skill is matched by its contract alone — never as a `subset` or
+`superset` — so to be learned and then answered locally a client keeps the
+question stable: the same `instructions` text, the same option ids with the
+same descriptions (`null` descriptions, and absent instructions, count as
+such and stay stable). Labels the oracle rarely picks stay *quarantined*: they
+count as known within the contract, get probability 0, and a text of such a
+label is expected to abstain on novelty and keep teaching it. A
+young auto-skill's gate comes from a handful of rows, so a local answer also
+needs `p_top ≥ learning.auto_tau` (0.90; `balanced` and `quality-first`) and
+its temperature is floored at `learning.auto_temperature_min` (0.02 — a clean
+calibration subset fits T at its lower bound, where every `p_top` is 1);
+whatever abstains escalates and teaches, and later attempts refit the whole
+skill under a regression gate. *Exploration* is the one exception to the hard
+rule that a gate-accepted question never reaches the oracle: while a label of
+an auto-skill is quarantined, a question the gate accepted is escalated anyway
+when the text's hash says so (`u64le(sha256(φ_P))` ≡ 0 modulo
+`learning.auto_explore_every`, 8 — one text in eight; 0 turns it off), because
+the gate confidently names a quarantined label's texts as a neighbour and they
+would otherwise never teach it. The oracle's answer is served (`action:
+oracle` / `cache`, `decision_path` `escalate→…`, the flag `explore`, the
+`gate` block still `accepted: true`) and learned as usual, so the rare label
+collects examples at a quarter of its traffic until it activates; a refused
+or failed call leaves the local answer. Only auto-skills explore, only while
+a label is quarantined, and only when the answer could teach (the oracle
+consented, learning on, a key with `learning_allowed`); `cortiq decide`,
+shadow mode and `/v1/route` never do. Its rubric is the contract — the first
+caller's instructions and criteria verbatim — visible through
+`/v1/skills/{id}` to every key (as cached answers are shared across accounts).
+Limits, who teaches and the state directory's compatibility:
+[ORACLE.md](ORACLE.md#auto-skills). When a data skill and auto-skills fit a
+question equally, the data skill answers.
+
 `cmf.skill` names the skill and skips the search (an unknown id is 400). An
 answer is `certified: true` only for an exact match on a string `state`, the
 `balanced` or `quality-first` profile, a skill whose gate is certified, and a
@@ -408,7 +458,7 @@ curl -s "$CORTIQ/v1/skills/banking77" -H "Authorization: Bearer $KEY" \
 | `source`, `skill`, `match`, `certified` | where the answer came from; section 3.2 |
 | `gate` | `accepted`, `p_top` and `tau`, `novelty` and `theta`, `is_novel`, `margin`, `profile` |
 | `errors` | reconstruction errors of the 5 best labels (all of them with `cmf.explain`) |
-| `flags` | e.g. `oracle_disabled`, `no_key` (with `oracle_disabled`: the key variable is not set), `bad_key` (with `oracle_disabled`: it holds something that is not a key), `consent_off`, `budget`, `stopped`, `oracle_unavailable`, `pii_redacted` |
+| `flags` | e.g. `oracle_disabled`, `no_key` (with `oracle_disabled`: the key variable is not set), `bad_key` (with `oracle_disabled`: it holds something that is not a key), `consent_off`, `budget`, `stopped`, `oracle_unavailable`, `pii_redacted`, `explore` (a gate-accepted question of an auto-skill answered by the oracle for exploration, section 3.2) |
 | `confident` | the answer can be used as is (gate accepted and not novel, or a valid oracle answer) |
 | `complexity` | `{score, tier, factors: {base, ambiguity, novelty, margin, length}}`, the cortiq-router formula |
 | `routing` | `{target, reason}` when `routing_tiers` maps the tier |
@@ -530,10 +580,10 @@ curl -s "$CORTIQ/v1/feedback" -H "Authorization: Bearer $KEY" \
 | Path | Access | Returns |
 |---|---|---|
 | `GET /v1/models` | open | the model in the shape of an OpenRouter provider listing (`id`, `pricing` as USD-per-token strings, `context_length` 512, `max_output_length` 255) plus `cmf.skills` with each gate |
-| `GET /v1/skills` | key | every skill: `id`, `taxonomy_version`, `labels`, `certified`, `tau`, `theta`, `temperature` and `has_rubric` (whether it has a rubric; the rubric itself is per skill) |
+| `GET /v1/skills` | key | every skill: `id`, `taxonomy_version`, `labels` (the active ones), `certified`, `tau`, `theta`, `temperature`, `has_rubric` (whether it has a rubric; the rubric itself is per skill) and, since 0.8.6, `auto` (an auto-skill), `active_labels` (their count), `quarantined_labels` (the labels not scored yet) and `examples` (learned rows); auto-skills are listed after the file's skills |
 | `GET /v1/skills/{id}` | key | the same plus `tasks`, the whole `gate` and `rubric` (`instructions`, `criteria`; `null` without one) |
 | `GET /v1/usage` | key | the caller's account (router format; `x-cmf-extensions: 1` adds token and cost totals) |
-| `GET /healthz` | open | status, model, generation, skills, `oracle` (on/off) and `oracle_status` (`ready`, `no_key`, `bad_key`, `disabled`, `budget_exhausted`, `budget_too_small`, `stopped: <reason>`; section 5) |
+| `GET /healthz` | open | status, model, generation, skills (every served skill), `auto_skills` (0.8.6), `oracle` (on/off) and `oracle_status` (`ready`, `no_key`, `bad_key`, `disabled`, `budget_exhausted`, `budget_too_small`, `stopped: <reason>`; section 5) |
 
 Without `--jev-compatible`, `/v1/models` has only the native OpenRouter-style
 listing shape; Cortiq Decision is not listed on OpenRouter. With the adapter
@@ -762,7 +812,7 @@ unset the admin API answers 404 `ADMIN_DISABLED`.
 | `POST / GET /v1/admin/keys`, `DELETE /v1/admin/keys/{account}`, `DELETE /v1/admin/keys/hash/{hash12}` | create (raw key returned once), list, revoke |
 | `GET /v1/admin/usage` | usage of every account |
 | `GET /v1/admin/oracle`, `POST /v1/admin/oracle {"enabled", "budget_usd", "max_calls"}` | oracle status: `status` (`ready`; `no_key` — the key variable is unset or empty; `bad_key` — it holds something that is not a key, never sent; `disabled` — not configured or switched off by the admin; `budget_exhausted` — something was spent and the budget (what is left cannot hold the smallest possible call, or a call it refused) or `max_calls` is used up; `budget_too_small` — nothing spent, and the budget cannot hold one call, `min_call_usd`, or `max_calls` is 0; `stopped: <reason>` — a stop rule), `configured` = `oracle.enabled` of the configuration, `enabled` = not switched off by a stop rule or the admin, `key_present`, `key_ok`, `key_problem` (by position and length, never a byte of the key), `key_trimmed` (surrounding whitespace was trimmed), `key_env` (the variable's name, never its value), `model`, `max_price`, `min_call_usd` (the least budget a call needs: the smallest possible call's reservation, or, once the budget refused a longer call — a budget below the smallest call included — that call's, the larger), spent, calls, stop reason, `last_error` (the code of the last failed call; both from a closed code set, another value read back from `oracle.state` is `unknown_code`); switch it and lower limits within the configuration (kept in `oracle.state` across restarts: when one of them refuses the next call, `cmf.hint` and the startup line name it and the `POST /v1/admin/oracle` that lifts it) |
-| `GET /v1/admin/learning` | buffer, cache, quarantine, attempts, promotions, recent events |
+| `GET /v1/admin/learning` | buffer, cache, quarantine, attempts, promotions, recent events; since 0.8.6 `auto_contracts`, `auto_skipped` and `auto_skills` (per contract: `id`, `labels`, `examples` per label — the rows an attempt fits: the served learned rows plus the pending buffer examples, each row once — `created_unix`, `served`) |
 | `GET /v1/admin/generations`, `POST /v1/admin/rollback {"generation": N}` | generations; serve generation N (0 = the base file) |
 | `GET /v1/admin/shadow` | agreement statistics in shadow mode, and the routed requests not compared since the start (section 8) |
 
@@ -803,7 +853,11 @@ unknown key is an error. The defaults:
              "max_tokens_per_question": 64, "deadline_s": 30, "budget_usd": 1.0, "max_calls": 10000,
              "max_errors": 30, "redact_pii": true, "title": "cortiq-decision", "data_collection": null},
   "cache": {"enabled": true, "threshold": 0.97, "cap": 50000},
-  "learning": {"enabled": true, "refit_min_new": 25, "dedup": 0.995, "cold_start": true, "synchronous": false},
+  "learning": {"enabled": true, "refit_min_new": 25, "dedup": 0.995, "cold_start": true, "synchronous": false,
+               "auto_skills": true, "auto_min_rows": 10, "auto_k": 8, "auto_tau": 0.9,
+               "auto_min_agreement": 0.8, "auto_min_coverage": 0.8, "auto_max_skills": 256,
+               "auto_max_labels": 64, "auto_max_examples_per_label": 1000,
+               "auto_temperature_min": 0.02, "auto_explore_every": 8},
   "feedback": {"pending_cap": 50000},
   "complexity_weights": {"base": 0.4, "ambiguity": 0.25, "novelty": 0.15, "margin": 0.1, "length": 0.1},
   "complexity_tiers": [{"tier": "low", "max": 0.33}, {"tier": "medium", "max": 0.66}, {"tier": "high", "max": 1.0}],
@@ -817,7 +871,22 @@ part of the file: the admin token and the OpenRouter key are read from the
 environment variables it names. `auth.plans` may override the plan table.
 `oracle.base_url` must be https, or plain http to a loopback address only (a
 local proxy). The oracle, cache and learning sections are explained in
-[ORACLE.md](ORACLE.md). `cortiq serve --oracle MODEL` and its companions
+[ORACLE.md](ORACLE.md). The `learning.auto_*` keys govern auto-skills
+(section 3.2, [ORACLE.md](ORACLE.md#auto-skills)): `auto_skills` learns
+untrained choice contracts at all; `auto_min_rows` fit rows (≥ 2) a label
+needs to be active; `auto_k` the rank of its topologies; `auto_tau` the
+confidence floor of a local answer (0..1; a serving parameter — `cortiq
+decide` uses the default); `auto_min_agreement` the macro agreement with the
+oracle required to activate and `auto_min_coverage` the share of the
+contract's examples its active labels must hold (both 0..1); `auto_max_skills`
+how many contracts are learned at most and `auto_max_labels` (2..255) how
+many options one may have — a contract past either is answered by the
+oracle and not learned; `auto_max_examples_per_label` (≤ 5000) replaces the
+per-label buffer cap for auto-skills; `auto_temperature_min` (0..1) floors
+the gate temperature an attempt records (0: the fitted T, whose lower bound
+makes every `p_top` 1); `auto_explore_every` (an integer, 0 = off) explores
+one text in that many while a label is quarantined — the one case in which a
+gate-accepted question reaches the oracle, auto-skills only. `cortiq serve --oracle MODEL` and its companions
 override the `oracle` section (and `--no-oracle-learning` sets
 `learning.enabled` false); a file that sets `oracle.provider` keeps its
 `max_price` unless `--oracle-max-price` is given.

ORACLE.md

@@ -72,7 +72,9 @@ says first whether it is ready — [Check your setup](#check-your-setup)).
 
 Superset questions (a skill's labels plus new ones) are decided by the oracle
 only; for a caller with `learning_allowed` the answer teaches that skill, and
-a new label starts a cold start.
+a new label starts a cold start. A choice question no skill fits is learned
+into an *auto-skill* of its own ([Auto-skills](#auto-skills)), for such a
+caller only.
 
 ## Self-learning
 
@@ -127,7 +129,130 @@ a new label starts a cold start.
   every label that got new examples, however few (there is no threshold of
   25 here), checks each challenger on the holdout and re-certifies the gate
   once at the end; if a certified gate would be lost, every promotion of the
-  skill is undone (see API.md, section 7).
+  skill is undone (see API.md, section 7). An auto-skill (below) learns
+  online only.
+
+### Auto-skills
+
+A choice question whose option ids match no skill (`match: untrained`, not
+an ambiguous one — several skills fit — and not one forced with `cmf.skill`)
+goes to the oracle on every new text and, until 0.8.5, was never learned. Since
+0.8.6 its *contract* becomes a skill of its own:
+
+* **Key.** The contract is the question itself: `auto-` + the first 12 hex
+  characters of the sha256 of the canonical JSON of `{type, instructions,
+  criteria}` — the instructions as sent (`null` when absent), the criteria
+  object with its keys sorted and every description kept. The criteria in
+  any order are one contract; other instructions, a changed description, a
+  dropped or an added option are each another contract, learned separately.
+  The id set alone is not the key: two clients using `{yes, no}` under
+  different questions get two skills, and a multiple-choice benchmark whose
+  positional ids `{A, B, C, D}` carry a new description per request is never
+  learned as one skill (every request is its own contract, bounded by
+  `auto_max_skills`). To be learned, a client keeps its question stable —
+  the same instructions text, the same ids with the same descriptions. The
+  first request's instructions and criteria are stored verbatim as the
+  skill's rubric (the id is recomputed from it), visible through
+  `/v1/skills/{id}` to every key. The contract is written to `learn.log`
+  before its first example, so a restart rebuilds it.
+* **Who teaches.** Only a caller whose key has `learning_allowed` (the explicit
+  open mode `auth.require: false` has it; the implicit open mode of a loopback
+  address never teaches). The rule "a question that is exactly the skill's own
+  teaches it" does not apply to auto-skills: their rubric *is* the contract
+  every caller sends. `learning.auto_skills: false` turns the learning off;
+  the oracle answers as before.
+* **Examples.** Each answer is an example of the auto-skill under the
+  contract's label (the dedup of 0.995 applies). The labels are closed: a
+  feedback with a label outside the contract is refused (a request with one
+  more id is another contract, learned on its own).
+  At most `learning.auto_max_examples_per_label` (1000) examples per label.
+* **Activation.** When a label collects 25 new examples (the usual trigger)
+  the whole skill is attempted: one row in five — by a hash of the row itself,
+  so a row never moves between the fit and the calibration subset as more
+  arrive — is held out; a label with ≥ `auto_min_rows` (10) fit rows is
+  *eligible* and fitted (rank `auto_k`, 8), the others stay quarantined; the
+  attempt needs two eligible labels including the triggering one and, the
+  first time, the eligible labels must hold ≥ `auto_min_coverage` (0.8) of
+  the contract's rows. T and θ are certified on the held-out rows as at build
+  time (τ certifies only with 100 accepted odd rows, so the skill reports
+  `certified: false` for a long time, and its winners are `cold_start` ones —
+  never certified). The first activation (`auto_start`) requires a macro
+  agreement of the fitted skill with the oracle's labels on the held-out rows
+  ≥ `auto_min_agreement` (0.8) and ≥ 0.5 per label; a later attempt
+  (`auto_refit`, the whole skill refitted) must not regress the champion on
+  the same rows (the holdout rule). A promotion writes a generation that
+  carries the auto-skill entirely; every other skill is byte for byte the
+  same.
+* **Serving.** The contract is then decided locally, `match: exact`,
+  `skill: auto-…`, `certified: false`. An auto-skill is matched by its
+  contract alone: a part of its ids, a superset, other instructions or another
+  description is another contract (`untrained`, learned on its own), never a
+  `subset` or `superset` of the auto-skill. A quarantined label still counts
+  as known within the contract: it gets probability 0, and a text of it is
+  expected to abstain on novelty and keep teaching it. Because θ and T of a
+  young auto-skill come from a handful of rows (θ is close to the largest
+  novelty of the even half, T may sit at a bound), a local answer also needs
+  `p_top ≥ learning.auto_tau` (0.9) under `balanced` and `quality-first`
+  (`cost-saver` keeps its θ-only rule), and the T an attempt records is
+  floored at `learning.auto_temperature_min` (0.02): a clean held-out subset
+  fits T at its lower bound, where every `p_top` is 1 and the `auto_tau`
+  floor never bites. An abstention escalates and teaches as any other. The
+  agreement gate is the real guard while the subset is small; the gate
+  improves with every refit. Rejected examples are never forgotten: a
+  contract polluted by a noisy oracle needs more consistent examples, not a
+  reset (or a rollback to a generation before it).
+* **Exploration.** The hard rule — a question the gate accepted never
+  reaches the oracle — has exactly one exception, for auto-skills only and
+  only while one of their labels is quarantined: the gate names such a
+  label's texts as a neighbour with full confidence (measured on a stand: all
+  21 texts of a quarantined label answered locally as another label at
+  `p_top` 1), so without it the label could never learn. One text in
+  `learning.auto_explore_every` (8; `u64le(sha256(φ_P))` modulo it is 0, a
+  property of the text, never of the order; 0 turns it off) is escalated
+  although accepted: the oracle's answer is served (`action: oracle` /
+  `cache`, the flag `explore`, the `gate` block still reports `accepted:
+  true`) and learned as usual, so the rare label collects examples at a
+  quarter of its traffic until the next attempt activates it, after which
+  the contract explores no more. A refused or failed call leaves the local
+  answer. Exploration draws only when the answer could teach: the oracle
+  consented, learning and `auto_skills` on, a key with `learning_allowed`;
+  data skills, `cortiq decide`, shadow mode and `/v1/route` never explore.
+* **Limits.** A contract is learned only with 2..`auto_max_labels` (64) option
+  ids and while fewer than `auto_max_skills` (256) contracts are registered;
+  otherwise the oracle answers and nothing is recorded (`auto_skipped` in
+  `GET /v1/admin/learning`, one warning an hour, never the ids). Every
+  generation re-carries every auto-skill's rows: with many contracts run
+  `cortiq decision materialize` from time to time and serve the materialised
+  file (it holds the auto-skills as ordinary skills; `decide --labels` and
+  `verify` read it; such a file needs cortiq ≥ 0.8.6). A served auto-skill
+  keeps learning wherever its contract comes from: the contract record of
+  `learn.log`, or — a materialised file on a fresh state directory, a
+  `learn.log` lost while the generations were kept — the skill's own labels
+  and rubric, registered (and written to `learn.log`) at start and after a
+  rollback.
+* **Routing.** `/v1/route` without `taxonomy_id` still means the file's only
+  data skill; an auto-skill is routed by its id (its question is then the
+  whole contract from the rubric). `cortiq decide --labels` never names an
+  auto-skill by labels alone: `--skill auto-…` names it and its rubric
+  supplies the contract. When a data skill and auto-skills fit a question
+  equally, the data skill answers; two auto-skills of one contract cannot
+  exist (the contract is the id).
+* **Admin.** `GET /v1/admin/learning` lists `auto_skills` (id, labels,
+  `examples` per label — the rows the next attempt fits: the served learned
+  rows of the label plus the buffer examples not among them, each row once,
+  so their sum is the attempt's `rows.total` — what is served),
+  `auto_contracts` and `auto_skipped`;
+  attempts carry `kind: auto_start | auto_refit` and an `auto` block with the
+  eligible and quarantined labels, the rows and the agreement. `/healthz`
+  adds `auto_skills`. Rollback to an earlier generation drops the skill from
+  the served model (the contract is untrained again; its examples stay and
+  start it over at the next trigger).
+* **Compatibility.** A 0.8.5 binary opening a 0.8.6 state directory truncates
+  `learn.log` at the first contract record and refuses a generation that
+  carries an auto-skill (the same rule as the `LOCK` of 0.7.9: never run an
+  older binary on a newer state directory). The cache scope of a contract
+  changes from the contract to the skill at its activation, so its first
+  requests after that miss the cache (they are answered locally anyway).
 
 ## Budget and stop rules
 
@@ -535,7 +660,8 @@ warning: removed the LOCK of state directory cortiq-decision.cmf.state left by p
   (`oracle_allowed: true`, like imported router keys); `--oracle-allowed=false`
   makes one that never escalates, and `--oracle-budget-usd` caps one key's
   oracle spending. Teaching the shared skills is a separate permission,
-  `--learning-allowed` (off by default). Keys made through `POST
+  `--learning-allowed` (off by default); it alone lets a caller's untrained
+  contracts become [auto-skills](#auto-skills). Keys made through `POST
   /v1/admin/keys` keep the router's default (`oracle_allowed` only when the
   body says so).