feat(home/pi): add blind intuition probe skill

Assisted-by: pi (openai-codex/gpt-5.6-sol)
2026-08-24 10:04:09 -05:00 · 2026-07-17 17:21:10 -03:00
parent 83aa5f2c8d
commit 93fc009c5e
4 changed files with 154 additions and 0 deletions
@@ -0,0 +1,43 @@
+---
+name: intuition-probe
+description: Blind-test the intuitiveness of an API, CLI, config format, or UI by asking fresh isolated Pi processes what interface they expect before inspecting the real artifact. Use for API/DX/UX intuition probes, expected affordances, and "make the API whatever the LLM guesses".
+---
+
+# Intuition probe
+
+Use a model's prior as a cheap sample of familiar interface conventions. A divergent guess is a design candidate, not merely a model error. Prefer conforming the interface to a convergent familiar guess unless safety or a hard invariant forbids it.
+
+This is adapted for Pi from Jeremy Theocharis's [intuition-probe](https://gist.github.com/JeremyTheocharis/83c76da5a10bcf495d4298c70fee91b4), itself based on Anselm Eickhoff's Jazz principle: “make the API whatever the LLM guesses.”
+
+## Guardrails
+
+- Ordering is the experiment: freeze the blind prompt, then inspect reality, then launch probes.
+- Never expose artifact paths, implementation details, exact identifiers, or the answer key to a probe.
+- One sample produces candidates only. Call something a priority only when at least two independent samples converge.
+- The probes are isolated `pi -p` processes with tools, extensions, skills, context files, prompt templates, and sessions disabled. Do not use a subagent tool.
+- Never edit the artifact under test as part of this skill; report recommendations only.
+
+## Procedure
+
+1. Ask for the artifact, rough outcome, optional read-set, and sample count (default 1). Do not inspect the artifact yet.
+2. Rewrite the outcome without leaked method names, keys, flags, labels, paths, or implementation clues. Ask the user to confirm the sanitized goal.
+3. Read [references/blind-prompt.md](references/blind-prompt.md). Fill its placeholders in a temporary prompt file. A read-set is allowed verbatim; fetch URL content and inline it rather than giving the probe a URL. Freeze this file now.
+4. Only now inspect the real API signatures, schema/examples, CLI help/definitions, or UI routes/components. This is the answer key and stays in the orchestrator context.
+5. Run the probes from this skill directory:
+
+   ```bash
+   bash scripts/run-blind.sh /path/to/frozen-prompt.md N
+   ```
+
+   The script prints an output directory containing `probe-*.json` and logs. Read every result. If a probe emits invalid JSON, preserve it as a failed sample; do not quietly repair its design choices.
+6. Compare every decision against reality using [references/scoring.md](references/scoring.md). Fold duplicate/hedged decisions explicitly rather than dropping them.
+7. Group equivalent guesses across samples. Convergence means the same semantic shape, not merely similar wording.
+8. Report:
+   - mode (`cold` or `doc-informed`), model, and N;
+   - the candidate interface/spec the probes reached for;
+   - divergences with bucket, confidence/convergence, familiar anchor, and conform-first recommendation;
+   - matches briefly;
+   - rejected conform moves and the concrete safety/invariant reason;
+   - limitations: same-model samples are repeated draws, not independent human usability evidence.
+
+For N=1, label the report **candidate only — not a confirmed familiar default**. Offer to save it outside the repository under test.
@@ -0,0 +1,32 @@
+# Blind probe prompt
+
+Replace `{{SYSTEM}}`, `{{GOAL}}`, and the optional read-set block before launching the probe. Do not include the artifact's path or real implementation.
+
+---
+You are a developer encountering {{SYSTEM}} for the first time.
+
+Outcome you want: {{GOAL}}
+
+{{OPTIONAL_READ_SET}}
+
+Write the code, configuration, commands, or interaction you expect to work from intuition and prior knowledge alone. Capture your first instinct; do not hedge toward every plausible design.
+
+You have no access to the real implementation or documentation beyond material quoted above. Do not ask to inspect it.
+
+For each independent choice, record the exact interface you reached for, your honest confidence, and the familiar API/library/idiom that anchored the guess. Split choices such as operation name, argument shape, return shape, placement, and workflow when they can vary independently.
+
+Output exactly one JSON object and nothing else:
+
+{
+  "decision_points": [
+    {
+      "decision": "short label",
+      "guess": "exact code/config/command/interaction",
+      "familiar_anchor": "known API, library, or idiom; 'none' only when appropriate",
+      "confidence": "high | medium | low",
+      "reasoning": "one sentence"
+    }
+  ],
+  "wished_existed": ["expected affordance not already covered"]
+}
+---
@@ -0,0 +1,17 @@
+# Scoring
+
+Map each decision to exactly one bucket. When between buckets, choose the one implying the larger interface change; hindsight must not rationalize a divergence into a match.
+
+| Bucket | Meaning | Default move |
+|---|---|---|
+| `match` | Reality has the expected name and shape | Keep it |
+| `naming-mismatch` | Correct concept/shape, different name | Rename or add the expected alias |
+| `shape-mismatch` | Different structure, arguments, return, placement, or idiom | Reshape toward the familiar expectation |
+| `missing-affordance` | The expected capability does not exist | Build it |
+| `hallucinated` | A confidently invented call/key/interaction does not exist | Treat the invention as a candidate spec and build it |
+
+Use the probe's pre-reality confidence as the base signal. A named, widely familiar anchor strengthens the result. With multiple probes, convergence matters more than any one confidence label. Mark priority only when at least two probes independently converge.
+
+Conforming is the default, not a law. Keep the unfamiliar design and teach it in docs only when conforming would be unsafe, violate a hard invariant, or impose unacceptable compatibility cost; name the constraint explicitly and consider rerunning a doc-informed probe after changing the docs.
+
+Include strong `wished_existed` entries as `missing-affordance` or `hallucinated` findings. Do not count matches as divergences.
@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  echo "usage: $0 FROZEN_PROMPT [COUNT]" >&2
+  echo "optional env: PI_INTUITION_MODEL=provider/model, PI_INTUITION_CONCURRENCY=4" >&2
+  exit 2
+}
+
+[[ $# -ge 1 && $# -le 2 ]] || usage
+prompt_file=$(realpath "$1")
+count=${2:-1}
+concurrency=${PI_INTUITION_CONCURRENCY:-4}
+[[ -f "$prompt_file" ]] || { echo "prompt not found: $prompt_file" >&2; exit 2; }
+[[ "$count" =~ ^[1-9][0-9]*$ ]] || usage
+[[ "$concurrency" =~ ^[1-9][0-9]*$ ]] || usage
+
+run_dir=$(mktemp -d "${TMPDIR:-/tmp}/pi-intuition-probe.XXXXXX")
+cp -- "$prompt_file" "$run_dir/frozen-prompt.md"
+printf '%s\n' "${PI_INTUITION_MODEL:-<default model>}" > "$run_dir/model.txt"
+
+run_probe() {
+  local i=$1
+  local -a model_args=()
+  [[ -z ${PI_INTUITION_MODEL:-} ]] || model_args=(--model "$PI_INTUITION_MODEL")
+
+  (
+    cd "$run_dir"
+    pi --print --no-session --offline \
+      --no-tools --no-extensions --no-skills --no-prompt-templates --no-context-files \
+      --system-prompt 'Follow the user prompt exactly. Return only the requested JSON. You have no tools and must not seek additional context.' \
+      "${model_args[@]}" \
+      "$(cat frozen-prompt.md)"
+  ) > "$run_dir/probe-$i.raw" 2> "$run_dir/probe-$i.stderr"
+
+  if jq -e '.decision_points | type == "array"' "$run_dir/probe-$i.raw" > "$run_dir/probe-$i.json" 2>/dev/null; then
+    printf 'probe %s: valid\n' "$i"
+  else
+    printf 'probe %s: invalid JSON (kept as .raw)\n' "$i" >&2
+    return 1
+  fi
+}
+export -f run_probe
+export run_dir prompt_file PI_INTUITION_MODEL
+
+status=0
+running=0
+for i in $(seq 1 "$count"); do
+  run_probe "$i" &
+  running=$((running + 1))
+  if (( running >= concurrency )); then
+    wait -n || status=1
+    running=$((running - 1))
+  fi
+done
+while (( running > 0 )); do
+  wait -n || status=1
+  running=$((running - 1))
+done
+
+printf 'outputs: %s\n' "$run_dir"
+exit "$status"