mirror of
https://github.com/Gitlawb/openclaude.git
synced 2026-08-24 10:14:19 -05:00
c461a0363d
* feat: merge knowledge graph + conversation arc into memdir
Replace the standalone KG/ARC system (SQLite + JSON + Orama storage)
with direct integration into the existing auto-memory directory.
What changed:
- New memdir/vectorIndex.ts — Orama full-text index over all memory/ .md
files, replacing the separate knowledge.orama binary
- New memdir/autoExtractFacts.ts — auto-detects env vars, paths,
versions, URLs, IPs, backtick concepts from conversation and writes
them as structured .md files into memory/.facts/ with frontmatter
- conversationArc.ts now persists arc state (goals, decisions,
milestones, phase) to memory/.arc.json sidecar instead of the KG
- knowledgeGraph.ts gutted from 728→165 lines — now a thin
compatibility layer that reads .facts/ files from memdir and
delegates vector search to vectorIndex.ts
- build.ts: enabled CONVERSATION_ARC and MULTI_TURN_CONTEXT feature
flags (previously undefined → dead-code eliminated in production)
Removed:
- src/utils/storage/ (SQLiteProvider, JSONProvider, 3 test files) —
unused after KG migration
- src/utils/knowledgeGraph.test.ts, .stress.test.ts
- src/utils/conversationArc.test.ts, .perf.test.ts
- ~1700 lines of redundant storage code
Benefits:
- Single memory system (memdir) instead of two parallel systems
- Auto-extracted facts are plain .md files — visible to the model,
discoverable by the existing Sonnet prefetch
- Vector search indexes real memory content, not a separate DB
- Arc state survives across sessions via .arc.json
- ~700 lines removed from the production bundle
* fix: type errors, add test suites, fix resetArc() disk-write bug
- Fix vectorIndex.ts: parseFrontmatter returns nested {frontmatter, content},
Orama DB typed as 'any' matching existing code pattern
- Fix resetArc(): was overwriting .arc.json on disk — now clears only in-memory state
- Add vectorIndex.test.ts (6 tests): build/search/persist/rebuild
- Add autoExtractFacts.test.ts (10 tests): env vars, paths, versions,
URLs, backtick concepts, PascalCase, React/Redux, file signatures, frontmatter
- Add conversationArc.test.ts (14 tests): arc init, persistence, goals,
decisions, milestones, phase detection, arc summary, finalize, stats
- Update verify-kg-merge.sh: add test suite check (33 tests)
* fix: remove generic type param from restore() call
* fix: address CodeRabbit findings — YAML injection, secrets leak, cache invalidation, frontmatter parsing, test weakness
* fix: redact URL credentials/query/hash in endpoint extraction; add persistence + reindex regression test
* test: add regression for URL credential/query/hash redaction in fact extraction
* feat: wire getOrchestratedMemory into query.ts prompt; remove dead promises array in autoExtractFacts
* feat: enhance memory management by adding clearArcArtifacts function and integrating it into the clear command; implement file count tracking in vector index
* feat: enhance fact extraction by adding tests for absolute paths, backtick concepts, technical terms, project file signatures, and IP addresses; implement clearArcArtifacts function in tests
* Review-fix: scoped IP tagging, scrubbedContent, arcMemoryDir null, vector-index cleanup
* Review-fix: isAutoMemoryEnabled gate, scrubbed paths, clearIndex in cleanup
* Review-fix: URL-stripped path scan, digit-key/quoted-value env redaction, rm isAutoMemory gate
* Review-fix: quoted multi-token env values fully redacted, regression test
* Review-fix: freshness check on each search, integration test, typecheck in verifier
* Review-fix: missing-index-file reinit, full-pipeline integration test
* Review-fix: gate arc/RAG on isAutoMemoryEnabled, add integration+stale-index tests
Addresses three P1/P2 findings from code review:
1. P1: Honor auto-memory opt-out before writing arc facts
- Add isAutoMemoryEnabled() checks in query.ts before calling
updateArcPhase() and getOrchestratedMemory()
- Add same check inside conversationArc.ts extractFactsAutomatically()
- Prevents .facts file writes when auto-memory disabled via --bare,
CLAUDE_CODE_DISABLE_AUTO_MEMORY=1, or memory.autoWrite: false
2. P2: Add query-level integration test coverage
- New test in conversationArc.test.ts verifies query.ts path
- Confirms arc functions called behind feature gates and results
appended to system prompt (lines 555-575)
3. P2: Add stale-index regression tests
- 6 new tests in vectorIndex.test.ts cover:
* Searching after adding files
* Searching after editing files
* Searching after removing files
* Searching when .vector-index missing
* Searching when .vector-index-meta.json missing
* Mixed stale conditions
All tests pass (31 total, 99 assertions), typecheck clean.
* Review-fix: use >= for mtime staleness check to catch same-ms edits
The verification agent discovered a timing-dependent bug in the stale
index detection. When a file edit and index save occur within the same
millisecond, latestMtime equals indexMtime, causing the check
`latestMtime > indexMtime` to return false. The stale index is not
refreshed and searches miss the updated content.
Changed line 220 from `>` to `>=` in the mtime comparison. The file
count check catches add/remove operations, but edits that don't change
the file count rely on mtime comparison.
All 12 vector index tests now pass consistently, including the
"searching after editing a file picks up changes" test that previously
failed intermittently.
* Review-fix: move auto-memory gate to persistence layer
Addresses inline review comment: query.ts was incorrectly skipping
updateArcPhase() entirely when isAutoMemoryEnabled() returned false,
leaving the in-memory arc state stale.
Fixed by:
- Removed isAutoMemoryEnabled() gate from query.ts line 449
- Added gate inside conversationArc.ts updateArcPhase() at persistence
layer (line 223), so phase advances but only persistence is disabled
- Arc state tracking now works regardless of auto-memory setting
- Only disk writes (.arc.json, .facts files, index rebuilds) are gated
The 2ms delay in vectorIndex.test.ts is kept as a pragmatic fix for the
timing race. Content-hash detection would be ideal but adds complexity;
the mtime check works reliably in production.
All 31 tests pass, typecheck clean.
* Refactor memory handling: consolidate metadata retrieval and improve auto-memory checks
* Fix: update expectation to use toEqual for prompt comparison in conversationArc tests
* fix: detect same-size content changes via content hash; handle text-block user messages in arc query
* fix: skip symlinked dirs in vector index walk; enable arc/multi-turn flags; add production-path tests
* fix: follow symlinked dirs in vector index walk per review
* fix: skip symlinked dirs in vector index walk; add symlink-boundary regression tests
* fix: skip indexing symlinked directories and ensure only files are processed
* fix: show cmd output on failure in verifier; add multi-turn coverage; clear mempath cache; cover arc reset in knowledge clear test
* fix: clear memoized auto-mem path after teardown in knowledge + conversationArc tests
* fix: isolate vector index per memdir, filter secrets from backtick facts, clean trailing ws
* fix: extend credential filter for AWS/GitLab tokens; reject all symlinks in vector index
* fix: use repo's redactSecretSubstringsForDisplay; add npm/glpat/AKIA/ASIA/xox to shared patterns; cover NPM+JWT in tests
* fix: P1 backtick credential safety + untrusted-data boundary; P2 legacy migration + non-fatal writes; P3 type safety + build regression
* fix: B1-B5, M6, M8 — no message mutation, migration data loss, empty-file, non-fatal writes, probe, rebuildIndex resilience, dead import
* fix: address 8 reviewer findings (R1-R8)
P1:
- Keep retrieved facts in DATA ONLY block with strict system instruction
- Catch lowercase config secrets (api_key=...) in env scrubber
- Migrate SQLite working store (knowledge.db) before deleting provider
- /knowledge clear atomically archives legacy sources
P2:
- Run legacy migration on getOrchestratedMemory retrieval path
- Preserve entity attributes in migration frontmatter
- Only rebuild vector index when facts actually changed
- Fix feature-flag verifier --define syntax (declare const)
* fix: close 9 memory findings — approval gate, non-fatal writes, secret scrub, per-project migration, attribute/relation preservation, WAL cleanup, cheaper index
- Gate auto fact extraction on isMemoryWriteApprovalRequired() + isAutoMemoryEnabled() so default projects cannot silently persist conversation content
- Make ensureFactsDir/writeFactMemory degrade non-fatally (no turn-breaking throw on read-only dirs)
- Scrub token-like URL/path/hyphenated segments from durable facts via looksLikeSecret (reuses providerSecrets.looksLikeSecretValue)
- Restore passive project-rule extraction as rule facts
- Honor isAutoMemoryEnabled() before legacy migration; scope the migration guard per project (Set) instead of a single global
- Preserve legacy entity attributes and relations through migration (indented attributes + relation fact file, reconstructed in getGlobalGraph)
- Clear SQLite WAL/SHM sidecars on /knowledge clear
- Replace per-turn content hashing in vectorIndex getMdStats with size+mtime metadata; drop redundant initMemdirIndex call in getOrchestratedMemory
- Add knowledgeGraph tests covering P1#2/P1#4/P2#5/P2#8 and extend autoExtractFacts tests
* test: avoid global cwd pollution in knowledgeGraph tests
Replace process.chdir with a per-test setFsImplementation mock cwd that is
reset via setOriginalFsImplementation in afterEach, so the test no longer
leaks a changed process.cwd() into other test files. Assert the auto-memory
gate via the project-specific legacy file rather than the shared resolved
memdir dir (which bun runs concurrently across it blocks).
* fix: address 15 reviewer findings (R1-R15)
P1:
- Gate saveArcToDisk/finalizeArcTurn/saveIndex/migration on memory-write approval
- Retire legacy sources after successful migration (rmSync, backup preserved)
- Slugify entity.type and summary.id in migration filenames (path traversal)
- Fall back to JSON when SQLite has zero entities
- Scrub rule-fact extraction on scrubbedContent + looksLikeSecret/redact check
P2:
- Track skipped vs completed migration; re-enable clears skip marker
- Walk back to latest human text for tool-round vector queries
- Auto-extract goals/decisions from user messages in updateArcPhase
- /knowledge clear message says durable wipe, not session-only
- Bind arc state to projectKey (re-resolve on cwd change)
- clearIndex(memoryDir?) scopes to one memdir
- yamlQuote all migration frontmatter fields
- Real SHA-256 contentHash alongside fileFingerprint for same-size edits
- Stop claiming production-pipeline coverage in tests/verifier
* test: isolate governance mock by removing afterEach clear
Remove setGovernancePolicySettingsForSourceForTesting(null) from afterEach
in autoExtractFacts, conversationArc, and knowledgeGraph test files. The
module-level mock is set in each file's beforeEach and since there is no
afterEach cleardown, parallel test execution can no longer corrupt the
mock state across files.
This fixes 26 CI test failures caused by one file's afterEach clearing
the mock that another concurrently-running file had set in its beforeEach.
* fix: isolate governance mock per async context via executionAsyncId tracking
Replace the module-level variable in governancePolicy.ts with an
executionAsyncId-keyed Map and an async_hooks.createHook that propagates
the override from parent to child async resources. This ensures each
concurrent test's beforeEach/afterEach cannot corrupt the override set
by another test file, even when test bodies directly mutate the mock.
Also restore setGovernancePolicySettingsForSourceForTesting(null) calls
in afterEach hooks (removed in 39c68376), which are now safe because
each afterEach only clears its own async context.
Fixes 26 CI test failures across knowledgeGraph (2), conversationArc (7),
and autoExtractFacts (17/22, governance-gate tests).
* Refactor knowledge graph legacy migration, secure secrets and IP octets, optimize build-time feature flags, cap conversation arc collections, and resolve all verification check gaps
* Fix governancePolicy enablement in full test runs by checking for test runner globals
* fix: ensure auto memory is enabled in conversationArc, knowledgeGraph, autoExtractFacts tests
Delete CLAUDE_CODE_DISABLE_AUTO_MEMORY and CLAUDE_CODE_SIMPLE env vars
in beforeEach hooks so tests are not blocked when CI sets these vars.
Also revert governancePolicy.ts to the simple module-level variable
(removing the async ID tracking approach that broke with Bun v1.3.13).
Fixes 33 CI test failures where isAutoMemoryEnabled() returned false
causing all disk-persistence guards to fire.
* fix: address P1/P2 review findings — redaction, bounds, legacy backup
[P1] Redact goal/decision descriptions with redactLikelySecrets before
persisting to .arc.json, session summaries, and prompt summaries so
credentials captured by auto-extraction regexes are not durably stored.
[P1] Bound multi-turn tool input serialization to 2000 chars and apply
redactLikelySecrets, preventing oversized/credential-bearing tool inputs
from overflowing the next provider request or exposing secrets.
[P2] Archive the non-selected legacy store (JSON or SQLite) and its WAL
sidecars before retiring both sources, ensuring a recoverable snapshot
exists if generated fact files are incomplete or a migration bug surfaces.
* fix: address P1 findings — safe legacy retirement, multi-turn aggregate budget
[P1] Do not retire a legacy store unless every existing source was
successfully archived. Track archived sources in a Set and skip deletion
of any source whose backup failed (knowledgeGraph.ts).
[P1] Archive the selected SQLite WAL/SHM sidecars alongside its migration
backup, since committed state may reside only in the WAL file and the
advertised recovery backup would otherwise be incomplete (knowledgeGraph.ts).
[P1] Bound the aggregate multi-turn tool replay to 10KB total and stop
appending further turns once the budget is exceeded, preventing many
Agent/MCP calls per turn from adding unbounded text to system prompts
(conversationArc.ts).
* fix: address P1/P2 review findings — rule extraction gate, SQLite read status, atomic WAL/SHAM, KG status gate, byte budgets
* fix: tighten looksLikeOpaqueToken to avoid flagging compound model names
* fix: address P1/P2 review findings — rebase, test isolation, SQLite retry, attribute redaction, entity aliases, body content, project-scoped multiturn, skip unchanged writes, knowledge list gate
* fix: address P1/P2/P3 review findings — secret scrub on migrate, lean decision gate, git-root legacy lookup, backup retention, index rebuild chaining, single vector search, drop unreferenced fixture
* fix: scrub secret entity names on migrate, exclude summary facts from entities, reset multi-turn on /knowledge clear
* fix: address P1 review findings for legacy graph redaction, recovery-safe clear, stable change guards
- Redact embedded secrets in migrated legacy knowledge-graph entities,
summaries, and rules via shared sanitizeLegacyText() policy
- Preserve legacy artifacts (json/db/wal/shm) as migration-backup before
/knowledge clear; resetGlobalGraph returns { archived, failures }
- Skip rewrite + index rebuild on unchanged turns by stripping the volatile
detectedAt timestamp (facts and arc session summaries)
- Always recompute the authoritative content hash in getMdStats so edits of
equal size with preserved mtime are detected and served correctly
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix: redact protocol-relative URL userinfo in legacy migration
new URL() throws for scheme-less URLs, so the catch branch now redacts
obvious //user:pass@host userinfo instead of persisting credentials.
* fix(memory): harden memdir migration and retrieval
---------
Co-authored-by: Gravirei <gravirei@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Kevin Codex <kevin@gitlawb.com>