> For the complete documentation index, see [llms.txt](https://docs.gitloom.cloud/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.gitloom.cloud/documentation/memory-improvements.md).

# GitLoom Memory — Improvement Plan

**Status:** analysis, 2026-08-01 **Scope:** retrieval quality, ranking, ingestion fidelity, schema, storage, observability **Basis:** code read at `fix/deterministic-retrieval` (`c502279`), plus a 30-query manual probe against the deployed API

***

## 0. How to read this

GitLoom's architecture is sound and unusually well-chosen for the problem:

* **Git is the source of truth.** Memories are `.md` files with typed YAML frontmatter under four tiers (`facts/`, `incidents/`, `rules/`, `skills/`). Every write is a commit, so provenance, diff, and time-travel are free.
* **SQLite is a pure derived cache.** `schemaVersion` (`internal/index/schema.go:5`) gates a full rebuild from git on any layout change. Nothing in the DB is authoritative except embeddings, which are keyed by `blob_hash` so they survive rebuilds.
* **Retrieval is three complementary arms** — BM25 over FTS5, dense vectors over cues *and* bodies, and a relationship graph — fused by reciprocal rank, with no LLM in the loop.

The problems below are **not** architectural. They are a small number of concrete defects in the query layer, plus a set of capabilities the schema anticipates but nothing yet uses (`supersedes`, `confidence`, `updated`, section-level nodes). Fixing the top five items should move measured quality more than any redesign would.

Severity key: **P0** breaks correctness today · **P1** materially caps quality · **P2** matters at scale or for the next capability tier.

***

## 1. Executive summary

| #   | Area      | Issue                                                                            | Sev    |
| --- | --------- | -------------------------------------------------------------------------------- | ------ |
| R1  | Lexical   | Single-char token `i` is prefix-matched, matching `is`/`in`/`it` in every doc    | **P0** |
| R3  | Fusion    | No relevance floor — every query returns the whole corpus, never abstains        | **P0** |
| G1  | Storage   | `WriteBatch` stages `All: true`, committing the SQLite index into git history    | **P0** |
| R5  | Fusion    | Lexical arm is counted twice and runs twice per query                            | **P1** |
| R4  | Semantic  | `vecScoreMargin = 0.35` is wider than most models' dynamic range                 | **P1** |
| R6  | Scoring   | Exposed `score` is pure RRF — carries no relevance signal, cannot be thresholded | **P1** |
| I1  | Ingest    | Unrelated facts merged into one file (GPU wattage inside a camera purchase)      | **P1** |
| I5  | Ingest    | One commit + one full index sync per memory                                      | **P1** |
| R9  | Ranking   | `supersedes`, `confidence`, `updated` stored but never used in ranking           | **P1** |
| R12 | Retrieval | No entity index — entity-pivot questions have no path to succeed                 | **P1** |
| R7  | Graph     | Graph expansion contributes \~nothing (0–10 ms, weight `1/160`)                  | **P1** |
| S2  | Schema    | No temporal validity interval; contradiction handling is implicit                | **P2** |
| R15 | Perf      | Brute-force float32 scan, twice per query, unquantized                           | **P2** |
| R16 | Quality   | No reranking stage anywhere                                                      | **P2** |
| S7  | Data      | Raw transcripts discarded — cannot re-extract with a better model                | **P2** |

***

## 2. Retrieval and ranking

### R1 — Single-character prefix matching poisons every first-person query · **P0**

`internal/search/lexical.go:17` tokenizes on alphanumerics. `:104` and `:54` emit every token as an FTS5 **prefix** term:

```go
parts = append(parts, `"`+v+`"*`)
```

The query `where do i work` therefore becomes:

```
"where"* OR "do"* OR "i"* OR "work"*
```

`"i"*` matches **is, in, it, its, if, I** — tokens present in essentially every stored memory. BM25 then ranks the entire corpus on stopword-fragment frequency.

This is directly visible in the probe output: the returned snippets bracket `[is]`, `[in]`, `[it]` as *matched terms*. On `where do i work`, the current employer (`joined-orbital-ai`) landed at **rank 7** while an unrelated conference talk took rank 1.

The stopword list at `:69` would have caught `is`/`in`/`it` — but it is applied only in ANY mode, and only to *whole* tokens. The prefix expansion smuggles them back in through `i`.

Compounding: the list omits every interrogative and first-person form — `i`, `my`, `me`, `do`, `did`, `does`, `what`, `who`, `where`, `when`, `why`, `how`, `will`, `am`. Those are the entire lexical surface of a natural memory question.

**Fix**

1. Do not prefix-match tokens shorter than 3 characters — emit them as exact terms or drop them. The FTS table already uses `porter unicode61` (`schema.go:88`), so stemming covers the morphology that prefixing was compensating for.
2. Extend the ANY-mode stopword set with interrogatives, first-person pronouns, and auxiliaries.
3. Apply stopword filtering in AND mode too, or at minimum stop letting stopwords satisfy the all-terms requirement — a memory should not qualify as an exact match because it contains "is".

**Validation:** assert that `FTSQueryAny("where do i work", nil)` contains neither `"i"*` nor a bare-`i` term. Add a corpus-level test: no query in a fixture set may return more than 60% of the corpus from the lexical arm alone.

***

### R3 — No relevance floor; the system cannot say "I don't know" · **P0**

`engine/retrieve.go:202`:

```go
ranked := rankTop(fused, limit)
```

`limit` defaults to 24. There is no score threshold anywhere after fusion. With a 9-document corpus, all 9 documents return on every query, always, ranked by a spread of roughly 7%.

Observed consequence: `who is bruno` (no Bruno in the corpus) returned nine hits led by a camera purchase. `what automobile do i own` returned nine hits about cameras and Japan. Across 14 genuinely unanswerable probe queries, the abstention rate was **0/14**.

To a downstream answering model these are indistinguishable from evidence. This is the single largest hallucination surface in the system, and it is worse than returning nothing because it is *confidently* wrong.

**Fix**

Introduce a post-fusion evidence test. RRF rank alone cannot support one — a document at rank 1 of a garbage list scores identically to rank 1 of a perfect list. The floor must consult **raw** per-arm scores:

```go
// Sketch. A hit is evidence if it clears an absolute bar in at least one arm,
// or a lower bar in two.
type armScores struct{ bm25, cue, body float64 }

func isEvidence(a armScores) bool {
    strong := a.bm25 >= bm25Floor || a.cue >= cueFloor || a.body >= bodyFloor
    corroborated := agreeCount(a) >= 2
    return strong || corroborated
}
```

Return an empty hit list when nothing clears it, and surface that as an explicit `"hits": []` rather than a degraded ranking. Calibrate the floors on the 499-instance LongMemEval run: pick the operating point that maximises F1 over (answerable → non-empty, unanswerable → empty).

**Validation:** add abstention as a first-class benchmark metric. LongMemEval's `_abs` (abstention) instances measure exactly this and are currently free points being dropped.

***

### R5 — The lexical arm is fused twice and executed twice · **P1**

`engine/retrieve.go:106` runs the lexical arm:

```go
lexical, lexErr = e.index.Search(ctx, base)
```

Concurrently, `:123` runs:

```go
semantic, semErr = e.index.HybridSearch(ctx, base, vecs[0])
```

But `HybridSearch` (`internal/index/vector.go:304`) **runs the identical lexical search again internally** and fuses it with cues and bodies:

```go
lexQ := q
lexQ.Limit = deep
lexical, err := s.Search(ctx, lexQ)   // same `base` query, second execution
...
for i, r := range lexical { fused[r.Node.Path] += 1 / (rrfK + float64(i+1)) }
```

The outer fusion then computes `RRF(lexical) + RRF(hybrid)`, so BM25 influences roughly two-thirds of the final ordering while the vector arms split the rest. Given R1, that is the worst possible weighting.

It also explains the reported timings: `vector_ms` of 70–80 ms against `lexical_ms` of 2–3 ms out of a \~78 ms total. The "semantic" timer is wrapping a second full lexical scan.

**Fix**

Pick one composition and delete the other. Preferred: make `Retrieve` fuse three *peer* arms itself —

```go
lexical := index.Search(base)          // BM25
cues    := index.CueHits(qv, deep)     // question-shaped vectors
bodies  := index.VectorHits(qv, deep)  // prose vectors
fused   := weightedRRF(lexical, cues, bodies, weights)
```

— exporting `cueHits`/`vectorHits` and reducing `HybridSearch` to a convenience wrapper for `engine.SemanticSearch`. This also makes per-arm weights tunable, which they currently are not, and makes per-arm raw scores available for R3 and R6.

***

### R4 — The vector relevance margin is wider than the signal · **P1**

`internal/index/vector.go:252`:

```go
const vecScoreMargin = 0.35
```

`trimIrrelevant` cuts at `best - 0.35` on unit-normalised cosine. The comment is candid ("Wide on purpose: embedding models pack all text into a narrow similarity band"), but 0.35 is wider than the *entire* band most sentence embedders produce over short factual text — typically 0.45–0.75 across a mixed corpus. The trim therefore never fires, which is precisely the failure `c502279` set out to fix.

Evidence: the probe's `who lives in kyoto` returned a clean `1/61, 1/62 … 1/69` ladder — one arm, nine documents, monotone, untrimmed.

**Fix**

Make the cut adaptive to the observed distribution rather than absolute:

```go
// Cut where the score distribution actually falls off, not at a fixed offset.
func cutoff(hits []scoredPath) float64 {
    best, med := hits[0].Score, median(hits)
    return math.Max(best-0.5*(best-med), absoluteFloor)
}
```

Add `absoluteFloor` as a hard minimum so a corpus with *no* relevant document trims to empty rather than to its own least-bad member. Both constants belong in config, calibrated per embedding model — the right value for `bge-*` is not the right value for an OpenAI model, and the code currently assumes one number fits all.

***

### R6 — Exposed scores carry no relevance information · **P1**

The `score` field returned to clients is pure RRF. Verified arithmetic from the probe: top hits are exactly `2/61 = 0.032786885` (rank 1 in both arms); the second is exactly `1/62 + 1/64`; unmatched queries produce the exact ladder `1/(60+n)`.

Consequences:

* A perfect match and a random document differ by \~7%. No client can threshold.
* The value cannot be cached, compared across queries, or surfaced as confidence.
* Debugging is blind: nothing in the response says *which arm* produced a hit or *how strongly*.

**Fix**

Extend `model.SearchResult` with a provenance breakdown, and return it:

```go
type SearchResult struct {
    Node    Node
    Score   float64      // fused rank score (unchanged, for ordering)
    Raw     ArmScores    // bm25, cue cosine, body cosine, graph hops
    Arms    []string     // which arms produced this hit
    Snippet string
}
```

Keep RRF for ordering — it is the right fusion primitive — but stop pretending it is a relevance measure.

***

### R7 — Graph expansion is effectively inert · **P1**

`engine/retrieve.go:192` admits neighbours at a fixed:

```go
fused[p] += 1 / (60.0 + 100.0)   // 0.00625
```

That is one-fifth of a rank-1 contribution, below every directly-matched hit by construction ("they are context, not evidence"). In the probe, `graph_ms` was 0–10 ms across all 30 queries and no result was attributable to graph expansion; the two graph-only hits that appeared ranked last both times.

The multi-hop case the graph exists for — session `friend` ("Arjun lives in Kyoto") plus session `friend2` ("During my Japan trip I will stay with **him**") — produced no linked fact at all. The pronoun was never resolved across the session boundary, so no edge was ever written.

**Fix**

Three changes, in order of value:

1. **Make edges earn rank.** A neighbour reached from a *strong* hit via a *labelled* edge is evidence, not decoration. Score it as a function of the source's fused score and hop count: `neighbourScore = α · sourceScore / (hops + 1)`, with `α ≈ 0.5`. A dangling or unlabelled edge stays at the current floor.
2. **Expand selectively, not uniformly.** Expansion currently fires on the top 6 hits regardless of whether the query needs it. Multi-hop and enumeration questions need it; lookup questions do not. Gate on query shape (see R13) or on whether the top hits are weak.
3. **Fix cross-session coreference at ingest** (see I7) — the graph cannot traverse an edge that was never written.

***

### R8/R9/R10 — Stored ranking signals that nothing reads · **P1**

The schema carries four signals that the query path completely ignores:

| Column       | Where stored                              | Used in ranking?                  |
| ------------ | ----------------------------------------- | --------------------------------- |
| `updated`    | `nodes.updated`, indexed (`schema.go:49`) | **No** — only as a filter bound   |
| `supersedes` | `nodes.supersedes` (`schema.go:28`)       | **No** — never read at query time |
| `confidence` | `nodes.confidence` (`schema.go:24`)       | **No** — scanned, then dropped    |
| `tier`       | `nodes.tier`, indexed                     | **No** — filter only, no prior    |

This is why `where do i work` can rank a stale employer above a current one on lexical noise, and why `was i always non vegetarian` has no mechanism to prefer the *earliest* record over the latest.

**Fix**

Add a post-fusion prior, small enough not to override genuine relevance:

```go
// Applied after fusion, before truncation.
score *= recencyBoost(node.Updated, now)   // mild log-decay, e.g. 1.0 → 0.85 over a year
score *= confidencePrior(node.Confidence)  // default 1.0 when unset
score *= tierPrior(node.Tier, queryShape)  // rules outrank incidents for "should I…"
if isSuperseded(node.Path) { score *= supersededPenalty }  // e.g. 0.3
```

`isSuperseded` needs a reverse index — add `CREATE INDEX idx_nodes_supersedes ON nodes(supersedes)` so "is anything superseding me?" is a lookup rather than a scan.

Make every multiplier configurable and default them to 1.0, so the change is inert until calibrated on the benchmark. **Do not ship priors you have not measured** — a recency boost is actively wrong for `was i always…` questions, which is why R13 (query shape) should land alongside it.

***

### R12 — No entity index · **P1**

Questions that pivot on an entity — `who lives in kyoto`, `who is bruno`, `where does arjun live` — have no retrieval path that can reliably succeed, even with the right data stored. They depend on a rare token appearing verbatim in the body or in an LLM-authored cue. When it does, retrieval is excellent (`who is mellob` returned exactly one hit, the correct one). When the cue writer phrased it differently, the memory is unreachable.

The graph in `links` is **document-to-document only**. There are no entity nodes, so "everything GitLoom knows about Arjun" is not a query the system can express.

**Fix**

Add an entity layer, derived from git like everything else:

```sql
-- Entities mentioned by memories. Derived from frontmatter `entities:` and
-- resolved [[wikilinks]]; rebuildable, therefore not a source of truth.
CREATE TABLE entities (
  id           INTEGER PRIMARY KEY,
  canonical    TEXT NOT NULL,     -- "Arjun"
  kind         TEXT NOT NULL,     -- person | place | org | product | event
  UNIQUE(canonical, kind)
);
CREATE TABLE entity_aliases (
  alias     TEXT NOT NULL,        -- lowercased surface form
  entity_id INTEGER NOT NULL,
  PRIMARY KEY(alias, entity_id)
);
CREATE TABLE entity_mentions (
  entity_id INTEGER NOT NULL,
  path      TEXT NOT NULL,        -- memory that mentions it
  role      TEXT NOT NULL DEFAULT '',  -- subject | object | location | …
  PRIMARY KEY(entity_id, path, role)
);
CREATE INDEX idx_mentions_path ON entity_mentions(path);
```

This reuses the existing `vocab`/`vocab_alias` pattern (`schema.go:94-112`), which already solves alias resolution for terms — entities are the same problem with a different namespace.

Then add a fourth retrieval arm: detect entity mentions in the query, look up mentions, and fuse. `who lives in kyoto` becomes "entities of kind `place` named Kyoto → memories mentioning them", which is exact rather than probabilistic.

Ingestion already runs a relations pass (`cortex/prompts.go:60`, `relationsSystem`) that identifies "the same real-world entity". Extend that call to *name* the entity rather than only linking documents — it is nearly free, since the model is already reasoning about entity identity.

***

### R13 — No query understanding · **P2**

Every query is treated identically: same arms, same weights, same expansion, same limit. But the probe queries fall into distinct shapes with incompatible needs:

| Shape        | Example                           | Needs                                      |
| ------------ | --------------------------------- | ------------------------------------------ |
| Lookup       | `what lens do i own`              | Precision, latest value, no expansion      |
| Temporal     | `where did i work **before**`     | Ordering by validity, *not* recency        |
| Persistence  | `was i **always** non vegetarian` | Earliest + latest, contradiction detection |
| Multi-hop    | `who am i staying with`           | Graph expansion, entity resolution         |
| Aggregation  | `how much rent do i pay`          | Numeric extraction, unit awareness         |
| Unanswerable | `what automobile do i own`        | Confident abstention                       |

**Fix**

A cheap deterministic classifier (regex/keyword, no model call) that sets retrieval parameters:

```go
type QueryShape struct {
    Temporal    bool  // "before", "used to", "still", "always", "first", "now"
    MultiHop    bool  // pronoun with no antecedent, or 2+ entity mentions
    Enumerating bool  // "all", "which ones", "list", "how many"
    Comparative bool  // "instead of", "rather than", "over"
}
```

Then: `Temporal` disables the recency prior and widens the date window; `MultiHop` raises `ExpandHops` and the graph weight; `Enumerating` raises `limit` and lowers the floor. This is a few dozen lines and unlocks most of R8's value safely.

***

### R14 — The AND→ANY fallback is a cliff · **P2**

`internal/index/query.go:259`:

```go
const minAndHits = 5
```

If the all-terms search yields fewer than 5 hits, an ANY-mode search backfills beneath it. This is binary: at 4 hits the whole noisy ANY list is appended; at 5, none of it is. There is no blending, and the constant is unexplained.

**Fix** Blend rather than switch: always run both, fuse by RRF with the AND list weighted higher (it is strictly more precise). This removes the cliff and the magic number, and costs one extra FTS query that was already running half the time.

***

### R15 — Vector scan cost and duplication · **P2**

`cueHits` (`vector.go:93`) and `vectorHits` (`vector.go:211`) each perform a full table scan, decode every BLOB into a fresh `[]float32`, and compute a scalar dot product in pure Go. Two scans per query, no reuse.

At today's scale (thousands of nodes) this is genuinely fine and the CGo-free tradeoff documented at `schema.go:132-139` is the right call. It stops being fine at \~100k memories × 1024 dims — 400 MB of allocation and traversal per query.

**Fix, in order:**

1. **Stop allocating per row.** `embed.Decode` (`embed.go:146`) allocates a new slice for every vector on every query. Decode into a reusable buffer, or store vectors in one contiguous `mmap`-ed arena keyed by row offset.
2. **Scalar-quantise to int8.** 4× memory reduction, \~4× throughput, negligible recall loss at these dimensions. Keep float32 as the rebuild source.
3. **One scan, two scores.** Cues and bodies live in separate tables; a combined vector table with a `kind` column halves the traversal.
4. **Only then** consider ANN. A flat IVF over quantised vectors in pure Go is \~200 lines and preserves the CGo-free property; do not reach for `sqlite-vec` and give up `modernc.org/sqlite`.

***

### R16 — No reranking stage · **P2**

Retrieval goes fusion → truncate → return. There is no reranking, so the top-k ordering is exactly what RRF produced. Given that P\@1 measured 75% while Recall\@3 measured 94%, **the correct document is usually retrieved but sometimes mis-ordered** — which is precisely the gap a reranker closes.

**Fix** Add an optional rerank stage over the top \~20 fused hits, behind the existing `Embedder`-style injection so the engine stays model-agnostic:

```go
type Reranker interface {
    Rerank(ctx context.Context, query string, docs []string) ([]float64, error)
}
```

A small cross-encoder adds \~20 ms and typically converts most of a Recall\@3-minus-P\@1 gap. Keep it optional: `nil` reranker must degrade to today's behaviour exactly.

***

### R17 — Section-level retrieval is built but unused · **P2**

The schema models sections as first-class nodes (`kind='section'`, with `level`, `start_line`, `end_line`, `parent_path`, `summary` — `schema.go:32-38`), and `Tree()` exposes a PageIndex-style navigable table of contents. Cues can already target a section (`cues.section`). But retrieval returns file-level hits, and snippets are file-level excerpts.

Today's memories are short enough that this doesn't bite. It will as soon as consolidation (I8) starts merging facts into longer documents — at which point returning a whole file as "the hit" wastes context and dilutes the vector.

**Fix** Prefer the section address when a cue targets one (`cueHits` already computes this at `vector.go:118-122` and then the engine discards the distinction). Return section paths with their line spans so callers can quote precisely.

***

## 3. Ingestion and corpus quality

Retrieval quality at ingest time compounds into corpus quality, because the librarian's merge decisions are made from retrieved candidates. **Every R-fix above improves ingestion for free.** These are the ingest-specific issues.

### I1 — Unrelated facts merged into one file · **P1**

Observed in the probe corpus: `facts/camera-gear/sony-a7iii-purchase-92820861.md` contains

> *"\[2026-06-18] User's RTX 5090 GPU draws nearly 600 watts under load."*

— a GPU fact inside a camera purchase record. Separately, `facts/japan-trip/japan-trip-planned-for-october-2026` absorbed camera-readiness state.

Root cause: `cortex/candidates.go:41` shows the librarian four candidates:

```go
const candidateLimit = 4
```

Those four come from `findCandidates` → `engine.Retrieve` with `AnyTerms: true` — the noisiest possible configuration of the noisiest arm. With R1 unfixed, the four candidates are near-random, and the reconciler folds facts into whichever plausible-looking neighbour it was handed.

The guard rails work as designed (`stage()` at `ingest.go:369-381` rejects hallucinated targets and double-claims), but they only verify that the target was *among the candidates* — they cannot tell that the candidate set was garbage.

**Fix**

1. Land R1 and R4 first — most of this dissolves.
2. Add a **subject-agreement check** before honouring an update: require lexical or entity overlap between the new fact and the target's existing body. A camera purchase and a GPU wattage share no subject; reject the merge and file as new. Cheap, deterministic, no model call.
3. Show the librarian *why* each candidate was retrieved (arm + raw score, from R6). A model told "this matched at cosine 0.41, weakest arm only" merges far less aggressively than one shown a bare list.

***

### I4 — `supersedes` is never written · **P1**

`model.Frontmatter.Supersedes` exists (`frontmatter.go:23`), the column exists (`schema.go:28`), and `Get` reconstructs it. But `cortex/ingest.go:299-318` builds its write args as:

```go
args := toolkit.Args{"path": it.path, "content": content, "tags": tags, "date": dateStr}
// + message, cues, related — never supersedes
```

Cortex never populates it. The contradiction-handling story is therefore entirely implicit: when a fact changes, the reconciler *appends* a dated line to the existing body (visible in the probe: the Acme file carries both `[2026-06-10] joined` and `[2026-07-30] left`).

That is actually elegant — the file becomes a mini-timeline and git holds the diff — but it means **no structured signal of which statement is current**. Retrieval sees one document containing two contradictory claims and has no way to weight them.

**Fix**

1. Have the reconciler emit an explicit verdict per fact: `extends` | `corrects` | `supersedes` | `unrelated`. It is already making this judgment; capture it.
2. On `supersedes`, write the frontmatter field and let R9's penalty demote the old record.
3. On `extends`, keep today's append behaviour.
4. Consider per-line validity in the body — `[2026-06-10 .. 2026-07-30]` rather than a bare start date — so a timeline is machine-readable rather than prose.

***

### I5 — One commit and one full index sync per memory · **P1**

`cortex/ingest.go:319` calls `remember` once per staged item, inside the loop. Each `remember` → `engine.Write` → `store.Write` (one commit, `gitstore.go:131`) → `index.Sync` (`engine.go:284`).

A session extracting 8 facts produces **8 commits and 8 index syncs**. `gitstore.WriteBatch` (`gitstore.go:158`) exists and does exactly the right thing, but its doc comment says it is "used to import synthetic corpora for benchmarking" — the real ingest path does not use it.

Given the memory note that ingestion dominates benchmark cost, this is likely a meaningful share of wall-clock on the 499-instance runs.

**Fix**

Stage all of a session's writes and commit once — one commit per session is also a *better* history granularity than one per fact, since a session is the natural unit of "what the user told us". Sync the index once afterward. Requires a `remember_batch` toolkit op and a batch-aware `engine.WriteBatch` that validates every frontmatter before staging any file.

**Blocked on G1** — `WriteBatch` is currently unsafe (see below).

***

### I6 — Filenames carry a global counter · **P2**

`cortex/ingest.go:399`:

```go
n := atomic.AddInt64(counter, 1)
items = append(items, staged{mem: m, path: fmt.Sprintf("facts/%s/%s-%d.md", topic, title, n)})
```

Every new memory gets a monotonically increasing suffix. Two consequences:

* **No stable identity.** The same fact re-extracted in a later run lands at a different path, so git history shows an add rather than a modification, and `related:` targets written by an earlier session can silently rot.
* **Duplicate accumulation.** `sony-a7iii-purchase-92820861` and a later `sony-a7iii-purchase-92820862` are, to the filesystem, unrelated files.

**Fix** Derive the suffix from a content hash of the fact's *subject* (title + primary entity), not a counter. Identical subjects then collide by construction, which routes them into the update path instead of creating a sibling. Keep a counter only as a collision breaker for genuinely distinct facts.

***

### I7 — Cross-session coreference is lost · **P1**

Within a session, coreference resolution works well — the probe showed `"It consumes nearly 600 watts"` correctly stored as `"User's RTX 5090 GPU draws nearly 600 watts."`

Across sessions it fails silently. Session `friend2` contained only *"During my Japan trip I will stay with him."* No fact was stored. `him` had no antecedent in that session, and `extract` (`ingest.go:186`) sees only the current session's text.

This is structural: extraction is in the **unordered** phase precisely so sessions can run in parallel (`ingest.go:3-20`), and it therefore cannot see prior memories.

**Fix**

Do not move extraction into the ordered phase — that would forfeit the parallelism which is the whole performance story. Instead, add a **repair pass**:

1. Extraction flags facts containing unresolved references (a pronoun or definite description with no in-session antecedent) rather than dropping them.
2. In the ordered phase — where candidates from prior sessions are already retrieved — resolve the reference against those candidates and rewrite the fact before staging.

This costs nothing when there are no dangling references and reuses the candidate retrieval already happening at `ingest.go:256`.

***

### I8 — No consolidation or compaction · **P2**

Memories only ever accumulate. There is no pass that merges near-duplicates, collapses a superseded chain into a summary, or promotes a repeatedly-confirmed fact's confidence. `ExpireIncidents` (`engine.go:350`) handles TTL'd incidents, and nothing else ever shrinks.

Over a long-lived memory this degrades retrieval directly: N near-identical records of the same fact split the vector signal N ways and crowd the top-k.

**Fix** A periodic, git-native consolidation job:

* Cluster by cue-vector similarity above a high threshold, within a topic.
* Merge clusters into one memory with a full timeline body; write `supersedes` on the retired paths.
* Commit as one `consolidate:` commit so the whole operation is a single reviewable, revertible diff.

Because git is the source of truth, this is safe in a way it would not be in a mutable store — a bad consolidation is one `git revert` away. This is a genuine architectural advantage worth exploiting.

***

### I9 — Dates are prose, not structure · **P2**

`cortex/ingest.go:293`:

```go
if !strings.Contains(content, dateStr) {
    content = "[" + dateStr + "] " + content
}
```

The session date is prepended into the body text. It is also in frontmatter `created`, but the *fact's* date (as opposed to the record's) exists only as a bracketed string inside prose.

So `what month am i visiting japan` must match "October 2026" lexically or semantically; there is no date-typed field to filter or sort on, and a body accumulating several dated lines has no machine-readable ordering.

**Fix** Extract dated assertions into a structured sidecar in frontmatter — e.g. `timeline: [{date: 2026-06-10, text: "joined Acme Robotics"}, …]` — indexed into a `assertions(path, date, text)` table. Keep the prose rendering for the model to read; add the structure for the engine to reason over. This is the substrate temporal queries (R13) need.

***

## 4. Schema and data model

### S2 — No temporal validity interval · **P2**

The frontmatter has `created`, `updated`, `ttl`, `expires_at`. All describe **when the record was written**, not **when the fact was true**. Employment from June to July, a trip in October, a dietary change in June — none of these are representable.

**Fix** Add optional `valid_from` / `valid_to` to `Frontmatter` and to `nodes`. Combined with `created`/`updated` this gives proper bitemporality: *when we learned it* × *when it was true*. Then `where did i work before` becomes a range query rather than a lexical gamble, and `was i always non vegetarian` becomes "is there more than one interval for this subject".

Frontmatter is the right home — it keeps git as the source of truth and the columns purely derived, consistent with the existing design.

### S4 — Cue vectors do not survive rebuild · **P2**

`cues` carries `blob_hash` (`schema.go:160`), but `PendingCues` (`vector.go:33`) selects on `vec IS NULL` alone. Since a schema bump wipes and re-inserts all cues, every cue vector is recomputed on every rebuild — `engine.go` acknowledges this ("cue vectors are recomputed once"). Node embeddings, by contrast, correctly survive via `blob_hash` (`vector.go:147`).

At benchmark scale that is thousands of redundant embedding calls per schema change.

**Fix** Mirror the embeddings pattern: keep a `cue_vectors(text_hash, dim, vec)` table keyed by a hash of the cue *text*, and join on rebuild. Cue text is stable across rebuilds even when row IDs are not, and identical cues across memories then share one vector.

### S6 — Body embeddings concatenate three fields · **P2**

`PendingEmbeddings` (`vector.go:143`) embeds:

```sql
TRIM(COALESCE(n.title,'') || ' ' || COALESCE(n.summary,'') || ' ' || n.body)
```

For a long or multi-fact body (which I1 actively produces), the resulting vector is an average that represents no single fact well. This is the standard chunking problem, and the schema already has the answer: sections are first-class nodes.

**Fix** Embed per section when a file has sections, per file otherwise. `PendingEmbeddings` already iterates `nodes`, which includes section rows — the filter just needs to stop collapsing them.

### S7 — Raw transcripts are discarded · **P2**

Only extracted facts are stored. If the extraction prompt improves, or a better model becomes available, there is no way to re-derive memories from the original conversations — the source is gone.

**Fix** Store transcripts under a non-indexed tier (`sessions/<date>/<id>.md`), excluded from retrieval by `walk`'s tier filter (`walk.go:202` already excludes non-tier directories, so this is nearly free). Git compresses them well, and it makes the whole corpus reproducible: *re-extract everything with the new model and diff the result* becomes a real operation. For a system whose thesis is "git as source of truth", discarding the actual source is the one place the thesis isn't honoured.

***

## 5. Git and storage

### G1 — `WriteBatch` commits the SQLite index into git history · **P0**

`internal/gitstore/gitstore.go:171`:

```go
if err := wt.AddWithOptions(&git.AddOptions{All: true}); err != nil {
```

`All: true` stages **everything in the worktree**, not just `files`. The index database lives at `RepoDir/_index/gitloom.db` by default (`engine.go:46`).

Unless `_index/` is gitignored in every repo GitLoom ever opens, `WriteBatch` commits the SQLite file — including every embedding BLOB — into git history. Because git history is append-only, this is **permanent and unrecoverable without a history rewrite**, and it grows with every batch commit.

The single-file `Write` path is safe (`wt.Add(path)`, `:142`), which is why this has not surfaced yet: only the benchmark corpus importer uses `WriteBatch`. But I5 proposes routing all ingestion through it.

**Fix — do this before I5:**

```go
for _, f := range files {
    if err := util.WriteFile(wt.Filesystem, f.Path, f.Content, 0o644); err != nil { … }
    if _, err := wt.Add(f.Path); err != nil { … }   // stage explicitly
}
```

Then, defensively: have `engine.Open` write a `.gitignore` containing `_index/` if absent, and add a test asserting no commit ever contains a path outside a known tier or `vocab/`.

### G2 — Commit granularity · **P1**

See I5. One commit per fact makes history noisy and `git log` on the repo unusable as a review surface. One commit per session, with the session date as the commit date, would make the memory's history readable — and would let `Log()` answer "what did we learn on this day" directly.

### G3 — No packing or GC strategy · **P2**

Nothing ever runs `git gc`. A long-lived memory with per-fact commits accumulates loose objects indefinitely, degrading every read.

**Fix** Repack on a schedule or after N commits. Worth pairing with I8 (consolidation), since both are "maintenance passes over a git-backed store" and can share a lock and a cadence.

***

## 6. Operations and observability

### O1 — The API drops `namespace` · **P0** *(service layer, not this repo)*

The probe posted and queried `namespace=benchmark`; every response echoed `"namespace":"test"`. The parameter is being ignored or overridden server-side, so writes and reads addressed different corpora. This invalidated an entire evaluation run before any retrieval logic was exercised.

Note that the engine in this repo has **no namespace concept at all** — isolation lives entirely in the service layer above it, with nothing in `engine` or `index` to enforce it. That is a reasonable split, but it means the engine cannot catch a tenancy bug. Worth an explicit contract test at the service boundary.

### O2 — Deployed build lagged HEAD by a critical commit · **P0**

The probe output exhibits exactly the pathology `c502279 "Make deterministic retrieval actually retrieve"` describes and claims to have fixed — full-corpus vector returns, and distractors scoring `0.0325` against a correct match at `0.0309`. Those are the commit message's own numbers.

The deployed service predates its own fix. **Any benchmark number collected before verifying the deployed SHA is unreliable.**

**Fix** Expose the build SHA in the retrieve response (`"build": "c502279"`) and assert it in the benchmark harness. A benchmark that cannot identify what it measured is not a benchmark.

### O3 — No retrieval trace · **P1**

The response gives per-arm *timings* but not per-arm *contributions*. When a result is wrong there is no way to tell whether lexical, cues, bodies, or the graph produced it. Every diagnosis in this document required reading source and reverse-engineering RRF arithmetic from the score values.

**Fix** Ship R6's `ArmScores` and add an opt-in `?explain=1` that returns, per hit: which arms matched, raw score and rank in each, whether it arrived by graph expansion and from where, and which cue matched.

### O4 — No evaluation gate · **P1**

`longmemeval/` exists and is substantial, but nothing prevents a retrieval change from regressing quality. The git history shows this has already happened at least twice — `8bff282 "Test richer retrieval context; it regresses — revert"` and `ff4a327 "Revert to the best-measured configuration"`.

**Fix** A fast retrieval-only gate in CI: a fixed 100-question fixture with known gold documents, asserting P\@1, Recall\@5, MRR, and abstention rate against committed baselines. No LLM calls, so it runs in seconds. The full 499-instance run stays a manual pre-release step.

Track these as a committed baseline file so a regression shows up as a *diff*, not a number someone has to remember.

***

## 7. Roadmap

### Phase 1 — Correctness (days)

Nothing here is speculative; all four are bugs.

1. **G1** — explicit staging in `WriteBatch`, plus `.gitignore` guarantee and a test.
2. **R1** — no prefix-matching below 3 chars; extend stopwords with interrogatives and pronouns.
3. **R5** — collapse the double lexical fusion into three peer arms.
4. **O2** — expose the build SHA; verify the deployment.

*Expected:* P\@1 from 75% → high 80s; precision\@k from 11% to something meaningful.

### Phase 2 — Calibration (1–2 weeks)

5. **R6** — carry raw per-arm scores through to the response.
6. **R4** — adaptive vector cutoff, calibrated per embedding model.
7. **R3** — post-fusion evidence floor and true abstention.
8. **O4** — retrieval-only CI gate with committed baselines.

*Expected:* abstention from 0% to a measured operating point; every subsequent change becomes measurable.

### Phase 3 — Ranking and ingest quality (2–4 weeks)

9. **R13** — deterministic query-shape classifier.
10. **R8/R9/R10** — recency, confidence, tier, and supersession priors, gated on query shape.
11. **I1** — subject-agreement check before honouring a merge.
12. **I4** — reconciler emits explicit verdicts; write `supersedes`.
13. **I5 + G2** — one commit and one sync per session.
14. **I7** — cross-session coreference repair pass.

### Phase 4 — New capability (1–2 months)

15. **R12** — entity index and a fourth retrieval arm.
16. **S2 + I9** — temporal validity intervals and structured assertions.
17. **R7** — graph edges that earn rank, selectively expanded.
18. **R16** — optional reranker.
19. **I8** — consolidation and compaction.
20. **S7** — store transcripts; make the corpus reproducible.

### Phase 5 — Scale (as needed)

21. **R15** — quantisation, arena storage, single-pass scan, then ANN.
22. **G3** — repack and GC cadence.
23. **S4/S6** — durable cue vectors, section-level embeddings.

***

## 8. Metrics to track

Today's harness reports end-to-end QA accuracy, which conflates retrieval and generation. Split them:

**Retrieval (no LLM, runs in CI):**

| Metric                      | Why                                                        |
| --------------------------- | ---------------------------------------------------------- |
| P\@1, P\@5                  | Top-of-list quality — currently 75% on answerable queries  |
| Recall\@5, Recall\@20       | Ceiling for any reranker — currently 94% @3                |
| MRR                         | Ordering quality — currently 0.85                          |
| Abstention precision/recall | Currently 0% recall; the largest single gap                |
| Arm attribution             | Share of correct hits by arm — is each earning its keep?   |
| Corpus-return ratio         | Fraction of the corpus returned per query — currently 100% |

**Ingestion:**

| Metric                      | Why                                                                                   |
| --------------------------- | ------------------------------------------------------------------------------------- |
| Fact recall                 | Facts stated vs. facts stored — the probe corpus was missing \~40% of posted sessions |
| Subject purity              | Fraction of memories containing exactly one subject — catches I1                      |
| Duplicate rate              | Near-identical memories per 100 — catches I6/I8                                       |
| Coreference resolution rate | In-session vs. cross-session — catches I7                                             |

**Operational:** p50/p95 retrieval latency by arm · commits per session · index rebuild time · embedding calls per ingested fact.

***

## 9. What is already right

Worth recording, because these are the hard parts and they should not be disturbed by any of the above:

* **Git as source of truth, SQLite as pure cache.** The `schemaVersion` rebuild contract (`schema.go:5`) is enforced and honest. Every table except `embeddings` is reconstructible, and `embeddings` survives via `blob_hash` — exactly the right exception, correctly documented.
* **Cue-based retrieval.** Embedding question-shaped keys rather than prose aligns the vector space with queries. The probe's cleanest result (`who is mellob` → exactly one hit, correct) came from this. FTS weights it 8× body (`query.go:317`), which is the right call.
* **The CGo-free tradeoff.** Brute-force scan over `modernc.org/sqlite` in exchange for a pure-Go build is correctly reasoned and correctly documented (`schema.go:132-139`). Keep it through R15.
* **Ordered/unordered ingest split.** Extraction parallel, reconciliation serial in date order (`ingest.go:3-20`) — the right decomposition, and the reason ingest is tractable at all.
* **Write-path guard rails.** `stage()` rejecting hallucinated merge targets and double-claims (`ingest.go:369-381`), and `matchCandidate`'s exact-then-unique-basename resolution (`candidates.go:182`), are exactly the defensive posture an LLM-driven write path needs.
* **Graceful degradation.** A nil `Embedder` yields a working lexical-only system, and an embedding failure never breaks a query (`retrieve.go:119`, `engine.go:183`). This is why R1 was survivable rather than fatal.
* **Dangling-link healing.** Edges stored unresolved and re-resolved on every sync (`schema.go:114-118`) means forward references fix themselves. Correct, and unusual to get right.
* **Intra-session coreference and implicit timelines.** *"It consumes nearly 600 watts"* → *"User's RTX 5090 GPU draws nearly 600 watts"*, and the Acme file carrying both its join and leave dates. The bitemporal instinct is right; it just needs structure (S2) and a ranking consumer (R9).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.gitloom.cloud/documentation/memory-improvements.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.