Skip to content

Commit 564d4d5

Browse files
Document that corpus.json is always written
The save format section predated corpus.json becoming unconditional, so it described the file as optional and written only when the index has a corpus. Both are now wrong: the file is always written and it is the contents that are optional. Also call out that saving without a corpus clears any corpus already at that path, since that is the part a caller can be surprised by. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent d82e5eb commit 564d4d5

1 file changed

Lines changed: 11 additions & 6 deletions

File tree

README.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -375,18 +375,23 @@ Sentinel supports both local file storage and S3 storage:
375375

376376
The storage is abstracted using `smart_open`, making it seamless to switch between storage backends.
377377

378-
A saved index is a directory of up to three files:
378+
A saved index is a directory of three files:
379379

380380
| File | Contents |
381381
|------|----------|
382382
| `sentinel_local_index_config.json` | Encoder model name, encoding kwargs, model card |
383383
| `embeddings.safetensors` | The positive and negative embedding tensors |
384-
| `corpus.json` | The original texts behind those embeddings — **optional** |
384+
| `corpus.json` | The original texts behind those embeddings — contents **optional** |
385385

386-
`corpus.json` is written whenever the index has a corpus, and it is what lets
387-
explanations name the matched sentence after a reload. Without it, `explanations`
388-
falls back to reporting the row number of the match instead of its text. Indices
389-
saved before this file existed simply lack it and continue to load normally.
386+
`corpus.json` is what lets explanations name the matched sentence after a reload.
387+
Without those texts, `explanations` falls back to reporting the row number of the
388+
match instead of its text.
389+
390+
The file itself is always written, holding nulls when the index has no corpus, so
391+
that it always describes the embeddings saved beside it. Saving an index without a
392+
corpus therefore clears any corpus already at that path, rather than leaving
393+
behind texts that describe rows which no longer exist. Indices saved before this
394+
file existed simply lack it and continue to load normally.
390395

391396
## Examples
392397
To run the notebook examples

0 commit comments

Comments
 (0)