🚸 Enable linting scripts to check whether they produce complete data lineage by sheetalgiri · Pull Request #3846 · laminlabs/lamindb

sheetalgiri · 2026-08-06T13:17:44Z

Added verify_lineage() function that parses a Python script with ast and checks:
- lineage tracking calls (ln.track(...) / ln.finish(...) or imported equivalents)
- Finds lines in the code that contain a file/folder path/URL and track them, and check if these these are eventually used in a LaminDB call to verify lineage tracking
Updated skills.md so that for every python script that the agent creates/plans to run, it verifies the lineage by calling the function and doing assertions on the verification results, make sure everything passes. If there is a failure, it goes back and makes modifications to the script and runs the function again.
Update skills.md to create artifacts from in-memory objects when possible using from_dataframe(), from_anndata(),etc instead of saving it to disk and doing save()

Why:

Agents can be unreliable when it comes to doing the lineage tracking properly

Limitations:

Only supports python scripts

codecov · 2026-08-06T13:28:57Z

Codecov Report

❌ Patch coverage is 91.66667% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.04%. Comparing base (112244e) to head (be32063).
⚠️ Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
lamindb/core/_verify_lineage.py	91.62%	19 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3846      +/-   ##
==========================================
- Coverage   91.55%   87.04%   -4.52%     
==========================================
  Files          88       89       +1     
  Lines       15747    16066     +319     
==========================================
- Hits        14417    13984     -433     
- Misses       1330     2082     +752

Files with missing lines	Coverage Δ
lamindb/core/__init__.py	`62.50% <100.00%> (-11.42%)`	⬇️
lamindb/core/_context.py	`89.41% <ø> (ø)`
lamindb/core/_verify_lineage.py	`91.62% <91.62%> (ø)`

... and 20 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-08-06T13:29:36Z

Deployment URL: https://6a5bda96.lamindb.pages.dev

…d checks: - lineage tracking calls (ln.track(...) / ln.finish(...) or imported equivalents) - LaminDB input retrieval patterns (e.g. ln.Artifact.get(...), ln.Collection.filter(...), etc.) - LaminDB output persistence patterns (.save() / ln.save(...)) - suspicious non-LaminDB input reads (e.g. pd.read_*, open, np.load, etc.)

- Instead of having checks like has_lamindb_inputs and has_lamindb_outputs, use checks has_external_inputs and has_external_outputs. in this case even if there is no input/output a script can be correct - if there are external inputs/outputs printing those function calls, to help guide the agent in what it should fix

lamin-cli: points to main (#254 merged). lamin-skills: points to the latest commit on sheetalgiri-patch-1 (includes the --session-id finish fix on top of verify_lineage).

…taframe(), etc

lamin-cli: points to main (#255 merged). lamin-skills: points to the latest commit on the still-open sheetalgiri-patch-1 branch.

- checks for file names in a script like 'x.txt'or 's.npy'or anything and whether or not they are associated with a lamin call or not , regardless of what it is, directly or indirectly, if all file names have this kind of association the function passes the script

falexwolf · 2026-08-13T13:00:39Z

+      `ln.track()` and `ln.finish()`.
+    - Extracts path-like strings from literals, variables, call arguments,
+      and simple path joins (string `+` or path `/` operations).
+    - Classifies path usage as tracked when it appears in LaminDB calls


I see quite a lot of logic related to this. Why not just flag paths that are not going through lamindb? Wouldn't this enable us to eliminate a lot of the code?

This was my initial idea but in practice I saw agents using non-lamindb write functions in combination with lamindb save. Its also inevitable in general because we would have to do this for file formats other than those where we have ln.Artifact.from_* methods like from_dataframe or from_anndata

falexwolf · 2026-08-13T13:00:54Z

+      and simple path joins (string `+` or path `/` operations).
+    - Classifies path usage as tracked when it appears in LaminDB calls
+      (`ln.*`, `lamindb.*`, including `ln.Artifact(...).save()` patterns),
+      and flags paths that only appear in non-LaminDB calls.


This is the important thing.

falexwolf · 2026-08-13T13:03:02Z

+    assert result.missing_lineage == ()
+
+
+def test_verify_lineage_positive_output_from_dataframe(tmp_path: Path):


I would combine this test and the previous one through parametrization. Both tests are very simple baseline tests.

falexwolf · 2026-08-13T13:04:21Z

+    assert result.missing_lineage == ()
+
+
+def test_verify_lineage_imported_symbols_match_current_behavior(tmp_path: Path):


What is the reason for testing this false positive behavior? 🤔

this is a mistake, it should be tracked

falexwolf · 2026-08-13T13:04:56Z

+    assert "Missing ln.finish() call in script." in result.missing_lineage
+
+
+def test_verify_lineage_positive_zero_io_script(tmp_path: Path):


I would merge this into tests 1 & 2 through parametrization as the structure is the same

falexwolf · 2026-08-13T13:06:08Z

+    assert result.missing_lineage == ()
+
+
+def test_verify_lineage_negative_missing_lineage_tracking_calls(tmp_path: Path):


What about @ln.flow()? If I had a script that used @ln.flow() instead of ln.track() and ln.finish(), would that be also a false negative (meaning negative is "not tracked").

falexwolf · 2026-08-13T13:07:40Z

+    assert "Missing ln.finish() call in script." in result.missing_lineage
+
+
+def test_verify_lineage_negative_external_input_even_when_script_has_lamindb_io(tmp_path: Path):


Great test! I think creating a catalog of these negative examples and then running a parametrized test against them is the most important

What matters is to detect scripts that don't produce lineage with high confidence

falexwolf · 2026-08-13T13:08:08Z

+    assert any("./local_input.csv" in item for item in result.missing_lineage)
+
+
+def test_verify_lineage_negative_external_output_write(tmp_path: Path):


falexwolf · 2026-08-13T13:08:13Z

+    assert any("./local_output.txt" in item for item in result.missing_lineage)
+
+
+def test_verify_lineage_negative_open_read_untracked(tmp_path: Path):


falexwolf · 2026-08-13T13:08:48Z

+    assert any("./local_input.txt" in item for item in result.missing_lineage)
+
+
+def test_verify_lineage_positive_local_write_then_lamindb_save(tmp_path: Path):


That's cool that this is properly classified. Impressive!

falexwolf · 2026-08-13T13:12:46Z

Cool! 😄

(1) What about scripts that track lineage via @ln.flow()?
(2) I think I understand now why we care about classifying positive usage of ln.Artifact() etc.: because one needs to detect whether an invalid call writes a temp file that's later ingested, right?
(3) I would use parametrized tests a bit more heavily since this hinges so much on being a catalog of scripts with their error types
(4) What about docs and public API? Is this supposed to be a public API or do we want to first use this internally only? If it's a public API we should add a screenshot of the docs and a nice example. ☺️

sheetalgiri · 2026-08-14T09:59:02Z

@falexwolf thanks for the thorough feedback !

(1) What about scripts that track lineage via @ln.flow()?

This is something I missed, will add it

(2) I think I understand now why we care about classifying positive usage of ln.Artifact() etc.: because one needs to detect whether an invalid call writes a temp file that's later ingested, right?

Exactly, when the path is written to a temp file it adds it to the 'untracked' list. Once it sees that the same path is saved on Lamindb, it adds it to the 'tracked' list and then its considered resolved.

(3) I would use parametrized tests a bit more heavily since this hinges so much on being a catalog of scripts with their error types

Makes sense 👍

(4) What about docs and public API? Is this supposed to be a public API or do we want to first use this internally only? If it's a public API we should add a screenshot of the docs and a nice example. ☺️

I would say we use it internally first and get more feedback/experience using it with agents, what do you think?

falexwolf · 2026-08-14T11:05:14Z

I would say we use it internally first and get more feedback/experience using it with agents, what do you think?

Agreed!

sheetalgiri · 2026-08-14T15:24:15Z

@falexwolf I made the changes, could you please have another look?

sheetalgiri · 2026-08-14T15:31:33Z

While testing this, I noticed that our skills are kind of bad at using ln.flow(). There is no mention of using it in the SKILLS.md and then it also uses it incorrectly. It generated:

import lamindb as ln

# 1. Start global script tracking
ln.track()

# 2. Use @ln.flow() for functions nested inside an ln.track() session
@ln.flow()
def process_sample(sample_id: str):
    df = ln.examples.datasets.mini_immuno.get_dataset1()
    return ln.Artifact.from_dataframe(df, key=f"processed_{sample_id}.parquet").save()

# 3. Call the step function
artifact = process_sample(sample_id="batch_01")

# 4. Finish the global tracking session
ln.finish()

which caused the error

RuntimeError: Please use @ln.step() or clear the global run context before using @ln.flow(): no `ln.track()` or `@ln.flow(global_run='clear')`

So, ln.flow() should never be used inside ln.track(). Is that correct @falexwolf ?

I could add the skills for this but I would need to understand better when exactly we want agents to use ln.flow(), is it something meant to be used for notebooks and not for normal scripts?

FYI @Ebad371

falexwolf · 2026-08-14T15:50:41Z

So, ln.flow() should never be used inside ln.track(). Is that correct @falexwolf ?

Yes, that's correct.

While ln.flow() is ultimately the more powerful and better pattern, we can also be pragmatic for now and just focus on ln.track() and ln.finish(). We can push out dealing with @ln.flow() in the upcoming weeks, when the need arises. I use it in most of my work, so I'll run into this sooner or later.

github-actions Bot deployed to preview August 6, 2026 13:29 View deployment

github-actions Bot deployed to preview August 7, 2026 13:45 View deployment

github-actions Bot deployed to preview August 7, 2026 16:31 View deployment

github-actions Bot deployed to preview August 7, 2026 16:59 View deployment

github-actions Bot deployed to preview August 10, 2026 10:49 View deployment

github-actions Bot deployed to preview August 10, 2026 12:58 View deployment

github-actions Bot deployed to preview August 10, 2026 13:32 View deployment

github-actions Bot deployed to preview August 10, 2026 14:55 View deployment

sheetalgiri and others added 20 commits August 11, 2026 10:42

Add tests for verify_lineage

949db16

📝 Remove Copilot dropdown from ln.track() docstring; bump submodules

64cb5ad

⬆️ Bump lamin-skills submodule to latest main

04a022f

⬆️ Bump lamin-skills submodule to latest main

880bdb9

⬆️ Bump lamin-skills submodule to latest main

b678068

⬆️ Bump lamin-skills submodule to latest main

78817a1

Updated skills to use verify_lineage()

f39f621

rename missing to missing_lineage for clarity

1766e4d

Updated skills

dbfb00e

🔖 Bump lamin-cli to main, lamin-skills to sheetalgiri-patch-1 latest

015c769

lamin-cli: points to main (#254 merged). lamin-skills: points to the latest commit on sheetalgiri-patch-1 (includes the --session-id finish fix on top of verify_lineage).

Allow saving files externally if it is later saved to lamindb

fafd22a

programmatically get output methods of lamindb like save () , from_da…

011138f

…taframe(), etc

Added tests

e381282

🔖 Bump lamin-cli to main, lamin-skills to sheetalgiri-patch-1 latest

4130281

lamin-cli: points to main (#255 merged). lamin-skills: points to the latest commit on the still-open sheetalgiri-patch-1 branch.

🔖 Bump lamin-cli to main (#256 merged)

cabf1ee

Additional tests

a95a1da

Updates skills

e9b096a

sheetalgiri requested a review from falexwolf August 13, 2026 08:41

Ebad371 approved these changes Aug 13, 2026

View reviewed changes

falexwolf changed the title ~~🚸 Verify lineage for scripts created by agents using a deterministic function~~ 🚸 Enable linting scripts to check whether they produce complete data lineage Aug 13, 2026

falexwolf reviewed Aug 13, 2026

View reviewed changes

Support lineage tracking using imported lamindb symbols

0e873d8

sheetalgiri added 3 commits August 14, 2026 13:32

Add ln.flow support

b765608

Parameterize tests

b9f9708

ruff

7f85167

github-actions Bot deployed to preview August 14, 2026 13:35 View deployment

github-actions Bot deployed to preview August 14, 2026 13:37 View deployment

sheetalgiri added 2 commits August 14, 2026 15:51

Don't hardcode 'ln' alias, support any other alias for lamindb

0a73699

Additional tests

3e0bb73

github-actions Bot deployed to preview August 14, 2026 14:04 View deployment

Updated comments

be32063

github-actions Bot deployed to preview August 14, 2026 14:23 View deployment

		assert result.missing_lineage == ()


		def test_verify_lineage_positive_output_from_dataframe(tmp_path: Path):

		assert result.missing_lineage == ()


		def test_verify_lineage_imported_symbols_match_current_behavior(tmp_path: Path):

		assert "Missing ln.finish() call in script." in result.missing_lineage


		def test_verify_lineage_positive_zero_io_script(tmp_path: Path):

		assert result.missing_lineage == ()


		def test_verify_lineage_negative_missing_lineage_tracking_calls(tmp_path: Path):

		assert "Missing ln.finish() call in script." in result.missing_lineage


		def test_verify_lineage_negative_external_input_even_when_script_has_lamindb_io(tmp_path: Path):

		assert any("./local_input.csv" in item for item in result.missing_lineage)


		def test_verify_lineage_negative_external_output_write(tmp_path: Path):

		assert any("./local_output.txt" in item for item in result.missing_lineage)


		def test_verify_lineage_negative_open_read_untracked(tmp_path: Path):

		assert any("./local_input.txt" in item for item in result.missing_lineage)


		def test_verify_lineage_positive_local_write_then_lamindb_save(tmp_path: Path):

Uh oh!

Conversation

sheetalgiri commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

falexwolf commented Aug 13, 2026

Uh oh!

sheetalgiri commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

falexwolf commented Aug 14, 2026

Uh oh!

sheetalgiri commented Aug 14, 2026

Uh oh!

sheetalgiri commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

falexwolf commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sheetalgiri commented Aug 6, 2026 •

edited

Loading

codecov Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 6, 2026 •

edited

Loading

sheetalgiri commented Aug 14, 2026 •

edited

Loading

sheetalgiri commented Aug 14, 2026 •

edited

Loading