Skip to content

redesign of updateStandaloneArtwork() #1637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
darrell-k wants to merge 7 commits into
base: artwork-scan-db
Choose a base branch
from
Draft

redesign of updateStandaloneArtwork() #1637

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
50c8587
redesign of updateStandaloneArtwork()
darrell-k Aug 15, 2026
af8169b
some fixes
darrell-k Aug 16, 2026
4da574e
rename scanned_pics columns
darrell-k Aug 17, 2026
6a5a6bd
add MySql support
darrell-k Aug 18, 2026
7c62c4d
simplify SQLITE online artwork update, I misunderstood the example co…
darrell-k Aug 18, 2026
faafb9f
fix addition & removal of embedded artwork
darrell-k Aug 19, 2026
d6932bd
Some rework
darrell-k Aug 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion SQL/SQLite/schema_optimize.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
-- This is done here as it is faster to do in sql than in the server.
--

DELETE FROM scanned_pics;
--DELETE FROM scanned_pics; ### Removed temporarily for debugging

@michaelherger michaelherger Aug 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Time to re-enable?


-- XXX This appears to not be needed anymore as contributors are properly
-- removed by the new scanner
Expand Down
18 changes: 15 additions & 3 deletions SQL/SQLite/schema_scanner.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,20 @@ CREATE INDEX scannedUrlIndex ON scanned_files (url);

DROP TABLE IF EXISTS scanned_pics;
CREATE TABLE scanned_pics (
url text NOT NULL,
folder text,
full_path text NOT NULL,
timestamp int(10),
filesize int(10)
filesize int(10),
coverid char(8),
status char(1) CHECK (status IN ('D', 'E', 'N'))
-- D = image deleted (ie used in the database (tracks.cover) but no longer existing on disk)
-- E = an existing image present in tracks.cover
-- N = a new image (eg on disk but not used in tracks.cover - might actually have been on disk before, but passed over in a previous scan)
-- NULL = we'll set status 'E' to NULL if after n&c music files have been processed the image is still being used.
-- This improves performance as we'll only process the tracks if there's also a N(ew) image
);
CREATE INDEX scannedPicUrlIndex ON scanned_pics (url);
CREATE INDEX scannedPicUrlIndex ON scanned_pics (full_path);
CREATE INDEX scannedPicDirIndex ON scanned_pics (folder);
create index scannedPicStatusidx on scanned_pics(status);
Comment on lines +23 to +25

@michaelherger michaelherger Aug 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think SQLite would only use one index per table in a request. Should we therefore have combined indexes on status and full_path, and status and folder, instead of just status? We potentially have thousands of hits with the same status, then want to further filter or group by one of the other columns.


CREATE INDEX IF NOT EXISTS trackscoveridx ON tracks(cover);

@michaelherger michaelherger Aug 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add this to one of the versioned files, too? If it's only used in the scanner (for now) we can probably get away adding it to the latest existing up files, avoiding another full wipe & rescan.

@darrell-k darrell-k Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That reminds me, I changed the INSERT to check tracks using coverid rather than cover, in case the user has updated an image without changing the file name. So I don't think this index is required any more.

https://github.com/darrell-k/slimserver/blob/4da574ef3d256ac970f2baeb026895dd28f535b8/Slim/Utils/Scanner/Local/Async.pm#L50-L57

@michaelherger michaelherger Aug 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You still might be using it around the update album artwork to first track coverid for remote and embedded images. query. But as that's also using album ID and coverid checks, I'm not sure it has much of an impact? Did you measure performance?

In any case: if you wanted to keep it, please name it, as it's not an index on the cover ID.

And for all index statements add a space or remove it everywhere between the table name and the index field 😉 .

@darrell-k darrell-k Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"idx" means index. I'll change it to "index"

@michaelherger michaelherger Aug 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hah! Right! Keep the naming convention and upper/lower/camel casing consistent to help my aging and tired eyes and brain.

2 changes: 1 addition & 1 deletion SQL/mysql/schema_optimize.sql
View file
Open in desktop

@michaelherger michaelherger Aug 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ignore MySQL - I'll remove it

Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
-- This is done here as it is faster to do in sql than in the server.
--

DELETE FROM scanned_pics;
--DELETE FROM scanned_pics; ### Removed temporarily for debugging

-- XXX This appears to not be needed anymore as contributors are properly
-- removed by the new scanner
Expand Down
18 changes: 15 additions & 3 deletions SQL/mysql/schema_scanner.sql
View file
Open in desktop

@michaelherger michaelherger Aug 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ignore MySQL

Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,20 @@ CREATE INDEX scannedUrlIndex ON scanned_files (url);

DROP TABLE IF EXISTS scanned_pics;
CREATE TABLE scanned_pics (
url text NOT NULL COLLATE NOCASE, -- URL must be case insensitive, or we might duplicate tracks if the filename changes case only (https://github.com/LMS-Community/slimserver/issues/705#issuecomment-1026229542)
folder text,
full_path text NOT NULL,
timestamp int(10),
filesize int(10)
filesize int(10),
coverid char(8),
status char(1) CHECK (status IN ('D', 'E', 'N'))
-- D = image deleted (ie used in the database (tracks.cover) but no longer existing on disk)
-- E = an existing image present in tracks.cover
-- N = a new image (eg on disk but not used in tracks.cover - might actually have been on disk before, but passed over in a previous scan)
-- NULL = we'll set status 'E' to NULL if after n&c music files have been processed the image is still being used.
-- This improves performance as we'll only process the tracks if there's also a N(ew) image
);
CREATE INDEX scannedPicUrlIndex ON scanned_pics (url);
CREATE INDEX scannedPicUrlIndex ON scanned_pics (full_path);
CREATE INDEX scannedPicDirIndex ON scanned_pics (folder);
create index scannedPicStatusidx on scanned_pics(status);

CREATE INDEX IF NOT EXISTS trackscoveridx ON tracks(cover);
Loading