redesign of updateStandaloneArtwork() by darrell-k · Pull Request #1637 · LMS-Community/slimserver

darrell-k · 2026-08-15T20:27:26Z

As discussed. I hope it all makes sense.

The diff generated by git for updateStandaloneArtwork() is a bit of a mess, probably best to view the new routine as a complete replacement for the old one.

This redesign enhances the new scanned_pics table so that it can drive updateStandaloneArtwork().

I've added acoverid column so that we can read it directly from the table (in the scanner process) when we need to update tracks or albums. In order for this to work, all external coverid generation will now use the image path, not the music file URL.
There is a new status column so we can differentiate new, existing and deleted images.
The url column is renamed to path as it will now hold the file system path of the image, not a file:// URL. This makes things much easier.
There is a new dir column as discussed.

In performance testing, this runs faster, even though we are now calling findStandaloneArtwork() for every track where an image change has been detected, rather than only once for each album/image group.

This change enables TitleFormatter to do its work correctly in cases when the user has specified a variable cover id which includes a "sub-album" field like discnumber or grouping. This means that disc or grouping-specific images can be applied to tracks using this existing mechanism when everything for the album is in the same directory.

I've added some comments to new/changed code in order to aid understanding.

I'm sure at this stage there is stuff I've missed.

Signed-off-by: darrell-k <darrell@darrell.org.uk>

michaelherger

Thanks a lot! I hope to find time to actually test this later today. All my comments are just of theoretical nature. Haven't even pulled this change yet. Bear with me.

michaelherger · 2026-08-16T06:31:08Z

 		# XXX how best to delete files in non-recursive mode?
 		# Delete the directory itself and all children
 		$dbh->do("DELETE FROM scanned_files WHERE url = '${file}' OR url LIKE '${file}/%'");
+		$dbh->do("DELETE FROM scanned_pics WHERE dir LIKE '${path}/%'");


Is there a need for this? I believe the pictures table has a different use than the files: the latter really is there to iterate over and process all records. The former (scaned_pics) is a helper to look up things for the tracks. In my plans/ideas this will be more than just cover artwork, but eg. artist pictures too. We shouldn't delete that data before we're really done. Wouldn't we potentially need it at a later stage to look up box set artwork, too?

BTW: I first wanted to complain about the use of variables in the SQL statement, instead of using prepared statements. That's a typical target for SQL injection. A folder name of drop table <table name>; -- or similar could potentially cause harm... something we should probably clean up at some point. But please try to avoid using variables potentially containing user data as much as possible.

Probably not. Also with the introduction of schema_scanner.sql you're also recreating scanned_files so the existing DELETE could be removed, too.

But this has prompted a thought: without the change which I assumed was temporary for debugging, to not run schema_scanner.sql unless we're in the scanner process, we'll also clear scanned_files when schema.pm is initialised in the main process. At the moment scanned_files remains populated until a full rescan. Might this affect things like autorescanning?

michaelherger · 2026-08-16T06:33:26Z

-  filesize int(10)
+  filesize int(10),
+  coverid char(8),
+  status char(1)


please document the possible status values

michaelherger · 2026-08-16T06:41:10Z

-  filesize int(10)
+  filesize int(10),
+  coverid char(8),
+  status char(1)


Please document the possible status values.

I also talked to my bot buddy about this, and how to potentially optimise this. A first suggestion was defining the possible values:

status TEXT NOT NULL CHECK (status IN ('D', 'E', 'N'))

(or whatever the flags!)

Not that it added much to readability or performance, but DB level validation. We probably need NULL, but you get the point.

I was also wondering about using number, and then human readable constants in code. But that would make the query definitions somewhat more cumbersome.

Anyway: if the status is well defined somewhere even I should be able to learn the few characters.

michaelherger · 2026-08-16T06:43:32Z

+				WHERE NOT EXISTS (
+					SELECT path FROM scanned_pics
+					WHERE scanned_pics.path = tracks.cover
+				)


Would we need a clause here to exclude online pictures (imported from music service) from being considered deleted?

It's there immediately below. Also excludes embedded covers (the numeric check) and includes only tracks within the currently processing base directory:

AND cover NOT LIKE 'https%' AND CAST(CAST(cover AS INTEGER) AS TEXT) <> cover AND url LIKE '$basedir%'

Oh my... 🤦🏻.

Hopefully SQLite is smart enough to do these cheap checks before doing the path lookup in scanned_pics.

Why would you have to do the double casting?

Oh my... 🤦🏻.

Hopefully SQLite is smart enough to do these cheap checks before doing the path lookup in scanned_pics.

Why would you have to do the double casting?

The is no 'is numeric' function in SQLITE. But if the value survives being cast to integer and back again, it is numeric.

michaelherger · 2026-08-16T06:48:38Z

 		VALUES
-		(?, ?, ?)
+		(?, ?, ?, ?, ?,
+		CASE WHEN (SELECT COUNT(*) FROM tracks WHERE tracks.cover = ?) = 0 THEN 'N' ELSE 'E' END


I guess that's the reason for the new index on cover?

Doing a full count might be easier to read, but it's somewhat wasteful, as the DB would always have to count all the occurrences, even if we're only interested in the existence of at least one record.

CASE WHEN ( SELECT EXISTS ( SELECT 1 FROM tracks WHERE tracks.cover = ? ) ) THEN 'N' ELSE 'E' END;

Supposedly is more efficient.

Yes, I usually would use EXISTS, don't know what happened here. But this needs changing anyway because we need to check the image hasn't been updated with another of the same name. I'll be pushing a fix soon.

michaelherger · 2026-08-16T06:57:18Z

+				$file,
+				$mtime,
+				$size,
+				substr( safe_md5_hex( $file . $mtime . $size ), 0, 8 ),


Could you please add another helper for this in Artwork.pm, and use it wherever we do this calculation? It's so specific and non-obvious, having an understandable function name would not only help making sure we're always doing the same thing, but also reading the code. I had to search existing code to understand what this was.

michaelherger · 2026-08-16T06:58:31Z

+### I might have missed it, but I can't see where this might be called in main process async mode.
+### If it is, we'll need more work to populate scanned_pics in the main process or just keep a version of the old subroutine for that use.


Please don't remove this just yet... I'm a bit anxious we might be missing something. I want to double check this.

Signed-off-by: darrell-k <darrell@darrell.org.uk>

michaelherger · 2026-08-18T15:14:01Z

+CREATE INDEX scannedPicDirIndex ON scanned_pics (folder);
+create index scannedPicStatusidx on scanned_pics(status);
+
+CREATE INDEX IF NOT EXISTS trackscoveridx ON tracks(cover);


Should we add this to one of the versioned files, too? If it's only used in the scanner (for now) we can probably get away adding it to the latest existing up files, avoiding another full wipe & rescan.

That reminds me, I changed the INSERT to check tracks using coverid rather than cover, in case the user has updated an image without changing the file name. So I don't think this index is required any more.

https://github.com/darrell-k/slimserver/blob/4da574ef3d256ac970f2baeb026895dd28f535b8/Slim/Utils/Scanner/Local/Async.pm#L50-L57

Signed-off-by: darrell-k <darrell@darrell.org.uk>

…de I used Signed-off-by: darrell-k <darrell@darrell.org.uk>

Signed-off-by: darrell-k <darrell@darrell.org.uk>

darrell-k · 2026-08-19T18:04:49Z


+		# update album artwork to first track coverid
+		### I considered adding rows to scanned_pics for remote images so that they'd be processed in the loop above, but I think this is more efficient.
+		#there's a different syntax for MySql.


Moved this to after the loop and changed it to update the albums table for everything that's changed. This is part of the fix needed when adding/removing embedded artwork and also simplifies the logic. Hopefully performance won't be impacted, because we're no longer having to update the albums table in the loop.

darrell-k · 2026-08-19T18:08:39Z

-			performance => 1,
-			grouping => 1,
-			discsubtitle => 1,
-			musicbrainz_id => 1,


I've renamed to $defaultCols and updated the logic below so we can specify in the hash what the value should be in the absence of a tag from the music file.

I still need to test with performance/grouping etc, might need to set the default value in the hash to undef rather than '' but I don't think it will matter either way.

michaelherger · 2026-08-20T15:44:33Z

+		#there's a different syntax for MySql.
+		my $sql = IS_SQLITE


Is this distinction strictly necessary, or is there a performance advantage in one vs. the other? Could there be a universal solution if we sacrificed some performance? As 90% or more is identical between the two, I'd prefer not to have to deal with those subtleties.

In SQLITE, you can't issue an UPDATE over a JOIN, so they came up with their own extension to SQL, UPDATE ... FROM, which MySQL would not understand. (MySQL does allow an UPDATE over a JOIN).

The alternative would be to finally, finally, finally abandon support for MySQL.

Thinking ahead to getting the album image from the common parent folder (the separate album artwork/box set problem), we may need to move the album update for local files back into the fetch loop, we wouldn't be able to simply use the image from the first track as this one hit SQL does. (It would remain for remote tracks/images only).

Should I do this now? What do you think?

Let's drop MySQL. Just ignore it, and I'll rip it out when I'm in the mood of ripping out some stuff.

The changelog for 9.1 says "Remove support for MySQL". It's time to go beyond the warning.

redesign of updateStandaloneArtwork()

50c8587

Signed-off-by: darrell-k <darrell@darrell.org.uk>

michaelherger reviewed Aug 16, 2026

View reviewed changes

darrell-k added 2 commits August 16, 2026 15:09

some fixes

af8169b

Signed-off-by: darrell-k <darrell@darrell.org.uk>

rename scanned_pics columns

4da574e

Signed-off-by: darrell-k <darrell@darrell.org.uk>

michaelherger reviewed Aug 18, 2026

View reviewed changes

darrell-k added 3 commits August 18, 2026 17:31

add MySql support

6a5a6bd

Signed-off-by: darrell-k <darrell@darrell.org.uk>

simplify SQLITE online artwork update, I misunderstood the example co…

7c62c4d

…de I used Signed-off-by: darrell-k <darrell@darrell.org.uk>

fix addition & removal of embedded artwork

faafb9f

Signed-off-by: darrell-k <darrell@darrell.org.uk>

darrell-k commented Aug 19, 2026

View reviewed changes

michaelherger reviewed Aug 20, 2026

View reviewed changes

		### I might have missed it, but I can't see where this might be called in main process async mode.
		### If it is, we'll need more work to populate scanned_pics in the main process or just keep a version of the old subroutine for that use.

Uh oh!

Conversation

darrell-k commented Aug 15, 2026

Uh oh!

michaelherger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darrell-k Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

darrell-k Aug 16, 2026 •

edited

Loading