Parallel CinemaDNG export + Dual ISO optimizations and RCD interpolation by fijha · Pull Request #316 · ilia3101/MLV-App

fijha · 2026-02-25T16:54:48Z

A pretty significant export speed improvement (especially for Dual ISO).

Parallel CinemaDNG export:

Based on Parallel DNG export #312 and Make Cinema DNG frames export multithread #297.
Includes fixes for white/black or partially corrupted frames, random app crashes related to dark-frame subtraction, and issues inside the applyLLRawProcObject() function.
Smoother, less jittery ETA estimation.
Safe with Dual ISO #pragma omp parallel processing, because by default omp_get_nested() == 0 and omp_get_max_active_levels == 1.
Seems stable, but further testing is appreciated.

Dual ISO optimizations:

Small but noticeable improvements.
Not as scary as it looks :) (carefully tested).

Dual ISO RCD interpolation:

This was actually easier than expected. Only a small fix was needed to make RCD work with Dual ISO or any bit depth.
The fixed 16-bit 65536.f scale was unnecessary – it’s actually better without it. No precision is lost.

Based on this 2 pull requests: ilia3101#312 ilia3101#297

masc4ii · 2026-02-28T19:25:53Z

Thank you for another PR!

In the first test, I got one very wrong frame in a 250 frames export: the frame was cropped in x and y. Never have seen something like this.
In a second test, I got a white frame, same as in my try #312

For the RCD thing: so you now took the librtprocess RCD in the same way as AMaZE, and it works with the small patch you did in librtprocess? Funny. But I can't tell, why they did this 16bit thing...
However, the RCD interpolation seems to work fine. Faster than AMaZE but without noticable different output.

fijha · 2026-02-28T21:50:50Z

I'm still doing some tests. I don't get any white/black or cropped frames anymore, but one thing that still concerns me is that multiple threads are reading from the same MLV file.

I think there is an issue with dng_get_frame(). The file_set_pos(), fseek(), and fread() could be the culprits.

Maybe just putting them inside #pragma omp critical could solve the issue. Can you try this?

static int dng_get_frame(mlvObject_t * mlv_data, dngObject_t * dng_data, uint32_t frame_index, const char *prop_filename)
{
    int ret = 0;

    FILE *fd = mlv_data->file[mlv_data->video_index[frame_index].chunk_num];

    if (isMcrawLoaded(mlv_data))
    {
        size_t stored_size;

#pragma omp critical
{
        /* Move to start of frame in file and read the RAW data */
        file_set_pos(fd, mlv_data->video_index[frame_index].block_offset, SEEK_SET);

        mr_item_t item = {};

        if (fread(&item, sizeof(mr_item_t), 1, fd) != 1)
        {
#ifndef STDOUT_SILENT
            printf("Can not read raw frame from %s\n", mlv_data->path);
#endif
            return -1;
        }

        stored_size = item.size;

        if (stored_size > dng_get_image_size(mlv_data, IMG_SIZE_UNPACKED, frame_index)) {
            dng_data->image_buf2 = realloc(dng_data->image_buf2, stored_size);
        }

        if (fread(dng_data->image_buf2, stored_size, 1, fd) != 1)
        {
#ifndef STDOUT_SILENT
            printf("Can not read raw frame from %s\n", mlv_data->path);
#endif
            return -1;
        }
}
        int64_t ret = mr_decode_video_frame((uint8_t*)dng_data->image_buf_unpacked,
                                            (uint8_t*)dng_data->image_buf2,
                                            stored_size,
                                            mlv_data->RAWI.xRes,
                                            mlv_data->RAWI.yRes,
                                            mlv_data->compression_type);

        if (ret <= 0)
        {
#ifndef STDOUT_SILENT
            printf("mcraw decoder: Failed with error code (%ld)\n", ret);
#endif
            return -1;
        }

        /* apply low level raw processing to the unpacked_frame */
        applyLLRawProcObject(mlv_data, dng_data->image_buf_unpacked, dng_data->image_size_unpacked);

        if (dng_data->raw_output_state == COMPRESSED_RAW || dng_data->raw_output_state == COMPRESSED_ORIG)
        {
            ret = dng_compress_image(dng_data->image_buf,
                                     dng_data->image_buf_unpacked,
                                     &dng_data->image_size,
                                     mlv_data->RAWI.xRes,
                                     mlv_data->RAWI.yRes,
                                     mlv_data->RAWI.raw_info.bits_per_pixel);
        }
        else   // uncompressed and fast pass
        {
            dng_pack_image_bits(dng_data->image_buf,
                                dng_data->image_buf_unpacked,
                                mlv_data->RAWI.xRes,
                                mlv_data->RAWI.yRes,
                                mlv_data->RAWI.raw_info.bits_per_pixel,
                                1);
        }
    }
    else
    {
        if (dng_data->raw_input_state == COMPRESSED_RAW) /* If lossless, decompress or pass trough */
        {
#pragma omp critical
{
            /* Move to start of frame in file and read the RAW data */
            file_set_pos(fd, mlv_data->video_index[frame_index].frame_offset, SEEK_SET);

            dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_LOSLESS, frame_index);
            if(fread(dng_data->image_buf, dng_data->image_size, 1, fd) != 1)
            {
#ifndef STDOUT_SILENT
                printf("Can not read raw frame from %s\n", mlv_data->path);
#endif
            }
}
            if(dng_data->raw_output_state == COMPRESSED_ORIG)
            {
                // do nothing, compressed raw data is ready to save unchanged
            }
            else
            {
                ret = dng_decompress_image(dng_data->image_buf_unpacked,
                                           dng_data->image_buf,
                                           dng_data->image_size,
                                           mlv_data->RAWI.xRes,
                                           mlv_data->RAWI.yRes,
                                           mlv_data->RAWI.raw_info.bits_per_pixel);

                /* apply low level raw processing to the unpacked_frame */
                applyLLRawProcObject(mlv_data, dng_data->image_buf_unpacked, dng_data->image_size_unpacked);

                if(dng_data->raw_output_state == COMPRESSED_RAW)
                {
                    ret = dng_compress_image(dng_data->image_buf,
                                             dng_data->image_buf_unpacked,
                                             &dng_data->image_size,
                                             mlv_data->RAWI.xRes,
                                             mlv_data->RAWI.yRes,
                                             (llrpHQDualIso(mlv_data)) ? 16 : mlv_data->RAWI.raw_info.bits_per_pixel);
                }
                else
                {
                    if(!llrpHQDualIso(mlv_data))
                    {
                        dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_PACKED, frame_index);
                        dng_pack_image_bits(dng_data->image_buf,
                                            dng_data->image_buf_unpacked,
                                            mlv_data->RAWI.xRes,
                                            mlv_data->RAWI.yRes,
                                            mlv_data->RAWI.raw_info.bits_per_pixel,
                                            1);
                    }
                    else
                    {
                        dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_UNPACKED, frame_index);
                        memcpy(dng_data->image_buf, dng_data->image_buf_unpacked, dng_data->image_size);
                    }
                }
            }
        }
        else /* If uncompressed, unpack to 16bit or pass trough */
        {
#pragma omp critical
{
            /* Move to start of frame in file and read the RAW data */
            file_set_pos(fd, mlv_data->video_index[frame_index].frame_offset, SEEK_SET);

            dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_PACKED, frame_index);
            if(fread(dng_data->image_buf, dng_data->image_size, 1, fd) != 1)
            {
#ifndef STDOUT_SILENT
                printf("Can not read raw frame from %s\n", mlv_data->path);
#endif
            }
}
            if(dng_data->raw_output_state == UNCOMPRESSED_ORIG)
            {
                dng_reverse_byte_order(dng_data->image_buf, dng_data->image_size);
            }
            else
            {
                dng_unpack_image_bits(dng_data->image_buf_unpacked,
                                      dng_data->image_buf,
                                      mlv_data->RAWI.xRes,
                                      mlv_data->RAWI.yRes,
                                      mlv_data->RAWI.raw_info.bits_per_pixel);

                /* apply low level raw processing to the unpacked_frame */
                applyLLRawProcObject(mlv_data, dng_data->image_buf_unpacked, dng_data->image_size_unpacked);

                if(dng_data->raw_output_state == COMPRESSED_RAW)
                {
                    ret = dng_compress_image(dng_data->image_buf,
                                             dng_data->image_buf_unpacked,
                                             &dng_data->image_size,
                                             mlv_data->RAWI.xRes,
                                             mlv_data->RAWI.yRes,
                                             (llrpHQDualIso(mlv_data)) ? 16 : mlv_data->RAWI.raw_info.bits_per_pixel);
                }
                else
                {
                    if(!llrpHQDualIso(mlv_data))
                    {
                        dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_PACKED, frame_index);
                        dng_pack_image_bits(dng_data->image_buf,
                                            dng_data->image_buf_unpacked,
                                            mlv_data->RAWI.xRes,
                                            mlv_data->RAWI.yRes,
                                            mlv_data->RAWI.raw_info.bits_per_pixel,
                                            1);
                    }
                    else
                    {
                        dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_UNPACKED, frame_index);
                        memcpy(dng_data->image_buf, dng_data->image_buf_unpacked, dng_data->image_size);
                    }
                }
            }
        }
    }

    dng_fill_header(mlv_data, dng_data, frame_index, prop_filename);
    return ret;
}

fijha · 2026-02-28T23:35:34Z

Actually, we already have the mlv_data->main_file_mutex, so this might be slightly better:

static int dng_get_frame(mlvObject_t * mlv_data, dngObject_t * dng_data, uint32_t frame_index, const char *prop_filename)
{
    int ret = 0;

    uint16_t chunk = mlv_data->video_index[frame_index].chunk_num;

    FILE *fd = mlv_data->file[chunk];

    pthread_mutex_lock(mlv_data->main_file_mutex + chunk);

    if (isMcrawLoaded(mlv_data))
    {
        /* Move to start of frame in file and read the RAW data */
        file_set_pos(fd, mlv_data->video_index[frame_index].block_offset, SEEK_SET);

        mr_item_t item = {};

        if (fread(&item, sizeof(mr_item_t), 1, fd) != 1)
        {
#ifndef STDOUT_SILENT
            printf("Can not read raw frame from %s\n", mlv_data->path);
#endif
            pthread_mutex_unlock(mlv_data->main_file_mutex + chunk);
            return -1;
        }

        size_t stored_size = item.size;

        if (stored_size > dng_get_image_size(mlv_data, IMG_SIZE_UNPACKED, frame_index)) {
            dng_data->image_buf2 = realloc(dng_data->image_buf2, stored_size);
        }

        if (fread(dng_data->image_buf2, stored_size, 1, fd) != 1)
        {
#ifndef STDOUT_SILENT
            printf("Can not read raw frame from %s\n", mlv_data->path);
#endif
            pthread_mutex_unlock(mlv_data->main_file_mutex + chunk);
            return -1;
        }

        pthread_mutex_unlock(mlv_data->main_file_mutex + chunk);

        int64_t ret = mr_decode_video_frame((uint8_t*)dng_data->image_buf_unpacked,
                                            (uint8_t*)dng_data->image_buf2,
                                            stored_size,
                                            mlv_data->RAWI.xRes,
                                            mlv_data->RAWI.yRes,
                                            mlv_data->compression_type);

        if (ret <= 0)
        {
#ifndef STDOUT_SILENT
            printf("mcraw decoder: Failed with error code (%ld)\n", ret);
#endif
            return -1;
        }

        /* apply low level raw processing to the unpacked_frame */
        applyLLRawProcObject(mlv_data, dng_data->image_buf_unpacked, dng_data->image_size_unpacked);

        if (dng_data->raw_output_state == COMPRESSED_RAW || dng_data->raw_output_state == COMPRESSED_ORIG)
        {
            ret = dng_compress_image(dng_data->image_buf,
                                     dng_data->image_buf_unpacked,
                                     &dng_data->image_size,
                                     mlv_data->RAWI.xRes,
                                     mlv_data->RAWI.yRes,
                                     mlv_data->RAWI.raw_info.bits_per_pixel);
        }
        else   // uncompressed and fast pass
        {
            dng_pack_image_bits(dng_data->image_buf,
                                dng_data->image_buf_unpacked,
                                mlv_data->RAWI.xRes,
                                mlv_data->RAWI.yRes,
                                mlv_data->RAWI.raw_info.bits_per_pixel,
                                1);
        }
    }
    else
    {
        /* Move to start of frame in file and read the RAW data */
        file_set_pos(fd, mlv_data->video_index[frame_index].frame_offset, SEEK_SET);

        if (dng_data->raw_input_state == COMPRESSED_RAW) /* If lossless, decompress or pass trough */
        {
            dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_LOSLESS, frame_index);
            if(fread(dng_data->image_buf, dng_data->image_size, 1, fd) != 1)
            {
#ifndef STDOUT_SILENT
                printf("Can not read raw frame from %s\n", mlv_data->path);
#endif
            }

            pthread_mutex_unlock(mlv_data->main_file_mutex + chunk);

            if(dng_data->raw_output_state == COMPRESSED_ORIG)
            {
                // do nothing, compressed raw data is ready to save unchanged
            }
            else
            {
                ret = dng_decompress_image(dng_data->image_buf_unpacked,
                                           dng_data->image_buf,
                                           dng_data->image_size,
                                           mlv_data->RAWI.xRes,
                                           mlv_data->RAWI.yRes,
                                           mlv_data->RAWI.raw_info.bits_per_pixel);

                /* apply low level raw processing to the unpacked_frame */
                applyLLRawProcObject(mlv_data, dng_data->image_buf_unpacked, dng_data->image_size_unpacked);

                if(dng_data->raw_output_state == COMPRESSED_RAW)
                {
                    ret = dng_compress_image(dng_data->image_buf,
                                             dng_data->image_buf_unpacked,
                                             &dng_data->image_size,
                                             mlv_data->RAWI.xRes,
                                             mlv_data->RAWI.yRes,
                                             (llrpHQDualIso(mlv_data)) ? 16 : mlv_data->RAWI.raw_info.bits_per_pixel);
                }
                else
                {
                    if(!llrpHQDualIso(mlv_data))
                    {
                        dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_PACKED, frame_index);
                        dng_pack_image_bits(dng_data->image_buf,
                                            dng_data->image_buf_unpacked,
                                            mlv_data->RAWI.xRes,
                                            mlv_data->RAWI.yRes,
                                            mlv_data->RAWI.raw_info.bits_per_pixel,
                                            1);
                    }
                    else
                    {
                        dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_UNPACKED, frame_index);
                        memcpy(dng_data->image_buf, dng_data->image_buf_unpacked, dng_data->image_size);
                    }
                }
            }
        }
        else /* If uncompressed, unpack to 16bit or pass trough */
        {
            dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_PACKED, frame_index);
            if(fread(dng_data->image_buf, dng_data->image_size, 1, fd) != 1)
            {
#ifndef STDOUT_SILENT
                printf("Can not read raw frame from %s\n", mlv_data->path);
#endif
            }

            pthread_mutex_unlock(mlv_data->main_file_mutex + chunk);

            if(dng_data->raw_output_state == UNCOMPRESSED_ORIG)
            {
                dng_reverse_byte_order(dng_data->image_buf, dng_data->image_size);
            }
            else
            {
                dng_unpack_image_bits(dng_data->image_buf_unpacked,
                                      dng_data->image_buf,
                                      mlv_data->RAWI.xRes,
                                      mlv_data->RAWI.yRes,
                                      mlv_data->RAWI.raw_info.bits_per_pixel);

                /* apply low level raw processing to the unpacked_frame */
                applyLLRawProcObject(mlv_data, dng_data->image_buf_unpacked, dng_data->image_size_unpacked);

                if(dng_data->raw_output_state == COMPRESSED_RAW)
                {
                    ret = dng_compress_image(dng_data->image_buf,
                                             dng_data->image_buf_unpacked,
                                             &dng_data->image_size,
                                             mlv_data->RAWI.xRes,
                                             mlv_data->RAWI.yRes,
                                             (llrpHQDualIso(mlv_data)) ? 16 : mlv_data->RAWI.raw_info.bits_per_pixel);
                }
                else
                {
                    if(!llrpHQDualIso(mlv_data))
                    {
                        dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_PACKED, frame_index);
                        dng_pack_image_bits(dng_data->image_buf,
                                            dng_data->image_buf_unpacked,
                                            mlv_data->RAWI.xRes,
                                            mlv_data->RAWI.yRes,
                                            mlv_data->RAWI.raw_info.bits_per_pixel,
                                            1);
                    }
                    else
                    {
                        dng_data->image_size = dng_get_image_size(mlv_data, IMG_SIZE_UNPACKED, frame_index);
                        memcpy(dng_data->image_buf, dng_data->image_buf_unpacked, dng_data->image_size);
                    }
                }
            }
        }
    }

    dng_fill_header(mlv_data, dng_data, frame_index, prop_filename);
    return ret;
}

masc4ii · 2026-03-03T19:14:06Z

In new tests I did not get any errors anymore.

It looks there are many many changes - also in dualiso code. I hope you know what you're doing :-D as it worked very nicely in latest PRs.

Why darkframe was changed? You wrote about crashes? How to get them?

For the StatusDialog: the averaging just is usefull if all clips have the same amout of processing, right? In my tests I first had a heavy clip, then a clip with small resolution... for the first the ETA was nice, for the second it was a joke (very very wrong numbers). Haha. ;-)

fijha · 2026-03-03T20:02:05Z

Thanks for testing! The changes in darkframe.c prevent loading and freeing the same dark-frame if it is already loaded. This is important in our parallel processing because the free(video->llrawproc->dark_frame_data); would otherwise cause a race condition and random crashes. It is also an optimization. We don't need to load the same dark-frame for every single frame, but only once or if it has changed.

I also don't get any errors anymore. It works really well. The Dual ISO changes, as crazy as they look :D, are just small optimizations, but together they make a difference in speed. I tested and compared every single change.

fijha · 2026-03-03T20:36:55Z

StatusDialog: Yes, the ETA can't be accurate and it is almost impossible to calculate if there are very different clips (especially a mix of Dual ISO and non-Dual ISO, or different resolutions). However, I'm already working on a better status dialog that shows the progress of the current clip as well as the total progress. The averaging will reset with every clip. I also added the ability to pause/resume the CinemaDNG export. Just need a bit more testing :).

Display progress and remaining/elapsed time of the current clip, as well as total progress. Ability to pause/resume CinemaDNG export. Use a monospace font for the timer label to prevent text "dancing". Show a confirmation dialog when clicking the "Abort" button.

fijha · 2026-03-05T14:53:02Z

Improved StatusDialog with ability to pause/resume CinemaDNG export:

Display progress and remaining/elapsed time of the current clip, as well as total progress.
Ability to pause/resume CinemaDNG export.
Use a monospace font for the timer label to prevent text "dancing" :)
Show a confirmation dialog when clicking the "Abort" button. This also acts as a pause feature for non-CinemaDNG exports, as the confirmation dialog pauses the for loop.

TODO: Calculate theoretical CPU load for each clip based on resolution and recipe settings and use it for accurate total remaining time estimation. I think this is possible :)

Add m_isLoopRunning flag in StatusDialog to prevent double-clicks on the pause button while the parallel loop is still running.

Remove LJ92 flag check for raw scaling. This ensures low-range raw data is normalized regardless of the LJ92 flag.

fijha added 3 commits February 25, 2026 15:43

Dual ISO optimizations

21e8549

Parallel CinemaDNG export

20e914e

Based on this 2 pull requests: ilia3101#312 ilia3101#297

Dual ISO RCD interpolation

9c6c87f

fijha changed the title ~~Parallel CinemaDNG export + Dual ISO optimizations~~ Parallel CinemaDNG export + Dual ISO optimizations and RCD interpolation Feb 27, 2026

Slightly more refined and safer StatusDialog::drawTimeFromToDoFrames()

21ae4b4

Fix race condition in dng_get_frame() by adding per-chunk file mutex

f688c41

fijha added 14 commits March 5, 2026 23:57

Naming convention fix

1f03916

StatusDialog: prevent double-clicks on the pause button

8387c8b

Add m_isLoopRunning flag in StatusDialog to prevent double-clicks on the pause button while the parallel loop is still running.

Cleanup whitespace in recently modified files

bf09130

Fix missing tr() for translations

3ea3998

Replace "x" with the proper multiplication sign "×"

c679dd5

Minor changes/modifications

e565939

Check disk space before export

5c8c4f1

Show elapsed time in export finished notification

0c3f254

Fix Dual ISO raw scaling in uncompressed MLV

0fa3944

Remove LJ92 flag check for raw scaling. This ensures low-range raw data is normalized regardless of the LJ92 flag.

Minor improvements

863e566

Fix Dual ISO raw scaling in uncompressed MLV

bd2a27e

Cleanup whitespace

5504949

Small revert/update in dualiso.c

f49e578

Merge remote-tracking branch 'upstream/master' into Optimizations

eeeaccb

Navigation Menu

Parallel CinemaDNG export + Dual ISO optimizations and RCD interpolation - #316

Parallel CinemaDNG export + Dual ISO optimizations and RCD interpolation#316
fijha wants to merge 20 commits into
ilia3101:masterfrom
fijha:Optimizations

fijha commented Feb 25, 2026 •

edited

Loading

Uh oh!

masc4ii commented Feb 28, 2026

Uh oh!

fijha commented Feb 28, 2026

Uh oh!

fijha commented Feb 28, 2026

Uh oh!

masc4ii commented Mar 3, 2026 •

edited

Loading

Uh oh!

fijha commented Mar 3, 2026

Uh oh!

fijha commented Mar 3, 2026

Uh oh!

fijha commented Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

fijha commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

masc4ii commented Feb 28, 2026

Uh oh!

fijha commented Feb 28, 2026

Uh oh!

fijha commented Feb 28, 2026

Uh oh!

masc4ii commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fijha commented Mar 3, 2026

Uh oh!

fijha commented Mar 3, 2026

Uh oh!

fijha commented Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fijha commented Feb 25, 2026 •

edited

Loading

masc4ii commented Mar 3, 2026 •

edited

Loading