encode & decode 32-bit float (IEEE 754 standard) samples. - #842
encode & decode 32-bit float (IEEE 754 standard) samples.#842aliheidary1381 wants to merge 1 commit into
Conversation
|
Hi, Many thanks for filing a PR. I haven't tested it, and as you might understand, this will need a lot of testing :) I've tried doing this in the past, but didn't get very far. float PCM coding has been a often-requested feature, so this would be a great addition in that regard. I have some "objections on principal grounds" to floating point audio in FLAC. For example, floating point audio isn't meant for playback. Also, because this breaks forward compatibility. It feels to me like this will cause confusion for a lot of users, as these FLAC files will not playback on current equipment. But then again, that was also the case (and still is for some newly manufactured equipment) for 24-bit audio. I'm not saying this won't be merged, but I'll need to hear a few opinions of other people involved in the FLAC project. Still, I already know a lot of people want this. I am surprised you didn't need any new coding methods (subframe types or residual coding methods). I think this is really nice. FLAC is also used for compression measurement signals (for scientific experiments and such) where this also might be useful. Not sure whether your bit manipulation works for that as well. After I test this myself (which might take a while), I would like to propose this change through various channels with FLAC enthousiasts. However, before that, I would like to make the format changes future-proof. The problem is that the reserved bit after the bit depth bits that you use, is the last remaining reserved bit. Use of this makes it impossible for the frame header to extended any further in the future. I would like this bit to have a different function: signal an extension to the frame header with one byte of extra flag bits (or feature bits) and a channel mask. Anyway, many thanks for taking on this task. Please be patient, as merging might take quite a while, and publishing a release after that a while longer. |
|
By the way, perhaps we should also define a new STREAMINFO metadata block variant. |
|
Thx!
There's also another reserved bit in the frame header, just after the sync bits at the very beginning (
I don't think it'd be necessary. A bps of 1-3 is forbidden, according to the current standard rfc. Using them seems like a no-brainer to me. It also suggests (to the older decoders) to stop playing these (cause of the forbidden value), and requires minimal changes in the standard and other implementations. p.s. Sorry for closing the PR, wrong button 😅 |
No, there isn't. The code still refers to it, but the RFC made it part of the sync code. That is to make it distinguishable from MPEG. See here: https://lists.xiph.org/pipermail/flac-dev/2008-December/002607.html So, we only have one bit remaining.
When it is necessary to deviate from the standard, I'd like to do it in a clear and conscious way. So, there'll be more features incorporated in a new streaminfo metadata block, like increasing the max number of samples or perhaps increasing the max samplerate and the total number of samples. Still really niche stuff, so it is really only necessary for exotic stuff. The thing is, like using floats (which is a niche), there are others using FLAC for stuff it wasn't made for, like RF captures. So, this will take a long time. |
Oh, OK. |
|
I completely agree both on
There is also the 64-bit float format (in both endiannesses) - not that it is any more urgently needed for listening, but for compatibility (edit: with DAW plugins, for example) it wouldn't be a bad thing for a FLAC plugin to be able to handle "everything" the application could save. (64-bits would likely need more than 5-bit Rice, but who cares if that element also makes today's decoders err out on something they cannot decode.) Here is a part of a possible solution, if we adopt the following view:
The FLAC format admits sample rate "0" for non-audio. It could also be used for "audio in files that are stored as non-audio", with the FLAC format being used to compress the file and not just the audio stream: interpret the use of "0" as "files you need to treat as a full file", and a player/DAW should then skip it unless it knows what it is doing. There are a bunch of APPLICATION block types left to be used. You got ones for foreign metadata already, but here you might consider ones for mandatory file headers/footers (I think footer would be potentially more crucial for float than for integer, with possible metadata chunks for volume?!), and maybe one for audio properties the decoder needs to reconstruct the files (endianness, and signendness although that isn't applicable for float) - and source file extension (like WavPack does)? So the workflow of a "player"/DAW would then be:
This could also make it possible to store AIFF(/CAF) with non-integer sampling rates - or object-based audio in the BW64 format (like Monkey's and WavPack do), as there would be no need for FLAC to "understand" the objects metadata. Sure the _en_coder does need to know what a sample and a channel is, and if it is not aware of the input format it could be force-fed with "raw format options" plus OptimFROG-stype --headersize and --tailsize. As long as the source file is structured in the order header--audio--footer and nothing between audio and audio, then it would future-proof against new file types? (And past-proof enough to encode .au and A-law and µ-law ... just what the world has not been waiting for.)
In frame headers then:
And more:
|
|
How does WavPack (or its patented competitor, DST) compress 1-bit PWM streams, I don't know. I suppose we also need to reserve some space for defining new subframe types later? AFAIK, with As for the frame header, I will use bit_depth= I'm more in favour of defining a new STREAMINFO_EXTENSION metadata block type (instead of wrapping it inside an APPLICATION block), with the same workflow you said.
|
|
Oof my bad, I wrote APPLICATION when I meant "metadata". I agree yes, different (new) type - there are 120 left, not running out soon. STREAMINFO_EXTENSION to inform a "new audio type-aware" decoder/player how to play, and a "this block type-aware but won't play" decoder how to order the decoded bits wrt. endianness and signedness and interleaving. Then I suggest types to inform about and store file header and footer when those are "mandatory" to get output right; If full file headers/footers are indeed indispensable, there should be a way to get them past the 16MiB by spanning them over several blocks if necessary; whether the STREAMINFO_EXTENSION contains the info about that or there are just block types for "first" and "continued" header and ditto footer ... there are many ways.
Though I tend to disagree (I think that "subset" should not become more permissible, that would put new demands on finalized software that claims to decode subset), and maybe it has to be sorted out before committing, but ... not needed yet. |
|
Well, now that we're modifying the STREAMINFO block, we can also replace the 3-byte block size indicator with an elias gamma representation... |
|
Sorry guys... Been busy lately. |
|
This needs a really thorough review and a lot of work to make it happen. Lately, I haven't had a lot of time to work on this, and it might take quite a while before I do. |
|
The only backwards compatible way to extend the 16 MiB block size is to add a new block type, e.g. FLAC__METADATA_TYPE_EXTENDED. This is then chained one after another, and is a container for other blocks, which can then be defined with a new header. For example, a 40 MiB metadata block would need to be split up in 3 of those extended blocks. But I don't think this is needed for this float proposal, which mostly needs a new FLAC__METADATA_TYPE_STREAMINFO2 block type. +1 for 64-bit float support, but if we're thinking ahead, don't forget (b)float16_t, float128_t. Extending the blocksize is useful for big pictures, but also for future-proofing. It would be useful to have a new way of encoding metadata blocks, since you could add optional checksums, compression, etc. I can start a new discussion topic about this, maybe others have good ideas about it too. |
| subtracting an (automatically recognised) DC offset from the "exponents" channel | ||
| and storing the offset in each frame header (i.e. unsigned to signed conversion) | ||
| could be better (haven't tried it yet. hard for me to implement). | ||
| my guess would be a *consistent* ~60% ratio, at least. |
There was a problem hiding this comment.
a 10% improvement in compression might be worth investigating a bit further? Would need more discussions on how to split up the signal in "sub-channels", maybe too much complexity to be worth it.
How does the current (choice 2) compare to other coders like WavPack? If it is already as good or very close, then it might not be worth it.
There was a problem hiding this comment.
The current choice (2) that I've implemented performs a little bit better than Monkey's Audio .ape, which also implements option 2, but performs worse than WavPack, which implements option 3. It IS worth it. I don't know if I'll be able to do it tho.
41998ed to
c7dfe69
Compare
…standard) samples. I've also added support to encode & decode raw, wave riff & aiff float formats. The float encoding feature achieves a near 70% compression ratio, which is better than nothing. no oss-fuzz or tests have been added yet (sorry). the replay gain feature should also be expanded in the future to support the new feature. sorry for the big PR. it's pretty readable tho! I tried to make it as modular & independent as possible. documentation is good. should this make it to a release version, an update to the standard RFC could also be considered. the changes are backwards-compatible and are as follows: 1. when storing float samples, the bps bits in the streaminfo metadata block should be 0b00000. 2. when storing float samples, the zero-padded bit, originally reserved, after the bit-depth bits of each frame header should be 1. 3. when storing float samples, the actual data samples stored in each subframe are obtained by doing some bit manipulation before encoding and after decoding. That part is in src/libFLAC/transform_float.c and is necessary to boost the compression (more info in the file comments). Monkey's Audio .ape does something similar, but this one seems to achieve better ratios (~8% improvement). sorry for the unorthodox type conversions. will fix it if I see it being considered for a merge.
c7dfe69 to
284eacc
Compare
|
Hey guys. Sorry for taking this long to respond. I resolved the issues noted in the review. Thx for the kind and thorough review! I also removed anything related to changing the frame header bits. As I've mentioned before, float is already outside of the streamable subset ('cause it's 32 bits). There was no need to change anything in frame headers. |
|
@aliheidary1381 Thanks for giving this a go ! CI reports conflicts, but should be easy |
|
Yes, resolving the conflict should be easy. My biggest concern here is our discussion about STREAMINFO_EXTENSION here. A summary of the discussions about it: Since we're already deviating from the standard, and older decoders won't be able to read the new float stream types anyway, we might as well introduce a STREAMINFO "extension", future-proofing other potential breaking changes that FLAC could benefit from. Possible future extensions discussed are "1-bit PWM streams", "immersive/3d moving sound objects", and "better channel masks and sample rates". The STREAMINFO structure must be kept the same, so that the "standard" streams generated by the new encoder versions would still be readable by older decoders. We're using forbidden values in the original STREAMINFO (like sample_rate=0 or bps=1-3) to suggest to the older decoders not to play it, and also to indicate that there's a STREAMINFO_EXTENSION block on the way. I'll try to write a proposal for the new header in the following week, and will TRY to implement it in the following month or so, if nobody puts an objection to the proposal. |
|
@ktmf01 (and @aliheidary1381 and anyone):
Is it a given that we can do without new residual coding? Especially in the latter situation, we could have several NaN showing up that might not (I don't know this though!) show up in audio test files. Just testing it on audio that has presumably originated as integer-PCM and then been through some float math in the manipulation might not pinpoint the issues that may show up when you get NaNs in. If a new residual coding method turns out to be necessary or warranted (like "variable-bit Rice" and possibly with extensions to say, pack at the end of the frame single sample corrections), that could solve something. Old players would just stop at unknown residual methods, so reserve a new method and forbid the two subframes that won't need residuals (VERBATIM and CONSTANT ... or assign two of the twenty-five remaining subframe types) |
|
The STREAMINFO_EXTENTION proposal: The presence of the STREAMINFO_EXTENTION metadata block MUST be indicated in the STREAMINFO block by setting the bps bits to 0b00000 (meaning 1bps, which was forbidden before). This also suggests to the older decoders to refuse decoding the new stream specifications. If present, the new decoders MUST replace every valid information from STREAMINFO_EXTENTION with anything inside the STREAMINFO block. The block, 128 bits in total, contains the following bits:
When necessary (i.e. when STREAMINFO_EXTENTION is present and it contains valid info), the sample rate in frame headers MUST be 0b0000, the bit depth bits in frame headers MUST be 0b000, and the channel mask bits MUST be 0b1111, to defer parameters to the STREAMINFO_EXTENSION block. |
|
I still insist that extending the frame header, or adding a new subframe type is outside the scope of this PR. Current discussions about the 16MiB block size limitation doesn't seem to benefit from an extended streaminfo. I think we can put that one outside this PR's scope too. If PDM sample format support would be added in the near future, we could add: If the sample format is PDM, the bit depth MUST be 0. If the sample format is PCM, the bit depth MUST NOT be 0. It is possible to extend this proposal for 3D objects using the reserved bits, even after a merge. I intentionally didn't mention it in the proposal. Based on my research, I believe that a new 3D object metadata standard just for FLAC won't be adopted by consumers (especially commercial software like DAWs). DTS:X tried this once with MDA. The most adopted open-source alternative is IAMF, which is a container that supports FLAC already. Nothing we'd need to do here. |
Yes this makes sense, as the new block enables float types, the decoder should have no problems reading this value. This also enables lossless transformation from CAF, which also uses 64-bit float, and near-lossless from AIFF, which uses 80-bit. But then we have to decide what to do when decoding to .wav, should it give an error, or we round/truncate to an integer value. A potential problem is for hardware decoder without float support. But manually decoding the float should be fine, since that is how everyone does it for AIFF's samplerate 80-bit extended float.
256 channels is nice. But how will that work in practice with regards to the channel mask or layout?
Yes this is a good idea since it is now commonly supported. This means we can also get rid of the WAVEFORMATEXTENSIBLE_CHANNEL_MASK vorbis comment.
I don't think 6 bits is enough for anything useful though. It is maybe better to define a whole new block type for advanced channel layouts in that case. Might be worth checking out CAF Channel Layout, Channel Description and related chunks:
Maybe allow for 16 and 128 in a future version, due to the existence of float16_t/float128_t in C/C++ and f16/f128 in Rust.
What if the STREAMINFO_EXTENTION contains mostly valid data for an older decoder, e.g. only a fractional samplerate of 44100.123 is set, and so the STREAMINFO block contains 44100.
Another idea is to not include STREAMINFO at all, but that might mean rejecting the file as a flac file entirely. |
The reason to do this at the same time as STREAMINFO_EXTENTION is this is the one chance for any backwards incompatible format changes. It is better to plan out all potential "flac 2.0" format changes, because I think it will be much harder to do changes to the frame header later. I agree about separating out the changes, not everything has to be crammed into the same PR. But then these changes should be protected behind some #ifdef until everything is ready. A separate branch is another way to do it. One idea for a changed format header would be to extend the channel decorrelation. Now it is limited to 2 channels only. For example, a simple left-right/front-back pairings could improve compression for surround audio. But this feels a bit unexplored though. Maybe it doesn't give so much compression benefits, or there is not enough 8+ channels source material to draw some conclusions from.
Yes I agree, I don't know if there's even a need for it, or interest. |
|
Thx for the wonderful feedback.
What do you think about using f128 (for sample rate) instead? That would make encoding AIFF streams lossless, and will still work on non-x86 architectures (which don't support f80). The downside is that almost no CPU architecture natively supports f128, and the C compiler would fall back to software emulation on most architectures.
You're right. I'll bring it down to just 2 bits to hint to the coder to look for an additional block with explicit channel layouts (like an array of labels/coordinates like CAF).
Agreed. Will bump it to 7 bits instead.
Seems like CAF's layout tags have some advantages over channel bitmaps like WAVEFORMATEXTENSIBLE. The most interesting one for me is differentiating use cases (e.g. standard stereo vs. headphone stereo). I'd keep the remaining 3 bits reserved for this use case.
I understand. Convinced me. I guess that'd mean that we'd have to mandate the "extended" frame header whenever the extended streaminfo is present (like an "if and only if" situation). Just to summarize: The following ideas have been presented here by now: |
f128 would be overkill. flac's 1 MHz samplerate range is a bit limited though, since it doesn't even reach up to the lowest DSD(64) samplerate of 2.8 MHz (64*44100). Interms of decoding complexity, a 32:32 fixed point format could also work. Then there's no worry about NAN, INF, or subnormals. |
|
Understood. |
|
The frame header is self-contained, and usually doesn't need the STREAMINFO. With the extension, we should also choose wether to let go of this advantage and defer the info to the STREAMINFO extension block (which breaks seeking on raw streams), or to add some frame header overhead (3-16 additional bytes on disk for each frame). |
More important than the streaming ability to play without file headers, I think is to avoid wrong playback. A "current integer-only flac player" must be able/forced to discard a float frame. Let's take the frame header in order:
|
5 participants
I've also added support to encode & decode raw, wave riff, & aiff float formats.
The float encoding feature achieves a near 70% compression ratio, which is better than nothing. No oss-fuzz or tests have been added yet (sorry). The replay gain feature should also be expanded in the future to support the new feature. Sorry for the big PR. It's pretty readable tho! I tried to make it as modular & independent as possible. Documentation is good. Should this make it to a release version, an update to the standard RFC could also be considered. The changes are backwards-compatible and are as follows: