Q: vbz_decompress_signal return "Input data not compressed by zstd"

[Alteroldis]

Dear developers,

Sorry for the stupid questions, but my student and I are thinking about ways to compress a signal for long-term storage, but we immediately ran into a problem: don't understand whether the signal is compressed..

viewing signal_table.schema

reader = pod5.Reader(pod5file)
reader.signal_table.schema
signal: large_binary
  -- field metadata --
  ARROW:extension:metadata: ''
  ARROW:extension:name: 'minknow.vbz'

So, signal is VBZ compressed, is it right?
And if I execute pod5 inspect summary, I get File uses VBZ compression.

But if I try to decompress signal, I raised error:

decompress signal with vbz_decompress_signal

with pod5.DatasetReader(pod5file) as dataset:
     for read_record in dataset: break
read_record.signal
array([313, 391, 402, ..., 578, 605, 637], shape=(423702,), dtype=int16)
dsig = pod5.vbz_decompress_signal( read_record.signal, read_record.sample_count )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/Tools/pod5-v0.3.24/lib/python3.11/site-packages/pod5/signal_tools.py", line 35, in vbz_decompress_signal
    p5b.decompress_signal(compressed_signal, signal)
RuntimeError: Invalid: Input data not compressed by zstd: (18446744073709551614 Unspecified error code)

Signal values are in the range -4096 to 4095.
I upload my pod5 file to Google Drive.

[cjw85]

When you do:

reader = pod5.Reader(pod5file)
reader.signal_table.schema

and see:

signal: large_binary
  ARROW:extension:name: 'minknow.vbz'

that tells you that, under the hood, the Arrow column “signal” is stored using VBZ (Zstandard-based) compression — you can also confirm this with pod5 inspect summary returning “File uses VBZ compression.”

However, by the time you call:

with pod5.DatasetReader(pod5file) as dataset:
    for read in dataset:
        break
read.signal  # → numpy array of int16

that array is already the decompressed waveform (i.e. raw ADC counts in –4096…+4095). The convenience reader automatically applies VBZ decompression for you; what you see in read.signal is the uncompressed data.

The helper function pod5.vbz_decompress_signal() is intended for use on the compressed byte stream, not on the already‐expanded int16 buffer that DatasetReader returns. Passing an uncompressed int16 array into the VBZ decompressor will therefore fail with “Input data not compressed by zstd.”

In summary:

• Yes, POD5 uses VBZ compression on the stored signal column  .
• The Python API’s DatasetReader automatically decompresses it for you; read.signal is already raw samples.