Decompressing zip stream

[Leonardo M. Ramé]

Hi, I posted this same question in fpc-devel list a week ago, with no
answer. I hope somebody can help me here.

The attached file, is a WAV audio compressed with TurboPower Abbrevia
using Delphi 7, then stored in a blob field in a database.

I'm trying to migrate an app to Lazarus that must read and write those files
using the same format as the Delphi version. I cannot use Abbrevia here,
because of some incompatibilities with Linux 64bits (the new app must
run in 32/64bits linux/win/mac).

While trying to decompress the streams using TInflater class from zipper unit
using this code, I get "buffer error" message when calling DeCompress
method. I must add that the error only happens with a group of streams,
specially with the small ones:

var
lUnzip: TInflater;
lInStream: TMemoryStream;
lOutStream: TMemoryStream;
begin
lInStream := TMemoryStream.Create;
lOutStream := TMemoryStream.Create;

lInStream.LoadFromFile('inputfile.file');
lInStream.Position:= 0;
lUnzip := TInflater.Create(lInStream, lOutStream, lInStream.Size);
try
lUnzip.DeCompress;
lOutStream.Position:=0;
lOutStream.SaveToFile('output.wav');
finally
lInStream.Free;
lOutStream.Free;
lUnzip.Free;
end;
end;

I'm using fpc 2.7.1 from today and Lazarus 0.9.31 Rev 32139 from SVN on
Linux 64bits.

Any hint?

[Ludo Brands]

Post by Leonardo M. Ramé
While trying to decompress the streams using TInflater class
from zipper unit using this code, I get "buffer error"
message when calling DeCompress method. I must add that the
error only happens with a group of streams, specially with

Tdecompressionstream decompresses in chunks using an outputbuffer with the
size you specify in TInflater.Create. When this size is too small to
decompress a chunk you get an "buffer error". I suggest you try with a
bigger buffersize for small files, instead of lInStream.Size. Wav files have
a high compression ratio so a small input gives a large output.

Ludo


--

[Leonardo M. Ramé]

Post by Ludo Brands

Post by Leonardo M. Ramé
While trying to decompress the streams using TInflater class
from zipper unit using this code, I get "buffer error"
message when calling DeCompress method. I must add that the
error only happens with a group of streams, specially with

Tdecompressionstream decompresses in chunks using an outputbuffer with the
size you specify in TInflater.Create. When this size is too small to
decompress a chunk you get an "buffer error". I suggest you try with a
bigger buffersize for small files, instead of lInStream.Size. Wav files have
a high compression ratio so a small input gives a large output.
Ludo

Thanks Ludo for your answer.

The file size is 68127 bytes, I tried with buffer size of 2048 bytes and 100000 bytes.
In both cases I got the 'buffer error' message.

[Ludo Brands]

Post by Leonardo M. Ramé
Thanks Ludo for your answer.
The file size is 68127 bytes, I tried with buffer size of
2048 bytes and 100000 bytes.
In both cases I got the 'buffer error' message.

Seems something is wrong with the data. Tried to decode with
TZDecompressionStream in ZLibEx for Delphi and I'm getting a data error.
I also traced into paszlib/src/zinflate.pas and the 'buffer error' is
returned quite early with apparently plenty of output buffer available. That
also points into the direction of corrupted data.
Can you attach a (small) file that decompresses succesfully? Just to
validate my test with TZDecompressionStream.
What Abbrevia unit was used to compress the files?

Ludo


--

[Leonardo M. Ramé]

Post by Ludo Brands

Post by Leonardo M. Ramé
Thanks Ludo for your answer.
The file size is 68127 bytes, I tried with buffer size of
2048 bytes and 100000 bytes.
In both cases I got the 'buffer error' message.

Seems something is wrong with the data. Tried to decode with
TZDecompressionStream in ZLibEx for Delphi and I'm getting a data error.
I also traced into paszlib/src/zinflate.pas and the 'buffer error' is
returned quite early with apparently plenty of output buffer available. That
also points into the direction of corrupted data.
Can you attach a (small) file that decompresses succesfully? Just to
validate my test with TZDecompressionStream.
What Abbrevia unit was used to compress the files?
Ludo

Ludo, I attached a .tar.gz file containing one file that can be decompressed successfully, and
its output as .wav.

To compress AbZipPrc unit was used, and AbUnzPrc to decompress.

[Ludo Brands]

Post by Leonardo M. Ramé
Ludo, I attached a .tar.gz file containing one file that can
be decompressed successfully, and its output as .wav.
To compress AbZipPrc unit was used, and AbUnzPrc to decompress.
--

The delphi test with TZDecompressionStream failed also with ok.file. So that
test isn't valid.
Recompiled the packages with debug info and found that error.file is too
short. The inflate "engine" hasn't encountered the end of stream and wants
more input.
Perhaps you need to take a look at the database code that reads the blobs.

Ludo


--

[Ludo Brands]

Post by Ludo Brands
The delphi test with TZDecompressionStream failed also with
ok.file. So that test isn't valid.
Recompiled the packages with debug info and found that
error.file is too short. The inflate "engine" hasn't
encountered the end of stream and wants more input.
Perhaps you need to take a look at the database code that
reads the blobs.

Is it possible to write the data that AbUnzPrc gets to a file and compare
this to your error.file. Then you are sure where the problem is. Or feed
error.file to AbUnzPrc.

Ludo


--

[Ludo Brands]

Post by Ludo Brands
Is it possible to write the data that AbUnzPrc gets to a file
and compare this to your error.file. Then you are sure where
the problem is. Or feed error.file to AbUnzPrc.

Tried error.file with AbUnzPrc and am getting a 'no more compressed data in
stream [TAbDfInBitStream.DiscardBits]' error. Ok.file decompresses fine.

Something wrong with the database part.

Ludo


--

[Leonardo M. Ramé]

Post by Ludo Brands

Post by Ludo Brands
Is it possible to write the data that AbUnzPrc gets to a file
and compare this to your error.file. Then you are sure where
the problem is. Or feed error.file to AbUnzPrc.

Tried error.file with AbUnzPrc and am getting a 'no more compressed data in
stream [TAbDfInBitStream.DiscardBits]' error. Ok.file decompresses fine.
Something wrong with the database part.
Ludo

Thanks! Ludo. You are right, the problem was the way I was reading the
blob field from the database.

Thank you very much!.

[Hans-Peter Diettrich]

Post by Leonardo M. Ramé
To compress AbZipPrc unit was used, and AbUnzPrc to decompress.

Just a note on Abbrevia: I've been working on that project some years
ago, and found it very buggy. During that work I even found an bug in
the C sources, from which the Abbrevia units are derived. This bug had
been lurking in the C sources for many years, and was fixed only a few
days before I reported it to the C developers. Since that time I don't
like manual ports from C to Pascal, because their maintenance requires
too much work :-(

Even if compressed files decompress properly with Abbrevia, the files
may be invalid. When decompression with other tools (7zip...) works,
then the Lazarus units may be buggy, otherwise Abbrevia itself is the
trouble maker.

DoDi


--

[Hans-Peter Diettrich]

Post by Leonardo M. Ramé
While trying to decompress the streams using TInflater class from zipper unit
using this code, I get "buffer error" message when calling DeCompress
method. I must add that the error only happens with a group of streams,
var
lUnzip: TInflater;
lInStream: TMemoryStream;
lOutStream: TMemoryStream;

I'd use TFileStream here, at least for the output stream. This should
reduce buffer size problems.

DoDi


--

[Leonardo M. Ramé]

Post by Leonardo M. Ramé
While trying to decompress the streams using TInflater class from zipper unit
using this code, I get "buffer error" message when calling DeCompress
method. I must add that the error only happens with a group of streams,
var
lUnzip: TInflater;
lInStream: TMemoryStream;
lOutStream: TMemoryStream;

I'd use TFileStream here, at least for the output stream. This should reduce
buffer size problems.
DoDi

I did a quick test using TFileStream, and indeed, it is a little better
than TMemoryStream, but it also raises the "buffer error" message.

Using TFileStream, at least I can see that *part of* the file was decompressed.