Python: Unpack arbitary length bits for database storage

Posted by sberry2A on Stack Overflow See other posts from Stack Overflow or by sberry2A
Published on 2010-06-14T14:19:26Z Indexed on 2010/06/14 14:22 UTC
Read the original article Hit count: 250

Filed under:
|
|
|

I have a binary data format consisting of 18,000+ packed int64s, ints, shorts, bytes and chars. The data is packed to minimize it's size, so they don't always use byte sized chunks. For example, a number whose min and max value are 31, 32 respectively might be stored with a single bit where the actual value is bitvalue + min, so 0 is 31 and 1 is 32. I am looking for the most efficient way to unpack all of these for subsequent processing and database storage.

Right now I am able to read any value by using either struct.unpack, or BitBuffer. I use struct.unpack for any data that starts on a bit where (bit-offset % 8 == 0 and data-length % 8 == 0) and I use BitBuffer for anything else.

I know the offset and size of every packed piece of data, so what is going to be the fasted way to completely unpack them?

Many thanks.

© Stack Overflow or respective owner

Related posts about python

Related posts about binary