Data compression (help)

asmboy · Joined: 20 Nov 2007 Posts: 2128 Location: albany ny

ok so you are talking about cell phone users of the data?

who is going to write the APP that decodes your "compressed" data for the end user?

lucasromeiro · Joined: 27 Mar 2010 Posts: 167

Ttelmah · Joined: 11 Mar 2010 Posts: 20061

I think you need to go away, and do some very basic reading on data transmission.

Data is sent normally as 'lumps' referred to as bytes. 8 binary digits. Not always - some systems use 7, some use 9, and historically there were even coding's that used 5. The hardware does the job of sending this byte 'bit by bit', and adding the synchronisation bits.

Now you can use such a lump in a lot of ways.
You have a number stored in your computer. 120000 say. This 'requires' seventeen bits to store (2^17 = 131072). It could be sent over an 8 bit transmission medium therefore as 3 bytes (24 bits), or sent as text (ASCII) - 6 bytes, or sent as ASCII hex (6 bytes), or sent as binary (17 bytes). All equally easy to do, all transferring the same data, but all having different advantages/disadvantages.

The text form's only real advantage, is that it is human readable. Downside is that it only uses just over three bits out of every eight, to actually transfer useful data. Hex retains some degree of human readability and increases this to four bits our of eight (there is a slight saving for larger numbers 4000000000 for example uses just eight hex digits). However not a lot better. Binary would not normally be used for transmission over an 8bit link.

The point is though, that if you send the raw data stored in your computer, as bytes, this makes the best use of the transmission medium. Then there are ways of making even bigger improvements by thinking in computer terms rather than human. The clock is a classic example. The human form has separate hours, minutes, seconds, day of month, month of year, and year counters. Fourteen characters plus separators. A single 'seconds since midnight year 2000' counter, instead allows any time for over a century to be sent as just four bytes. Then if (for instance) a number only goes up to 9999, there are potentially 2 bits still 'left' out of a pair of bytes. If other things have to be sent like a status flag for what the instrument is doing, then this could potentially use these spare bits.

Now the downside of packing data really tightly comes when something goes wrong. A 'number' based text transmission, may still receive a legitimate number if a bit is damaged, but may well not. You have a high probability of being able to detect that something has gone wrong by receiving a non numeric result. With a form that sends everything packed tightly every possible value is used in the bytes, and you cannot detect an error. So the solution then is to add a CRC. This 'wastes' a byte (or more), but gives an increasing chance of detecting an error, and rejecting the packet.

The the very nature of your data may allow for less to be sent, So (for instance) if values generally ramp slowly, instead of sending the int32 'value' each time, you could send just an int8 'delta' . This limits how fast the data can change, but reduces the transmission space needed to just 1/4, for this particular value. Even better you could use a 'variable slope' form of this. However this all depends on the actual nature of your data.

Your original request for 'compression', shows significant ignorance. Except for making efficient use of the bytes, no compression is really possible on small amounts of data. Where compression starts to work, is where there is a huge amount of data, and some of it, by it's nature 'repeats'. It then becomes possible to to not send the repeated parts. I can't think of any compression algorithm that would actually reduce such a small amount of data. Most would actually make it larger.....