Data logging and retrieval

[Guest]

Maybe off the topic, but I was wondering what people do when retrieving logged data when there is no clear "termination" of the data. For example, say I am logging ADC data to an external EEPROM. After 1, 10, ...., 10,000 captures of the data, I read the data back and show it on a display, send it out over RS232 for plotting on a PC, or something like that. Now, if the logging system is shut down in a pre-determined manner (like flipping a switch), the EEPROM can be stamped with something that denotes the end of the capture. Then when uploading the logged data later, one can check for the pre-determined end-of-capture marker.

But what do you do when the data logger is unceremoniously shut down due to power failure, or an unforseen disaster? The data logging stops and now when I read back the data, I don't know where "real" data stops and whatever random data is in EEPROM begins. I thought I could populate the entire EEPROM with zeros or ones, but this is tedious and in some cases the real data could be zeros for an extended time. The problem sounds easy, but lets say there are 10's of thousands of points and you want to know where they "really" end. Download to a PC, plot it, and try to find a discontinuity in the data? There must be a smarter way than this, especially when there are tons of data.

Any ideas on this? Do you add something like a special (checksum?) bit to each data before logging it? Then during readback check for these bits? This eats up space, but nothing is free. I would like to know what people think about this in general. Thanks!

Bill

[Neutone]

Log your data in blocks that the EEPROM is based upon (write time is greatly reduced with page writes). Put time stamps in each block as well as a CRC. When readings data into a PC read all the blocks and use logic in the PC to display data in a logical manner. This also extends the life of the EEPROM if your always writing the the block following the last one written.

[Kasper]

how about storing the address you have last written to in internal eeprom, then use that to determine how much data you should clock out of it