CCS C Software and Maintenance Offers
FAQFAQ   FAQForum Help   FAQOfficial CCS Support   SearchSearch  RegisterRegister 

ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

CCS does not monitor this forum on a regular basis.

Please do not post bug reports on this forum. Send them to support@ccsinfo.com

Handling big data array in ROM

 
Post new topic   Reply to topic    CCS Forum Index -> General CCS C Discussion
View previous topic :: View next topic  
Author Message
adcor



Joined: 21 Feb 2008
Posts: 31

View user's profile Send private message

Handling big data array in ROM
PostPosted: Wed Dec 30, 2020 6:52 pm     Reply with quote

I am using a dspic33ch128mp505 with 128k flash to display on a ST7735 GLCD 128x160 resolution (compiler 5.097)
On full resolution the data takes 20.5k pixels x 16bit space.It is declared as ROM unsigned int16 array[] , it compiles fine but the uP is resetting in a loop. (some memory violation probably).
If I drop the resolution to 70x110 (~8k pixels), everything is ok.
I tried to place the data array use the #org 0x1000 after the IVT but no matter the size is the array I have compiler errors:

Fonts.c:163:6: Info#300 More info: Segment at 00000-0FFFE (0000 used)
Fonts.c:163:6: Info#300 More info: Segment at 10000-15EFE (0000 used)
Fonts.c:163:6: Info#300 More info: Attempted to create: 01000-00000 for #org
Fonts.c:163:6: Error#126 Invalid ORG range

Is any way I can fit this amount of data in the rom space?
thanks
Ttelmah



Joined: 11 Mar 2010
Posts: 19195

View user's profile Send private message

PostPosted: Thu Dec 31, 2020 2:38 am     Reply with quote

The way I have done a large data block, is to directly import it, and not
access it as a 'variable', but access it directly.

So:
Code:

#IMPORT(FILE=.\Logo.bin, RAW, LOCATION=where_stored, BPI=3)
//This physically imports the file, and sets 'where_stored' as the
//address where it is stored.

//Then I read the data from this with:
//routine to give access to ROM stored data - used for bitmap
union access {
    unsigned int32 whole;
    unsigned int8 bytes[4];
};

BYTE getval(unsigned int32 locn, unsigned int32 index)
{
    unsigned int32 calc;
    union access rdg;
    unsigned int8 bval;
    //Now I need to calculate the actual cell I need to read, fetch this
    //then extract the byte
    calc=(index/3)*2;
    //This gives the actual cell offset from locn.
    read_program_memory(locn+calc, &rdg, 4);
    //Gives the 32bit value containing the byte required.
    bval=index % 3;
    //Now gives the byte number required
    return rdg.bytes[bval];
    //return the physical byte
}


This is called with 'where_stored', and the index of the byte required.

This is efficient since it stored 3 bytes per instruction (BPI=3).

This merrily handles over a 100K data array for me.

There is a very large 'caveat' in what you were doing. It may be taking
much more ROM than you expect. A 16bit value stored into the ROM space
has to be 'word aligned'. So each 16bit value will be using a whole 3byte
instruction word. A chip with 128K of program memory only has about
44000 words of memory. Using over 20000 of these up on a single
array may well give problems.... Using byte alignment as I show, uses
only 66% of the space.
adcor



Joined: 21 Feb 2008
Posts: 31

View user's profile Send private message

PostPosted: Fri Jan 01, 2021 3:40 am     Reply with quote

Thanks for your solution, I'll try it.
So, index variable is total number of bytes from the raw file and basically in 8 bytes used in ROM I can retrieve 3 x int16 data.
I could also try to boost the ROM to 512k chip, or to split the big bitmap file into smaller ones to see if the compiler maps them differently.
Ttelmah



Joined: 11 Mar 2010
Posts: 19195

View user's profile Send private message

PostPosted: Fri Jan 01, 2021 3:56 am     Reply with quote

getval automatically handles the indexing. If you look it calculates the
physical ROM address by taking the 'offset' it is given and dividing this
by three and multiplying by 2 (to give a 'byte' based address. It then
extracts the needed byte from this address.

So if you call getval(where_stored,0) it returns the first stored byte
setval)where_stored,1), the second byte, getval(where_stored,2), the third
byte, and getval(where_stored,3) gets the third byte. Since the index and
address are int32, it can handle any number of bytes that can be stored.
temtronic



Joined: 01 Jul 2010
Posts: 9081
Location: Greensville,Ontario

View user's profile Send private message

PostPosted: Fri Jan 01, 2021 6:28 am     Reply with quote

Assuming the bitmaps are 'fonts for alphabets', you could split the array into 'uppercase' and 'lowercase' arrays. I did this decades ago for a 'kbd code converter' project in order to fit everything into a 16C84.
Whenever I see 'fonts and bitmaps... I just see a LOT of typing involved and even more misstyping..... Crying or Very sad
adcor



Joined: 21 Feb 2008
Posts: 31

View user's profile Send private message

PostPosted: Fri Jan 01, 2021 9:29 pm     Reply with quote

Thanks, it is a picture in my case 128w x 151h pixels.
I tried the #import way but it draws some garbage, sort of horizontal
bands.
Since I have an int16 for each pixel, it means I have 256 bytes per row, or 38656 bytes in the raw file (bmp file was stripped from its header).
Below is how I declared the function to map the pixels. Am I doing something wrong?

Code:
#IMPORT(FILE=.\test_16.raw, RAW, LOCATION=0x1000, BPI=3)
//This physically imports the file, and sets 'where_stored' as the
//address where it is stored.

//Then I read the data from this with:
//routine to give access to ROM stored data - used for bitmap
union access {
    unsigned int32 whole;
    unsigned int8 bytes[4];
};

BYTE getval(unsigned int32 locn, unsigned int32 index)  //max index = (128x151x2)=38656 bytes
{
    unsigned int32 calc;
    union access rdg;
    unsigned int8 bval;
    //Now I need to calculate the actual cell I need to read, fetch this
    //then extract the byte
    calc=(index/3)*2;
    //This gives the actual cell offset from locn.
    read_program_memory(locn+calc, &rdg, 4);
    //Gives the 32bit value containing the byte required.
    bval=index % 3;
    //Now gives the byte number required
    return rdg.bytes[bval];
    //return the physical byte
}

void display_import() {
    unsigned int8 hi,lo,buff,col,row;
    unsigned int16 color;
    unsigned int32 pos;
    for (pos=0;pos<38656;pos++) {
        buff=getval(0x1000,pos);
        if (pos%2 == 0) lo=buff;
        else {
              hi=buff;color=make16(hi,lo);
              row=pos/256;
              col=pos%256;
              drawPixel(col, row, color);}
    }
}
   
Ttelmah



Joined: 11 Mar 2010
Posts: 19195

View user's profile Send private message

PostPosted: Sun Jan 03, 2021 9:19 am     Reply with quote

Right, first question. Are you sure your 'raw' file format is right. This needs to be basic binary data containing the array contents LSBMSBLSBMSBetc..

I'd test first, by doing a basic read of a few elements and comparing with
what the int16 array was returning.

Also use where_stored as I show rather than trying to force an address.
This allows more flexibility on where the compiler places the data. Better
for memory usage.

Now honestly, I'd encapsulate a array_read function. So, something like:
Code:

#IMPORT(FILE=.\test_16.raw, RAW, LOCATION=where_stored, BPI=3)
//This physically imports the file, and sets 'where_stored' as the
//address where it is stored.

//Then I read the data from this with:
//routine to give access to ROM stored data - used for bitmap
union access {
    unsigned int32 whole;
    unsigned int8 bytes[4];
};

BYTE getval(unsigned int32 locn, unsigned int32 index)  //max index = (128x151x2)=38656 bytes
{
    unsigned int32 calc;
    union access rdg;
    unsigned int8 bval;
    //Now I need to calculate the actual cell I need to read, fetch this
    //then extract the byte
    calc=(index/3)*2;
    //This gives the actual cell offset from locn.
    read_program_memory(locn+calc, &rdg, 4);
    //Gives the 32bit value containing the byte required.
    bval=index % 3;
    //Now gives the byte number required
    return rdg.bytes[bval];
    //return the physical byte
}

#define ROW_SIZE 256

unsigned int16 read_from(int32 row, int col)
{
    //Using int32, for the row, so I can do the maths in this variable
    unsigned int16 rval;
    row*=ROW_SIZE;
    row+=col;
    //Now have index required/2 in 'row'
    row*=2;
    rval=make16(getval(where_stored,row+1), getval(where_stored,row));
    return rval;
}

void display_import() {
    unsigned int8 col,row;
    unsigned int16 color;
    for (row=0;row<151;row++)
    {
        for (col=0;col<128;col++)
        {
           color=read_from(row,col);
           drawPixel(col, row, color);
        }
    }
}
adcor



Joined: 21 Feb 2008
Posts: 31

View user's profile Send private message

PostPosted: Mon Jan 04, 2021 3:47 pm     Reply with quote

Thanks Mr TT, I made some progress with your math. For a full screen resolution I do have some sort of image showing somewhere in the middle and squeezed...
However I wanted to get some grasp on this issue and create a small 4 rows hex data F8 00 ( blue color). Import address is 0x1000, but it proves it does not matter where I set it, the lst result is the same, it starts at 015C54 for some reason. Could be a compiler issue?

Code:
015C54: F800 0000 00F8 00F8 F800 0000 00F8 00F8    ................
015C5C: F800 0000 00F8 00F8 F800 0000 00F8 00F8    ................
015C64: F800 0000 00F8 00F8 F800 0000 00F8 00F8    ................
015C6C: F800 0000 00F8 00F8 F800 0000 00F8 00F8    ................
015C74: F800 0000 00F8 00F8 F800 0000 00F8 00F8    ................
015C7C: F800 0000 00F8 00F8 F800 0000 00F8 00F8    ................
PCM programmer



Joined: 06 Sep 2003
Posts: 21708

View user's profile Send private message

PostPosted: Mon Jan 04, 2021 5:45 pm     Reply with quote

Why not try importing at 0x16000 ?
adcor



Joined: 21 Feb 2008
Posts: 31

View user's profile Send private message

PostPosted: Mon Jan 04, 2021 6:10 pm     Reply with quote

At 0x16000 I think there is some memory violation...the chip is resetting in a loop.
temtronic



Joined: 01 Jul 2010
Posts: 9081
Location: Greensville,Ontario

View user's profile Send private message

PostPosted: Mon Jan 04, 2021 6:37 pm     Reply with quote

Hmmm...maybe 'something' has protected banks of memory and 15c54 is the first available address ??
Try a very, very small program (org and store, main() loops forever) and see what happens ??
adcor



Joined: 21 Feb 2008
Posts: 31

View user's profile Send private message

PostPosted: Mon Jan 04, 2021 9:18 pm     Reply with quote

No luck, I compiled this program where "test1_16.raw" was either 1kB or 16 bytes size and the compiler puts the data in the #015xxx address area for some reason. However I changed the processor to 33CH512MP505 version and the ROM address moved to # 057EF4.

Code:
#include <33CH128MP505.h>
#device ADC=12
#IMPORT(FILE=.\test1_16.raw,RAW,LOCATION=0x1000,BPI=3)

#use delay(internal=180000000)

#FUSES NOWDT                    //No Watch Dog Timer
#FUSES CKSFSM                   //Clock Switching is enabled, fail Safe clock monitor is enabled
#FUSES S1_CKSFSM                //Slave core Clock Switching is enabled, fail Safe clock monitor is enabled

union access {
    unsigned int32 whole;
    unsigned int8 bytes[4];
};

BYTE getval(unsigned int32 locn,unsigned int32 index) 
{
    unsigned int32 calc;
    union access rdg;
    unsigned int8 bval;
    //Now I need to calculate the actual cell I need to read, fetch this
    //then extract the byte
    calc=(index/3)*2;
    //This gives the actual cell offset from locn.
    read_program_memory(locn+calc,&rdg,4);
    //Gives the 32bit value containing the byte required.
    bval=index%3;
    //Now gives the byte number required
    return rdg.bytes[bval];
    //return the physical byte
}

#define ROW_SIZE 256

unsigned int16 read_from(unsigned int32 row,unsigned int8 col)
{
    //Using int32, for the row, so I can do the maths in this variable
    unsigned int16 rval;
    row*=ROW_SIZE;
    row+=col;
    //Now have index required/2 in 'row'
    row*=2;
    rval=make16(getval(0x1000,row+1), getval(0x1000,row));
    return rval;
}

void test_import() {
    unsigned int8 col,row;
    unsigned int16 color;
    for (row=0;row<4;row++)
    {
        for (col=0;col<128;col++)
        {
           color=read_from(row,col);
           }
    }
}   
void main()
{
   setup_timer1(TMR_INTERNAL | TMR_DIV_BY_256, 0);
   setup_adc_ports(sAN0);
   setup_adc(ADC_CLOCK_SYSTEM | ADC_CLOCK_DIV_1 | ADC_WARMTIME_16 | ADC_SHARED_CLOCK_DIV_2 | ADC_SHARED_TAD_MUL_2);

    while (1) {
        test_import();
    }
}

Ttelmah



Joined: 11 Mar 2010
Posts: 19195

View user's profile Send private message

PostPosted: Tue Jan 05, 2021 3:18 am     Reply with quote

I don't see anywhere where #import says you can use a location as you
are trying to do.
From the manual:
Quote:

LOCATION=id - The identifier is made a constant with the start address of the imported data.

No mention of putting the data at a location with this....

You could put it at a specified location, by having the data at 0x1000 in
the file, and using RANGE for the import, or probably by using a #ORG
for the import. By default the compiler places imported data at the top
of the available memory, to leave as much uninterrupted space as possible.
This is what you are seeing happen.

Use #import the way I do it.
adcor



Joined: 21 Feb 2008
Posts: 31

View user's profile Send private message

PostPosted: Fri Jan 08, 2021 4:27 pm     Reply with quote

Thanks MrTT, the only way I could make it work is to look in the lst file to see where the compiler put the ROM data and use that address as starting range as in "locn".
There is no correlation between LOCATION=id or RANGE=start:end in import directive and locn from getval(unsigned int32 locn,unsigned int32 index), the compiler is lost in translation.
adcor



Joined: 21 Feb 2008
Posts: 31

View user's profile Send private message

SOLVED -Handling big data array in ROM
PostPosted: Fri Jan 08, 2021 5:46 pm     Reply with quote

I got it working by not using LOCATION=0x4000 as a physical address interpretation rather as a label id e.g. LOCATION=ph1 and then use this label into the function retrieving the data from the ROM. Maybe the compiler was confused somehow.
Thanks all for support.
Display posts from previous:   
Post new topic   Reply to topic    CCS Forum Index -> General CCS C Discussion All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group