Upload File via FTP SIM7080G

[Marco27293]

Hi,

I'm trying to upload 64.022KB data to a txt file using FTP protocol.
Following this doc (SIM7080_Series_FTP_Application_Note_V1.01.pdf) I used FTP Extend PUT Method.
All seems working fine because I got:

+APP PDP: 0,ACTIVE

OK

OK

OK

OK

OK

OK

OK

+FTPEXTPUT: 0,64022

OK

OK

+FTPPUT: 1,0

OK


But when I open the file uploaded to my FTP host it has 1360 bytes size, which is actually the maxium size that at+ftpput=1 command can transfer.

Could I upload all my data at once using extended FTP PUT method?
If yes, how ? (please give me a complete set of commands and a step by step guideline)

Regards

Marco

[benoitstjean]

Hi Marco,

This has nothing to do with PIC MCU but rather a question for SIMCOM. I use the standard PUT method.

FYI, been working long enough with SIMCOM products to say that their documentation is not accurate and very convoluted. They are getting better but sometimes it is very frustrating trying to understand it.

However, I use the SIM7600 modems and I use FTP and it works well but it is not that simple of a process. Many steps are involved and I cannot give you my code unfortunately for a few reasons.

Basically, my files are located on an SD card. The PIC MCU reads chunks of data from the file, passes-it to the modem over the UART which in turn passes-it to its FTP stack.

Here are the only commands I use (within the main code state machine):

AT+CFTPSSTART
AT+CFTPSSTOP
AT+CFTPSLOGOUT

sprintf( Command, "AT+CFTPSLOGIN=\"%s\",%s,\"%s\",\"%s\",%u", Address, Port, Username, Password, Type );
sprintf( Command, "AT+CFTPSPUT=\"%s\"", Filename );
sprintf( Command, "AT+CFTPSABORT" );


Each command also returns a whole slew of responses and you have to understand also how all the mechanism works.

Good luck.

Ben

[Ttelmah]

I think you are misunderstanding.
The FTPEXTPUT method, extends the output file. You still have to loop
outputting the blocks that the transfer actually sends, till the file is
complete.
Your extput call will return how many bytes it actually transfers. You
then have to call it again starting this many bytes further into the data,
and keep doing this till you reach the end of the data.

[benoitstjean]

@Ttelmah

Agree. That's why I indicated to the user that it's not as easy as it sounds to implement. There's lots of code involved and he will need timers also in there in case a response is not received etc etc. SIMCOM modems are really kick-a** when you manage to understand the poor documentation. They're really solid and reliable.

Cheers!

Ben

[Ttelmah]

Absolutely.
I just thought he was misunderstanding what the difference between the
standard and 'EXT' FTP calls did. I think he thinks it offers some form of
'extended' PUT, supporting magically sending an enormous block....

[Marco27293]

Thanks,

how can I loop to upload a whole file (size 64.022KB)?

Now, I do:

....

AT+FTPEXTPUT=1

OK

AT+FTPEXTPUT=2,0,64022,100000

+FTPEXTPUT: 0,64022

//send 64.022KB data to sim7080g via uart

OK

AT+FTPPUT=1

OK

+FTPPUT: 1,0
...

Which command I need to loop? There is any error?

Please, give me pseudo code in order to understand how I can implement a correct loop to upload my file.

Regards,

Marco

[benoitstjean]

Hi,

As stated in my previous email, I cannot really give you code because it is not that simple in my application, I probably have 75,000 lines of code if not more. I don't have just that code running therefore it's not as easy as a copy-paste.

I also just realized that the AT+FTPEXTPUT command does not apply to the SIM7600 (my modem). The information I can give you pertains to the SIM7600 but I'm sure it will be similar with yours.

You haven't specified if this modem is tied to a PIC microcontroller or not. Just a friendly reminder that this forum pertains to PIC micro users using the CCS C compiler :)

You have not specified where the file resides (SD card or the modem's EFS).

In my case, the file resides on an SD card and the PIC is tied to the modem via a UART.

The PIC initiates the FTP transfer by first enabling the FTP stack on the modem.

Then it sends the FTP login command. That's when the modem attempts the connection to the remote FTP server. Make sure that you can in fact access the FTP server from a known working computer to at least test the link. Be aware also that there are a multitude of responses that the modem will return if it fails to login, fails to reach the server etc. You must test all these scenarios before and during the transfer to account for all possible types of failures.

The PIC then needs to initiates the FTP transfer and in return, the modem will send '>' to the PIC. That's when you need to start passing the data from the file to the modem. The modem is in 'waiting' mode. All data that the modem receives from this point-on will be sent to the FTP server.

From here, the PIC needs to read blocks of data from the file on the SD card and passes-it to the modem.

When the entire file has been read and transferred, the PIC must send CTRL+Z (ASCII character \x1A) to tell the modem that the FTP transfer is complete. Then the modem will return in normal mode.

Note sure if this helps but that's the mechanism.

Good luck.

Ben

[Marco27293]

Hi Ben,

I use PIC18F47J53 (5.090 PCH ccs compiler) tied to SIM7080g via uart, file data are stored in microSD card.

I've already post a 1KB file to my FTP host with success.
BUT: I need to post 64KB file!
Using simple FTP PUT I was able to upload multiple 1KB packets but on one hand the process is too slow and on the other if I make my loop faster, I lose some packets during transmission...

How can I solve this problem?
I thought 4G LTE module would have been fast in upload large data size with a negligible power consumption...

Marco

[benoitstjean]

Hi Marco,

Well you're using a PIC18 which is not the fastest. I'm using a PIC24EP512 with a 29.4912 MHz external oscillator but internally on the MCU, it's overclocked at 129.024 MHz.

I don't use a loop to read and send the data, it's all timed based on external clocking interrupts from other devices. My circuit runs like a swiss watch in a sense that I have a few external clocks tied to interrupts and everything runs off these interrupts. I went this way because this was good enough for my situation but because of other components connected to the PIC that need to get serviced in real-time.

In my case, the data read from the SD card is transmitted from the PIC24 to modem at 115200 bps and it does in fact take a little while, around a minute and a half, for a 525KB file or something like that. These are the biggest files I have on my SD card. I know it sounds slow but I had to go with how my architecture is done. And while the FTP data is being transmitted, the MCU does other stuff in the background servicing interrupts and whatnot.

On my to-do list, I want to increase the UART and FTP transmission speeds but for now it works so I'll keep it that way. Next revision I guess.

What speed is you UART set to for the transmission of data between the PIC and modem?

And how slow is slow? You say it's too slow but how much time does it take to send a 64KB file?

The other alternative you can explore is to have the data stored on the modem's EFS. If it's there and you inhibit the FTP transfer, I'm not sure but maybe the modem will take care of handling the data by itself so maybe that will be super fast? I haven't explored that option, it's just a suggestion.

Ben

[Marco27293]

It takes about 1 minute.
The worst issue is: Why during transmission some data packets are lost ?

If I check sim7080g log all seems ok.

Then I open my file on FTP host and it has a wrong size (less than 64KB) and some data are absent.

If I extend the delay between two packet transmission the situation improve, but I do not understand why sim7080g module does NOT give me any error...

[benoitstjean]

I guess you should first start with a smaller buffer, like 260 bytes and in it, insert something that has a trend that can easily be spotted like 'abcdefghijklmnopqrstuvwxyz' and repeat this pattern 10 times. Test with that. The remote side should contain all the data.

What kind of data do you have in your file? Because if you are sending 0x00 - 0xFF, then if you are sending a valid data byte 0x1A which corresponds to CTRL-Z, then that will terminate the transmission. I had this exact problem since my files were raw data.

This is going a bit far back but if this is the problem you are encountering, the solution is that as the data blocks are read from the file, each byte has to be checked, byte by byte, and stored in a secondary buffer.

Then, everytime a \x1A is encountered as part of the data, it must be preceeded with \x03. So if you have the following raw string:

\x00\x01\xA8\x1A\x47, you must pass the following to the modem \x00\x01\xA8\x03\x1A\x47

So when the modem sees a string of data and \x03\x1A is encountered, it knows that it has to send the \x1A byte as data and ignore the \x03 byte.

If the data happens to be \x03\x1A like this: \x00\x01\xA8\x03\x1A\x47

then you must preceed it with \x03 as well like this: \x00\x01\xA8\x03\x03\x1A\x47

where the first \x03 is an actual \x03 and the second one is the one indicating that \x1A is a valid data byte.

Something along those lines. Like I said, its been a little while. Not sure if that's the problem you're encountering but I'm sharing useful information since I've been through this process already and it was a pain!

Ben

[Ttelmah]

Problem is you have to handle things being sometimes being different.

When you send:

AT+FTPEXTPUT=2,0,64022,100000

You may if you are lucky get back

+FTPEXTPUT: 0,64022

However on most networks you are much more likely to get something
like:

+FTPEXTPUT: 0,5644

The response number at the end of the line is the maximum number the
network can accept at this instant. It'll change quite frequently but will
not normally go below 1466.

You then need to transmit just the number allowed, and try again for
the next lot of data.

[benoitstjean]

[Marco27293]

Thanks Ben,

it is not my problem because at the moment I send packets composed only by 'M' char....I'm doing some tests using very simple textual files...

[benoitstjean]

Ok then perhaps TTelmah's response is the solution.