SPIFFS: From image creation to read & write through flashing

Bernard
Posts: 7
Joined: Tue Apr 17, 2018 10:49 pm
Location: France

SPIFFS: From image creation to read & write through flashing

Postby Bernard » Wed Jul 11, 2018 5:22 pm

As I fight a little bit to be able to use SPIFFS on RTOS SDK and did not find too many information on the web, here are my findings:

Compile mkspiffs (tool used to generate a SPIFFS image)
Use mkspiffs tool: https://github.com/igrr/mkspiffs
git clone --recurse-submodules git@github.com:igrr/mkspiffs.git to clone (with sub module)

Note: the sub module here is spiffs source. Unfortunately, these spiffs sources do not  match the ones from RTOS SDK. That's why you have to copy the following files:
/path/to/ESP8266_RTOS_SDK/third_party/spiffs/*.c (but not esp_spiffs.c) to /path/to/mkspiffs/spiffs/src/
/path/to/ESP8266_RTOS_SDK/include/spiffs/spiffs.h to /path/to/mkspiffs/spiffs/src/
/path/to/ESP8266_RTOS_SDK/include/spiffs/spiffs_nucleus.h to /path/to/mkspiffs/spiffs/src/

And you have to update the file /path/to/mkspiffs/include/spiffs_config.h with the values from /path/to/ESP8266_RTOS_SDK/include/spiffs/spiffs_config.h
You should keep anyway the value of SPIFFS_TEST_VISUALISATION to 1 in order to be able to compile.

Then compile with make dist
Normally you should not get any errors, but if you get some, you will have to fix them.

Create SPIFFS image
Now put all you files you want in your image in a folder called data/
Note: There should be no subfolder as SPIFFS only works with a flat hierarchy. Max lenght for a file name is 31 characters. When you want to open a file from esp8266, you should add a / before file name (see code)

Generate the image by typing:
/path/to/mkspiffs/mkspiffs –p 256 –b 8192 –s 0xfa000 –c /path/to/data/ spiffs-image.bin
Where:
-p 256 indicates the size of the page (should match what you will put in esp8266 code)
-b 8192 indicates the size of the block (should match what you will put in esp8266 code)
-s 0xfa000 ndicates the size of the image (should match what you will put in esp8266 code). Here our image will go from 0x300000 to 0x3fa000 as from 0x3fc000 we have other images (system param)

Flash SPIFFS image
esptool.py --port /dev/ttyUSB0 --baud 460800 write_flash 0x300000 spiffs-image.bin
Image is flashed at adress 0x300000

Read SPIFFS image from ESP8266
Here is a code:

Code: Select all

/*
* user_spiffs.c
*
*/
 
#include "esp_common.h"
#include "spiffs.h"

#include <fcntl.h>
 
#define FS1_FLASH_SIZE      (1000*1024)                // Size of SPIFFS image 0xfa000
#define FS1_FLASH_ADDR      (3*1024*1024)            // Start of SPIFFS image in FLASH memory space 0x300000
#define SECTOR_SIZE         (8*1024)
#define LOG_BLOCK           (SECTOR_SIZE)            // Size of a logical block
#define LOG_PAGE            (256)                    // Size of a logical page
#define FD_BUF_SIZE         32*4
#define CACHE_BUF_SIZE      (LOG_PAGE + 32)*8
 
void spiffs_fs1_init(void)
{
    s32_t ret;
    struct esp_spiffs_config config;
 
    config.phys_size = FS1_FLASH_SIZE;
    config.phys_addr = FS1_FLASH_ADDR;
    config.phys_erase_block = SECTOR_SIZE;
    config.log_block_size = LOG_BLOCK;
    config.log_page_size = LOG_PAGE;
    config.fd_buf_size = FD_BUF_SIZE * 2;
    config.cache_buf_size = CACHE_BUF_SIZE;
 
    ret = esp_spiffs_init(&config);
    if (ret)
        os_printf("ERR: Can't init spiffs image!\n");
}

void spiffs_main()
{
    char out[20] = { 0 };
    int pfd;
    int res;

    // Mount SPIFFS image
    spiffs_fs1_init();
 
    // Create a new file
    char *buf = "hello world";

    pfd = open("myfile", O_TRUNC | O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
    if (pfd <= 3) {
        printf("open file error\n");
    }
     
    // Write to new file
    int write_byte = write(pfd, buf, strlen(buf));
    if (write_byte <= 0) {
        printf("write file error\n");
    }
    close(pfd);
 
    // Read previous created file
    pfd = open("myfile", O_RDWR);
    if (pfd < 3)
        os_printf("ERR: Can't open file! %d\n", pfd);
    else {
        if (read(pfd, out, 20) < 0)
            os_printf("ERR: Can't read file!\n");
        close(pfd);
        printf("--> %s <--\n", out);
    }
 
    // Read file from generated image with mkspiffs tool
    pfd = open("/readme.txt", O_RDWR);            // !! Don't forget to add the / before the name of the file
    if (pfd < 3)
        os_printf("ERR: Can't open file! %d\n", pfd);
    else {
        if (read(pfd, out, 20) < 0)
            os_printf("ERR: Can't read file!\n");
        close(pfd);
        printf("--> %s <--\n", out);
    }
}

Note on page, block, address and size parameters
These parameters depend heavily on the memory you are using.
In my case I am using a 4MB flash (Winbond W25Q32FV).
Page size and block size (256 and 8192) are fine for this memory but other values may work also.
For a 4MB flash, address range is from 0x000000 to 0x400000. I decided to put my SPIFFS image at address 0x300000, but you can put it somewhere else if you prefer (0x100000, 0x200000).
I decided to used a size of 1MB (0x100000) for SPIFFS image but as I am flashing sytem parameter (esp_init_data_default.bin) and blank.bin at resp. 0x3fc000 and 0x3fe000 I need to reduce the size to 0xfa000 (size has to be a multiple of block size)


Hope it will be helpful for people.
Last edited by Bernard on Wed Apr 24, 2019 6:17 pm, edited 2 times in total.
Regards, Bernard

Website: https://www.ochrin.com

razuel

Re: SPIFFS: From image creation to read & write through flashing

Postby razuel » Fri Mar 15, 2019 7:54 pm

thanks for your sample

i try build using RTOSSDK IDFstyle , and using esp826612E 4MB Flash (1MB SPIFFS)

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 7196, room 16
0x40100000: _stext at ??:?

tail 12
chksum 0x51
ho 0 tail 12 room 4
load 0x3ffe8408, len 24, room 12
tail 12
chksum 0xd8
ho 0 tail 12 room 4
load 0x3ffe8420, len 3548, room 12
tail 0
chksum 0xa2
I (51) boot: ESP-IDF v3.2-dev-137-g93e3a3f5 2nd stage bootloader
I (51) boot: compile time 08:31:07
I (51) qio_mode: Enabling default flash chip QIO
I (59) boot: SPI Speed : 40MHz
I (66) boot: SPI Mode : QIO
I (72) boot: SPI Flash Size : 2MB
I (78) boot: Partition Table:
I (84) boot: ## Label Usage Type ST Offset Length
I (95) boot: 0 nvs WiFi data 01 02 00009000 00006000
I (106) boot: 1 phy_init RF data 01 01 0000f000 00001000
I (118) boot: 2 factory factory app 00 00 00010000 000f0000
I (130) boot: End of partition table
I (136) esp_image: segment 0: paddr=0x00010010 vaddr=0x40210010 size=0x35ae0 (219872) map
I (223) esp_image: segment 1: paddr=0x00045af8 vaddr=0x3ffe8000 size=0x00548 ( 1352) load
I (224) esp_image: segment 2: paddr=0x00046048 vaddr=0x3ffe8548 size=0x0011c ( 284) load
I (235) esp_image: segment 3: paddr=0x0004616c vaddr=0x40100000 size=0x06004 ( 24580) load
0x40100000: _stext at ??:?

I (256) boot: Loaded app from partition at offset 0x10000
I (277) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (287) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (487) reset_reason: RTC reset 2 wakeup 0 store 0, reason is 2
open file error
write file error
ERR: Can't open file! 0
ERR: Can't open file! 0


and this is my code



/*
* user_spiffs.c
*
*/
#include "esp_spiffs.h"
#include "spiffs.h"

#include <fcntl.h>

#define FS1_FLASH_SIZE (1000*1024) // Size of SPIFFS image 0xfa000
#define FS1_FLASH_ADDR (3*1024*1024) // Start of SPIFFS image in FLASH memory space 0x300000
#define SECTOR_SIZE (8*1024)
#define LOG_BLOCK (SECTOR_SIZE) // Size of a logical block
#define LOG_PAGE (256) // Size of a logical page
#define FD_BUF_SIZE 32*4
#define CACHE_BUF_SIZE (LOG_PAGE + 32)*8

void spiffs_fs1_init(void)
{
s32_t ret;
struct esp_spiffs_config config;

config.phys_size = FS1_FLASH_SIZE;
config.phys_addr = FS1_FLASH_ADDR;
config.phys_erase_block = SECTOR_SIZE;
config.log_block_size = LOG_BLOCK;
config.log_page_size = LOG_PAGE;
config.fd_buf_size = FD_BUF_SIZE * 2;
config.cache_buf_size = CACHE_BUF_SIZE;

ret = esp_spiffs_init(&config);
if (ret)
os_printf("ERR: Can't init spiffs image!\n");
}

void spiffs_main()
{
char out[20] = { 0 };
int pfd;
//int res;

// Mount SPIFFS image
spiffs_fs1_init();

// Create a new file
char *buf = "hello world";

pfd = open("myfile", O_TRUNC | O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
if (pfd <= 3) {
printf("open file error\n");
}

// Write to new file
int write_byte = write(pfd, buf, strlen(buf));
if (write_byte <= 0) {
printf("write file error\n");
}
close(pfd);

// Read previous created file
pfd = open("myfile", O_RDWR);
if (pfd < 3)
os_printf("ERR: Can't open file! %d\n", pfd);
else {
if (read(pfd, out, 20) < 0)
os_printf("ERR: Can't read file!\n");
close(pfd);
printf("--> %s <--\n", out);
}

// Read file from generated image with mkspiffs tool
pfd = open("/readme.txt", O_RDWR); // !! Don't forget to add the / before the name of the file
if (pfd < 3)
os_printf("ERR: Can't open file! %d\n", pfd);
else {
if (read(pfd, out, 20) < 0)
os_printf("ERR: Can't read file!\n");
close(pfd);
printf("--> %s <--\n", out);
}
}

void app_main(){
spiffs_main();
}

Bernard
Posts: 7
Joined: Tue Apr 17, 2018 10:49 pm
Location: France

Re: SPIFFS: From image creation to read & write through flashing

Postby Bernard » Mon Mar 18, 2019 12:50 am

Unfortunately I did test only with version 2.x.x of esp8266_rtos_sdk. I did not try with version 3.x.
So I can't tell you what is wrong.
If I have time, I will give it a try and let you know my result.
Regards, Bernard

Website: https://www.ochrin.com

Bernard
Posts: 7
Joined: Tue Apr 17, 2018 10:49 pm
Location: France

Re: SPIFFS: From image creation to read & write through flashing

Postby Bernard » Wed Apr 24, 2019 6:13 pm

I retried my tutorial and found out that there was a mistake: the spiffs_config.h file which should be modified is not the one in default folder but the one in /path/to/mkspiffs/include.
And even more it is better to update value in the file than do a direct copy of this file from RTOS_SDK.
You should also keep SPIFFS_TEST_VISUALISATION to 1 in order to be able to compile.
I have updated my first post.

Anyway, I don't think this fix will solve your problem.
Regards, Bernard

Website: https://www.ochrin.com

ELIOTRADISA
Posts: 7
Joined: Mon Apr 20, 2020 2:44 am

Re: SPIFFS From image creation to read write through flashing

Postby ELIOTRADISA » Wed Jul 15, 2020 6:59 am

Mariewet wrote:Use the IrTool program to decode Pronto hex, but in this case I did it for you.

They are both Panasonic protocol, device 160, subdevice 0.
The first was command 63 EFC=026.
The second was command 62 EFC=022.

/path/to/ESP8266_RTOS_SDK/third_party/spiffs/*.c (but not esp_spiffs.c) to /path/to/mkspiffs/spiffs/src/

rajubhaiya
Posts: 1
Joined: Thu Sep 17, 2020 4:09 am

Re: SPIFFS: From image creation to read & write through flashing

Postby rajubhaiya » Thu Sep 17, 2020 4:23 am


roundcubenine
Posts: 1
Joined: Sun Sep 27, 2020 2:59 pm

Re: SPIFFS: From image creation to read & write through flashing

Postby roundcubenine » Sun Sep 27, 2020 4:13 pm

To create a flashable NAND root file system image, follow these steps on your host PC. In these steps, we use phytec-tisdk-image-am5728-phycore-rdk.tar.xz as an example source root file system.

Nielson
Posts: 6
Joined: Wed Dec 09, 2020 4:15 pm

Re: SPIFFS: From image creation to read & write through flashing

Postby Nielson » Wed Dec 09, 2020 4:19 pm

SPIFFS let's you access the flash memory as if it was a normal file system like the one on your computer (but much simpler of course): you can read and write files, create folders ...https://gbapps.net/gbwhatsapp-apk/

The easiest way to learn how to use SPIFFS is to look at some examples. But a file server with no files to serve is pretty pointless,

dote81
Posts: 1
Joined: Mon Feb 08, 2021 6:44 pm

Re: SPIFFS: From image creation to read & write through flashing

Postby dote81 » Mon Feb 08, 2021 6:58 pm

Did you know each ESP8266 allows you to partition the system flash such that you can use it to store both code and support a file system?
This filing system can be used to store infrequently changing data such as; web pages, configurations, sensor calibration data etc.
This short Instructable details how to use SPIFFs, configure your Arduino IDE to upload files direct to your ESP8266 and figure out how much flash your device has.

Source: best microwave toaster oven combo 2021

julie128
Posts: 1
Joined: Wed Mar 24, 2021 12:15 pm
Contact:

Re: SPIFFS: From image creation to read & write through flashing

Postby julie128 » Wed Mar 24, 2021 12:24 pm

Up until now, we've always included the HTML for our web pages as string literals in our sketch. This makes our code very hard to read, and you'll run out of memory rather quickly.
If you remember the introduction, I mentioned the Serial Peripheral Interface Flash File System, or SPIFFS for short. It's a light-weight file system for microcontrollers with an SPI flash chip. The on-board flash chip of the ESP8266 has plenty of space for your webpages, especially if you have the 1MB, 2MB or 4MB version.

SPIFFS let's you access the flash memory as if it was a normal file system like the one on your computer (but much simpler of course): you can read and write files, create folders ...

The easiest way to learn how to use SPIFFS is to look at some examples. But a file server with no files to serve is pretty pointless, so I'll explain how to upload files to the SPIFFS first.
Uploading files to SPIFFS
To select the right files to upload, you have to place them in a folder called data, inside the sketch folder of your project: Open your sketch in the Arduino IDE, and hit CTRL+K. Wait for a file explorer window to open, and create a new folder named data. Copy your files over to this folder. (Only use small files like text files or icons. There's not enough space for large photos or videos.)
Next, select all files in the folder (CTRL+A) and check the size of all files combined (don't forget subfolders). Go to the Arduino IDE again, and under Tools > Flash Size, select an option with the right flash size for your board, and a SPIFFS size that is larger than the size of your data folder.
Then upload the sketch. When that's finished, make sure that the Serial Monitor is closed, then open the Tools menu, and click ESP8266 sketch data upload. If your ESP has auto-reset and auto-program, it should work automatically, if you don't have auto-program, you have to manually enter program mode before uploading the data to SPIFFS. The procedure is exactly the same as entering program mode before uploading a sketch.

If you get an error saying SPIFFS_write error(-10001): File system is full , this means that your files are too large to fit into the SPIFFS memory. Select a larger SPIFFS size under Tools > Flash Size, or delete some files.
Even if your computer says that the files are smaller than the selected SPIFFS size, you can still get this error: this has to do with block sizes, and metadata like file and folder names that take up space as well.

If you change the SPIFFS size, you have to reupload your sketch, because when you change the SPIFFS size, the memory location will be different. The program has to know the updated SPIFFS address offset to be able to read the files.
SPIFFS File Server
The following example is a very basic file server: it just takes the URI of the HTTP request, checks if the URI points to a file in the SPIFFS, and if it finds the file, it sends it as a response.

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266mDNS.h>
#include <ESP8266WebServer.h>
#include <FS.h> // Include the SPIFFS library

ESP8266WiFiMulti wifiMulti; // Create an instance of the ESP8266WiFiMulti class, called 'wifiMulti'

ESP8266WebServer server(80); // Create a webserver object that listens for HTTP request on port 80

String getContentType(String filename); // convert the file extension to the MIME type
bool handleFileRead(String path); // send the right file to the client (if it exists)

void setup() {
Serial.begin(115200); // Start the Serial communication to send messages to the computer
delay(10);
Serial.println('\n');

wifiMulti.addAP("ssid_from_AP_1", "your_password_for_AP_1"); // add Wi-Fi networks you want to connect to
wifiMulti.addAP("ssid_from_AP_2", "your_password_for_AP_2");
wifiMulti.addAP("ssid_from_AP_3", "your_password_for_AP_3");

Serial.println("Connecting ...");
int i = 0;
while (wifiMulti.run() != WL_CONNECTED) { // Wait for the Wi-Fi to connect
delay(250);
Serial.print('.');
}
Serial.println('\n');
Serial.print("Connected to ");
Serial.println(WiFi.SSID()); // Tell us what network we're connected to
Serial.print("IP address:\t");
Serial.println(WiFi.localIP()); // Send the IP address of the ESP8266 to the computer

if (MDNS.begin("esp8266")) { // Start the mDNS responder for esp8266.local
Serial.println("mDNS responder started");
} else {
Serial.println("Error setting up MDNS responder!");
}

SPIFFS.begin(); // Start the SPI Flash Files System

server.onNotFound([]() { // If the client requests any URI
if (!handleFileRead(server.uri())) // send it if it exists
server.send(404, "text/plain", "404: Not Found"); // otherwise, respond with a 404 (Not Found) error
});

server.begin(); // Actually start the server
Serial.println("HTTP server started");
}

void loop(void) {
server.handleClient();
}

String getContentType(String filename) { // convert the file extension to the MIME type
if (filename.endsWith(".html")) return "text/html";
else if (filename.endsWith(".css")) return "text/css";
else if (filename.endsWith(".js")) return "application/javascript";
else if (filename.endsWith(".ico")) return "image/x-icon";
return "text/plain";
}

bool handleFileRead(String path) { // send the right file to the client (if it exists)
Serial.println("handleFileRead: " + path);
if (path.endsWith("/")) path += "index.html"; // If a folder is requested, send the index file
String contentType = getContentType(path); // Get the MIME type
if (SPIFFS.exists(path)) { // If the file exists
File file = SPIFFS.open(path, "r"); // Open it
size_t sent = server.streamFile(file, contentType); // And send it to the client
file.close(); // Then close the file again
return true;
}
Serial.println("\tFile Not Found");
return false; // If the file doesn't exist, return false
}
As you can see, we don't use server.on in this example. Instead, we use server.onNotFound : this will match any URI, since we didn't declare any specific URI handlers like in the previous server examples.
When a URI is requested, we call the function handleFileRead . This function checks if the URI of the HTTP request is the path to an existing file in the SPIFFS. If that's the case, it sends the file back to the client. If the path doesn't exist, it returns false, and a 404 (Not Found) HTTP status will be sent.

The MIME type for the different files is based on the file extension.
You could add other file types as well. For instance:
String getContentType(String filename){
if(filename.endsWith(".htm")) return "text/html";
else if(filename.endsWith(".html")) return "text/html";
else if(filename.endsWith(".css")) return "text/css";
else if(filename.endsWith(".js")) return "application/javascript";
else if(filename.endsWith(".png")) return "image/png";
else if(filename.endsWith(".gif")) return "image/gif";
else if(filename.endsWith(".jpg")) return "image/jpeg";
else if(filename.endsWith(".ico")) return "image/x-icon";
else if(filename.endsWith(".xml")) return "text/xml";
else if(filename.endsWith(".pdf")) return "application/x-pdf";
else if(filename.endsWith(".zip")) return "application/x-zip";
else if(filename.endsWith(".gz")) return "application/x-gzip";
return "text/plain";
}

This example is adapted from the FSBrowser example by Hristo Gochkov.
Compressing files
The ESP8266's flash memory isn't huge, and most text files, like html, css etc. can be compressed by quite a large factor. Modern web browsers accept compressed files as a response, so we'll take advantage of this by uploading compressed versions of our html and icon files to the SPIFFS, in order to save space and bandwidth.

To do this, we need to add the GNU zip file type to our list of MIME types:
String getContentType(String filename){
if(filename.endsWith(".html")) return "text/html";
else if(filename.endsWith(".css")) return "text/css";
else if(filename.endsWith(".js")) return "application/javascript";
else if(filename.endsWith(".ico")) return "image/x-icon";
else if(filename.endsWith(".gz")) return "application/x-gzip";
return "text/plain";
}
And we need to change our handleFileRead function as well:
bool handleFileRead(String path){ // send the right file to the client (if it exists)
Serial.println("handleFileRead: " + path);
if(path.endsWith("/")) path += "index.html"; // If a folder is requested, send the index file
String contentType = getContentType(path); // Get the MIME type
String pathWithGz = path + ".gz";
if(SPIFFS.exists(pathWithGz) || SPIFFS.exists(path)){ // If the file exists, either as a compressed archive, or normal
if(SPIFFS.exists(pathWithGz)) // If there's a compressed version available
path += ".gz"; // Use the compressed version
File file = SPIFFS.open(path, "r"); // Open the file
size_t sent = server.streamFile(file, contentType); // Send it to the client
file.close(); // Close the file again
Serial.println(String("\tSent file: ") + path);
return true;
}
Serial.println(String("\tFile Not Found: ") + path);
return false; // If the file doesn't exist, return false
}
Now, try compressing some of the files to the GNU zip format (.gz), and uploading them to SPIFFS. Or you can just download the new data folder (unzip it first).
Every time a client requests a certain file, the ESP will check if a compressed version is available. If so, it will use that instead of the uncompressed file. The output in the Serial Monitor should look something like this:
handleFileRead: /
Sent file: /index.html.gz
handleFileRead: /main.css
Sent file: /main.css
handleFileRead: /JavaScript.js
Sent file: /JavaScript.js
handleFileRead: /folder/JavaScript.js
Sent file: /folder/JavaScript.js
handleFileRead: /favicon.ico
Sent file: /favicon.ico.gz
It automatically detected that it had to send the compressed versions of index.html and favicon.ico.

Source - WhatsApp Group Link

Who is online

Users browsing this forum: No registered users and 2 guests