ESP8266 IDF based development - my thoughts

[jhinkle]

This post is more like a blog post on my path/experiences and thoughts on using the new IDF development process.

Prior SDK's came with binary blobs of the internal code which could not be changed and user code was wrapped around it. I've used the RTOS SDK for a couple of years now but was motivated to look at the IDF process when a lwip issue on a commercial product could not be fixed by user code.

The IDF process removes most all of the binary blobs and exposes most all of the internals (like the lwip and freertos implementations). Espressif has packaged most of the available software packages/libraries as "components" - each with their our make file. This allows the developer to examine and modify/change the code if required. All of this is tied together via an interactive configuration menu designed to allow the developer to customize the ESP8266 (and ESP32 ) to their specif needs. I think the idea is fantastic - but the current config menu process is flawed and depends on Espressif to modify and fix (more on this in a moment).

The config menu asks if you are using the ESP8266 or the ESP32. It attempts to make all possible configuration setting for all of the "components" available in one spot. It tries to accomplish this by running a menuconfig make process that reads a component configuration file located in each of the component folders which adds configuration variables to the menuconfig process. After you execute the menuconfig process (a make process) and select all your component configuration settings, the menuconfig process outputs a "sdkconfig" file which holds all of your selections and generates a sdkconfig.h file used in the user code and component compilation process. The current menuconfig does NOT allow for component exclusion so you constantly compile components you are not interested in.

My opinion: The menuconfig process hides potential component implementation issues and depends on Espressif to make changes - neither is acceptable in my opinion (I can't wait weeks for Espressif to respond to an issue request). (examples in a moment)

My opinion: Get rid of the menuconfig process all together and release the sdkconfig.h file as a general configuration file which the developer can modify as required with an editor. This is what I did.

My opinion: Here are the problems with the current menuconfig process implementation.

The current menuconfig process tries to to modify the lwip's and freertos's configuration files by having the lwip and freertos configuration .h files be modified by the new IDF selection variable (the lwip implantation is flawed). IMO - the lwip and freertos config files should NOT be IDF configurable and left to the developer to configure as would be expected in any application that currently uses those libraries.

There are three (3) major players in "component" code used in the IDF process. LWIP, FreeRtos, and Espressif (they have multiple components). When a component/library is released (lwip and freertos as an example), their config .h file will use selection variables in their components code. Those selection variables are implemented using either #ifdef (is it define or not defined) or #if (it should be defined and should have a value associated with it (usually 1 or zero for an enable or disabled selection). lwip and freertos has been around for years and their build process works as long as their config .h files are properly implemented. Espressif's components are flawed in my opinion and suggest multiple developers were involved with no set standard on #ifdef or #if usage. The current Espressif components use a combination of both for the same selection variable leading to questionable compilation results. The current menuconfig process does NOT generate selection variables for ALL variables required by lwip or freertos which can lead to issues if those software products use a #if and expect to acquire a value (the compiler is make a guess at the value (usually 0) ) - which may or may not implement the behavior you expected.

The current make process does NOT expose these possible issues - currently everything compiles without issues for their components.

I modified the project.mk file (more in a moment) to include the -Wundef flag. This tells gcc to warn about all undefined variables that are associated with #if (usually) but are missing. When you do this - a ton of potential issues are exposed. Using project.mk with the added -Wundef flag will also identify multiple "defines" that are missing that could also lead to issues. I modified project.mk to add the following to its "COMMON_FLAGS" variable:
-D DEBUG=0 \
-D __GNU_VISIBLE=1 \
-D __have_long32=1 \
-D __have_long64=1 \
-D __BSD_VISIBLE=1

This is what I did to implement the new ESP8266 IDF process.

Run "make menuconfig". This will build Espressif's menuconfig and allow you to make selections. The important selections to me are the Bootloader and code generation selections --- NOT --- the component selections.

I went into each of the "component" folders and opened their associated "Kconfig" file. You can easily see how the menuconfig selection variables are created as well as Espressif's default options for each.

I created my our "My_sdkconfig.h" file and replaced "sdkconfig.h" with it in any of the component source files where it was used (this is were an editor that can search multiple files for specific text is priceless).

I then searched all of the source files associated with every component - for each of these config selection variables. Where I found Espressif using #ifdef and #if with the same variable, I modified it to only use #if.

I made lwip's and freertos's config .h files be stand alone (removed Espressif's selection CONFIG_variable) and set the selection I wanted per the way they wanted it. lwip's config file "lwipopts.h" was the took the most work. Many of the required lwip selection variables that wanted a value (used in #if) were missing which meant gcc normally assigned a value of 0. I found some missing altogether: Menufig variable Config_LWIP_MAX_SOCKETS (sets the number of sockets available) was not used. It's meant to set the value associated with ESP_SOCKET_MAX. I suspect the current non-modified IDF compilation assigned a value of 0 to this variable.

My config file "My_sdkconfig.h" holds all configuration variables and have them either defined/not defined or has a value set depending on how the source code used that variable.

The current IDF process does not provide a way of NOT using/implementing a component. The make process will build all components found in the components folder. For my IDF implementation, I removed all components I did NOT want and saved them in an Unused_Components folder. (I removed components cjson, mqtt, pthread, etc as an example).

I also changed esp8266.common.ld located in esp8266.ld folder - replacing "sdkconfig.h" with "My_sdfconfig.h" - that allows my config method to modify the build process - not using the one associated with Espressif's menuconfig process.

The end result is that I can NOW build a ESP8266 application with MY selections and not be dependent on Espressif for changes. YES - I have modified component source files so a git update is a little harder (I git to a new folder and the use my editor's multiple file compare capability to see what changes Espressif may have been made to their ESP8266 RTOS product and update my source as required). I'll file "issues" to the Espressif's ESP8266_RTOS_IDF git site but I no longer need to wait for them to address any issue I can solve myself).

As a note for anyone developing code from the ESP32 and the ESP8266 IDF -- Espressif wants an global ENV variable set which identifies the path to the IDF package to be used (either ESP8266 or ESP32). The way I worked around this is to NOT use a ENV variable at all. Each ESP8266 or ESP32 IDF project has an associated project make file (makefile). I added the path as part of the project. This way the make process will use to proper SDK. The contents of an example modified project make is:

*******************************************************************************************
#
# This is a project Makefile. It is assumed the directory this Makefile resides in is a
# project subdirectory.
#

PROJECT_NAME := project_template

export IDF_PATH=C:/msys32/home/Joe/esp/ESP8266_RTOS_SDK

include $(IDF_PATH)/make/project.mk

*******************************************************************************************

Espressif states that the new IDF process should not be used with existing applications - new ones only.

https://github.com/espressif/ESP8266_RTOS_SDK

My approach to updating an existing project:

Most all Espressif related function names in the SDK have changed from those used in the binary-blob SDKs (example - function to change CPU clock speed, etc). There is not a lot of documentation on the new IDF functions available -- but you have access to the code to see what it does and how it works.

For me - I am currently rewriting my ESP8266 startup process that gets the device up and running is Access Point mode, Station mode, or AP+STA mode - This take some time!!! After I get THAT implemented - most all of my own code should import easily since it is NOT dependent of Espressif's functions. I'm not done yet - but I would suggest you look into your own application code to see the time impact associated with conversion.

I hope this information can help anyone that wants to develop a ESP8266 RTOS application and have total control over 95% of the code used to generate that application.

Joe

[pjs]

Helpful post. New to ESP8266 and just set up ESP8266 IDF based tools and SDK (ESP8266_RTOS_SDK). A bit confusing of a process but looks like I have things set up and configured correctly as I can build, flash and run the simple WiFi example app on dev kit. even configured Eclipse to import and build projects. Can you clarify a few things for me?

1. I got the SDK from git repository( ESP8266_RTOS_SDK v3.0) and yes there seems no formal documentation other then source code. The Espressif web site documents ESP32 APIs fairly extensively. I see a document for ESP8266 2.0 API. But there seems to be quite a difference with the ESP8266 3.0 API. What is the purpose of the 3.0 release (vs for example 2.0), just to make it compatible with IDF? I keep wondering if 3.0 / IDF is the correct starting point?

2. Sounds like you are trying to write app that brings up device in AP mode. This was my first thought for a starting point to better understand the development environment, and would also like to serve a few simple web pages. Having a hard time getting started. I did start with the simple wifi example that starts in AP mode, now would like to make it a http server so I can understand how to serve up web pages. Could not find any examples for that, have you?