(RTOS_SDK) mbedtls_ssl_write low memory issue

[gustavomassa]

Hello,

The function mbedtls_ssl_write get stuck forever when there is low heap memory, about 3-4kb of heap. The ssl_write will block forever until you reset the device, it only occurs when the heap memory is low. The interrupts still works while the write is blocking forever. Please investigate.

File -> RTOS_SDK\third_party\mbedtls\library\ssl_tls.c
Function -> int mbedtls_ssl_write( mbedtls_ssl_context *ssl, const unsigned char *buf, size_t len )

Digging a little bit more on the issue, I've added vPortEnterCritical() and vPortExitCritical between the "ssl_write" call, then the code always crashes and prints ShowCritical:1 message on the uart.

[pratik]

So this is an RTOS SDK issue I guess? Because mbedTLS on nonOS SDK works fine near minimum heap usage limits too, as long as it can allocate memory for operations. And if it cannot, a memory allocation error is generated (and sending fails).
Which version of SDK are you using? Can you attach a log?
I would like to know if this failure occurs during a callback or just before a callback.

For temporarily fixing this, try to check free heap before calling the function that causes failure with low heap space.

[Her Mary]

Without enough memory, you can do nothing.
Which version of ESP8266 RTOS SDK are you using? Maybe you can try the latest one.

[ememberus]

Without enough memory, you can do nothing.

Running an infinite loop and waiting gives developers no clue to what is wrong and how to resolve the issue.
In case of low memory, you can return an error code or message indicating this explicitly.
Silent default behavior will poison everything and should be completely eliminated.

[pratik]

There is not enough information for your issue. If you are using RTOS SDK, then you will need to make your stack use comply with max stack usage allowed for tasks. Do let me know of which version of SDK you are using and stack usage. I suspect that is what is happening.
Note that no silent action is ever taken up by SDK. You will get a reboot if there is an issue, always. Unless it is power supply issue, of course.

[ememberus]

Note that no silent action is ever taken up by SDK.

I dare to disagree with this statement, because I personally
encountered these in non-RTOS library.

Endless loops while waiting for resources is a doubtful practice.
Such loops should be interrupted and error generated.

Apart from hiding such internal issues from developers,
lots of other things in SDK are also not documented.
We often have hard time figuring out what went wrong
and how to resolve it.

Of course, we can certainly make and "educated guess", but
do you really think this is how SDK issues should be handled?

[ememberus]

You will get a reboot if there is an issue, always.

in the order of importance:

1. in this particular case, the system freezes, but does not reboot,
simply because it is running an internal loop that resets WD timer.

2. the system should never reboot unless the error is fatal.
A "fatal error" means that the system cannot function whatsoever.
Low memory should not be a fatal error.
Give the application some time to complete its current
task so it would release previously allocated memory.

3. system reboot should only be last resort when where is no other
way to recover from an error.

[pratik]

I've never encountered such issues, it would be nice if you could share the code that is causing the problem. All WiFi networking functions will return with error instead of waiting for resource allocation.
You can look at the code for LwIP or mbedTLS (they are open), if that would help. With FreeRTOS, you might want to make sure you are not exceeding the stack size allotted to a task.
Also, do not starve the idle task, because that's the one which handles memory deallocation.

[ememberus]

Pratik, here are few clarifying notes

- In order to avoid memory leaks, I no longer use secure layer.

- Problem: TCP stack in SDK is unstable by itself - it fails to open
new connections after a number of previous disconnects on WLAN level
(this was previously reported by others).

- Hypothesis: this seems to me like a connection resource leak, when
broken TCP connections are not released back into connection pool.

- Hint: this is likely caused by a race condition, because it is not
100% reproducible, but sooner or later the SDK eventually goes into
this state and becomes dysfunctional.

- Question: you seem to have access to the SDK code.
Could you take a look into this?
I am using ESP8266_NONOS_SDK-2.1.0

Thanks

[davydnorris]

Hi @ememberus,

The source for the lwip and mbedtls libraries has now been released in the NONOS SDK on Github:
https://github.com/espressif/ESP8266_NONOS_SDK

Have a look under third_party.

There have also very recently been some fixes and a couple of pull requests addressing leaks and connections bugs in the RTOS Github repo versions of these libraries. They haven't been merged into the NONOS version yet but you could do it yourself:
https://github.com/espressif/ESP8266_RTOS_SDK/commits/master

I am busy with other things but I want to try this myself - if you get a chance please let us know if it solves your problem.