IllegalInstructionCause in system_get_os_print

[gailu]

Hi Experts,

We are using ESP12-E successfully for some time and are devices are deployed in field. We are facing random crashes when prints are enabled in the firmware. Following are the details.

Crash Logs:
Fatal exception 0(IllegalInstructionCause):
epc1=0x40201c10, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

ets Jan 8 2013,rst cause:2, boot mode:(3,7)

load 0x40100000, len 1396, room 16

We checked the address epc1 address 0x40201c10 in user1.4096.new.4.S and following is the details from symbol file

40201c10 <system_get_os_print>:
40201c10: fffb21 l32r a2, 40201bfc <spi_flash_set_read_func+0x8>
40201c13: 000222 l8ui a2, a2, 0
40201c16: f00d ret.n
40201c18: feb344 excw
40201c1b: 3f .byte 0x3f
40201c1c: 1cec bnez.n a12, 40201c41 <system_get_os_print+0x31>
40201c1e: 404020 excw
40201c21: 402327 blt a3, a2, 40201c65 <system_get_os_print+0x55>
40201c24: b3ec bnez.n a3, 40201c53 <system_get_os_print+0x43>
...

Can you please suggest how to fix this crash
We are using SDK 1.4

Looking forward to your expert advice as some of our live devices are impacted by this.

[pratik]

Hello,

Illegal Instruction error is almost always caused by data corruption in flash or glitches in power supply that affect flash read operation. So you should look at the FOTA (update) process for your product if you have implemented that. Make sure that is not the cause of the problem.
Erasing the flash completely, then reprogramming the application may also help. I have an article on that here:
http://www.pratikpanda.com/completely-f ... sh-memory/

As for power glitches - it may happen when you are printing stuff out. Make sure your device power supply is properly decoupled and there are weak pullups or pulldowns on ESP8266 pins, wherever necessary. Typically, one would not use a resistor lower than 10K for pull-up/down. You could also debug by reducing flash access speed to below 40MHz.

Personally, I have only encountered this type of exception with the above 2 causes only. Also, if your firmware writes too often to the flash memory... note that the flash write cycles are limited and the flash may not function properly around the end of its serviceable cycles.

[gailu]

Problem due to FOTA will only happen when you trigger the FOTA update. Yes, we do have FOTA implementation but this problem is definitely not due to FOTA. Power supply is also properly decoupled

[gailu]

Tried internal pull up on all unused pins with no help at all. Crashing at same place

[gailu]

changing flash speed also has no effect at all.

I hope someone from espressif team must have an answer to it I believe crash is really a serious issue.

[pratik]

Okay, so that rules out all hardware and FOTA issues that may result in corruption.
Does this also occur with other SDK versions? v.1.4 is not the latest and this might have been resolved in subsequent versions. I am assuming you are using the RTOS version of the SDK?

It might be the print function then. I will inform a firmware developer to get back to you on this.
Thanks for using the ESP8266 in your design!

[gailu]

Hi Prateek,

We are using following SDK.

esp_iot_sdk_v1.4.0_15_09_18

[ESP_Rubin]

please provide the reproduece code

[gailu]

Dear ESP_Rubin,

It is not possible to provide complete source code of our IP otherwise I would have shared it in the first post itself. We have relays in our hardware and we see that it crashes when os_printf is called while switching relays ON/OFF probably some disturbances but not able to nail down problem.

I am wondering if system_get_os_print or other print functions uses ICACHE_FLASH_ATTR and can I implement printf without this attribute?

[pratik]

If you get the error at the exact same code address every time then it is not likely to be related to power. Can you please run the code leaving Tx and Rx pins of the ESP8266 floating?

If you feel relays have something to do with this, please scope the power rails of the ESP8266 module when the relay is triggered. Use AC coupling and see if relay triggering affects the power rails too much. 10% tolerance is what you should have on the power rails to the module.
Other troubleshooting steps:
- Use a separate power source for the relay temporarily to see if things work any better.
- Make sure you are using an appropriate diode to suppress freewheeling current around relay circuitry. Not having this will definitely cause problems.

You can implement your own print functions using the UART registers directly, in which case you can have the function send characters directly without APIs and completely reside in iRAM instead of iCACHE.
Attached is UART register summary in case you want to do it that way.