Cannot recover from Fatal exception (0): problem during boot

[beegee1962]

Used hardware: Adafruit HUZZAH ESP8266 ESP12

Used software:

- Arduino IDE V1.6.7 with esp8266 board V2.0.0
- AT firmware (latest from github https://github.com/espressif/ESP8266_AT)
- NodeMCU (0.9.6-dev_20150704 from github https://github.com/nodemcu/nodemcu-firmware/releases

Used flash tools:

Espressif ESP FLASH DOWNLOAD TOOL V2.4
esptool.py

What happened:

- Tested my own project (written with Arduino IDE)
- Used Arduino OTA (device is in difficult accessible location)
- OTA worked several times
- OTA failed, device not connecting to WiFi anymore, even after power off/on cycle
- Removed ESP module, try to reflash over serial port with FDDI cable
- Still ESP module fail to boot

Found error codes:
On serial monitor (74880 baud) module shows always endless loop of

Code: Select all

Fatal exception (0):
epc1=0x40211668, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000]

Trial to recover:
a) Reflash module with working code (works on different ESP12 module). ==> fail, same error, only epc1=0x... shows different address
Terminal output:

Code: Select all

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x4010f000, len 1264, room 16
tail 0
chksum 0x42
csum 0x42
~ld
Fatal exception (0):
epc1=0x402147a4, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

b) Reflash module with NodeMCU 0.9.6 ==> fail, same error, only epc1=0x... shows different address

Code: Select all

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x4010f000, len 1264, room 16
tail 0
chksum 0x42
csum 0x42
~ld
Fatal exception (0):
epc1=0x402147a4, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

c) Reflash module with lates AT firmware found on github => fail, same error, only epc1=0x... shows different address

Code: Select all

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x40100000, len 612, room 16
tail 4
chksum 0x12
load 0x3ffe8000, len 788, room 4
tail 0
chksum 0x50
load 0x3ffe8314, len 264, room 8
tail 0
chksum 0x4a
csum 0x4a

2nd boot version : 1.1
  SPI Speed      : 40MHz
  SPI Mode       : QIO
  SPI Flash Size : 32Mbit
jump to run user1

Fatal exception (0):
epc1=0x40211558, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

d) Erase complete flash with esptool.py. Reflash module with boot and esp_init_data_default.bin (no user1.bin or user2.bin flashed!) from latest AT firmware ==> no more Fatal Exception

Code: Select all

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x40100000, len 612, room 16
tail 4
chksum 0x12
load 0x3ffe8000, len 788, room 4
tail 0
chksum 0x50
load 0x3ffe8314, len 264, room 8
tail 0
chksum 0x4a
csum 0x4a

2nd boot version : 1.1
  SPI Speed      : 40MHz
  SPI Mode       : QIO
  SPI Flash Size : 32Mbit
jump to run user1

user code done

Try to eliminate power supply problem
Tried 4 different power supplies (Adafruit Huzzah ESP8266 has onboard power regulator, input voltage range 3.3V to 16V) and 3 different Adafruit Huzzah ESP8266 modules

5V USB supply from iPhone, 5V, 1A
5V USB supply from Samsung Note 10.1, 5V, 2A
9V wall plug with additional caps (470uF elec. & 100nF ceramic), 9V 1A
5V from LM317 build power supply, 5V 1.5A

=> all the 4 power supplies work fine on the 2 ESP modules without problem

Try to check for flash failure
a) Erase and read back

- erase complete flash with esptool.py erase_flash
- read back complete flash with esptool.py read_flash 0 4194304 erased.bin
- check erased.bin with hex editor ==> complete content is 0xFF

b) Flash and read back

- erase complete flash with esptool.py erase_flash
- flash AT firmware with ESP FLASH DOWNLOAD TOOL V2.4
- used addresses:

• 0x000000 boot_v1.1.bin
• 0x001000 user1.bin
• 0x07c000 esp_init_data_default.bin
• 0x3fc000 esp_init_data_default.bin
• 0x07e000 blank.bin
• 0x3fe000 blank.bin
  - use CombineBin to create combined file target.bin
  - read back complete flash with esptool.py read_flash 0 4194304 at.bin
  - compare at.bin with target.bin ==> complete content is identical

==> seems the flash is working fine

I went to a lot of forums and searched for this problem, but found no recovery solution that worked in my case.
Anybody has an idea what else I can try to recover the module?

[ESP_Faye]

Hi,

Please try:
0x000000 boot_v1.5.bin
0x001000 user1.bin
0x3fc000 esp_init_data_default.bin
0x3fe000 blank.bin

Flash size in ESP FLASH Download Tool to be 32Mbit.

Here is the latest firmware ESP8266_NONOS_SDK_V1.5.1.

If your problem is still unsolved, please provide the captured picture of ESP FLASH Download Tool while downloading.

Thanks for your interest in ESP8266 !

[beegee1962]

Hi,

Please try:
0x000000 boot_v1.5.bin
0x001000 user1.bin
0x3fc000 esp_init_data_default.bin
0x3fe000 blank.bin

Flash size in ESP FLASH Download Tool to be 32Mbit.

Here is the latest firmware ESP8266_NONOS_SDK_V1.5.1.

If your problem is still unsolved, please provide the captured picture of ESP FLASH Download Tool while downloading.

Thanks for your interest in ESP8266 !

Thanks for your reply.
I tried what you proposed, but no difference, still get Fatal exception(0): after boot (if a user1.bin was flashed).

One question: in the readme.txt in the sdk tool it is written:

Code: Select all

Flash size 32Mbit: 512KB+512KB
boot_v1.2+.bin         0x00000
user1.1024.new.2.bin     0x01000
esp_init_data_default.bin   0x3fc000 (optional)
blank.bin              0x7e000 & 0x3fe000

Is this the one I should use with the ESP12 module? But you said blank.bin needs only to be flashed at 0x3fe000?

I attached three screenshots. Recover-ESP8266-1.JPG is the ESP FLASH Download tool just before flashing, Recover-ESP8266-2.JPG & Recover-ESP8266-3.JPG is the output in the terminal window of the Download tool. Recover-ESP8266-4.JPG is the Download tool after flashing.

I see several EFUSE warnings/errors in the log. Maybe my module is really damaged:

Code: Select all

...
======================
EFUSE NORMAL MODE
======================
bit[1:0] error
bit[59:56] error
...
bit[47:24] crc warning
efuse warning found
...
======================
EFUSE NORMAL MODE
======================
EFUSE VAL ERROR...
EFUSE VAL WARNING...
EFUSE CRC WARNING...
...

I don't know what the EFUSE is doing, but there seems to be something wrong.

[ESP_Faye]

Hi,

Sorry that it seems your module is broken. Please try it with another module..

[beegee1962]

Hi,

Sorry that it seems your module is broken. Please try it with another module..

I was afraid it is broken. What exactly is the "EFUSE" doing? Any chance to repair the board by e.g. replacing the flash chip?

[ESP_Faye]

Hi,

Did you use 5V power supply to connect to ESP8266 directly ?

Could you help us debug the hardware of your module following below steps :
1. Measure the voltage of ESP8266's pin VDD.
2. Use hot air gun to remove the wifi Chip from the PCB board and measure the resisitence of each pin to the ground to check if the chip is broken.
3. Power on the ESP8266 module, measure the voltage of ESP8266's 3V3 pin.

Thanks for your interest in ESP8266 !

[beegee1962]

Hi,

Did you use 5V power supply to connect to ESP8266 directly ?

Could you help us debug the hardware of your module following below steps :
1. Measure the voltage of ESP8266's pin VDD.
2. Use hot air gun to remove the wifi Chip from the PCB board and measure the resisitence of each pin to the ground to check if the chip is broken.
3. Power on the ESP8266 module, measure the voltage of ESP8266's 3V3 pin.

Thanks for your interest in ESP8266 !

I am using an Adafruit HUZZAH ESP8266 module. This module has an onboard 3.3V regulator.

I measured the voltage on ESP8266 VDD pin
=> 3.3V

I measured all pins of the ESP8266 module against ground
=> all are high resistance, nothing shorted to ground

I removed the metal cover on the ESP8266 module and measured all pins of the Espressif chip against GND
=> all high resistance

I powered the board and measured all VDD pins at the 32pin QFN chip (Tensilica L106??)
Pin 1 + 30 VDDA = 3.3V
Pin 3 + 4 VDD3P3 = 3.3V
Pin 5 VDD_RTC = 0.08V
Pin 12 + 17 VDDPST = 3.3V
Pin 29 VDDD = 3.3V

But something else has changed before I measured all these values and before I removed the metal cover.

A few days ago, when I powered up the module, the blue LED was light up and the red LED was dimmed light up. When I pressed RESET while keeping GPIO0 connected to ground, the blue LED stayed off, the red LED was dimmed light up and I could get the bootloader message over serial connection. I could even program the module, but the program didn't work (see my first post).

Today when I powered up the module, none of the LEDs was light up nothing is visible on the serial connection (even at 74880 baud) and even with RESET and GPIO0 on ground I cannot get the module into programming mode anymore.

I guess the module is now complete dead. Looks like I need to buy a new module. Hope it is working longer than this one.

Thanks for your support.

[Admin]

I think it's possible that the flash device might have failed. the other possibility is with the regulator.
yes, while it's possible that ESP8266 might have failed, it's extremely unlikely. If it has failed, then probably it is due to very high voltage ESD on the VDD_RTC pin, or VDD33 pin. it's best not to remove the cover.

[beegee1962]

I think it's possible that the flash device might have failed. the other possibility is with the regulator.
yes, while it's possible that ESP8266 might have failed, it's extremely unlikely. If it has failed, then probably it is due to very high voltage ESD on the VDD_RTC pin, or VDD33 pin. it's best not to remove the cover.

The module failed while it was mounted in a housing, no possibility that ESD would have touched any pin.
The cover was removed after the module died.

As a last try I separated the ESP8266 module now from the Adafruit carrier board. The carrier board works fine, 3.3V generation is ok, level shifters work fine, switches work fine.
Tried to supply the ESP8266 module just with 3.3V and see if I can get any reaction from it, but it is complete dead now.

RIP

[Admin]

Would you like to ship your dead module to us? we have very strict RMA policies and we would like to conduct a postmortem on this. There are many possibilities and for this case, the manufacturing process has been rather convoluted from a module house to Adafruit to you. it may be hard for us to pin-point the failure mode. But the usual suspects are ESD or ill-designed power supplies, failure of the flash, or even the capacitor and of course, failure of the chip is a possibility. But because we have done so much testing on our chip, we think that this is absolutely the least probable cause.

Espressif's chips are exclusively manufactured by TSMC, which in the area of CMOS technology, probably has the best reliability, RF device performance and power consumption in the industry based on what we have seen so far.

We have shipped 50 million WiFi parts in various industries without issues. While it's possible that what you have seen is a statistical blip, it is more probable that some part of the module has been damaged, or there is over-voltage or ESD from the pins of the module, since these pins are directly connected to the chip?

For your case, we suspect that we would see over-voltage, because of the presence of EFUSE errors, which means that a high voltage has occurred and burnt out some EFUSES such that our firmware's CRC detects an error. Burning out EFUSE is probably not possible without entering into a test mode, or direct application of a very high voltage.

On Reliability and Quality:
Our WiFi chip product lines are qualified by accredited external test houses, for ESD performance, reliable operation for 20 years under normal operating conditions and normal usage. The test results are available upon request from us.

The WROOM modules marketed by Espressif are also qualified by accredited external test houses, which means that we qualify all the parts on the module, including capacitors, resistors, crystal, and flash. There are a couple of versions of WROOM modules with different operating temperature range, but they are all qualified for 10 years of reliable operations. Test results are available upon request.

If you are using a module other than WROOM modules, it's possible that some manufacturers are not using Espressif's recommended BOM list.

We would be glad to replace your part, if you could send it to us, at our expense, for more analysis.