SSL Authentication to AWS IOT Service

[Deadolus]

Hello,

I am trying to connect the ESP8266 to the AWS IOT Cloud for mqtt publishing.
I received multiple certification files from Amazon and trying to convert everything to the ESP, but I have trouble understanding the documentation.
As a basis I use https://github.com/tuanpmt/esp_mqtt with SECURITY defined as "1" (SSL).
I can successfully publish stuff to the cloud with this command

Code: Select all

mosquitto_pub --cafile VeriSign-Class\ 3-Public-Primary-Certification-Authority-G5.pem --cert thing_esp8266/a05f990520-certificate.pem.crt --key thing_esp8266/a05f990520-private.pem.key -h A2UPMX25ELLWO1.iot.us-west-2.amazonaws.com -p 8883 -q 1 -d -t topic/test -i esp8266 -m "hello world"


So I know which files the ESP needs to work properly.

I created a folder "ca_key" in which I copied the VeriSign.....pem file and the make_cacert.py file from the example.
I then converted the Verisign license with the command

Code: Select all

openssl x509 -in VeriSign-Class\ 3-Public-Primary-Certification-Authority-G5.pem -outform DER -out TLS.ca_x509.cer

Then I ran the make_cacert.py command with

Code: Select all

python make_cacert.py

This generated "esp_ca_cert.bin", which seems to contain binary data.

I created another folder named "client_key" in which I copied the a05....certificate.pem.crt and the a05....private.pem.key and the make_cert.py script from the example.
I converted the private key with

Code: Select all

openssl rsa -in a05f990520-private.pem.key -out private_key.key1024 -outform DER

and the certificate with

Code: Select all

 openssl x509 -in a05f990520-certificate.pem.crt -outform DER -out certificate.cer


I then ran the make_cert.py command with

Code: Select all

python make_cert.py

which generated "esp_cert_private_key.bin", which now also contains binary data(beginning with 0x70 0x72).

I then modified the esp_mqtt program to work with SSL, mainly I added:

Code: Select all

            espconn_secure_ca_enable(0x01, CA_FLASH_SECTOR); //connect as client
            espconn_secure_cert_req_enable(0x01, CERT_FLASH_SECTOR); //connect as client

where CA_FLASH_SECTOR is defined as 0x3A and CERT_FLASH_SECTOR as 0x7E.

I figured that if one always blanks those sectors I could securely reuse them as holding my certification files (not sure if that's true, I can't seem to figure out how exactly the code will be written in to the flash and where it is safe to flash my own things).
I then flashed the program and the two binary certificates (at 0x3A000 and 0x7E000 respectively) in to the ESP and started the program up.
The output of the program is as follows:

Code: Select all

dhcp client start...
STATION_IDLE
STATION_IDLE
STATION_IDLE
STATION_IDLE
STATION_IDLE
STATION_IDLE
STATION_IDLE
ip:10.0.0.11,mask:255.255.255.0,gw:10.0.0.1
TCP: Connect to domain A21PT0FIM59RP0.iot.us-west-2.amazonaws.com:8883
DNS: found ip 54.187.143.164
TCP: connecting...
TLS.ca_x509.cer 4d7
the file is not a PEM file.
private_key 4a8
certificate 35d
the file is not a PEM file.
the file is not a PEM file.
client handshake start.
client handshake failed
Error: SSL error 40
pm open,type:2 0


So it seems that I do something, somewhere wrong - I would appreciate any help.
I am particularly thrown by the ESP complaining that "the file is not a PEM file" and reporting private_key with 4a8 (I figured this might be the beginning of the location and thus might be wrong).
It would be wonderful if we could manage to connect the ESP to the Cloud via SSL!

[Deadolus]

I debugged further and found that the ESP does not seem to send a client certificate.

I started a local ssl server with (device 3 is another device I defined in AWS IoT)

Code: Select all

openssl s_server -accept 8883 -cert ~/device3/device3_certificate.pem -key ~/device3/device3_private.pem -CAfile ~/VeriSign-Class\ 3-Public-Primary-Certification-Authority-G5.pem -msg -verify 1

I get

Code: Select all

<<< TLS 1.1 Handshake [length 0007], Certificate
    0b 00 00 03 00 00 00


From the ESP8266 as a "client certificate.
The output on the ESP is

Code: Select all

STATION_IDLE
ip:10.0.0.11,mask:255.255.255.0,gw:10.0.0.1
TCP: Connect to ip  10.0.0.3:8883
client handshake start.
client handshake failed
Error: SSL error 10
pm open,type:2 0


So it doesn't complain anymore about "this is not a PEM file", but instead shows a new error.
I always get the same number as client certificate, no matter what certificate I burn in to the flash or which Sector number I burn it in to.
Incidentally 3B is the address defined in the SSL Manual:

uint8 flash_sector : Flash sector in which CA certificate
(esp_ca_cert.bin) is written into. For example, parameters 0x3B
should be written into Flash 0x3B000 in the flash. Please be noted
that sectors used for storing codes and system parameters must not
be covered.

I don't know if this is just pure incident (as the Client certificate is not just 0x3b but has more zeroes in it), or if the subroutine maybe does not read in my actual client certificate and instead just sends the default value of where it assumes the client certificate should be(?).


edit:
The correct way of a connection is:

Code: Select all

openssl s_client -host A21PT0FIM59RP0.iot.us-west-2.amazonaws.com -port 8883 -cert ~/device3/device3_certificate.pem -key ~/device3/device3_private.pem -CAfile ~/VeriSign-Class\ 3-Public-Primary-Certification-Authority-G5.pem -msg -verify 1


The received server certificate is 2653 bytes long (0x0a5d)

<<< TLS 1.2 Handshake [length 0a5d], Certificate
.......

The ESP should send a certificate of 872 bytes (0x368)

>>> TLS 1.2 Handshake [length 0368], Certificate
.....

[ESP_Faye]

Hi,

Please disable the CA and cert, and have a try first.

Code: Select all

           // espconn_secure_ca_enable(0x01, CA_FLASH_SECTOR); //connect as client
           // espconn_secure_cert_req_enable(0x01, CERT_FLASH_SECTOR); //connect as client


If it is still fail to connect to your server, please provide the log for debugging.

Here is a demo of SSL http://bbs.espressif.com/viewtopic.php?f=31&t=389.

[Deadolus]

Hello Faye,

thanks for looking in to this.
I have disabled the two lines again, but of course it still failed.

What the demo does is encryption of the TCP channel - no real certificate required here, as only the connection gets encrypted (so nobody else can read its data directly.

But what I need to achieve with the Amazon Cloud is also authentication.
For this I need a valid certificate exchange. The server needs to read my client key and will challenge me (so this is why I also need the private certificate).
This is why I added the two lines in the code.

I am probably still doing something wrong.
I have tried to document all my steps as clearly and simple to follow, so you can verify what I am doing.
Alternatively, but unlikely this might also be a bug in the SDK.
This post: viewtopic.php?f=7&t=1425&p=4709&hilit=amazonaws#p4709 seemed to have similar problems, but the poster never indicated if he was able to solve the issue or not.

edit:
Thanks to quant67's post https://github.com/tuanpmt/esp_mqtt/iss ... -173525131
I get a different output on the ESP.
So I now don't convert the certificates received by Amazon anymore and instead let make_cert.py and make_cacert.py Scripts directly on the (properly renamed) files.
So now the python scripts just seem to escape some characters in the Ascii coded certificates.
When I upload these certificates I get

STATION_IDLE
STATION_IDLE
STATION_IDLE
STATION_IDLE
ip:10.0.0.11,mask:255.255.255.0,gw:10.0.0.1
TCP: Connect to domain A21PT0FIM59RP0.iot.us-west-2.amazonaws.com:8883
DNS: found ip 54.200.238.99
Loaded certificates, trying to connect...TCP: connecting...
verisign 6c4
the file is a PEM file.
private_key 68b
certificate 4c8
the file is a PEM file.
the file is a PEM file.
client handshake start.
client handshake failed
Error: SSL error 40
pm open,type:2 0

When connecting to the Amazon Cloud and

STATION_IDLE
STATION_IDLE
ip:10.0.0.11,mask:255.255.255.0,gw:10.0.0.1
TCP: Connect to ip 10.0.0.3:8883
verisign 6c4
the file is a PEM file.
private_key 68b
certificate 4c8
the file is a PEM file.
the file is a PEM file.
client handshake start.
distinguished names: [<null>], [AWS IoT Certificate]
before 2995055104, tv_sec 2995055113, after 2693241344
client handshake failed
Error: Cert has expired
TCP: Reconnect to 10.0.0.3:8883

when connecting to the local server.
I think the local error can be explained by using "device" certificates instead of proper server certificates.
But on the upside, it seems to read the server certificate (the handshake fails before exchanging the client key).

Also it now says that this is a PEM file ("the file is a PEM file"), which seems to be some progress to me.

edit2:
When I comment (disabling) the

Code: Select all

espconn_secure_ca_enable

line the connection to the local ssh server works

>>> TLS 1.1 Handshake [length 00dd], CertificateRequest
...
<<< TLS 1.1 Handshake [length 0368], Certificate
....
depth=0 CN = AWS IoT Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = AWS IoT Certificate
verify error:num=27:certificate not trusted
verify return:1
depth=0 CN = AWS IoT Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
<<< TLS 1.1 Handshake [length 0106], ClientKeyExchange
...
<<< TLS 1.1 Handshake [length 0106], CertificateVerify
...
<<< TLS 1.1 ChangeCipherSpec [length 0001]
01
<<< TLS 1.1 Handshake [length 0010], Finished
...
>>> TLS 1.1 ChangeCipherSpec [length 0001]
01
>>> TLS 1.1 Handshake [length 0010], Finished
...
-----BEGIN SSL SESSION PARAMETERS-----
...
-----END SSL SESSION PARAMETERS-----
Client certificate
-----BEGIN CERTIFICATE-----
....
-----END CERTIFICATE-----
subject=/CN=AWS IoT Certificate
issuer=/OU=Amazon Web Services O=Amazon.com Inc. L=Seattle ST=Washington C=US
Shared ciphers:AES128-SHA:AES256-SHA:RC4-SHA:RC4-MD5
CIPHER is AES128-SHA
Secure Renegotiation IS NOT supported

Not sure how to proceed from here on forwards at the moment.
All it seems to be left is a certification error.
I don't quite remember if the SSL protocol needs the client to verify the CA or if it can just assume it is ok to bypass this (will have to reread the specification).
I am also not really sure if everything with the key exchange is ok or not. As openssl spits out some "verify error"s.
Will investigate further...

[chaeplin]

question: Is tls1.2 supported by esp ?

http://docs.aws.amazon.com/iot/latest/d ... n-iot.html
TLSv1.2
SHA-256 RSA certificate signature validation
One of the cipher suites from the TLS cipher suite support section

[Deadolus]

@chaeplin Very good point.

When I test it with TLS1.2, using openssl

Code: Select all

openssl s_server -accept 8883 -cert ~/device3/device3_certificate.pem -key ~/device3/device3_private.pem -CAfile ~/VeriSign-Class\ 3-Public-Primary-Certification-Authority-G5.pem -msg -verify 1 -tls1_2


I get

<<< TLS 1.2 Handshake [length 0033], ClientHello
...
>>> TLS 1.1 Alert [length 0002], fatal protocol_version
...
ERROR
140387270956704:error:1408A10B:SSL routines:SSL3_GET_CLIENT_HELLO:wrong version number:s3_srvr.c:956:
shutting down SSL
CONNECTION CLOSED

and on the ESP

Error: SSL error 70

Also https://github.com/esp8266/Arduino/issues/43 indicates that the ESP does not (yet?) support TLS1.2 and thus can't be connected using the standard SDK ssl library.

[ESP_Faye]

Hi,

Yes, ESP8266 does not support TLS 1.2 now.
Are you trying to connect "A1QQ280DDHPJDD.iot.us-east-1.amazonaws.com" ? It seems to be TLS 1.2 which ESP8266 does not support now.
In your log, it is the server send an ALERT packet after ESP8266 sent SSL_CLIENT_HELLO packet.
ESP8266 can connect to https://aws.amazon.com but fail to connect to the IOT server. Or you can try to check the difference between these servers.

[cforsberg]

But how about https://twitter.com/TimMattison/status/ ... 6289049600 ?

[Tomasz_K]

@chaeplin Very good point.

When I test it with TLS1.2, using openssl

Code: Select all

openssl s_server -accept 8883 -cert ~/device3/device3_certificate.pem -key ~/device3/device3_private.pem -CAfile ~/VeriSign-Class\ 3-Public-Primary-Certification-Authority-G5.pem -msg -verify 1 -tls1_2


I get

<<< TLS 1.2 Handshake [length 0033], ClientHello
...
>>> TLS 1.1 Alert [length 0002], fatal protocol_version
...
ERROR
140387270956704:error:1408A10B:SSL routines:SSL3_GET_CLIENT_HELLO:wrong version number:s3_srvr.c:956:
shutting down SSL
CONNECTION CLOSED

and on the ESP

Error: SSL error 70

Also https://github.com/esp8266/Arduino/issues/43 indicates that the ESP does not (yet?) support TLS1.2 and thus can't be connected using the standard SDK ssl library.

Deadolus can you run it?
You have solved the problem?

[electronicsguy]

@chaeplin Very good point.

When I test it with TLS1.2, using openssl

Code: Select all

openssl s_server -accept 8883 -cert ~/device3/device3_certificate.pem -key ~/device3/device3_private.pem -CAfile ~/VeriSign-Class\ 3-Public-Primary-Certification-Authority-G5.pem -msg -verify 1 -tls1_2


I get

<<< TLS 1.2 Handshake [length 0033], ClientHello
...
>>> TLS 1.1 Alert [length 0002], fatal protocol_version
...
ERROR
140387270956704:error:1408A10B:SSL routines:SSL3_GET_CLIENT_HELLO:wrong version number:s3_srvr.c:956:
shutting down SSL
CONNECTION CLOSED

and on the ESP

Error: SSL error 70

Also https://github.com/esp8266/Arduino/issues/43 indicates that the ESP does not (yet?) support TLS1.2 and thus can't be connected using the standard SDK ssl library.

Deadolus can you run it?
You have solved the problem?

@Deadolus @Tomasz_K I don't know about AWS, but I am able to connect esp8266 to Google Script and Google Calendar services without any problems. I am able to directly write into spreadsheet or fetch calendar events. It uses TLS 1.0/1.1 though. Using Arduino development environment.