【Feedback】SSL problems

[narayan]

It is not very clear how well tested the SSL library is.
Our product does not use the AT interface.
The call sequence is:

espconn_secure_set_size (1,xxx) - Tried 2k to 5k - 2k is fine for small transfers, but not for large transfer
espconn_secure_cert_req_enable (1,sector) - This is OK
espconn_secure_ca_enable (1, sector) - Always Crashes - causing reboot - I am not using now
espconn_secure_connect ()

Note that for large transfers, we seem to need 5k at least. This causes
espconn_secure_connect () to fail.
If I use less than 5k (4k for example) - SSL fails during data-transfer with invalid client protocol
(The connection succeed, I am able to do a POST - that is received on the other side,
but when the other side sends, everything fails).
When I use 4.5k - The system reboots- with crash in "rc4_skip"
Small transfers are fine.

Please help - I notice that my last SSL posting for support did not get a response.
I note that you released liblwip.a sources. Can you please release the sources for libssl.a.
This will really help.

[narayan]

More info:
If I set the ssl_buffer_size on the server side to 512 bytes, then
I am able to get the transfer to succeed about 50% of the time.
The other half of the time, I get:

client's data invalid protocol
Error: invalid mac

It seems to me like the processing of data is timing dependent, and the SSL
input routines are not managing the TCP windows correctly.

[ESP_Faye]

Hi,

What is the version of your SDK ? Please have a try with the latest ESP8266_NONOS_SDK_V1.5.2.

To call espconn_secure_ca_enable you have to download esp_ca_cert.bin into Flash. You can refer to the documentation 5A-ESP8266__SDK__SSL_User_Manual and the example TLS_BiDirectVerif_Demo.zip and test_cert_and_creat_information.zip.

If your problem is still unsolved, please provide your test code and your esp_ca_cert.bin, we will help debug it.

Thanks for your interest in ESP8266 !

[narayan]

I use version 1.5.2 (including the latest released libssl/libmain patch).
I did follow the example methods in the SSL manual (5A-xxx) and the scripts (server-ca/make_cacert.py)
for creating the esp_ca_cert.bin file.
(Steps: creat PEM, convert to DER. Rename it as TLS_ca_x509.cer, Run make_cacert.py on it)
I put esp_ca_cert.bin file in sector 3, and the device cert in sector 2 (which works very well).
The steps I took are exactly as I specified. I have 2 problems:

1. Device crash every time i call espconn_secure_ca_enable () - I can work past this for now, but needs to be solved
2. To make SSL I/O to work, I changed on the server (remote), the ssl_buffer_size to 512 bytes. Anything else
causes a lot of problems. The failure during the transfer (after about 100k bytes - this varies)

client's data invalid protocol
Error: invalid mac

This happens after 100+kbytes of input data. I am transferring 400k (image). 1 in 2 times it
fails with this message.

Since the TCP-PUSH bit is not treated as a record boundary, the SSL input
is getting more data from espconn input routines. This often happens during re-transmits and
TCP window opening/shrinking. Most likely due to a bug in the input handling
of espconn to SSL, There is extra left over, so when
the next buffer comes from server, you get the above message.

Our code base is very large, I have given the synopsis of the area of concern to you.
I know you have many customers and our problem wont get solved quickly. I really would
very much like to solve my own problem. Please release your libssl.a sources too. I can
then support myself.

I can send you directly the esp_ca_cert.bin - I do not wish to post it in the open forum.

(BTW -Please do not confuse me to be a rookie to networking. I have been
writing RFCs and TCP/IP protocol for 35 years - in fact the first 6 commercial
TCP/IP implementations are mine)

[ESP_Faye]

Hi,

1. Sorry that the libssl is not open source. Here is a simple example of espconn_secure_ca_enable, please download the \ca_enable_test\esp_ca_cert.bin in 0x50000 and have a try.

2. Please provide your test code about the "Error: invalid mac" issue, we will help debug it.

Thanks for your interest in ESP8266 !

[narayan]

Thanks for your input. Parts of my source that deal with connections are below
(See the section that says ===My connection code below=== - Not i do no use
Keep-alive in it)

1. Error with espconn_secure_ca_enable () - I am using it very much like you.
The difference is that both Server and Client need to authenticate each other.
So I use both espconn_secure_ca_enable () and espconn_secure_cert_req_enable
Both the certs (bin files) have been generated per your python scripts and
been loaded into the appropriate sectors. When both are used, the
code crashes.

2. I want to re-iterate that the source below throws away the input from the SSL
(after it process the HTTP header). The problem with:
client's data invalid protocol
Error: invalid mac

occurs after about 100+K of data input (once in 2 times). The server is local and is sending data fast.
I have restricted the server to use 512 byte buffers. Note if I make this 1K byte buffers, then
the error shows up much more quickly.
My observations:
1. You have disabled the TF_DELAYACK in your lwip - Not sure if this is OK
2. Your espconn input routines in espconn_client_recv (liblwip.a) does a os_malloc
and if os_malloc fails (because of TCP streaming full ethernet packets), then the code
is not dealing with it or telling if there is a problem. You should really change the
code to "NO os_malloc" and just call as follows (no copy, noalloc, no free)

struct pbuf *pthis, *pnext;
for (pthis = p; pthis; pthis = pnext) {
pnext = pthis->next;
precv_cb->pespconn->recv_callback (precvcb->pespconn, pthis->payload, pthis->len);
}

===My connection code below====
/*
* Start the TCP connection to the remote host
*/
static void _ICF_
tcpc_start_tcp_conn (struct tcpconn *req)
{
struct espconn *esp = &req->esp;
int rval;
struct httpc_req *hreq = (struct httpc_req *)req;

DPRF (APPM_CONN, ("tcpc:cid=%d,starting con to %s\n",
req->cid, req->is_http ? hreq->url->host : ip_to_str (&req->ip)));
esp->type = ESPCONN_TCP;
esp->state = ESPCONN_NONE;
esp->proto.tcp = &req->tcp;
if (!esp->proto.tcp->local_port)
esp->proto.tcp->local_port = espconn_port ();
esp->proto.tcp->remote_port = req->port;
os_memcpy (&esp->proto.tcp->remote_ip, &req->ip, 4);
espconn_regist_connectcb (esp, tcpc_conn_cb);
espconn_regist_disconcb (esp, tcpc_disconn_cb);
espconn_regist_reconcb (esp, tcpc_conn_err_cb);
espconn_regist_recvcb (esp, tcpc_recv_cb);
espconn_regist_sentcb (esp, tcpc_sent_cb);
/*
* While this is not the best modular stuff,
* Keep Alive is meant for a global basis
*/
if (dev_config.flags & DC_TCPKA_ENA) {
/*
* Enable Keep Alive
*/
uint32_t val;
espconn_set_opt (esp, ESPCONN_KEEPALIVE);
val = dev_config.tcpka_init_timo;
espconn_set_keepalive (esp, ESPCONN_KEEPIDLE, &val);
val = dev_config.tcpka_retry_timo;
espconn_set_keepalive (esp, ESPCONN_KEEPINTVL, &val);
val = dev_config.tcpka_retry_count;
espconn_set_keepalive (esp, ESPCONN_KEEPCNT, &val);
}
DPRF (APPM_CONN, ("tcpc:cid=%d,start %s conn to %s:%d\n",
req->cid,
(req->is_ssl ? "SSL" : "TCP"),
ip_to_str (&req->ip),
req->port));
req->state = HTS_CONNECTING;
/*
* get certs for HTTPS
*/
if (req->is_ssl) {
/* TODO - talk to Espressif about this */
/*
* The following espconn_secure_set_size () is
* broken - It seems to swallow buffer-space
* Taking the default value of 2K seems to work for now
* for small transfers. Large transfer (such
* as OTA - big F%#k-up)
*/
#if 0
if (!espconn_secure_set_size (1, 8*512/*dev_config.ssl_buf_sz*/)) {
DPRF (APPM_ERR, (CLR_RED "*ERR*SSL set-size\n" CLR_OFF));
} else {
//OS_PRINTF ("SSL:buf-sz=%d\n", dev_config.ssl_buf_sz);
}
#endif
if (req->use_dev_cert) {
OS_PRINTF ("CRT:sect %d\n", SFLASH_DEV_CRT_BLK);
if (!espconn_secure_cert_req_enable (ESPCONN_CLIENT, SFLASH_DEV_CRT_BLK)) {
OS_PRINTF ("*ERR*sec_cert_req\n");
}
/* This seems broken - Narayan talk to ESP
* about this.
*/
//OS_PRINTF (CLR_GRAY "CA:sect %d\n" CLR_OFF, SFLASH_CA_CRT_BLK);
//if (!espconn_secure_ca_enable (ESPCONN_CLIENT, SFLASH_CA_CRT_BLK)) {
// OS_PRINTF ("*ERR*sec_ca_req\n");
//}
}
rval = espconn_secure_connect (esp);
} else {
rval = espconn_connect (esp);
}
if (rval != 0) {
DPRF (APPM_ERR, ("*ERR*conn:rval=%d\n", rval));
req->state = HTS_FAIL;
req->err = AERR_CONN;
app_evt_post (AMI_TCPCONN, TCPCONN_SIG_TCP_FAIL, req);
}
/* Completion will happen in one of the call-backs */
}

[ESP_Faye]

Hi,

Here is a simple example of both espconn_secure_ca_enable and espconn_secure_cert_req_enable are calling.
Download \bi_auth_test\esp_ca_cert.bin in 0x50000, \bi_auth_test\esp_cert_private_key.bin in 0x51000 and have a try.

Please notice that SSL needs a lot of memory, maybe your crash is due to the lack of memory, you can call system_get_free_heap_size to get the available size of heap.

We need more information about "Error: invalid mac" issue, please send your esp_ca_cert.bin, esp_cert_private_key.bin and the IP, port of your SSL server to feedback@espressif.com. If you can provide a simple test project to duplicate your problem, it will be even better.

Thanks for your interest in ESP8266 !

[wahaha]

I have an another question like this.
First , i don't know how to set the address which won't cover the system codes.
Second , i call espconn_secure_ca_enable(0x01,0x3B) and espconn_secure_cert_req_enable(0x01,0x3A) , and both of them are success , but the debug log dose not print anything about ssl . I can't understand.
Please help me!

[wahaha]

I have an another question like this.
First , i don't know how to set the address which won't cover the system codes.
Second , i call espconn_secure_ca_enable(0x01,0x3B) and espconn_secure_cert_req_enable(0x01,0x3A) , and both of them are success , but the debug log dose not print anything about ssl . I can't understand.
Please help me!

[wahaha]

I have an another question like this.
First , i don't know how to set the address which won't cover the system codes.
Second , i call espconn_secure_ca_enable(0x01,0x3B) and espconn_secure_cert_req_enable(0x01,0x3A) , and both of them are success , but the debug log dose not print anything about ssl . I can't understand.
Please help me!