ESP8266 Developer Zone The Official ESP8266 Forum 2016-10-21T04:06:35+08:00 https://bbs.espressif.com:443/feed.php?f=16&t=2900 2016-10-21T04:06:35+08:00 2016-10-21T04:06:35+08:00 https://bbs.espressif.com:443/viewtopic.php?t=2900&p=10219#p10219 <![CDATA[TCP AT+CIPSEND and +IPD:length inconsitencies?]]>
I'm sending the AT requests with an FTDI adapter from a little Java program doing serial communication. My goal is to use an Arduino Mega to send these requests to wunderground to get JSON weather data back. Initially I used this library https://github.com/itead/ITEADLIB_Ardui ... ee/develop to do the requests. But this has failed so far, so I'm taking the Mega out of the picture and working with the mentioned Java serial port app.

In my new tests I try getting the HTML from api.wunderground.com and http://www.google.com. The WeatherUndeground does send inconsistent +IPD package length data and the Google response is correct. I also tried http://www.bing.com which has a mix of correct and incorrect packages. Everything is done with the same Java Serial port client.

I have a hunch that this is related to the CR/LFs, but even if I ignore either CR or LFs, the numbers still don't match up. Maybe the handling is different in the HTTP headers and body? Does the ESP8266 look at the content?

Wunderground request:

Code:

AT+CIPSTART="TCP","api.wunderground.com",80

CONNECT

OK
AT+CIPSEND=93


OK
> GET / HTTP/1.1
Host: api.wunderground.com
Accept-Charset: ISO-8859-1
Connection: close


Wunderground response:

Code:

+IPD,1460:HTTP/1.1 200 OK
Server: Apache/2.2.15 (CentOS)
Access-Control-Allow-Origin: http://www.wunderground.com
Access-Control-Allow-Credentials: true
X-CreationTime: 0.320
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=UTF-8
Expires: Thu, 20 Oct 2016 19:23:05 GMT
Cache-Control: max-age=0, no-cache
Pragma: no-cache
Date: Thu, 20 Oct 2016 19:23:05 GMT
Transfer-Encoding:  chunked
Connection: close
Connection: Transfer-Encoding
Set-Cookie: DT=1476991384:21984:365-e6; path=/; expires=Fri, 01-Jan-2020 00:00:00 GMT; domain=.wunderground.com
Set-Cookie: Prefs=FAVS:1|WXSN:1|PWSOBS:1|WPHO:1|PHOT:1|RADC:0|RADALL:0|HIST0:NULL|GIFT:1|PHOTOTHUMBS:50|EXPFCT:1|; path=/; expires=Fri, 01-Jan-2020 00:00:00 GMT; domain=.wunderground.com

00006000

<!DOCTYPE html>
<!--[if IE 9]><html class="no-js ie9"> <![endif]-->
<!--[if gt IE 9]><!--> <html class="no-js "> <!--<![endif]-->
   <head>
      <title>Weather Forecast & Reports - Long Range & Local | Wunderground | Weather Underground</title>
      <link href="//icons.wxug.com/" rel="dns-prefetch" />
      <link href="//api-ak.wunderground.com/" rel="dns-prefetch" />
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
   <meta name="viewport" content="width=device-width, initial-scale=1">
      <meta name="description" content="Weather Underground provides local & long range Weather Forecast, weather reports, maps & tropical weather conditions for locations worldwide."
+IPD,1460: />
      <meta name="keywords" content="Weather, Weather Underground, Weather Forecast & Reports - Long Range & Local | Wunderground, forecasts, current conditions, rain, snow, hurricane, tornado, storm, thunderstorm, tropical storm, wundermap, ski conditions, ski report, surfing, marine, aviation, photographs, photos, severe weather" />
   <meta property="og:image" content="https://icons.wxug.com/i/wu/HomePageRedesign/wunderground_new_homepage.png" />
[.. SNIP ... there's more text]   


The number of bytes from after the first IPD package to the next "+IPD" string is 1471 instead of 1460. I measure that by copying from the console to Notepad++, save it and look at the file size. (Note: It looks like the posted content is different and you have a different data size if you check the posted content). Looking at it in a HEX there's CR/LF at all lines. There are 30 lines in the first package, so that doesn't relate to the difference of 11 bytes.

Google request:

Code:

AT+CIPSTART="TCP","www.google.com",80

CONNECT

OK
AT+CIPSEND=59


OK
> GET / HTTP/1.1
Host: www.google.com
Connection: close



Recv 59 bytes

SEND OK


Google response:

Code:

+IPD,1430:HTTP/1.1 200 OK
Date: Thu, 20 Oct 2016 19:46:47 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=89=RrW-P3fZ_lafKL_coJlY0CK7EBV4R8CEAxJSbV2yylEsX1yN75g0FghqlaqCoDQ3c5pdoOeNPLKlV5mglCMvyRsXcG8TEBZfRPJhSGIVwR5W-BvRQ9pXMCHJn8qk5qSqDi_toyp908lSKyY; expires=Fri, 21-Apr-2017 19:46:47 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Connection: close

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>Google</title><script>(function(){window.google={kEI:'Jx8JWKSOK4uf0gKLgIfYDw',kEXPI:'1351634,1351828,1351901,3700311,3700400,4028875,4029815,4031109,4032678,4036527,4038012,4039268,4041899,4043492,4045841,4048347,4052304,4055745,4065787,4067859,4067929,4068550,4069838,4069841,4070327,4071230,4072289
+IPD,1430:,4072364,4072602,4072705,
[... SNIP .. there's more text.. ]


All the IPD lengths seem to be correct for the google response.

Can anybody reproduce this? How am I supposed to parse the response? Do certain bytes need to be ignored in certain places.

Thanks in advance!

Statistics: Posted by dirkw — Fri Oct 21, 2016 4:06 am

]]>