Lab 01b

The questions below are due on Tuesday February 06, 2018; 08:25:00 PM.


Partners: You have not yet been assigned a partner for this lab.
You are not logged in.

If you are a current student, please Log In for full access to the web site.
Note that this link will take you to an external site (https://oidc.mit.edu) to authenticate, and then you will be redirected back to this page.

Music for this Lab

 

Goals:

In this lab we'll get more practice with state machines, as well as start to perform rudimentary HTTP GET requests, as an initial foray into the "I" part of IOT. We'll then merge these two topics to implement a basic number fact reporter. In particular we'll:
  • First, explore pulling some information from the web using HTTP GET requests with our ESP32 and a bare-bones public API
  • Build a more complicated state-machine to allow us to request information from an API based on user input.
  • There's no need to wire up new hardware in this lab so you can keep your wire kits in your bag, unless you like how they look.

Unless otherwise stated, all checkoffs in this lab are due. Also we expect groups to work together on labs so we during the checkoff we expect to engage with both partners in discussion.

1) The GET Method

Download and extract the code found in lab01b.zip. Open the file up, compile it, and then upload it to your ESP32 (the code as-written will only work in the lab on the 6S08 access point. If you are doing this lab elsewhere, you'll likely need to change some parameters, which we'll learn about below). Your OLED should flash some text, and if you monitor its output on the Serial monitor you should see the a bunch of text flying by and after a little bit, interspersed factoids about numbers. Your little microcontroller is talking to the web!

1.1) HTTP

At the core of any communication protocol is an agreed-upon syntax which enables both parties to effectively convey information. This can be generalized up to things like human language, and obsessed over at the level of individual bits, and EECS as a field covers all of that. The internet is no exception, with a vast array of entities using standardized protocols. Initially those entities were supposed to be just standard "computers" (in the 1980s sense), but in recent years as computation has become more ubiquitous, the limits (size, cost, availability) of what exactly is needed to join that system have been greatly relaxed. This is really where the term "IOT" comes from...it is a new period in the history of the internet where arbitrary "things" can be on the internet in addition to your Dell Latitude laptop.

What exactly comprises a "web-communication"? Well, a lot of web communication is built on top of the Hypertext Transfer Protocol (HTTP) which is itself generally built on top of another protocol (Transmission Control Protocol (TCP)) which is itself built on other "layers" (take 6.033 if this stuff interests you...it should since it is pretty cool).

HTTP is far from the only Application Layer that is available. Things like SSH, FTP, IMAP, and MQTT are all other acronyms you might encounter in your day-to-day millenial lives, which refer to particular communication protocols that have other intentions. We'll talk a bit about a select few of these throughout the semester.

1.2) The HTTP GET

One of the simplest things you can do over HTTP is a "GET". We actually already know how to perform an HTTP GET using a web browser. You do that simply by visiting a website, believe it or not. But what actually happens behind the scenes?

First the client (your laptop or phone) must establish a connection to the server of interest (the host). The host can either be an IP address or a domain name like google.com. When we connect we are a client of the service.

If the connection is successful (and a lot goes on behind the scenes to ensure that this happens), we next need to send the actual HTTP GET request. When you visit a site in a browser, this is almost always a GET request behind the scenes and actually involves sending a text message that looks similar to the following:.

GET /resource HTTP/1.1
Host: server

Line-by-line this corresponds to:

GET /resource HTTP/1.1

which is the HTTP operation that we're doing and at what Uniform Resource Identifier (of which URL, Uniform Resource Locator, is a subtype) on the server we're interested in. We also specify which version of the HTTP protocol we want (just use 1.1).

The next line:

Host: server

lists the server we're talking with/interested in.

Finally we must always end a GET request with a single blank line which we accomplish like so


We should make a note on two special characters at this point that have proven invisible so far: \r and \n. Both of these characters should be thought of as individual characters even though we express them using two typed-characters. They are special since they're usually rendered as invisible when we show a string featuring them but you can think of them the same as the character "a" or "T" since they still influence how things get rendered. Those two symbols are:

  • \r Originally called a "carriage return", some OS's use it to indicate the end of a line.
  • \n Often called a "new-line" or "line-feed", many other OS's use this as a standard end of line/new line indicator (*Nix, Mac). In C and Python, if you want to include a new line in a string you are creating, this is also how you do it

On the internet (for various historical reasons), the way to terminate a line is the two character sequence: \r\n. That means the above GET request can be (and is) expressed as a single sequence of characters in the following way:

GET /resource HTTP/1.1\r\nHost: server\r\n\r\n

We will come back to that a bit later in the lab when we start doing this on the ESP32.

1.3) Specifying Resources

Now when you go into a browser like Chrome and visit google.com, what your browser is really doing behind the scenes is establishing a connection with the google.com host, and then sending a GET request of the following:

GET  HTTP/1.1
Host: google.com

Note there is nothing after the GET in this case since we are just going to the root of the host (google.com)

If we wanted to visit a more specific URI within the google.com, for example Google's "about" page (yes they have one) in the browser field you'd type in: https://google.com/about/ while behind the scenes it is performing the following GET:

GET /about/ HTTP/1.1
Host: google.com

We'll get tons of experience with this, but what's really neat is if we wanted to perform a search (even though we usually only use GUI buttons and our mouse to do it for us) we can in the browser search for "cat" by manually typing: https://google.com/search?q=cat, and when you press enter you'll show up on Google's search results page for 'cat'. Behind the scenes that looked like the following:

GET /search?q=cat HTTP/1.1
Host: google.com

The ? part indicates a query and the q=cat is the key-value pair of the argument we provide.

2) Doing this on the ESP32

The ESP32 lacks a browser, but once we realize that a web browser is really just sugar coating on top of the client/user side of a robust method of requesting and receiving information over HTTP, we can start to figure out how to do this on our microcontroller. To do this on our ESP32 there's a few ways. First and foremost we need to use Wifi to establish a connection to the greater network in the world (the so-called "world-wide-web" everyone's been talking about lately). We'll do that by including a library that allows the ESP32 to connect to a WiFi network. Rather appropriately that library is called WiFi. We include that in our code with the following syntax:

#include <WiFi.h>

Next when the code first runs (therefore in the setup function) we need to provide credentials to connect to our network of choice (just the same as when you connect your laptop). The example below will connect to the class router in the 6.08 room (yes there is an 's' in the SSID and password).

WiFi.begin("6s08","iesc6s08");

As an aside if you want to connect to an open network, you can leave the second argument off (the password). So for example, a common open network in Building 38/36 is EECS-MTL-RLE, so you could join it by doing WiFi.begin("EECS-MTL-RLE");.

Be careful joining open networks on MIT's campus since often times there are many access points labeled "MIT" or "EECS-MTL-RLE" and if you tell the ESP32 to just join one of them it may pick the weakest one and have a repeatedly hard time connecting. There are ways around this by using the unique identifying numbers of each access point (the MAC address), but we'll not go into that this week.

Next, (also in setup) we'll need to pause the code until we establish a connection. This can be done in tons of ways, but a short convenient one that gives a bit of debugging information is shown below - while we are NOT connected, wait for about 3 seconds or until we get connected (whichever comes first).

int count = 0;
while (WiFi.status() != WL_CONNECTED && count<6) {
  delay(500);
  Serial.print(".");
  count++;
}

Then afterwards, let's check to see if we are connected. If we are, print out over serial the credentials assigned to our device by the network for our own records, but if not, restart the device (ESP.restart(), which will proceed to run the setup function again, and try to connect again. (Note for right now, if you try to connect to a non-existent AP, the system will just repeatedly run this try/fail/restart loop, so finding the strongest WiFi network is good here.) The piece of code that implements this connection check, credential printing, and restart looks like:

if (WiFi.isConnected()) {
  Serial.println(WiFi.localIP().toString() + " (" + WiFi.macAddress() + ") (" + WiFi.SSID() + ")");
} else {
  Serial.println(WiFi.status());
  ESP.restart(); // just restart and try again
}

Once we're through that we are connected to a network and into the loop function.

In order to connect to a host, we must create a client object. A client as we've mentioned earlier is the name for the entity connecting to a host (a server). In most of our daily techno-consumerist existence we are clients, and entities like Google and Facebook are the hosts.

WiFiClient client;

We then instruct the client to try and connect to the host we're interested in. The first value can either be an IP address or a domain name (like google.com), and the second argument is a port number. For today and at least a little while into the future, we'll be using port 80 which is a particular communication channel reserved for HTTP traffic. (Later we'll use port 443 for HTTPS).

For today's lab we're going to connect to a host called numbersapi.com which provides a free Application Programming Interface (API) for interesting information about numbers. Yes that might seem silly, but you are at MIT, and trivia about numbers should appeal to you.

So to connect:

client.connect("numbersapi.com", 80)

This member function connect will return a true or false. You can use this to do some error handling if you'd like or try to reconnect. Assuming it does connect, however, it means a channel of communication exists from client to host. In order to perform a GET request we need to simply use the member function print and/or println which work very similarly to the Serial.print and Serial.println except that instead of printing to the Serial monitor they are sent over the socket connecting our client to the host in interest. Specifically print prints only the string provided while println appends a \r\n to the line. A three-line example of sending a GET request is:

client.println("GET http://numbersapi.com/51/trivia HTTP/1.1");
client.println("Host: numbersapi.com");
client.print("\r\n");

which sends up the GET request to the host of the following form (which should look very familiar to what we saw earlier):

GET http://numbersapi.com/51/trivia HTTP/1.1
Host: numbersapi.com

which when written in one line is:

GET http://numbersapi.com/51/trivia HTTP/1.1\r\nHost: numbersapi.com\r\n\r\n

Try Now:

Make sure you know where each \r\n pair came from in the code above.

Note also that this HTTP GET is equivalent and redundant with (sometimes you'll see the host specified in the URI and other times you won't):

GET /51/trivia HTTP/1.1
Host: numbersapi.com

The host (numbersapi.com) receives this request and parses it. In future weeks we'll work on performing the type of operations that go on behind the scenes there, but let's not worry about that now. Instead, let's treat it as a black box that takes in an input and generates an response.

An example response that will come back from the server looks like the following:

HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Sun, 04 Feb 2018 14:48:06 GMT
Content-Type: text/plain; charset="UTF-8"; charset=utf-8
Content-Length: 36
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With
X-Numbers-API-Number: 51
X-Numbers-API-Type: trivia
Pragma: no-cache
Cache-Control: no-cache
Expires: 0

51 is the atomic number of antimony.

Now a lot of the information present in this response from the host is usually hidden from us in our web browsing experience. The first line for example is the HTTP response status code. 200 means essentially "all good". We'll rarely, if ever, see a 200 in browser, because if we get that it means things have been returned correctly and the browser instead just returns the content. More often we'll see something like a 404 appear which means that while the client could talk to the server/host, the host was unable to locate the appropriate resource. There's a whole list of HTTP status codes which can be used to dictate how to behave in regards to an HTTP response. We won't worry too much about that here and instead just focus on the response body.

How do we determine the response body?

The end of the header is indicated by an extra blank line in the response. If we look for a line that is nothing more than a \r\n, we'll be able to know where our actual content is. We can collect and parse the host response using a set of client member functions:

  • client.connected(): returns true if the client is still connected to the host (server). This will be true when data is being transferred to the host and back from the host as well as sometimes afterwards as well. Otherwise returns false.
  • client.available(): returns true if there are characters to read from the data buffer of the client object, otherwise false.
  • client.read(): Reads out one character from the data buffer of the client object.
  • client.readStringUntil('\n'): Reads out characters from the data buffer of the client object until a \n is encountered. This is convenient since individual pieces of information are usually separated by known characters, in this case we can use the newline character "\n".

When placed all together the following bit of code will wait for up to six seconds while the client is connected and at first print out over Serial (visible in the Serial monitor) incoming lines of data until it encounters a line containing only a "blank" (a \r). After that it will build up a String object that it will use to contain the body of the response.

while (client.connected()) {
  String line = client.readStringUntil('\n');
  Serial.print(line);
  if (line == "\r") { //found a line containing only a \r\n
    //header must be done...move on to collect content!
    break;
  }
  if (millis()-count>6000) break; //timeout
}
String op; //create new empty string
while (client.available()) { //while there are still bytes to read...
  op+=(char)client.read(); //concatenate String object op with incoming letters
}
Serial.println(op); //print data overall!

In review, the transfer of data during a simple HTTP GET request is illustrated below:

client and host messages

Illustrated example of HTTP communication between client (in this case an ESP32) and the host (in this case a server called numbersapi.com) where we perform an HTTP GET request for trivia about the number about the numer 31 and the host responds back with trivia about the number 31.

Checkoff 1:
Describe the HTTP response listening code. Briefly verify and discuss with a staff member the code from above. What is the total response? What is the header? What is the body? How are they separated? In the code running on your micrcontroller, where do the numbers it is requesting come from?

3) Numbers of Interest

During the upcoming design experience in this lab, the following concepts might prove helpful for this assignment:

3.1) Timeouts

Sometimes we only want to do something for a little while. Maybe that means check an input, or Wifi connectivity, or something else. This usually means we'll run the task we want with a timeout that corresponds to some check for the amount of time that has passed since a given starting point. In C++ on our microcontroller we can implement a timeout as shown below:

const int timeout = 400;
unsigned long timeo = millis(); //
while (millis()-timeo < timeout){
  //do things in here while waiting (e.g. check inputs, etc...)
  //you may also just do nothing depending on the situation
}
The millis() function is an internal counter in the microcontroller and is useful for timing events. The code above will do an action for timeout milliseconds, assuming that the time to execute the code within the while loop is short (<<timeout), which it usually is. Another way to implement a timeout which can work better in the context of our loop function (since it allows loop to keep running) is:
const int timeout = 400;
unsigned long timeo = millis(); //

void setup(){
  timeo = millis();
}
void loop(){
  if (millis()-timeo < timeout){
    //do things in here while waiting (e.g. check inputs, etc...)
    //you may also just do nothing depending on the situation
  }else{
    //do other things after time is up
  }
}

3.2) Switch Statements

Sometimes as you deal with lots of situations or things to check (such as when you get more states in your state machine) it will get annoying to do nested if-else-if things. Instead in C/C++ you can use a switch statement:
int var1;
//var1 gets assigned to be something.

switch(var1){
  case 2: 
    //stuff you'd like to do when var1==2;
    break;
  case 3:
    //stuff you'd like to do when var1==3;
    break;
  default:
    //if individual values of var1 are not covered do this
    break;
}
When written like shown, a switch statement can provide a slightly more readable way to perform a one-of-many decision in your code.
Do not forget the break;! in your switch statement's individual cases! Failure to do so will result in fall-through where the following cases will be checked as well.

3.3) Definitions

Finally, sometimes as we start to introduce more and more states it gets annoying to have to remember what each state's number means. We can instead start to use variables or definitions for this (these two things have very different implications). One common convention is to use a definition:

#define IDLE 0
#define PUSHONE 1
#define RELEASE 2

Then in usage you can do something like the following:

//state starts as some value!
loop(){
  switch(state){
    case IDLE:
      //do your idle state stuff
      break;
    case PUSHONE:
      //do you stuff while waiting for state
      break;
    case RELEASE:
      //do stuff here
      break;
  }
}

During code compilation, the compiler will replace each of the keywords (IDLE, etc.) with the relevant number (0, etc.). Using DEFINE statements has the benefit of making analyzing your flow-control (if-else, switch-statments, etc.) much more readable, i.e. you won't be wondering what "4" means anymore if you give it a nice name.

3.4) The Design

Ok, so we've got a button, an OLED, and a really powerful microcontroller with internet access. We've also seen that there's a simple API we can talk to where we can give it a number and receive trivia about that number. Right now those numbers are randomly generated. Your assignment for this lab is to build a number-fact looker-upper. The number requested will be based on the number of button presses a user provides as input in quick succession. To be more specific:

  1. Your system will start at rest.
  2. Upon pressing and releasing your button, the system will wait for up to one second for another press and release. If the press-release sequence occurs, the system will restart its timeout and wait a second for another press-release, all the while tallying the number of times the button has been pressed and then released.
  3. Following at least one press-release sequence, if the button remains unpressed for more than one second, the system is to look up trivia on the number of presses that occured and display it on the OLED, before returning to rest.

A high-definition video of the system working is shown below. This is the functionality we're looking for:

Design your system as a state machine (draw out a diagram)

As a first step in your design, draw your finite state machine diagram, sketch some pseudo-code1, and if needed discuss with a staff member how you're implementing it. Not sure where to start? Consider packaging the entire WiFi request into one large function so that you can keep the number of lines in your primary loop function to a minimum for the sake of clarity and readability! Please ask for help if you get confused!

There is no starting-code skeleton. Create a new file and move over portions of code from the WiFiGetter.ino file and/or any code from Lab 01A that might prove helpful!

When ready, ask for the checkoff below to discuss your design prior to implementation:

Checkoff 2:
Show your state machine diagram and design idea to a staff member and discuss your design.

Finally, when ready, implement your system in code and then show it working. Clip the returned text so that only the first ~20 characters get printed to the OLED (failing to do that will result in garbled text when very long prints end up wrapping around the screen.)

Since this will be the first debugging on an embedded system for many of you, here are a few tips to get started:

  • When you click the right arrow to compile and upload your code, you may end up with an error. The error messages will show up in the bottom part of the Arduino IDE window (the black region). Error messages will show up red (same color as the messages telling you the code was succesfully sent to the ESP). What you'll want to do is to increase the size of the black bottom region, and if you get an error, scroll up to the first error you see. That's the one to fix. Then try again to recompile, until no more errors show up and success is upon you.

  • When debugging, you'll often want to print out the values of certain variables at different points of execution. Serial.println and Serial.print are great ways to do this. You can also print to the OLED, though that takes a tiny bit more work.

Checkoff 3:
Show your working system to a staff member.


 
Footnotes

1 By pseudo-code we mean don't obsess over syntactic correctness but get the general idea and flow down. (click to return to text)