Arduino ESP8266 Softwareserial not enough buffer size for HTTP get request

Question

I am working on a project in which I use arduino uno and ESP8266ex as wifi module on it. The connections of wires:

Arduino 5V --> 3.3V Regulator --> ESP: CH_PD (with 10k resistor) and VCC Arduino GND --> 3.3V Regulator --> ESP: GND and RST (reset is connected via a button and resistance) Arduino RX --> ESP TX Arduino TX --> Voltage divider (2k 1k resistors) --> ESP RX 5uF Capacitor --> Voltage regulator to prevent ESP to reset itself.

Now let me explain the problem I am having. I have two codes working on which I use ESP8266 as wifi module for arduino uno. In my first program I sent commands by hand:

#define ard_rx_esp_tx 2
#define ard_tx_esp_rx 3

#include <SoftwareSerial.h>

SoftwareSerial ESPserial(ard_rx_esp_tx, ard_tx_esp_rx); // RX | TX

void setup()
{
  int i = 0;
  Serial.begin(9600);     // communication with the host computer
  while (!Serial);

  // Start the software serial for communication with the ESP8266
  ESPserial.begin(9600);
  Serial.println("");
  Serial.println(F("Remember to to set Both NL & CR in the serial monitor."));
  Serial.println(F("Ready"));
  Serial.println(F(""));
  Serial.println(F("start"));
  delay(1000);
}
void loop()
{

  if ( ESPserial.available() )   {
    char c = ESPserial.read();
    Serial.print(c);
  }

  if ( Serial.available() )    {
    ESPserial.write( Serial.read() );
  }
}

I succeeded to open TCP connection with a server, to send a GET request which has a long (over 600 chars) and handle all the long response via SoftwareSerial read() function and print them all to the serial monitor. In short, this code can handle 600+ char response of a server which is:

The aim is to send these AT commands via "SoftwareSerial.print()" and put the whole response in a character array to parse its API-KEY. The code I have written for this so far:

#define ard_rx_esp_tx 2
#define ard_tx_esp_rx 3
char response[625];
#include <SoftwareSerial.h>
SoftwareSerial ESPserial(ard_rx_esp_tx, ard_tx_esp_rx); // RX | TX
int i;

void setup() 
{

    Serial.begin(9600);     // communication with the host computer
    while (!Serial);

    // Start the software serial for communication with the ESP8266
    ESPserial.begin(9600);  

    Serial.println("");
    Serial.println(F("Remember to to set Both NL & CR in the serial monitor."));
    Serial.println(F("Ready"));
    Serial.println(F(""));
    Serial.println(F("start"));
    delay(1000);

    ESPserial.println("AT+CIPSTART=\"TCP\",\"domainname\",80");
    delay(5000);

    i = 0;
    while ( ESPserial.available() ) {
      response[i] = ESPserial.read();
      i++;
    }
    response[i++] = '\0';
    Serial.println(response);
    for (i = 0; i < 625; i++) {
      response[i] = '\0';
    }

    ESPserial.println("AT+CIPSEND=107");
    delay(5000);

    i = 0;
    while ( ESPserial.available() ) {
      response[i] = ESPserial.read();
      i++;
    }
    response[i++] = '\0';
    Serial.println(response);
    for (i = 0; i < 625; i++) {
      response[i] = '\0';
    }

    ESPserial.println("GET request to the server which has length 107 as indicated");
    delay(5000);

    i = 0;
    while ( ESPserial.available() ) {
      response[i] = ESPserial.read();
      i++;
    }
    response[i++] = '\0';
    Serial.println(response);
    for (i = 0; i < 625; i++) {
      response[i] = '\0';
    }
}
void loop() {
  // put your main code here, to run repeatedly:

}

Which prints the response before at the end of the "setup()" scope. Let me also put the photo of the output:

In conclusion the problem is that: SoftwareSerial has 64 byte buffer which can be increased up to 256 bytes and when I increase it the program is able to print 256 chars this time, however, in my first code, in which I sent AT commands manually, it can handle whole response in spite of its 64 bytes buffer and is able to print it to the serial monitor. In the second one I could not handle and store the whole response into a char array.

I hope I could explain my problem and pointed out at where exactly I am on my process with details.

What do you suggest me to do. What can I do while handling this big response and putting it into a character array? How can I handle whole response which stays on ESP8266ex' buffer initially and read by Arduino RX pin via SoftwareSerial class in which the function read() has 64 bytes array and can be increased up to 256 but no more?

Answer 1

So, the problem here is all about timing. You know that you have a limit on your buffer size for the software serial (Which is also true for any hardware UART, too), which is 256 bytes, at a baud rate of 9600 bits per second.

Since there is a start bit, 8 data bits, and a stop bit (Assuming you use 9600 8N1 here as it's the most common), you will be receiving a byte of data every (1 / 9600) * 10 - seconds, or 1.04 milliseconds. Therefore, to receive 256 bytes, it should take about 266 milliseconds. This means after 266 milliseconds, your buffer will be completely full, and anything you receive afterwards will start removing previously received data.

And there in lies the crux of your problem - You are sending a command to the ESP to receive data from a server, and then going to sleep for 5 seconds, meaning nothing is pulling data from the buffer, so it wraps around, leading to data loss. The point of a serial buffer isn't to hold the entire dataset you will receive at a single point, but, to hold it just long enough until you can read it out, which is why they're generally quite small.

What you need to do is send the command, and have your Arduino immediately run code to retrieve data as fast as it can from the buffer until it finds an expected end or times out.

Something basic like this would get you going:

char espBuffer[1024] = {0};
int readCount = 0;
long startTime = millis();

ESPserial.println("AT+CIPSTART=\"TCP\",\"domainname\",80");

while (millis() - startTime < 5000) { // Run for at least 5 seconds 
  // Check to make sure we don't exceed espBuffer's boundaries
  if (ESPserial.available() > readCount + sizeof espBuffer - 1) 
    break;
  readCount += ESPserial.readBytes(espBuffer + readCount, ESPserial.available());
}

Serial.println(espBuffer);

Now, you'd want to modify this code to end when it has received all the data it expects. Additionally, this simple setup limits you to a maximum size of 1023 bytes for the response, which isn't too helpful either. Ideally you'd keep reading until you find, say, the HTTP body, and discard everything else, meaning the buffer to find the data is small, and the buffer to actually store the body can be much larger.