Get the changed HTML content after it's updated by Javascript? (htmlunit)

Question

I'm having some trouble figuring out how to get the content of some HTML after javascript has updated it.

Specifically, I'm trying to get the current time from US Naval Observatory Master Clock. It has an h1 element with the ID of USNOclk in which it displays the current time.

When the page first loads, this element is set to display "Loading...", and then javascript kicks in and updates it to the current time via

function showTime()
    {
        document.getElementById('USNOclk').innerHTML="Loading...<br />";
        xmlHttp=GetXmlHttpObject();
        if (xmlHttp==null){
            document.getElementById('USNOclk').innerHTML="Sorry, browser incapatible. <BR />";
            return;
        } 
        refresher = 0;
        startResponse = new Date().getTime();
        var url="http://tycho.usno.navy.mil/cgi-bin/time.pl?n="+ startResponse;
        xmlHttp.onreadystatechange=stateChanged;
        xmlHttp.open("GET",url,true);
        xmlHttp.send(null);
    }  

So, the problem is that I'm not sure how to get the updated time. When I check the element, I see the "Loading..." as the content of the h1 element.

I've double checked that javascript is enabled, and I've tried calling the waitForBackgroundJavaScript function on the webclient as well hoping that it would give the javascript time to start updating stuff. However, no success as of yet.

My Current Code:

import com.gargoylesoftware.htmlunit._
import com.gargoylesoftware.htmlunit.html.HtmlPage

object AtomicTime {

  def main(args: Array[String]): Unit = {
    val url = "http://tycho.usno.navy.mil/what.html"
    val client = new WebClient(BrowserVersion.CHROME)
    
    println(client.isJavaScriptEnabled()) // returns true
    client.waitForBackgroundJavaScript(10000)
//    client.waitForBackgroundJavaScriptStartingBefore(10000) //tried this one too without success
    var response: HtmlPage = client.getPage(url)
    println(response.asText())
  }
}

How do I trigger the javascript to update the HTML?

Answer 1

I figured it out!

HtmlPage objects have an executeJavaScript(String) which can be used to kick off the showTime script. Then, once the script has actually started, that's when waitForBackgroundJavaScript becomes relevant.

The code I ended up with:

import com.gargoylesoftware.htmlunit._
import com.gargoylesoftware.htmlunit.html.HtmlPage
import com.gargoylesoftware.htmlunit.html.DomElement

object AtomicTime {

  def main(args: Array[String]): Unit = {
    val url = "http://tycho.usno.navy.mil/what.html"
    val client = new WebClient(BrowserVersion.CHROME)

    var response: HtmlPage = client.getPage(url)
    response.executeJavaScript("showTime")

    printf("Current AtomicTime: %s", getUpdatedRespose(response, client))
  }

  def getUpdatedRespose(page: HtmlPage, client: WebClient): String = {
    while (page.getElementById("USNOclk").asText() == "Loading...") {
      client.waitForBackgroundJavaScript(200)
    }
    return page.getElementById("USNOclk").asText()
  }
}