simple script to check if a webpage has been updated

bash web scrape
Programmer picture Programmer · Jan 31, 2012 · Viewed 24.8k times · Source

There is some information that I am waiting for on a website. I do not wish to check it every hour. I want a script that will do this for me and notify me if this website has been updated with the keyword that I am looking for.

Answer

Chris Seymour picture Chris Seymour · Jan 31, 2012

Here is a basic bash script for checking if the webpage www.nba.com contains the keyword Basketball. The script will output www.nba.com updated! if the keyword is found, if the keyword isn't found the script waits 10 minutes and checks again.

#!/bin/bash

while [ 1 ];
do
    count=`curl -s "www.nba.com" | grep -c "Basketball"`

    if [ "$count" != "0" ]
    then
       echo "www.nba.com updated!"
       exit 0   
    fi
    sleep 600   
done

We don't want the site or the keyword hard-coded into the script, we can make these arguments with the following changes.

#!/bin/bash

while [ 1 ];
do
    count=`curl -s "$1" | grep -c "$2"`

    if [ "$count" != "0" ]
    then
       echo "$1 updated!"
       exit 0
    fi
    sleep 600
done

Now to run the script we would type ./testscript.sh www.nba.com Basketball. We could change the echo command to have the script send an email or any other preferred way of notification. Note we should check that arguments are valid.

Related questions

How to concatenate string variables in Bash

In PHP, strings are concatenated together as follows: $foo = "Hello"; $foo .= " World"; Here, $foo becomes "Hello World". How is this accomplished in Bash?

Read More
How can I check if a directory exists in a Bash shell script?

What command can be used to check if a directory exists or not, within a Bash shell script?

Read More
How do I tell if a regular file does not exist in Bash?

I've used the following script to see if a file exists: #!/bin/bash FILE=$1 if [ -f $FILE ]; then echo "File $FILE exists." else echo "File $FILE does not exist." fi What's the correct syntax to use if I only want …

Read More