Compute average and standard deviation with awk

awk standard-deviation
PLM picture PLM · Sep 13, 2013 · Viewed 32.8k times · Source

I have a 'file.dat' with 24 (rows) x 16 (columns) data.

I have already tested the following awk script that computes de average of each column. 

touch aver-std.dat
awk '{   for (i=1; i<=NF; i++) { sum[i]+= $i } }
END { for (i=1; i<=NF; i++ )  
{ printf "%f \n", sum[i]/NR} }' file.dat >> aver-std.dat

The output 'aver-std.dat' has one column with these averages.

Similarly as the average computation I would like to compute the standard deviation of each column of the data file 'file.dat' and write it in a second column of the output file. Namely I would like an output file with the average in the first column and the standard deviation in the second column.

I have been making different tests, like this one

touch aver-std.dat
awk '{   for (i=1; i<=NF; i++) { sum[i]+= $i }}
END { for (i=1; i<=NF; i++ )  
{std[i] += ($i - sum[i])^2 ; printf "%f %f \n", sum[i]/NR, sqrt(std[i]/(NR-1))}}' file.dat >> aver-std.dat

and it writes values in the second column but they are not the correct value of the standard deviation. The computation of the deviation is not right somehow. I would appreciate very much any help. Regards

Answer

Hari Menon picture Hari Menon · Sep 13, 2013

Standard deviation is 

stdev = sqrt((1/N)*(sum of (value - mean)^2))

But there is another form of the formula which does not require you to know the mean beforehand. It is:

stdev = sqrt((1/N)*((sum of squares) - (((sum)^2)/N)))

(A quick web search for "sum of squares" formula for standard deviation will give you the derivation if you are interested)

To use this formula, you need to keep track of both the sum and the sum of squares of the values. So your awk script will change to:

    awk '{for(i=1;i<=NF;i++) {sum[i] += $i; sumsq[i] += ($i)^2}} 
          END {for (i=1;i<=NF;i++) {
          printf "%f %f \n", sum[i]/NR, sqrt((sumsq[i]-sum[i]^2/NR)/NR)}
         }' file.dat >> aver-std.dat

Related questions

How can I use ":" as an AWK field separator?

Given the following command, echo "1: " | awk '/1/ -F ":" {print $1}' why does AWK output: 1: ?

Read More
How to do a recursive find/replace of a string with awk or sed?

How do I find and replace every occurrence of: subdomainA.example.com with subdomainB.example.com in every text file under the /home/www/ directory tree recursively?

Read More
How do I use shell variables in an awk script?

I found some ways to pass external shell variables to an awk script, but I'm confused about ' and ". First, I tried with a shell script: $ v=123test $ echo $v 123test $ echo "$v" 123test Then tried awk: $ awk 'BEGIN{print "…

Read More