Poor Man’s Network Traffic Meter

Set out tonight to find a way to log “network traffic” through the interfaces on my solaris box. What I was wanting was the actually amount of traffic going through the interfaces. First thought was to use netstat. But that only shows “packets” and the packets could be differing sizes. So I ended up using kstat. I wrote this simple little script to grab the interface names, and then use kstat to get the data out of the network module for each card:

#!/bin/ksh
#Get list of Ethernet Cards in machine:
MyHOST="`hostname`"
OS="`uname -r`"
if [ ${OS} == "5.10" ] ; then
   MyETHERS="`/usr/sbin/dladm show-dev | awk '{print $1}'`"
else
   MyETHERS="`/usr/sbin/ifconfig -a | awk '{print $1}' | grep \":\" | awk -F':' '{print $1}' | sort -u | grep -v \"^lo0\"`"
fi
COUNT=0
while [ $COUNT -lt 800 ]; 
  do
  for i in `echo $MyETHERS`
  do
    OBYTES="`/usr/bin/kstat -p -c net -n $i -s obytes64 | awk '{print $2}'`"
    RBYTES="`/usr/bin/kstat -p -c net -n $i -s rbytes64 | awk '{print $2}'`"
    SNAPTIME="`perl -e \"print(time());\"`"
    echo "${MyHOST},${i},${SNAPTIME},${OBYTES},${RBYTES}"
    OBYTES=
    RBYTES= 
    SNAPTIME=
  done
  sleep 10
  COUNT="`expr $COUNT + 1`"
done

You have to be root to run this, but that is only because of the dladm command I am using on Solaris 10. If you don’t want to run it as root, then comment out the if statement and just leave the line that uses ifconfig. When you run it, it will produce an output like this:

gonzo,elxl0,1252806095,37255837,715035
gonzo,rge0,1252806096,605012664015,863919572622
gonzo,elxl0,1252806106,37255837,715035
gonzo,rge0,1252806107,605012664377,863919573090

The output is formated as hostname, ethernet, time of the run, sending bytes, and receiving bytes. (The time is the epoch time.) The above script will only run 800 times, pausing 10 seconds between each run of the kstat. You can change how long it runs by changing the line:

while [ $COUNT -lt 800 ]; 

Just change the 800 to some other number. The second item to change is the “interval” time and that is controled by the :

sleep 10

You probably don’t want to run this every second. Every 10 is about right, as it will allow me to get the traffic with out much overhead.

The second script I did, was a little php script (but can be done in probably any language, but I use php for just about everything. This script takes output from the file you created above (just run the above script, redirect it to a file) and gives you a human readable output.

Note if you have more than one ethernet card active in your system, currently you will need to
“grep” out each card to it’s own file. If you have a bunch of machines, you should probably import the data from above in to a mysql db, and then modify this script to pull the info from it.

Here is the script to just parse one network card:

< ?php
date_default_timezone_set("EST");
$fp=fopen("Netstat.csv",r);
if ($fp) {
  $i=0;
  while (!feof($fp)) {
    $buffer=fgets($fp);
    if ($buffer) { 
      list($hostname&#91;$i&#93;,$ethernet&#91;$i&#93;,$time&#91;$i&#93;,$sending&#91;$i&#93;,$receiving&#91;$i&#93;) = explode(",",$buffer);
      $newtime=date('r',$time&#91;$i&#93;);
      if ($i != 0 ) {
        $TDIFF=($time&#91;$i&#93;-$time&#91;$i-1&#93;);
        $SDIFF=($sending&#91;$i&#93;-$sending&#91;$i-1&#93;)/$TDIFF/1024/1024;
        $RDIFF=($receiving&#91;$i&#93;-$receiving&#91;$i-1&#93;)/$TDIFF/1024/1024;
        printf("%s|%s|%s|%3.3f|%3.3f\n",$hostname&#91;$i&#93;,$ethernet&#91;$i&#93;,$newtime,$SDIFF,$RDIFF);
        $SDIFF="";
        $RDIFF="";
        $TDIFF="";
      }
      $i++;
    }
  }
}
fclose($fp);
?>

In the above, I named my redirected output to be Netstat.csv. What the above script outputs will look like this:

gonzo|rge0|Sat, 12 Sep 2009 15:44:38 -0500|0.000|0.000
gonzo|rge0|Sat, 12 Sep 2009 15:44:49 -0500|0.000|0.007
gonzo|rge0|Sat, 12 Sep 2009 15:45:04 -0500|6.677|0.065
gonzo|rge0|Sat, 12 Sep 2009 15:45:18 -0500|3.148|0.027
gonzo|rge0|Sat, 12 Sep 2009 15:45:41 -0500|5.377|0.076
gonzo|rge0|Sat, 12 Sep 2009 15:45:55 -0500|8.678|0.111
gonzo|rge0|Sat, 12 Sep 2009 15:46:16 -0500|9.499|0.117
gonzo|rge0|Sat, 12 Sep 2009 15:46:30 -0500|8.861|0.117
gonzo|rge0|Sat, 12 Sep 2009 15:46:46 -0500|9.183|0.120
gonzo|rge0|Sat, 12 Sep 2009 15:47:02 -0500|10.783|0.139
gonzo|rge0|Sat, 12 Sep 2009 15:47:15 -0500|7.103|0.093
gonzo|rge0|Sat, 12 Sep 2009 15:47:29 -0500|7.165|0.100
gonzo|rge0|Sat, 12 Sep 2009 15:47:44 -0500|6.995|0.095
gonzo|rge0|Sat, 12 Sep 2009 15:48:01 -0500|6.986|0.099
gonzo|rge0|Sat, 12 Sep 2009 15:48:15 -0500|5.678|0.069
gonzo|rge0|Sat, 12 Sep 2009 15:48:28 -0500|6.530|0.090
gonzo|rge0|Sat, 12 Sep 2009 15:48:53 -0500|3.477|0.046
gonzo|rge0|Sat, 12 Sep 2009 15:49:14 -0500|6.459|0.083
gonzo|rge0|Sat, 12 Sep 2009 15:49:31 -0500|7.754|0.105
gonzo|rge0|Sat, 12 Sep 2009 15:49:58 -0500|9.416|0.121
gonzo|rge0|Sat, 12 Sep 2009 15:50:10 -0500|10.854|0.139
gonzo|rge0|Sat, 12 Sep 2009 15:50:21 -0500|11.922|0.152
gonzo|rge0|Sat, 12 Sep 2009 15:50:31 -0500|12.556|0.165
gonzo|rge0|Sat, 12 Sep 2009 15:50:43 -0500|12.813|0.170
gonzo|rge0|Sat, 12 Sep 2009 15:50:54 -0500|14.783|0.188
gonzo|rge0|Sat, 12 Sep 2009 15:51:05 -0500|12.729|0.168
gonzo|rge0|Sat, 12 Sep 2009 15:51:16 -0500|12.018|0.148
gonzo|rge0|Sat, 12 Sep 2009 15:51:27 -0500|10.786|0.141
gonzo|rge0|Sat, 12 Sep 2009 15:51:38 -0500|13.566|0.167
gonzo|rge0|Sat, 12 Sep 2009 15:51:49 -0500|11.234|0.144
gonzo|rge0|Sat, 12 Sep 2009 15:52:01 -0500|12.914|0.165

The output is : hostname, ethernet, time of query,sending speed in Mbps, receiving speed in Mbps. As you can see from the above, I was copying some large amounts of data.

PHP 1, PERL & SED 0

So I have been trying to figure out how to get PERL and SED to replace a set of random characters in a line with a fixed set of characters. I tried for about 2 days on Perl and sed, but could not get it to work right. So last night I tried PHP and got it in one shot. So say I have a file that has a part of a line that looks like this:

Variable1=test&Variable2=test%2312&Variable3=92ns10i9js

and I want to replace what ever is in the Variable2 value to be just a bunch of X’s or some other fixed value. Nothing I could do in sed and Perl would work, I ended up with this in PHP:

$buffer2=ereg_replace(“&Variable2=(.*)&Variable3″,”&Variable2=XXXXXXXX&Variable3”,$buffer);

So can any one tell me how to do it in Perl or sed?

To 64 bit or to 32 bit that is the question

Well, I started last night trying to get my Apache+PHP+MySQL+OpenSSL+IMAP compiled to run on the new X2100. What a nightmare. For some reason some of the software compiled as 32bit, some as 64 bit and nothing would link together right. I ended up putting Sun Studio 11 on the machine and trying it all again. Got OpenSSL, MySQL and IMAP to compile as 64 bit. Then was gettext and libxml2. gettext refuses to do the whole compile as 64 bit. Part of it does 32 bit and I can’t find out where. I keep getting those little “wrong ELF class: ELFCLASS64” or “wrong ELF class: ELFCLASS32”. I tried striping everything out of my path and environment variables and still does not work. Any one out there know exactly what needs set to make it all 32 or all 64 bit? I found that if I had /usr/ccs/bin in the path that it was trying to use the 32 bit linker, so I had to put /usr/ccs/bin/amd64 before the /usr/ccs/bin. I also added the CFLAGS environment variable of “-xarch=amd64”..

I even tried compiling everything on a Dual Pentium 3 machine (which is not 64 bit) and copying it over, but that failed too.

/Silently goes to bed with head down and wishing I could have afforded a T2000.

Java Studio Creator 2

I have decided to try and teach my self some java by using the free Sun Java Studio Creator 2 application. (My background is computer hardware engineering, so I never really played with Java while I was going through school. What I did get some what good at was Assembly, C, and Ada. Since I was not in Computer Science but rather in the Engineering school, they did not have us do any Java. Java was considered an elective, and when my time to take electives came around I did a lot of network hardware and network software electives. One of the classes I took was a class on network software programming. Having never done any socket programming before I took this class, it was sort of scary when on the first day the teacher says “your first project is to make a HTTP Proxy server, you have 2 weeks.” Needless to say it was one of the coolest classes I ever took at the university and probably one of the ones I learned the most in.) Anyways, I have decided to see if I can replicate my Syswatch web interface in JSP just as a proof of concept. It will probably take me a while as I said earlier I have 0 java/jsp experience.

99.9% of all the web apps I have written have been done in PHP. I have been using PHP since version 2, when is it called PHP/FI… I wrote a couple of apps that would help manage DNS and DHCP for a department I was working in at the time. We had about 10,000 IP’s and I found the easiest way to keep inventory of them was to build a web based front in to a MySQL DB and have it control BIND and DHCP. Later on I rewrote the entire thing to support dynamic registration of PC’s that the students brought to the dorms. Needless to say, PHP is the language that I use for pretty much any web app I have been doing lately.

So I downloaded Java Studio Creator 2 last night and set it up on my fastest machine (which is running windows at the moment). Installed MySQL and copied over a copy of my Syswatch DB. I have been able to get a table to display some of the content from the database, all with out reading any manuals yet. So we will see how far I can get before I get stuck.

Syswatch check versions script

So i previously mentioned that all the current version information is stored in a MySQL database on the main syswatch server. So how does the client code get updated? Pretty easy, each time before the program runs it runs this script to check the version info between the local files and what is on the server.


< ? mysql_connect("syswatch-server","syswatch") or die(mysql_error()); mysql_select_db("syswatch"); //Get list of files to check to see if they exist //Get OS so only ones required by this os are downloaded $OSTYPE=@system("/bin/uname"); $result=mysql_query("select * from versions where os ='ALL' or os='$OSTYPE'"); $num=mysql_num_rows($result); $i=0; echo "There are $num function files\n"; while ($i < $num ) { $File[$i]=mysql_result($result,$i,"function"); $Version[$i]=mysql_result($result,$i,"version"); $MD5Sum[$i]=mysql_result($result,$i,"md5file"); $fp = @fopen($File[$i],r); if ($fp) { //File Exists while (!feof($fp)) { $buffer=fgets($fp); if (ereg("//SYSVERSION:",$buffer)) { //Should be a version line explode on the : list($PreHead,$VERSION)=explode(":",$buffer); $VERSION=ereg_replace("\n","",$VERSION); if ("$VERSION" != "$Version[$i]" ) { echo "$File[$i] is downrev, it is $VERSION, current is $Version[$i]\n"; echo "Downloading new version\n"; //File is downrev, need to get new file $newfp=fopen("http://syswatch-server:8080/syswatch/$File[$i].txt",r); fclose($fp); unlink($File[$i]); $fp=fopen($File[$i],w); while (! feof($newfp)) { $buffernew=fgets($newfp); fputs($fp,$buffernew); } //Check the md5 sum of the file to make sure it has not been changed; $NewFile=md5_file($File[$i]); if ($NewFile!=$MD5Sum[$i]) { //If MD5's don't match issue warning and stop. echo "Local File does not match MD5 of stored function, stopping now\n"; echo "File name: $File[$i]\n"; echo "Local MD5: $NewFile\n"; echo "Store MD5: $MD5Sum[$i]\n"; exit (1); } } else { $NewFile=md5_file($File[$i]); if ($NewFile!=$MD5Sum[$i]) { //If MD5's don't match issue warning and stop. echo "Local File does not match MD5 of stored function, stopping now\n"; echo "File name: $File[$i]\n"; echo "Local MD5: $NewFile\n"; echo "Store MD5: $MD5Sum[$i]\n"; exit (1); } echo "$File[$i] is at current version\n"; fclose($fp); break; } } } } else { //File does not exist, need to download it echo "$File[$i] does not exist, downloading now\n"; $newfp=fopen("http://syswatch-server:8080/syswatch/$File[$i].txt","rb"); while (!feof($newfp)) { $contents.=fgets($newfp,1024); } fclose($newfp); $fp2=@fopen($File[$i],"w"); fwrite($fp2,$contents); fclose($fp2); $NewFile=md5_file($File[$i]); if ($NewFile!=$MD5Sum[$i]) { //If MD5's don't match issue warning and stop. echo "Local File does not match MD5 of stored function, stopping now\n"; echo "File name: $File[$i]\n"; echo "Local MD5: $NewFile\n"; echo "Store MD5: $MD5Sum[$i]\n"; exit (1); } } $contents=""; $i++; }