Poor Man’s Network Traffic Meter

Set out tonight to find a way to log “network traffic” through the interfaces on my solaris box. What I was wanting was the actually amount of traffic going through the interfaces. First thought was to use netstat. But that only shows “packets” and the packets could be differing sizes. So I ended up using kstat. I wrote this simple little script to grab the interface names, and then use kstat to get the data out of the network module for each card:

#!/bin/ksh
#Get list of Ethernet Cards in machine:
MyHOST="`hostname`"
OS="`uname -r`"
if [ ${OS} == "5.10" ] ; then
   MyETHERS="`/usr/sbin/dladm show-dev | awk '{print $1}'`"
else
   MyETHERS="`/usr/sbin/ifconfig -a | awk '{print $1}' | grep \":\" | awk -F':' '{print $1}' | sort -u | grep -v \"^lo0\"`"
fi
COUNT=0
while [ $COUNT -lt 800 ]; 
  do
  for i in `echo $MyETHERS`
  do
    OBYTES="`/usr/bin/kstat -p -c net -n $i -s obytes64 | awk '{print $2}'`"
    RBYTES="`/usr/bin/kstat -p -c net -n $i -s rbytes64 | awk '{print $2}'`"
    SNAPTIME="`perl -e \"print(time());\"`"
    echo "${MyHOST},${i},${SNAPTIME},${OBYTES},${RBYTES}"
    OBYTES=
    RBYTES= 
    SNAPTIME=
  done
  sleep 10
  COUNT="`expr $COUNT + 1`"
done

You have to be root to run this, but that is only because of the dladm command I am using on Solaris 10. If you don’t want to run it as root, then comment out the if statement and just leave the line that uses ifconfig. When you run it, it will produce an output like this:

gonzo,elxl0,1252806095,37255837,715035
gonzo,rge0,1252806096,605012664015,863919572622
gonzo,elxl0,1252806106,37255837,715035
gonzo,rge0,1252806107,605012664377,863919573090

The output is formated as hostname, ethernet, time of the run, sending bytes, and receiving bytes. (The time is the epoch time.) The above script will only run 800 times, pausing 10 seconds between each run of the kstat. You can change how long it runs by changing the line:

while [ $COUNT -lt 800 ]; 

Just change the 800 to some other number. The second item to change is the “interval” time and that is controled by the :

sleep 10

You probably don’t want to run this every second. Every 10 is about right, as it will allow me to get the traffic with out much overhead.

The second script I did, was a little php script (but can be done in probably any language, but I use php for just about everything. This script takes output from the file you created above (just run the above script, redirect it to a file) and gives you a human readable output.

Note if you have more than one ethernet card active in your system, currently you will need to
“grep” out each card to it’s own file. If you have a bunch of machines, you should probably import the data from above in to a mysql db, and then modify this script to pull the info from it.

Here is the script to just parse one network card:

< ?php
date_default_timezone_set("EST");
$fp=fopen("Netstat.csv",r);
if ($fp) {
  $i=0;
  while (!feof($fp)) {
    $buffer=fgets($fp);
    if ($buffer) { 
      list($hostname&#91;$i&#93;,$ethernet&#91;$i&#93;,$time&#91;$i&#93;,$sending&#91;$i&#93;,$receiving&#91;$i&#93;) = explode(",",$buffer);
      $newtime=date('r',$time&#91;$i&#93;);
      if ($i != 0 ) {
        $TDIFF=($time&#91;$i&#93;-$time&#91;$i-1&#93;);
        $SDIFF=($sending&#91;$i&#93;-$sending&#91;$i-1&#93;)/$TDIFF/1024/1024;
        $RDIFF=($receiving&#91;$i&#93;-$receiving&#91;$i-1&#93;)/$TDIFF/1024/1024;
        printf("%s|%s|%s|%3.3f|%3.3f\n",$hostname&#91;$i&#93;,$ethernet&#91;$i&#93;,$newtime,$SDIFF,$RDIFF);
        $SDIFF="";
        $RDIFF="";
        $TDIFF="";
      }
      $i++;
    }
  }
}
fclose($fp);
?>

In the above, I named my redirected output to be Netstat.csv. What the above script outputs will look like this:

gonzo|rge0|Sat, 12 Sep 2009 15:44:38 -0500|0.000|0.000
gonzo|rge0|Sat, 12 Sep 2009 15:44:49 -0500|0.000|0.007
gonzo|rge0|Sat, 12 Sep 2009 15:45:04 -0500|6.677|0.065
gonzo|rge0|Sat, 12 Sep 2009 15:45:18 -0500|3.148|0.027
gonzo|rge0|Sat, 12 Sep 2009 15:45:41 -0500|5.377|0.076
gonzo|rge0|Sat, 12 Sep 2009 15:45:55 -0500|8.678|0.111
gonzo|rge0|Sat, 12 Sep 2009 15:46:16 -0500|9.499|0.117
gonzo|rge0|Sat, 12 Sep 2009 15:46:30 -0500|8.861|0.117
gonzo|rge0|Sat, 12 Sep 2009 15:46:46 -0500|9.183|0.120
gonzo|rge0|Sat, 12 Sep 2009 15:47:02 -0500|10.783|0.139
gonzo|rge0|Sat, 12 Sep 2009 15:47:15 -0500|7.103|0.093
gonzo|rge0|Sat, 12 Sep 2009 15:47:29 -0500|7.165|0.100
gonzo|rge0|Sat, 12 Sep 2009 15:47:44 -0500|6.995|0.095
gonzo|rge0|Sat, 12 Sep 2009 15:48:01 -0500|6.986|0.099
gonzo|rge0|Sat, 12 Sep 2009 15:48:15 -0500|5.678|0.069
gonzo|rge0|Sat, 12 Sep 2009 15:48:28 -0500|6.530|0.090
gonzo|rge0|Sat, 12 Sep 2009 15:48:53 -0500|3.477|0.046
gonzo|rge0|Sat, 12 Sep 2009 15:49:14 -0500|6.459|0.083
gonzo|rge0|Sat, 12 Sep 2009 15:49:31 -0500|7.754|0.105
gonzo|rge0|Sat, 12 Sep 2009 15:49:58 -0500|9.416|0.121
gonzo|rge0|Sat, 12 Sep 2009 15:50:10 -0500|10.854|0.139
gonzo|rge0|Sat, 12 Sep 2009 15:50:21 -0500|11.922|0.152
gonzo|rge0|Sat, 12 Sep 2009 15:50:31 -0500|12.556|0.165
gonzo|rge0|Sat, 12 Sep 2009 15:50:43 -0500|12.813|0.170
gonzo|rge0|Sat, 12 Sep 2009 15:50:54 -0500|14.783|0.188
gonzo|rge0|Sat, 12 Sep 2009 15:51:05 -0500|12.729|0.168
gonzo|rge0|Sat, 12 Sep 2009 15:51:16 -0500|12.018|0.148
gonzo|rge0|Sat, 12 Sep 2009 15:51:27 -0500|10.786|0.141
gonzo|rge0|Sat, 12 Sep 2009 15:51:38 -0500|13.566|0.167
gonzo|rge0|Sat, 12 Sep 2009 15:51:49 -0500|11.234|0.144
gonzo|rge0|Sat, 12 Sep 2009 15:52:01 -0500|12.914|0.165

The output is : hostname, ethernet, time of query,sending speed in Mbps, receiving speed in Mbps. As you can see from the above, I was copying some large amounts of data.

Why everyone should use bart (AKA do the Bart Man)

If you are using Solaris 10, and you have not used bart yet, you should stop everything and take a look at it.

For those who don’t know what bart is, it is the Basic Auditing and Reporting Tool that is in Solaris 10.

In a quick synopsis bart will create a report that shows all files/directories on a solaris machine. This report contains the permissions, owners, sizes, modify times and md5 hashes of all files on the system, along with acl’s if you are using ZFS.

So why is bart so important? First, it can be used as a security tool. When you install a new Solaris 10 system, the first thing you should do after you get it installed and patched and before it is placed on the network is run a bart on the system and save the report to a cd. This will be the “baseline” image of the system. Then every week/month you should run a bart against the machine again and then use the compare option to see what files have changed, added or deleted from the system. Where this comes in really handy is if your think that your machine has been hacked or compromised. You can use the comparison to determine which files may have been modified by the hacker.

But there is a non-security use for bart as well that is VERY useful. This use is one that I had not thought of until I needed it the other day. So what is this use? Reseting the permissions on files that were accidentally changed by an in-experienced UNIX person thinking that a “chmod -R 777 *” is the best way to fix their problems.

The first thing that came to my mind when I saw this happen was oh no, the machine had not even been backed up yet, and a day’s worth of work would have been lost. Even if the machine had been backed up, do you realize how long it would take to restore a file system with 40,000+ files, just because the permissions were screwed up. ( Note, the permissions on the various files were very different and even included some setuid, and setgiud files which were wiped out as well.)

So how did bart save the day? Luckly I had taken a bart of the machine before the work had begun on the file system. So after the chmod command was issued, I then took a bart of the file system again. I now could run a bart compare against the control and test manifest and see exactly what all had changed.

Once I had this output, I could then create a script to change the permissions of the files/directories back to the original values. All told after I finished tweaking my script it took about 20 minutes to reset the permissions on all the files and directories.

So here is a quick start to getting your first bart manifest of your system:

1. Create a bart_rules file. If you do not create a rules file, your output will only have Files and not directories listed in it. My simple bart_rules file looks like this:

/
CHECK ALL
/home
IGNORE ALL

I ignore the /home file system as in my case it was nfs mounted. In reality you would want to include all local file systems.

2. Create the bart, I keep the rules file in /root/bart_rules so I would run the command:

bart create -R / -r /root/bart_rules > /tmp/bart.output

This will create a bart manifest and output it to /tmp/bart.output. Looking at the first couple of lines of it looks like this:

unixwiz@sungeek:/home/unixwiz> head -20 /tmp/bart.out
! Version 1.0
! Saturday, May 17, 2008 (21:24:27)
# Format:
#fname D size mode acl dirmtime uid gid
#fname P size mode acl mtime uid gid
#fname S size mode acl mtime uid gid
#fname F size mode acl mtime uid gid contents
#fname L size mode acl lnmtime uid gid dest
#fname B size mode acl mtime uid gid devnode
#fname C size mode acl mtime uid gid devnode
/ D 1024 40755 user::rwx,group::r-x,mask:r-x,other:r-x 481d0e43 0 0
/.ICEauthority F 310 100600 user::rw-,group::---,mask:---,other:--- 44c581c2 0 0 3eb63faf448e8a2b2c1a7b2019a8bde3
/.Xauthority F 99 100600 user::rw-,group::---,mask:---,other:--- 44c560e0 0 0 5ffe2e5f4b6f73e662001f62f7cae4d3
/.bash_history F 649 100600 user::rw-,group::---,mask:---,other:--- 481d1109 0 0 9132e0e798d5d05644cafc90c2aa876a
/.dt D 512 40755 user::rwx,group::r-x,mask:r-x,other:r-x 44c560e0 0 0
/.dt/appmanager D 512 40755 user::rwx,group::r-x,mask:r-x,other:r-x 44c5534d 0 0
/.dt/help D 512 40755 user::rwx,group::r-x,mask:r-x,other:r-x 44c5534d 0 0
/.dt/icons D 512 40755 user::rwx,group::r-x,mask:r-x,other:r-x 44c5534d 0 0
/.dt/sessionlogs D 512 40755 user::rwx,group::r-x,mask:r-x,other:r-x 44c5534c 0 0
/.dt/sessionlogs/sungeek_DISPLAY=:0 F 132 100644 user::rw-,group::r--,mask:r--,other:r-- 44c560e0 0 0 6d4e62fc972046a7a85fdb36a0ce21fd

The first part of the file, the part that begins with #fname is a legend as to how each type of line is formed.
So looking at the first actual line of the contents :
/ D 1024 40755 user::rwx,group::r-x,mask:r-x,other:r-x 481d0e43 0 0
We see that the fnmae is /, it is a directory, with a size of 1024. Its mode is 755, the last modified time is the “481d0e43” and it is owned by uid 0 and gid 0.

Looking at a file in particular we see this:

/httpd/htdocs/index.html F 10 100644 user::rw-,group::r--,mask:r--,other:r-- 463d4f4b 0 0 b7a9369d4cc9f82ed707bce91ced8af8

In the above, we see that the file is 10 bytes, has a permissions of 644 and is owned by root/root.

Now suppose that I for some reason by accident was in the /httpd/htdocs directory and did a chmod -R 777 *. Since I had my control manifest, I would then run another bart and then use the compare option. What I would get is something like this:

#bart compare /tmp/bart.output /tmp/bart.output2
/httpd/htdocs/index.html:
mode control:100644 test:100777
acl control:user::rw-,group::r--,mask:r--,other:r-- test:user::rwx,group::rwx,mask:rwx,other:rwx

Here we can see that the permissions has changed from 644 to 777. But the output is not really easy to parse with a script. So we need to use the “-p” option on the bart compare:

#bart compare -p /tmp/bart.output /tmp/bart.output2
/httpd/htdocs/index.html mode 100644 100777 acl user::rw-,group::r--,mask:r--,other:r-- user::rwx,group::rwx,mask:rwx,other:rwx

In the above, since the only thing that was changed was the mode, that is the only thing that is listed.

here are some other examples:

/var/samba/locks/browse.dat mtime 482f8544 482f8800
/var/samba/locks/unexpected.tdb contents 7c3404e9622749702e3df56caf26fe72 72983947ada3260a236394a51aef0d31

The first line shows that the file browse.dat modify time changed, but nothing else. The second line shows that the unexpected.tdb had it’s contents change. This can been see by the 2 different hashes.

Here is another example of the index.html file above, after it had been edited:

bash-3.00# bart compare /tmp/bart.out /tmp/bart.out3
/httpd/htdocs/index.html:
size control:10 test:26
mode control:100644 test:100777
acl control:user::rw-,group::r--,mask:r--,other:r-- test:user::rwx,group::rwx,mask:rwx,other:rwx
mtime control:463d4f4b test:482f8b89
contents control:b7a9369d4cc9f82ed707bce91ced8af8 test:1567caf683e3859cb5da7335c35438f7

Once again this is in the “human” readable format, the “machine” readable looks like :

bash-3.00# bart compare -p /tmp/bart.out /tmp/bart.out3
/httpd/htdocs/index.html size 10 26 mode 100644 100777 acl user::rw-,group::r--,mask:r--,other:r-- user::rwx,group::rwx,mask:rwx,other:rwx mtime 463d4f4b 482f8b89 contents b7a9369d4cc9f82ed707bce91ced8af8 1567caf683e3859cb5da7335c35438f7

(the above is actually all on one line.)

Once you have the output of the bart after the “oops” you will need to run the bart compare with options to ignore some items. Since I am only interested in the mode, the size, mtime and contents can be ignored. I used the following:

bash-3.00# bart compare -i size,mtime,contents,uid,gid -p /tmp/bart.out /tmp/bart.out2

This only shows files that have had their mode changed:

bash-3.00# bart compare -i size,mtime,contents,uid,gid -p /tmp/bart.out /tmp/bart.out2
/httpd/htdocs/index.html mode 100644 100777 acl user::rw-,group::r--,mask:r--,other:r-- user::rwx,group::rwx,mask:rwx,other:rwx

You should redirect this output to a file, so that it can then be used to generate a script.
With the output in a file I then did this:

cat /tmp/bart.compare | awk '{print "chmod "$3" "$1}' > /tmp/CHANGEPERMS

So basicly I cat the file and print the chmod command allong with the 3rd field (100644) and then the first field (/httpd/htdocs/index.html) and redirect this to a new file. Once I spot check this file, you can then run it and it will “reset” the permissions back.

Now everything I have shown above is based on the machine having a UFS file system. If you run bart against a file system that is ZFS, you will get a manifest that looks something like this:


/home/unixwiz/bin/php F 10587732 100755 owner@::deny,owner@:read_data/write_data/append_data/write_xattr/execute/write_attributes/write_acl/write_owner:allow,group@:write_data/append_data:deny,group@:read_data/execute:allow,everyone@:write_data/append_data/write_xattr/write_attributes/write_acl/write_owner:deny,everyone@:read_data/read_xattr/execute/read_attributes/read_acl/synchronize:allow 4743a7fa 100 14 9b8cfb15ed069bd6e43d7c2ae11a3e23

It shows the ZFS extended acl’s.

So if you haven’t started using bart, you should start as soon as possible.

How to calculate yesterday

I was working on a shell script last night and needed to calculate the value for yesterday. I did not have access to GNU date, so using that is out of the question. All I could use was what was available to me in a default install of Solaris 10. So I decided to use Perl as such (note that the YESTERDAY should all be on one line):


#!/bin/bash
YESTERDAY=$(perl -e '@y=localtime(time()-86400);
printf "%04d%02d%02d",$y[5]+1900,$y[4]+1,$y[3];$y[3];')

What this will do is store the value of yesterday in a shell variable called YESTERDAY
Now I have not done perl in a long while so here is an explanation of what it does:
1. Runs the perl function time which will find the current time, then subtract 86400 from it (24 hours).
2. Next it is run through the localtime function which creates an array that has the following values:

Array Element Value
0 Seconds
1 Minutes
2 Hour
3 Day of Month
4 Month of year (0=January)
5 Year (starting at 1900)
6 Day of week (0=sunday)
7 Day of Year (0..364 or 0..365 if leap)
8 Is Daylight savings time active

So in my little script above, we are looking for fields 5, 4 and 3. I add 1900 to the value of 5 (in this case 5 = 108). I add 1 to the value of 4 to get the current month (3+1 = 4 = April). The values are then pushed through printf so that we have a 4 digit year with leading 0’s, a 2 digit month with leading 0’s and a 2 digit day with leading 0’s. So the value of my YESTERDAY variable will now show 20080418.

Hope this helps some one else.

Quick Script to see missing patches on Solaris

Today I needed to see what patches were missing on a ton of machines. Instead of trying to start the Patch manager of the month for solaris, I wrote this little script that would produce me a HTML page of the current patches installed and ones that needed to be installed. This script is based off of the patchdiag.xref available from Sun. I know there are many other tools out there such as PCA (Patch check advanced) but in the environment I was in today, I could not use any third party programs so I wrote my own. The output looks like this:

output of patch checking script

What the script does is the following:

  1. Get a list of current patches on the machine
  2. Find the latest version of each patch that is installed on the machine and compare it to the latest version available according to the patchdiag.xref
  3. Generate one line of HTML code, listing the patch, the current installed revision and the current available revision and a description of the patch. It will also place a link to the patch on sunsolve.
  4. At the end it will list a summary of patches installed, missing, obsolete, and how many security and recommended patches there are.
  5. It will then compare the list of currently available patches against what is installed to see if there are patches that are available but not installed on the system
  6. If a patch has never been installed it will then list the line of HTML code showing the patch number, revision, flag (security,recommended) and its description
  7. At the end it will display the total number of patches that are not installed and how many are recommended and/or security

Basically it is a very simple script. It should work on all versions of solaris from 7+. It ONLY looks at Solaris specific patches and not those that are unbundled (i.e. Sun Studio, Web Server, etc.)

Here is the script, it may not be the cleanest or most efficient, but it was a quick job…

Shell script to analyze solaris patches