unixwiz

If you are using Solaris 10, and you have not used bart yet, you should stop everything and take a look at it.

For those who don’t know what bart is, it is the Basic Auditing and Reporting Tool that is in Solaris 10.

In a quick synopsis bart will create a report that shows all files/directories on a solaris machine. This report contains the permissions, owners, sizes, modify times and md5 hashes of all files on the system, along with acl’s if you are using ZFS.

So why is bart so important? First, it can be used as a security tool. When you install a new Solaris 10 system, the first thing you should do after you get it installed and patched and before it is placed on the network is run a bart on the system and save the report to a cd. This will be the “baseline” image of the system. Then every week/month you should run a bart against the machine again and then use the compare option to see what files have changed, added or deleted from the system. Where this comes in really handy is if your think that your machine has been hacked or compromised. You can use the comparison to determine which files may have been modified by the hacker.

But there is a non-security use for bart as well that is VERY useful. This use is one that I had not thought of until I needed it the other day. So what is this use? Reseting the permissions on files that were accidentally changed by an in-experienced UNIX person thinking that a “chmod -R 777 *” is the best way to fix their problems.

The first thing that came to my mind when I saw this happen was oh no, the machine had not even been backed up yet, and a day’s worth of work would have been lost. Even if the machine had been backed up, do you realize how long it would take to restore a file system with 40,000+ files, just because the permissions were screwed up. ( Note, the permissions on the various files were very different and even included some setuid, and setgiud files which were wiped out as well.)

So how did bart save the day? Luckly I had taken a bart of the machine before the work had begun on the file system. So after the chmod command was issued, I then took a bart of the file system again. I now could run a bart compare against the control and test manifest and see exactly what all had changed.

Once I had this output, I could then create a script to change the permissions of the files/directories back to the original values. All told after I finished tweaking my script it took about 20 minutes to reset the permissions on all the files and directories.

So here is a quick start to getting your first bart manifest of your system:

1. Create a bart_rules file. If you do not create a rules file, your output will only have Files and not directories listed in it. My simple bart_rules file looks like this:

I ignore the /home file system as in my case it was nfs mounted. In reality you would want to include all local file systems.

2. Create the bart, I keep the rules file in /root/bart_rules so I would run the command:

This will create a bart manifest and output it to /tmp/bart.output. Looking at the first couple of lines of it looks like this:

The first part of the file, the part that begins with #fname is a legend as to how each type of line is formed.
So looking at the first actual line of the contents :

We see that the fnmae is /, it is a directory, with a size of 1024. Its mode is 755, the last modified time is the “481d0e43″ and it is owned by uid 0 and gid 0.

Looking at a file in particular we see this:

In the above, we see that the file is 10 bytes, has a permissions of 644 and is owned by root/root.

Now suppose that I for some reason by accident was in the /httpd/htdocs directory and did a chmod -R 777 *. Since I had my control manifest, I would then run another bart and then use the compare option. What I would get is something like this:

Here we can see that the permissions has changed from 644 to 777. But the output is not really easy to parse with a script. So we need to use the “-p” option on the bart compare:

In the above, since the only thing that was changed was the mode, that is the only thing that is listed.

here are some other examples:

The first line shows that the file browse.dat modify time changed, but nothing else. The second line shows that the unexpected.tdb had it’s contents change. This can been see by the 2 different hashes.

Here is another example of the index.html file above, after it had been edited:

Once again this is in the “human” readable format, the “machine” readable looks like :

(the above is actually all on one line.)

Once you have the output of the bart after the “oops” you will need to run the bart compare with options to ignore some items. Since I am only interested in the mode, the size, mtime and contents can be ignored. I used the following:

This only shows files that have had their mode changed:

You should redirect this output to a file, so that it can then be used to generate a script.
With the output in a file I then did this:

So basicly I cat the file and print the chmod command allong with the 3rd field (100644) and then the first field (/httpd/htdocs/index.html) and redirect this to a new file. Once I spot check this file, you can then run it and it will “reset” the permissions back.

Now everything I have shown above is based on the machine having a UFS file system. If you run bart against a file system that is ZFS, you will get a manifest that looks something like this:

It shows the ZFS extended acl’s.

So if you haven’t started using bart, you should start as soon as possible.

I was working on a shell script last night and needed to calculate the value for yesterday. I did not have access to GNU date, so using that is out of the question. All I could use was what was available to me in a default install of Solaris 10. So I decided to use Perl as such (note that the YESTERDAY should all be on one line):

What this will do is store the value of yesterday in a shell variable called YESTERDAY
Now I have not done perl in a long while so here is an explanation of what it does:
1. Runs the perl function time which will find the current time, then subtract 86400 from it (24 hours).
2. Next it is run through the localtime function which creates an array that has the following values:

So in my little script above, we are looking for fields 5, 4 and 3. I add 1900 to the value of 5 (in this case 5 = 108). I add 1 to the value of 4 to get the current month (3+1 = 4 = April). The values are then pushed through printf so that we have a 4 digit year with leading 0’s, a 2 digit month with leading 0’s and a 2 digit day with leading 0’s. So the value of my YESTERDAY variable will now show 20080418.

Hope this helps some one else.

Today I needed to see what patches were missing on a ton of machines. Instead of trying to start the Patch manager of the month for solaris, I wrote this little script that would produce me a HTML page of the current patches installed and ones that needed to be installed. This script is based off of the patchdiag.xref available from Sun. I know there are many other tools out there such as PCA (Patch check advanced) but in the environment I was in today, I could not use any third party programs so I wrote my own. The output looks like this:

What the script does is the following:

1. Get a list of current patches on the machine
2. Find the latest version of each patch that is installed on the machine and compare it to the latest version available according to the patchdiag.xref
3. Generate one line of HTML code, listing the patch, the current installed revision and the current available revision and a description of the patch. It will also place a link to the patch on sunsolve.
4. At the end it will list a summary of patches installed, missing, obsolete, and how many security and recommended patches there are.
5. It will then compare the list of currently available patches against what is installed to see if there are patches that are available but not installed on the system
6. If a patch has never been installed it will then list the line of HTML code showing the patch number, revision, flag (security,recommended) and its description
7. At the end it will display the total number of patches that are not installed and how many are recommended and/or security

Basically it is a very simple script. It should work on all versions of solaris from 7+. It ONLY looks at Solaris specific patches and not those that are unbundled (i.e. Sun Studio, Web Server, etc.)

Here is the script, it may not be the cleanest or most efficient, but it was a quick job…

Shell script to analyze solaris patches

I decided to add a modules to my Syswatch system that would do basic hardware inventory (Processor[speed,number,type, etc] and Memory). Basicly this was so that DBA’s could look at a web page to see what OS and hardware a machine was running. So far each OS has it’s own way of doing this and this is how I am getting the information.

Solaris:

What the above code does is populate the following Variables:

$NumberOfPhysical : Number of Physical processors in the machine
$NumberOfVirutal: Number of Cores/threads that are in the machine
$FirstProcessor: The Numeric ID of the first processor in the machine. For example in an E25K your first processor could be a non 0 number
$MemorySize: Size of memory in MB
$ProcessorType: The Processor type, i.e. UltraSPARC-III+
$ProcessorSpeed: Speed of the Processor in MHz

For AIX the commands are done a little differently:

Same variables are availabe as in Solaris.

For Apple MacOSX you do it like this:

Mac was the most intersting to do as I looked around for a while to try and find a command that would output what I wanted. I figured out how to do it while watching the Activity monitor and ran the gui system _profiler command. I decided to run “man system_profiler” and found that you could run it via the command line. Made it really easy to get info out. What I use above is just 1 of many data types that system_profiler will return.

I still have to write the Linux part to this module, and it will use the “/proc” file system with the cpuinfo and meminfo files to get info. Once I get that done next week I will post it as well.

My first set of OS quirks that I have found from writing my System Monitoring System is how to get system load information out of MacOSX. Where as Solaris, AIX and Linux have a pretty detailed output of the vmstat command, the same command on OSX does not provide any of the information I am looking for.

For example a run of “vmstat 1 5″ on Solaris might output something like this:

 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m1 m1 m1 m1   in   sy   cs us sy id
 0 0 0 9766688 2621504 42 182 184 3 3 0  0  1  1  1  0  871  860  667  8  3 89
 0 0 0 5264424 448816 8 174  0  0  0  0  0  0  0  0  0 5507 165436 4029 36 29 35
 1 0 0 5263392 447784 1558 4497 8 0 0 0  0  1  0  1  0 5465 157436 4628 47 36 17
 0 0 0 5246296 427976 1 737  0  0  0  0  0  0  0  0  0 3592 12346 2550 51 4 45
 0 0 0 5240480 422168 1 499 16  0  0  0  0  0  0  0  0 6027 16409 3344 51 5 44

However on MacOSX the “vmstat” command is actually called “vm_stat” and provides this:

Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                   917147.
Pages active:                  53505.
Pages inactive:                12689.
Pages wired down:              65238.
"Translation faults":      515506895.
Pages copy-on-write:        42753315.
Pages zero filled:         136121644.
Pages reactivated:                 0.
Pageins:                     8237820.
Pageouts:                      10953.
Object cache: 18010609 hits of 25242212 lookups (71% hit rate)

Needless to say, it did not provide me any of the User, Sys, Idle, or run/block queue information I was looking for. So I scraped using vm_stat for MacOSX. Next I looked at iostat. Needless to say it is not the same either. An example of “iostat 1 5″ on Solaris looks like :

   tty        md10          md11          md12          md15           cpu
 tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv   us sy wt id
   0   59   7   1   12    5   1    9    5   1    8    3   0    4    8  3  1 89
   0  234   0   0    0    0   0    0    0   0    0    0   0    0   25  1  0 74
   0   80   0   0    0    0   0    0    0   0    0    0   0    0   25  0  0 74
   0   80  12   3   19   12   3   10   12   3    8    0   0    0   33  4  1 62
   0   80   0   0    0    0   0    0    0   0    0    0   0    0   48  2  0 49

However running the same command on MacOSX gives you:

          disk3           disk0           disk1       cpu
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us sy id
 51.71   0  0.00  13.42   1  0.01  12.73   1  0.01   0  1 99
  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0 99
  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  1 99
  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0100
  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  2 98

So I at least get the User, Sys and Idle time. but we now have a new problem. For what ever reason the us,sy,id fields are a fixed 3 character width, which is fine except for when you see what is above. Where the machine is 100% idle, the 100 is right next to the 0 of the sys. Well, when I am using AWK to sum up the columns and average them, this “0100″ produces some weird output. So to get around it I run it as follows:

The reason I only look at the last 5 lines is the first line is a summary line since the machine has been booted which does not give me a good snapshot of the current load on the box. The 0:0:0…:0 represents values that I can get from vmstat on other machines but is not available on MacOSX, so they just get logged in to the DB as 0’s.

If any one knows of other tools to get performance info out of MacOSX please leave a comment.