unixwiz

If you are using Solaris 10, and you have not used bart yet, you should stop everything and take a look at it.

For those who don’t know what bart is, it is the Basic Auditing and Reporting Tool that is in Solaris 10.

In a quick synopsis bart will create a report that shows all files/directories on a solaris machine. This report contains the permissions, owners, sizes, modify times and md5 hashes of all files on the system, along with acl’s if you are using ZFS.

So why is bart so important? First, it can be used as a security tool. When you install a new Solaris 10 system, the first thing you should do after you get it installed and patched and before it is placed on the network is run a bart on the system and save the report to a cd. This will be the “baseline” image of the system. Then every week/month you should run a bart against the machine again and then use the compare option to see what files have changed, added or deleted from the system. Where this comes in really handy is if your think that your machine has been hacked or compromised. You can use the comparison to determine which files may have been modified by the hacker.

But there is a non-security use for bart as well that is VERY useful. This use is one that I had not thought of until I needed it the other day. So what is this use? Reseting the permissions on files that were accidentally changed by an in-experienced UNIX person thinking that a “chmod -R 777 *” is the best way to fix their problems.

The first thing that came to my mind when I saw this happen was oh no, the machine had not even been backed up yet, and a day’s worth of work would have been lost. Even if the machine had been backed up, do you realize how long it would take to restore a file system with 40,000+ files, just because the permissions were screwed up. ( Note, the permissions on the various files were very different and even included some setuid, and setgiud files which were wiped out as well.)

So how did bart save the day? Luckly I had taken a bart of the machine before the work had begun on the file system. So after the chmod command was issued, I then took a bart of the file system again. I now could run a bart compare against the control and test manifest and see exactly what all had changed.

Once I had this output, I could then create a script to change the permissions of the files/directories back to the original values. All told after I finished tweaking my script it took about 20 minutes to reset the permissions on all the files and directories.

So here is a quick start to getting your first bart manifest of your system:

1. Create a bart_rules file. If you do not create a rules file, your output will only have Files and not directories listed in it. My simple bart_rules file looks like this:

I ignore the /home file system as in my case it was nfs mounted. In reality you would want to include all local file systems.

2. Create the bart, I keep the rules file in /root/bart_rules so I would run the command:

This will create a bart manifest and output it to /tmp/bart.output. Looking at the first couple of lines of it looks like this:

The first part of the file, the part that begins with #fname is a legend as to how each type of line is formed.
So looking at the first actual line of the contents :

We see that the fnmae is /, it is a directory, with a size of 1024. Its mode is 755, the last modified time is the “481d0e43″ and it is owned by uid 0 and gid 0.

Looking at a file in particular we see this:

In the above, we see that the file is 10 bytes, has a permissions of 644 and is owned by root/root.

Now suppose that I for some reason by accident was in the /httpd/htdocs directory and did a chmod -R 777 *. Since I had my control manifest, I would then run another bart and then use the compare option. What I would get is something like this:

Here we can see that the permissions has changed from 644 to 777. But the output is not really easy to parse with a script. So we need to use the “-p” option on the bart compare:

In the above, since the only thing that was changed was the mode, that is the only thing that is listed.

here are some other examples:

The first line shows that the file browse.dat modify time changed, but nothing else. The second line shows that the unexpected.tdb had it’s contents change. This can been see by the 2 different hashes.

Here is another example of the index.html file above, after it had been edited:

Once again this is in the “human” readable format, the “machine” readable looks like :

(the above is actually all on one line.)

Once you have the output of the bart after the “oops” you will need to run the bart compare with options to ignore some items. Since I am only interested in the mode, the size, mtime and contents can be ignored. I used the following:

This only shows files that have had their mode changed:

You should redirect this output to a file, so that it can then be used to generate a script.
With the output in a file I then did this:

So basicly I cat the file and print the chmod command allong with the 3rd field (100644) and then the first field (/httpd/htdocs/index.html) and redirect this to a new file. Once I spot check this file, you can then run it and it will “reset” the permissions back.

Now everything I have shown above is based on the machine having a UFS file system. If you run bart against a file system that is ZFS, you will get a manifest that looks something like this:

It shows the ZFS extended acl’s.

So if you haven’t started using bart, you should start as soon as possible.

Forgot to mention that I have tried using TSM (Tivoli Storage Manager) client on Solaris againest a ZFS file system. From what I can tell it works fine… I have backed up and restored some files with it. When looking at the TSM Server it lists the file system as “unknown”, but so far everything has been ok with it.

Technorati Tags: Solaris, ZFS, TSM, Tivoli

So I switch the 4 drives around so the large zfs pool is on the onboard controller and the smaller zfs pool is on the PCI controller. I then reran bonnie and look at the results now:

Old Slow zfs pool:

# ./bonnie
File './Bonnie.581', size: 104857600
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          100 15822 51.9 21557 18.7 12929 13.9 25061 97.6 23125 12.0 284.8  3.0

Is now nice and fast (or as fast as my on board controller will handle. The FTP now stays at 10M/s solid which is what I was looking for (this pool is used to serve Video to my Replay TV units).

Now the old fast pool is the new slow pool:

File './Bonnie.585', size: 104857600
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          100   975  3.0  1125  1.1  1541  1.5 26689 99.9 209631 99.9  42.6  0.6

as you can see that controller card is awful.. Now I understand why my growfs took so long when I added the drives a while back (it took over 2 hours to run). So I guess now, I need to do one of the following:

1. Find a new server that has SATA support on the mobo
2. Find a new controller card that has a BIOS that I can force the UltraDMA to be on
3. Get some IDE to scsi convertors to attach the drives to the scsi card I have
4. Get a SATA card and new SATA drives to extend the pool with

All in all, it is going to be a small amount of money for any of the options, but probably getting SATA cards will be the next step I do. I just can’t see getting rid of a perfectly fine Dual PIII when all it is doing is file serving and running a Zone for DVArchive.

Technorati Tags: Solaris, Adaptec, DVArchive

Seems I have a little performance problem with the Adaptec ASH-1233 controller and Solaris 10. For what ever reason, the drives are falling back to MultiwordDMA mode 2 (bad mmmkay!). What is weird, on a fresh boot, I can start copying large files to the ZFS mirror that is on the ASH-1233. It starts out at 10M/s (using FTP from another machine). After a little while, it suddenly drops to about 6M/s, then a little while later it drops to 3M/s, then it drops to about 1M/s and then finally 150K/s and then starts timeing out.. In contrast if I FTP the same file to a zfs mirror that is not on that controller I get a sustained 10+ M/s transfer rate. So I went and downloaded bonnie to run on it intresting results:

ZFS mirror on the ASH-1233:

# ./bonnie
File './Bonnie.682', size: 104857600
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          100 22989 90.5  1360  2.0  1591  2.6  3071 14.9  3905  2.7 124.8  1.3

Here is a zfs mirror using the onboard IDE controller test using bonnie:

# /home/bonnie/bonnie
File './Bonnie.900', size: 104857600
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          100 33636 100.0 39261 33.5 115563 100.0 27222 100.0 229894 100.0 10663.1 186.2

See the difference? I sure do. Needless to say my next step is to export the zfs pools and swap the cables and see how that works. Hopefully it will give me some better performance. (No the machine is not really starved for horse power, it is a Dual Pentium III with 1.5 gig of RAM and 1.1 TB of disk space on it.

Time to Backup 300+ gig of data: 8 hours
Time to install Solaris 10 06/06 : 1 hour
Time to fix the PATA controller card: 10 minutes
Time to create a mirrored 387 gig ZFS file system: 5.2 seconds

For some people MS Windows is the only thing they know, but for others if you use Solaris and ZFS, things go much faster and are much better.

Now that I have a ZFS file system, I can clone my zones instead of having to install each one. I really like zfs!

One note, that I came across creating a zfs file system. One of the disks previously had a ufs file system on it and it would not let me create the pool. to get around it I did this:

The error that caused me to do this was: