30 minutes with Solaris 11 Express

Havn’t really posted anything Solaris / Unix related in a while, but Oracle gave me a reason to do so today. They released Solaris 11 Express today. So like any “Sun Geek” I downloaded all the different installers; text, automated, live cd, and the full ips package stuff. First up on the install was the text based installer. Needless to say if you are used to the old Solaris text based installer, this one is almost 100% different. The colors are different, there is next to no customization of the install (you can’t do any network config other than auto(dhcp) or none), can’t pick what Filesystem you will use for root (glad I started doing all zfs roots a couple of months back.)

Some of the things I have noticed:

1. Sudo is now installed, and the first user you create (during the install) is automatically given full “root” access via sudo.

2. Seems all of the commands that were in /usr/sfw/bin are now in /usr/bin with symlinks in /usr/sfw/bin

3. There is a new /usr/gnu structure that has a lot of the GNU based commads, one cool thing [date “+%s”] now works and prints out the date since the EPOCH.

4. $PATH has /usr/gnu/bin:/usr/bin:/usr/sbin:/sbin by default. Which means doing an ls -la looks different than when using the /usr/bin/ls -la. This means it may break scripts ….

5. It seems that when installing in a VMWare environment (I was using Fusion at the moment and will try with ESX later this weekend) that on the first reboot, it will hang indefinitely. You have to do a force reboot or shutdown and restart to get it to “boot”

6. The graphical startup is sort of cool, but it “hides” all the boot messages unless you hit a key to show them

7. cc and gcc are NOT installed by default. 🙁

8. showrev doesn’t exist any more.

9. Secure by default is enabled

10. IPfilter is enabled by default (no rules though)

that is just a few.. more later.

See the What’s New doc for more info and the Release notes.

ZFS + PCA, goodbye UFS

ZFS has been around for a while now.. I have used it for some data partitions, but when Sun added the ability to use it as the root filesystem, I was a little hesitant to start using it there. Part of it was because, I know if I get a root disk that crashes and it is on UFS, I can get in to it pretty well. ZFS was different and I was never really comfortable about using it for root, until last night. I have been looking for a way to keep a lot of Solaris machines up to date with the Recommended and Security patches and doing it with UFS seemed to be taking for ever. Part of the problem I had with keeping them updated with UFS was the shear downtime it required to install the cluster in single user mode. Multiply that by X number of machines and it is a never ending chore to update them.

This weekend I started looking at the PCA tool, since I have seen a lot of people mention good things about it. So off to my test machine and I installed a new VM with Solaris 10 10/09 ( update 8 ) in it. After the install was finished using a ZFS root, I decided to set up a PCA proxy server on another machine. The purpose of the PCA Proxy server is that it will be the one with access to the Internet to download the patches from sunsolve. It was extremely easy to do this, (in fact I have it running in a zone on my main server.)

  1. Created a new plain zone (can be on anything, but I wanted to keep it seperate).
  2. Configure the apache2 instance on the machine, by copying the /etc/apache2/httpd.conf-example to /etc/apache2/httpd.conf
  3. Edit the httpd.conf and change the line that says “Timeout 300” to be “Timeout 1800”. You need to make it at least 1800, if not more depending on the speed of your Internet connection. At 22Mb/s 1800 was ok for me.
  4. Create a directory /var/apache2/htdocs/patches, make it owned by webservd:webservd and 755 as the permissions.
  5. Download and save a copy of pca in /var/apache2/cgi-bin and call it pca-proxy.cgi. Make it owned by webservd:webservd and 755 as the permissions.
  6. Create a file in /etc called pca-proxy.conf. In it place the following:
    xrefdir=/var/apache2/htdocs/patches
    patchdir=/var/apache2/htdocs/patches
    user=sunsolveusername
    passwd=sunsolvepassword
    
  7. In order to make the proxy run a little faster on the first use, I decided to download and “cache” the latest security and recommended patch cluster. (You don’t need to do this, but if the patches are missing the pca proxy server will download them. Considering my machine needed 156 patches, this was faster…) Once the recommended and security patches were downloaded, I placed them in a temp place and unzipped the cluster. Once the cluster is unzipped, I needed to make zip files of each patch (so that the pca client can download the zip file). To do this, I went in to tmp/10_x86_Recommended/patches and ran the following:
    for i in `cat patch_order`
    do
    zip -r $i $i
    done
    
  8. Once the zipping is done, move all the patch zip files in to the /var/apache2/htdocs/patches directory.
  9. Start up the apache2 service “svcadm enable apache2”
  10. Now it is time to configure the client, copy the pca script to the client machine and place it some place, I used /root.
  11. Next create a config file /etc/pca.conf in it with the following:
    patchurl=http://pca-host/cgi-bin/pca-proxy.cgi
    xrefurl=http://pca-host/cgi-bin/pca-proxy.cgi
    syslog=local7
    safe=1
    

    The first two lines tells pca where to find the patches and the patchdiag.xref file. The syslog line tells it to log all activity to local7 syslog facaility. The last line “safe=1” means: Safe patch installation. Checks all files for local modifications before installing a patch. A patch will not be installed if files with local modifications would be overwritten.

  12. Now that the config file is created, make sure that syslog is set to handle local7 info, I have mine set to local7.info going to /var/adm/local7.log. PCA will log the patch installation stuff to that log (i.e.:
    Apr 11 17:10:50 zfstest2 pca: [ID 702911 local7.notice] Installed patch 124631-36 (SunOS 5.10_x86: System Administration Applications, Network, and C)
    Apr 11 19:07:04 zfstest2 pca: [ID 702911 local7.notice] Failed to install patch 118246-21 (comm_dssetup 6.4-5.05_x86: core patch) rc=15

Now comes the part that makes ZFS worth using… We are going to create a new “boot environment” and then patch that environment”

  1. First we need to create a new BE;
    lucreate -n p20100411

    The p20100411 can be anything, I used today’s date since I patched the machine today.. Makes it easy to remember when the last time the machine was patched.

  2. Now we need to mount it
    lumount p20100411 /.alt.root 
  3. Now we can start patching;
     pca -i -R /.alt.root
  4. Because I cached most of the patches locally on my pca proxy, it should not take too long for it to download, unzip and install the patches in the alt root
  5. Once the patching is done, it will give you a summary line telling you how many patches were downloaded and installed:
    Download Summary: 156 total, 156 successful, 0 skipped, 0 failed
    Install Summary : 156 total, 156 successful, 0 skipped, 0 failed
    
  6. Now we need to unmount the alt root and activate it to boot:
    luumount p20100411
    luactivate p20100411
    
  7. Now just reboot the machine. You MUST use init or shutdown, if you don’t then it won’t boot in to the new boot environment. I use
    shutdown -g0 -i6 -y
  8. Depending on how long it takes for your machine to boot, when it comes back up it should be on the new ZFS file system:
    bash-3.00# df -h
    Filesystem             size   used  avail capacity  Mounted on
    rpool/ROOT/p20100411    49G   6.6G    38G    15%    /
    
  9. Now you can run that new patched system for how ever long it takes to verify your patches didn’t break anything. Once you are sure everything is ok, then you can delete the old install, in my case:
    ludelete s10x_u8wos_08a
    

    This should let you recover a little bit of space. In my case it was about 1.5 gig.

The only thing left is to set up a bunch of scripts to do “pca -l” about once a month to see what patches need installed and to log that. PCA has a lot of other functions than I went over here, in a couple of words, it seems to be kick ass. On top of that it is free! The ability to create new BE’s will definitely hope any one with the right amount of disk space be able to keep their system up to date.

One Tip, make sure you watch the output of the luactivate command. This is what is displayed:

**********************************************************************

The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.

**********************************************************************

In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:

1. Boot from Solaris failsafe or boot in single user mode from the Solaris
Install CD or Network.

2. Mount the Parent boot environment root slice to some directory (like
/mnt). You can use the following command to mount:

     mount -Fzfs /dev/dsk/c1t0d0s0 /mnt

3. Run  utility with out any arguments from the Parent boot
environment root slice, as shown below:

     /mnt/sbin/luactivate

4. luactivate, activates the previous working boot environment and
indicates the result.

5. Exit Single User mode and reboot the machine.

**********************************************************************

Bad Oracle, Leave Solaris free

I just read Ben Rockwood’s post about Solaris No Longer Free. All I can say is I am severely disappointed in how Oracle has pretty much killed Sun and it’s products. One of the best things that Sun ever did was allow people to use Solaris for free. The caveat was you only got the Security patches for free. About a month or so ago, Oracle decided that you couldn’t get any patches unless you had a support contract. Ok I can sort of see your position on that, but why the hell are you now going to start charging for the OS as well. You have taken one of the best OS’ in the world for servers and basically killed it. What you are going to do is push more people to Linux (eck, I hate linux..). I am not sure why a company that has the number one OS would ever push people away from its OS. Linux is still immature in many ways and can’t scale at all unless you want to scale out and use up more power/floor space.

I hope that the Solaris user community will come around like they did when Sun tried to kill Solaris X86, and let Oracle know what a bad idea this was.

Seeing Red – RIP Sun

Not sure how I feel now that Sun is no longer Sun. Kinda weird watching the Oracle web cast of their purchase of Sun and what is going to happen. One thing I can’t understand is they are all wearing badges saying “We’re Hiring!”. But yet they let go SO many good Sun employees. I just don’t think it is going to be the same old Sun that I have used since 1994.

RIP Sun 1982 – 2010

From http://blogs.sun.com/jag/

Poor Man’s Network Traffic Meter

Set out tonight to find a way to log “network traffic” through the interfaces on my solaris box. What I was wanting was the actually amount of traffic going through the interfaces. First thought was to use netstat. But that only shows “packets” and the packets could be differing sizes. So I ended up using kstat. I wrote this simple little script to grab the interface names, and then use kstat to get the data out of the network module for each card:

#!/bin/ksh
#Get list of Ethernet Cards in machine:
MyHOST="`hostname`"
OS="`uname -r`"
if [ ${OS} == "5.10" ] ; then
   MyETHERS="`/usr/sbin/dladm show-dev | awk '{print $1}'`"
else
   MyETHERS="`/usr/sbin/ifconfig -a | awk '{print $1}' | grep \":\" | awk -F':' '{print $1}' | sort -u | grep -v \"^lo0\"`"
fi
COUNT=0
while [ $COUNT -lt 800 ]; 
  do
  for i in `echo $MyETHERS`
  do
    OBYTES="`/usr/bin/kstat -p -c net -n $i -s obytes64 | awk '{print $2}'`"
    RBYTES="`/usr/bin/kstat -p -c net -n $i -s rbytes64 | awk '{print $2}'`"
    SNAPTIME="`perl -e \"print(time());\"`"
    echo "${MyHOST},${i},${SNAPTIME},${OBYTES},${RBYTES}"
    OBYTES=
    RBYTES= 
    SNAPTIME=
  done
  sleep 10
  COUNT="`expr $COUNT + 1`"
done

You have to be root to run this, but that is only because of the dladm command I am using on Solaris 10. If you don’t want to run it as root, then comment out the if statement and just leave the line that uses ifconfig. When you run it, it will produce an output like this:

gonzo,elxl0,1252806095,37255837,715035
gonzo,rge0,1252806096,605012664015,863919572622
gonzo,elxl0,1252806106,37255837,715035
gonzo,rge0,1252806107,605012664377,863919573090

The output is formated as hostname, ethernet, time of the run, sending bytes, and receiving bytes. (The time is the epoch time.) The above script will only run 800 times, pausing 10 seconds between each run of the kstat. You can change how long it runs by changing the line:

while [ $COUNT -lt 800 ]; 

Just change the 800 to some other number. The second item to change is the “interval” time and that is controled by the :

sleep 10

You probably don’t want to run this every second. Every 10 is about right, as it will allow me to get the traffic with out much overhead.

The second script I did, was a little php script (but can be done in probably any language, but I use php for just about everything. This script takes output from the file you created above (just run the above script, redirect it to a file) and gives you a human readable output.

Note if you have more than one ethernet card active in your system, currently you will need to
“grep” out each card to it’s own file. If you have a bunch of machines, you should probably import the data from above in to a mysql db, and then modify this script to pull the info from it.

Here is the script to just parse one network card:

< ?php
date_default_timezone_set("EST");
$fp=fopen("Netstat.csv",r);
if ($fp) {
  $i=0;
  while (!feof($fp)) {
    $buffer=fgets($fp);
    if ($buffer) { 
      list($hostname&#91;$i&#93;,$ethernet&#91;$i&#93;,$time&#91;$i&#93;,$sending&#91;$i&#93;,$receiving&#91;$i&#93;) = explode(",",$buffer);
      $newtime=date('r',$time&#91;$i&#93;);
      if ($i != 0 ) {
        $TDIFF=($time&#91;$i&#93;-$time&#91;$i-1&#93;);
        $SDIFF=($sending&#91;$i&#93;-$sending&#91;$i-1&#93;)/$TDIFF/1024/1024;
        $RDIFF=($receiving&#91;$i&#93;-$receiving&#91;$i-1&#93;)/$TDIFF/1024/1024;
        printf("%s|%s|%s|%3.3f|%3.3f\n",$hostname&#91;$i&#93;,$ethernet&#91;$i&#93;,$newtime,$SDIFF,$RDIFF);
        $SDIFF="";
        $RDIFF="";
        $TDIFF="";
      }
      $i++;
    }
  }
}
fclose($fp);
?>

In the above, I named my redirected output to be Netstat.csv. What the above script outputs will look like this:

gonzo|rge0|Sat, 12 Sep 2009 15:44:38 -0500|0.000|0.000
gonzo|rge0|Sat, 12 Sep 2009 15:44:49 -0500|0.000|0.007
gonzo|rge0|Sat, 12 Sep 2009 15:45:04 -0500|6.677|0.065
gonzo|rge0|Sat, 12 Sep 2009 15:45:18 -0500|3.148|0.027
gonzo|rge0|Sat, 12 Sep 2009 15:45:41 -0500|5.377|0.076
gonzo|rge0|Sat, 12 Sep 2009 15:45:55 -0500|8.678|0.111
gonzo|rge0|Sat, 12 Sep 2009 15:46:16 -0500|9.499|0.117
gonzo|rge0|Sat, 12 Sep 2009 15:46:30 -0500|8.861|0.117
gonzo|rge0|Sat, 12 Sep 2009 15:46:46 -0500|9.183|0.120
gonzo|rge0|Sat, 12 Sep 2009 15:47:02 -0500|10.783|0.139
gonzo|rge0|Sat, 12 Sep 2009 15:47:15 -0500|7.103|0.093
gonzo|rge0|Sat, 12 Sep 2009 15:47:29 -0500|7.165|0.100
gonzo|rge0|Sat, 12 Sep 2009 15:47:44 -0500|6.995|0.095
gonzo|rge0|Sat, 12 Sep 2009 15:48:01 -0500|6.986|0.099
gonzo|rge0|Sat, 12 Sep 2009 15:48:15 -0500|5.678|0.069
gonzo|rge0|Sat, 12 Sep 2009 15:48:28 -0500|6.530|0.090
gonzo|rge0|Sat, 12 Sep 2009 15:48:53 -0500|3.477|0.046
gonzo|rge0|Sat, 12 Sep 2009 15:49:14 -0500|6.459|0.083
gonzo|rge0|Sat, 12 Sep 2009 15:49:31 -0500|7.754|0.105
gonzo|rge0|Sat, 12 Sep 2009 15:49:58 -0500|9.416|0.121
gonzo|rge0|Sat, 12 Sep 2009 15:50:10 -0500|10.854|0.139
gonzo|rge0|Sat, 12 Sep 2009 15:50:21 -0500|11.922|0.152
gonzo|rge0|Sat, 12 Sep 2009 15:50:31 -0500|12.556|0.165
gonzo|rge0|Sat, 12 Sep 2009 15:50:43 -0500|12.813|0.170
gonzo|rge0|Sat, 12 Sep 2009 15:50:54 -0500|14.783|0.188
gonzo|rge0|Sat, 12 Sep 2009 15:51:05 -0500|12.729|0.168
gonzo|rge0|Sat, 12 Sep 2009 15:51:16 -0500|12.018|0.148
gonzo|rge0|Sat, 12 Sep 2009 15:51:27 -0500|10.786|0.141
gonzo|rge0|Sat, 12 Sep 2009 15:51:38 -0500|13.566|0.167
gonzo|rge0|Sat, 12 Sep 2009 15:51:49 -0500|11.234|0.144
gonzo|rge0|Sat, 12 Sep 2009 15:52:01 -0500|12.914|0.165

The output is : hostname, ethernet, time of query,sending speed in Mbps, receiving speed in Mbps. As you can see from the above, I was copying some large amounts of data.