This weekend I started looking at the PCA tool, since I have seen a lot of people mention good things about it. So off to my test machine and I installed a new VM with Solaris 10 10/09 ( update 8 ) in it. After the install was finished using a ZFS root, I decided to set up a PCA proxy server on another machine. The purpose of the PCA Proxy server is that it will be the one with access to the Internet to download the patches from sunsolve. It was extremely easy to do this, (in fact I have it running in a zone on my main server.)

1. Created a new plain zone (can be on anything, but I wanted to keep it seperate).
2. Configure the apache2 instance on the machine, by copying the /etc/apache2/httpd.conf-example to /etc/apache2/httpd.conf
3. Edit the httpd.conf and change the line that says “Timeout 300″ to be “Timeout 1800″. You need to make it at least 1800, if not more depending on the speed of your Internet connection. At 22Mb/s 1800 was ok for me.
4. Create a directory /var/apache2/htdocs/patches, make it owned by webservd:webservd and 755 as the permissions.
5. Download and save a copy of pca in /var/apache2/cgi-bin and call it pca-proxy.cgi. Make it owned by webservd:webservd and 755 as the permissions.
6. Create a file in /etc called pca-proxy.conf. In it place the following:
   xrefdir=/var/apache2/htdocs/patches
   patchdir=/var/apache2/htdocs/patches
   user=sunsolveusername
   passwd=sunsolvepassword
7. In order to make the proxy run a little faster on the first use, I decided to download and “cache” the latest security and recommended patch cluster. (You don’t need to do this, but if the patches are missing the pca proxy server will download them. Considering my machine needed 156 patches, this was faster…) Once the recommended and security patches were downloaded, I placed them in a temp place and unzipped the cluster. Once the cluster is unzipped, I needed to make zip files of each patch (so that the pca client can download the zip file). To do this, I went in to tmp/10_x86_Recommended/patches and ran the following:
   for i in `cat patch_order`
   do
   zip -r $i $i
   done
8. Once the zipping is done, move all the patch zip files in to the /var/apache2/htdocs/patches directory.
9. Start up the apache2 service “svcadm enable apache2″
10. Now it is time to configure the client, copy the pca script to the client machine and place it some place, I used /root.
11. Next create a config file /etc/pca.conf in it with the following:
    patchurl=http://pca-host/cgi-bin/pca-proxy.cgi
    xrefurl=http://pca-host/cgi-bin/pca-proxy.cgi
    syslog=local7
    safe=1

    The first two lines tells pca where to find the patches and the patchdiag.xref file. The syslog line tells it to log all activity to local7 syslog facaility. The last line “safe=1″ means: Safe patch installation. Checks all files for local modifications before installing a patch. A patch will not be installed if files with local modifications would be overwritten.

12. Now that the config file is created, make sure that syslog is set to handle local7 info, I have mine set to local7.info going to /var/adm/local7.log. PCA will log the patch installation stuff to that log (i.e.:
    Apr 11 17:10:50 zfstest2 pca: [ID 702911 local7.notice] Installed patch 124631-36 (SunOS 5.10_x86: System Administration Applications, Network, and C)
    Apr 11 19:07:04 zfstest2 pca: [ID 702911 local7.notice] Failed to install patch 118246-21 (comm_dssetup 6.4-5.05_x86: core patch) rc=15

Now comes the part that makes ZFS worth using… We are going to create a new “boot environment” and then patch that environment”

1. First we need to create a new BE;
   lucreate -n p20100411

   The p20100411 can be anything, I used today’s date since I patched the machine today.. Makes it easy to remember when the last time the machine was patched.

2. Now we need to mount it
   lumount p20100411 /.alt.root
3. Now we can start patching;
   pca -i -R /.alt.root
4. Because I cached most of the patches locally on my pca proxy, it should not take too long for it to download, unzip and install the patches in the alt root
5. Once the patching is done, it will give you a summary line telling you how many patches were downloaded and installed:
   Download Summary: 156 total, 156 successful, 0 skipped, 0 failed
   Install Summary : 156 total, 156 successful, 0 skipped, 0 failed
6. Now we need to unmount the alt root and activate it to boot:
   luumount p20100411
   luactivate p20100411
7. Now just reboot the machine. You MUST use init or shutdown, if you don’t then it won’t boot in to the new boot environment. I use
   shutdown -g0 -i6 -y
9. Depending on how long it takes for your machine to boot, when it comes back up it should be on the new ZFS file system:
   bash-3.00# df -h
   Filesystem             size   used  avail capacity  Mounted on
   rpool/ROOT/p20100411    49G   6.6G    38G    15%    /
10. Now you can run that new patched system for how ever long it takes to verify your patches didn’t break anything. Once you are sure everything is ok, then you can delete the old install, in my case:
    ludelete s10x_u8wos_08a

    This should let you recover a little bit of space. In my case it was about 1.5 gig.

The only thing left is to set up a bunch of scripts to do “pca -l” about once a month to see what patches need installed and to log that. PCA has a lot of other functions than I went over here, in a couple of words, it seems to be kick ass. On top of that it is free! The ability to create new BE’s will definitely hope any one with the right amount of disk space be able to keep their system up to date.

One Tip, make sure you watch the output of the luactivate command. This is what is displayed:

**********************************************************************

The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.

**********************************************************************

In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:

1. Boot from Solaris failsafe or boot in single user mode from the Solaris
Install CD or Network.

2. Mount the Parent boot environment root slice to some directory (like
/mnt). You can use the following command to mount:

     mount -Fzfs /dev/dsk/c1t0d0s0 /mnt

3. Run  utility with out any arguments from the Parent boot
environment root slice, as shown below:

     /mnt/sbin/luactivate

4. luactivate, activates the previous working boot environment and
indicates the result.

5. Exit Single User mode and reboot the machine.

**********************************************************************

I hope that the Solaris user community will come around like they did when Sun tried to kill Solaris X86, and let Oracle know what a bad idea this was.

RIP Sun 1982 – 2010

From http://blogs.sun.com/jag/

You have to be root to run this, but that is only because of the dladm command I am using on Solaris 10. If you don’t want to run it as root, then comment out the if statement and just leave the line that uses ifconfig. When you run it, it will produce an output like this:

gonzo,elxl0,1252806095,37255837,715035
gonzo,rge0,1252806096,605012664015,863919572622
gonzo,elxl0,1252806106,37255837,715035
gonzo,rge0,1252806107,605012664377,863919573090

The output is formated as hostname, ethernet, time of the run, sending bytes, and receiving bytes. (The time is the epoch time.) The above script will only run 800 times, pausing 10 seconds between each run of the kstat. You can change how long it runs by changing the line:

Just change the 800 to some other number. The second item to change is the “interval” time and that is controled by the :

You probably don’t want to run this every second. Every 10 is about right, as it will allow me to get the traffic with out much overhead.

The second script I did, was a little php script (but can be done in probably any language, but I use php for just about everything. This script takes output from the file you created above (just run the above script, redirect it to a file) and gives you a human readable output.

Note if you have more than one ethernet card active in your system, currently you will need to
“grep” out each card to it’s own file. If you have a bunch of machines, you should probably import the data from above in to a mysql db, and then modify this script to pull the info from it.

Here is the script to just parse one network card:

In the above, I named my redirected output to be Netstat.csv. What the above script outputs will look like this:

gonzo|rge0|Sat, 12 Sep 2009 15:44:38 -0500|0.000|0.000
gonzo|rge0|Sat, 12 Sep 2009 15:44:49 -0500|0.000|0.007
gonzo|rge0|Sat, 12 Sep 2009 15:45:04 -0500|6.677|0.065
gonzo|rge0|Sat, 12 Sep 2009 15:45:18 -0500|3.148|0.027
gonzo|rge0|Sat, 12 Sep 2009 15:45:41 -0500|5.377|0.076
gonzo|rge0|Sat, 12 Sep 2009 15:45:55 -0500|8.678|0.111
gonzo|rge0|Sat, 12 Sep 2009 15:46:16 -0500|9.499|0.117
gonzo|rge0|Sat, 12 Sep 2009 15:46:30 -0500|8.861|0.117
gonzo|rge0|Sat, 12 Sep 2009 15:46:46 -0500|9.183|0.120
gonzo|rge0|Sat, 12 Sep 2009 15:47:02 -0500|10.783|0.139
gonzo|rge0|Sat, 12 Sep 2009 15:47:15 -0500|7.103|0.093
gonzo|rge0|Sat, 12 Sep 2009 15:47:29 -0500|7.165|0.100
gonzo|rge0|Sat, 12 Sep 2009 15:47:44 -0500|6.995|0.095
gonzo|rge0|Sat, 12 Sep 2009 15:48:01 -0500|6.986|0.099
gonzo|rge0|Sat, 12 Sep 2009 15:48:15 -0500|5.678|0.069
gonzo|rge0|Sat, 12 Sep 2009 15:48:28 -0500|6.530|0.090
gonzo|rge0|Sat, 12 Sep 2009 15:48:53 -0500|3.477|0.046
gonzo|rge0|Sat, 12 Sep 2009 15:49:14 -0500|6.459|0.083
gonzo|rge0|Sat, 12 Sep 2009 15:49:31 -0500|7.754|0.105
gonzo|rge0|Sat, 12 Sep 2009 15:49:58 -0500|9.416|0.121
gonzo|rge0|Sat, 12 Sep 2009 15:50:10 -0500|10.854|0.139
gonzo|rge0|Sat, 12 Sep 2009 15:50:21 -0500|11.922|0.152
gonzo|rge0|Sat, 12 Sep 2009 15:50:31 -0500|12.556|0.165
gonzo|rge0|Sat, 12 Sep 2009 15:50:43 -0500|12.813|0.170
gonzo|rge0|Sat, 12 Sep 2009 15:50:54 -0500|14.783|0.188
gonzo|rge0|Sat, 12 Sep 2009 15:51:05 -0500|12.729|0.168
gonzo|rge0|Sat, 12 Sep 2009 15:51:16 -0500|12.018|0.148
gonzo|rge0|Sat, 12 Sep 2009 15:51:27 -0500|10.786|0.141
gonzo|rge0|Sat, 12 Sep 2009 15:51:38 -0500|13.566|0.167
gonzo|rge0|Sat, 12 Sep 2009 15:51:49 -0500|11.234|0.144
gonzo|rge0|Sat, 12 Sep 2009 15:52:01 -0500|12.914|0.165

The output is : hostname, ethernet, time of query,sending speed in Mbps, receiving speed in Mbps. As you can see from the above, I was copying some large amounts of data.

1. Virtual Point-to-Point (Tun) and Ethernet (TAP) devices driver. I got the version 1.1 from http://vtun.sourceforge.net/tun/ in source code form.
2. LZO version 1.08 compression software from : http://www.oberhumer.com/opensource/lzo/download/LZO-v1/
3. OpenVPN software, I am using the version 2.1RC because I wanted the version to match what I am going to run on the Mac. It can be downloaded from http://openvpn.net/index.php/open-source/downloads.html

Once I got everything downloaded, just compile the LZO, Tun, and OpenVPN:
I decided to have everything related to the vpn installed in /opt/vpn. One thing to note, I tried using the new version 2.x of LZO, and OpenVPN would not find it, so I had to use Version 1 even though 2 is supposed to be supported. So I did the following to compile LZO:

Next was to compile TUN

Only issue with tun was that it did not use the –prefix, it puts everything where it needs to be in /usr/kernel/drv on solaris.

Next is openvpn:

Once that is installed I did the simple 1 to 1 vpn connection (static key) for just testing to see if it would work. So in the /opt/vpn/openvpn/sbin directory I did this:

I then copy that key to my client via some “secure” means

Then created a server.conf that looks like this:

dev tun
ifconfig 10.8.0.1 10.8.0.2
secret static.key
cipher AES-256-CBC
keepalive 10 120

On my client (MacOSX) I downloaded Tunnelblick from http://code.google.com/p/tunnelblick/downloads/list and installed it. Next I copied that static.key from the server to the client and put it in ~/Library/openvpn. I also created a openvpn.conf in that directory that looked like this:

remote a.b.c.d
dev tun
ifconfig 10.8.0.2 10.8.0.1
secret static.key
cipher AES-256-CBC
route 10.0.0.0 255.255.255.0

In the above, a.b.c.d represents my public IP address for my solaris server.

Now when you start tunnelblick it will search that directory and find that config file and ask if you want to load it. But we are not quite ready to start yet. The next thing I had to do was forward port 1194 UDP off of my router to my OpenVPN server. I will leave this exercise to you. You will also need to make sure IP forwarding is enabled on the Solaris 10 server (because I only have 1 network card in it, but “two” different networks on the box. IP Forwarding will allow your remote machine to be able to see your local network. And since my OpenVPN server is not the router for the entire network, I had to add a static route on my router to say that 10.8.0.0 is available via the openvpn servers local network address, I.e. 10.0.0.1.

You should be able to start the openvpn server now:

Once it is started you can use tunnelblick to connect. Once you are connected, you should see that is is connected and the icon has changed from this:

to look like this:

You should also see a tun0 device show up:

You should now be able to see all your hosts on the “remote” network. Next up I am going to work on doing the pki infrastructure so I can hopefully link other clients both static and dynamic.

This make is really nice to be able to see your “home” network while you are away.

The kicker was getting VMWare Fusion to actually allocate the network cards the way I needed them. I gave the VM 2 nics (the Mac Pro has 2), however the only options that VMWare Fusion let you do is NAT, Host-Only, and Bridged. None of which will work if I want a private network for the Sun Ray’s. To fix this you will need to go and edit some files that VMWare Fusion uses. What I had to do was the following:

1. Open up the Terminal app
2. Edit the file /Library/Application Support/VMware Fusion/boot.sh

3. Comment out the following line:

And then add 2 lines directly below that line, which tells vmware to bind the en0 physical device to the vmnet0 virtual device, same for en1 to vmnet2. Note you can not use vmnet1 or vmnet8 as those are for NAT and Host-only connections.

Once done, do the following:

Now go in to your Mac System Preferences and configure the second network card for a private subnet (i.e. 192.168.128.0/24, and set the IP to be something like 192.168.128.254).

Now make sure that your VM is NOT started and is in a powered off state. Go in to the VM and under the settings for that VM add another network adapter, make sure it is selected as “Connected” it doesn’t matter what the device is configured to as we will change it later to an option that is not shown in that list.

Now you need to change the .VMX file so that it can use the new network device. So go in to the directory where you have your VM’s at and then cd in to the machine.vmwarevm directory (For example mine is called SolarisDev.vmwarevm)

Once in there you will need to edit the vmx file, mine is called SolarisDev.vmx. The first thing we are going to change is the ethernet0.connectionType property. Right now it could be any of the ones listed (host-only,bridged, nat). But we are going to change this to “custom”:

Next find the entry for ethernet0.vnet, if it doesn’t exist create it and make it look like the line below. If it does exist and doesn’t match that below, make it match that:

Now we need to do the same for the ethernet1 entries. The only difference to what is above is vmnet0 changes to vmnet2. Once the changes are made you can save the file and start up your Solaris VM. Now what ever network is on your en0 connection on your Mac should be what is connected to the e1000g0 network on the Solaris side. I used the e1000g0 as the “public” side of the Sun Ray server. The e1000g1 interface will be what ever is connected on the en1 connection on the Mac side. I used this adapter for the private Sun Ray Lan.

You should be able to finish following the instructions on the Sun Ray wiki now and get everything configured.

To test the soft client, I set up LAN Connections on the Sun Ray Server:

I then installed the soft client in another VM on the same machine that only had access to the public network. You then can tell the soft client what the IP of the Sun Ray server is and it will connect. Pretty darn cool that the soft client works with minimal config.

This can probably be done on a MacBook Pro as well, if you use the wireless connection as the public side and the wired as the private side. Nice way to do a little demo in one computer.

For reference here is what my network section of the .vmx file looks like :

References:
Sun Ray Software 4.2 Wiki: http://wikis.sun.com/display/SRSS4dot2/Home

ufs: [ID 879645 kern.notice] NOTICE: /: unexpected free inode 48714, run fsck(1M) -o f

Well this is a little hard to fix especially if you have root under solaris volume manager control. So how do you do it? Sort of easy, but it assumes you have either a solaris boot cd or a jumpstart server you can boot off of to get the box into single user mode.

Once you have the box in single user mode off of either the CD or the jumpstart server, you will need to mount in readonly mode one side of the root mirror. Say /dev/dsk/c1t0d0s0. :

Once that is mounted (if it won’t mount then you will have to fsck that side of the mirror to fix the file system if it is really screwed up to not even mount in read only mode) you need to copy some files from it to the “temp” root that you are booted from. But first we need to unload the md driver:

Now that the md driver is unloaded, you need to copy the following files:

Now unmount the /mnt

Now we need to restart the md driver

Now if you run metastat or metadb, you will get a generic error like there is no devices or databases set up. To fix this run a :

This does the following:

    -r              Only used in a shell script  at  boot  time.
                     Sets up all metadevices that were configured
                     before the system crashed or was shut  down.
                     The  information about previously configured
                     metadevices  is  stored  in  the  metadevice
                     state database (see metadb(1M)).

You can now run metastat, but all your devices will say they need maintenance. To fix this run :

This will sync all the mirrors back up. Now we are finally able to run an fsck againest a mirrored slice that we weren’t able to run if the machine is up in full user mode.

So now I ran :

And I keep running fsck on the device till it comes back clean with no errors. Then lather, rinse, repeat for the other slices.

Now once all the slices are done, make sure to update your boot archive (if on an x86 machine) and then you can restart the machine:

If on a SPARC box, just make sure all the file systems you mounted off of the disks are unmounted and then restart the machine.

1. Install one of the VM’s from scratch from either a CD or ISO Image. (You can upload an ISO image to the server by going in to to the Configuration section of the vSphere Client, and clicking on storage and then browse the datastore. What I did was make a directory for the ISO Images, and then you can click the upload button and find the file on the local machine and it will upload it to the server. It makes it a lot faster to install from an ISO image vs the CDRom).

2. Once the OS is installed, I put the latest Security/Recommended patches on it. I also installed the VMWare Tools.

3. Do any “configuration” that you want to be done for all the VM’s. For example, modify the /etc/issue,/etc/ssh/sshd_config, create non root accounts, enable auditing, disable services, etc.

4. The next thing I did was ran a “sys-unconfig”. This will remove the root user’s password, clear the hostname, clear all IP information and the naming service info. It will leave other local created accounts though.

5. The final step of the sys-unconfig, halts the VM. This will be the “last” time you will boot this image. We will now use it as a poor man’s template for our other VM’s. (The reason I say last is unless you are going to update it to be used to “clone” new vm’s, you should never boot it again.) You should also probably rename this virtual machine to be something like “Solaris – Template – DNB” (DNB=Do Not Boot… )

6. Back in the Datastore browser, create a new folder for the to be “cloned” vm.

7. Now go in to the folder of the original vm you made that is in the “unconfiged” state. Once in there, select ALL the files in there and right click and select copy.

8. Now go to the newly created folder, and go in to it and right click and select paste. For a VM with a 20gb disk this takes about a minute or so to copy and paste.

9. Now go in to the cloned directory and right click on the VMX file and select “Add to Inventory”. It will pop up a dialog box and will let you “name” the new vm. Type in the new name of the new vm. (It won’t change the name of the files in that directory though.)

10. Next you can power on the new “cloned” vm. It will give you a pop up box asking if you have “moved” it or “copied” it or something else. Just select copied, and it will load the VM.

11. Now when Solaris boots, it will look like a clean machine, it will ask you for the hostname, ip, name service, etc, and then reboot after you enter in the information.

All said I did about 12 of these and had them all running at the same time and didn’t have a problem with them. If you need some different “hardware” for a vm, before you power it on, you can change the settings just as if you had installed it from scratch. One thing with the new vSphere it will support adding more than just a disk if the OS supports it. So you should be able to add processors, disk and possibly memory on the fly to a solaris VM.

Hope that helps people with the i version of ESX…

1. Download the latest version of the “file” program from ftp://ftp.astron.com/pub/file/ (my case the current is 5.03)
2. Unzip/untar the file
3. Configure and run make:

3. There are a bunch of other requirements for mediatomb, the easiest way to get them is to use www.blastwave.org. The packages that I installed are:
CSWbdb4
CSWbzip2
CSWcurl
CSWcurlrt
CSWexpat
CSWfaac
CSWfaad2
CSWfconfig
CSWffmpeg
CSWffmpeglib
CSWftype2
CSWgcc3corert
CSWgcc3g++rt
CSWgcrypt
CSWggettext
CSWgpgerr
CSWiconv
CSWid3lib
CSWimlib2
CSWisaexec
CSWlame
CSWliba52
CSWlibid3tag
CSWlibidn
CSWlibnet
CSWlibogg
CSWlibsdl
CSWlibssh2
CSWlibtool
CSWlibtoolrt
CSWlibx11
CSWlibxau
CSWlibxcb
CSWlibxdmcp
CSWncurses
CSWoldaprt
CSWossl
CSWossldevel
CSWosslrt
CSWosslutils
CSWpixman
CSWpng
CSWsasl
CSWsdlmixer
CSWsqlite3
CSWsqlite3dev
CSWstl4
CSWsunmath
CSWtaglibgcc
CSWtheora
CSWtiff
CSWungif
CSWvorbis
CSWxvid
CSWzlib

4. Once these are installed, you can download mediatomb and compile it. http://mediatomb.cc/pages/download

However before you can run gmake, you need to edit a couple of files. One is the src/main.cc, you need to comment out lines 128 through 141. This is not needed for Solaris. The second file to edit is a result of this:

During the compile I got an error that looked like this:

To fix it edit the src/url.cc file and on line 78 change it from this:

to this :

Then rerun gmake . Once the compile is finished, su to root and do a gmake install, it will place all the media tomb stuff in /mediatomb. (I am using a Zone on a Solaris 10 machine so / has plenty of space).

I then created a user for mediatomb to be run under, so a user and group called mediatmb were created, and all the /mediatomb directories and files were changed to be owned by mediatmb

Once that was done, login as the mediatmb user and create a script in /mediatmb/bin with the following in it:

where x.x.x.x is the IP address of the machine you are running it on. In Constantin’s blog, he mentioned also using the interface, but I found that it had problems since this was a zone. So instead of using the interface, I used the IP address of the zone.

The changes I made to the config.xml in the ~/.mediatomb directory are as follows:

Now all you have to do is login to the web interface and add the media, following mediatomb’s documentation.

Some interesting things I have found with it:

1. If your PS3 is on wireless it may have problems streaming mp4, but not mpeg2. Weird I know as the mpeg2 was solid streaming at 370kb/s, but it couldn’t handle the mp4. Switching to a hardwired connection fixed that problem.

2. If you happen to be a ReplayTV user [cause TiVO is a wanna be replay ] you can use MediaTomb in conjunction with DVArchive, which also runs nicely in my Solaris zone. Just point a media directory at the Local_Guide directory and mediatomb will stream every MPEG2 file in that directory to your PS3. Which is pretty damn cool.

3. The PS3 is very pickly about it’s mp4 type files. Some that I made with handbrake don’t work, but ones I did with ffmpegx worked once I put the hardwired connection in.

4. I have not tried any of the transcoding stuff. I would rather do that before and not bog down my server doing that.

5. I need to do a lot of reading on how to make folders and the such so that my collection is organized and not just all under one directory.

For those interested:
My compile environment is setup like this:

Great.. now what. It appears that you have to have the boot info in the first 1023 cylinders of the harddrive. Well this big 500GB harddrive has over 60,000+ cylinders on it. Guess what, the Solaris installer put the swap on 3-263 and then put the root on 58884 – some high number.. So obviously grub is going to have a problem.

So how do you fix this. There is 2 possible ways that I can think of and possibly a third.

The first would be for Sun to change the installer to not automatically place root at the end of the disk.

The second is when you are getting ready to do the install, pre-layout the disks using format from the command line.

The third is what I am trying, since the remaining of the disk has not been used yet, I am going to move the partitions around with dd. The first thing I did was delete the swap partition on slice 1. The second was to create a new temp partition on slice 3 that is the exact same size as slice 0. Now using dd I did the following:

Hopefully this won’t take too long as the partition is only 15gb in size. You could have done the same thing with tar and have both partitions mounted. Once the dd finishes. I will delete the slice 0 and recreate it starting at cylinder 3 and give it the same size as it was before, and then dd the info back. Once that is done, delete the temp slice and recreate a swap partition. Hopefully this works.