Changing passwords? lets make it as difficult as we can…

In this day and age of computer hacks and security problems, why do companies make it awkward to change usernames and or passwords? One example of an awkward procedure to change a password is on the VMware vCenter server. If like any good security minded person you have all  your passwords set to expire every 28 days or so, to change the password on the vCenter server you have to do some “command line fu” to change it. Heaven forbid that you have to change the username as well. So how do you do it? Well if you are running vCenter on a Windows 2008 server and connecting to a Oracle server (that actually holds all the data) there are a couple of things you need to do:

  1. Shutdown the vCenter server (disable it in the Services Control panel)
  2. Change the password for your vCenter user in the oracle DB
  3. Now here it the BIG gotcha. On the windows side you have to run a CMD prompt as an admin user. Just clicking on it in the start menu won’t do it. You have to right click on it and do “Run as Administrator”. If you fail to do this, the next step will fail and just piss you off even more. (The reason for this is the username and password are stored in the registry and I guess running cmd as normal user revokes all privs to modify the registry.)
  4. Now go to the location where VMware vCenter is installed and run the vpxd command with either a -p or a -P. If you use the lower case -p it will prompt you for the new database user password. If you use the -P option, right after the P you can put the new password on the command line.
  5. Now you should be able to start back up the vCenter processes.

Now if you need to change the userid, you need to use Regedit and go to :

  • HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter\DB (under My Computer)
  • HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\VMware, Inc.\VMware VirtualCenter\DB for 64 bit versions of Windows.

and change #2 to be the new userid.

This is documented in the VMware KB Article : Changing the vCenter database userid and password. But if you don’t pay attention go the run as part, you will spend a lot of time trying to figure it out even if you are logged in as an administrator.

 

If your password expires in Oracle while vCenter is up and running, it appears to continue to work while it is up. But if you reboot the vCenter server or restart the vCenter processes, it will “hang” and never start. They also need to make their error messages a little more detailed as to why it is ‘failing’ to start.

Why Thin Provisioning is bad

In this day and age everyone is trying to squeeze the last little drop out of every technological advance that they can. One of the technologies that is “big” is called Thin Provisioning. Basic in short terms, thin provisioning is where you tell a computer that you have X GB of disk (usually from a SAN or in VMware) but in reality you only have <X GB of disk backing it. This is big right now in SAN and VMware because enterprise disk is “expensive”. But is it really worth the cost? No!

See the main reason people (SAN or VMware admins) use Thin Provisioning is to “save” disk space. Say you have a server that performs one function and does not really use a lot of disk space, say a DNS server (either virutalized or physical booting from a SAN).  Now most admins usually like to keep all their servers with a standard config. So for the sake of this post, lets say the boot disk for this server is 50GB. Now once the OS and app is installed on it, it may only be using 4 GB of that 50GB disk.

Before thin provisioning that 50GB as far as a SAN admin is concerned is 50GB used. So in comes Thin Provisioning, now the SAN admin says “hey mister computer here is your 50GB disk ;-)” But in reality it only allocates as much space as being used by the server. So now on the SAN instead of a full 50GB “used” only 4GB would be used. Sounds awesome in theory, but what happens when  you add other servers in that same SAN pool (say the pool is 100GB in size). So the server admin gets another “50GB” disk from the SAN, doesn’t realize thin provisioning is in use, so they go on and install that server. Now we have 8GB in use out of the 100GB pool, but in reality all 100GB has been allocated as far as the 2 servers are concerned.

The next part is when the whole process starts to drown. The server admin asks for another disk, this time 200Gb for say a database or code repository server. Well the SAN administrator says “ok here is your 200GB disk ;-)” But put the disk in the same 100GB pool that the other two servers are in because “he knows” you won’t use all “200GB”. We have now over committed disk however the server admin does not know this has happened. Once the third servers OS has been installed (another 4GB) everything seems to be fine, and technically it is because we are only using 12 GB out of the 100GB pool. But in reality the servers are using 300GB of disk, because they are unaware that there is no space issues.

Where the fun starts is when you start loading data in to those disks. Lets say the second server was going to be a small database server, so we load Oracle and create some table spaces. We end up using up about 40 of the 50GB alloted to it. (So now we are up to 48GB of disk used in the 100GB pool). Still technically ok, but with only 52GB free we need to really start worrying about the disks and the servers. The fun begins when we start loading data on to the server with the 200GB disk. Once we get up to 52 GB used in this we have some problems. Basically all the servers will start reporting write errors or other weird issues. The server admin can’t figure out what the problem is because when he looks at the servers he see plenty of “free” space on the servers. When stuff gets really weird is when processes start dying and they won’t start when you try to restart them (maybe they write to a log file, etc). So the first thing the Server admin will try to do is reboot the server. This is where all hell breaks loose…

See when you start rebooting servers it can’t flush out writes to the disk because there is “no” space left to write to. So the file-systems end up becoming corrupted. When the server reboots, it will try to write more to the disk thinking that it has plenty of free space, but again can’t, so stuff starts hanging. So of course a reboot is done again, and again, etc…

So now you start seeing write errors showing up every where on the other servers, and from the looks it may be a SAN issue, like the disk has disappeared. So you call the SAN admin only to find out that you have been thin provisioned.

This my friends is why thin provisioning is bad and should NEVER be used. Yes it may save you some money on disk, but what you save there will be wasted when you have down time rebuilding servers and restoring data.

VMWare and Sun Gigaswift Ethernet cards

I began setting up my new sun server and san at home the other day.. (Picked up a Sun V20Z and a Sun T3 SAN Disk array very cheap)… Because I am going to be doing some IPv6 testing as well, I installed a Sun Gigaswift (aka Sun’s ce, Cassini Ethernet) card in to the machine along with the fibre channel card. I put the VMWare vSphere 4i cd in and went on with the install. But didn’t really pay attention that it did not see the CE card, just the two broadcom cards.. So I went ahead any ways thinking I will fix it later. But it seems that there is no drivers on the interweb for the Sun CE card for vmware? If any one knows of a place to get them let me know? Otherwise I will have to find a new card to use in it’s place.

VMWare Fusion tip

For some reason last night my Windows XP image in VMWare fusion locked up during a update to the Microsoft Security stuff. I tried doing the “Virtual Machine -> Shutdown” which looked like it was going to work. However Windows just set for ever on the “Shutting Down Windows” screen. Well if you hold down the Option key when you click on the “Virtual Machine” menu item, the word “Force” shows up in front of all the options. This is the equivalent of pressing the power button when you click “Force Shutdown”. With out that, VMWare is trying to do a “nice” shutdown. So I forced a “reset” and everything came up fine… Hope this helps some one else who is “hung”

Sun Ray 5 Early Access part 2

I finally got time again to start playing with Sun Ray 5 Early Access software. Now my current setup probably should not be used for any type of test more than simple single/dual user testing. But I did not want to test the software on the current working server. So I decided to install it in a VMWare image on my Mac Pro. The Mac Pro is more than suited to handle it and had plenty of free memory/processor/storage to use so there was no contention (I gave the VM 4 processors and 8 gig of ram)..

The kicker was getting VMWare Fusion to actually allocate the network cards the way I needed them. I gave the VM 2 nics (the Mac Pro has 2), however the only options that VMWare Fusion let you do is NAT, Host-Only, and Bridged. None of which will work if I want a private network for the Sun Ray’s. To fix this you will need to go and edit some files that VMWare Fusion uses. What I had to do was the following:

1. Open up the Terminal app
2. Edit the file /Library/Application Support/VMware Fusion/boot.sh

sudo vi /Library/Application\ Support/VMware\ Fusion/boot.sh

3. Comment out the following line:

"$LIBDIR/vmnet-bridge" -d /var/run/vmnet-bridge-vmnet0.pid vmnet0 ''

And then add 2 lines directly below that line, which tells vmware to bind the en0 physical device to the vmnet0 virtual device, same for en1 to vmnet2. Note you can not use vmnet1 or vmnet8 as those are for NAT and Host-only connections.

"$LIBDIR/vmnet-bridge" -d /var/run/vmnet-bridge-vmnet0.pid vmnet0 en0
"$LIBDIR/vmnet-bridge" -d /var/run/vmnet-bridge-vmnet2.pid vmnet2 en1

Once done, do the following:

sudo /Library/Application\ Support/VMware\ Fusion/boot.sh --restart

Now go in to your Mac System Preferences and configure the second network card for a private subnet (i.e. 192.168.128.0/24, and set the IP to be something like 192.168.128.254).

Now make sure that your VM is NOT started and is in a powered off state. Go in to the VM and under the settings for that VM add another network adapter, make sure it is selected as “Connected” it doesn’t matter what the device is configured to as we will change it later to an option that is not shown in that list.

Now you need to change the .VMX file so that it can use the new network device. So go in to the directory where you have your VM’s at and then cd in to the machine.vmwarevm directory (For example mine is called SolarisDev.vmwarevm)

Once in there you will need to edit the vmx file, mine is called SolarisDev.vmx. The first thing we are going to change is the ethernet0.connectionType property. Right now it could be any of the ones listed (host-only,bridged, nat). But we are going to change this to “custom”:

ethernet0.connectionType="custom"

Next find the entry for ethernet0.vnet, if it doesn’t exist create it and make it look like the line below. If it does exist and doesn’t match that below, make it match that:

ethernet0.vnet = "vmnet0"

Now we need to do the same for the ethernet1 entries. The only difference to what is above is vmnet0 changes to vmnet2. Once the changes are made you can save the file and start up your Solaris VM. Now what ever network is on your en0 connection on your Mac should be what is connected to the e1000g0 network on the Solaris side. I used the e1000g0 as the “public” side of the Sun Ray server. The e1000g1 interface will be what ever is connected on the en1 connection on the Mac side. I used this adapter for the private Sun Ray Lan.

You should be able to finish following the instructions on the Sun Ray wiki now and get everything configured.

To test the soft client, I set up LAN Connections on the Sun Ray Server:

/opt/SUNWut/sbin/utadm -L on

I then installed the soft client in another VM on the same machine that only had access to the public network. You then can tell the soft client what the IP of the Sun Ray server is and it will connect. Pretty darn cool that the soft client works with minimal config.

This can probably be done on a MacBook Pro as well, if you use the wireless connection as the public side and the wired as the private side. Nice way to do a little demo in one computer.

For reference here is what my network section of the .vmx file looks like :

ethernet0.addressType = "generated"
ethernet0.connectionType = "custom"
ethernet0.generatedAddress = "00:0c:29:f8:29:3b"
ethernet0.generatedAddressOffset = "0"
ethernet0.linkStatePropagation.enable = "TRUE"
ethernet0.pciSlotNumber = "32"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.vnet = "vmnet0"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet1.addressType = "generated"
ethernet1.connectionType = "custom"
ethernet1.generatedAddress = "00:0c:29:f8:29:45"
ethernet1.generatedAddressOffset = "10"
ethernet1.linkStatePropagation.enable = "TRUE"
ethernet1.pciSlotNumber = "35"
ethernet1.present = "TRUE"
ethernet1.virtualDev = "e1000"
ethernet1.vnet = "vmnet2"
ethernet1.wakeOnPcktRcv = "FALSE"

References:
Sun Ray Software 4.2 Wiki: http://wikis.sun.com/display/SRSS4dot2/Home