Category: Interesting

Posted on

Cash for Cache

I decided to build a new VMware host for my home “lab” last week to replace the HP workstation I had been using. (The real motive was to turn the HP workstation in to a large NAS since it has 12 SATA ports on it, but more on that later.) So off to part out my new server. What I ended up purchasing was the following (Prices as of 3/24/2015 in USD):

 

The plan was to set this system up with VMware vSphere 6 and then migrate everything from my VMware 5.1 system to this. So I began building it as the parts arrived last friday night. Everything was going swimmingly until I forgot that the LSI2308 SAS/SATA RAID card doesn’t have any cache. What I found was that the 2 480GB SSD drives in a RAID 1 on that card were fast, extremely fast, as in I could boot a Windows 7 or Windows 2012R2 VM in about 3 seconds. However the 2 2TB SATA drives that I made a RAID 1 on there were slow as hell. (Same as the issue I was having with the HPXW8600 system.) I had originally thought it was just the RAID rebuilding, so I left it at the RAID bios over night rebuilding the array.

Well after leaving it at 51% completed and going to bed, waking up 8 hours later and it was only at 63%, I knew that I would never be able to use the SATA drives as a hardware mirror on that device. So I powered it down and disconnected them from the LSI2308 and moved them over to the Intel SATA side of the motherboard. This is where things get interesting, as I really wanted to have a large 2TB mirrored datastore for some of my test vm’s that I didn’t run 24×7 (the ones I do are on the SSD RAID 1.) In order to achieve this I had to do some virtualization of my storage…

The easiest way I could get the “mirrored” datastore to work was to do the following:

  1. Install FreeNAS vm on the SSD drive (pretty simple a small 8GB disk with 8GB of ram, which would leave me 24GB of ram for my other VM’s.)
  2. On each of the 2TB disks, create a VMware datastore, I called them nas-1 and nas-2, but it can be anything you want.
  3. Next create a VMDK that takes up nearly the full 2TB �(or smaller in my case, I created two 980GB VMDK’s per each 2TB disk.)
  4. Now present the VMDK’s to the FreeNAS VM.
  5. Next create a new RAID 1 volume in FreeNAS using the 2 disks (or 4 in my case) presented to it.
  6. Create a new iSCSI share of the new RAID 1 volume.

Now comes the part that gets a little funky. Because I didn’t want the iSCSI traffic to affect my physical 1GB on the motherboard I created a new vSwitch but didn’t assign any physical adapters to it. I then created a VMkernel Port on it and assigned the local vSphere host to it with a new IP in a different subnet. I then added another ethernet (e1000) card to the FreeNAS VM and placed it in that same vSwitch and assigned it an IP in the same subnet as the vSphere host.

With the networking “done”, it is now time to add the iSCSI software adapter:

  1. In the vSphere Client, click on the vSphere host, and then configuration
  2. Under Hardware, select Storage Adapter, then click Add in the upper right.
  3. The select the iSCSI adapter and hit ok. You should now have another adapter called iSCSI Software Adapter, in my case it was called vmhba38.
  4. Click on the new adapter and then click Properties
  5. Next I clicked on the Dynamic Discovery tab and clicked Add.
  6. In the iSCSI Server address I ended the IP address I made on the FreeNAS box on the second interface (the one on the “internal vSwitch”)
  7. Click ok (assuming you didn’t change the port from 3260)
  8. Now if you go back and click Rescan All at the top, you should see your iSCSI device.
  9. Now we just need to make a datastore out of it, so click on Storage under the Hardware box
  10. Then Add Storage…
  11. Then follow through adding the Disk/LUN and the naming stuff.

You should now have a new iSCSI datastore on the 2 disks that were not able to be “hardware” mirrored. Using HD Tune in a Windows 7 VM on that datastore I got this:

HD Tune running in Windows 7

As you can see, the left side of the huge spike was actually the writing portion of the test, which got drowned out by the read side of the test. Needless to say the cache on the FreeNAS makes it read extremely fast. As an example a cold boot of this Windows 7 VM took about 45 seconds to get to the login screen from power on. However a reboot is about 15�seconds or less..

Now on the FreeNAS side here is what the CPU utilization looked like during the test:

FreeNAS CPU usage

You can see that is barely touched the CPU’s while the test was running. So lets look at the disk’s to see how they dealt with it:

FreeNAS disks

It looks like the writes were averaging around 17MB/s, which for a SATA/6Gbps drive is a little slow, but we are also doing a software raid, with cacheing being handled in memory on the FreeNAS side. The reads looked to be about double the writes, which is expected in a RAID 1 config.

The final graph I have from the FreeNAS is the internal network card:

FreeNAS Network

Here we can see the transfer rates appear to be pretty close to that of the disk side. This is however on the e1000 card. I have yet to try it with the VMXNET3 driver to see if I get any faster speeds or not.

While the above may not show very “high” transfer speeds, the real test was when I was transferring the VM’s from the HP box to the new one. Before I created the iSCSI datastore and was just using the straight LSI2308 RAID1 on the 2x 2TB disks, the write speed was so bad that it was going to take hours to move a simple 10GB VM. After making the switch, it was down to minutes. In fact the largest one I moved, was 123GB in size and took 138 minutes to copy using the ovftool method.

So why did I title this post Cash for Cache, quite simple, if I had more cash to spend on a RAID controller that actually had a lot of cache on it, and a BBU, I wouldn’t have had to go the virtualized FreeNAS route. I should also mention that I would NEVER recommend some one doing this in a production environment as their is a HUGE catch 22. If you only have one vSphere host and no shared storage, when you power off the vSphere side (and consequently the FreeNAS VM) you will lose the iSCSI datastore (which would be expected). The problem is when you power it back on, you have to go and rescan to find the iSCSI datastore(s) after �you boot the FreeNAS vm back up. Sure you could have the FreeNAS boot automatically, but I have not tested that yet and to see if vSphere will automatically scan the iSCSI again to find the FreeNAS share.

 

Looking to the future, if SSD’s drop in price to where they are about equal to current spindle disks, I will likely replace all the SATA hard drives with SSD drives and then this would be the fastest VMware server ever.

 

Posted on

Comcast vs TiVO Roamio

As most of the world knows TiVO released their new DVR called the Roamio. Which in all shapes and forms appears to be the most awesome DVR yet. With the ability to do 6 tuners and stream live TV to the TiVO Mini, it alone will save me hundreds of thousands of dollars in stupid hardware rental fees from Comcast. So before I put down nearly $1000 for the new Roamio and a lifetime subscription I decided to ask Comcast if their Morgantown, WV system would support it. (I had seen some people on the interwebs saying there were issues with some cable systems not supporting all 6 tuners yet.)

So the first place I went was to twitter to ask them (@comcastcares) if they supported it and if there were any hoops I had to jump through to switch it from my Premiere to the Roamio. Well they wrote back and said to contact [email protected]. So I sent them an email asking about the support for the Roamio and whether it was required for a tech to come out to do the install.

So I got the typical boiler plate email back saying they would review my concerns.. Pretty typical.

Today I got a call from their Executive relations group while I was at work. So I called them when I got home and here is roughly how the conversation went:

ER: Hello, I was calling to address the email you sent us.

ME: Ok, well I am looking at getting a Roamio and was wanting to make sure it was supported before buying it.

ER: Well does it support 3 cable cards? We only have cable cards that support 2 tuners, so if it doesn’t have 3 cable card slots then it probably won’t work.

ME: No it only has one slot for a M-CARD. (Thinking to my self, yeah if I had to get 3 cards that is extra money to you.)

ER: Oh, ok. well we didn’t even know that TiVO had a DVR that did 6 tuners. (Thinking well, they have had a 4 tuner one for a couple of years now.) I have some calls in to our warehouses to verify if we have a cable card that supports that many tuners, but right now the only thing we support is 2 tuners.

ME: Ok, well from what I was reading it is just a firmware issue.

ER: Hmm, hmm, hmm, ok, ok, ok (don’t have a clue what he was doing) {he then repeats about checking with the warehouse people}

 

He then addressed my issue with doing a self install and said yes you can do it, but you have to call them to activate it. (Which I knew but was confirming it again.)

I then brought up the issue where the website says that for each customer owned piece of equipment you should get a $2.50 credit to your bill. I told him I had 2 TiVO’s and therefore I should see a $5.00 credit on it. To which he explained that I do get the credit but it isn’t reflected on the bill. He then told me that the cable card fee is actually the same as the other box fees ($9.95), but they subtract the $2.50 from it (which is the “cost of the box”)� to make it $7.45 (which is the cost of the “service”).� I told him that the Comcast website doesn’t say that and even the paper that comes with the bill doesn’t show that the Cable cards are $9.95..

The funny thing was that I told him that my friend sees the $2.50 credit on his bill. He immediately said “well different parts of the country does billing a different way.” I sort of laughed and said “well, he lives 2 miles away from me. So your hypothesis doesn’t work.” He couldn’t figure out why mine didn’t show it but others did.

He ended the call with saying he would call me back once he hears back from the warehouse and whether they would or ever support a TiVO with 6 tuners. I said “well I sure hope you do as it is going to save me hundreds of dollars a year in rental fees.” He didn’t really say anything after I said that.

 

So long story short, TiVO has released something that is far superior to anything Comcast could ever offer their own customers. So now they are going to probably give out false information to make sure that customers don’t purchase the new Roamio. Just another reason why Comcast is evil, and making billions a year from people from hardware rental fees. Shit I have had 2 Scientific Atlanta 3100 standard def boxes since 2001. The interface is slow, they put ads on the guide screen, and I have paid probably close to $1,500 in rental fees on them since then.

Posted on

set default DISK$DATA:[OS.EOL]

Seeing the news the other day of HP is discontinuing OpenVMS brought back some memories. Mostly of all the different operating system’s I have used that are no longer around or have changed a lot. Back in my undergrad days, one of the first OS’ we used to program on was OpenVMS on a VAX. It was for an engineering class and we had to use Fortran 77. I remember our quota’s used to be about 5MB in size, which at the time was “huge”.

So to list some of the other OS’ I have seen gone by the way side (and other computer related items that had a huge effect on where I am at today.)

1. OpenVMS (used it between 1994 and 1999)

2. VM/ESA (not really gone, now called zOS, but haven’t touched it since about 2001)

3. Gopher (this is the “original” web…)

4. IRIX (SGI’s UNIX platform. I still have 2 SGI Indy’s and a copy of 5.3 and I believe 6.5, but haven’t had them on in years, maybe a vacation project some time.)

5. SunOS (not Solaris, but the old BSD based SunOS 4.x) my things have changed in the 19 years that I have been doing Solaris work

6. mSQL (mini sql). Not really gone, but surpassed by other’s (mysql, mariadb, etc). I used msql as my first PHP/FI + DB + Apache installation on a Solaris 2.6 box. I wrote a network management application that controlled DNS, DHCP, etc for university dorm connection management.

7. Trumpet Winsock, for the good old Windows 3.1 days when you needed a way to do TCP/IP over modem or ethernet.

8. NCSA Mosaic, the web browser that is credited with popularizing the WWW. Used to use this on some old SGI and DEC machines.

9. ULTRIX, DEC’s version of UNIX. It was on a lot of DECstations in the Engineering department and one computer in the CS department. Used to have a teacher that made us make sure everything compiled on it vs the Solaris or Linux hosts.

10. AltaVista, Search engine to use before Google came around. Now it is just a “front end” to Yahoo search 🙁

11. Atari 400, used to have one of these at the grandparents house to tinker on.

12. Commodore 64, used to have a couple of these when I lived at home. We I learned some BASIC programming. (Later went on to try Visual Basic programming on Windows 3.11 on a 80486 DX4-100 AMD PC.)

13. BeOS, was a really neat idea, excellent media support, unfortunately it was around the time of the PC vs Mac battle so getting buy in was hard.

 

This all also brings back memories how of rudimentary computers were back then and the lack of security. There was no SSH, everything on the VM, OpenVMS and UNIX machines was done through telnet. There was no SSL, and people didn’t think twice about typing in a credit card number on a web site.

I also remember doing web surfing with Lynx on various UNIX systems. And what goes along with Web browsing then email, the first GUI email client I remember using was Pegasus Mail on a Novell Netware based mail system. Once people started doing POP3 mail, people switched over to Eudora Mail. Which I used for a while, but not a lot. I for some reason stuck with Pine a text based mail reader, mostly because I used it on the server that received all the mail.  (And to totally geek out, there were times were I would telnet in to the POP3 port on the mainframe and read my mail by issuing the pop commands by hand.)

As for personal computers, I have had quite a few since my first one. My first computer only had a 40MB hard drive in it. It was a KLH brand 80386 SX 16 that I bought from Phar-Mor. I think I had it maxed out a 4MB of Ram which at the time was huge. I remember trying to play some game on it (I keep thinking it was SimCity, but may be wrong) and it needed more Video RAM cause it only came with 128K of video ram. So I had to buy more to up it to like I think 384K.

As a list of what I have had or still have, here goes:

  1. KLH 80386SX 16MHz – First, no longer have it, came with a 40MB hd, and a EGA 15inch monitor.
  2. AMD 80486DX4 100MHz – Used this to run Windows 3.11, Linux and later Solaris 2.6. It came with a 320MB hard drive. I later paid close to $300 for a 1.6GB hard drive for it. It had a VESA Local Bus video card and a Sound Blaster 16 sound card. No longer have this computer.
  3. Intel Pentium II 266MHz – Bought this in 1997 from a company called Vektron (who later went out of business, like all fly by night computer places back in the early days). It had 32MB of ram and a 500MB hard drive. It ran Windows 95, Windows NT, BeOS, Solaris and Linux. (I had bought bigger and more hard drives later, just can’t remember what all was in it.) I actually still have this machine, it’s most recent use was as a router for my home network running Solaris 10 with 3 NIC’s (one on Comcast, one on Verizon and one on my home network). The hard drive died in it a couple of years ago, so I turned it off, it is still sitting in a rack thought.
  4. Sun SPARCstation 2 – This was my first “workstation”. I got it second hand from a friend’s company. It was where I cut my teeth on Solaris. It ran Solaris 2.5 when I got it, and over the years I upgraded it to Solaris 7. Ironically it only had a 40MHz processor and 64 MB of ram. It had 2 huge external 800MB disk packs and a freakishly heavy 17 inch Sony monitor that used 13W3 connector with BNC ends. I still have this one, but the disk packs both died, so it hasn’t been on in years.
  5. Sun Ultra5 – 360MHz, 128MB of ram. One of the first “IDE” based lower end workstations from Sun. I still have this, but I think the power supply is bad, as I can’t get it to turn on :(. When it ran, I had Solaris 9 on it.
  6. SGI Indy – 2 of these 133MHz with 96MB of ram. One of the coolest “workstations” I ever owned. I believe they both still run, but haven’t been on in years. One ran IRIX 5.3 and the other ran IRIX 6.5
  7. Dual Intel Pentium III 933MHz – Bought this in probably 2001 I think. It is huge, it was a full tower with onboard IDE raid (which only works with Windows because of driver issues.). Right now it has 1.5GB of ram in it, ~2TB of disk and runs Solaris 10 with 7 zones running on it.
  8. IBM Thinkpad i1100, Celeron 500MHz. This one was given to me as a result of work being done for a company. It was my first laptop, and I still have it today. However it’s stats are very underwhelming by today’s point of view. The monitor is an LCD one, but not TFT, so that means there are all kinds of shadows and the picture isn’t crisp. It also only had a 5GB hard drive in it. Which means after installing Windows 2000 on it, there was only maybe a gig free. It also had no floppy drive, and no network ports. So I bought a Linksys WAP11 back in the day (probably in 2002 when I got this) for upwards of $300 so I could have wireless internet on it.
  9. ThinkPad A22p – 900MHz Pentium III. I bought this one as a replacement of the first. Side by side this one is HUGE, as it has a 15 inch display that runs at 1600×1200. It also had a 30GB hard drive (which was split in to 3 10GB chunks, one for Windows XP NTFS, One for Solaris 10 and one for FAT 32 to share files between the two OS’).
  10. AMD 3600+ – Got this one in 2005. It currently runs a combination of Windows XP and Windows 7. Has about 2.5 TB of disk on it.
  11. Sun X2100 – This server. Currently running Solaris 10, with a surprisingly small 160GB of disk with 4 zones on it.
  12. Apple MacBook Pro 2.0GHZ – This was one of the first Intel based Mac’s that was released in 2006. It had a Dual Core 2.0 GHz processor, 2GB of ram an a 100GB hard drive. It did have it’s issues (mostly battery and power adapter ones), but it ran solid for about 5 years. In the fall of 2011 the logic board “died” and it will no longer run in full “user” mode. (I think it is the graphics part of the board.) Still have it hoping for a price drop of replacement boards some day.
  13. Apple Mac Pro – Dual Quad Xeon 2.8GHz with 10 GB of ram. This is the best desktop I have ever had. It is fast and quiet. Right now I think I have close to 13GB of disk on it (both internal and external). I also dual boot it with MacOSX 10.8 and Windows 7 (for a couple of games)
  14. Apple MacBook Pro 2.8GHz iCore7 – the replacement for the one that died above. It is hands down probably 4 to 8 times faster than the 2.0 one that I had before.
  15. Sun V20z – Used to run VMware ESX 3.5 with a Sun T3 fibre connected Disk array. The V20z is fully loaded with processor (2) and ram (16GB). One loud machine…
  16. IBM X3550 – Dual Quad Xeon with 8GB of ram. Used to run VMware vSphere 5.0. Used it to play around with doing virtualization of my house servers. Unfortunately it is too loud to leave running 24×7, so it is only on when needed.
  17. HP XW8600 workstation – Dual Quad Xeon with 16GB of ram. This is my “production” VMware server at  home. It has 3 TB of disk it in and runs probably 11VM’s all the time. It was used to replace the noisy IBM one, and it is super quiet.

As for a list of operating systems I keep current with, it is many and with VMware it is possible to have “test” versions of everything sitting around which helps a lot. Basically the following is what I keep running:

  1. MacOSX 10.7 and 10.8
  2. Windows XP, 7, 8, 2008, 2008R2, 2012
  3. CentOS 6.3
  4. Solaris 10, 11
  5. OpenIndiana 151
  6. pfSense (freebsd)
  7. OpenBSD
  8. Ubuntu Linux

Well that is about enough nostalgia for tonight. Trying to think of other things to put back on the blog to start updating it more often. If you have any idea’s leave a comment (open for 30 days only to keep the spammers away..)

Posted on

How R-Studio for Mac saved my ass

I have an external Seagate Firewire 800 drive that I use on my Mac Pro that has over 700GB of VMware images on it. Pretty much anything I work on I have an image on there, everything from a Windows XP client to Microsoft Exchange servers, and Solaris, Linux and the such. I have had the drive for a couple of years and it has always been rock solid and fast too. (I bought it when Windows 7 screwed up my internal drives.)

Well today I  was wanting to run a VM off of that drive to test something, and noticed that the drive did not appear on my Desktop. Weird, it as plugged in, the light was flashing, but no icon. Hmm, where the hell did it go? So I unplugged it and plugged it back in. Still no go. So i tried switching power supplies, still no go. Then if I left it sit for a while I would get the error that it could not use the drive, or that it needed initialized. Holy crap, that isn’t good.

I popped up the command prompt, diskutil would list that there was a drive there, but no partitions on it. The gui Disk Utility would see the disk, and again no partitions and wouldn’t let me do anything with it. gpt wouldn’t let me read it. So I thought to my self, did Windows 7 screw the disk up again (it was working the other day when I had booted in to windows, but forgot to unplug it before doing so 🙁 ). So I booted in to Windows 7, it could see the drive but said it was unformatted. Double shit. So back to MacOSX, I went out searching for some data recovery programs. The first one was Data Rescue 3 while the graphics were gimmicky it didn’t even look like the demo version could even see 1 file on the drive. So I uninstalled it and started looking for another program.

In the past I have used the R-Studio for NTFS & FAT and both have worked wonders. I did a google search, and they now have a Mac version.  Now we are talking! So I downloaded the demo, and with in about 2 minutes of starting it, it showed me the entire disk and all the files that were on it. But since it was a demo it would only restore 10 files under 64kb.. So I bought it for $79.99. 2 minutes after buying it, it was busy restoring the files to another external 2TB USB drive. 6 hours later, 100% of my files were restored from the dead firewire drive, and my VM started up just like nothing had happen.  Needless to say it saved me hundreds of hours of reinstalling and setting up my VM environment. Now I just need to go get another drive to make a backup of this one.

 

So if you are ever needing to restore MacOS, HSFS, NTFS, FAT, UFS, EXT file systems, definitely check out r-tools technology and their R-Studio products. http://www.r-tt.com/  For $79.99 it was more than worth it!

 

 

Posted on

What happens in Vegas, should have stayed in Vegas

Last week, I went to VMworld 2011 in Las Vegas. The conference was great, 20,000+ people all there and focused on one thing, VMware and every product they offer. This was my first time at the VMworld conference, and hopefully will get to go again some time in the future. The main reason I went was because of the recently released vSphere 5 and seeing what all it offered and what all was changed. Needless to say, there are many cool new features that were added, I am only going to mention a few here, but the full list is available in this PDF.

The first cool feature is : Auto Deploy. Simply said, (wish they would have chose a different name) it is PXE boot of the vSphere image from a TFTP server, so no local disk is required to “run” vSphere. For example if you have a “shit ton” of blades and don’t want to have to go update and install all of them, just get their MAC address, setup the host in DHCP with a couple of DHCP options to tell it where to boot from and have the blade boot from the network. It will download the image from the TFTP server and run automagically. Once up and running all config is stored in vCenter 5 (a requirement!). So need to upgrade your hosts? Just reboot them after updating the image. A couple of notes for this, make sure you have logging set up to go to your syslog server, and that you set up the Dump Collector incase of a PSOD.

Another cool feature is: vSphere 5 supports Apple Xserve servers running OS X Server 10.6 (Snow Leopard) as a guest operating system. This is because vSphere now supports UEFI “bios”. Now “supposedly” this does not require Xserve’s (since Apple no longer sells them), but it “requires” them because of Apple’s EULA for use of Mac OS X.

There are many other features that have been upgraded, or are new.. Too bad the conference wasn’t a little longer, as the amount of sessions I wanted to go to were greater than the amount of time I had available to go to said sessions. (I.E. only one instance of a session and 2 sessions I wanted to see were at the same time.)

The Hands on Lab area was “freaking huge”. There were over 800 workstations set up where you could do 1 of 16 LABS (you could do more, just had to stand in line, I was only able to do 1 in the week I was there). Ironically each “lab” station was a Wyse “chubby client” that had dual monitors so you could rdesktop to some windows XP and servers to do the work. The HOL area, sort of reminded me of the CTF area at DefCon, a huge big room, with nearly no light what so ever and hundreds of thousands of screens.

The most interesting part of the conference is that they have grown so big, that next year they have to go to San Francisco to host the event, as there is no place in Vegas that is big enough to house them. This year it was at the Venetian with some spill over to Wynn. They also had the Sands Expo hall, which is connected to the Venetian. The “dining” room was 1.5 million sq ft alone, you could barely see from one end to the other.

I will have to say out of the many conferences I have been to by different vendors, I will have to say so far VMware has been the best. Some of the things that has made it stand out from the rest:

  1. Food, while not “the greatest ever” it was far better than I have had at other places. They gave us breakfast and lunch every day. In addition the break periods between sessions had different items every day. One day they had fresh hot made pretzel sticks with cheese and different sauces.
  2. Hang out area: Most conferences if there is “downtime” you usually end up either walking around or going back to the hotel. VMware set up a “hang space” where they had a basketball court, badmitten court, huge chess sets, fake grass to sit on in front of a big screen (like 20+feet) TV. A Twitter vMeetup place, where you could meet other people that you have met on twitter.
  3. Scheduled sessions. While I was skeptical at first on “pre-registering” for the sessions you want to attend, I think in the end it was a good idea, as it “guaranteed” your spot in the session as long as you showed up 3 minutes before it started. (There were gaps between end and start, so you really had no reason not to be there.)
  4. Group Discussion: in some conferences, I have seen “group discussion” be these “huge” groups where it ends up being a more Q&A session. VMware had group discussions, where there were maybe max 30 people in a room, each one had a clicker, and everyone voted on how the session went and it was a free form for questions. One of the best ones was the Oracle on VMware vSphere one. I learned a lot from that session.
  5. P.A.R.T.Y. : By far the best conference / vendor party I have ever been to. First was the food, you name it, they probably had it. I didn’t realize this till I had already ate a couple of slices of pizza. Then I saw a station where they were making fresh cut cheese-steak sandwiches, another was doing fresh made crab cakes. Like I said, name it, and it was probably there. In addition, a huge open bar (not that I drink, but it was there). So now that we got past the food, they had at least 4 different acts during the night. Two people doing fire tricks, then the openers was Recycled Percussion, which I didn’t realize who they were till I got back to the hotel room that night, but they were on the America’s Got Talent show, and previously had a show nightly in Vegas. The headliners were The Killers. They played for an hour and did all the “popular” songs along with some that I hadn’t heard before.
    This part of the party ended around 9PM. Which was the start time to the “after party” which was at the Venetian pool. I did not go to it, but it sounded like people had a bunch of fun there too.

So if you are still reading by now, you are probably trying to figure out the second part of the title “… should have stayed in Vegas”. Well, it seems that some time either on Sunday or early Monday morning I either sprained or got a stress fracture in my left foot. Needless to say, the 30+miles of walking I did, (cause my hotel was 2 miles away from the conference hotel, it is a damn long walk from Planet Hollywood to the Venetian even if you take the monorail when your foot it hurting like a Mofo) did not help it any. By the time I got home it was still hurting and I noticed that the top of my foot started to have some swelling and bruising. I just iced it on Saturday and Sunday, but as of today it was still hurting and didn’t seem to change much, so I ended up going to the doctor to have it X-ray’d. They said it didn’t show any fractures, but thought it was just a really bad sprain or a damaged ligament. So it is more ice, and a ankle air cast for a while. So that is what I “wish that it should have stayed in Vegas.”