Pano View of the Strip from Augustus Tower at Caesar's Palace in Las Vegas

Another year has past and another trip to Blackhat 2010 and Defcon 18 has been completed. Overall both conferences were great. However Defcon 18 (aka Barely Legal) did have its problems. One was the amount of new people who showed up. During the closing Ceremony DarkTangent asked who all was here for the first time, most of the people in the room I was in rose their hands. When asked if they would return next year, almost all of them stayed up.  The problem that this caused was that one, they ran out of Human badges (They only made 7,000 of them). Luckly I got one of them, and once again Kingpin did a awesome job making the badge this year. So there were thousands of people who received the “paper” badges. From what I have been reading there are estimates that there were between 10 and 12,000 people at the Con this year.

Because of the massive amount of people there, there were talks that I was unable to get in to that I really wanted to see. Hopefully they will be up on iTunes soon and I will be able to see them. The one big announcement that DarkTangent made was that this would be the last year for Defcon at the Riviera Hotel. Next year it is going to be at the Rio.. Which is Freaking awesome. In addition to being at the Rio, they will be doing 5 TV channels in the hotel that will be run to the rooms so that if one of the talks gets full, you can go to your room and actually watch it live on TV. I see this becoming a huge benefit and success for Defcon 19.

The one bad part about the trip, USAirways… We had originally set up our return flight to only have about a 1 hour layover in Phoenix. Unfortunately, USAir decided that was too short, and changed the flight so that it had a 4 hour layover. Which turned in a trip that if they still flew direct from being 4 hours to being almost 9 hours. My total travel time for my return  home, 16 hours… WTF.

The first thing we did was check the current one that was put in almost a week prior. Yup, still not activated. So we pulled it and put a “new” one in. Jason called the Comcast office and they “sent the signal” well we waited and chatted for a about 30 minutes, and it never got the signal. So another call back and this time a different person. He seemed to think he was “better” than the last person and that the last person didn’t know what they were doing. Well this guy didn’t either, as the signal he sent to the card was the “kill one” which basically inactivated the card. But we did not know this until another 30 minutes past and we called in again because no signal had been received.

So they had us put in the another card and they sent the signal again. Still no go, so they put the final card in. I am not entirely sure what they did on their side, but the card finally said it was activated (the woman on the phone kept talking to some other techs in the back about what to send to it to make it ‘work’.) Well it finally started “working” and I could finally get all the channels I pay for, (which is basically everything) so Jason left and I finally have full HD cable in the bed room. All told there were 4 cable cards used, 2 onsite techs, probably 6 phone techs and 2 online techs that worked on getting my new TiVo working.

First I went to their web site, yup no info on how much they cost, etc. So I opened a chat session with them and they told me it was free (government mandate) and that I could go pick one up at the local Comcast office. I asked if they had any in stock so I could make sure I got on, and they told me I had to call the local office to see. Ok, so the next day I went to call the local office, surprise, the local number is “disconnected” and tells you to call the 800 number..

So I call the 800 number, they then proceed to tell me that I can’t pick one up, and they are not free ($3 a month) and it also will cost me $16 to have them come put the card in the TiVo. Holy shit what a different story. So on my way home one night I stop by the local office to ask yet a third person. She tells me that I can’t pick it up, they have to come install it and now it is $25 for some one to come install it. Damn can we say they don’t have a single answer they give every one. So now I am pretty well pissed at them. I go home and get back on the chat (so I can log it all to later send back to comcast as a complaint). The sales person I talk to this time tells me yeah the card is free and I can go to the local office to pick it up. When I tell them that I tried to do that and they told me I couldn’t he all of the sudden changed his story. So I asked about the whole install fee thing and he said he could waive that and give me the card for free. I said ok, but I need it installed on the weekend because I am not wasting 8 hours waiting for some one to come to my house during the week.

The online tech scheduled my appointment for 12 ~ 4 PM today. I received automated calls from Comcast on Thursday and today around 11AM verifying that I would be here and available. Well around 3:50PM I get a call from Comcast saying “our tech is running really late (no shit sherlock) and won’t be there till 5:30PM. We will give you a $20 credit because we missed our window. Do you still want him to come?” To which I replied yeah… So 5:30 rolls around, no tech. About 5:40 he shows up and he brings the card in side. We plug it in and he calls Comcast in Pittsburgh to activate it. Well 15 ~20 minutes pass, no “hit”.. 1 hour passes no luck.. So he calls back and they tell him either the card is bad or it is just taking a while. So he leaves at 7:40PM… (Yes almost 8 hours waiting on this install).

I then run out to the market for a while and come home, and try to see if it got the signal, no such luck. After a quick shower I sit down and fire up the good old Comcast chat and sit on “hold” for 20 minutes. Finally “Roel” comes in and here is the transcript:

Me > Cable card not working in TiVo
Roel > Hello Me_, Thank you for contacting Comcast Live Chat Support. My name is Roel. Please give me one moment to review your information.
Me_ > ok
Roel > I see here that you have a concern or problem with your cable card, I understand the trouble that this has caused you and I want you to know how sorry I am for the inconvenience.
Roel > As your service representative today, I want you to know that your satisfaction is of my topmost priority and I assure you that we can resolve this issue together on this chat, Me.
Roel > To ensure the integrity of your account information, kindly provide me your 14 to 16 digits Account Number and the Last 4 digits of your Social Security Number.
Me_ > XXXXXXXXXXXXXXXXXXXXXXXXX
Roel > Thank you for that information.
Roel > Let me check that here on my end.
Roel > By the way, while waiting for your account to pull up, I will share you a feature of Comcast that you can truly benefit. Are you aware of the customer self-help on comcast.com?
Roel > Comcast.com has an extensive series of Frequently Asked Questions (FAQs) that cover all of our products. Customers do not have to sign in to access the FAQs. Quick steps to do it…Open a web browser window and go to http://www.comcast.com/. On the home page, the navigation menus are on the left side of the window and click on Customers then Help and Support.
Roel > I will be resolving this issue for you today, can you give me 2-3 minutes to work on this?
Me_ > yeah, self help is of no use..
Roel > Thank you for waiting.
Roel > May I ask if your Cable Card SN is this :XXXXXXXX ?
Me_ > let me check
Me_ > yes that is correct XXXXXXXX
Roel > That is okay.
Roel > May I ask what trouble shooting have you done so far?
Me_ > the card was “installed” today . the tech called the “activation” center and they supposedly sent a couple of hits to it. That was about 4:30 hours ago, it still has not activated. I have rebooted the tivo and it still has not received the activation signal
Me_ > key verification says “success” but provisioning says none, and auth state says disabled
Me_ > the card was also removed and re-inserted by the tech when he was here.
Roel > I see, thank you for confirming.
Roel > At this time, I am going to send a notice to our higher department since normally this should be done within 45 minutes, but the thing is the maximum time to wait for this when it is fully activated during the first installation would be less than 24 hours.
Roel > So I really do hope that you still have the patience to have it completely done for the process of installing.
Roel > Need not to worry, I am going to make a follow up on this so that you won’t have to wait for that long.
Me_ > yeah, just none of the numbers are changing like the person who the tech talked to on the phone.
Roel > I see, I’m sorry to know that and need not to worry about this anymore.
Roel > I am making sure that your cable card will be activated less than 24 hours.
Roel > What I am hoping for is that you will still have the patience for that.
Me_ > yeah.. already wasted 9 hours today waiting, guess a couple more isn’t going to hurt.
Roel > I am glad I was able to assist you by sending a notice to our higher department for the follow up of your activation process..
Roel > It has been my pleasure serving you today and I truly appreciate your understanding and cooperation.
Roel > Do you have other concerns for me today?  I will be glad to assist you further.
Me_ > no that is all.
Roel > It is with gratitude to have you on this chat and I appreciate the opportunity you’ve given us today to resolve your cable card activation process concern.  Enjoy the rest of your day and take care.
Roel > Do you want to watch full TV shows and movies online? Go to http://www.fancast.com. Thank you for choosing Comcast as your cable TV provider and have a great day! Comcast appreciates your business and values you as a customer. Our goal is to provide you with excellent service. If you need further assistance, you can chat with one of our Customer Support Specialists 24 hour a day, 7 days a week at http://www.comcastsupport.com/videochat. To close this chat, please click the end session button at the top of your chat window.
Roel > Analyst has closed chat and left the room

Based on the above, absolutely nothing of use occurred.. Half the stuff he said didn’t even make sense.. And nothing has been fixed with the cable card.. So I guess I will wait some more time and see if it does activate..

Now here are some things that piss me off about Comcast today.

1. The tech’s don’t have spare cable cards with them. Why the hell would you not let your techs carry spare cards. This is like not having a spare wheel in your car. So now if this “activation” does not work I have to waste another 8 hours waiting on a tech to bring a card that may or may not work. (My friend took 3 techs coming out to get his done.)

2. You don’t have any transparency in anything you do. None of  your people give the same answer as the others.

3. You are greatly understaffed in the Morgantown area. When the techs show up almost 2 hours late… get a clue.

4. Your tech’s used my phone to make long distance calls to Pittsburgh and Massachusetts . (over 43 minutes of them) You going to pay me for that?

This weekend I started looking at the PCA tool, since I have seen a lot of people mention good things about it. So off to my test machine and I installed a new VM with Solaris 10 10/09 ( update 8 ) in it. After the install was finished using a ZFS root, I decided to set up a PCA proxy server on another machine. The purpose of the PCA Proxy server is that it will be the one with access to the Internet to download the patches from sunsolve. It was extremely easy to do this, (in fact I have it running in a zone on my main server.)

1. Created a new plain zone (can be on anything, but I wanted to keep it seperate).
2. Configure the apache2 instance on the machine, by copying the /etc/apache2/httpd.conf-example to /etc/apache2/httpd.conf
3. Edit the httpd.conf and change the line that says “Timeout 300″ to be “Timeout 1800″. You need to make it at least 1800, if not more depending on the speed of your Internet connection. At 22Mb/s 1800 was ok for me.
4. Create a directory /var/apache2/htdocs/patches, make it owned by webservd:webservd and 755 as the permissions.
5. Download and save a copy of pca in /var/apache2/cgi-bin and call it pca-proxy.cgi. Make it owned by webservd:webservd and 755 as the permissions.
6. Create a file in /etc called pca-proxy.conf. In it place the following:
   xrefdir=/var/apache2/htdocs/patches
   patchdir=/var/apache2/htdocs/patches
   user=sunsolveusername
   passwd=sunsolvepassword
7. In order to make the proxy run a little faster on the first use, I decided to download and “cache” the latest security and recommended patch cluster. (You don’t need to do this, but if the patches are missing the pca proxy server will download them. Considering my machine needed 156 patches, this was faster…) Once the recommended and security patches were downloaded, I placed them in a temp place and unzipped the cluster. Once the cluster is unzipped, I needed to make zip files of each patch (so that the pca client can download the zip file). To do this, I went in to tmp/10_x86_Recommended/patches and ran the following:
   for i in `cat patch_order`
   do
   zip -r $i $i
   done
8. Once the zipping is done, move all the patch zip files in to the /var/apache2/htdocs/patches directory.
9. Start up the apache2 service “svcadm enable apache2″
10. Now it is time to configure the client, copy the pca script to the client machine and place it some place, I used /root.
11. Next create a config file /etc/pca.conf in it with the following:
    patchurl=http://pca-host/cgi-bin/pca-proxy.cgi
    xrefurl=http://pca-host/cgi-bin/pca-proxy.cgi
    syslog=local7
    safe=1

    The first two lines tells pca where to find the patches and the patchdiag.xref file. The syslog line tells it to log all activity to local7 syslog facaility. The last line “safe=1″ means: Safe patch installation. Checks all files for local modifications before installing a patch. A patch will not be installed if files with local modifications would be overwritten.

12. Now that the config file is created, make sure that syslog is set to handle local7 info, I have mine set to local7.info going to /var/adm/local7.log. PCA will log the patch installation stuff to that log (i.e.:
    Apr 11 17:10:50 zfstest2 pca: [ID 702911 local7.notice] Installed patch 124631-36 (SunOS 5.10_x86: System Administration Applications, Network, and C)
    Apr 11 19:07:04 zfstest2 pca: [ID 702911 local7.notice] Failed to install patch 118246-21 (comm_dssetup 6.4-5.05_x86: core patch) rc=15

Now comes the part that makes ZFS worth using… We are going to create a new “boot environment” and then patch that environment”

1. First we need to create a new BE;
   lucreate -n p20100411

   The p20100411 can be anything, I used today’s date since I patched the machine today.. Makes it easy to remember when the last time the machine was patched.

2. Now we need to mount it
   lumount p20100411 /.alt.root
3. Now we can start patching;
   pca -i -R /.alt.root
4. Because I cached most of the patches locally on my pca proxy, it should not take too long for it to download, unzip and install the patches in the alt root
5. Once the patching is done, it will give you a summary line telling you how many patches were downloaded and installed:
   Download Summary: 156 total, 156 successful, 0 skipped, 0 failed
   Install Summary : 156 total, 156 successful, 0 skipped, 0 failed
6. Now we need to unmount the alt root and activate it to boot:
   luumount p20100411
   luactivate p20100411
7. Now just reboot the machine. You MUST use init or shutdown, if you don’t then it won’t boot in to the new boot environment. I use
   shutdown -g0 -i6 -y
9. Depending on how long it takes for your machine to boot, when it comes back up it should be on the new ZFS file system:
   bash-3.00# df -h
   Filesystem             size   used  avail capacity  Mounted on
   rpool/ROOT/p20100411    49G   6.6G    38G    15%    /
10. Now you can run that new patched system for how ever long it takes to verify your patches didn’t break anything. Once you are sure everything is ok, then you can delete the old install, in my case:
    ludelete s10x_u8wos_08a

    This should let you recover a little bit of space. In my case it was about 1.5 gig.

The only thing left is to set up a bunch of scripts to do “pca -l” about once a month to see what patches need installed and to log that. PCA has a lot of other functions than I went over here, in a couple of words, it seems to be kick ass. On top of that it is free! The ability to create new BE’s will definitely hope any one with the right amount of disk space be able to keep their system up to date.

One Tip, make sure you watch the output of the luactivate command. This is what is displayed:

**********************************************************************

The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.

**********************************************************************

In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:

1. Boot from Solaris failsafe or boot in single user mode from the Solaris
Install CD or Network.

2. Mount the Parent boot environment root slice to some directory (like
/mnt). You can use the following command to mount:

     mount -Fzfs /dev/dsk/c1t0d0s0 /mnt

3. Run  utility with out any arguments from the Parent boot
environment root slice, as shown below:

     /mnt/sbin/luactivate

4. luactivate, activates the previous working boot environment and
indicates the result.

5. Exit Single User mode and reboot the machine.

**********************************************************************