Archive

Posts Tagged ‘Work’

Interseting Sun T2000 crash

October 10th, 2006
Comments Off

Got an email tonight from our operations group that one of our mail servers was not responding… So I looked at the /var/adm/messages and see this in it..

Oct 10 20:46:42 mailserver unix: [ID 836849 kern.notice]
Oct 10 20:46:42 mailserver ^Mpanic[cpu4]/thread=30002e89c40:
Oct 10 20:46:42 mailserver unix: [ID 165833 kern.notice] CONN_DEC_REF: connp(6001918b940) has ref = 0
Oct 10 20:46:42 mailserver unix: [ID 100000 kern.notice]
Oct 10 20:46:42 mailserver genunix: [ID 723222 kern.notice] 000002a101cb9380 ip:squeue_drain+2e0 (6001dba5180, 10, 3101fcb3, 6000115fc00, 0, 0)
Oct 10 20:46:42 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000000000000011 000006001429bc80 0000000000000002 0000000000000000
Oct 10 20:46:42 mailserver %l4-7: 00000000018a5a58 000000007bea2a90 0000000000000000 000006001918b940
Oct 10 20:46:43 mailserver genunix: [ID 723222 kern.notice] 000002a101cb9440 ip:squeue_enter+338 (6000115fc00, 6001dba5180, 7beaa54c, 30007e2ed00, 0, 6001927e150)
Oct 10 20:46:43 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 000006001429bc80 0000000000000000 000006001429bc80
Oct 10 20:46:43 mailserver %l4-7: 0000000000000002 0000000000000001 000000007beab790 0000000000000000
Oct 10 20:46:44 mailserver genunix: [ID 723222 kern.notice] 000002a101cb94f0 unix:putnext+218 (6001927e248, 6001927e150, 6001dba5180, 100, 6001c8f0e28, 0)
Oct 10 20:46:44 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 0000000000000000 0000000000000000 00000000000055a0
Oct 10 20:46:44 mailserver %l4-7: 000000000000010d 0000000070042a68 000000007beab790 fffffd5efe34c000
Oct 10 20:46:44 mailserver genunix: [ID 723222 kern.notice] 000002a101cb95a0 genunix:strput+1d8 (3000711c028, 0, 2a101cb9928, 2a101cb9860, 0, 0)
Oct 10 20:46:44 mailserver genunix: [ID 179002 kern.notice] %l0-3: 000006001c8f0e28 0000000000000000 0000000000000000 0000000000000000
Oct 10 20:46:44 mailserver %l4-7: 000002a101cb9860 000000000000c000 0000000000000000 0000000000000000
Oct 10 20:46:45 mailserver genunix: [ID 723222 kern.notice] 000002a101cb97a0 genunix:strwrite+1f4 (60012321440, 2a101cb9928, 600035c11e0, 1000000, 6001c8f0e28, 3000711c0a8)
Oct 10 20:46:45 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000000000000003 0000000000000850 0000000000000000 000003000711c028
Oct 10 20:46:45 mailserver %l4-7: 000002a101cb9860 000000000000c000 0000000000000001 0000000000000000
Oct 10 20:46:46 mailserver genunix: [ID 723222 kern.notice] 000002a101cb9870 genunix:writev+350 (a, 30007280e10, 5, 3, 3, 1)
Oct 10 20:46:46 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000060012321440 0000000000000000 00000300070811a8 0000000000000000
Oct 10 20:46:46 mailserver %l4-7: 0000000000000005 0000000000000000 00000000000000b7 0000000000000000
Oct 10 20:46:46 mailserver unix: [ID 100000 kern.notice]
Oct 10 20:46:46 mailserver genunix: [ID 672855 kern.notice] syncing file systems…
Oct 10 20:47:16 mailserver unix: [ID 836849 kern.notice]
Oct 10 20:47:16 mailserver ^Mpanic[cpu4]/thread=30002e89c40:
Oct 10 20:47:16 mailserver unix: [ID 715357 kern.notice] panic sync timeout
Oct 10 20:47:16 mailserver unix: [ID 100000 kern.notice]
Oct 10 20:47:16 mailserver genunix: [ID 111219 kern.notice] dumping to /dev/dsk/c3t0d0s1, offset 65536, content: kernel
Oct 10 20:48:58 mailserver genunix: [ID 409368 kern.notice] ^M100% done: 107116 pages dumped, compression ratio 2.92,
Oct 10 20:48:59 mailserver genunix: [ID 851671 kern.notice] dump succeeded
Oct 10 20:48:30 mailserver genunix: [ID 540533 kern.notice] ^MSunOS Release 5.10 Version Generic_118822-23 64-bit
Oct 10 20:48:30 mailserver genunix: [ID 943906 kern.notice] Copyright 1983-2005 Sun Microsystems, Inc. All rights reserved.
Oct 10 20:48:30 mailserver Use is subject to license terms.
Oct 10 20:48:30 mailserver genunix: [ID 678236 kern.info] Ethernet address = 0:14:4f:e:ef:7c
Oct 10 20:48:30 mailserver unix: [ID 673563 kern.info] NOTICE: Kernel Cage is ENABLED
Oct 10 20:48:30 mailserver unix: [ID 389951 kern.info] mem = 8380416K (0x1ff800000)
Oct 10 20:48:30 mailserver unix: [ID 930857 kern.info] avail mem = 8379768832
Oct 10 20:48:30 mailserver rootnex: [ID 466748 kern.info] root nexus = Sun Fire T200
Oct 10 20:48:30 mailserver rootnex: [ID 349649 kern.info] pseudo0 at root

Did a search on sunsolve and found the references to the CONN_DEC_REF, but most just say “see comments” but there is nothing there. Guess I will look at debuging the crash file tomorrow.. Looks like it is in the IP stack, and the load on the machine was sort of high when it happened… The machine does handle a ton of mail. In fact since July 11, 2006, it has handled 34,469,007 messages and 327,542,184K of email.

Solaris, Work ,

Solaris 10 with zones and patching

September 24th, 2006
Comments Off

One little draw back I have noticed about using zones on solaris 10 is the amount of time it takes to patch a machine. Right now I am waiting on a SunFire 890 with 8 processors and 16gb of ram with 12 zones (counting the global) to finish patching. I started it at around 8:54 this morning and it is now 11:16 and there are still 2 or 3 patches left to go. Since all the zones are basicly sparse zones, I wonder why it takes so much longer to do the patching? I also hope all this patching fixes my Power supply problem We have replaced the power supplies a couple of times, and the power distribution bored. I put the latest OBP on it this morning and it did not seem to fix it either. So hopefully after this set of patches are done, I will have a better idea whether it is a hardware or software problem.

Solaris, Sun, Work, Zones/Containers , , ,

Interesting Sun Ray problem

September 14th, 2006
Comments Off

I got called the other night by our operations group because the keyboard and mouse would not work on their 3 head group of Sun Ray 150′s. So I went in and killed their session and had them restart it, did not work. So I went looking in the log files and saw this:

Sep 11 17:53:41 [10.198.11.221.2.2] 0×0.0x1c392b7 0:3:ba:3c:1b:c1 USB: enable change: 2 lost enable state!
Sep 11 17:53:41 [10.198.11.221.2.2] 0×0.0x1c392b7 0:3:ba:3c:1b:c1 USB: enable change: 4 lost enable state!
Sep 11 20:28:44 [10.198.11.221.2.2] 0×0.0x2a1 0:3:ba:3c:1b:c1 USB: usb port 1 overcurrent
Sep 11 20:28:46 [10.198.11.221.2.2] 0×0.0×307 0:3:ba:3c:1b:c1 USB: usb port 2 overcurrent
Sep 11 20:28:46 [10.198.11.221.2.2] 0×0.0x36d 0:3:ba:3c:1b:c1 USB: usb port 3 overcurrent
Sep 11 20:28:47 [10.198.11.221.2.2] 0×0.0x3d3 0:3:ba:3c:1b:c1 USB: usb port 4 overcurrent
Sep 11 20:28:48 [10.198.11.221.2.2] 0×0.0×439 0:3:ba:3c:1b:c1 USB: usb port 5 overcurrent
Sep 11 20:45:34 [10.198.11.221.2.2] 0×0.0×291 0:3:ba:3c:1b:c1 USB: usb hub port 4 overcurrent!
Sep 11 20:45:35 [10.198.11.221.2.2] 0×0.0x2f9 0:3:ba:3c:1b:c1 USB: usb hub port 1 overcurrent!
Sep 11 20:45:36 [10.198.11.221.2.2] 0×0.0x35f 0:3:ba:3c:1b:c1 USB: usb hub port 2 overcurrent!
Sep 11 20:45:37 [10.198.11.221.2.2] 0×0.0x3c5 0:3:ba:3c:1b:c1 USB: usb hub port 3 overcurrent!
Sep 11 20:45:38 [10.198.11.221.2.2] 0×0.0x42b 0:3:ba:3c:1b:c1 USB: usb hub port 5 overcurrent!
Sep 11 20:46:21 [10.198.11.221.2.2] 0×0.0×304 0:3:ba:3c:1b:c1 USB: usb hub port 1 overcurrent!
Sep 11 20:46:22 [10.198.11.221.2.2] 0×0.0x36a 0:3:ba:3c:1b:c1 USB: usb hub port 2 overcurrent!
Sep 11 20:46:23 [10.198.11.221.2.2] 0×0.0x3d0 0:3:ba:3c:1b:c1 USB: usb hub port 3 overcurrent!
Sep 11 20:46:24 [10.198.11.221.2.2] 0×0.0×436 0:3:ba:3c:1b:c1 USB: usb hub port 4 overcurrent!
Sep 11 20:46:25 [10.198.11.221.2.2] 0×0.0x49c 0:3:ba:3c:1b:c1 USB: usb hub port 5 overcurrent!

Well that could not be good. So I ended up going in to the office. Tried unplugging the Sun Ray and plugging it back in. This is when I saw the 9 D error icon. Nice little icon with a picture of a USB connector and a yellow triangle. So I unplugged it and disconnected the keyboard and mouse and then plugged it back in. Still got the same error. The funny thing about the error is, it is listed as this in the docs:

This is an over current condition on the USB bus, i.e., the total number of devices draws too much current . Consider using a powered hub.

So now I ended up swaping it out with one that was in my office and rebuilding the multi-head group, and they were all set. The interesting thing about it is that the status LED stayed green, instead of turning amber. So the next morning I tried it on a different server (the original server it was attached to is running SRSS 2.0 still) that was running SRSS 3.1, this time nothing showed up in the log files, but the Sun Ray still showed the USB 9 icon and the keyboard and mouse did not work. So I ended up calling it in for replacement. It is nice that the Sun Ray’s have a long warrenty period. This one was bought 2 or 3 years ago.

In an unrelated note, I have to go in early to get a power backplane replaced in one of our V890′s because we have went through three power supplies in the PS0 slot in under a month. The bad part about this is the 890 has 11 zones on it and 1TB of disk, so we are going to have some services out while Sun replaces the backplane and power supply. Hopefully this will fix it though.

Computer Hardware, Interesting, Solaris, Sun, Sun Ray, Thin Clients, Work, Zones/Containers , , , , , , ,

How many tool bars can you install in IE?

September 6th, 2006

This afternoon, Justin and I decided to take a fresh installation of Windows XP and see if we could “junk” it up like some students do with tool bars, spyware, and ad ware. To say the least, we found it hard to find some of the stuff we wanted to install, but we did get a lot installed. (The whole point for this exercise was to test our student portal system because we have had reports from students that stuff does not work right. So we decided to put everything we could on a machine to see what all would break it.) So here is a screen shot of our Windows XP machine with some tool bars installed (click on it to get the bigger version).

Small window

We only found one so far that actually prevented us from doing stuff. It was the one that has the “CleanMyPC” line. Once we disabled it’s popup blocker everything worked fine.

As for what we installed, here is a list not in any particular order:

  • Google Toolbar
  • Yahoo Toolbar
  • Alexa Toolbar
  • Viewpoint Toolbar
  • AOL Toolbar
  • Dogpile Toolbar
  • Search Hippo Toolbar
  • Norton Internet Security (30 day trial from Yahoo)
  • Weatherbug
  • The Weather Channel toolbar and desktop
  • Hotbar
  • Newest AOL IM client
  • SideStep Side bar (not in this picture)

Needless to say after we installed SideStep and went to Travelocity to do a search, Internet Explorer just went away. No error messages, No “send this bug to MS” messages, just vanished. We also had it at one point where the actual viewable web portion of the screen was about only 1 inch high. So now we are looking for what the students have actually installed so we can see if we can reproduce what is on their machines.

Funny, Interesting, M$ Winders, Work , , ,

Large Sun Domain

March 23rd, 2006
Comments Off

We installed some new UltraSparc IV+ boards in our E25K yesterday, and to “test” the new boards, we decided to make one huge domain. (it is not every day I get to play with a large domain like this, usually they are 8 or 16 way). Well the domain has 24 cores, and 98GB of ram, and 1.1 TeraBytes of disk. What better way to test it then install Solaris 10 and create as many zones as I can. I started the process this morning, and each zone took about 3.5 minutes to create. (I was doing one at a time, assigning 254 zones per ethernet card, and with 12 ethernet cards, I figured I could get about 3048 zones on the machine. After 7 hours I have 120 zones on the machine, but unfortunatly I have to shut the domain down as it appears one of our System Controllers (SC1) fried it self, and we can’t turn it off or on, so Sun will be in tonight and we are going to shut the whole box down and replace the bad SC and then boot it back up. I think once that is done, I will start the script back up again and let it run all night, 3 of them at a time. Needless to say this box “spanked” my v890 with 8 x 1.2GHZ UltraVI cores with 16 gig of ram. I created a zone on it today and it took about 30 minutes to do on that machine.

Output from creating one zone on the E25K:

# zonecfg -z oscar -f oscar.cfg
# time zoneadm -z oscar install
Preparing to install zone .
Creating list of files to copy from the global zone.
Copying <2573> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <917> packages on the zone.
Initialized <917> packages on zone.
Zone
is initialized.
The file contains a log of the zone installation.

real 3:29.7
user 57.9
sys 1:18.3

Interesting, Solaris, Work, Zones/Containers , , ,