Interseting Sun T2000 crash
Got an email tonight from our operations group that one of our mail servers was not responding… So I looked at the /var/adm/messages and see this in it..
Oct 10 20:46:42 mailserver ^Mpanic[cpu4]/thread=30002e89c40:
Oct 10 20:46:42 mailserver unix: [ID 165833 kern.notice] CONN_DEC_REF: connp(6001918b940) has ref = 0
Oct 10 20:46:42 mailserver unix: [ID 100000 kern.notice]
Oct 10 20:46:42 mailserver genunix: [ID 723222 kern.notice] 000002a101cb9380 ip:squeue_drain+2e0 (6001dba5180, 10, 3101fcb3, 6000115fc00, 0, 0)
Oct 10 20:46:42 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000000000000011 000006001429bc80 0000000000000002 0000000000000000
Oct 10 20:46:42 mailserver %l4-7: 00000000018a5a58 000000007bea2a90 0000000000000000 000006001918b940
Oct 10 20:46:43 mailserver genunix: [ID 723222 kern.notice] 000002a101cb9440 ip:squeue_enter+338 (6000115fc00, 6001dba5180, 7beaa54c, 30007e2ed00, 0, 6001927e150)
Oct 10 20:46:43 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 000006001429bc80 0000000000000000 000006001429bc80
Oct 10 20:46:43 mailserver %l4-7: 0000000000000002 0000000000000001 000000007beab790 0000000000000000
Oct 10 20:46:44 mailserver genunix: [ID 723222 kern.notice] 000002a101cb94f0 unix:putnext+218 (6001927e248, 6001927e150, 6001dba5180, 100, 6001c8f0e28, 0)
Oct 10 20:46:44 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 0000000000000000 0000000000000000 00000000000055a0
Oct 10 20:46:44 mailserver %l4-7: 000000000000010d 0000000070042a68 000000007beab790 fffffd5efe34c000
Oct 10 20:46:44 mailserver genunix: [ID 723222 kern.notice] 000002a101cb95a0 genunix:strput+1d8 (3000711c028, 0, 2a101cb9928, 2a101cb9860, 0, 0)
Oct 10 20:46:44 mailserver genunix: [ID 179002 kern.notice] %l0-3: 000006001c8f0e28 0000000000000000 0000000000000000 0000000000000000
Oct 10 20:46:44 mailserver %l4-7: 000002a101cb9860 000000000000c000 0000000000000000 0000000000000000
Oct 10 20:46:45 mailserver genunix: [ID 723222 kern.notice] 000002a101cb97a0 genunix:strwrite+1f4 (60012321440, 2a101cb9928, 600035c11e0, 1000000, 6001c8f0e28, 3000711c0a8)
Oct 10 20:46:45 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000000000000003 0000000000000850 0000000000000000 000003000711c028
Oct 10 20:46:45 mailserver %l4-7: 000002a101cb9860 000000000000c000 0000000000000001 0000000000000000
Oct 10 20:46:46 mailserver genunix: [ID 723222 kern.notice] 000002a101cb9870 genunix:writev+350 (a, 30007280e10, 5, 3, 3, 1)
Oct 10 20:46:46 mailserver genunix: [ID 179002 kern.notice] %l0-3: 0000060012321440 0000000000000000 00000300070811a8 0000000000000000
Oct 10 20:46:46 mailserver %l4-7: 0000000000000005 0000000000000000 00000000000000b7 0000000000000000
Oct 10 20:46:46 mailserver unix: [ID 100000 kern.notice]
Oct 10 20:46:46 mailserver genunix: [ID 672855 kern.notice] syncing file systems…
Oct 10 20:47:16 mailserver unix: [ID 836849 kern.notice]
Oct 10 20:47:16 mailserver ^Mpanic[cpu4]/thread=30002e89c40:
Oct 10 20:47:16 mailserver unix: [ID 715357 kern.notice] panic sync timeout
Oct 10 20:47:16 mailserver unix: [ID 100000 kern.notice]
Oct 10 20:47:16 mailserver genunix: [ID 111219 kern.notice] dumping to /dev/dsk/c3t0d0s1, offset 65536, content: kernel
Oct 10 20:48:58 mailserver genunix: [ID 409368 kern.notice] ^M100% done: 107116 pages dumped, compression ratio 2.92,
Oct 10 20:48:59 mailserver genunix: [ID 851671 kern.notice] dump succeeded
Oct 10 20:48:30 mailserver genunix: [ID 540533 kern.notice] ^MSunOS Release 5.10 Version Generic_118822-23 64-bit
Oct 10 20:48:30 mailserver genunix: [ID 943906 kern.notice] Copyright 1983-2005 Sun Microsystems, Inc. All rights reserved.
Oct 10 20:48:30 mailserver Use is subject to license terms.
Oct 10 20:48:30 mailserver genunix: [ID 678236 kern.info] Ethernet address = 0:14:4f:e:ef:7c
Oct 10 20:48:30 mailserver unix: [ID 673563 kern.info] NOTICE: Kernel Cage is ENABLED
Oct 10 20:48:30 mailserver unix: [ID 389951 kern.info] mem = 8380416K (0x1ff800000)
Oct 10 20:48:30 mailserver unix: [ID 930857 kern.info] avail mem = 8379768832
Oct 10 20:48:30 mailserver rootnex: [ID 466748 kern.info] root nexus = Sun Fire T200
Oct 10 20:48:30 mailserver rootnex: [ID 349649 kern.info] pseudo0 at root
Did a search on sunsolve and found the references to the CONN_DEC_REF, but most just say “see comments” but there is nothing there. Guess I will look at debuging the crash file tomorrow.. Looks like it is in the IP stack, and the load on the machine was sort of high when it happened… The machine does handle a ton of mail. In fact since July 11, 2006, it has handled 34,469,007 messages and 327,542,184K of email.

