test.chruetertee.ch dumped core - see /var/crash/vmcore.0 Tue Sep 23 21:09:43 CEST 2008 FreeBSD test.chruetertee.ch 8.0-CURRENT-200809 FreeBSD 8.0-CURRENT-200809 #0: Sun Sep 7 22:13:46 UTC 2008 root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 panic: kmem_malloc(131072): kmem_map too small: 35979264 total allocated GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: panic: kmem_malloc(131072): kmem_map too small: 35979264 total allocated cpuid = 0 KDB: enter: panic panic: from debugger cpuid = 0 Uptime: 8m42s Physical memory: 115 MB Dumping 66 MB: 51 35 19 3 Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko #0 doadump () at pcpu.h:221 221 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:221 #1 0xc07b357e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc07b3843 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:572 #3 0xc0496c17 in db_panic (addr=Could not find the frame base for "db_panic". ) at /usr/src/sys/ddb/db_command.c:446 #4 0xc0497648 in db_command (last_cmdp=0xc0c7e270, cmd_table=0x0, dopager=1) at /usr/src/sys/ddb/db_command.c:413 #5 0xc049777a in db_command_loop () at /usr/src/sys/ddb/db_command.c:466 #6 0xc0498f6d in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:228 #7 0xc07e0ae6 in kdb_trap (type=3, code=0, tf=0xc88dd6fc) at /usr/src/sys/kern/subr_kdb.c:534 #8 0xc0ac94bb in trap (frame=0xc88dd6fc) at /usr/src/sys/i386/i386/trap.c:694 #9 0xc0aad2bb in calltrap () at /usr/src/sys/i386/i386/exception.s:165 #10 0xc07e0c6a in kdb_enter (why=0xc0b6a898 "panic", msg=0xc0b6a898 "panic") at cpufunc.h:70 #11 0xc07b382c in panic ( fmt=0xc0b8fca1 "kmem_malloc(%ld): kmem_map too small: %ld total allocated") at /usr/src/sys/kern/kern_shutdown.c:556 #12 0xc09d67d0 in kmem_malloc (map=0xc1471084, size=131072, flags=2) at /usr/src/sys/vm/vm_kern.c:303 #13 0xc09ce0d7 in page_alloc (zone=0x0, bytes=131072, pflag=0xc88dd7f3 "\002", wait=2) at /usr/src/sys/vm/uma_core.c:952 #14 0xc09cf130 in uma_large_malloc (size=131072, wait=2) at /usr/src/sys/vm/uma_core.c:2706 #15 0xc07a30d8 in malloc (size=131072, mtp=0xc2248560, flags=2) at /usr/src/sys/kern/kern_malloc.c:393 #16 0xc21c5bb0 in zfs_kmem_alloc (size=131072, kmflags=2) at /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:74 #17 0xc2223299 in zio_data_buf_alloc (size=131072) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:237 #18 0xc21d5e95 in arc_get_data_buf (buf=0xc365bc6c) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1714 #19 0xc21d6150 in arc_buf_alloc (spa=0xc1dfb000, size=131072, tag=0xc29695d8, type=ARC_BUFC_DATA) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:881 #20 0xc21dac50 in dbuf_read (db=0xc29695d8, zio=0xc1e2c000, flags=34) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:501 #21 0xc21e7e6f in dmu_tx_check_ioerr (zio=0xc1e2c000, dn=Variable "dn" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:160 #22 0xc21e8138 in dmu_tx_count_write (txh=0xc36585c0, off=27656192, len=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:201 #23 0xc21e85a3 in dmu_tx_hold_write (tx=0xc245c880, object=9, off=27656192, len=65536) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:314 #24 0xc22382e0 in zfs_freebsd_write (ap=0xc88ddbc4) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:740 #25 0xc0ad5ce5 in VOP_WRITE_APV (vop=0xc2249d60, a=0xc88ddbc4) at vnode_if.c:691 #26 0xc08475f3 in vn_write (fp=0xc1ea5380, uio=0xc88ddc58, active_cred=0xc2577800, flags=Variable "flags" is not available. ) at vnode_if.h:364 #27 0xc07f4b45 in dofilewrite (td=0xc1e36230, fd=4, fp=0xc1ea5380, auio=0xc88ddc58, offset=-1, flags=0) at file.h:239 #28 0xc07f5178 in kern_writev (td=0xc1e36230, fd=4, auio=0xc88ddc58) at /usr/src/sys/kern/sys_generic.c:444 #29 0xc07f528f in write (td=0xc1e36230, uap=0xc88ddcf8) at /usr/src/sys/kern/sys_generic.c:360 #30 0xc0ac8c13 in syscall (frame=0xc88ddd38) at /usr/src/sys/i386/i386/trap.c:1090 #31 0xc0aad320 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261 #32 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) ------------------------------------------------------------------------ ps -axl UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 8 0 0 0 - DLs ?? 1391799:06.00 [kernel] 0 1 0 0 44 0 1888 0 wait DLs ?? 12070683:38.00 [init] 0 2 0 0 -8 0 0 0 - DL ?? 10452386:19.00 [g_event] 0 3 0 0 -8 0 0 0 - DL ?? 27113920:52.00 [g_up] 0 4 0 0 -8 0 0 0 - DL ?? -9640195:-31.55 [g_down] 0 5 0 0 -16 0 0 0 ccb_sc DL ?? 0:00.00 [xpt_thrd] 0 6 0 0 -16 0 0 0 idle DL ?? 0:00.00 [mpt_recove 0 7 0 0 -16 0 0 0 - DL ?? 711066:23.00 [fdc0] 0 8 0 0 -60 0 0 0 waitin DL ?? 19377:06.00 [sctp_itera 0 9 0 0 76 0 0 0 psleep DL ?? 495831:32.00 [pagedaemon 0 10 0 0 -16 0 0 0 audit_ DL ?? 0:00.00 [audit] 0 11 0 0 171 0 0 0 - RL ?? -14517072:-52.55 [idle] 0 12 0 0 -48 0 0 0 - WL ?? 28015585:22.00 [intr] 0 13 0 0 -16 0 0 0 - DL ?? 12264369:01.00 [yarrow] 0 14 0 0 -16 0 0 0 usbevt DL ?? 5611:13.00 [usb0] 0 15 0 0 -16 0 0 0 usbtsk DL ?? 0:00.00 [usbtask-hc 0 16 0 0 -16 0 0 0 usbtsk DL ?? 0:00.00 [usbtask-dr 0 17 0 0 -16 0 0 0 usbevt DL ?? 16642:34.00 [usb1] 0 18 0 0 76 0 0 0 psleep DL ?? 2607:53.00 [vmdaemon] 0 19 0 0 76 0 0 0 pgzero DL ?? 10881:00.00 [pagezero] 0 20 0 0 76 0 0 0 psleep DL ?? 1915968:36.00 [bufdaemon] 0 21 0 0 44 0 0 0 syncer DL ?? 1989739:34.00 [syncer] 0 22 0 0 44 0 0 0 vlruwt DL ?? 1862125:41.00 [vnlru] 0 23 0 0 76 0 0 0 sdflus DL ?? 1750289:59.00 [softdepflu 0 123 1 0 76 0 1464 0 pause Ds ?? 87191:59.00 [adjkerntz] 0 483 1 0 76 0 3212 0 select Ds ?? 246077:06.00 [dhclient] 65 499 1 0 76 0 3212 0 select Ds ?? 95533:15.00 [dhclient] 0 500 1 0 44 0 1888 0 select Ds ?? 850127:23.00 [devd] 0 646 1 0 44 0 3272 0 select Ds ?? 5222530:31.00 [syslogd] 0 864 1 0 47 0 6584 0 select Ds ?? 898945:43.00 [sshd] 0 871 1 0 44 0 5976 0 select Ds ?? 3491830:54.00 [sendmail] 25 875 1 0 76 0 5976 0 pause Ds ?? 659498:42.00 [sendmail] 0 881 1 0 44 0 3300 0 nanslp Ds ?? 1203702:04.00 [cron] 0 895 1 0 44 0 3376 0 select Ds ?? 16368956:37.00 [moused] 0 947 1 0 47 0 3720 0 wait Ds ?? 5934821:07.00 [login] 0 948 1 0 45 0 3720 0 wait Ds ?? 5328270:26.00 [login] 0 949 1 0 45 0 3720 0 wait Ds ?? 5695942:36.00 [login] 0 950 1 0 45 0 3720 0 wait Ds ?? 6175204:47.00 [login] 0 951 1 0 45 0 3720 0 wait Ds ?? 5774667:10.00 [login] 0 952 1 0 45 0 3720 0 wait Ds ?? 6198060:25.00 [login] 0 953 1 0 44 0 3720 0 wait Ds ?? 7733065:51.00 [login] 0 954 1 0 76 0 3272 0 tty in Ds+ ?? 751052:15.00 [getty] 0 955 947 0 44 0 4592 0 pause D ?? 9767513:30.00 [csh] 0 966 0 0 76 0 0 0 tq->tq DL ?? 33579:42.00 [system_tas 0 967 0 0 47 0 0 0 d DL ?? -26708973:-43.55 [arc_reclai 0 983 0 0 76 0 0 0 tq->tq DL ?? 17688:17.00 [spa_zio_is 0 984 0 0 76 0 0 0 tq->tq DL ?? 589:38.00 [spa_zio_in 0 985 0 0 76 0 0 0 tq->tq DL ?? 513:15.00 [spa_zio_is 0 986 0 0 44 0 0 0 tq->tq DL ?? 5024570:07.00 [spa_zio_in 0 987 0 0 44 0 0 0 tq->tq DL ?? 26100670:17.00 [spa_zio_is 0 988 0 0 45 0 0 0 db->db DL ?? -10289913:00.55 [spa_zio_in 0 989 0 0 76 0 0 0 tq->tq DL ?? 489:00.00 [spa_zio_is 0 990 0 0 76 0 0 0 tq->tq DL ?? 480:07.00 [spa_zio_in 0 991 0 0 76 0 0 0 tq->tq DL ?? 498:30.00 [spa_zio_is 0 992 0 0 76 0 0 0 tq->tq DL ?? 484:23.00 [spa_zio_in 0 993 0 0 76 0 0 0 tq->tq DL ?? 490:00.00 [spa_zio_is 0 994 0 0 76 0 0 0 tq->tq DL ?? 485:37.00 [spa_zio_in 0 996 0 0 76 0 0 0 tx->tx DL ?? 7362605:31.00 [txg_thread 0 997 0 0 45 0 0 0 zio->i DL ?? 29942124:55.00 [txg_thread 0 998 0 0 76 0 0 0 tx->tx DL ?? 343320:18.00 [txg_thread 0 1000 0 0 44 0 0 0 tq->tq DL ?? 6944142:20.00 [zil_clean] 0 1001 0 0 76 0 0 0 tq->tq DL ?? 783:00.00 [zil_clean] 0 1030 955 0 54 0 3212 0 nokva D+ ?? -20357387:-46.55 [dd] 0 1033 948 0 46 0 4592 0 pause D ?? 7092247:00.00 [csh] 0 1038 1033 0 54 0 3212 0 nokva D+ ?? 8032824:27.00 [dd] 0 1039 949 0 45 0 4592 0 pause D ?? 6044339:25.00 [csh] 0 1044 1039 0 44 0 3212 0 nokva D+ ?? 12810202:08.00 [dd] 0 1045 950 0 44 0 4592 0 pause D ?? 7031232:52.00 [csh] 0 1053 1045 0 46 0 3212 0 zfs D+ ?? -28282511:-54.55 [dd] 0 1056 951 0 44 0 4592 0 pause D ?? 8417280:14.00 [csh] 0 1076 1056 0 44 0 3212 0 tx->tx D+ ?? -7386989:-30.55 [cp] 0 1077 952 0 48 0 4592 0 pause D ?? 13868295:11.00 [csh] 0 1137 1077 0 44 0 3604 0 select D+ ?? 26765780:07.00 [top] 0 1138 953 0 49 0 4592 0 pause D ?? 11267995:27.00 [csh] 0 1151 1138 0 44 0 3212 0 - R ?? -18262593:-37.55 [cp] 0 1152 1138 0 44 0 3212 0 nokva D ?? -12605795:-57.55 [cp] 0 1154 1138 0 44 0 3212 0 tx->tx D ?? -5526665:-43.55 [cp] 0 1158 1138 0 44 0 3604 0 select D+ ?? 20678434:30.00 [top] 0 1165 0 0 44 0 0 0 vgeom: DL ?? 2110230:57.00 [vdev:worke ------------------------------------------------------------------------ vmstat -s 0 cpu context switches 0 device interrupts 0 software interrupts 0 traps 0 system calls 0 kernel threads created 0 fork() calls 0 vfork() calls 0 rfork() calls 0 swap pager pageins 0 swap pager pages paged in 0 swap pager pageouts 0 swap pager pages paged out 0 vnode pager pageins 0 vnode pager pages paged in 0 vnode pager pageouts 0 vnode pager pages paged out 0 page daemon wakeups 0 pages examined by the page daemon 1119 pages reactivated 0 copy-on-write faults 0 copy-on-write optimized faults 0 zero fill pages zeroed 0 zero fill pages prezeroed 0 intransit blocking page faults 0 total VM faults taken 0 pages affected by kernel thread creation 0 pages affected by fork() 0 pages affected by vfork() 0 pages affected by rfork() 1167 pages cached 0 pages freed 0 pages freed by daemon 109769 pages freed by exiting processes 9971 pages active 1973 pages inactive 43 pages in VM cache 11172 pages wired down 5691 pages free 4096 bytes per page 24600 total name lookups cache hits (83% pos + 11% neg) system 0% per-directory deletions 0%, falsehits 0%, toolong 0% ------------------------------------------------------------------------ vmstat -m Type InUse MemUse HighUse Requests Size(s) pfs_nodes -20 -1K - 0 CAM SIM -2 0K - 0 GEOM 1189233 -1173K - 1190672 64,1024,4096,16384 isadev -18 0K - 0 CAM periph 2072 -1K - 2096 32,64,256 CAM XPT 36290 -36K - 36432 16,32,64,128,256,512,1024 ata_generic -3 -2K - 0 ad_driver -2 0K - 0 cdev -15 0K - 0 ar_driver 37878 -36K - 37888 32,128 sigio -1 0K - 0 filedesc 277104 -291K - 278272 16,32,64,128,256,512,16384 kenv -69 -6K - 48 32 kqueue 27600 -26K - 27648 256,512 proc-args 26270 -26K - 26800 32,64,128,512,1024,2048,4096 ithread -71 -4K - 0 prison -1 0K - 0 acd_driver -1 -1K - 0 KTRACE -100 -11K - 0 kbdmux -6 -7K - 0 linker 12874 -787K - 13056 32,128,1024 lockf 640 0K - 672 64,128 ip6ndp -4 0K - 0 temp 7454145092 -7281280K - 7455966048 128,4096,32768,262144,524288,1048576,2097152,8388608,16777216 devbuf 40701 -2242K - 40976 32 module -442 -26K - 0 mtx_pool -1 -3K - 0 subproc 4451101 -4684K - 4452352 16,32,64,128,256,512,16384 proc -2 0K - 0 session 296 0K - 320 16,64 pgrp 6390 -7K - 6528 32,64,512,1024 cred 773890 -766K - 776960 16,32,128,256,1024,2048,4096,8192,32768 uidinfo 120 0K - 128 64 plimit 61946 -64K - 62208 16,32,256,512,1024,2048 sysctltmp 9742 -8K - 10032 32,512,4096 sysctloid -1452 -70K - 976 16,64,128,256,512 sysctl 63290 -63K - 65888 32,64,512,8192,32768 umtx -112 -6K - 0 p1003.1b -1 0K - 0 SWAP -2 -275K - 0 bus-sc 6484491 -6362K - 6486016 64,1024,2048,4096,16384 bus 644694 -671K - 648320 128,256,512,8192,32768 clist -54 -5K - 0 devstat -8 -15K - 0 eventhandler -61 -1K - 0 kobj 173701 -757K - 174080 16,64,256,1024 rman 28008 -36K - 28608 16,32,64,128,256,512,2048,4096 sbuf 699692 -682K - 700304 64,512,1024,8192 ddb_capture -1 -47K - 0 stack 254 0K - 256 32 taskqueue -11 0K - 0 Unitno 949 0K - 1024 1024 CAM dev queue -2 0K - 0 Witness -1 -103K - 0 iov 41006 -39K - 41344 32,256,1024,4096 select -19 0K - 0 ioctlops 289260 -284K - 292592 64,4096,16384,32768 msg -4 -24K - 0 sem -4 -5K - 0 shm -1 -11K - 0 tty -22 -10K - 0 mbuf_tag 310 0K - 320 32,128 shmfd -1 -3K - 0 pcb 538 -77K - 592 16,64,512 soname 24799 -23K - 25088 32,64,128,256,4096 biobuf 4089 -13K - 4096 32 vfscache -1 -31K - 0 vfs_hash -1 -15K - 0 vnodes -2 0K - 0 vnodemarker 130305 -126K - 130560 16,32,64,128,256,512,1024,2048 mount 1544 -3K - 1696 16,32,128,256,512 BPF -8 -7K - 0 ether_multi 314 0K - 336 16,32,64 ifaddr 224 -6K - 256 16 ifnet -4 -2K - 0 clone -5 -19K - 0 arpcom -1 0K - 0 routetbl 12341 -13K - 12416 64,256,512 in_multi 61 0K - 64 16 sctp_iter 254 0K - 256 32 sctp_ifn -2 0K - 0 sctp_ifa -3 0K - 0 sctp_vrf -1 0K - 0 sctp_a_it 30 0K - 32 32 hostcache -1 -15K - 0 syncache -1 -71K - 0 in6_multi -16 0K - 0 CAM queue 2769 -1K - 2960 16,128,256,512,2048 nfss_daemon -1 -7K - 0 audit_evclass 387 -2K - 592 16,64,512 savedino 5100 -4K - 5120 64,256 dirrem 775 0K - 800 16,128,256 mkdir 372 0K - 384 64,128 diradd 2016 -1K - 2048 512 freefile 713 0K - 736 16,32,64,256 freeblks 4590 -3K - 4608 32,256 freefrag 62 0K - 64 32 allocdirect 4699 -3K - 4736 16,64,512 bmsafemap 693 0K - 704 16,32,128 newblk 2330 -1K - 2368 16,64,512 inodedep 6476 -21K - 6528 16,32,256,512 pagedep 881 -3K - 896 32,64,128 ufs_dirhash -24 -3K - 0 ufs_mount -12 -24K - 0 UMAHash 254 0K - 256 16 ppbusdev -3 0K - 0 vm_pgdata -2 -7K - 0 entropy -1024 -63K - 0 atkbddev -2 0K - 0 UART -6 -1K - 0 USBdev -9 -2K - 0 USB 2513 -9K - 2560 32,64 DEVFS1 3471 -27K - 3584 32,64,128 io_apic -1 0K - 0 memdesc -1 -3K - 0 DEVFS3 275 -12K - 384 16,32 nexusdev -4 0K - 0 DEVFS 2 0K - 16 16 DEVFSP -1 0K - 0 acpica 1373906 -1416K - 1397408 16,128,512,2048,4096,8192,16384,65536,262144 acpitask 62 0K - 64 32 acpidev -37 0K - 0 acpisem -15 0K - 0 pci_link -8 0K - 0 apmdev -1 0K - 0 solaris 8419483101 -8302251K - 8473341664 16,32,64,128,1024,4096,8192,16384,262144,524288,1048576,4194304,16777216,33554432,268435456,536870912 kstat_data -1 0K - 0 ------------------------------------------------------------------------ vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 124, 0, 93, 27, 93, 0 UMA Zones: 480, 0, 93, 3, 93, 0 UMA Slabs: 64, 0, 715, 52, 51539, 0 UMA RCntSlabs: 104, 0, 69, 5, 69, 0 UMA Hash: 128, 0, 5, 25, 6, 0 16 Bucket: 76, 0, 31, 19, 58, 0 32 Bucket: 140, 0, 18, 10, 67, 0 64 Bucket: 268, 0, 24, 18, 166, 7 128 Bucket: 524, 0, 39, 45, 31161, 170 VM OBJECT: 124, 0, 1347, 79, 16306, 0 MAP: 132, 0, 7, 22, 7, 0 KMAP ENTRY: 68, 7392, 145, 359, 569417, 0 MAP ENTRY: 68, 0, 908, 100, 30669, 0 DP fakepg: 68, 0, 0, 0, 0, 0 mt_zone: 1032, 0, 258, 126, 258, 0 16: 16, 0, 2647, 195, 9001654, 0 32: 32, 0, 1971, 63, 22502495, 0 64: 64, 0, 3742, 93, 13419717, 0 128: 128, 0, 637, 83, 4445249, 0 256: 256, 0, 1207, 173, 4469549, 0 512: 512, 0, 121, 15, 4448, 0 1024: 1024, 0, 153, 79, 6489, 0 2048: 2048, 0, 323, 13, 1518, 0 4096: 4096, 0, 131, 14, 1829391, 0 Files: 56, 0, 81, 120, 5522, 0 TURNSTILE: 72, 0, 113, 79, 113, 0 umtx pi: 52, 0, 0, 0, 0, 0 PROC: 668, 0, 78, 6, 1165, 0 THREAD: 548, 0, 106, 6, 106, 0 SLEEPQUEUE: 40, 0, 113, 121, 113, 0 VMSPACE: 224, 0, 36, 32, 1069, 0 cpuset: 40, 0, 2, 182, 2, 0 audit_record: 856, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 0, 128, 9, 0 mbuf: 256, 0, 1, 141, 306, 0 mbuf_cluster: 2048, 4672, 128, 6, 131, 0 mbuf_jumbo_page: 4096, 2336, 0, 2, 2, 0 mbuf_jumbo_9k: 9216, 3504, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 2336, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0, 0 g_bio: 132, 0, 0, 145, 177751, 0 ttyinq: 152, 0, 540, 110, 2520, 0 ttyoutq: 256, 0, 279, 51, 1302, 0 ata_request: 200, 0, 1, 86, 108937, 0 ata_composite: 180, 0, 0, 0, 0, 0 VNODE: 268, 0, 489, 15, 513, 0 VNODEPOLL: 56, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 20, 11571, 0 S VFS Cache: 68, 0, 452, 52, 1120, 0 L VFS Cache: 291, 0, 0, 26, 2, 0 DIRHASH: 1024, 0, 34, 2, 34, 0 NFSMOUNT: 556, 0, 0, 0, 0, 0 NFSNODE: 448, 0, 0, 0, 0, 0 pipe: 384, 0, 1, 19, 595, 0 ksiginfo: 80, 0, 69, 75, 69, 0 itimer: 220, 0, 0, 0, 0, 0 KNOTE: 68, 0, 0, 112, 56, 0 socket: 392, 1870, 18, 12, 204, 0 unpcb: 172, 1886, 12, 34, 51, 0 ipq: 32, 226, 0, 0, 0, 0 udpcb: 176, 1870, 2, 42, 141, 0 inpcb: 176, 1870, 3, 41, 7, 0 tcpcb: 484, 1864, 3, 13, 7, 0 tcptw: 52, 432, 0, 144, 1, 0 syncache: 108, 15372, 0, 72, 1, 0 hostcache: 76, 15400, 1, 99, 1, 0 tcpreass: 20, 338, 0, 0, 0, 0 sackhole: 20, 0, 0, 0, 0, 0 sctp_ep: 804, 1865, 0, 0, 0, 0 sctp_asoc: 1436, 40000, 0, 0, 0, 0 sctp_laddr: 24, 80040, 0, 145, 2, 0 sctp_raddr: 412, 80001, 0, 0, 0, 0 sctp_chunk: 92, 400008, 0, 0, 0, 0 sctp_readq: 76, 400000, 0, 0, 0, 0 sctp_stream_msg_out: 60, 400050, 0, 0, 0, 0 sctp_asconf: 24, 400055, 0, 0, 0, 0 sctp_asconf_ack: 24, 400055, 0, 0, 0, 0 ripcb: 176, 1870, 0, 0, 0, 0 rtentry: 120, 0, 9, 55, 10, 0 selfd: 28, 0, 35, 219, 1697, 0 Mountpoints: 672, 0, 7, 5, 7, 0 FFS inode: 124, 0, 429, 36, 452, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 429, 6, 452, 0 SWAPMETA: 276, 14476, 0, 0, 0, 0 taskq_ent_cache: 32, 0, 608, 183, 1242, 0 taskq_cache: 196, 0, 15, 25, 27, 0 zio_cache: 548, 0, 409, 18, 9139476, 0 dmu_buf_impl_t: 136, 0, 78, 241, 44011, 0 dnode_t: 448, 0, 47, 16, 529, 0 arc_buf_hdr_t: 124, 0, 298, 229, 63157, 0 arc_buf_t: 20, 0, 231, 445, 65046, 0 zil_lwb_cache: 176, 0, 1, 87, 532, 0 zfs_znode_cache: 208, 0, 12, 26, 12, 0 ------------------------------------------------------------------------ vmstat -i interrupt total rate irq1: atkbd0 2552 27 irq6: fdc0 13 0 irq12: psm0 3804 40 irq14: ata0 107854 1147 irq15: ata1 57 0 irq16: le0 70 0 irq17: mpt0 17 0 cpu0: timer 1040626 11070 Total 1154993 12287 ------------------------------------------------------------------------ pstat -T 81/1864 files 0M/2047M swap space ------------------------------------------------------------------------ pstat -s Device 1K-blocks Used Avail Capacity /dev/ad0s1b 2097024 0 2097024 0% ------------------------------------------------------------------------ iostat iostat: kvm_read(_tk_nin): invalid address (0x0) iostat: disabling TTY statistics iostat: kvm_getcptime: invalid address (0x0) iostat: disabling CPU time statistics ad0 ad1 KB/t tps MB/s KB/t tps MB/s 13.38 14 0.19 51.61 1119 56.41 ------------------------------------------------------------------------ ipcs -a Message Queues: T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME Shared Memory: T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME Semaphores: T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME ------------------------------------------------------------------------ ipcs -T msginfo: msgmax: 16384 (max characters in a message) msgmni: 40 (# of message queues) msgmnb: 2048 (max characters in a message queue) msgtql: 40 (max # of messages in system) msgssz: 8 (size of a message segment) msgseg: 2048 (# of message segments in system) shminfo: shmmax: 33554432 (max shared memory segment size) shmmin: 1 (min shared memory segment size) shmmni: 192 (max number of shared memory identifiers) shmseg: 128 (max shared memory segments per process) shmall: 8192 (max amount of shared memory in pages) seminfo: semmap: 30 (# of entries in semaphore map) semmni: 10 (# of semaphore identifiers) semmns: 60 (# of semaphores in system) semmnu: 30 (# of undo structures in system) semmsl: 60 (max # of semaphores per id) semopm: 100 (max # of operations per semop call) semume: 10 (max # of undo entries per process) semusz: 92 (size in bytes of undo structure) semvmx: 32767 (semaphore maximum value) semaem: 16384 (adjust on exit max value) ------------------------------------------------------------------------ nfsstat Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 0 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 0 0 0 0 0 0 0 0 BioRLHits Misses BioD Hits Misses DirE Hits Misses 0 0 0 0 0 0 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Server Ret-Failed 0 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 0 0 0 ------------------------------------------------------------------------ netstat -s tcp: 31 packets sent 19 data packets (3400 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 9 ack-only packets (5 delayed) 0 URG only packets 0 window probe packets 0 window update packets 3 control packets 31 packets received 23 acks (for 3403 bytes) 2 duplicate acks 0 acks for unsent data 21 packets (3400 bytes) received in-sequence 0 completely duplicate packets (0 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 0 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 1 connection request 1 connection accept 0 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 2 connections established (including accepts) 4 connections closed (including 0 drops) 2 connections updated cached RTT on close 2 connections updated cached RTT variance on close 0 connections updated cached ssthresh on close 0 embryonic connections dropped 23 segments updated rtt (of 22 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 3 correct ACK header predictions 3 correct data packet header predictions 1 syncache entry added 0 retransmitted 0 dupsyn 0 dropped 1 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 1 cookie sent 0 cookies received 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window udp: 33 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 0 with no checksum 0 dropped due to no socket 5 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 28 delivered 28 datagrams output 0 times multicast source filter matched ip: 65 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 64 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 1 packet not forwardable 0 packets received for unknown multicast group 0 redirects sent 59 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 0 calls to icmp_error 0 errors not generated in response to an icmp message 0 messages with bad code fields 0 messages less than the minimum length 0 messages with bad checksum 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored 0 message responses generated 0 invalid return addresses 0 no return routes igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent ip6: 0 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 0 redirects sent 4 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 0 multicast packets which we don't join Mbuf statistics: 0 one mbuf 0 one ext mbuf 0 two or more ext mbuf 0 packets whose headers are not continuous 0 tunneling packets that can't find gif 0 packets discarded because of too many headers 0 failures of source address selection 0 forward cache hit 0 forward cache miss Source addresses selection rule applied: 2 first candidate 2 same address icmp6: 0 calls to icmp6_error 0 errors not generated in response to an icmp6 message 0 errors not generated because of rate limitation Output histogram: multicast listener report: 4 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 0 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes rip6: 0 messages received 0 checksum calcurations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output ------------------------------------------------------------------------ netstat -m 1/269/270 mbufs in use (current/cache/total) 0/134/134/4672 mbuf clusters in use (current/cache/total/max) 0/128 mbuf+clusters out of packet secondary zone in use (current/cache) 0/2/2/2336 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/3504 9k jumbo clusters in use (current/cache/total/max) 0/0/0/2336 16k jumbo clusters in use (current/cache/total/max) 18014398509481983K/343K/343K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines ------------------------------------------------------------------------ netstat -id Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll Drop le0 1500 00:0c:29:56:d0:4f 35 0 31 0 0 0 le0 1500 192.168.204.0 192.168.204.128 33 - 28 - - - plip0 1500 0 0 0 0 0 0 lo0 16384 31 0 31 0 0 0 lo0 16384 fe80:3::1 fe80:3::1 0 - 0 - - - lo0 16384 localhost ::1 0 - 0 - - - lo0 16384 your-net localhost 31 - 31 - - - ------------------------------------------------------------------------ netstat -anr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 192.168.204.2 UGS 0 0 le0 127.0.0.1 127.0.0.1 UH 0 31 lo0 192.168.204.0/24 link#1 UC 0 0 le0 192.168.204.2 00:50:56:e4:00:8e UHLW 2 28 le0 1122 Internet6: Destination Gateway Flags Netif Expire ::1 ::1 UHL lo0 fe80::%lo0/64 fe80::1%lo0 U lo0 fe80::1%lo0 link#3 UHL lo0 ff01:3::/32 fe80::1%lo0 UC lo0 ff02::%lo0/32 fe80::1%lo0 UC lo0 ------------------------------------------------------------------------ netstat -anA Active Internet connections (including servers) Tcpcb Proto Recv-Q Send-Q Local Address Foreign Address (state) c1fb7974 tcp4 0 0 127.0.0.1.25 *.* LISTEN c1fb7d3c tcp4 0 0 *.22 *.* LISTEN c1fb8000 tcp6 0 0 *.22 *.* LISTEN c1f3cb00 udp4 0 0 *.514 *.* c1f3cbb0 udp6 0 0 *.514 *.* Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr c1f4a000 stream 0 0 c1f4b648 0 0 0 /var/run/devd.pipe c1f49e1c dgram 0 0 0 c1f492b0 0 c1f49ec8 c1f49ec8 dgram 0 0 0 c1f492b0 0 c1f49560 c1f49560 dgram 0 0 0 c1f492b0 0 c1f49d70 c1f49d70 dgram 0 0 0 c1f492b0 0 c1f49cc4 c1f49cc4 dgram 0 0 0 c1f492b0 0 c1f4960c c1f4960c dgram 0 0 0 c1f492b0 0 c1f49158 c1f49158 dgram 0 0 0 c1f492b0 0 c1f490ac c1f49000 dgram 0 0 0 c1f4935c 0 0 c1f490ac dgram 0 0 0 c1f492b0 0 0 c1f492b0 dgram 0 0 c1fa253c 0 c1f49e1c 0 /var/run/logpriv c1f4935c dgram 0 0 c1fa2648 0 c1f49000 0 /var/run/log ------------------------------------------------------------------------ netstat -aL Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0/0/10 localhost.smtp tcp4 0/0/128 *.ssh tcp6 0/0/128 *.ssh ------------------------------------------------------------------------ fstat USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root vdev:worker ad1 1165 root / 2 drwxr-xr-x 512 r root vdev:worker ad1 1165 wd / 2 drwxr-xr-x 512 r root top 1158 root / 2 drwxr-xr-x 512 r root top 1158 wd /mnt/crash 3 drwxr-xr-x 12 r root top 1158 text /usr 47397 -r-xr-xr-x 46204 r root top 1158 0 /dev 57 crw------- ttyv6 rw root top 1158 1 /dev 57 crw------- ttyv6 rw root top 1158 2 /dev 57 crw------- ttyv6 rw root top 1158 4 /dev 32 crw-rw-rw- null r root top 1158 5 /dev 32 crw-rw-rw- null r root cp 1154 root / 2 drwxr-xr-x 512 r root cp 1154 wd /mnt/crash 3 drwxr-xr-x 12 r root cp 1154 text / 16 -r-xr-xr-x 15936 r root cp 1154 0 /dev 57 crw------- ttyv6 rw root cp 1154 1 /dev 57 crw------- ttyv6 rw root cp 1154 2 /dev 57 crw------- ttyv6 rw root cp 1154 3 /mnt/crash 8 -rw-r--r-- 78643200 r root cp 1154 4 /mnt/crash 11 -rw-r--r-- 37748736 w root cp 1152 root / 2 drwxr-xr-x 512 r root cp 1152 wd /mnt/crash 3 drwxr-xr-x 12 r root cp 1152 text / 16 -r-xr-xr-x 15936 r root cp 1152 0 /dev 57 crw------- ttyv6 rw root cp 1152 1 /dev 57 crw------- ttyv6 rw root cp 1152 2 /dev 57 crw------- ttyv6 rw root cp 1152 3 /mnt/crash 7 -rw-r--r-- 471421952 r root cp 1152 4 /mnt/crash 10 -rw-r--r-- 31326208 w root cp 1151 root / 2 drwxr-xr-x 512 r root cp 1151 wd /mnt/crash 3 drwxr-xr-x 12 r root cp 1151 text / 16 -r-xr-xr-x 15936 r root cp 1151 0 /dev 57 crw------- ttyv6 rw root cp 1151 1 /dev 57 crw------- ttyv6 rw root cp 1151 2 /dev 57 crw------- ttyv6 rw root cp 1151 3 /mnt/crash 6 -rw-r--r-- 401266688 r root cp 1151 4 /mnt/crash 9 -rw-r--r-- 27656192 w root csh 1138 root / 2 drwxr-xr-x 512 r root csh 1138 wd /mnt/crash 3 drwxr-xr-x 12 r root csh 1138 text / 17 -r-xr-xr-x 320152 r root csh 1138 15 /dev 57 crw------- ttyv6 rw root csh 1138 16 /dev 57 crw------- ttyv6 rw root csh 1138 17 /dev 57 crw------- ttyv6 rw root csh 1138 18 /dev 57 crw------- ttyv6 rw root csh 1138 19 /dev 57 crw------- ttyv6 rw root top 1137 root / 2 drwxr-xr-x 512 r root top 1137 wd /mnt/crash 3 drwxr-xr-x 12 r root top 1137 text /usr 47397 -r-xr-xr-x 46204 r root top 1137 0 /dev 56 crw------- ttyv5 rw root top 1137 1 /dev 56 crw------- ttyv5 rw root top 1137 2 /dev 56 crw------- ttyv5 rw root top 1137 4 /dev 32 crw-rw-rw- null r root top 1137 5 /dev 32 crw-rw-rw- null r root csh 1077 root / 2 drwxr-xr-x 512 r root csh 1077 wd /mnt/crash 3 drwxr-xr-x 12 r root csh 1077 text / 17 -r-xr-xr-x 320152 r root csh 1077 15 /dev 56 crw------- ttyv5 rw root csh 1077 16 /dev 56 crw------- ttyv5 rw root csh 1077 17 /dev 56 crw------- ttyv5 rw root csh 1077 18 /dev 56 crw------- ttyv5 rw root csh 1077 19 /dev 56 crw------- ttyv5 rw root cp 1076 root / 2 drwxr-xr-x 512 r root cp 1076 wd /mnt/crash 3 drwxr-xr-x 12 r root cp 1076 text / 16 -r-xr-xr-x 15936 r root cp 1076 0 /dev 55 crw------- ttyv4 rw root cp 1076 1 /dev 55 crw------- ttyv4 rw root cp 1076 2 /dev 55 crw------- ttyv4 rw root cp 1076 3 /mnt/crash 4 -rw-r--r-- 776243200 r root cp 1076 4 /mnt/crash 8 -rw-r--r-- 78643200 w root csh 1056 root / 2 drwxr-xr-x 512 r root csh 1056 wd /mnt/crash 3 drwxr-xr-x 12 r root csh 1056 text / 17 -r-xr-xr-x 320152 r root csh 1056 15 /dev 55 crw------- ttyv4 rw root csh 1056 16 /dev 55 crw------- ttyv4 rw root csh 1056 17 /dev 55 crw------- ttyv4 rw root csh 1056 18 /dev 55 crw------- ttyv4 rw root csh 1056 19 /dev 55 crw------- ttyv4 rw root dd 1053 root / 2 drwxr-xr-x 512 r root dd 1053 wd / 12741 drwxr-xr-x 512 r root dd 1053 text / 19 -r-xr-xr-x 20180 r root dd 1053 0 /dev 54 crw------- ttyv3 rw root dd 1053 1 /dev 54 crw------- ttyv3 rw root dd 1053 2 /dev 54 crw------- ttyv3 rw root dd 1053 3 /dev 33 crw-rw-rw- zero r root dd 1053 4 /mnt/crash 7 -rw-r--r-- 471421952 rw root csh 1045 root / 2 drwxr-xr-x 512 r root csh 1045 wd / 12741 drwxr-xr-x 512 r root csh 1045 text / 17 -r-xr-xr-x 320152 r root csh 1045 15 /dev 54 crw------- ttyv3 rw root csh 1045 16 /dev 54 crw------- ttyv3 rw root csh 1045 17 /dev 54 crw------- ttyv3 rw root csh 1045 18 /dev 54 crw------- ttyv3 rw root csh 1045 19 /dev 54 crw------- ttyv3 rw root dd 1044 root / 2 drwxr-xr-x 512 r root dd 1044 wd / 12741 drwxr-xr-x 512 r root dd 1044 text / 19 -r-xr-xr-x 20180 r root dd 1044 0 /dev 53 crw------- ttyv2 rw root dd 1044 1 /dev 53 crw------- ttyv2 rw root dd 1044 2 /dev 53 crw------- ttyv2 rw root dd 1044 3 /dev 8 crw-rw-rw- random r root dd 1044 4 /mnt/crash 6 -rw-r--r-- 401266688 rw root csh 1039 root / 2 drwxr-xr-x 512 r root csh 1039 wd / 12741 drwxr-xr-x 512 r root csh 1039 text / 17 -r-xr-xr-x 320152 r root csh 1039 15 /dev 53 crw------- ttyv2 rw root csh 1039 16 /dev 53 crw------- ttyv2 rw root csh 1039 17 /dev 53 crw------- ttyv2 rw root csh 1039 18 /dev 53 crw------- ttyv2 rw root csh 1039 19 /dev 53 crw------- ttyv2 rw root dd 1038 root / 2 drwxr-xr-x 512 r root dd 1038 wd /mnt/crash 3 drwxr-xr-x 12 r root dd 1038 text / 19 -r-xr-xr-x 20180 r root dd 1038 0 /dev 52 crw------- ttyv1 rw root dd 1038 1 /dev 52 crw------- ttyv1 rw root dd 1038 2 /dev 52 crw------- ttyv1 rw root dd 1038 3 /dev 8 crw-rw-rw- random r root dd 1038 4 /mnt/crash 5 -rw-r--r-- 528339968 rw root csh 1033 root / 2 drwxr-xr-x 512 r root csh 1033 wd /mnt/crash 3 drwxr-xr-x 12 r root csh 1033 text / 17 -r-xr-xr-x 320152 r root csh 1033 15 /dev 52 crw------- ttyv1 rw root csh 1033 16 /dev 52 crw------- ttyv1 rw root csh 1033 17 /dev 52 crw------- ttyv1 rw root csh 1033 18 /dev 52 crw------- ttyv1 rw root csh 1033 19 /dev 52 crw------- ttyv1 rw root dd 1030 root / 2 drwxr-xr-x 512 r root dd 1030 wd /mnt/crash 3 drwxr-xr-x 12 r root dd 1030 text / 19 -r-xr-xr-x 20180 r root dd 1030 0 /dev 51 crw------- ttyv0 rw root dd 1030 1 /dev 51 crw------- ttyv0 rw root dd 1030 2 /dev 51 crw------- ttyv0 rw root dd 1030 3 /dev 33 crw-rw-rw- zero r root dd 1030 4 /mnt/crash 4 -rw-r--r-- 776243200 rw root zil_clean 1001 root / 2 drwxr-xr-x 512 r root zil_clean 1001 wd / 2 drwxr-xr-x 512 r root zil_clean 1000 root / 2 drwxr-xr-x 512 r root zil_clean 1000 wd / 2 drwxr-xr-x 512 r root txg_thread_enter 998 root / 2 drwxr-xr-x 512 r root txg_thread_enter 998 wd / 2 drwxr-xr-x 512 r root txg_thread_enter 997 root / 2 drwxr-xr-x 512 r root txg_thread_enter 997 wd / 2 drwxr-xr-x 512 r root txg_thread_enter 996 root / 2 drwxr-xr-x 512 r root txg_thread_enter 996 wd / 2 drwxr-xr-x 512 r root spa_zio_intr_5 994 root / 2 drwxr-xr-x 512 r root spa_zio_intr_5 994 wd / 2 drwxr-xr-x 512 r root spa_zio_issue_5 993 root / 2 drwxr-xr-x 512 r root spa_zio_issue_5 993 wd / 2 drwxr-xr-x 512 r root spa_zio_intr_4 992 root / 2 drwxr-xr-x 512 r root spa_zio_intr_4 992 wd / 2 drwxr-xr-x 512 r root spa_zio_issue_4 991 root / 2 drwxr-xr-x 512 r root spa_zio_issue_4 991 wd / 2 drwxr-xr-x 512 r root spa_zio_intr_3 990 root / 2 drwxr-xr-x 512 r root spa_zio_intr_3 990 wd / 2 drwxr-xr-x 512 r root spa_zio_issue_3 989 root / 2 drwxr-xr-x 512 r root spa_zio_issue_3 989 wd / 2 drwxr-xr-x 512 r root spa_zio_intr_2 988 root / 2 drwxr-xr-x 512 r root spa_zio_intr_2 988 wd / 2 drwxr-xr-x 512 r root spa_zio_issue_2 987 root / 2 drwxr-xr-x 512 r root spa_zio_issue_2 987 wd / 2 drwxr-xr-x 512 r root spa_zio_intr_1 986 root / 2 drwxr-xr-x 512 r root spa_zio_intr_1 986 wd / 2 drwxr-xr-x 512 r root spa_zio_issue_1 985 root / 2 drwxr-xr-x 512 r root spa_zio_issue_1 985 wd / 2 drwxr-xr-x 512 r root spa_zio_intr_0 984 root / 2 drwxr-xr-x 512 r root spa_zio_intr_0 984 wd / 2 drwxr-xr-x 512 r root spa_zio_issue_0 983 root / 2 drwxr-xr-x 512 r root spa_zio_issue_0 983 wd / 2 drwxr-xr-x 512 r root arc_reclaim_thread 967 root / 2 drwxr-xr-x 512 r root arc_reclaim_thread 967 wd / 2 drwxr-xr-x 512 r root system_taskq 966 root / 2 drwxr-xr-x 512 r root system_taskq 966 wd / 2 drwxr-xr-x 512 r root csh 955 root / 2 drwxr-xr-x 512 r root csh 955 wd /mnt/crash 3 drwxr-xr-x 12 r root csh 955 text / 17 -r-xr-xr-x 320152 r root csh 955 15 /dev 51 crw------- ttyv0 rw root csh 955 16 /dev 51 crw------- ttyv0 rw root csh 955 17 /dev 51 crw------- ttyv0 rw root csh 955 18 /dev 51 crw------- ttyv0 rw root csh 955 19 /dev 51 crw------- ttyv0 rw root getty 954 root / 2 drwxr-xr-x 512 r root getty 954 wd / 2 drwxr-xr-x 512 r root getty 954 text /usr 353290 -r-xr-xr-x 21832 r root getty 954 0 /dev 58 crw------- ttyv7 rw root getty 954 1 /dev 58 crw------- ttyv7 rw root getty 954 2 /dev 58 crw------- ttyv7 rw root login 953 root / 2 drwxr-xr-x 512 r root login 953 wd / 12741 drwxr-xr-x 512 r root login 953 text /usr 47318 -r-sr-xr-x 21712 r root login 953 0 /dev 57 crw------- ttyv6 rw root login 953 1 /dev 57 crw------- ttyv6 rw root login 953 2 /dev 57 crw------- ttyv6 rw root login 953 3* local dgram c1f49e1c <-> c1f492b0 root login 952 root / 2 drwxr-xr-x 512 r root login 952 wd / 12741 drwxr-xr-x 512 r root login 952 text /usr 47318 -r-sr-xr-x 21712 r root login 952 0 /dev 56 crw------- ttyv5 rw root login 952 1 /dev 56 crw------- ttyv5 rw root login 952 2 /dev 56 crw------- ttyv5 rw root login 952 3* local dgram c1f49ec8 <-> c1f492b0 root login 951 root / 2 drwxr-xr-x 512 r root login 951 wd / 12741 drwxr-xr-x 512 r root login 951 text /usr 47318 -r-sr-xr-x 21712 r root login 951 0 /dev 55 crw------- ttyv4 rw root login 951 1 /dev 55 crw------- ttyv4 rw root login 951 2 /dev 55 crw------- ttyv4 rw root login 951 3* local dgram c1f49560 <-> c1f492b0 root login 950 root / 2 drwxr-xr-x 512 r root login 950 wd / 12741 drwxr-xr-x 512 r root login 950 text /usr 47318 -r-sr-xr-x 21712 r root login 950 0 /dev 54 crw------- ttyv3 rw root login 950 1 /dev 54 crw------- ttyv3 rw root login 950 2 /dev 54 crw------- ttyv3 rw root login 950 3* local dgram c1f49d70 <-> c1f492b0 root login 949 root / 2 drwxr-xr-x 512 r root login 949 wd / 12741 drwxr-xr-x 512 r root login 949 text /usr 47318 -r-sr-xr-x 21712 r root login 949 0 /dev 53 crw------- ttyv2 rw root login 949 1 /dev 53 crw------- ttyv2 rw root login 949 2 /dev 53 crw------- ttyv2 rw root login 949 3* local dgram c1f49cc4 <-> c1f492b0 root login 948 root / 2 drwxr-xr-x 512 r root login 948 wd / 12741 drwxr-xr-x 512 r root login 948 text /usr 47318 -r-sr-xr-x 21712 r root login 948 0 /dev 52 crw------- ttyv1 rw root login 948 1 /dev 52 crw------- ttyv1 rw root login 948 2 /dev 52 crw------- ttyv1 rw root login 948 3* local dgram c1f4960c <-> c1f492b0 root login 947 root / 2 drwxr-xr-x 512 r root login 947 wd / 12741 drwxr-xr-x 512 r root login 947 text /usr 47318 -r-sr-xr-x 21712 r root login 947 0 /dev 51 crw------- ttyv0 rw root login 947 1 /dev 51 crw------- ttyv0 rw root login 947 2 /dev 51 crw------- ttyv0 rw root login 947 3* local dgram c1f49158 <-> c1f492b0 root moused 895 root / 2 drwxr-xr-x 512 r root moused 895 wd / 2 drwxr-xr-x 512 r root moused 895 text /usr 306306 -r-xr-xr-x 36100 r root moused 895 0 /dev 32 crw-rw-rw- null rw root moused 895 1 /dev 32 crw-rw-rw- null rw root moused 895 2 /dev 32 crw-rw-rw- null rw root moused 895 3 /dev 44 crw-rw-rw- psm0 rw root moused 895 4 /dev 67 crw------- consolectl rw root moused 895 5 /var 94234 -rw------- 3 w root cron 881 root / 2 drwxr-xr-x 512 r root cron 881 wd /var 70656 drwxr-x--- 512 r root cron 881 text /usr 306236 -r-xr-xr-x 34088 r root cron 881 0 /dev 32 crw-rw-rw- null rw root cron 881 1 /dev 32 crw-rw-rw- null rw root cron 881 2 /dev 32 crw-rw-rw- null rw root cron 881 3 /var 94233 -rw------- 3 w smmsp sendmail 875 root / 2 drwxr-xr-x 512 r smmsp sendmail 875 wd /var 164871 drwxrwx--- 512 r smmsp sendmail 875 text /usr 353339 -r-xr-sr-x 683412 r smmsp sendmail 875 0 /dev 32 crw-rw-rw- null r smmsp sendmail 875 1 /dev 32 crw-rw-rw- null w smmsp sendmail 875 2 /dev 32 crw-rw-rw- null w smmsp sendmail 875 3* local dgram c1f49000 <-> c1f4935c smmsp sendmail 875 4 /var 164873 -rw------- 49 w root sendmail 871 root / 2 drwxr-xr-x 512 r root sendmail 871 wd /var 164868 drwxr-xr-x 512 r root sendmail 871 text /usr 353339 -r-xr-sr-x 683412 r root sendmail 871 0 /dev 32 crw-rw-rw- null r root sendmail 871 1 /dev 32 crw-rw-rw- null w root sendmail 871 2 /dev 32 crw-rw-rw- null w root sendmail 871 3* internet stream tcp c1fb7974 root sendmail 871 4* local dgram c1f490ac <-> c1f492b0 root sendmail 871 5 /var 94232 -rw------- 78 w root sshd 864 root / 2 drwxr-xr-x 512 r root sshd 864 wd / 2 drwxr-xr-x 512 r root sshd 864 text /usr 306183 -r-xr-xr-x 223900 r root sshd 864 0 /dev 32 crw-rw-rw- null rw root sshd 864 1 /dev 32 crw-rw-rw- null rw root sshd 864 2 /dev 32 crw-rw-rw- null rw root sshd 864 3* internet6 stream tcp c1fb8000 root sshd 864 4* internet stream tcp c1fb7d3c root syslogd 646 root / 2 drwxr-xr-x 512 r root syslogd 646 wd / 2 drwxr-xr-x 512 r root syslogd 646 text /usr 306389 -r-xr-xr-x 35864 r root syslogd 646 0 /dev 32 crw-rw-rw- null rw root syslogd 646 1 /dev 32 crw-rw-rw- null rw root syslogd 646 2 /dev 32 crw-rw-rw- null rw root syslogd 646 3 /var 94223 -rw------- 3 w root syslogd 646 4* local dgram c1f4935c root syslogd 646 5* local dgram c1f492b0 root syslogd 646 6* internet6 dgram udp c1f3cbb0 root syslogd 646 7* internet dgram udp c1f3cb00 root syslogd 646 8 /dev 6 crw------- klog r root syslogd 646 10 - - bad - root syslogd 646 11 /var 117771 -rw-r--r-- 45749 w root syslogd 646 12 /var 117773 -rw------- 59 w root syslogd 646 13 /var 117766 -rw------- 1380 w root syslogd 646 14 /var 117770 -rw-r----- 543 w root syslogd 646 15 /var 117769 -rw-r--r-- 59 w root syslogd 646 16 /var 117774 -rw------- 59 w root syslogd 646 17 /var 117767 -rw------- 369 w root syslogd 646 18 /var 117768 -rw------- 59 w root syslogd 646 19 /var 117772 -rw-r----- 59 w root devd 500 root / 2 drwxr-xr-x 512 r root devd 500 wd / 2 drwxr-xr-x 512 r root devd 500 text / 38235 -r-xr-xr-x 380172 r root devd 500 0 /dev 32 crw-rw-rw- null rw root devd 500 1 /dev 32 crw-rw-rw- null rw root devd 500 2 /dev 32 crw-rw-rw- null rw root devd 500 3 /dev 5 crw------- devctl r root devd 500 4* local stream c1f4a000 root devd 500 5 /var 94219 -rw------- 3 w _dhcp dhclient 499 root /var 141312 dr-xr-xr-x 512 r _dhcp dhclient 499 wd /var 141312 dr-xr-xr-x 512 r _dhcp dhclient 499 jail /var 141312 dr-xr-xr-x 512 r _dhcp dhclient 499 text / 38237 -r-xr-xr-x 78316 r _dhcp dhclient 499 0 /dev 32 crw-rw-rw- null rw _dhcp dhclient 499 1 /dev 32 crw-rw-rw- null rw _dhcp dhclient 499 2 /dev 32 crw-rw-rw- null rw _dhcp dhclient 499 4* local stream c1f4a000 _dhcp dhclient 499 5* route raw 0 c1f3a7a8 _dhcp dhclient 499 6* pipe c1eb7834 <-> c1eb7780 0 rw _dhcp dhclient 499 7 /var 188428 ---------- 912 w _dhcp dhclient 499 8 /dev 14 crw------- bpf rw root dhclient 483 root / 2 drwxr-xr-x 512 r root dhclient 483 wd / 2 drwxr-xr-x 512 r root dhclient 483 text / 38237 -r-xr-xr-x 78316 r root dhclient 483 0 /dev 32 crw-rw-rw- null rw root dhclient 483 1 /dev 32 crw-rw-rw- null rw root dhclient 483 2 /dev 32 crw-rw-rw- null rw root dhclient 483 4* local stream c1f4a000 root dhclient 483 5* pipe c1eb7780 <-> c1eb7834 0 rw root adjkerntz 123 root / 2 drwxr-xr-x 512 r root adjkerntz 123 wd / 2 drwxr-xr-x 512 r root adjkerntz 123 text / 38224 -r-xr-xr-x 7360 r root adjkerntz 123 0 /dev 32 crw-rw-rw- null rw root adjkerntz 123 1 /dev 32 crw-rw-rw- null rw root adjkerntz 123 2 /dev 32 crw-rw-rw- null rw root init 1 root / 2 drwxr-xr-x 512 r root init 1 wd / 2 drwxr-xr-x 512 r root init 1 text / 38260 -r-xr-xr-x 653392 r root kernel 0 root / 2 drwxr-xr-x 512 r root kernel 0 wd / 2 drwxr-xr-x 512 r ------------------------------------------------------------------------ dmesg Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-CURRENT-200809 #0: Sun Sep 7 22:13:46 UTC 2008 root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2998.62-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf38 Stepping = 8 Features=0xfebfbff Features2=0x11 real memory = 134217728 (128 MB) avail memory = 117182464 (111 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.10.5.10 (AR5210, AR5211, AR5212, AR5416, RF5111, RF5112, RF2413, RF5413, RF2133, RF2425, RF2417) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] uhci0: port 0x1060-0x107f irq 19 at device 7.2 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered pci0: at device 7.3 (no driver attached) vgapci0: port 0x1400-0x140f mem 0xf0000000-0xf7ffffff,0xe8000000-0xe87fffff at device 15.0 on pci0 mpt0: port 0x1080-0x10ff mem 0xe8800000-0xe8800fff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 pcib2: mem 0xe8801000-0xe8801fff at device 17.0 on pci0 pci2: on pcib2 pci2: at device 0.0 (no driver attached) ehci0: mem 0xe8900000-0xe8900fff irq 19 at device 1.0 on pci2 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb1: EHCI version 1.0 usb1: on ehci0 usb1: USB revision 2.0 uhub1: <(0x15ad) EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb1 uhub1: 6 ports with 6 removable, self powered le0: port 0x2000-0x207f irq 16 at device 2.0 on pci2 le0: 16 receive buffers, 4 transmit buffers le0: Ethernet address: 00:0c:29:56:d0:4f le0: [ITHREAD] acpi_acad0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 cpu0: on acpi0 acpi_throttle0: on cpu0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xdc000-0xdffff,0xe0000-0xe3fff pnpid ORM0000 on isa0 ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 ppbus0: [ITHREAD] plip0: on ppbus0 plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 uart0: [FILTER] uart1: <16550 or compatible> at port 0x2f8-0x2ff irq 3 on isa0 uart1: [FILTER] vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 2998615875 Hz quality 800 Timecounters tick every 1.000 msec Waiting 5 seconds for SCSI devices to settle ad0: 8192MB at ata0-master UDMA33 ad1: 8192MB at ata0-slave UDMA33 acd0: CDROM at ata1-master UDMA33 WARNING: WITNESS option enabled, expect reduced performance. lock order reversal: (sleepable after non-sleepable) 1st 0xc1e9601c struct mount mtx (struct mount mtx) @ /usr/src/sys/kern/vfs_subr.c:345 2nd 0xc1e96000 vfslock (vfslock) @ /usr/src/sys/kern/vfs_subr.c:372 KDB: stack backtrace: db_trace_self_wrapper(c0b6da3f,c1b70b0c,c07f1dc5,4,c0b693d1,...) at db_trace_self_wrapper+0x26 kdb_backtrace(4,c0b693d1,c0dea230,c1c76ea8,c1b70b64,...) at kdb_backtrace+0x29 _witness_debugger(c0b70309,c1e96000,c0b76adb,c1c76ea8,c0b77080,...) at _witness_debugger+0x25 witness_checkorder(c1e96000,1,c0b77080,174,c0b693d1,...) at witness_checkorder+0x7c9 __lockmgr_args(c1e96000,200100,c1e9601c,0,0,...) at __lockmgr_args+0x230 vfs_busy(c1e96000,200,0,1,c1cbed20,...) at vfs_busy+0x1bc vfs_mount_alloc(0,c0c3c740,c0b76e26,c1cc3800,c08318a0,...) at vfs_mount_alloc+0x74 vfs_mountroot(c0cafbb0,4,c0b6533b,264,0,...) at vfs_mountroot+0x272 start_init(0,c1b70d38,c0b66d1e,322,c1cbcd0c,...) at start_init+0x65 fork_exit(c077a0c0,0,c1b70d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc1b70d70, ebp = 0 --- lock order reversal: (sleepable after non-sleepable) 1st 0xc1e05e10 vnode interlock (vnode interlock) @ /usr/src/sys/fs/devfs/devfs_vnops.c:286 2nd 0xc1e05df4 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2051 KDB: stack backtrace: db_trace_self_wrapper(c0b6da3f,c1b70a2c,c07f1dc5,4,c0b693d1,...) at db_trace_self_wrapper+0x26 kdb_backtrace(4,c0b693d1,c0dea230,c1c76fe0,c1b70a84,...) at kdb_backtrace+0x29 _witness_debugger(c0b70309,c1e05df4,c0b60c4e,c1c76fe0,c0b77080,...) at _witness_debugger+0x25 witness_checkorder(c1e05df4,9,c0b77080,803,c1b70ab6,...) at witness_checkorder+0x7c9 __lockmgr_args(c1e05df4,80100,c1e05e10,0,0,...) at __lockmgr_args+0x790 vop_stdlock(c1b70b88,c0b60e2b,c0b63fdf,80100,c1e05d9c,...) at vop_stdlock+0x62 VOP_LOCK1_APV(c0c3c820,c1b70b88,c0c7cea0,c1e05d9c,80100,...) at VOP_LOCK1_APV+0xa5 _vn_lock(c1e05d9c,80100,c0b77080,803,8,...) at _vn_lock+0x5e vget(c1e05d9c,80100,c1cbed20,11f,c0b60da1,...) at vget+0xc9 devfs_allocv(c1db5880,c1e96000,c1b70c20,c1cbed20,c0dea230,...) at devfs_allocv+0x11a devfs_root(c1e96000,80000,c0e15d54,c1cbed20,4,...) at devfs_root+0x51 set_rootvnode(c0e15d40,0,c0b769dd,5fa,c08318a0,...) at set_rootvnode+0x2d vfs_mountroot(c0cafbb0,4,c0b6533b,264,0,...) at vfs_mountroot+0x356 start_init(0,c1b70d38,c0b66d1e,322,c1cbcd0c,...) at start_init+0x65 fork_exit(c077a0c0,0,c1b70d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc1b70d70, ebp = 0 --- Trying to mount root from ufs:/dev/ad0s1a lock order reversal: (sleepable after non-sleepable) 1st 0xc1e05aec vnode interlock (vnode interlock) @ /usr/src/sys/kern/vfs_hash.c:79 2nd 0xc1e05ad0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2051 KDB: stack backtrace: db_trace_self_wrapper(c0b6da3f,c1b708ec,c07f1dc5,4,c0b693d1,...) at db_trace_self_wrapper+0x26 kdb_backtrace(4,c0b693d1,c0de9520,c1c77118,c1b70944,...) at kdb_backtrace+0x29 _witness_debugger(c0b70309,c1e05ad0,c0b648d3,c1c77118,c0b77080,...) at _witness_debugger+0x25 witness_checkorder(c1e05ad0,9,c0b77080,803,c1b70974,...) at witness_checkorder+0x7c9 __lockmgr_args(c1e05ad0,80100,c1e05aec,0,0,...) at __lockmgr_args+0x790 ffs_lock(c1b70a50,c0b7666f,c0b63fdf,80100,c1e05a78,...) at ffs_lock+0x8a VOP_LOCK1_APV(c0c62900,c1b70a50,c0c7cea0,c1e05a78,80100,...) at VOP_LOCK1_APV+0xa5 _vn_lock(c1e05a78,80100,c0b77080,803,4,...) at _vn_lock+0x5e vget(c1e05a78,80100,c1cbed20,50,0,...) at vget+0xc9 vfs_hash_get(c1e95d20,2,80000,c1cbed20,c1b70b64,...) at vfs_hash_get+0xe6 ffs_vgetf(c1e95d20,2,80000,c1b70b64,0,...) at ffs_vgetf+0x49 ffs_vget(c1e95d20,2,80000,c1b70b64,c0b769dd,...) at ffs_vget+0x2e ufs_root(c1e95d20,80000,c0e15d54,c1cbed20,4,...) at ufs_root+0x28 set_rootvnode(c1e05e10,0,c0b769dd,619,c0b63fe1,...) at set_rootvnode+0x2d vfs_mountroot_try(c0b76fdc,c0b63fe1,c0b5c08b,1,c08318a0,...) at vfs_mountroot_try+0x2cd vfs_mountroot(c0cafbb0,4,c0b6533b,264,0,...) at vfs_mountroot+0x418 start_init(0,c1b70d38,c0b66d1e,322,c1cbcd0c,...) at start_init+0x65 fork_exit(c077a0c0,0,c1b70d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc1b70d70, ebp = 0 --- lock order reversal: (sleepable after non-sleepable) 1st 0xc1e05b20 bufobj interlock (bufobj interlock) @ /usr/src/sys/kern/vfs_bio.c:2429 2nd 0xc1a71280 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2443 KDB: stack backtrace: db_trace_self_wrapper(c0b6da3f,c1b70728,c07f1dc5,4,c0b693d1,...) at db_trace_self_wrapper+0x26 kdb_backtrace(4,c0b693d1,c0de9578,c1c74670,c1b70780,...) at kdb_backtrace+0x29 _witness_debugger(c0b70309,c1a71280,c0b759a6,c1c74670,c0b74f55,...) at _witness_debugger+0x25 witness_checkorder(c1a71280,9,c0b74f55,98b,c1b707a4,...) at witness_checkorder+0x7c9 __lockmgr_args(c1a71280,81900,c1e05b20,c0b75902,50,...) at __lockmgr_args+0x790 getblk(c1e05a78,0,0,800,0,...) at getblk+0x153 breadn(c1e05a78,0,0,800,0,...) at breadn+0x44 bread(c1e05a78,0,0,800,0,...) at bread+0x4c ffs_blkatoff(c1e05a78,0,0,0,c1b709a4,...) at ffs_blkatoff+0xd1 ufs_lookup(c1b709e8,c1e05a78,c1b70b38,c1e05a78,c1b70a08,...) at ufs_lookup+0x2e6 VOP_CACHEDLOOKUP_APV(c0c62900,c1b709e8,c1b70b38,c1b70b24,c1cc3800,...) at VOP_CACHEDLOOKUP_APV+0xa5 vfs_cache_lookup(c1b70a68,c1b70a68,500000c,80000,c1e05a78,...) at vfs_cache_lookup+0xd0 VOP_LOOKUP_APV(c0c62900,c1b70a68,c0b7678b,1b0,c1b70b24,...) at VOP_LOOKUP_APV+0xa5 lookup(c1b70b0c,c0b7678b,d8,c0,c1c8d22c,...) at lookup+0x57e namei(c1b70b0c,c1b70b24,c07f1c0c,c0de9588,c0b76e22,...) at namei+0x44b kern_unlinkat(c1cbed20,ffffff9c,c0b76e22,1,c1b70c5c,...) at kern_unlinkat+0x46 kern_unlink(c1cbed20,c0b76e22,1,635,0,...) at kern_unlink+0x27 vfs_mountroot_try(c0b76fdc,c0b63fe1,c0b5c08b,1,c08318a0,...) at vfs_mountroot_try+0x478 vfs_mountroot(c0cafbb0,4,c0b6533b,264,0,...) at vfs_mountroot+0x418 start_init(0,c1b70d38,c0b66d1e,322,c1cbcd0c,...) at start_init+0x65 fork_exit(c077a0c0,0,c1b70d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc1b70d70, ebp = 0 --- lock order reversal: (sleepable after non-sleepable) 1st 0xc1e058d4 vnode interlock (vnode interlock) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:167 2nd 0xc1df7600 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:178 KDB: stack backtrace: db_trace_self_wrapper(c0b6da3f,c87cf790,c07f1dc5,4,c0b693d1,...) at db_trace_self_wrapper+0x26 kdb_backtrace(4,c0b693d1,c0de9578,c1c77180,c87cf7e8,...) at kdb_backtrace+0x29 _witness_debugger(c0b70309,c1df7600,c0b8e3b1,c1c77180,c0b8e04a,...) at _witness_debugger+0x25 witness_checkorder(c1df7600,1,c0b8e04a,b2,c87cf80c,...) at witness_checkorder+0x7c9 __lockmgr_args(c1df7600,200100,c1e058d4,0,0,...) at __lockmgr_args+0x230 ufsdirhash_build(c1e9ce0c,a00,0,c1dc1af0,c0b76072,...) at ufsdirhash_build+0x1d8 ufs_lookup(c87cf9c4,c1e05860,c87cfbac,c1e05860,c87cf9e4,...) at ufs_lookup+0xf0 VOP_CACHEDLOOKUP_APV(c0c62900,c87cf9c4,c87cfbac,c87cfb98,c1cc3800,...) at VOP_CACHEDLOOKUP_APV+0xa5 vfs_cache_lookup(c87cfa44,c87cfa44,c87cfb80,80000,c1e05860,...) at vfs_cache_lookup+0xd0 VOP_LOOKUP_APV(c0c62900,c87cfa44,c0b7678b,1b0,c87cfb98,...) at VOP_LOOKUP_APV+0xa5 lookup(c87cfb80,c0b7678b,d8,c0,c1eb592c,...) at lookup+0x57e namei(c87cfb80,0,c0b66b4c,169,c1eb4538,...) at namei+0x44b kern_execve(c1dc1af0,c87cfc58,0,810432c,8104348,c7799000,c7799000,c7799027,c7799057,c77d9000,3ffa9,3,3,0) at kern_execve+0x267 execve(c1dc1af0,c87cfcf8,c,c87cfd38,c0c40388,...) at execve+0x4c syscall(c87cfd38) at syscall+0x2a3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (59, FreeBSD ELF32, execve), eip = 0x2815340b, esp = 0xbfbfe9ec, ebp = 0xbfbfea08 --- Entropy harvesting: interrupts ethernet point_to_point kickstart . /dev/ad0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad0s1a: clean, 125964 free (2980 frags, 15373 blocks, 1.5% fragmentation) /dev/ad0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad0s1e: clean, 253809 free (41 frags, 31721 blocks, 0.0% fragmentation) /dev/ad0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad0s1f: clean, 1515463 free (5015 frags, 188806 blocks, 0.3% fragmentation) /dev/ad0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad0s1d: clean, 1012902 free (78 frags, 126603 blocks, 0.0% fragmentation) Starting Network: lo0 le0. DHCPREQUEST on le0 to 255.255.255.255 port 67 DHCPACK from 192.168.204.254 bound to 192.168.204.128 -- renewal in 900 seconds. Sep 23 20:38:34 syslogd: /var/log/slip.log: No such file or directory Configuring syscons: keymap blanktime . Tue Sep 23 20:38:39 CEST 2008 Sep 23 20:38:44 test login: ROOT LOGIN (root) ON ttyv0 This module (opensolaris) contains code covered by the Common Development and Distribution License (CDDL) see http://opensolaris.org/os/licensing/opensolaris_license/ WARNING: ZFS is considered to be an experimental feature in FreeBSD. ZFS WARNING: Recommended minimum RAM size is 512MB; expect unstable behavior. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 6 ZFS storage pool version 6 lock order reversal: (sleepable after non-sleepable) 1st 0xc21a0df4 zfs (zfs) @ /usr/src/sys/kern/vfs_lookup.c:432 2nd 0xc1f2ed04 vnode interlock (vnode interlock) @ /usr/src/sys/kern/vfs_cache.c:442 3rd 0xc1f2ece8 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2051 KDB: stack backtrace: db_trace_self_wrapper(c0b6da3f,c88478dc,c07f1dc5,4,c0b693d1,...) at db_trace_self_wrapper+0x26 kdb_backtrace(4,c0b693d1,c0de91b0,c1c77528,c8847934,...) at kdb_backtrace+0x29 _witness_debugger(c0b70322,c1f2ece8,c224594d,c1c77528,c0b77080,...) at _witness_debugger+0x25 witness_checkorder(c1f2ece8,9,c0b77080,803,c0c46684,...) at witness_checkorder+0x7c9 __lockmgr_args(c1f2ece8,80100,c1f2ed04,0,0,...) at __lockmgr_args+0x790 vop_stdlock(c8847a38,c8847a3c,c07a625c,80100,c1f2ec90,...) at vop_stdlock+0x62 VOP_LOCK1_APV(c2249d60,c8847a38,c0c7cea0,c1f2ec90,80100,...) at VOP_LOCK1_APV+0xa5 _vn_lock(c1f2ec90,80100,c0b77080,803,4,...) at _vn_lock+0x5e vget(c1f2ec90,80100,c21ab000,1bb,c8847ab8,...) at vget+0xc9 cache_lookup(c21a0d9c,c8847c14,c8847c28,c8847c14,c215d100,...) at cache_lookup+0x4c0 vfs_cache_lookup(c8847b54,c8847b54,c8847bfc,80000,c21a0d9c,...) at vfs_cache_lookup+0xa7 VOP_LOOKUP_APV(c2249d60,c8847b54,c0b7678b,1b0,c8847c14,...) at VOP_LOOKUP_APV+0xa5 lookup(c8847bfc,c0b7678b,d8,c0,c289e42c,...) at lookup+0x57e namei(c8847bfc,c07f4b45,c1ea4bd0,c8847c58,c215d100,...) at namei+0x44b kern_pathconf(c21ab000,bfbfdd53,0,3b,c8847d2c,...) at kern_pathconf+0x41 pathconf(c21ab000,c8847cf8,8,c0b590be,c0c40fe8,...) at pathconf+0x29 syscall(c8847d38) at syscall+0x2a3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (191, FreeBSD ELF32, pathconf), eip = 0x28146d6f, esp = 0xbfbfd8fc, ebp = 0xbfbfe178 --- Sep 23 20:40:13 test sshd[1024]: error: PAM: authentication error for root from localhost Sep 23 20:40:38 test login: ROOT LOGIN (root) ON ttyv1 Sep 23 20:40:52 test login: ROOT LOGIN (root) ON ttyv2 Sep 23 20:41:09 test login: ROOT LOGIN (root) ON ttyv3 Sep 23 20:41:40 test login: ROOT LOGIN (root) ON ttyv4 Sep 23 20:42:08 test login: ROOT LOGIN (root) ON ttyv5 Sep 23 20:43:36 test login: ROOT LOGIN (root) ON ttyv6 panic: kmem_malloc(131072): kmem_map too small: 35979264 total allocated cpuid = 0 KDB: enter: panic panic: from debugger cpuid = 0 Uptime: 8m42s Physical memory: 115 MB Dumping 66 MB: 51 35 19 3 ------------------------------------------------------------------------ kernel config config: File /boot/kernel/kernel doesn't contain configuration file. Either unsupported, or not compiled with INCLUDE_CONFIG_FILE