Login or Register to Ask a Question and Join Our Community


Solaris

vmstat and waiting kernel threads

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Solaris vmstat and waiting kernel threads
# 1  
Old 02-16-2006
vmstat and waiting kernel threads
I am running Apache:

# httpd -version
Server version: Apache/1.3.26 (Unix)
Server built: Jul 15 2002 03:27:01

On a V280:

# uname -a
SunOS bsmweb01 5.8 Generic_108528-23 sun4u sparc SUNW,Sun-Fire-280R
#

I am assessing the server for additional applications. I ran vmstat 10 and a high number of waiting processes were reported; this output was typical of the vmstat output over a 30 minute period:

procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s6 sd sd -- in sy cs us sy id
0 0 42 2007552 1646280 0 0 0 0 0 0 0 0 0 0 0 130 47 123 0 0 100
0 0 42 2007552 1646280 0 0 0 0 0 0 0 0 2 0 0 130 42 122 0 1 99
0 0 42 2007552 1646280 0 0 0 0 0 0 0 0 0 0 0 131 46 125 0 1 99
0 0 42 2006280 1644288 106 320 73 0 0 0 0 0 0 8 0 151 1373 179 6 5 89
0 0 42 2007552 1646176 1 2 2 0 0 0 0 0 4 2 0 145 60 135 0 1 99
0 0 42 2006448 1645240 120 259 0 1 1 0 0 0 1 0 0 127 691 145 1 1 98
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 124 49 121 0 0 100
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 7 0 0 144 43 126 0 1 99
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 133 45 121 0 0 100
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 128 44 120 0 0 100
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 129 40 121 0 1 99
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 132 45 123 0 0 100
0 0 42 2003376 1640672 117 415 111 0 0 0 0 0 0 11 0 162 1606 213 11 1 87
0 0 42 2007552 1646080 0 0 0 0 0 0 0 0 4 2 0 134 39 124 0 1 99

CPU was consistently reporting 98-100% idle, negligible scan rate. Ample free memory was also reported. The is ample disk space on the server. Runing prstat gave no CPU/memory intensive processes over the same time period.
What causes the kernel threads to be waiting?
Will this affect server performance?
I have looked at netstat - a and found:

# netstat -a

UDP: IPv4
Local Address Remote Address State
-------------------- -------------------- -------
*.sunrpc Idle
*.* Unbound
*.32771 Idle
*.lockd Idle
*.* Unbound
*.* Unbound
*.32772 Idle
*.syslog Idle
*.* Unbound
*.630 Idle
*.2024 Idle
*.32782 Idle
*.nfsd Idle
*.177 Idle
*.32783 Idle
*.161 Idle
*.* Unbound
*.name Idle
*.biff Idle
*.talk Idle
*.time Idle
*.echo Idle
*.discard Idle
*.daytime Idle
*.chargen Idle
*.35517 Idle
*.35518 Idle
*.35519 Idle
*.35520 Idle
*.35521 Idle
*.35522 Idle
*.35523 Idle
*.35524 Idle
*.* Unbound

UDP: IPv6
Local Address Remote Address State
If
--------------------------------- --------------------------------- ---------- -
----
*.time Idle
*.echo Idle
*.discard Idle
*.daytime Idle
*.chargen Idle

TCP: IPv4
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ----- ------ ----- ------ -------
*.* *.* 0 0 24576 0 IDLE
*.sunrpc *.* 0 0 24576 0 LISTEN
*.* *.* 0 0 24576 0 IDLE
*.lockd *.* 0 0 24576 0 LISTEN
*.32771 *.* 0 0 24576 0 LISTEN
*.32773 *.* 0 0 24576 0 LISTEN
*.fs *.* 0 0 24576 0 LISTEN
*.2201 *.* 0 0 24576 0 LISTEN
*.2200 *.* 0 0 24576 0 LISTEN
*.5987 *.* 0 0 24576 0 LISTEN
*.898 *.* 0 0 24576 0 LISTEN
*.32783 *.* 0 0 24576 0 LISTEN
*.wnn6 *.* 0 0 24576 0 LISTEN
*.9010 *.* 0 0 24576 0 LISTEN
*.8888 *.* 0 0 24576 0 LISTEN
*.32784 *.* 0 0 24576 0 LISTEN
*.nfsd *.* 0 0 24576 0 LISTEN
*.32785 *.* 0 0 24576 0 LISTEN
*.32786 *.* 0 0 24576 0 LISTEN
*.smtp *.* 0 0 24576 0 LISTEN
*.smtp *.* 0 0 24576 0 LISTEN
*.submission *.* 0 0 24576 0 LISTEN
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.bpdbm *.* 0 0 24576 0 LISTEN
*.bpjobd *.* 0 0 24576 0 LISTEN
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.vmd *.* 0 0 24576 0 LISTEN
*.tldcd *.* 0 0 24576 0 LISTEN
*.bprd *.* 0 0 24576 0 LISTEN
*.46284 *.* 0 0 24576 0 LISTEN
*.ftp *.* 0 0 24576 0 LISTEN
*.telnet *.* 0 0 24576 0 LISTEN
*.shell *.* 0 0 24576 0 LISTEN
*.shell *.* 0 0 24576 0 LISTEN
*.login *.* 0 0 24576 0 LISTEN
*.exec *.* 0 0 24576 0 LISTEN
*.exec *.* 0 0 24576 0 LISTEN
*.uucp *.* 0 0 24576 0 LISTEN
*.finger *.* 0 0 24576 0 LISTEN
*.time *.* 0 0 24576 0 LISTEN
*.echo *.* 0 0 24576 0 LISTEN
*.discard *.* 0 0 24576 0 LISTEN
*.daytime *.* 0 0 24576 0 LISTEN
*.chargen *.* 0 0 24576 0 LISTEN
*.46654 *.* 0 0 24576 0 LISTEN
*.printer *.* 0 0 24576 0 LISTEN
*.dtspc *.* 0 0 24576 0 LISTEN
*.bpcd *.* 0 0 24576 0 LISTEN
*.vnetd *.* 0 0 24576 0 LISTEN
*.vopied *.* 0 0 24576 0 LISTEN
*.bpjava-msvc *.* 0 0 24576 0 LISTEN
*.hengili-poly *.* 0 0 24576 0 LISTEN
*.hengite-poly *.* 0 0 24576 0 LISTEN
*.midsoli-poly *.* 0 0 24576 0 LISTEN
*.zedali-poly *.* 0 0 24576 0 LISTEN
*.aegonli-poly *.* 0 0 24576 0 LISTEN
*.aegonte-poly *.* 0 0 24576 0 LISTEN
*.aegonde-poly *.* 0 0 24576 0 LISTEN
*.ofgemli-poly *.* 0 0 24576 0 LISTEN
*.ofgemde-poly *.* 0 0 24576 0 LISTEN
*.ofgemte-poly *.* 0 0 24576 0 LISTEN
*.reabcli-poly *.* 0 0 24576 0 LISTEN
*.reabcde-poly *.* 0 0 24576 0 LISTEN
*.midsote-poly *.* 0 0 24576 0 LISTEN
*.zedate-poly *.* 0 0 24576 0 LISTEN
*.56914 *.* 0 0 24576 0 LISTEN
*.56915 *.* 0 0 24576 0 LISTEN
*.56916 *.* 0 0 24576 0 LISTEN
*.56918 *.* 0 0 24576 0 LISTEN
*.56919 *.* 0 0 24576 0 LISTEN
*.80 *.* 0 0 24576 0 LISTEN
bsmweb01.telnet 172.17.100.187.1162 63741 1 24820 0 ESTABLISHED
*.* *.* 0 0 24576 0 IDLE
bsmweb01.52741 172.17.100.187.6000 64512 0 24820 0 ESTABLISHED
bsmweb01.telnet 172.17.100.187.1817 63291 0 24820 0 ESTABLISHED
bsmweb01.80 172.16.50.113.4734 63204 0 24820 0 FIN_WAIT_2
bsmweb01.80 172.16.50.113.4735 63981 0 24820 0 FIN_WAIT_2
bsmweb01.midsoli-poly bsmweb01.53877 32768 0 32768 0 TIME_WAIT
bsmweb01.53878 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.midsoli-poly bsmweb01.53879 32768 0 32768 0 TIME_WAIT
bsmweb01.53880 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.bpjobd bsmweb01.53881 32768 0 32768 0 TIME_WAIT
bsmweb01.951 bsmweb01.bpcd 32768 0 32768 0 TIME_WAIT
bsmweb01.914 bsmweb01.593 32768 0 32768 0 TIME_WAIT
bsmweb01.692 bsmweb01.bpcd 32768 0 32768 0 TIME_WAIT
bsmweb01.625 bsmweb01.532 32768 0 32768 0 TIME_WAIT
bsmweb01.bpcd bsmweb01.591 32768 0 32768 0 TIME_WAIT
bsmweb01.957 bsmweb01.768 32768 0 32768 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53883 32768 0 32768 0 TIME_WAIT
bsmweb01.53884 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.zedali-poly bsmweb01.53885 32768 0 32768 0 TIME_WAIT
bsmweb01.53886 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53887 32768 0 32768 0 TIME_WAIT
bsmweb01.53888 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53889 32768 0 32768 0 TIME_WAIT
bsmweb01.53890 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53891 32768 0 32768 0 TIME_WAIT
bsmweb01.53892 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53893 32768 0 32768 0 TIME_WAIT
bsmweb01.53894 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.bpjobd bsmweb01.53896 32768 0 32768 0 TIME_WAIT
bsmweb01.785 bsmweb01.bpcd 32768 0 32768 0 TIME_WAIT
bsmweb01.576 bsmweb01.995 32768 0 32768 0 TIME_WAIT
bsmweb01.866 bsmweb01.bpcd 32768 0 32768 0 TIME_WAIT
bsmweb01.908 bsmweb01.954 32768 0 32768 0 TIME_WAIT
bsmweb01.bpcd bsmweb01.900 32768 0 32768 0 TIME_WAIT
bsmweb01.nfsd bsmapp01.866 24820 0 24820 0 ESTABLISHED
bsmweb01.828 bsmweb01.670 32768 0 32768 0 TIME_WAIT
bsmweb01.80 192.168.127.9.35795 64929 0 24820 0 ESTABLISHED
*.* *.* 0 0 24576 0 IDLE

TCP: IPv6
Local Address Remote Address Swind Send-Q
Rwind Recv-Q State If
--------------------------------- --------------------------------- ----- ------
----- ------ ----------- -----
*.* *.* 0 0
24576 0 IDLE
*.smtp *.* 0 0
24576 0 LISTEN
*.ftp *.* 0 0
24576 0 LISTEN
*.telnet *.* 0 0
24576 0 LISTEN
*.shell *.* 0 0
24576 0 LISTEN
*.login *.* 0 0
24576 0 LISTEN
*.exec *.* 0 0
24576 0 LISTEN
*.finger *.* 0 0
24576 0 LISTEN
*.time *.* 0 0
24576 0 LISTEN
*.echo *.* 0 0
24576 0 LISTEN
*.discard *.* 0 0
24576 0 LISTEN
*.daytime *.* 0 0
24576 0 LISTEN
*.chargen *.* 0 0
24576 0 LISTEN
*.printer *.* 0 0
24576 0 LISTEN

Active UNIX domain sockets
Address Type Vnode Conn Local Addr Remote Addr
300035a1958 stream-ord 30003589720 00000000 /tmp/jd_sockV6
300035a1b08 stream-ord 30002fff8f0 00000000 /dev/ccv
300035a1cb8 stream-ord 30002fffac8 00000000 /dev/kkcv

The web applications on this server connect to oracle databases on another Sun server, bsmdb02 using port 1521. The status of alot of these connections is TIME_WAIT, although the OS process (from the prstat command) only exists for the duration of the http transaction.
Is the fact that there are connections reported at TIME_WAIT for completed transactions with no evidence from prstat causing waiting kernel threads to be reported?

Thanks,

Jabberwocky
# 2  
Old 05-04-2009
Reason for this problem
Hi,
Did you find out the reason for this problem? even I'm facing this kind of issue where waiting processes is around 39 constantly and I don't know what to do? or where i can get more information?

kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m2 m3 in sy cs us sy id
0 1 15 11161456 2852688 576 2174 1083 146 256 0 1017 6 5 5 12 31001 58111 18725 12 14 74
0 1 39 11239432 1948504 405 3883 0 55 55 0 0 0 0 0 0 24826 60924 17867 9 5 86
0 0 39 11236704 1945744 55 677 0 0 0 0 0 0 0 0 0 19912 42956 14285 7 3 90
0 1 39 11238120 1945776 56 652 0 24 24 0 0 0 0 0 0 14315 29272 10417 5 2 93
0 2 39 11239112 1948520 66 656 0 0 0 0 0 0 0 0 0 18306 39397 12786 6 3 91
0 2 39 11239536 1948456 207 1360 0 0 0 0 0 0 0 0 0 13299 30573 9671 6 3 92
0 1 39 11239232 1948464 407 3873 0 47 47 0 0 0 0 0 0 12782 35371 10256 5 3 92
0 1 39 11237952 1945608 56 682 0 0 0 0 0 36 35 35 0 26613 51200 17998 8 3 89
0 0 39 11237744 1945584 55 661 0 32 32 0 0 0 0 0 0 26910 55649 18525 9 4 87
0 1 39 11240456 1948072 8886 14208 0 8 8 0 0 0 0 0 0 32042 67780 21963 12 6 81
0 0 39 11241840 1948144 793 1816 0 0 0 0 0 0 0 0 0 29625 53072 20249 9 4 87
0 0 39 11238800 1948072 403 3884 0 55 55 0 0 0 0 0 0 28807 57615 20346 8 5 88
0 1 39 11237080 1945152 56 673 0 0 0 0 0 0 0 0 0 31001 56950 20858 10 4 86
0 0 39 11236016 1945120 55 640 0 39 39 0 0 0 0 0 0 30219 58501 20653 10 4 86
0 1 39 11239576 1947800 67 1031 0 0 0 0 0 0 0 0 0 36109 63614 24731 12 5 84

Please help.

Regards,
Roopa.
# 3  
Old 05-04-2009
what solaris OS release/version ?
# 4  
Old 05-05-2009
os version
SunOS 5.10
# 5  
Old 05-05-2009
MaxClients -defines how many simultaneous requests can be served.
Any connection requests from browsers that come in after that will be queued. Default hard limit of 256.
IT Is ok to set high for static content, but bad for rendered pages.

MaxSpareServers
MinSpareServers
StartServers
ServerLimit

Thrashing -where the system is just swapping pages from physical memory to virtual memory (on disk), and vice versa, without doing any real work.

Apache processes with modules (mod_perl, mod_python, mod_php) can easily be 21MB per process.Can improve with PHP op-cache/accelerator, then you can make each Apache process take as little as 12MB.

-this is where one_liner for RSS*sort processes by mem usage
# ps auwxx --sort rss
# icps "semiphores"

-if apache is high mem usage, check MaxRequestsPerChild and keepalives
Last edited by incredible; 05-05-2009 at 02:52 AM..
# 6  
Old 05-05-2009
Hi
the first post in this thread was from someone else. (jabberwocky)....
I posted my query here since the findings are similar.

Thanks.
Roopa.
# 7  
Old 05-05-2009
Quote:
Originally Posted by roopaprabhun
I'm facing this kind of issue where waiting processes is around 39 constantly and I don't know what to do?
These 39 processes have been swapped out in the past to free RAM because of a strong shortage at that time.
As these processes are inactive (likely unused daemons), they remain dormant. There is nothing particular to do.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Linux

Vmstat

I m checking idle time using vmstat, below are the results var=$(ssh wmtmgr@$hostname vmstat | tail -1 | awk '{print $15}') 89 and now im subtracting 89 with 100 & im getting expected results expr 100 - $var 11 Now How can I get the result 11 in one line code? (4 Replies)
Discussion started by: sam@sam
4 Replies

2. Linux

Unload kernel module at boot time (Debian Wheezy 7.2, 3.2.0-4-686-pae kernel)

Hi everyone, I am trying to prevent the ehci_hcd kernel module to load at boot time. Here's what I've tried so far: 1) Add the following line to /etc/modprobe.d/blacklist.conf (as suggested here): 2) Blacklisted the module by adding the following string to 3) Tried to blacklist the module... (0 Replies)
Discussion started by: gacanepa
0 Replies

3. AIX

AIX: mkdev is waiting in kernel at et_wait

Hi All, bash-3.2# ps -eaf | grep mkdev root 4378752 1548526 0 03:27:04 - 0:00 /usr/sbin/mkdev -c disk -s vscsi -t xyz -l virtualdev bash-3.2# Here we are trying to create a virtual device using mkdev and the device is already populated in the namespace and the corresponding ODMs... (1 Reply)
Discussion started by: Muppana Prasad
1 Replies

4. Programming

Number of threads waiting on a pthread mutex/rwlock

Using pthreads is there a way to determine how many threads are waiting on a locked resource? I mean, once a shared resource is protected using e.g. pthread_rwlock_t or pthread_mutex_t one thread grabs the lock and other threads will go to sleep waiting for the resource to be available again. Is... (0 Replies)
Discussion started by: muggu
0 Replies

5. Solaris

Which file is read by kernel to set its default system kernel parameters values?

Hi gurus Could anybody tell me which file is read by kernel to set its default system kernal parameters values in solaris. Here I am not taking about /etc/system file which is used to load kernal modules or to change any default system kernal parameter value Is it /dev/kmem file or something... (1 Reply)
Discussion started by: girish.batra
1 Replies

6. UNIX for Advanced & Expert Users

Passing socket struct between kernel threads - module programming

I write kernel module with kernel threads using linux/kthread.h on 2.6.* kernel I tried to passing data between two kernel threads with data argument of kthread_run( fun, data , NAME ); but this is not work I dont know why. I tried many possibility and nothing works. So I thought that... (0 Replies)
Discussion started by: marcintom
0 Replies

7. Linux

vmstat help

Hi everyone, I need to see some VM manager performance/behavior information on some Linux boxes regarding pages scanned/activation of the paging algorithm in order to get an idea if a given server needs more memory and is actually paging. In Aix servers, by using the vmstat cmd you... (1 Reply)
Discussion started by: jcpetela
1 Replies

8. UNIX for Advanced & Expert Users

Threads and Threads Count ?

Hi all, How can I get the list of all Threads and the Total count of threads under a particular process ? Do suggest !! Awaiting for the replies !! Thanks Varun:b: (2 Replies)
Discussion started by: varungupta
2 Replies

9. SuSE

max number of slabs per kernel module (kernel 2.6.17, suse)

Hi All, Is there a max number of slabs that can be used per kernel module? I'm having a tough time finding out that kind of information, but the array 'node_zonelists' (mmzone.h) has a size of 5. I just want to avoid buffer overruns and other bad stuff. Cheers, Brendan (4 Replies)
Discussion started by: Brendan Kennedy
4 Replies
Login or Register to Ask a Question