Detect Most CPU Utilizing Function in Linux

### top output

Tasks: 393 total, 3 running, 388 sleeping, 1 stopped, 1 zombie

Cpu(s): 87.7%us, 12.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st

Mem: 74251208k total, 73797492k used, 453716k free, 182852k buffers

Swap: 16678908k total, 9468772k used, 7210136k free, 2898440k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

7190 was 20 0 1986m 729m 25m S 73.8 1.0 120197:36 java

7542 was 20 0 2402m 1.0g 21m S 71.9 1.5 45086:14 java

### gathering data

[server1]~#pstack 7190 > /tmp/pstack_7190

[server1]~#pstack 7542 > /tmp/pstack_7542

### top output with threads

Tasks: 17837 total, 67 running, 17768 sleeping, 1 stopped, 1 zombie

Cpu(s): 70.8%us, 14.1%sy, 0.0%ni, 14.7%id, 0.1%wa, 0.0%hi, 0.3%si, 0.0%st

Mem: 74251208k total, 73823320k used, 427888k free, 183392k buffers

Swap: 16678908k total, 9468292k used, 7210616k free, 2909320k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

15769 was 20 0 1986m 730m 26m R 36.5 1.0 116006:07 WebContainer :

10219 was 20 0 2402m 1.0g 21m R 25.2 1.5 8849:17 WebContainer :

15452 was 20 0 2402m 1.0g 21m R 25.2 1.5 29222:10 Thread-15618

10214 was 20 0 2402m 1.0g 21m S 24.7 1.5 4569:09 WebContainer :

12263 root 20 0 30772 14m 1016 R 20.9 0.0 0:00.57 top

### examine 15769 inside /tmp/pstack_7190

Thread 46 (Thread 0x7f3e3e47e700 (LWP 15769)):

#0 0x00007f3e3e318a12 in ?? ()

#1 0x00000000033731f0 in ?? ()

#2 0x0000000000000003 in ?? ()

#3 0x000000000366a298 in ?? ()

#4 0x000000000366a790 in ?? ()

#5 0x00007f3e89f07c90 in ?? () from /usr/WebSphere/AppServer/java/jre/lib/amd64/compressedrefs/libj9gc26.so

#6 0x000000000318cf88 in ?? ()

#7 0x000000000318cfb8 in ?? ()

#8 0x0000000013601200 in ?? ()

#9 0x00007f3e89f07c90 in ?? () from /usr/WebSphere/AppServer/java/jre/lib/amd64/compressedrefs/libj9gc26.so

#10 0x000000000d3e11b0 in ?? ()

#11 0x00007f3e50f227b4 in ?? ()

#12 0x00007f3e95e31e00 in ?? () from /usr/WebSphere/AppServer/java/jre/lib/amd64/compressedrefs/libj9vm26.so

#13 0x000000000553f898 in ?? ()

#14 0x0000000013601200 in ?? ()

#15 0x0000000000000000 in ?? ()

libj9gc26.so show that application is making garbage collection. (gc is the abbreviation of garbage collection)

Garbage collection is a normal application process but in this sample, we found who was utilizing cpu the most. We add below parameters to our java processes, so we can be able to analyze the details of garbage collection to understand what causes this cpu utilization.

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps/ -Xloggc:/tmp/gc.log -XX:+PrintGCDetails

Feel free to communicate by bulent.yucesoy@gmail.com