I noticed that my servers are running out of memory after one whole day’s service.
At first, I removed all functions that eat up memory. But still the next morning when the traffic is down, the memory is still high. So it seems I have memory leak in my server.
After making some searches and reading some blog articles, I learned that I can try to find view all memory by using jmap and jhat.
First, I made the dump:
jmap -dump:file=heap.bin ####
It was about 800M file, as my server is running at 1.5G maximum memory on 2G-memory server. I tried to use jhat to give a memory view:
jhat -J-mx1536m heap.bin
It run but failed with OutOfMemoryError, as my service was still running and it had no enough memory. Fortunately, I had another server without service running, so I uploaded the dump file to another server, and tried to start jhat there.? It run but with lots of warning message like the following:
WARNING:? Failed to resolve object id 0xac2f1690 for field key (signature L)
WARNING:? Failed to resolve object id 0xacd1c208 for field value (signature L)
WARNING:? Failed to resolve object id 0xac08df00 for field value (signature L)
WARNING:? Failed to resolve object id 0xac2f3f28 for field key (signature L)
WARNING:? Failed to resolve object id 0xac08df00 for field value (signature L)
WARNING:? Failed to resolve object id 0xac2eea70 for field key (signature L)
WARNING:? Failed to resolve object id 0xacd1c208 for field value (signature L)
WARNING:? Failed to resolve object id 0xac08df00 for field value (signature L)
And after a long time, it still gave me an OutOfMemoryError!
So I made another dump from another server which was running on its early stage with less memory being used. So the dump file was a lot smaller. And I could run jhat on the same without uploading to another. And after a very long time of waiting I finally got message:
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.
Visiting http://###.###:7000/ , I noticed that jhat was slow. But it helped! I found my memory leak.
Great tools! I think I can reduce servers but still provide more stable services now.
