Major Description
CPU:40Cores, Mem:132GB, Hard Disk:300GB sys + 11* 2TB data(naked), Network Card: 4*GbE bond, ATS: 5.3.1 or 6.1.1
Recently I found possible memory leak in production host with ATS version from 5.3.1 to 6.1.1, the memory tracker log is as follows:
--------------------------------------------------------------------------------------------
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
67108864 | 33554432 | 2097152 | memory/ioBufAllocator[14]
41271951360 | 40564162560 | 1048576 | memory/ioBufAllocator[13]
570425344 | 566755328 | 524288 | memory/ioBufAllocator[12]
276824064 | 270794752 | 262144 | memory/ioBufAllocator[11]
155189248 | 150994944 | 131072 | memory/ioBufAllocator[10]
113246208 | 111607808 | 65536 | memory/ioBufAllocator[9]
1319108608 | 1278607360 | 32768 | memory/ioBufAllocator[8]
39845888 | 39501824 | 16384 | memory/ioBufAllocator[7]
78381056 | 55869440 | 8192 | memory/ioBufAllocator[6]
228589568 | 212971520 | 4096 | memory/ioBufAllocator[5]
262144 | 0 | 2048 | memory/ioBufAllocator[4]
131072 | 0 | 1024 | memory/ioBufAllocator[3]
65536 | 0 | 512 | memory/ioBufAllocator[2]
163840 | 256 | 256 | memory/ioBufAllocator[1]
1785856 | 768 | 128 | memory/ioBufAllocator[0]
0 | 0 | 512 | memory/FetchSMAllocator
0 | 0 | 592 | memory/ICPRequestCont_allocator
0 | 0 | 128 | memory/ICPPeerReadContAllocator
0 | 0 | 432 | memory/PeerReadDataAllocator
0 | 0 | 32 | memory/MIMEFieldSDKHandle
0 | 0 | 240 | memory/INKVConnAllocator
0 | 0 | 96 | memory/INKContAllocator
503808 | 492864 | 32 | memory/apiHookAllocator
0 | 0 | 128 | memory/socksProxyAllocator
17211392 | 17169856 | 704 | memory/httpClientSessionAllocator
180633600 | 170085888 | 8064 | memory/httpSMAllocator
2236416 | 2229024 | 224 | memory/httpServerSessionAllocator
0 | 0 | 48 | memory/CacheLookupHttpConfigAllocator
0 | 0 | 848 | memory/http2ClientSessionAllocator
0 | 0 | 128 | memory/RemapPluginsAlloc
0 | 0 | 48 | memory/CongestRequestParamAllocator
0 | 0 | 160 | memory/CongestionDBContAllocator
4325376 | 4245248 | 256 | memory/httpCacheAltAllocator
155451392 | 146763776 | 2048 | memory/hdrStrHeap
269221888 | 253577216 | 2048 | memory/hdrHeap
0 | 0 | 128 | memory/OneWayTunnelAllocator
0 | 0 | 96 | memory/hostDBFileContAllocator
296960 | 4640 | 2320 | memory/hostDBContAllocator
135424 | 33856 | 33856 | memory/dnsBufAllocator
163840 | 1280 | 1280 | memory/dnsEntryAllocator
0 | 0 | 16 | memory/DNSRequestDataAllocator
0 | 0 | 112 | memory/inControlAllocator
0 | 0 | 128 | memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 | memory/clusterVCAllocator
0 | 0 | 48 | memory/ClusterVConnectionCache::Entry
0 | 0 | 576 | memory/cacheContAllocator
0 | 0 | 48 | memory/evacuationKey
0 | 0 | 64 | memory/cacheRemoveCont
380928 | 380928 | 96 | memory/evacuationBlock
20003360 | 19959936 | 944 | memory/cacheVConnection
163840 | 156960 | 160 | memory/openDirEntry
2260992 | 2253408 | 96 | memory/RamCacheCLFUSEntry
0 | 0 | 64 | memory/RamCacheLRUEntry
0 | 0 | 880 | memory/sslNetVCAllocator
0 | 0 | 128 | memory/UDPIOEventAllocator
24399872 | 23407008 | 736 | memory/netVCAllocator
0 | 0 | 128 | memory/udpReadContAllocator
0 | 0 | 160 | memory/udpPacketAllocator
0 | 0 | 416 | memory/socksAllocator
7053312 | 7051072 | 64 | memory/ioBlockAllocator
7441920 | 7440432 | 48 | memory/ioDataAllocator
19355520 | 19350000 | 240 | memory/ioAllocator
3402720 | 3306880 | 80 | memory/mutexAllocator
147456 | 103680 | 96 | memory/eventAllocator
22937600 | 21599232 | 1024 | memory/ArenaBlock
6206100608 | 5329728512 | | TOTAL
Allocs | Frees | Size In-use | Avg Size | Location
-----------|------------|----------------------|------------|--------------------------------------------------------------------
39443091 | 39399441 | 178790400 | 4096 | memory/IOBuffer/ProtocolProbeSessionAccept.cc:66
61 | 61 | 0 | 0 | memory/IOBuffer/Transform.cc:791
4064 | 4064 | 0 | 0 | memory/IOBuffer/HttpClientSession.cc:211
10271910 | 10270634 | 23633920 | 18521 | memory/IOBuffer/HttpSM.cc:5831
261122 | 261122 | 0 | 0 | memory/IOBuffer/HttpSM.cc:5880
14785 | 14785 | 0 | 0 | memory/IOBuffer/HttpSM.cc:5401
524803 | 494019 | 1008730112 | 32768 | memory/IOBuffer/HttpSM.cc:6199
7365218 | 7365218 | 0 | 0 | memory/IOBuffer/HttpSM.cc:6021
11827313 | 11827056 | 1052672 | 4096 | memory/IOBuffer/HttpSM.cc:5750
51960076 | 51954854 | 170999808 | 32746 | memory/IOBuffer/HttpSM.cc:6302
1156 | 1156 | 0 | 0 | memory/IOBuffer/HttpSM.cc:6269
8806600 | 8803759 | 23273472 | 8192 | memory/IOBuffer/HttpServerSession.cc:86
708080 | 708074 | 768 | 128 | memory/IOBuffer/HttpTunnel.cc:106
34649 | 34648 | 256 | 256 | memory/IOBuffer/HttpTunnel.cc:111
8493 | 8101 | 108728320 | 277368 | memory/IOBuffer/Cache.cc:2476
31232454 | 31172959 | 41769619456 | 702069 | memory/IOBuffer/Cache.cc:2490
43284829184 | | TOTAL
---------------------------------------------------------------------------------------------
the possilbe memory leak position is in Cache.cc:2490, including new_CacheVC and free_CacheVC, and memory leak has 41769619456 bytes. I wonder some places forget to call free_CacheVC, or cont->buf.clear(), or anything like this, but not sure.