記一次 .NET 某風(fēng)控管理系統(tǒng)內(nèi)存泄漏分析
一、背景
1. 講故事
上個(gè)月中旬,星球里的一位朋友在微信找我,說他的程序跑著跑著內(nèi)存會(huì)不斷的緩慢增長并無法釋放,尋求如何解決 ?
得,看樣子星球還得好好弄!!!不管怎么說,先上 windbg 說話。
二、Windbg 分析
1. 經(jīng)驗(yàn)推理
從朋友的截圖看,有大量的 8216 字節(jié)的 byte[],這表示什么呢?追隨本系列的朋友應(yīng)該知道,有一篇 某三甲醫(yī)院 的內(nèi)存暴漲的dump中,也同樣有此 size= (8216-24=8192) 的 byte[] 數(shù)組, 他的問題是 Oracle 中讀取某大字段時(shí)sdk里的 OraBuf 出了問題,換句話說,這肯定又是底層或者第三方庫中的池對象搞出來的東西,接下來從 托管堆 看起。
2. 查看托管堆
- 0:000> !dumpheap -stat
 - Statistics:
 - 00007ffe107248f0 483707 15478624 System.Threading.PreAllocatedOverlapped
 - 00007ffe1079c160 483744 15479808 System.Threading.ThreadPoolBoundHandle
 - 00007ffe1079cff8 483701 23217648 System.Threading._IOCompletionCallback
 - 00007ffe106e7a90 483704 23217792 Microsoft.Win32.SafeHandles.SafeFileHandle
 - 00007ffe1079b088 483703 30956992 System.IO.FileSystemWatcher+AsyncReadState
 - 00007ffe1079ceb0 483707 34826904 System.Threading.OverlappedData
 - 00007ffe1079ccb0 483707 34826904 System.Threading.ThreadPoolBoundHandleOverlapped
 - 0000016c64651080 245652 1473128080 Free
 - 00007ffe105abf30 488172 3977571092 System.Byte[]
 
掃完托管堆,臥槽 ,byte[] 沒吸引到我,反而被 System.IO.FileSystemWatcher+AsyncReadState 吸引到了,畢竟被 System.IO.FileSystemWatcher 折騰多次了,它已經(jīng)深深打入了我的腦海。。。畢竟讓程序卡死,讓句柄爆高的都是它。。。這一回八成又是它惹的禍,看樣子還是有很多程序員栽在這里哈。
為做到嚴(yán)謹(jǐn),我還是從最大的 System.Byte[] 入手,按size對它進(jìn)行分組再按totalsize降序,丑陋的腳本我就不發(fā)了,直接上腳本的輸出結(jié)果。
- !dumpheap -mt 00007ffe105abf30
 - size=8216,count=483703,totalsize=3790M
 - size=8232,count=302,totalsize=2M
 - size=65560,count=6,totalsize=0M
 - size=131096,count=2,totalsize=0M
 - size=4120,count=11,totalsize=0M
 - size=56,count=301,totalsize=0M
 - size=88,count=186,totalsize=0M
 - size=848,count=16,totalsize=0M
 - size=152,count=85,totalsize=0M
 - size=46,count=242,totalsize=0M
 - size=279,count=38,totalsize=0M
 - !dumpheap -mt 00007ffe105abf30 -min 0n8216 -max 0n8216 -short
 - 0000016c664277f0
 - 0000016c66432a48
 - 0000016c6648ef88
 - 0000016c6649daa8
 - 0000016c6649fb00
 - 0000016c664a8b90
 - ...
 
從輸出結(jié)果看,size=8216 的 byte[] 有 48w 個(gè),然后腳本也列出了一些 8216 大小的 address 地址,接下來用 !gcroot 看下這些地址的引用。
- 0:000> !gcroot 0000016c664277f0
 - HandleTable:
 - 0000016C65FC28C0 (async pinned handle)
 - -> 0000016C6628DEB0 System.Threading.OverlappedData
 - -> 0000016C664277F0 System.Byte[]
 - Found 1 unique roots (run '!gcroot -all' to see all roots).
 - 0:000> !gcroot 0000016c667c80d0
 - HandleTable:
 - 0000016C65FB7920 (async pinned handle)
 - -> 0000016C663260F8 System.Threading.OverlappedData
 - -> 0000016C667C80D0 System.Byte[]
 
從輸出中可以看到這些 byte[] 都是 async pinned,也就是當(dāng)異步IO回來的時(shí)候需要給 byte[] 填充的存儲(chǔ)空間,接下來我們看看如何通過 OverlappedData 找到源碼中定義為 8192 大小的 byte[] 地方。
如果你了解 FileSystemWatcher ,反向查找鏈大概是這樣的 OverlappedData -> ThreadPoolBoundHandleOverlapped -> System.IO.FileSystemWatcher+AsyncReadState -> Buffer[], 這中間涉及到 ThreadPool 和 SafeHandle 的綁定。
- 0:000> !do 0000016C663260F8
 - Name: System.Threading.OverlappedData
 - MethodTable: 00007ffe1079ceb0
 - EEClass: 00007ffe107ac8d0
 - Size: 72(0x48) bytes
 - File: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Private.CoreLib.dll
 - Fields:
 - MT Field Offset Type VT Attr Value Name
 - 00007ffe106e3c08 40009ce 8 System.IAsyncResult 0 instance 0000000000000000 _asyncResult
 - 00007ffe104a0c68 40009cf 10 System.Object 0 instance 0000016c66326140 _callback
 - 00007ffe1079cb60 40009d0 18 ...eading.Overlapped 0 instance 0000016c663260b0 _overlapped
 - 00007ffe104a0c68 40009d1 20 System.Object 0 instance 0000016c667c80d0 _userObject
 - 00007ffe104af508 40009d2 28 PTR 0 instance 00000171728f66e0 _pNativeOverlapped
 - 00007ffe104aee60 40009d3 30 System.IntPtr 1 instance 0000000000000000 _eventHandle
 - 00007ffe104ab258 40009d4 38 System.Int32 1 instance 0 _offsetLow
 - 00007ffe104ab258 40009d5 3c System.Int32 1 instance 0 _offsetHigh
 - 0:000> !do 0000016c663260b0
 - Name: System.Threading.ThreadPoolBoundHandleOverlapped
 - MethodTable: 00007ffe1079ccb0
 - EEClass: 00007ffe107ac858
 - Size: 72(0x48) bytes
 - File: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Private.CoreLib.dll
 - Fields:
 - MT Field Offset Type VT Attr Value Name
 - 00007ffe1079ceb0 40009d6 8 ...ng.OverlappedData 0 instance 0000016c663260f8 _overlappedData
 - 00007ffe1079b818 40009c0 10 ...ompletionCallback 0 instance 0000016f661ab8a0 _userCallback
 - 00007ffe104a0c68 40009c1 18 System.Object 0 instance 0000016c667ca0e8 _userState
 - 00007ffe107248f0 40009c2 20 ...locatedOverlapped 0 instance 0000016c66326090 _preAllocated
 - 00007ffe104af508 40009c3 30 PTR 0 instance 00000171728f66e0 _nativeOverlapped
 - 00007ffe1079c160 40009c4 28 ...adPoolBoundHandle 0 instance 0000000000000000 _boundHandle
 - 00007ffe104a7238 40009c5 38 System.Boolean 1 instance 0 _completed
 - 00007ffe1079b818 40009bf 738 ...ompletionCallback 0 static 0000016f661ab990 s_completionCallback
 - 0:000> !do 0000016c667ca0e8
 - Name: System.IO.FileSystemWatcher+AsyncReadState
 - MethodTable: 00007ffe1079b088
 - EEClass: 00007ffe107a9dc0
 - Size: 64(0x40) bytes
 - File: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.IO.FileSystem.Watcher.dll
 - Fields:
 - MT Field Offset Type VT Attr Value Name
 - 00007ffe104ab258 400002b 30 System.Int32 1 instance 1 <Session>k__BackingField
 - 00007ffe105abf30 400002c 8 System.Byte[] 0 instance 0000016c667c80d0 <Buffer>k__BackingField
 - 00007ffe106e7a90 400002d 10 ...es.SafeFileHandle 0 instance 0000016c66326028 <DirectoryHandle>k__BackingField
 - 00007ffe1079c160 400002e 18 ...adPoolBoundHandle 0 instance 0000016c66326058 <ThreadPoolBinding>k__BackingField
 - 00007ffe107248f0 400002f 20 ...locatedOverlapped 0 instance 0000016c66326090 <PreAllocatedOverlapped>k__BackingField
 - 00007ffe1079b8c8 4000030 28 ...eSystem.Watcher]] 0 instance 0000016c66326078 <WeakWatcher>k__BackingField
 
上面的 
有了這些原理之后,接下來就可以問朋友是否有對 appsettings 設(shè)置了 reloadonchange=true 的情況,朋友找了下代碼,寫法大概如下:
- public object GetxxxFlag()
 - {
 - string value = AppConfig.GetConfig("appsettings.json").GetValue("xxxx", "0");
 - return new
 - {
 - state = 200,
 - data = value
 - };
 - }
 - public class AppConfig
 - {
 - public static AppConfig GetConfig(string settingfile = "appsettings.json")
 - {
 - return new AppConfig(settingfile);
 - }
 - }
 - public class AppConfig
 - {
 - private AppConfig(string settingfile)
 - {
 - _config = new ConfigurationBuilder().AddJsonFile(settingfile, optional: true, reloadOnChange: true).Build();
 - _settingfile = settingfile;
 - }
 - }
 
從源碼邏輯看,我猜測朋友將 GetConfig 方法標(biāo)記成 static 后就以為是單例化了,再次調(diào)用不會(huì)重復(fù) new AppConfig(settingfile),所以問題就出在這里。
不過有意思的是,前面二篇的 FileSystemWatcher 都會(huì)造成程序卡死,那這一篇為啥沒有呢?恰好他沒有在程序根目錄中放日志文件,不然的話。。。,可萬萬沒想到逃過了卡死卻沒逃過一個(gè) watcher 默認(rèn) 8byte 空間的靈魂拷問。。。
三、總結(jié)
總的來說,設(shè)置 reloadOnChange: true 一定要慎重, 可能它會(huì)造成你的程序卡死,句柄泄漏,內(nèi)存泄漏 等等!!!

















 
 
 


















 
 
 
 