.NET Memory Analysis with Linux

October 3, 2023

For Windows, there are various programs and tools for the analysis of memory/performance problems of .NET programs. These include the “official” programs from Microsoft such as Perfview, Visual Studio or WinDBG.

Under Linux, the choice is unfortunately very limited. Fortunately, Microsoft provides a whole set of tools for analyzing memory/performance problems of .NET programs with the dotnet diagnostic tools. These work under Linux as well as in Windows and with restrictions also under macOS.

dotnet-counters

The first step is to find out if there really is a problem with the managed memory (GC Heap).

This is easily done with dotnet counters. With the help of dotnet counters monitor -p <PROCESS ID> monitoring can be activated for any .NET process.

dotnet counters monitor -p 1234

Press p to pause, r to resume, q to quit.
    Status: Running

[System.Runtime]
    % Time in GC since last GC (%)                                         0
    Allocation Rate (B / 1 sec)                                        8,168
    CPU Usage (%)                                                          0.007
    Exception Count (Count / 1 sec)                                        0
    GC Committed Bytes (MB)                                              321.516
    GC Fragmentation (%)                                                   0.046
    GC Heap Size (MB)                                                  1,074.347

Watch for GC Heap Size (MB). If the value is always increasing but never decreasing, then the application has a memory leak.

dotnet-gcdump

A simple overview of all objects managed by the garbage collector (GC) can be displayed with dotnet gcdump. To do this, first create a memory dump of the desired .NET process with dotnet gcdump collect -p <PROCESS ID>.

dotnet gcdump collect -p 1234

The managed objects can now be viewed with dotnet gcdump report <DUMP FILE>.

dotnet gcdump report 20230906_205437_23141.gcdump

  1,283,668,344  GC Heap bytes
          4,315  GC Heap objects

   Object Bytes     Count  Type
      1,048,600     1,224  System.Byte[] (Bytes > 1M)  [System.Private.CoreLib.dll]
         28,588         1  System.String (Bytes > 10K)  [System.Private.CoreLib.dll]
         16,408         1  MemoryLeak.Lib.LeakData[] (Bytes > 10K)  [MemoryLeak.Lib.dll]
         16,344         1  System.Object[] (Bytes > 10K)  [System.Private.CoreLib.dll]
          8,184         4  System.Object[] (Bytes > 1K)  [System.Private.CoreLib.dll]

The report now shows a list of all objects including their size. The output is about the same as dumpheap -stat from the SOS debugging extension.

At the moment, more information cannot be displayed with dotnet gcdump report. For memory dumps created with dotnet gcdump, the only convenient way at the moment is under Windows with the help of Perfview or Visual Studio.

dotnet-dump

dotnet-dump is the most comprehensive tool for memory analysis of .NET applications. The tool can create memory dumps of .NET processes and provides a virtual environment for analyzing these dumps.

A memory dump of any .NET process is done with dotnet dump collect -p <PROCESS ID>

dotnet dump collect -p 1234

For the analysis of the dump a virtual environment is available afterwards which is started with dotnet dump analyse <DUMP FILE> .

dotnet dump analyze core_20230830_213024

The available SOS commands are documented here and can be displayed with the help command.

An overview of all objects can be displayed with dumpheap -stat. The output is analogous to the dotnet gcdump report <DUMP FILE> used above and shows the number of all managed objects as well as their size in memory.

dumpheap -stat

Statistics:
          MT Count     TotalSize Class Name
7f230ed2e098    18           984 System.String[]
7f230ed56608     7         3,292 System.Char[]
7f230ec4b9d8    14         4,760 System.Int32[]
7f230ebda508     7        36,016 System.Object[]
7f230f1aba78     3        49,224 MemoryLeak.Lib.LeakData[]
7f230ed01038   408        49,754 System.String
7f230f1aac60 2,200        52,800 MemoryLeak.Lib.LeakData
7f230f1ab2a0 2,200        52,800 MemoryLeak.Lib.Data
56490bf28700 2,280        93,408 Free

To get more information about the object of type MemoryLeak.Lib.Data, you can use dumpheap -mt 7f230f1ab2a0 to output each instance. Here in the example there are 2’200 instances of the type MemoryLeak.Lib.Data

dumpheap -mt 7f230f1ab2a0

7ee302816f70     7f230f1ab2a0             24
7ee302816ff0     7f230f1ab2a0             24
7ee302817020     7f230f1ab2a0             24

Now that we have an overview of all objects of the MemoryLeak.Lib.Data type, we can perform further analysis. For example, we can display the “content” of the object. For this we take any object from the list and execute do (dumpobject).

do 7ee302817020

Name:        MemoryLeak.Lib.Data
MethodTable: 00007f230f1ab2a0
EEClass:     00007f230f17be00
Tracked Type: false
Size:        24(0x18) bytes
File:        /home/stef/projects/performance/MemoryLeak/bin/Debug/net7.0/MemoryLeak.Lib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f230f01fcd0  400000a        8        System.Byte[]  0 instance 00007ee302e000b0 <Buffer>k__BackingField

The object contains only one byte array and is defined as follows:

internal class Data
{
    public byte[]? Buffer { get; set; }
}

Further we can display the used memory of this object with objsize:

objsize 7ee302817020

Objects which 7ee302817020(MemoryLeak.Lib.Data) transitively keep alive:

         Address               MT           Size
    7ee302817020     7f230f1ab2a0             24
    7ee302e000b0     7f230f01fcd0      1,048,600

Statistics:
          MT Count TotalSize Class Name
7f230f1ab2a0     1        24 MemoryLeak.Lib.Data
7f230f01fcd0     1 1,048,600 System.Byte[]
Total 2 objects, 1,048,624 bytes

It happens from time to time that the garbage collector does not free the memory anymore, because an object is still used somehow. Here gcroot helps. With gcroot you can see which objects reference a certain object.

gcroot 7ee302817020

HandleTable:
00007f238af413e8 (strong handle)
-> 7ee300000020 System.Object[]
-> 7ee302816f20 System.Collections.Generic.List<MemoryLeak.Lib.LeakData> (static variable: System.Random._random)
-> 7ee30201e750 MemoryLeak.Lib.LeakData[]
-> 7ee302817008 MemoryLeak.Lib.LeakData
-> 7ee302817020 MemoryLeak.Lib.Data

LLDB

For the analysis of a memory dump, LLDB can also be used instead of dotnet-dump. The advantage of LLDB is that the analysis of managed and native code is possible.

LLDB is probably available in all major linux distributions in the package sources and can be installed with the help of the package manager. Here’s an example of dnf (Fedora):

sudo dnf install lldb

The memory dump is also generated with dotnet dump collect -p <PROCESS ID>. LLDB uses the SOS debugging extension to debug managed code. It may be necessary to install dotnet-sos first so that the SOS debugging extension is found by LLDB.

dotnet tool install --global dotnet-sos
dotnet sos install

The memory dump can then be loaded into the interactive shell of LLDB with lldb –core DUMPFILE

lldb --core core_20231003_100537

Current symbol store settings:
-> Cache: /home/stef/.dotnet/symbolcache
-> Server: https://msdl.microsoft.com/download/symbols/ Timeout: 4 RetryCount: 0
(lldb) target create --core "core_20231003_100537"
Core file '/home/stef/projects/performance/dumps/core_20231003_100537' (x86_64) was loaded.
(lldb)

The memory analysis now works exactly as with dotnet dump. All SOS commands of the SOS debugging extension are available, like dumpheap -stat and so on.

More detailed information for using LLDB can be found in the official documentation from Microsoft.

dotMemory with JetBrains Rider

Of course, there are also commercial products like JetBrains Rider that are available for both Windows and Linux thanks to their platform independence. Since version 2023.2 of Rider you can collect memory dumps and analyze them directly in Rider, just like in the standalone version of dotMemory.

Stefan Geiger