A few years ago I wrote a WinDbg Cheat Sheet for .NET Developers
and later a walkthrough of .NET Memory Analysis with Linux.
The memory post covers the “the heap keeps growing” case in detail: dumpheap, dumpobj,
objsize, gcroot, dotnet-counters and dotnet-gcdump.
This post pulls the threads of those two older posts together and is really a runbook for my own
future self: the companion for everything you find in a production dump that isn’t a plain leak —
a hung request, a deadlock, an unhandled exception that took the process down, or a
finalizer thread that stopped doing its job. It is a lookup table, ordered by symptom, and
every command output below was produced on .NET 10 with dotnet-dump on Fedora — not copied
from the docs.
Note: This post was put together with the help of an AI assistant
SOS is just WinDbg without the !
SOS is the
debugger extension that teaches a native debugger about the CLR. On Windows you load it into WinDbg
and call every command with a leading ! (!clrstack, !dumpheap). On Linux the same commands
ship inside dotnet-dump,
so you type them without the !.
| Task | WinDbg | dotnet dump analyze |
|---|---|---|
| Managed call stack (current thread) | !clrstack |
clrstack |
| All managed threads | !threads |
clrthreads |
| Switch thread | ~<n>s |
setthread <n> |
| Managed stacks of all threads | ~*e !clrstack |
clrstack -all |
| Grouped (“parallel”) stacks | IDE only | pstacks |
Monitor / lock ownership |
!syncblk |
syncblk |
| Current exception | !pe |
pe |
| Objects on the stack | !dso |
dso |
| Finalizable objects | !finalizequeue |
finalizequeue |
| Runtime version | !eeversion |
eeversion |
Setup
Install the tool once (globally):
dotnet tool install --global dotnet-dump
On a minimal container image without ICU,
dotnetrefuses to start. ExportDOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1(or installlibicu) before running any of the commands below. See Linux Manual installation and .NET globalization and ICU
Collecting a dump of a running process:
dotnet dump collect -p <PID> --type Full
For a process that crashes, let the runtime write the dump for you. Set these environment variables before starting the app and you get a dump at the exact moment of the unhandled exception, with no debugger attached:
export DOTNET_DbgEnableMiniDump=1
export DOTNET_DbgMiniDumpType=4 # 4 = full dump
export DOTNET_DbgMiniDumpName=/tmp/crash.dmp
Then open the dump:
dotnet dump analyze /tmp/crash.dmp
That drops you into an interactive > prompt. Every heading below is one command you run there.
For scripting, pass commands with -c:
dotnet dump analyze crash.dmp -c "clrthreads" -c "pe -nested" -c "exit"
The sample program
All output below comes from one small console app that reproduces a scenario per command-line
argument (lock, deadlock, crash, finalizer, threads) and prints its PID on startup. The
interesting bits are shown inline with each section.
Download: Program.cs. To run it:
mkdir SosDemo && cd SosDemo
dotnet new console
curl -O https://www.stefangeiger.ch/assets/code/SosDemo/Program.cs
dotnet run lock # or: deadlock | crash | finalizer | threads
Threads & stacks
clrthreads lists every managed thread. The DBG column is the id you pass to setthread,
OSID is the native thread id (hex), and the last column flags a thread that currently has a
managed exception in flight.
> clrthreads
ThreadCount: 6
UnstartedThread: 0
BackgroundThread: 5
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
DBG ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Lock Count Apt Exception
0 1 b63a 000055D119475610 2020020 Preemptive ... 000055D11944C600 -00001 Ukn
5 2 b640 000055D119477D00 21220 Preemptive ... 000055D11944C600 -00001 Ukn (Finalizer)
6 3 b641 000055D119480860 21220 Preemptive ... 000055D11944C600 -00001 Ukn
8 4 b643 000055D1194CF160 2021220 Preemptive ... 000055D11944C600 -00001 Ukn
9 5 b644 000055D1194DC6D0 2021220 Preemptive ... 000055D11944C600 -00001 Ukn
10 6 b645 000055D1194DDDB0 2021220 Preemptive ... 000055D11944C600 -00001 Ukn
clrstack -all dumps the managed stack of every thread at once. This is usually the first thing
I look at — it tells you what the whole process was doing:
> clrstack -all
...excerpt
OS Thread Id: 0xb643
Child SP IP Call Site
00007F8048DFD990 ... System.Threading.Monitor.Wait(System.Object, Int32)
00007F8048DFDA40 ... System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F8048DFDAB0 ... System.Threading.ManualResetEventSlim.Wait()
00007F8048DFDAC0 ... SosDemo.Program.AcceptConnections() [Program.cs @ 157]
00007F8048DFDAD0 ... SosDemo.Program.HttpListenerLoop() [Program.cs @ 156]
00007F8048DFDAE0 ... System.Threading.Thread.StartCallback()
To dig into one thread, switch to it and print its stack:
> setthread 8
> clrstack
Lock contention
Symptom: requests hang, CPU is idle, one resource is a bottleneck. syncblk lists every
SyncBlock
that backs a CLR monitor (lock / Monitor.Enter) which is currently held:
> syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
1 00007F8230001958 7 1 00005650149D9220 b6e9 8 00007f827140c3d0 System.Object
-----------------------------
Total 1
Read it as: the monitor on object 00007f827140c3d0 is owned by thread b6e9 (DBG id 8).
MonitorHeld is 2 * waiters + 1 when the lock is taken, so 7 means three threads are
queued behind the owner.
pstacks groups threads by identical stack, which makes the contention obvious at a glance —
one thread sleeps while holding the lock, three sit in Monitor.Enter:
> pstacks
...excerpt
~~~~ b6e9
1 System.Threading.Thread.Sleep(Int32)
1 SosDemo.Program+<>c.<LockContention>b__4_0()
~~~~ b6ed,b6ee,b6ef
3 System.Threading.Monitor.Enter_Slowpath(Object)
3 System.Threading.Monitor.Enter(Object, Boolean ByRef)
3 SosDemo.Program.ChargeCustomer(Int32)
==> 5 threads with 2 roots
Now switch to the owner and find out why it is holding the lock for so long:
> setthread 8
> clrstack
Child SP IP Call Site
00007F8243FFEA10 ... System.Threading.Thread.Sleep(Int32)
00007F8243FFEAB0 ... SosDemo.Program+<>c.<LockContention>b__4_0() [Program.cs @ 37]
00007F8243FFEAE0 ... System.Threading.Thread.StartCallback()
Deadlocks
A deadlock is just lock contention with a cycle. Here syncblk shows two held monitors:
> syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
1 00007FE550002578 3 1 000055C97DAC47C0 b70f 9 00007fe58940c3e8 System.Object
2 00007FE5500025D0 3 1 000055C97DAB6970 b70e 8 00007fe58940c3d0 System.Object
Thread 8 owns object ...c3d0, thread 9 owns object ...c3e8, and MonitorHeld == 3
means one waiter on each. To see which lock each thread is blocked on, print the stack with
arguments and locals (clrstack -a). The obj parameter of Monitor.Enter is optimised away,
but the C# locals still hold the object references:
> setthread 8
> clrstack -a
...excerpt
00007FE55BFFEA80 ... System.Threading.Monitor.Enter(System.Object, Boolean ByRef)
00007FE55BFFEAA0 ... SosDemo.Program+<>c.<Deadlock>b__6_0() [Program.cs @ 70]
LOCALS:
0x00007FE55BFFEAC0 = 0x00007fe58940c3d0 <- holds this one (SyncBlock 2)
0x00007FE55BFFEAB0 = 0x00007fe58940c3e8 <- waiting for this one (SyncBlock 1, owned by thread 9)
> setthread 9
> clrstack -a
...excerpt
00007FE55B7FDAA0 ... SosDemo.Program+<>c.<Deadlock>b__6_1() [Program.cs @ 79]
LOCALS:
0x00007FE55B7FDAC0 = 0x00007fe58940c3e8 <- holds this one (SyncBlock 1)
0x00007FE55B7FDAB0 = 0x00007fe58940c3d0 <- waiting for this one (SyncBlock 2, owned by thread 8)
Thread 8 waits for a lock thread 9 owns, thread 9 waits for a lock thread 8 owns — the cycle is the deadlock.
Exception analysis (crash dumps)
With DOTNET_DbgEnableMiniDump=1 the dump is taken on the unhandled exception. clrthreads
immediately points at the faulting thread — note the Exception column:
> clrthreads
...excerpt
DBG ID OSID ThreadOBJ State GC Mode ... Exception
8 4 b7d0 000055EF9571ADE0 21020 Cooperative ... System.InvalidOperationException 00007f2de940c9b8 (nested exceptions)
Switch to it and use pe -nested (print exception, including inner exceptions). This gives you
the full chain with managed stack traces — usually everything you need:
> setthread 8
> pe -nested
Exception object: 00007f2de940c9b8
Exception type: System.InvalidOperationException
Message: Report generation failed for account 42
InnerException: System.Collections.Generic.KeyNotFoundException, Use printexception 00007F2DE940C890 to see more.
StackTrace (generated):
SP IP Function
00007F2DC13F9E90 00007F3D7CE92BED SosDemo.dll!SosDemo.Program.RunReport()+0x8d
00007F2DC13FDAE0 00007F3D7C0A9B65 System.Private.CoreLib.dll!System.Threading.Thread.StartCallback()+0x85
HResult: 80131509
Nested exception -------------------------------------------------------------
Exception object: 00007f2de940c890
Exception type: System.Collections.Generic.KeyNotFoundException
Message: Account 42 is not in the cache
StackTrace (generated):
SP IP Function
00007F2DC13FDA50 00007F3D7CE92CE9 SosDemo.dll!SosDemo.Program.LoadAccount(Int32)+0xd9
00007F2DC13FDAB0 00007F3D7CE92B8E SosDemo.dll!SosDemo.Program.RunReport()+0x2e
HResult: 80131577
dso (dump stack objects) walks the raw stack memory and lists every object reference it finds —
handy when the exception has already been caught-and-rethrown and pe on the current frame comes
up empty:
> setthread 8
> dso
SP/REG Object Name
7f2dc13f6478 7f2de940c9b8 System.InvalidOperationException
7f2dc13f6808 7f2de940c890 System.Collections.Generic.KeyNotFoundException
...excerpt
Finalizer queue
Symptom: memory grows slowly, ~Finalize / Dispose(false) code seems to never run, or
SafeHandles pile up. finalizequeue shows the state of the finalizable objects:
> finalizequeue
...
Heap 0
generation 0 has 0 objects
generation 1 has 24 objects
generation 2 has 0 objects
Ready for finalization 20,001 objects (55c0215edbb0->55c021614cb8)
------------------------------
Statistics:
MT Count TotalSize Class Name
...
7f4010518e20 20,000 480,000 SosDemo.PendingHandle
Total 20,025 objects, 481,872 bytes
“Ready for finalization” is the backlog: objects that are unreachable, have a finalizer, and
are waiting for the finalizer thread. 20,000 stuck objects means the finalizer thread is not
keeping up — or not running at all. Check it in clrthreads (it is the one marked
(Finalizer)) and print its stack:
> setthread 5
> clrstack
Child SP IP Call Site
00007F400D11C750 ... System.Threading.Thread.Sleep(Int32)
00007F400D11C7F0 ... SosDemo.StuckFinalizer.Finalize() [Program.cs @ 166]
00007F400D11C810 ... System.GC.RunFinalizers()
There it is: one badly-behaved finalizer blocks the single finalizer thread, so every other
finalizer behind it — and everything those objects keep alive — leaks. gcroot on one of the
stuck objects confirms it is held only by the queue:
> gcroot 7f307cc02048
Finalizer Queue:
000055c0215edbb8 (finalizer root)
-> 7f307cc02048 SosDemo.PendingHandle
Found 1 unique roots.
Runtime info
Quick orientation commands when you open an unfamiliar dump:
> eeversion
10.0.926.27113
Workstation mode
SOS Version: 10.0.14.31102
> clrmodules # loaded managed assemblies
> dumpdomain # AppDomain(s) and their assemblies
> eeheap -gc # GC heap size per generation
For a deeper look at the GC heap itself (leak hunting, retention paths, objsize), see the
.NET Memory Analysis with Linux post. And when
you need to mix managed and native frames in the same session, that post also shows how to load
SOS into LLDB.
The full list of SOS commands is in the
official documentation,
or type help at the dotnet dump prompt.