Some WinDbg tips

I’ve gathered some WinDbg tips over time (mostly for managed dump analysis) and this seems like as good a place as any to share them, so here you go.

Preparation (one time)

• Install the latest debugging tools from the Dev Center
  • Let’s assume you install them to c:debuggers
• Download sosex.dll and place it in c:debuggers
• Create an environment variable named _NT_SYMBOL_PATH with the value C:Symbols;srv*C:symbols*http://msdl.microsoft.com/download/symbols
  • Most Microsoft symbols should be found in the symbol server and will be cached in C:Symbols
  • You can copy any other symbols (PDBs) you have to the C:Symbols folder as well
• Create an environment variable named _NT_SOURCE_PATH with the value srv* 
  • This will enable source serving (when source indexing was included in the build) – simply double-click the relevant line in the Call Stack (calls) window and you should jump straight to the relevant line in the source code.
  • You might wonder how that’s possible when no server was specified. The answer is a bit surprising – strictly speaking, there is no such thing as a source server! The name is a bit misleading. What actually happens is that the source-indexed PDB contains the proper command(s) to retrieve the relevant source files. By default this “command” would be something like C:srcfoo.cs, and it would only work if the file in the correct version is actually there. However, if source-indexing was enabled in TFS build, it would look something like tf get MyClass.cs /version:C8 (i.e. the file will be retrieved directly from source, with the correct version).

Preparation (per debugging session)

• Open the dump file in WinDbg
  • be sure to match the architecture to the analyzed process – use WinDbg x86 for 32-bit processes and WinDbg x64 for 64-bit processes
• Enable Debugger Markup Language (DML) by issuing .prefer_dml 1
  • This will make the output of some commands contain convenient hyperlinks
• Load the SOSEX extension by issuing the command .load sosex.dll
  • Don’t load SOS.dll manually – the first command below (analyze -v) will load it with the correct version automatically

Debugging

• Automatic exception analysis: !analyze -v
  • In many cases this will suffice to find the root cause!
• Threads
  • !Threads – lists all the managed threads in the process
• Stack commands
  • !clrstack – provides a true stack trace for managed code only
  • !dumpstack – provides a verbose stack trace
  • !eestac – runs !dumpStack on all threads in the process
  • !dso – displays any managed objects within the bounds of the current stack
  • !sosex.mk – produces and displays a merged stack trace of managed and unmanaged frames. Note that in addition to the native offset, a managed (IL) offset is specified – this is extremely useful for debugging NullReferenceException’_s in that the exact offending IL instruction is indicated (actually it will be one instruction before the offending one in the specific case of _NullReferenceException)
  • !sosex.mdso – dumps object references on the stack and in CPU registers in the current context
  • !sosex.mdv – displays argument and local variable information for the current frame
  • !sosex.mframe – displays or sets the current managed frame for the !mdt and !mdv commands
• Heap commands
  • !eeheap – enumerates process memory consumed by internal CLR data structures
  • !DumpHeap – traverses the garbage collected heap
• Object commands
  • !do – allows you to examine the fields of an object, as well as learn important properties of the object
  • !dumparray – examines elements of an array object
  • !dumpvc – examines the fields of a value class
  • !sosex.mdt – displays the fields of the specified object or type
• Method commands
  • !dumpmt – examines a MethodTable
  • !DumpIL – prints the IL code associated with a managed method
  • !U – presents an annotated disassembly of a managed method
  • !sosex.muf – disassembles the method specified by the given MD or code address with interleaved source, IL, and assembly code
• Exception commands
  • !pe exceptionAddress – formats fields of any object derived from System.Exception
• GC commands
  • !GCRoot – looks for references (or roots) to an object
• SOS Help
  • !sos.help
  • !sos.help FAQ
• SOSEX Help
  • !sosexhelp or !sosex.help

Mismatched SOS.dll versions

!analyze -v  should get the correct SOS.dll version for you. Even if for some reason it doesn’t, SOS.dll warnings can be ignored most of the time, so long as the mscordacwks.dll version is correct (see the next section if that’s not the case). However there may be cases where the correct SOS.dll version is needed (or perhaps you just want to get rid of the warning). Here are some places to look for it (once you find it, copy it to your debugger folder and issue .load sos.dll):

• Your best bet is the machine on which the dump was taken. Provided that it’s accessible and wasn’t patched since the time the dump was taken, the correct version should be found at C:WindowsMicrosoft.NETFramework64v4.0.30319SOS.dll. Of course it should be there on your machine as well, and your local version may happen to match (or be close enough).
• You may find the version you’re looking for here (you should be able to extract the file from the update package itself using 7-zip).
• You may also want to try Psscor4.dll instead of SOS.dll (Psscor is a superset of SOS) – its version need not match the dump (aside of being .NET 4.0). Note that it is less maintained than SOS.
• For more information see https://stackoverflow.com/a/23244429/67824

Mismatched mscordacwks.dll versions

WinDBG should find the correct mscordacwks.dll automatically and download it from the symbol server. If that doesn’t happen, try and do it explicitly by running .cordll -ve -u -l (you’d might want to first run !sym noisy and/or !symfix in order to troubleshoot better – see the next section for details). Failing that, try and get the correct version from the following places, and run .cordll -u -ve -lp PathToFolderContainingMscorDAC once you have it.

• Again, your best bet is the machine on which the dump was taken (under the same caveats as above). It should be found at C:WindowsMicrosoft.NETFramework64v4.0.30319mscordacwks.dll. As before, it will be there on your machine so you could luck out.
• The following post lists many versions of CLR 4.0, you may be able to extract the correct version with the same method as above (use 7-zip to open the cab files inside the archive).
• For more information see https://stackoverflow.com/a/23244429/67824

Troubleshooting missing symbols and sources

• !sym noisy – increases symbol verbosity (always enable this when troubleshooting symbol issues)
• .srcnoisy 3 – increases source resolution vebosity (use it when double-clicking lines in the callstack window doesn’t bring up the correct sources)
• lm – displays the specified loaded modules
  • lme displays only modules that have a symbol problem (very useful)
• .reload /f – forces the reloading of all symbols
  • .reload /f SomeAssembly.dll – forces the reloading of a specified DLL

For further reading try to hit F1 inside WinDbg – the documentation is very good !