server | v4 switch to v3  

Photon Server Stack Overflow

Stack Overflow is a background exception that may cause silent craches. It is tricky to debug it. This document explains how to deal with stack overflow exceptions occurring due to custom code running on Photon Server.

Production Environment

This is the procedure we recommend to follow in order to gather dumps and a stack trace from it in case of a stack overflow.

Prerequisites

  1. Download and install Debug Diagnostic Tool. Download Latest version (v2.0 Update 2) or old v1.2.

  2. Download and install WinDBG. Windows Debugger is part of Windows Development Kit. Please download the appropriate dev. SDK for your Windows version and choose "Debugging Tools For Windows" as a feature to be installed during installation wizard.

Installing WinDBG for Windows 10
Installing WinDBG for Windows 10

Setup Debug Diagnostic Tool: Dump Collection

We are going to follow a step-by-step guide:

  1. Open Debug Diagnostic Tool ("DebugDiag 2 Collection"). By default a "Select Rule Type" wizard window should be open. Otherwise, please click "Add Rule". Select "Crash" as "Rule Type" and hit "Next".

    Select Rule Type
    Select Rule Type

  2. In "Select Target Type" window, select "A specific process" and hit "Next".

    Select Target Type
    Select Target Type

  3. In the "Select Target" window, choose "PhotonSocketServer.exe" from the list and hit "Next".

    Select Target Process
    Select Target Process

  4. In the "Advanced Configuration" window, "Advanced Settings" section click on "Exceptions".

    Advanced Configuration
    Advanced Configuration

  5. In the "First Chance Exception Configuration" window, click "Add Exception...".

    First Chance Exception Configuration
    First Chance Exception Configuration

  6. In the "Configure Exception" window, choose "Stack Overflow" from the list of exceptions. Then on the right, from the dropdown menu choose "Full Userdump" as "Action Type" and set "Action Limit" to 0. Once done, click "OK".

    Configure Exception
    Configure Exception

  7. A warning dialog will be shown to you about the Action Limit value 0. Continue by clicking "Yes".

    Action Limit 0 Warning
    Action Limit 0 Warning

  8. Now you are able to see the configure Stack Overflow exception added in the "First Chance Exception Configuration" window. Hit "Save & Close".

    First Chance Exception Configuration
    First Chance Exception Configuration

  9. Back to the "Advanced Configuration" window, just hit "Next".

    Advanced Configuration
    Advanced Configuration

  10. In the "Select Dump Location And Rule Name" window, you can change the default rule name. Set the "Userdump Location" to a temporary folder path of your choice and click "Next".

    Select Dump Location And Rule Name
    Select Dump Location And Rule Name

  11. The "Add Rule" wizard reaches its final window "Rule Completed". Choose "Activate the rule now" and hit "Finish".

    Rule Completed
    Rule Completed

  12. The Debug Diagnostic should prompt you to automatically set the "Symbol Search Path" to srv*c:\symcache*http://msdl.microsoft.com/download/symbols.

    Symbol Search Path for Debugging
    Symbol Search Path for Debugging

    In case it doesn't, please manually set this in "Tools" -> "Options & Settings" to get a better stacktrace.

    Set Debugging Symbols
    Set Debugging Symbols

  13. Back to the main Debug Diagnostic Tool window, you can see the configured rule.

    Crash Rule After Configuration
    Crash Rule After Configuration

Setup Debug Diagnostic Tool: Dump Analysis

  1. Now let's open Debug Diagnostic Tool ("Debug Diag 2 Analysis").

    Debug Diag 2 Analysis
    Debug Diag 2 Analysis

  2. Let's add the dump file using "Add Data Files".

    Adding collected dump file
    Adding collected dump file

  3. Once we have added the dump file, we can "Start Analysis".

    Starting analysis of dump file
    Starting analysis of dump file

  4. You should see the analysis loading which could take some time, then a report will be open in the browser.

    Loading Debug Diagnostic Analysis
    Loading Debug Diagnostic Analysis

Open the Dump using Visual Studio

Open the dump file in Visual Studio and run "Debug Mixed Mode".

Open dump file in Visual Studio
Open dump file in Visual Studio

The debugging will break when the stack overflow exception occurs. You will get the exception breakpoint and a full stack trace.

Breakpoint and stack trace from dump
Breakpoint and stack trace from dump

Open the Dump using WinDBG

Select Rule Type
Select Rule Type

Open the dump using WinDBG and set the following commands:

  • .sympath srv*C:\symcache*http://msdl.microsoft.com/download/symbols
  • .reload
  • .loadby sos clr. (Note: .loadby sos clr is if you are debugging .NET 4.0 or above, otherwise the command is .loadby sos mscorwks.)
  • !clrstack

Development Environment

You can notice that once the rule is added you can't attach Visual Studio debugger to "PhotonSocketServer" process. This is because the Debug Diagnostic Tool auto attaches as debugger to the process configured in the rule. If you happen to start Photon Server from Visual Studio when Debug Diagnostic Tool is running with an active rule for Photon Server, the latter will crash on startup without leaving any dump.

Unable to attach process error
Unable to attach process error

We recommend to remove the following registry keys to make sure the debugging tools don't get in the way of each other:

  • HKEYLOCALMACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug
  • HKEYLOCALMACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps

 To Document Top