Remember a UI crash analysis of a .NET lithium battery testing program for a new energy vehicle

More high-quality dry goods: see my GitHub: dotnetfly

One: background

1. Tell a story

Things in this world are strange. In the past two months, three friends have approached me and asked me to help analyze his program hangon phenomenon. These three dumps are related to: medical treatment, new energy, and POS system. Screenshot below:

Then why do you want to new energy in this article? Because this friend solved the problem most smoothly, after providing some clues, he found out the problem code smoothly.

To say a little off topic, I am not familiar with winform, but it has appeared in my field of vision over and over again, so I decided to write an article to summarize, because there are not many reference materials, the ability is limited, only I can try to interpret it myself.

Two: Windbg analysis

1. Procedural phenomena

wow64 about it. The dump files captured by these big guys are all 0613eb0b066d00, that is, 32-bit programs are captured with a 64-bit task manager. See the following output:

wow64cpu!CpupSyscallStub+0x9:
00000000`756d2e09 c3              ret

So it’s not easy to use windbg preview to analyze. First, you need to use !wow64exts.sw convert 64bit to 32bit. This article uses windbg10. Okay, since the UI is stuck, the first thing to do is to look at what is stuck in the UI thread. , You can use the command !clrstack to take a look.

0:000:x86> !clrstack 
OS Thread Id: 0x1d90 (0)
Child SP       IP Call Site
0019ee6c 0000002b [HelperMethodFrame_1OBJ: 0019ee6c] System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle, UInt32, Boolean, Boolean)
0019ef50 6c4fc7c1 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)
0019ef68 6c4fc788 System.Threading.WaitHandle.WaitOne(Int32, Boolean)
0019ef7c 6e094e7e System.Windows.Forms.Control.WaitForWaitHandle(System.Threading.WaitHandle)
0019efbc 6e463b96 System.Windows.Forms.Control.MarshaledInvoke(System.Windows.Forms.Control, System.Delegate, System.Object[], Boolean)
0019efc0 6e09722b [InlinedCallFrame: 0019efc0] 
0019f044 6e09722b System.Windows.Forms.Control.Invoke(System.Delegate, System.Object[])
0019f078 6e318556 System.Windows.Forms.WindowsFormsSynchronizationContext.Send(System.Threading.SendOrPostCallback, System.Object)
0019f090 6eef65a8 Microsoft.Win32.SystemEvents+SystemEventInvokeInfo.Invoke(Boolean, System.Object[])
0019f0c4 6eff850c Microsoft.Win32.SystemEvents.RaiseEvent(Boolean, System.Object, System.Object[])
0019f110 6eddb134 Microsoft.Win32.SystemEvents.OnUserPreferenceChanged(Int32, IntPtr, IntPtr)
0019f130 6f01f0b0 Microsoft.Win32.SystemEvents.WindowProc(IntPtr, Int32, IntPtr, IntPtr)
0019f134 001cd246 [InlinedCallFrame: 0019f134] 
0019f2e4 001cd246 [InlinedCallFrame: 0019f2e4] 
0019f2e0 6dbaefdc DomainBoundILStubClass.IL_STUB_PInvoke(MSG ByRef)
0019f2e4 6db5e039 [InlinedCallFrame: 0019f2e4] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
0019f318 6db5e039 System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr, Int32, Int32)
0019f31c 6db5dc49 [InlinedCallFrame: 0019f31c] 
0019f3a4 6db5dc49 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
0019f3f4 6db5dac0 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
0019f420 6db4a7b1 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
0019f434 003504a3 xxx.Program.Main()
0019f5a8 6f191366 [GCFrame: 0019f5a8] 

From the call stack, the code is Microsoft.Win32.SystemEvents.OnUserPreferenceChanged and then System.Windows.Forms.Control.WaitForWaitHandle at 0613eb0b066d9f. From the name of the former, you can see that OnUserPreferenceChanged (user preference) is a system-level Microsoft.Win32.SystemEvents event. What exactly is that? This caused the system event to be triggered. For this reason, I checked the information. I probably said: If the Control of the application registers these system-level events, then when windows sends out the WM_SYSCOLORCHANGE, WM_DISPLAYCHANGED, WM_THEMECHANGED (theme, preferences, interface display) message, these The handle of the Control registered for system-level events will be executed, such as refreshing itself.

If the text is more confusing, I try to draw a picture to clarify it.

In essence, it is an observer mode, but this has nothing to do with UI stuck. At best, it is the background knowledge that needs to be understood before solving the problem. There is another important concept that is not WindowsFormsSynchronizationContext , that is: 0613eb0b066df3.

2. Understand WindowsFormsSynchronizationContext

Why must we understand WindowsFormsSynchronizationContext? After understanding it, you will understand why it is stuck. We know that the UI thread of winform is a STA model. One of its characteristics is single thread. If other threads want to update the Control, they need to be dispatched to the Queue of the UI thread. , Does not exist and does not allow concurrent updates of Control, refer to the following:

0:000:x86> !t
ThreadCount:      207
UnstartedThread:  0
BackgroundThread: 206
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                         Lock  
       ID OSID ThreadOBJ    State GC Mode     GC Alloc Context  Domain   Count Apt Exception
   0    1 1d90 003e2430   2026020 Preemptive  00000000:00000000 003db8b8 0     STA 
   2    2 2804 003f0188     2b220 Preemptive  00000000:00000000 003db8b8 0     MTA (Finalizer) 

Winform also has a feature: it will assign a WindowsFormsSynchronizationContext synchronization context to the thread that creates the Control, which means that if other threads want to update the Control, the updated value must be dispatched to the thread that created it through the WindowsFormsSynchronizationContext. The thread here is not just the UI thread. With these basic knowledge, let’s analyze why it gets stuck.

3. The real cause of the stuck

OnUserPreferenceChanged -> WindowsFormsSynchronizationContext.Send -> Control.MarshaledInvoke -> WaitHandle.WaitOneNative again at the call stack of the main thread, its trend is like this: 0613eb0b066ee6, haha, do you see any problems? ? ?

WindowsFormsSynchronizationContext.Send eyed friends will find out why the main thread calls the 0613eb0b066f0c method? Isn't the Control registering the handler created by the main thread? To answer this question, you need to look at the value of the destinationThreadRef field of the WindowsFormsSynchronizationContext class. The source code is as follows:

public sealed class WindowsFormsSynchronizationContext : SynchronizationContext, IDisposable
{
    private Control controlToSendTo;

    private WeakReference destinationThreadRef;
}

You can use the !dso command to find out the WindowsFormsSynchronizationContext on the thread stack. The simplified output is as follows:

0:000:x86> !dso
OS Thread Id: 0x1d90 (0)
ESP/REG  Object   Name
0019ED70 027e441c System.Windows.Forms.WindowsFormsSynchronizationContext
0019EDC8 112ee43c Microsoft.Win32.SafeHandles.SafeWaitHandle
0019F078 11098b74 System.Windows.Forms.WindowsFormsSynchronizationContext
0019F080 1107487c Microsoft.Win32.SystemEvents+SystemEventInvokeInfo
0019F08C 10fa386c System.Object[]    (System.Object[])
0019F090 1107487c Microsoft.Win32.SystemEvents+SystemEventInvokeInfo
0019F0AC 027ebf60 System.Object
0019F0C0 10fa386c System.Object[]    (System.Object[])
0019F0C8 027ebe3c System.Object
0019F0CC 10fa388c Microsoft.Win32.SystemEvents+SystemEventInvokeInfo[]
...

0:000:x86> !do 11098b74
Name:        System.Windows.Forms.WindowsFormsSynchronizationContext
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6dbd8f30  4002567        8 ...ows.Forms.Control  0 instance 11098c24 controlToSendTo
6c667c2c  4002568        c System.WeakReference  0 instance 11098b88 destinationThreadRef

0:000:x86> !do 11098b88
Name:        System.WeakReference
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6c66938c  4000705        4        System.IntPtr  1 instance  86e426c m_handle

0:000:x86> !do poi(86e426c)
Name:        System.Threading.Thread
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6c663cc4  40018a5       24         System.Int32  1 instance        2 m_Priority
6c663cc4  40018a6       28         System.Int32  1 instance        7 m_ManagedThreadId
6c66f3d8  40018a7       2c       System.Boolean  1 instance        1 m_ExecutionContextBelongsToOuterScope

Sure enough, from the perspective of the hexagram, Thread=7 thread, so what kind of thread is Thread=7? It can be viewed through !t .

0:028:x86> !t
ThreadCount:      207
UnstartedThread:  0
BackgroundThread: 206
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                         Lock  
       ID OSID ThreadOBJ    State GC Mode     GC Alloc Context  Domain   Count Apt Exception
   0    1 1d90 003e2430   2026020 Preemptive  00000000:00000000 003db8b8 0     STA 
   2    2 2804 003f0188     2b220 Preemptive  00000000:00000000 003db8b8 0     MTA (Finalizer) 
  28    7 27f0 0b29cd30   3029220 Preemptive  00000000:00000000 003db8b8 0     MTA (Threadpool Worker) 

From the perspective of the hexagram: ID=7 is a thread pool thread, and it is in MTA mode. It stands to reason that it should create the control to the UI thread instead of creating it by itself, so the UI thread has been waiting at WaitOneNative for 7 No. thread message pump responds, so it causes an indefinite wait.

4. What control did thread 7 create?

This is another question that tests the underlying knowledge, and it has bothered me so far. It is too difficult. I tried to UserPreferenceChangedEventHandler all the handles on the 0613eb0b06707c incident, and wrote a script that looks like this:

"use strict";

// 32bit
let arr = ["xxxx"];

function initializeScript() { return [new host.apiVersionSupport(1, 7)]; }
function log(str) { host.diagnostics.debugLog(str + "\n"); }
function exec(str) { return host.namespace.Debugger.Utility.Control.ExecuteCommand(str); }

function invokeScript() {

    for (var address of arr) {
        var commandText = ".printf \"%04x\", poi(poi(poi(poi(" + address + "+0x4)+0xc)+0x4))";
        var output = exec(commandText).First();

        if (parseInt(output) == 0) continue; //not exists thread info

        commandText = ".printf \"%04x\", poi(poi(poi(poi(poi(" + address + "+0x4)+0xc)+0x4))+0x28)";
        output = exec(commandText).First();

        //thread id
        var tid = parseInt(output);

        if (tid > 1) log("Thread=" + tid + ",systemEventInvokeInfo=" + address);
    }
}

Output result:

||2:2:438>     !wow64exts.sw
Switched to Guest (WoW) mode
Thread=7,systemEventInvokeInfo=1107487c

The processing event systemEventInvokeInfo corresponding to thread No. was found from the output, and then traced as follows:

0:028:x86> !do 1107487c
Name:        Microsoft.Win32.SystemEvents+SystemEventInvokeInfo
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6c65ae34  4002e9f        4 ...ronizationContext  0 instance 11098b74 _syncContext
6c6635ac  4002ea0        8      System.Delegate  0 instance 1107485c _delegate

0:028:x86> !DumpObj /d 1107485c
Name:        Microsoft.Win32.UserPreferenceChangedEventHandler
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6c66211c  40002b0        4        System.Object  0 instance 110747bc _target
6c66211c  40002b1        8        System.Object  0 instance 00000000 _methodBase
6c66938c  40002b2        c        System.IntPtr  1 instance  6ebdc00 _methodPtr
6c66938c  40002b3       10        System.IntPtr  1 instance        0 _methodPtrAux
6c66211c  40002bd       14        System.Object  0 instance 00000000 _invocationList
6c66938c  40002be       18        System.IntPtr  1 instance        0 _invocationCount

0:028:x86> !DumpObj /d 110747bc
Name:        DevExpress.LookAndFeel.Design.UserLookAndFeelDefault

As you can see from the output, the last control is DevExpress.LookAndFeel.Design.UserLookAndFeelDefault . I thought I found the answer. I took this result and went to google. As a result, devExpress kicked the ball. The screenshot is as follows:

Cough, it seems that I can't check it up here. There are other information that Control will pass MarshalingControl when registering handlers across threads, so setting bp breakpoints in this control can be caught. The reference commands are as follows:

bp xxx ".echo MarshalingControl creation detected. Callstack follows.;!clrstack;.echo

I can't verify it here.

Three: Summary

Although I know that these three accidents are caused by the non-UI thread creating Control , it is a pity that I have tried my best knowledge and have not found the most important culprit, but I am happy that there is one based on the existing clues. A friend finally found the problem code, really happy for him😘😘😘, the solution is also very simple, the created control is dispatched to the UI thread for execution through Invoke. Screenshot below:

Through this case, I found that advanced debugging is really a journey of asceticism, and adjust and cherish it!