An analysis of the CPU burst of a .NET Web site of a travel agency

One: background

1. Tell a story

A friend of wx asked for help a few days ago. Its program memory often soars, and the cpu occasionally soars. I can't find the reason. I hope to help.

It is a pity that the dump sent is only 2G, and there are really two brushes if you can find the memory overflow in it. . . 😂😂😂, so I still hope that his program memory will rise to 5G+ and show it to me. Since the memory can’t be seen, let’s see what happens to the occasional soaring CPU? The old way, talk to windbg.

Two: windbg analysis

1. What is the CPU?

To view the CPU usage of the machine when this snapshot is generated, you can use the !tp command.

0:033> !tp
CPU utilization: 93%
Worker Thread: Total: 800 Running: 800 Idle: 0 MaxLimit: 800 MinLimit: 320
Work Request in Queue: 3203
    Unknown Function: 000007fefb551500  Context: 000000002a198480
    Unknown Function: 000007fefb551500  Context: 0000000028a70780
    Unknown Function: 000007fefb551500  Context: 000000002a182610
    Unknown Function: 000007fefb551500  Context: 00000000262a2700
    ...

I thought it was a simple command, but it turned out to be a bunch of hoops on the screen. . . It's a little unexpected. From the above diagram, the current CPU utilization rate is 93%, no problem. The CPU is indeed soaring. What's more surprising is that the upper limit of the thread pool is full of 800 threads, which is too tragic. . . What is even more tragic is thread pool queue. It can be guessed that the program is not only high CPU, but also hangs. . .

The next question is: what's wrong with these 800 heroes? The program is now in the use of people. If you want to find out the answer, you should follow my inertial thinking and check the synchronization block table.

2. Thread synchronization block table

To view the synchronization block table, you can use the !synblk command.

0:033> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
  188 0000000010defc28            1         1 000000001e8fb400 9f4 715   00000003ff1e3d80 System.Web.HttpApplicationStateLock
126159 000000001e424e28            1         1 0000000023425e00 1f14 695   0000000301210038 ASP.global_asax
126173 00000000281acaf8            1         1 0000000024b8ea70 24ec 785   00000000ff8c5e10 ASP.global_asax
126289 00000000247a4068            1         1 0000000027ee93c0 808 413   0000000306aca288 ASP.global_asax
126368 0000000027180dd8            1         1 0000000028005cb0 1e7c 650   00000002008d6280 ASP.global_asax
126489 0000000027211dd8            1         1 0000000026862420 ec4 220   000000030611a290 ASP.global_asax
126788 00000000247924b8            1         1 0000000021871ff0 2784 529   00000004039901a8 ASP.global_asax
126843 00000000285b8d28            1         1 000000001cbd6710 2170 456   00000004007ec748 ASP.global_asax
126934 0000000021b212b8            1         1 0000000026ca7590 16cc 472   000000030090e810 ASP.global_asax
127251 0000000024769188            1         1 000000002831eaf0 2b68 648   0000000207051038 ASP.global_asax
...

-----------------------------
Total           141781
CCW             2
RCW             4
ComClassFactory 0
Free            140270

When I go, there are a bunch of hula la, two pieces of information can be seen from the hexagram above:

• MonitorHeld: 1

Indicates that a thread is currently holding the lock.

• ASP.global_asax , System.Web.HttpApplicationStateLock

Represents the object held by the current thread.

However, it is a bit strange in general. Except for the first thread holding HttpApplicationStateLock , all the ASP.global_asax objects held by subsequent threads have different memory addresses: 0000000301210038,00000000ff8c5e10 . It feels that the lock object is not a thread-sharing static, but more like an instance. , Quite interesting, next draw two threads to look at its thread stack, such as here: 715,695 .

3. View the thread stack

To view the thread stack, you can use the !clrstack command.




From the perspective of these two thread stacks, they are System.Threading.Monitor.Enter(System.Object) and System.Threading.Monitor.ObjWait xxx.MvcApplication.Session_Start method respectively. In general, the Session_Start method here must be problematic, so we have to find a way to export the source code and take a look.

4. View the problem code

To export the Session_Start method, use the combined command !ip2md + !savemodule .

||2:2:1781> !ip2md 000007fe99c6f0c5
MethodDesc:   000007fe990fe080
Method Name:  xxx.xxx.xxx.MvcApplication.Session_Start(System.Object, System.EventArgs)
Class:        000007fe991ae0c0
MethodTable:  000007fe990fe238
mdToken:      0000000006000119
Module:       000007fe990fd750
IsJitted:     yes
CodeAddr:     000007fe99c6e1f0
Transparency: Critical
||2:2:1781> !savemodule 000007fe990fd750 E:\dumps\Session_Start.dll
3 sections in file
section 0 - VA=2000, VASize=17538, FileAddr=200, FileSize=17600
section 1 - VA=1a000, VASize=3ac, FileAddr=17800, FileSize=400
section 2 - VA=1c000, VASize=c, FileAddr=17c00, FileSize=200

Then use the ILSpy decompiler tool to check, because it is more sensitive, I am more vague, please forgive me!

After reading the above code, I am actually a bit puzzled. Since it is assigned to Application, why not extract it to Application_Start ? I guess it doesn't matter to developers, it doesn't matter how convenient they are. Next, let's look at the source code of Application.

public sealed class HttpApplicationState : NameObjectCollectionBase
{
    private HttpApplicationStateLock _lock = new HttpApplicationStateLock();

    public void Set(string name, object value)
    {
        _lock.AcquireWrite();
        try
        {
            BaseSet(name, value);
        }
        finally
        {
            _lock.ReleaseWrite();
        }
    }
}

internal class HttpApplicationStateLock : ReadWriteObjectLock
{
    internal override void AcquireWrite()
    {
        int currentThreadId = SafeNativeMethods.GetCurrentThreadId();
        if (_threadId == currentThreadId)
        {
            _recursionCount++;
            return;
        }
        base.AcquireWrite();
        _threadId = currentThreadId;
        _recursionCount = 1;
    }

    internal override void ReleaseWrite()
    {
        int currentThreadId = SafeNativeMethods.GetCurrentThreadId();
        if (_threadId == currentThreadId && --_recursionCount == 0)
        {
            _threadId = 0;
            base.ReleaseWrite();
        }
    }
}

internal class ReadWriteObjectLock
{
    internal virtual void AcquireWrite()
    {
        lock (this)
        {
            while (_lock != 0)
            {
                try
                {
                    Monitor.Wait(this);
                }
                catch (ThreadInterruptedException)
                {
                }
            }
            _lock = -1;
        }
    }
    internal virtual void ReleaseWrite()
    {
        lock (this)
        {
            _lock = 0;
            Monitor.PulseAll(this);
        }
    }
}

The code is a bit long, but in general, the code here is not simple. Application encapsulates a read-write lock through lock. It is not simple or simple, but what is the problem here? Even if the wrong place is written, it seems that it will not cause it. cpu burst high, right?

In fact, there is a concept involved here: that is lock convoys (lock escort)

5. lock convoys (lock escort)

Regarding what lock convoys , I found an article with a good explanation: Lock Escort , here I cut a picture, everyone carefully.

This is also lock-free programming has been criticizing.

Three: Summary

I looked at this Session_Start method, there are about 105 Application[xxx] , which means that there are 105 locks waiting for the current thread to pass through. . . At this time, nearly 800 threads have entered this method. In total, there are no less than 8W locks waiting for these threads to break through. With the forced mass CPU time slice switch, wake up and sleep again, and sleep again. Wake up, everyone staggered and lifted the cpu together.

The solution is simple. Try your best to reduce serial locks. It would be better if you can reduce it to one or even none 😄😄😄.

• All assignments to Application are extracted to Application_Start, after all, there is no competition when the program is started.
• Try to change single line assignment to batch assignment.

More high-quality dry goods: see my GitHub: dotnetfly