Race condition in MessageComposer causes NullReferenceException

Post Reply
shane
Posts: 5
Joined: Tue Apr 28, 2020 6:43 am

Race condition in MessageComposer causes NullReferenceException

Post by shane » Tue May 12, 2020 6:12 am

Hi,

When using MessageComposer in a stress testing project, I sometimes receive a NullReferenceException:

Code: Select all

Object reference not set to an instance of an object. System.NullReferenceException

 	Inetlab.SMPP.dll!Inetlab.SMPP.Common.ComposerItem.GetFullMessage() Line 392	C#
 	Inetlab.SMPP.dll!Inetlab.SMPP.Common.MessageComposer.OnFullMessageReceived(Inetlab.SMPP.Common.ComposerItem item) Line 239	C#
 	Inetlab.SMPP.dll!Inetlab.SMPP.Common.MessageComposer.AddMessage<Inetlab.SMPP.PDU.SubmitSm>(Inetlab.SMPP.PDU.SubmitSm message) Line 76	C#
 	TestNetCore.dll!TestNetCore.SmsRabbitLoadTest.LoadTestSmppServer.WhenServerReceivesPDU(object sender, Inetlab.SMPP.SmppServerClient serverClient, Inetlab.SMPP.PDU.SubmitSm data) Line 128	C#
 	Inetlab.SMPP.dll!Inetlab.SMPP.SmppServer.RaiseClientSubmitSm(Inetlab.SMPP.SmppServerClient client, Inetlab.SMPP.PDU.SubmitSm data) Line 581	C#
 	Inetlab.SMPP.dll!Inetlab.SMPP.SmppServer.client_evSubmitSm(object sender, Inetlab.SMPP.PDU.SubmitSm data) Line 421	C#
 	Inetlab.SMPP.dll!Inetlab.SMPP.SmppServerClient.OnSubmitSm(Inetlab.SMPP.PDU.SubmitSm data) Line 513	C#
 	Inetlab.SMPP.dll!Inetlab.SMPP.SmppServerClient..ctor.AnonymousMethod__21_0(Inetlab.SMPP.PDU.SmppRequest req) Line 133	C#
 	Inetlab.SMPP.dll!Inetlab.SMPP.SmppClientBase.HandleReceivedMessage(Inetlab.SMPP.PDU.SmppPDU pdu) Line 757	C#
 	Inetlab.SMPP.dll!Inetlab.SMPP.SmppClientBase.HandleData(byte[] buffer) Line 727	C#
 	Inetlab.SMPP.dll!Inetlab.SMPP.SmppClientBase.OnDataReceived.AnonymousMethod__1() Line 1149	C#
This also occurs in HasAllParts().

This is occurring because ComposerItem.Parts is null. This can only occur if the ComposerItem was disposed. I logged the call stack when Dispose() is called:

Code: Select all

   at Inetlab.SMPP.Common.ComposerItem.Dispose(Boolean disposing) in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\Common\MessageComposer.cs:line 423
   at Inetlab.SMPP.Common.ComposerItem.Dispose() in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\Common\MessageComposer.cs:line 434
   at Inetlab.SMPP.Common.MessageComposer.OnFullMessageReceived(ComposerItem item) in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\Common\MessageComposer.cs:line 231
   at Inetlab.SMPP.Common.MessageComposer.AddMessage[TSmppMessage](TSmppMessage message) in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\Common\MessageComposer.cs:line 76
   at TestNetCore.SmsRabbitLoadTest.LoadTestSmppServer.WhenServerReceivesPDU(Object sender, SmppServerClient serverClient, SubmitSm data) in c:\dev\poc\TestNetCore\TestNetCore\TestNetCore\SmsRabbitLoadTest\LoadTestSmppServer.cs:line 128
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at TestNetCore.SmsRabbitLoadTest.LoadTestSmppServer.WhenServerReceivesPDU(Object sender, SmppServerClient serverClient, SubmitSm data)
   at Inetlab.SMPP.SmppServer.RaiseClientSubmitSm(SmppServerClient client, SubmitSm data) in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\SmppServer.cs:line 581
   at Inetlab.SMPP.SmppServer.client_evSubmitSm(Object sender, SubmitSm data) in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\SmppServer.cs:line 421
   at Inetlab.SMPP.SmppServerClient.OnSubmitSm(SubmitSm data) in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\SmppServerClient.cs:line 513
   at Inetlab.SMPP.SmppServerClient.<.ctor>b__21_0(SmppRequest req) in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\SmppServerClient.cs:line 133
   at Inetlab.SMPP.SmppClientBase.HandleReceivedMessage(SmppPDU pdu) in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\SmppClientBase.cs:line 757
   at Inetlab.SMPP.SmppClientBase.HandleData(Byte[] buffer) in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\SmppClientBase.cs:line 727
   at Inetlab.SMPP.SmppClientBase.<>c__DisplayClass137_0.<OnDataReceived>b__1() in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\SmppClientBase.cs:line 1149
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.Tasks.Task.ExecuteEntry()
   at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task task)
   at Inetlab.SMPP.Common.ReceiveTaskScheduler.Execute() in C:\dev\_lib\Inetlab.SMPP.SourceCode.CS.2.8.0\Inetlab.SMPP\Common\ReceiveTaskScheduler.cs:line 106
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.<>c.<.cctor>b__10_0(Object s)
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)
My guess is that this is triggered when a duplicate pdu is received in quick succession.

After studying this for a while, my suggested fix would be:

- HasAllParts(), in the lock check if Parts == null and return false.
- OnFullMessageReceived, move "if (_itemStore.TryRemove(item.Key))" to surround the entire method implementation (so it doesn't try to interact with the ComposerItem unless it is the first time it is removed).

Cheers.
shane
Posts: 5
Joined: Tue Apr 28, 2020 6:43 am

Re: Race condition in MessageComposer causes NullReferenceException

Post by shane » Thu May 14, 2020 1:30 am

Alternate theory why this happens:

- two parts received quickly. asynchronous handler.
- in MessageComposer.AddMessage, two parts are being processed simultaneously by two threads
- both threads execute _itemStore.AddOrUpdate near simultaneously
- when both threads land on the next line (checking item.HasAllParts()) for the same message, they both are true hence OnFullMessageReceived is called twice for one message
- first thread executes OnnFullMessageReceived, removes from _itemStore and disposes the item
- second thread executes OnFullMessageReceived, attempts to interact with the item, but it's disposed already, hence NullReferenceException
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: Race condition in MessageComposer causes NullReferenceException

Post by alt » Thu May 14, 2020 2:07 pm

I have made ComposeItem non-disposable. The test pass where MessageComposer receives 7 parts with 100 duplicates for each of them from 700 different threads at the same time.

Please test with version 2.8.1-beta-2021.
developerlearn999
Posts: 85
Joined: Thu Jun 20, 2019 9:34 am

Re: Race condition in MessageComposer causes NullReferenceException

Post by developerlearn999 » Wed Jun 17, 2020 8:24 am

yes fixed!
developerlearn999
Posts: 85
Joined: Thu Jun 20, 2019 9:34 am

Re: Race condition in MessageComposer causes NullReferenceException

Post by developerlearn999 » Wed Jun 17, 2020 11:34 am

it's fixed but for some reason the Reference number jump's in odd number (not critical but just to know)
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: Race condition in MessageComposer causes NullReferenceException

Post by alt » Wed Jun 17, 2020 2:30 pm

Could you provide more information so that I can reproduce this.

What kind of conatenation do you use ?
- SAR in optional parameters
- UDH 8bit
- UDH 16bit

Do you set the reference number with your code or let you do it the library?
ahmedreza
Posts: 3
Joined: Sat Jun 20, 2020 5:31 pm

Re: Race condition in MessageComposer causes NullReferenceException

Post by ahmedreza » Sat Jun 20, 2020 5:32 pm

Can you suggest what code changes did you made for this fix ?
Post Reply