SMPP Server An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException:

Post Reply
ahmedreza
Posts: 3
Joined: Sat Jun 20, 2020 5:31 pm

SMPP Server An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException:

Post by ahmedreza » Sat Jun 20, 2020 5:52 pm

Out of Memory Exception when a clients attempts multiple binds to the server.

19.06.2020 12:10:12:INFO :2920: (SmppServerClient2817.Socket) XXXX:2465 => XXX:61928 Connection established.
19.06.2020 12:10:12:INFO :2920: (SmppServer1) [XXX:61928] connected.
19.06.2020 12:10:12:ERROR:2920: (SmppServerClient2817) Event evConnected failed
Exception: System.Threading.Tasks.TaskSchedulerException: An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
at System.Threading.Thread.Start(StackCrawlMark& stackMark)
at System.Threading.Thread.Start(Object parameter)
at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask(Task task)
at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlMark& stackMark)
at System.Threading.Tasks.TaskFactory.StartNew(Action action, TaskCreationOptions creationOptions)
at Inetlab.SMPP.Common.TimeLimitedQueue..ctor(Int32 capacity)
at Inetlab.SMPP.Common.RateGate.set_Rate(LimitRate value)
at SMPP_Server.frmServer.server_evClientConnected(Object sender, SmppServerClient client)
at Inetlab.SMPP.SmppServer.RaiseClientConnected(SmppServerClient client)
at Inetlab.SMPP.SmppServer.client_evConnected(Object sender, Boolean success)
at Inetlab.SMPP.SmppClientBase.OnConnected(Boolean success)
19.06.2020 12:10:12:ERROR:2920: (SmppServerClient2817.Socket) Event evConnected failed.
Exception: System.Threading.Tasks.TaskSchedulerException: An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
at System.Threading.Thread.Start(StackCrawlMark& stackMark)
at System.Threading.Thread.Start(Object parameter)
at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask(Task task)
at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlMark& stackMark)
at System.Threading.Tasks.TaskFactory.StartNew(Action action, TaskCreationOptions creationOptions)
at Inetlab.SMPP.Common.SmppSocket.StartReading()
at Inetlab.SMPP.Common.SmppSocket.OnConnected()
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: SMPP Server An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException:

Post by alt » Mon Jun 22, 2020 9:23 pm

Could you provide more details on your context?
What library version do you use?
How many concurrent connections your server accepts?

Consider to create memory dump with ProcDump.

Code: Select all

procdump -ma -e 1 -f OutOfMemoryException -w "YourProcessName"
Once you have a dump file, use DebugDiag to analyze it.
ahmedreza
Posts: 3
Joined: Sat Jun 20, 2020 5:31 pm

Re: SMPP Server An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException:

Post by ahmedreza » Tue Jun 23, 2020 10:37 am

We have this in bind section to limit 2 connections.

Library Version: ## [2.7.0] - 2019-12-11

//deny multiple connection with same smpp system id max 2 allowed
if (_server.ConnectedClients.Count(c => c.SystemID == data.SystemId) > 2)
{
data.Response.Header.Status = CommandStatus.ESME_RALYBND;
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: SMPP Server An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException:

Post by alt » Sat Jun 27, 2020 9:07 am

I suppose the client establishes new connection after receiving the response with status ESME_RALYBND, but doesn't disconnect previous connection. When it happens too often you get out of memory exception.

Could you tell me how many maximal connections your server had? It should be a lot of connections in Open state and ony some of them in Bound state.
You could write in log the number of Open and Bound connections in evClientConnected event handler.
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: SMPP Server An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException:

Post by alt » Tue Jun 30, 2020 4:12 pm

You can also change InactivityTimeout for connected clients. It helps to save server resources.
chengcti
Posts: 6
Joined: Wed Jul 21, 2021 12:24 pm

Re: SMPP Server An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException:

Post by chengcti » Fri Aug 27, 2021 3:56 pm

I also reproduce the same phenomenon.
My server only support one session, but customer connect SMPPServer with two session every second.
And I will terminate second session.
After 5~6 hours, the smppserver response out-of-memory.

The test condition.
1. SMPPclient -> Connect to server with username/password
2. SMPPclient -> Connect to server with step 1 condition every one second for 20 times.

parameter:
client.ReceiveBufferSize = 65535;
client.SendBufferSize = 65535;
client.InactivityTimeout = TimeSpan.FromSeconds(30);
client.ReceiveSpeedLimit = 40;
client.ReceiveTaskScheduler = TaskScheduler.Default;

version : 2.9.13
chengcti
Posts: 6
Joined: Wed Jul 21, 2021 12:24 pm

Re: SMPP Server An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException:

Post by chengcti » Wed Sep 01, 2021 1:53 am

Thank you ALT for your help.
Why I like your product and support it for a long time, because you always listen to the voice of your customers and solve it quickly.

Root Cause
SmppClientBase :
property ReceiveSpeedLimit
After the client disconnect in the smppserver object, it will cause a memory leak.

Solution
It be solved in the v2.9.14
Post Reply