SMPP Server An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException:

[ahmedreza]

Out of Memory Exception when a clients attempts multiple binds to the server.

19.06.2020 12:10:12:INFO :2920: (SmppServerClient2817.Socket) XXXX:2465 => XXX:61928 Connection established.
19.06.2020 12:10:12:INFO :2920: (SmppServer1) [XXX:61928] connected.
19.06.2020 12:10:12:ERROR:2920: (SmppServerClient2817) Event evConnected failed
Exception: System.Threading.Tasks.TaskSchedulerException: An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
at System.Threading.Thread.Start(StackCrawlMark& stackMark)
at System.Threading.Thread.Start(Object parameter)
at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask(Task task)
at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlMark& stackMark)
at System.Threading.Tasks.TaskFactory.StartNew(Action action, TaskCreationOptions creationOptions)
at Inetlab.SMPP.Common.TimeLimitedQueue..ctor(Int32 capacity)
at Inetlab.SMPP.Common.RateGate.set_Rate(LimitRate value)
at SMPP_Server.frmServer.server_evClientConnected(Object sender, SmppServerClient client)
at Inetlab.SMPP.SmppServer.RaiseClientConnected(SmppServerClient client)
at Inetlab.SMPP.SmppServer.client_evConnected(Object sender, Boolean success)
at Inetlab.SMPP.SmppClientBase.OnConnected(Boolean success)
19.06.2020 12:10:12:ERROR:2920: (SmppServerClient2817.Socket) Event evConnected failed.
Exception: System.Threading.Tasks.TaskSchedulerException: An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
at System.Threading.Thread.Start(StackCrawlMark& stackMark)
at System.Threading.Thread.Start(Object parameter)
at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask(Task task)
at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlMark& stackMark)
at System.Threading.Tasks.TaskFactory.StartNew(Action action, TaskCreationOptions creationOptions)
at Inetlab.SMPP.Common.SmppSocket.StartReading()
at Inetlab.SMPP.Common.SmppSocket.OnConnected()

[alt]

Could you provide more details on your context?
What library version do you use?
How many concurrent connections your server accepts?

Consider to create memory dump with ProcDump.

Code: Select all

procdump -ma -e 1 -f OutOfMemoryException -w "YourProcessName"

Once you have a dump file, use DebugDiag to analyze it.

[ahmedreza]

We have this in bind section to limit 2 connections.

Library Version: ## [2.7.0] - 2019-12-11

//deny multiple connection with same smpp system id max 2 allowed
if (_server.ConnectedClients.Count(c => c.SystemID == data.SystemId) > 2)
{
data.Response.Header.Status = CommandStatus.ESME_RALYBND;

[alt]

I suppose the client establishes new connection after receiving the response with status ESME_RALYBND, but doesn't disconnect previous connection. When it happens too often you get out of memory exception.

Could you tell me how many maximal connections your server had? It should be a lot of connections in Open state and ony some of them in Bound state.
You could write in log the number of Open and Bound connections in evClientConnected event handler.

[alt]

You can also change InactivityTimeout for connected clients. It helps to save server resources.

[chengcti]

I also reproduce the same phenomenon.
My server only support one session, but customer connect SMPPServer with two session every second.
And I will terminate second session.
After 5~6 hours, the smppserver response out-of-memory.

The test condition.
1. SMPPclient -> Connect to server with username/password
2. SMPPclient -> Connect to server with step 1 condition every one second for 20 times.

parameter:
client.ReceiveBufferSize = 65535;
client.SendBufferSize = 65535;
client.InactivityTimeout = TimeSpan.FromSeconds(30);
client.ReceiveSpeedLimit = 40;
client.ReceiveTaskScheduler = TaskScheduler.Default;

version : 2.9.13

[chengcti]

Thank you ALT for your help.
Why I like your product and support it for a long time, because you always listen to the voice of your customers and solve it quickly.

Root Cause
SmppClientBase :
property ReceiveSpeedLimit
After the client disconnect in the smppserver object, it will cause a memory leak.

Solution
It be solved in the v2.9.14