Ticket ID: 21478 Creation Date: 3/16/2015 9:49 PM Product: SiteRemote Attachment: -
TicketType: Support Request Version: 4.0.0 Language: English
Views: 26719 Last Modification Date: 4/16/2015 6:06 PM Platform:
Level: Closed

Support Request: Intermittent System.AccessViolationException

Description

We occasionally see this error logged:


Application: SiteRemoteServerService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
Stack:
at Provisio.Longinus.Server.Components.Xmpp.JobQueue.ManualJobQueue.<.ctor>b__0(System.Reactive.Notification`1<System.Reactive.Unit>)
at System.Reactive.Linq.Observable+WhereObservable`1+Observer[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Next(System.__Canon)
at System.Reactive.Observer`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnNext(System.__Canon)
at System.Reactive.Linq.Observable+WhereObservable`1+Observer[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Next(System.__Canon)
at System.Reactive.Linq.Observable+<>c__DisplayClass9c2`1+<>c__DisplayClass9c4[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].<Materialize>b__9c1()
at System.Reactive.Subjects.AsyncSubject`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].OnCompleted()
at Provisio.Longinus.Server.Components.Xmpp.JobQueue.AutoJobQueue.OnJobCompleted(Job, System.Exception)
at Provisio.Longinus.Server.Components.Xmpp.JobQueue.ManualJobQueue+<>c__DisplayClass10.<StartJob>b__c(System.Reactive.Unit)
at System.Reactive.Linq.Observable+<>c__DisplayClass326`1+<>c__DisplayClass328[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].<Return>b__325()
at System.Reactive.Concurrency.Scheduler.Invoke(System.Reactive.Concurrency.IScheduler, System.Action)
at System.Reactive.Concurrency.ImmediateScheduler.Schedule[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.__Canon, System.Func`3<System.Reactive.Concurrency.IScheduler,System.__Canon,System.IDisposable>)
at System.Reactive.AnonymousObservable`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].SubscribeCore(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.ObservableBase`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].Subscribe(System.IObserver`1<System.Reactive.Unit>)
at Provisio.Longinus.Server.Components.Xmpp.JobQueue.ManualJobQueue.StartJob(Job)
at Provisio.Longinus.Server.Components.Xmpp.JobQueue.ManualJobQueue.StartUpTo(Int32)
at Provisio.Longinus.Server.Components.Xmpp.JobQueue.AutoJobQueue.Add(System.Func`1<System.IObservable`1<System.Reactive.Unit>>, Boolean)
at Provisio.Longinus.Server.Components.Xmpp.StanzaActionLauncher.Execute(Provisio.Longinus.Server.IAppHost, Provisio.Xmpp.Jid, System.Func`2<Provisio.Longinus.Server.Components.Xmpp.XmppSessionContext,System.IObservable`1<System.Reactive.Unit>>, System.DateTime)
at Provisio.Longinus.Server.Components.Xmpp.StanzaActionLauncher.Execute(Provisio.Longinus.Server.IAppHost, Provisio.Xmpp.Jid, System.Func`2<Provisio.Longinus.Server.Components.Xmpp.XmppSessionContext,System.IObservable`1<System.Reactive.Unit>>, System.TimeSpan)
at Provisio.Longinus.Server.Components.Xmpp.StanzaActionLauncher.Execute(Provisio.Longinus.Server.IAppHost, Provisio.Xmpp.Jid, System.Func`2<Provisio.Longinus.Server.Components.Xmpp.XmppSessionContext,System.IObservable`1<System.Reactive.Unit>>)
at System.Linq.Enumerable+WhereSelectEnumerableIterator`2[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Reactive.Linq.Observable+<>c__DisplayClass331`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].<Subscribe>b__330(System.Action)
at System.Reactive.Concurrency.Scheduler.InvokeRec1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Reactive.Concurrency.IScheduler, Pair`2<System.__Canon,System.Action`2<System.__Canon,System.Action`1<System.__Canon>>>)
at System.Reactive.Concurrency.ImmediateScheduler.Schedule[[System.Reactive.Concurrency.Scheduler+Pair`2[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](Pair`2<System.__Canon,System.__Canon>, System.Func`3<System.Reactive.Concurrency.IScheduler,Pair`2<System.__Canon,System.__Canon>,System.IDisposable>)
at System.Reactive.Concurrency.Scheduler.Schedule[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Reactive.Concurrency.IScheduler, System.__Canon, System.Action`2<System.__Canon,System.Action`1<System.__Canon>>)
at System.Reactive.Linq.Observable.Subscribe[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.IObserver`1<System.__Canon>, System.Reactive.Concurrency.IScheduler)
at System.Reactive.AnonymousObservable`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SubscribeCore(System.IObserver`1<System.__Canon>)
at System.Reactive.ObservableBase`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Subscribe(System.IObserver`1<System.__Canon>)
at System.Reactive.Linq.Observable+<>c__DisplayClass38f`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].<Merge>b__388(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.AnonymousObservable`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].SubscribeCore(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.ObservableBase`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].Subscribe(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.Linq.Observable+<>c__DisplayClass33a`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].<Defer>b__339(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.AnonymousObservable`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].SubscribeCore(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.ObservableBase`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].Subscribe(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.Linq.Observable+<>c__DisplayClass3c9`2[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].<Catch>b__3c7(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.AnonymousObservable`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].SubscribeCore(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.ObservableBase`1+<>c__DisplayClassc5a[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].<Subscribe>b__c59()
at System.Reactive.Concurrency.Scheduler.Invoke(System.Reactive.Concurrency.IScheduler, System.Action)
at System.Reactive.Concurrency.ScheduledItem`2[[System.DateTimeOffset, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].InvokeCore()
at System.Reactive.Concurrency.ScheduledItem`1[[System.DateTimeOffset, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Invoke()
at System.Reactive.Concurrency.CurrentThreadScheduler+Trampoline.Run()
at System.Reactive.Concurrency.CurrentThreadScheduler.Schedule[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.__Canon, System.TimeSpan, System.Func`3<System.Reactive.Concurrency.IScheduler,System.__Canon,System.IDisposable>)
at System.Reactive.Concurrency.CurrentThreadScheduler.Schedule[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.__Canon, System.Func`3<System.Reactive.Concurrency.IScheduler,System.__Canon,System.IDisposable>)
at System.Reactive.ObservableBase`1[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].Subscribe(System.IObserver`1<System.Reactive.Unit>)
at System.Reactive.Linq.Observable+<>c__DisplayClass2fd`1+<>c__DisplayClass2ff[[System.Reactive.Unit, System.Reactive, Version=1.1.11111.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]].<SubscribeOn>b__2fc()
at System.Reactive.Concurrency.Scheduler.Invoke(System.Reactive.Concurrency.IScheduler, System.Action)
at System.Reactive.Concurrency.TaskPoolScheduler+<>c__DisplayClass2`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].<Schedule>b__0()
at System.Threading.Tasks.Task.Execute()
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
at System.Threading.Tasks.Task.ExecuteEntry(Boolean)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()


When this occurs, the service restarts without incident (due to Windows settings) but SiteKiosk Windows devices appear to not contact the SiteRemote server. The "Last contact" of these devices shows the time of the SiteRemoteServerService crash. Device contact times are not updated again until the entire server is restarted. SiteKiosk Android devices are not affected by this problem and appear to contact the SiteRemote server normally after such a crash.

What's going on here, and what can we do to prevent it from happening?

Thank you.

Answer: (4)
Re: Intermittent System.AccessViolationException 3/17/2015 12:28 PM
Hello,

I can’t say why the service has crashed (maybe it is related to other Windows issues you had: http://www.provisio.com/en-US/CustomerSupportCenter/ArticleDetails.aspx?ArticleID=21317) but even our programmers have done everything to comply with the requirements of Microsoft to release the port binding when the SiteRemote Server service closes unexpectedly it seems that on your system the port 8086 will not be released on service restart so that SiteKiosk Windows Client you cannot connect until rebooting the Windows system.

You should get another Event log message about that issue (that the binding to the port has failed because it is in use).


Our developer also has asked for further information:
Which operating system do you use (for SiteRemote)?
How often does this issue occur?
Since when do you run this SiteRemote Server?
Since when does this problem occur?
Did you install any additional software since this problem occurs?

Regards,
Michael Olbrich
Re: Intermittent System.AccessViolationException 3/18/2015 3:44 PM
Hello,

Thanks for your assistance. Regarding the port binding issue, there is no mention of it in the event logs, but the SiteRemote log itself contains a message about a SocketException caused by trying to bind to an already bound port.

I am now wondering if this binding problem is due to the VideoAudioAnalyzer child process that is spawned by the SiteRemoteServerService hanging onto a handle to the socket object it inherits from SiteRemoteServerSevice? I obviously have no idea if that child process actually does inherit handles from it's parent, but I do know that it is possible, and I know that VideoAudioAnalyzer is running after SiteRemoteServerService crashes because there are event logs about it leaking registry handles.

Also, here are the answers to your questions:

Which operating system do you use (for SiteRemote)?
>> Windows Server 2008 R2 Datacenter

How often does this issue occur?
>> Every 3-30 days. No obvious pattern.

Since when do you run this SiteRemote Server?
>> Since November 2012.

Since when does this problem occur?
>> Since at least July 2014. This is the earliest I have event logs for.

Did you install any additional software since this problem occurs?
>> New Relic .NET and Windows Server agents.
Re: Intermittent System.AccessViolationException 3/19/2015 11:19 AM
Hello,

Thank you for this information. I asked our developer and the VideoAudioAnalyzer child process should have nothing to do with the binding problem.

Even our programmers have done everything to comply with the requirements of Microsoft to release the port binding; on your system the port will not be released on SiteRemote Server service restart after a crash.

You may first check if there are any further .NET updates available and in case there are .NET updates available install them all.

If that doesn’t hale you may contact us via e-mail to get a workaround script that restarts the Windows system if the CPU usage falls below a certain value.

Regards,
Michael Olbrich
Re: Intermittent System.AccessViolationException 3/19/2015 5:09 PM
Hello,

We will investigate available updates. If we want to take advantage of the workaround offered, I'll send you an email as directed.

Thanks.
Post comment
My Account
Login
Language (Tickets):