| Author |
|
support
Administrator
Joined: January 26 2005
Location: United States
Posts: 1666
|
| Posted: August 14 2008 at 10:15am | IP Logged
|
|
|
Juice,
Have you built your VOIP app using a thread stack size that is smaller than the defaut????
Hmmm.....
Support
|
| Back to Top |
|
| |
juice
Vetran
Joined: December 05 2006
Location: United States
Posts: 139
|
| Posted: August 14 2008 at 9:15pm | IP Logged
|
|
|
Yes, we have in fact reduced the stack size according to the optimization help info for using the media engine. This was done for the following reasons:
* Be able to use all lines (see note above)
* Try to get more lines starting up when conference support is enabled (though, no matter what, seems we can only get around ~250 lines started before we get a memory error on startup).
We can try increasing the size of the stack, what is the minimum value you recommend (since it seems to be a problem)? If I recall, we lowered it to around 50k since we don't use the stack at all really.
|
| Back to Top |
|
| |
support
Administrator
Joined: January 26 2005
Location: United States
Posts: 1666
|
| Posted: August 15 2008 at 10:10am | IP Logged
|
|
|
Juice,
Not sure if a stack size of 50k bytes is ok for your app and the media engine – that may be a bit low. For testing purposes - try to increase the default stack size to 100k, create as many phone lines as you can and allow the testing to run again. If it crashes, increase the stack to 200k and retest. This way we can determine if the stack has anything to do with the crash. 200k should definitely be OK – but not sure if 50k is OK. If crashes occur, please upload mini dumps with full heap as needed. If we have a stack overflow situation, we would normally get a ‘stack overflow” exception and not an access violation. That being said, there could be stack corruption occurring for some reason. Unfortunately, the media engine build you have does not have runtime checking enabled. Something we may have to consider for later.
If we need to address the media engine’s ability to create a greater number of phone lines, we can manage that here in a number of different ways. There is more we can do to minimize memory usage – just have not had time to implement the changes.
As the result of inspection of your dump image:
An internal codec is attempting to convert 8k PCM data to 8k G729.
The proc that is generating the access violation is accessing the source PCM buffer via indirection using stack variables. If we can figure out why the access to the source PCM buffer is bad, we will be able to fix it.
If we can get mini dumps from you that contains full heap, that will really help. That way we can inspect much more post mortem.
Upload whatever dump files you feel will help us track this down.
We have to think...
Support
|
| Back to Top |
|
| |
juice
Vetran
Joined: December 05 2006
Location: United States
Posts: 139
|
| Posted: August 19 2008 at 8:50pm | IP Logged
|
|
|
Hmm, after going back and examining the setting, it is set for 500000. Both Reserve/Commit size. We had earlier experimented with lower settings to up the amount of lines we could reach with conference enabled (had no lock, was not able to get over 250ish).
So, shouldn't be from not having enough stack room. We will restart and see about getting a full heap dump. But, it's a little concerning, as our media engine has username/passwords of SIP endpoints as well as Database connection strings in memory, so we are not too sure we can produce such a dump. But, at least another mini dump without heap would see if the error happens in the same spot or jumps to some other random location. We will see what we can do to help find the issue.
|
| Back to Top |
|
| |
support
Administrator
Joined: January 26 2005
Location: United States
Posts: 1666
|
| Posted: August 20 2008 at 9:59am | IP Logged
|
|
|
Hi Juice,
Agreed – its not a stack issue then. Good to remove that possibility from the table.
Lets just keep working the issue until we find it. The full heap mini dump will give us the info we need to fully post-mortem the problem.
I understand your concern regarding a full heap dump of process space that is running with real world username and password data in memory.
What I can tell you is that we already have executed a mutual NDA between your group and us. The MNDA is currently active and binding. We absolutely ensure the protection of any information that flows between our groups. If we need to take additional steps to give your group additional confidence in our ability to protect your critical data, we are able to do so.
Don’t hesitate to have your boss contact our CTO to discuss this confidentiality issue. From this end, we just want to be able to resolve the issue as quickly as we can.
Thanks,
Support
|
| Back to Top |
|
| |
support
Administrator
Joined: January 26 2005
Location: United States
Posts: 1666
|
| Posted: September 02 2008 at 3:57pm | IP Logged
|
|
|
Juice,
We located an RTP transmitter issue that was reported by another customer that may have something to do with the “crash” you have reported.
See the end of the following support forum post:
Wave File Playback:
http://www.lanscapecorp.com/forum/forum_posts.asp?TID=503&PN =1
The bug is an uninitialized pointer in RTP transmitter logic and has the potential of causing a lot of headaches.
The bug could directly cause an access violation or depending on the uninitialized pointer value, could lead to heap corruption.
As stated in the above post:
If your app is:
1) Using license files (LanScapeVME.*) that have integrated DTMF capability enabled and…
2) Your app has integrated DTMF disabled when the media engine starts and…
3) Your app filters transmitted RTP media packets,
Your app will be affected by this RTP transmitter pointer bug.
We have updated your support FTP account with v5.12.8.9 of the media engine. If you can, please retest your VOIP application using this image and repost back as soon as you have updated test results.
Thanks,
Support
|
| Back to Top |
|
| |
support
Administrator
Joined: January 26 2005
Location: United States
Posts: 1666
|
| Posted: September 09 2008 at 12:43pm | IP Logged
|
|
|
Juice,
Did you ever place the v5.12.8.9 media engine release under test to see if it removes the random crash you reported?
Its important that we try to resolve this as quick as we can.
If you are still performing testing, you might as well grab the v5.12.8.10 image from your support FTP account. It contains another bug fix associated with the conference calling support that you want to use in your B2B billing server. You will definitely need this latest conference fix so that you no longer have to tie together phone lines yourself.
Note:
We are changing the way we disseminate “engineering release” version info. Engineering release info will now always be located here:
VOIP Media Engine - Engineering Release Notes:
http://www.lanscapecorp.com/DevResources/VOIP Media Engine - Engineering Release Notes.htm
Please take a look at the latest release notes to see what else may be of benefit to you.
Thanks,
Support
|
| Back to Top |
|
| |
support
Administrator
Joined: January 26 2005
Location: United States
Posts: 1666
|
| Posted: September 19 2008 at 11:58am | IP Logged
|
|
|
Juice,
Brief Update:
From the post in this thread dated "14 August 2008 at 9:50am", we finally came across a test scenario that looks like it duplicates the problem as we saw per your mini dump file. It took a really long time to get a test assembled that would allow us to reproduce the crash.
We are now actively looking into what is foobar. With any luck, it will not take too much longer. Looks like a very esoteric nasty bug.
We will repost when we identify the solution. Thanks for staying with us - we know it sucks.
Support
|
| Back to Top |
|
| |
support
Administrator
Joined: January 26 2005
Location: United States
Posts: 1666
|
| Posted: September 19 2008 at 3:38pm | IP Logged
|
|
|
Juice,
Question:
When your app experiences the crash, was a phone line in the process of performing an unattended call transfer?
Support
|
| Back to Top |
|
| |
juice
Vetran
Joined: December 05 2006
Location: United States
Posts: 139
|
| Posted: September 19 2008 at 6:37pm | IP Logged
|
|
|
Hmm, we don't do any call transfers with the app. It might be possible a user requested a sip transfer, but we were not configured to allow it. The app only:
* Has conference enabled
* Receives incoming sip calls (does some database lookups to see if they are authorized)
* Makes an outgoing leg and then will either cross lines using the conference API or the old method (depending on what voip service they were trying to do).
|
| Back to Top |
|
| |
support
Administrator
Joined: January 26 2005
Location: United States
Posts: 1666
|
| Posted: September 23 2008 at 2:06pm | IP Logged
|
|
|
Juice,
A few questions regarding the media engine app that is crashing:
1):
Is your app still using Tx IVR outputs?
2):
Is it possible in your app that the Tx IVR outputs are not being closed by the app before another call is started on the same phone line? This is important.
3):
Is your VOIP app handling both uLaw and G729 type calls as it operates?
Based on your mini dump from an above post dated “Posted: 13 August 2008 at 9:55pm”, we were able to construct a test that gave us a crash that has a very similar stack trace signature. It was pure luck that a test case here seemed to corroborate your mini dump image.
We are not sure that what we see here is exactly what your have experienced but it is very likely this is the same issue.
The possible cause of the crash:
If Tx IVR outputs are not closed before another call is assigned to the phone line, then it may be possible that reference counted format/rate converter support in the media engine is getting “confused”. We notice in our testing that it was possible for this scenario to occur:
1)
A call goes to the SipInCall state.
2)
The app opens the Tx IVR output it needs.
3)
The app does it “thing”….
4)
The call terminates (SipOnHook state) and the app handles “call cleanup” and the “close IVR output” logic in a deferred manner.
5)
If the app does not close the Tx IVR outputs before the phone line gets used again, internal format/rate conversion for RTP transmitter support may inadvertently use the previous call’s residual reference counted format/rate support from the previous call. If the last call used uLaw codec and the new call wants to use G729 codec, an explosion (crash) may occur.
If the above scenario is possible in your app, the immediate work around would be to ensure the Tx IVR outputs your app uses are closed synchronously in the media engine’s main event handler.
Do this by calling the CloseTxIvrChannel() API proc directly in the main media engine event handler when the the SipOnHook event is generated.
Hopefully you can answer the questions above. Your answers will give us a very good indication if we have identified the culprit.
Support
|
| Back to Top |
|
| |
support
Administrator
Joined: January 26 2005
Location: United States
Posts: 1666
|
| Posted: September 23 2008 at 3:32pm | IP Logged
|
|
|
Juice,
Ok, we should have re-read your last post, We seen that you have already answered question #1. Our mistake...
Support
|
| Back to Top |
|
| |
juice
Vetran
Joined: December 05 2006
Location: United States
Posts: 139
|
| Posted: September 23 2008 at 8:29pm | IP Logged
|
|
|
#2 Our app operates pretty close to the earlier demo we uploaded. It should as a result of SipOnHook be closing the transmit line. We have no indication of this not happening. We do not release the line number to assign to incoming until after call complete and transmit line is closed. Though, it might be good if your media engine would fail immediately if that was the case so we could get an error code and perhaps traceback an open connection (if that is the case).
#3 Yes, we are handling both Codecs. I can't say for sure which lines are using what codecs at the time of failure.
We are still testing your last release, and as far as I am aware, have not experienced the crash as often. I do have to confer with the rest of the team to make sure that is the case.
|
| Back to Top |
|
| |
Active Topics 
Member List 
Search 
Help 
Register 
Login
LanScape Support Forum -> LanScape VOIP Media Engine™ - Technical Support
Topic: Double SipInCall messages
