[jacorb-developer] Client Application of Jacorb3.0 hangs on IBM WebSphere

Phil Mesnier mesnierp at objectcomputing.com
Thu Mar 23 12:21:14 CET 2017 

Hi Prakash,

Please note that this community discussion board is a low priority channel.
If you are looking for commercial grade support, please contact one of the
providers listed on jacorb.org. I know that Avaya has worked with OCI in
the past.

Regarding known issues in JacORB 3.0, I don't know about implicit
deadlocks, but the request processor's thread pool management code was
refactored in JacORB 3.7 specifically to improve performance in high volume
uses.

You should get the latest release of JacORB and use that to try your
scenario again. If you still have trouble then we should take a deeper look.

Best regards,
Phil


On Tue, Mar 21, 2017 at 11:28 AM, Rungta, Prakash (Prakash) <
prungta at avaya.com> wrote:

> Hello Team,
>
> Would appreciate if anyone can look into this.
>
> Thanks,
> Prakash R
>
>
>
> _____________________________________________
> From: Rungta, Prakash (Prakash)
> Sent: Sunday, March 19, 2017 11:51 PM
> To: jacorb-developer at lists.spline.inf.fu-berlin.de
> Cc: Siddique, Md Anjum (Anjum); Rashinkar, Anand (Anand)
> Subject: Client Application of Jacorb3.0 hangs on IBM WebSphere
>
>
> Hello Team,
>
> I am developing a Jacorb 3.0 based Application (Client) . However during
> traffic conditions(Medium) the application(client application) hangs . I
> see the following errors in the client logs (Jacorb Logging increased)
> during start and at random times as well.
>
> 2017-03-19 16:19:28,843 [ClientMessageReceptor1] jacorb.poa.queue WARN -
> Request queue is full, consider increasing jacorb.poa.queue_max (currently:
> 100)
> 2017-03-19 16:19:28,844 [ClientMessageReceptor1] jacorb.poa.queue WARN -
> Request queue is full, consider increasing jacorb.poa.queue_max (currently:
> 100)
> 2017-03-19 16:19:28,844 [ClientMessageReceptor1] jacorb.poa.queue WARN -
> Request queue is full, consider increasing jacorb.poa.queue_max (currently:
> 100)
> 2017-03-19 16:19:28,844 [ClientMessageReceptor1] jacorb.poa.queue WARN -
> Request queue is full, consider increasing jacorb.poa.queue_max (currently:
> 100)
>
> 2017-03-19 16:19:28,845 [RequestController-2] jacorb.poa.controller WARN -
> Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max
> (currently: 20)
> 2017-03-19 16:19:28,845 [RequestController-2] jacorb.poa.controller WARN -
> Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max
> (currently: 20)
> 2017-03-19 16:19:28,845 [RequestController-2] jacorb.poa.controller WARN -
> Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max
> (currently: 20)
> 2017-03-19 16:19:28,845 [RequestController-2] jacorb.poa.controller WARN -
> Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max
> (currently: 20)
> 2017-03-19 16:19:28,846 [RequestController-2] jacorb.poa.controller WARN -
> Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max
> (currently: 20)
> 2017-03-19 16:19:28,846 [RequestController-2] jacorb.poa.controller WARN -
> Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max
> (currently: 20)
> 2017-03-19 16:19:28,846 [RequestController-2] jacorb.poa.controller WARN -
> Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max
> (currently: 20)
>
> On the Server side , I see the  CORBA TIMEOUT EXCEPTION being raised
>
>         e._info=system exception, ID 'IDL:omg.org/CORBA/TIMEOUT:1.0'
>
> When the client application hangs , I have generated a javacore , I see
> that there are lot of Thread blocking . Also there are lot of
> RequestProcessor threads which are in blocking state
>
> 3LKWAITNOTIFY            "RequestProcessor-53" (J9VMThread:
> 0x00000000A8742F00)
> 2LKMONINUSE      sys_mon_t:0x00007F933628F2D0 infl_mon_t:
> 0x00007F933628F348:
> 3LKMONOBJECT       org/jacorb/poa/RequestProcessor at 0x000000000B8E9B70:
> <unowned>
> 3LKNOTIFYQ            Waiting to be notified:
>
> 3XMTHREADINFO      "RequestProcessor-53" J9VMThread:0x00000000A8742F00,
> j9thread_t:0x00007F933DA77D30, java/lang/Thread:0x0000000006DC92C0,
> state:CW, prio=10
> 3XMJAVALTHREAD            (java/lang/Thread getId:0x30FC92, isDaemon:true)
> 3XMTHREADINFO1            (native thread ID:0x7A47, native priority:0xA,
> native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101)
> 3XMTHREADINFO2            (native stack address range
> from:0x00007F92B3BF6000, to:0x00007F92B3C37000, size:0x41000)
> 3XMCPUTIME               CPU usage total: 0.128906816 secs
> 3XMTHREADBLOCK     Waiting on: org/jacorb/poa/RequestProcessor at 0x0000000006DC92C0
> Owned by: <unowned>
> 3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
> 3XMTHREADINFO3           Java callstack:
> 4XESTACKTRACE                at java/lang/Object.wait(Native Method)
> 4XESTACKTRACE                at java/lang/Object.wait(Object.java:167(Compiled
> Code))
> 4XESTACKTRACE                at org/jacorb/poa/RequestProcessor.run(
> RequestProcessor.java:793(Compiled Code))
> 5XESTACKTRACE                   (entered lock: org/jacorb/poa/
> RequestProcessor at 0x0000000006DC92C0, entry count: 1)
> 3XMTHREADINFO3           Native callstack:
> 4XENATIVESTACK               (0x00007F9388B37912 [libj9prt26.so+0x13912])
>
> Application  overview
> Server is a C++ CORBA based and Client is  JAVA (Jacorb 3.0 based idl and
> runtime is both 3.0).Server does callback's to client application and
> client does processing.(For simplicity I have commented all the logic
> inside the client callback functions except some log statements during
> entering and leaving the functions.) still the issue is there.
>
> The client application  is deployed on RHEL 6.5 (64 bit ) on IBM WebSphere
> (8.5.5.4) .
>
> Following are some of Jacorb Properties that are set
>
> jacorb.net.server_socket_factory.port.min=52000
> jacorb.net.server_socket_factory.port.max=53000
> jacorb.security.support_ssl=on
>
> jacorb.security.ssl.client.supported_options=40
> jacorb.security.ssl.client.required_options=40
>
> jacorb.security.ssl.server.supported_options=0
> jacorb.security.ssl.server.required_options=0
>
> jacorb.security.jsse.server.key_manager_algorithm=IbmX509
> jacorb.security.jsse.server.trust_manager_algorithm=IbmX509
> jacorb.security.jsse.client.key_manager_algorithm=IbmX509
> jacorb.security.jsse.client.trust_manager_algorithm=IbmX509
>
> jacorb.poa.thread_pool_max=20
> jacorb.poa.thread_pool_min=5
>
> jacorb.poa.queue_wait=off
> jacorb.poa.queue_max=100
> jacorb.poa.queue_min=10
>
> Please let me know if there are any known issues with Jacorb with respect
> to thread hangs /Deadlocks and way to resolve this Jacorb issue.
>
> Thanks,
> Prakash R
>
>
>
>
> _______________________________________________
> jacorb-developer maillist  -  jacorb-developer at lists.spline.
> inf.fu-berlin.de
> https://lists.spline.inf.fu-berlin.de/mailman/listinfo/jacorb-developer
>



-- 

Phil Mesnier
Principal Engineer & Partner
OCI | WE ARE SOFTWARE ENGINEERS.
tel (314) 579-0066 x225 <(314)%20579-0066>
ObjectComputing.com
<http://objectcomputing.com/>

More information about the jacorb-developer mailing list