[jacorb-developer] Client Application of Jacorb3.0 hangs on IBM WebSphere

[Rungta, Prakash (Prakash)]

Hello Team,

I am developing a Jacorb 3.0 based Application (Client) . However during traffic conditions(Medium) the application(client application) hangs . I see the following errors in the client logs (Jacorb Logging increased) during start and at random times as well.

2017-03-19 16:19:28,843 [ClientMessageReceptor1] jacorb.poa.queue WARN - Request queue is full, consider increasing jacorb.poa.queue_max (currently: 100)
2017-03-19 16:19:28,844 [ClientMessageReceptor1] jacorb.poa.queue WARN - Request queue is full, consider increasing jacorb.poa.queue_max (currently: 100)
2017-03-19 16:19:28,844 [ClientMessageReceptor1] jacorb.poa.queue WARN - Request queue is full, consider increasing jacorb.poa.queue_max (currently: 100)
2017-03-19 16:19:28,844 [ClientMessageReceptor1] jacorb.poa.queue WARN - Request queue is full, consider increasing jacorb.poa.queue_max (currently: 100)

2017-03-19 16:19:28,845 [RequestController-2] jacorb.poa.controller WARN - Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max (currently: 20)
2017-03-19 16:19:28,845 [RequestController-2] jacorb.poa.controller WARN - Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max (currently: 20)
2017-03-19 16:19:28,845 [RequestController-2] jacorb.poa.controller WARN - Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max (currently: 20)
2017-03-19 16:19:28,845 [RequestController-2] jacorb.poa.controller WARN - Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max (currently: 20)
2017-03-19 16:19:28,846 [RequestController-2] jacorb.poa.controller WARN - Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max (currently: 20)
2017-03-19 16:19:28,846 [RequestController-2] jacorb.poa.controller WARN - Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max (currently: 20)
2017-03-19 16:19:28,846 [RequestController-2] jacorb.poa.controller WARN - Thread pool exhausted, consider increasing jacorb.poa.thread_pool_max (currently: 20)

On the Server side , I see the  CORBA TIMEOUT EXCEPTION being raised

        e._info=system exception, ID 'IDL:omg.org/CORBA/TIMEOUT:1.0'

When the client application hangs , I have generated a javacore , I see that there are lot of Thread blocking . Also there are lot of RequestProcessor threads which are in blocking state

3LKWAITNOTIFY            "RequestProcessor-53" (J9VMThread:0x00000000A8742F00)
2LKMONINUSE      sys_mon_t:0x00007F933628F2D0 infl_mon_t: 0x00007F933628F348:
3LKMONOBJECT       org/jacorb/poa/RequestProcessor at 0x000000000B8E9B70: <unowned>
3LKNOTIFYQ            Waiting to be notified:

3XMTHREADINFO      "RequestProcessor-53" J9VMThread:0x00000000A8742F00, j9thread_t:0x00007F933DA77D30, java/lang/Thread:0x0000000006DC92C0, state:CW, prio=10
3XMJAVALTHREAD            (java/lang/Thread getId:0x30FC92, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x7A47, native priority:0xA, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101)
3XMTHREADINFO2            (native stack address range from:0x00007F92B3BF6000, to:0x00007F92B3C37000, size:0x41000)
3XMCPUTIME               CPU usage total: 0.128906816 secs
3XMTHREADBLOCK     Waiting on: org/jacorb/poa/RequestProcessor at 0x0000000006DC92C0 Owned by: <unowned>
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Object.wait(Native Method)
4XESTACKTRACE                at java/lang/Object.wait(Object.java:167(Compiled Code))
4XESTACKTRACE                at org/jacorb/poa/RequestProcessor.run(RequestProcessor.java:793(Compiled Code))
5XESTACKTRACE                   (entered lock: org/jacorb/poa/RequestProcessor at 0x0000000006DC92C0, entry count: 1)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               (0x00007F9388B37912 [libj9prt26.so+0x13912])

Application  overview
Server is a C++ CORBA based and Client is  JAVA (Jacorb 3.0 based idl and runtime is both 3.0).Server does callback's to client application and client does processing.(For simplicity I have commented all the logic inside the client callback functions except some log statements during entering and leaving the functions.) still the issue is there.

The client application  is deployed on RHEL 6.5 (64 bit ) on IBM WebSphere (8.5.5.4) .

Following are some of Jacorb Properties that are set

jacorb.net.server_socket_factory.port.min=52000
jacorb.net.server_socket_factory.port.max=53000
jacorb.security.support_ssl=on

jacorb.security.ssl.client.supported_options=40
jacorb.security.ssl.client.required_options=40

jacorb.security.ssl.server.supported_options=0
jacorb.security.ssl.server.required_options=0

jacorb.security.jsse.server.key_manager_algorithm=IbmX509
jacorb.security.jsse.server.trust_manager_algorithm=IbmX509
jacorb.security.jsse.client.key_manager_algorithm=IbmX509
jacorb.security.jsse.client.trust_manager_algorithm=IbmX509

jacorb.poa.thread_pool_max=20
jacorb.poa.thread_pool_min=5

jacorb.poa.queue_wait=off
jacorb.poa.queue_max=100
jacorb.poa.queue_min=10

Please let me know if there are any known issues with Jacorb with respect to thread hangs /Deadlocks and way to resolve this Jacorb issue.

Thanks,
Prakash R