[jacorb-developer] Investigating COMM_FAILURE with abnormal closed connection at JacORB 3.5

[Amadeu Andrade Barbosa Junior]

Hi Nick, reply inline.

On 02/04/16 10:18, Nick Cross wrote:
> 
> Hi,
> 
> Reply inline.
> 
> On 18/03/16 19:25, Vinicius (Tec) wrote:
>> Hi everybody,
>> 
>> Sorry for bring this issue back but i'm still trying to figure out a better
>> solution for the bug reported at
>> http://www.jacorb.org/bugzilla/show_bug.cgi?id=1014 than using _duplicate
>> when sharing a Object between multiple threads. The stack error:
>> 
>> Exception id: IDL:omg.org/CORBA/NO_PERMISSION:1.0
>> Operation name: hello2
>> org.omg.CORBA.COMM_FAILURE: IOException: java.net.SocketException: Socket
>> closed  vmcid: 0x0  minor code: 0  completed: No
>>   at
>> org.jacorb.orb.etf.ConnectionBase.to_COMM_FAILURE(ConnectionBase.java:152)
>>   at
>> org.jacorb.orb.iiop.IIOPConnection.handleCommFailure(IIOPConnection.java:79)
>>   at
>> org.jacorb.orb.etf.StreamConnectionBase.flush(StreamConnectionBase.java:227)
>>   at
>> org.jacorb.orb.giop.GIOPConnection.sendMessage(GIOPConnection.java:1094)
>>   at
>> org.jacorb.orb.giop.GIOPConnection.sendRequest(GIOPConnection.java:1020)
>>   at
>> org.jacorb.orb.giop.ClientConnection.sendRequest(ClientConnection.java:309)
>>   at
>> org.jacorb.orb.giop.ClientConnection.sendRequest(ClientConnection.java:290)
>>   at org.jacorb.orb.Delegate._invoke_internal(Delegate.java:1371)
>>   at org.jacorb.orb.Delegate.invoke_internal(Delegate.java:1209)
>>   at org.jacorb.orb.Delegate.invoke(Delegate.java:1197)
>>   at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:475)
>>   at demo._HelloStub.hello(_HelloStub.java:35)
>>   at demo.Client.main(Client.java:58)
>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>   at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>   at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>   at java.lang.reflect.Method.invoke(Method.java:606)
>>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>> 
>> 
>> First, let me explain why using _duplicate, while it is surely a good
>> workaround for general use, is not a good solution for our project. We have
>> a Java SDK based on JacORB that must be used by the applications that need
>> to connect to our server. In order to apply the _duplicate solution
>> throughout the SDK code we'd have to know whether a Object will be shared
>> between multiple threads or not but such knowledge is restrict to the
>> application developer.
>> 
>> So, I'm beginning to analyse how JacORB manages its connection to discover
>> a better solution and, hopefully, implement it.
>> 
>> So far, a colleague of mine found what it seems to be a quick workaround
>> for this issue:
> 
> 
> This is an interesting problem and, as you say, would require some
> careful analysis of how Delegate uses the underlying connections to send
> and receive. There is also various thread local code already in Delegate
> - perhaps this needs to be 'enhanced’.

We’re interested in your (actually all list) thoughts how to enhance that code.

> Delegate may also benefit from using better java concurrency constructs. Another issue is what should
> be done with connection failure/rebind.
> Do you have any standalone test cases that reproduce this issue?

The file attached by Vinicius on bug 1014 reproduces the problem:
http://www.jacorb.org/bugzilla/attachment.cgi?id=467

By standalone do you mean an yet simpler test? No we don’t.

We used that “forward-request-bug.zip" to test our workaround on close semantics of ClientConnection.java.

>> diff --git a/core/src/main/java/org/jacorb/orb/giop/ClientConnection.java
>> b/core/src/main/java/org/jacorb/orb/giop/ClientConnection.java
>> index ab72273..90e7be7 100644
>> --- a/core/src/main/java/org/jacorb/orb/giop/ClientConnection.java
>> +++ b/core/src/main/java/org/jacorb/orb/giop/ClientConnection.java
>> @@ -326,7 +326,8 @@ public class ClientConnection
>> 
>>   public void close()
>>   {
>> -        connection.close();
>> +        if (!connection.hasPendingMessages())
>> +            connection.close();
>>   }
>> 
>> We test at bug 1014 and it seems to fix the problem. However, we did not
>> evaluate yet the side effects of this modification.
> 
> But won't the rest of the disconnect code in Delegate still be called?

Probably no. I didn’t find any other usages of connection.close() for that “connection” reference.
I suppose:
1) the connection reference will be collected by gc OR
2) the underlying connection (tcp or other) will be closed by some other cause (eg. when server close its socket)

I tried to identify some memory waste (eg. if the gc couldn’t collect that reference by some other code) but I got nothing.
I forgot to identify some socket waste (eg. if opened/active sockets are growing fast via netstat in Linux)!! :/ Sorry!

I could save all connections rebinded to try close them after has no pending messages. 
I didn’t try this. I did prefer to wait and to hear from you and the list for better ideas/tips.

Best,
Amadeu A. Barbosa Junior

PS: Vinicius and I work at same team

> Thanks
> 
> Nick
> 
> 
>> We're trying to evaluate the side effects of this modification. It isn't an
>> easy task, though. Delegate depends on ClientConnection and Delegate seems
>> designed to be used only by a single thread at time.
>> We really appreciate any insights about how to evolve Delegate class to a
>> multithread fashion behaviour.
> 
>> Best regards,
>> 
>> Vinicius
>> _______________________________________________
>> jacorb-developer maillist  -  jacorb-developer at lists.spline.inf.fu-berlin.de
>> https://lists.spline.inf.fu-berlin.de/mailman/listinfo/jacorb-developer
>>