[jacorb-developer] Client connection threads are never released

[Stout, Eric A.]

I just downloaded the JacORB 3.2 source, and compared the relevant part (org.jacorb.orb.giop.ClientGIOPConnection) to JacORB 2.3.  It hasn't changed.
    /**
     * Client-side implementation what to do when the underlying transport
     * is closed during a read operation.  We mark the transport as closed
     * and allow it to be reopened later, when the client retries.
     */
    protected void streamClosed()
    {
        if (logger.isDebugEnabled())
        {
            logger.debug (this.toString() + ": streamClosed()");
        }

        closeAllowReopen();

        if( connection_listener != null )
        {
            connection_listener.streamClosed();
        }
    }

    /**
     * Closes the underlying transport, but keeps this ClientGIOPConnection
     * alive.  If, subsequently, another request is sent to this connection,
     * it will try to reopen the transport.
     */
    public void closeAllowReopen()
    {
        if (logger.isDebugEnabled())
        {
            logger.debug (this.toString() + ": closeAllowReopen()");
        }

        try
        {
            //Solve potential deadlock caused by COMM_FAILURE.
            //The strategy is getting write_lock before sync
            //connect_sync when you need both of them.
            getWriteLock(0);
            synchronized (connect_sync)
            {
                transport.close();
                // We expect that the same transport can be reconnected
                // after a close, something that the ETF draft isn't
                // particularly clear about.
            }
        }
        finally
        {
            releaseWriteLock();
        }
    }

This leaves the client connection thread hanging around.  In contrast, the ServerGIOPConnection does this:

    /**
     * Server-side implementation what to do if the underlying transport
     * gets closed during a read operation. Since we're server-side and
     * can't reopen, we simply close completely.
     */
    protected void streamClosed()
    {
        if (logger.isDebugEnabled())
        {
            logger.debug (this.toString() + ": streamClosed()");
        }
        close();
    }

    /**
     * @see GIOPConnection#close()
     */
    public void close()
    {
        super.close();
        if( manager != null )
        {
            manager.unregisterServerGIOPConnection( this );
        }
    }

That's the behavior I want on the client side, but I don't see a way to get there from here.  (That's not the only pathway by which a connection gets closed, but the result is the same: ServerGIOPConnection overrides close() to remove the connection from the manager, but ClientGIOPConnection does not.

Eric

-----Original Message-----
From: Nick Cross [mailto:jacorb at goots.org]
Sent: Tuesday, February 12, 2013 1:25 AM
To: Discussions concerning CORBA development with JacORB
Cc: Stout, Eric A.
Subject: Re: [jacorb-developer] Client connection threads are never released


Hi,

Please retest using the current version of JacORB (3.2).

Thanks

Nick

On 11/02/13 23:38, Stout, Eric A. wrote:
> We have a mixed-platform, mixed-language, mixed-ORB system of about 1500 processes.  A few Java JacORB processes are clients to all, or virtually all, of the processes in the system.  I have set jacorb.connection.client.max_receptor_threads=2500, so the system starts up fine, and runs for a few days.  But processes are occasionally stopped and restarted, causing the all-process clients to make new connections and consume more threads.  Eventually, I end up with an error like:
>
> 2013/02/09 12:35:14.085 E_LOG Sys_Mgr_Server|main||@iccsfwprod0002 [Process Monitor: iccsprod0001, ISP_Shot_Director] ProcessMonitorTask: Ping received unexpected exception for process: [nif/iccsprod0001/ISP_Shot_Director]
>                  org.omg.CORBA.NO_RESOURCES: (Pool)[0/2500] no idle threads but maximum number of threads reached (2500)  vmcid: 0x0  minor code: 0  completed: No
>                  Stack trace: org.omg.CORBA.NO_RESOURCES: (Pool)[0/2500] no idle threads but maximum number of threads reached (2500)  vmcid: 0x0  minor code: 0  completed: No
>                  at org.jacorb.util.threadpool.ThreadPool.putJob(ThreadPool.java:183)
>                  at org.jacorb.orb.giop.MessageReceptorPool.connectionCreated(MessageReceptorPool.java:84)
>                  at org.jacorb.orb.giop.ClientConnectionManager.getConnection(ClientConnectionManager.java:121)
>                  at org.jacorb.orb.Delegate.bind(Delegate.java:335)
>                  at org.jacorb.orb.Delegate.bind(Delegate.java:290)
>                  at org.jacorb.orb.Delegate.getObjectKey(Delegate.java:852)
>                  at org.jacorb.orb.ORB.findPOA(ORB.java:510)
>                  at org.jacorb.orb.Delegate.resolvePOA(Delegate.java:883)
>                  at org.jacorb.orb.Delegate.is_really_local(Delegate.java:1504)
>                  at org.jacorb.orb.Delegate.is_local(Delegate.java:1493)
>                  at org.omg.CORBA.portable.ObjectImpl._is_local(ObjectImpl.java:372)
>                  at iccs.base.corba.ConnectionHandler._is_local(ConnectionHandler.java:77)
>                  at iccs.idl.Idl_Sys_Mgr.Process._AgentStub.ping(_AgentStub.java:33)
>                  at iccs.services.system_manager.server.ProcessMonitorTask.run(ProcessMonitorTask.java:94)
>
>  From looking at the JacORB code, it looks like client threads are never released; if the server goes away, the thread just waits around for it to maybe come back later.  But our processes always come up on a different port, and our JacORB processes come up with a unique implname, for reasons that I'm not going to go into.  So, we just slowly leak client threads.
>
> We are using JacORB 2.3.1.
>
> Is there a way to stop these threads from leaking?
>
> Thanks,
> Eric
> _______________________________________________
> jacorb-developer maillist  -  jacorb-developer at lists.spline.inf.fu-berlin.de
> https://lists.spline.inf.fu-berlin.de/mailman/listinfo/jacorb-developer
>