Uploaded image for project: 'Axis2'
  1. Axis2
  2. AXIS2-5453

Threads not getting closed at server side on High Load.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 1.5.5, 1.6.2
    • None
    • client-api
    • OS: win2k R2 server
      App server: apache-tomcat-7.0.32-windows-x64
      Axis2 version:- Axis2 1.5.5 at server side AND Axis2 1.6.2 at client side.
      Web service stubs generated using Axis2 1.6.2 jar files with ADB data binding.

    Description

      Hi,
      We are doing a load test on web Services. When we run a test for 25 thread and 25 cycles per thread with each cycle processing a few web service requests, we are seeing that the number of threads for tomcat server is reaching the max limit of 300 and the threads are not closed automatically by client side or server side. The stubs are generated using the ADB databinding. We are using new instance of the stub for different threads. We are sending the requests from a single host.
      Some config on client side include:
      -------------------------------------------------------------------------------------------------------------------------
      MultiThreadedHttpConnectionManager httpConnectionManager = new MultiThreadedHttpConnectionManager();
      HttpClient httpClient = new HttpClient(httpConnectionManager);

      stub._getServiceClient().getOptions().setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Constants.VALUE_TRUE);

      HttpConnectionManagerParams params = new HttpConnectionManagerParams();
      params.setDefaultMaxConnectionsPerHost(100); /* 20 per host is fine in practical scenario */
      params.setMaxTotalConnections(300);
      httpConnectionManager.setParams(params);

      stub._getServiceClient().getOptions().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
      stub._getServiceClient().getOptions().setProperty(HTTPConstants.SO_TIMEOUT, 10000);
      stub._getServiceClient().getOptions().setProperty(HTTPConstants.CONNECTION_TIMEOUT, 10000);
      stub._getServiceClient().getOptions().setProperty(HTTPConstants.CHUNKED, "false");

      // to avoid retries
      HttpMethodParams methodParams = new HttpMethodParams();
      DefaultHttpMethodRetryHandler retryHandler = new DefaultHttpMethodRetryHandler(0, false);
      methodParams.setParameter(HttpMethodParams.RETRY_HANDLER, retryHandler);
      stub._getServiceClient().getOptions().setProperty(HTTPConstants.HTTP_METHOD_PARAMS, methodParams);
      -------------------------------------------------------------------------------------------------------------------------

      After running the threads I am calling:
      -----------------------------------------------------------------------------
      stub._getServiceClient().cleanupTransport();
      stub._getServiceClient().cleanup();
      stub.cleanup();
      httpConnectionManager.closeIdleConnections(0);
      httpConnectionManager.deleteClosedConnections();
      httpConnectionManager.shutdown();
      stub = null;
      httpConnectionManager = null;
      -----------------------------------------------------------------------------

      After each web service call I am calling:
      -----------------------------------------------------------------
      stub._getServiceClient().cleanupTransport();
      -----------------------------------------------------------------

      When I shutdown the tomcat, I am getting 'severe' errors in the Tomcat log "catalina.2012-11-12.log". I have attached the log. All the threads are in "Waiting" state. I am not able to close the threads and server gets hanged and it cannot process any new requests.

      Regards,
      Aravind

      Attachments

        1. catalina.2012-11-12.log
          411 kB
          AravindPS
        2. tomcat_thdump.tgz
          5 kB
          cesare pietra

        Activity

          People

            Unassigned Unassigned
            arvindps24 AravindPS
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: