perfsonar development work

["Vedrin Jeliazkov" <>]

Hi Roman, Joe,

Roman Lapacz 
<>
 wrote:

> Vedrin Jeliazkov wrote:
> > Hi,
> >   
> 
> Hi,
> 
> > I've run some simple tests against the upgraded RRD MA at ESnet
> 
> Do they use eXist to store the metadata configuration file?

I guess no (there's no eXist listening at http://mea1.es.net:8680/exist), but
of course Joe should confirm (it might be listening on some other port and/or
a firewall might be blocking the access to it).

<snip>

> BTW. Vedrin, are you a member of the release team? (I remember the 
> discussion in Cambridge with the decision to ask you to have you there 
> as you are a very active perfSONAR member).

We had a discussion with Nicolas about this in late June and early July. We
agreed that I would be a member of the release management team (as suggested
at the Cambridge meeting) in order to provide advice whether "it is worth
delaying the release management process for feature X, Y or Z", as opposed to
"packaging stuff or going through loads of code", and provided that this
activity doesn't consume too much time ;-)

Regarding the performance and scalability issues I think that the automated
test suite for every perfSONAR service should be extended with some
benchmarking functions like stressing the service with a configurable number
of simultaneous requests (all supported message types), configurable number of
endpoints, and ability to gather processing time/memory consumption
statistics. Ideally, these tests should be maintained and run by service
developers in the first place, while the release management team should re-use
them against the release candidates. Maybe the attached code from TNO could be
used as a starting point for developing such benchmarking utilities.

I would like to use this opportunity to mention that during my second test
against ESnet's RRD MA yesterday evening (launching a "Retrieve all" operation
simultaneously from 10 instances of perfsonarUI) I noticed an exponential
increase in service response time (at that time it was processing 9
simultaneous MetadataKeyRequests and a sequence of SetupDataRequests). The
requests finally timed out and the service stopped responding (perhaps with an
OutOfMemory error of the Tomcat server, after the JVM has started doing
garbage collection most of the time):

http://netmon.acad.bg/smokeping?target=svc.prfs.RRDS.ESNET

Joe, sorry for this trouble - it was not intended. I'm just trying to
illustrate why it is important to benchmark the services... I think it's
reasonable to expect (at least) such loads, as the one I generated yesterday,
in production services.

Best regards,
Vedrin
import javax.xml.rpc.ServiceException;

import gnu.getopt.Getopt;

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.rmi.RemoteException;
import java.util.Vector;

import javax.xml.namespace.QName;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.rpc.soap.SOAPFaultException;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;

import org.apache.axis.client.Call;
import org.apache.axis.client.Service;
import org.apache.axis.message.SOAPBodyElement;

import org.jdom.input.DOMBuilder;
import org.jdom.output.XMLOutputter;

import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.w3c.dom.NamedNodeMap;
import org.xml.sax.SAXException;

/*
 * -s number of ms sleep in between requests
 * -i number of iterations
 * -c number of client threads java 
 * -e service endpoint
 * -f messagefile
 * -h help
 */


public class SOAPLoadClient {
        private class SOAPClientThread extends Thread {
                private int id;
                private int iterations;
                private long timeout;
                private String inputFilename;
                private String serviceEndPointUrl;
                private String xpathTime;
                
                public SOAPClientThread(int id,int iterations,long 
timeout,String serviceEndPointUrl,String inputFilename,String xpathTime)
                {
                        this.id = id;
                        this.iterations = iterations;
                        this.timeout = timeout;
                        this.inputFilename = inputFilename;
                        this.serviceEndPointUrl = serviceEndPointUrl;
                        this.xpathTime = xpathTime;
                }
                
                
                public void run() 
                {
                        Document request = null;
                        Call call = null;

                        try {
                                // set service elements 
                                Service service = new Service();
                                call = (Call)service.createCall();
                                call.setTargetEndpointAddress(new 
URL(this.serviceEndPointUrl));
                                call.setOperationName(new 
QName("http://soapinterop.org/","submit";));
                        } catch ( ServiceException e ) {
                                System.err.println("ServiceException " + 
e.getMessage());
                                return;
                        } catch ( MalformedURLException e ) {
                                System.err.println("MalFormedUrlexception " + 
e.getMessage());
                                return;
                        }

            
            try {
                DocumentBuilderFactory factory = 
DocumentBuilderFactory.newInstance();
                factory.setNamespaceAware(true);            
                DocumentBuilder builder = factory.newDocumentBuilder();
                request = builder.parse(new File(this.inputFilename));
            } catch ( ParserConfigurationException e ) {
                System.err.println("ParserConfigurationException " + 
e.getMessage());
                return;
            } catch ( IOException e ) {
                System.err.println("IOEXception " + e.getMessage());
                return;
            } catch ( SAXException e) {
                System.err.println("SAXException " + e.getMessage());
                return;
            }

            
            
            Node node = null;
            if ( this.xpathTime != null ) {
                XPath xpath = XPathFactory.newInstance().newXPath();
            
                try {
                        // //@timeValue
                        node = (Node) xpath.evaluate("//@timeValue", request, 
XPathConstants.NODE);
                                
                } catch ( XPathExpressionException e) {
                        System.err.println("XPathExpressionException " + 
e.getMessage());
                        return;
                }
                if ( node == null ) {
                        System.err.println("time node not found");
                        return;
                }
            }
            
           
            for(int i=0;(i<this.iterations);i++) {
                long thisTimeout = (long)(Math.random() * this.timeout * 
2.0);                                  
                try {
                        Thread.sleep(thisTimeout);                            
          
                } catch ( InterruptedException e) {
                        System.err.println("InteruptedException" + 
e.getMessage());
                        return;
                }
                
                if ( node != null ) {
                        node.setNodeValue((new 
Long(System.currentTimeMillis() / 1000)).toString());
                                }
                                                
                SOAPBodyElement requestMessage = new 
SOAPBodyElement(request.getDocumentElement());
                
                
                
                long startTime = System.currentTimeMillis();
                        
                boolean ok = false;
                try {
                        Object resultObject = call.invoke(new Object[] 
{requestMessage});
                        ok = true;
                    } catch ( SOAPFaultException sf) {
                        System.err.println("SOAPFaultException:" + 
sf.getMessage());                            
                    } catch ( RemoteException e) {
                        System.err.println("RemoteException:" + 
e.getMessage());
                    }
                    
                    
                        
                    long endTime = System.currentTimeMillis();
                    
                        
                    System.out.println(startTime + ",THREAD" + this.id + ","+ 
(ok ? "OK" : "ERROR") + "," + (endTime-startTime));
                    
                        
                        
                                        
                        }
                }
        }
        
        
        private void usage()
        {
                System.err.println(
                                "Commandline options\n" +
                                
"--------------------------------------------\n" +
                                " -s ms          | average timeout in ms in 
between two requests\n" +
                                " -i number      | number of iterations per 
client thread\n" +
                                " -c number      | number of concurrent 
client threads\n" +
                                " -e url         | service endpoint url\n" +
                                " -f path        | path to message sent in 
each request\n" +
                                " [-x xpath]     | xpath url to time node, 
changed in each request\n" +
                                " -h             | Display this message, and 
exit\n" +
                                
"--------------------------------------------\n");                            
  
                                
        }
        
        public void makeRequests(String[] args)
        {
                int iterations = 0;
                long timeout = 0;
                int numclients = 0;
                String serviceEndPointUrl = null; 
//"http://cluster2gw.telecom.tno.nl:8080/axis/services/MeasurementArchiveService";;
                String messageFilename = null; // 
{SONAR_HOME}/perfsonar/schema/example-instances/sonar/SetupDataRequest-FilterRRDSelect-2.xml";
                String xpathTime = null;
                
                Getopt g = new Getopt("SOAPLoadClient", args, 
"s:i:c:e:f:x:h");
                int c;
                while ((c = g.getopt()) != -1) {
                        switch (c) {
                        case 's':
                                timeout = Long.parseLong(g.getOptarg());
                                break;
                        case 'i':
                                iterations = Integer.parseInt(g.getOptarg());
                                break;
                        case 'c':
                                numclients = Integer.parseInt(g.getOptarg());
                                break;
                        case 'h':
                                usage();
                                return;
                        case 'x':
                                xpathTime = g.getOptarg();
                                break;
                        case 'e':
                                serviceEndPointUrl = g.getOptarg();
                                break;
                        case 'f':
                                messageFilename = g.getOptarg();
                                break;
                        default:
                                usage();
                                return;
                        }
                }
                
                if (( numclients == 0 ) || 
                        ( iterations == 0 ) ||
                        ( serviceEndPointUrl == null) ||
                        (messageFilename == null)) {
                        usage();
                        return;
                }

                SOAPClientThread clients[] = new SOAPClientThread[numclients];
                for(int i=0;(i<numclients);i++) {
                        clients[i] = new 
SOAPClientThread(i,iterations,timeout,serviceEndPointUrl,messageFilename,xpathTime);
                        clients[i].start();
                }               
                        
                
                boolean finished = false;
                while ( ! finished ) {
                        finished = true;
                        for(int i=0;(i<numclients);i++) {
                                if ( clients[i].isAlive()) {
                                        finished = false;
                                }
                        }
                        try {   
                                Thread.sleep(1000);
                        } catch ( Exception e ) {
                                System.err.println("Caught exception while 
waiting for clients to finish");
                                finished = true;
                        }
                }
                
                System.err.println("finished");
        }
        
    public static void main(String[] args) {

        SOAPLoadClient soapLoadClient = new SOAPLoadClient();
        soapLoadClient.makeRequests(args);        
    }
}