"Jarvis, sometimes you've to run before you can walk", said my favorite action super-hero last year. Unfortunately, the same is not true for most of us poor non-armored coders developing whatever software we've at hand. For us "whatever you don't test... will fail" holds true. Admittedly, the sentence lacks any glamour and it's more the murphy type of sentence than the super-heroe type.

Stress testing Plastic

No, I won't be talking about putting any kind of hardware about some kind of high physical pressure (as the folk after the first sentence in the article would be doing in his lab), but describing what we do to check how our version control system behaves under heavy load, and how we work to improve it's performance.

For the last month and a half I've been involved in stress testing Plastic SCM. We didn't only check how it behaves under heavy load, but also managed to increase its performance, making it up to 5 times faster under certain circumstances. We also used the test environment to benchmark against some well-known competitors.

The machines

How we did that? Well, first of all we needed a big number of machines. Ok, the meaning of big can probably vary depending on how wealthy your company is, but for us counting on about 100 test machines was enough.

We didn't have this amount of available machines at the office, so we contacted our local university, and we set up a collaboration to be able to use 100 boxes, all perfectly configured to start up remotely, with a very nicely set up network file system, and running Linux. The boxes belong to different labs, so they don't share the same specs (although each lab has the same configuration, they've different hardware on different labs), but they sum more than 750K bogomips together (I just hope Sarah Connor is not around :-P).

 

 

Then we set up a server, an Intel 64 bits QuadCore processor (not Itanium!) with 4GB RAM.

A test scenario

The goal of our load test is to simulate what would happen to a Plastic server when used concurrently by a high number of developers. So what each of the clients is going to do is to mimic the expected behavior of a developer:

• Create a branch
• Modify files
• Commit the changes to the branch
• Repeat

So the basic test follows the previous steps as depicted on the next figure:

 

Each of the client machines will run one of the tests described below, so we can emulate 100 concurrent users against the same server. In fact, we can run more than one of the tests on each machine, considerably increasing the number.

There's something important to note and it is the following: since there're will be no delays between each of the actions described, the bot client will be much faster than a real user. I mean, a real developer won't be able to create a branch, and make 50 changes in a few seconds, so with 100 machines we'll be actually simulating a much bigger load than what 100 human developers could ever achieve. In fact the load would better match what 10 times this number could accomplish.

PNUnit as distributed testing framework

Because we simulate Plastic users, we make extensive use of the cm command line tool included with the system. Each of the test steps above can be run with a cm command.

 

So one way to run the tests would be just gluing the previous commands with some scripting language and execute them remotely via ssh or something similar.

But then introducing things such as assertions and the like would be a little bit more complicated (well, you can always make use of the unit test facilities of some script-like language like ruby, but I'm not familiar enough with it so I stick to .NET).

Starting with NUnit 2.5 PNUnit, which stands for parallel nunit, is included with the framework, so writing these kind of tests is easier than ever. We wrote PNUnit back in January 2006 as the core of our internal testing system, in order to synchronize the launch of Plastic servers and clients on different machines and also being able to write test scenarios in C# as we were doing for regular unit tests. I wrote an article about it on DDJ long ago.

So, how would the previous sample look like when written in PNUnit?

Look at the following fully commented code listing for details:

 

using System;

using System.IO;

using System.Collections;

using System.Diagnostics;

using System.Threading;

 

using NUnit.Framework;

using PNUnit.Framework;

 

using Codice.Test;

 

namespace cmtest.BotTesting

{

    [TestFixture]

    public class DeveloperBot

    {

        [Test]

        public void Run()

        {

            CmdRunner.NoGetProcessTime = true;

            TestHelper.RunBotTest(

                "DeveloperBotSimplified",

                new ExecuteTestDelegate(DoDeveloperBotSimplified));

        }

 

        private void DoDeveloperBotSimplified(

            string testName,

            string repservername,

            string wkservername,

            string wkbasepath)

        {

            string wkpath = null;

            wkbasepath = Path.GetFullPath(wkbasepath);

 

            string repname = PNUnitServices.Get().GetTestParams()[2];

            long iterations = long.Parse(

                PNUnitServices.Get().GetTestParams()[3]);

 

            // We tune whether we want to run

            // parallel updates or not with the test name

            // A param would be better :-(

            bool bNoParallel = false;

            if (testName.ToLower().StartsWith("nop"))

                bNoParallel = true;

 

            try

            {

                // cm mkwk already handled by our internal test

                // library

                wkpath = TestHelper.CreateWorkspace(

                    testName + "wk", wkservername, wkbasepath);

 

                // and set slelector is also a

                // common action to do

                TestHelper.SetSelector(

                    new SelectorTest(

                        SelectorTypes.SELECTOR,

                        new string[] {repname}),

                    wkpath, wkservername);

 

                string paramparallel =

                    bNoParallel ? "--noparallel" : string.Empty;

 

                // CmdRunner is our class to

                // run commands

 

                // initial wk update

                CmdRunner.ExecuteCommand(

                    string.Format(

                        "cm update . {0} --silent -wks={1}",

                        paramparallel, wkservername),

                    wkpath);

 

                // Run the bot

                long cycle = 0;

 

                while( cycle++ < iterations )

                {

                    // Use PNUnitServices WriteLine to

                    // access the output since it will be

                    // handled by the framework and no

                    // direct access to the Console is

                    // normally allowed

                    PNUnitServices.Get().WriteLine(

                        string.Format("{0} running cycle {1}",

                        testName, cycle));

 

                    // create a branch

                    string brName = string.Format(

                        "task-{0}-{1}", testName, cycle);

                    string brFullName = string.Format(

                        "br:/main/{0}", brName);

 

                    CmdRunner.ExecuteCommand(

                        string.Format("cm mkbr {0} -wks={1}",

                            brFullName, wkservername),

                        wkpath);

 

                    int ini = Environment.TickCount;

 

                    // and switch to it

                    TestHelper.SetSelector(

                        new SelectorTest(

                            SelectorTypes.SELECTOR_CHILD,

                            new string[] {repname, brName}),

                        wkpath, wkservername);

 

                    CmdRunner.ExecuteCommand(

                        string.Format(

                            "cm update . {0} --silent -wks={1}",

                            paramparallel, wkservername),

                        wkpath);

 

                    PNUnitServices.Get().WriteLine(

                        string.Format("Wk updated {0} ms",

                            Environment.TickCount - ini));

 

                    ArrayList files = new ArrayList();

                    ArrayList dirs = new ArrayList();

 

                    FillContent(wkpath, ref files, ref dirs);

 

                    Assert.IsTrue(

                        files.Count > 0,

                        "Should find a non empty wk");

 

                    // number of files to modify

                    // at the beginning we used a

                    // random number to better emulate

                    // users, but now we stick to a

                    // fixed number to create repeatable

                    // tests when looking for performance

                    // boosts

                    int numFilesToModify = 10;

 

                    PNUnitServices.Get().WriteLine(

                        string.Format("Going to modify {0} files",

                            numFilesToModify));

 

                    Hashtable filesgenerated = new Hashtable();

 

                    for( int i = 0; i < numFilesToModify; ++i )

                    {

                        int number = rnd.Next(0, files.Count -1);

 

                        while( filesgenerated.Contains(number) )

                            number = rnd.Next(0, files.Count -1);

 

                        filesgenerated.Add(number, number);

 

                        string file = files[number] as string;

 

                        long numChanges = 3;

 

                        for( int j = 1; j <= numChanges; ++j )

                        {

                            CmdRunner.ExecuteCommand(

                                string.Format(

                                    "cm co \"{0}\" -wks={1}",

                                    file, wkservername),

                                wkpath);

 

                            FSHelper.AppendToFile(

                                file,

                                "added by " + testName);

 

                            if( j < numChanges )

                            {

                                CmdRunner.ExecuteCommand(

                                    string.Format(

                                        "cm ci \"{0}\" -wks={1}",

                                        file, wkservername),

                                    wkpath);

                            }

                        }

                    }

                    filesgenerated.Clear();

 

                    // find all the check outs and check them in

                    // in a "bigger" checkout

                    CheckInAllCOs(wkservername, wkpath);

                }

            }

            finally

            {

                // clean up the workspace

           

                if( wkpath != null )

                    FSHelper.DeleteDirectory(wkpath);

            }

        }

 

        private void CheckInAllCOs(string wkservername, string wkpath)

        {

            string cmdres = CmdRunner.ExecuteCommandWithStringResult(

                string.Format(

                    "cm findco --format={{4}} -wks={0}",

                    wkservername), wkpath);

 

            ArrayList cos = TestHelper.GetListResults(cmdres);

 

            string toci = string.Empty;

            foreach( string co in cos )

            {

                toci = toci + " \"" + co + "\"";

            }

 

            CmdRunner.ExecuteCommand(

                string.Format("cm ci {0} -wks={1}",

                    toci, wkservername),

                wkpath);

        }

 

        private void FillContent(

            string basepath,

            ref ArrayList files,

            ref ArrayList dirs)

        {

            files.AddRange(Directory.GetFiles(basepath));

 

            string[] localdirs = Directory.GetDirectories(basepath);

 

            dirs.AddRange(localdirs);

 

            foreach( string dir in localdirs )

                FillContent(dir, ref files, ref dirs);

        }

 

    }

}

What's next?

Well, there's still lots of work to do, we need to create a PNUnit XML file to tell the framework what to run and where, and I'll also be talking a little bit about our experience gathering results and debugging excessive memory use. So, stay tuned.