Adventures in Load Testing

Using R to Plot Universal Scaling Law Curve Automagically

2010-01-28T19:31:00.005-06:00

The past two days I was doing some benchmarking of a system that might be used to stub back end systems and I was very interested in generating an USL curve based upon the benchmark data I was producing.

In previous posts I had shown how to use R to generate the kappa and sigma coefficients for use with the USL equation.

I start with the data that I've compiled on the system based upon the number of concurrent threads and the throughput measured:

   1:  p,x

   2:  1,1

   3:  3,2.998108449

   4:  5,4.995428752

   5:  7,6.928278689

   6:  10,7.539249685

   7:  13,11.50488651

   8:  15,11.79476671

   9:  20,15.65936318

  10:  30,13.61349306

  11:  40,16.50567465

  12:  50,14.83716898

The number of threads is represented by "p" and the throughput is represented by "x"

I wrote the following R function to automagically create a plot of the data points for the above file and generate the USL curve along the data points and highlight the maximum theoretical point for the USL:

   1:  # Example usage:

   2:  # plotUSL("c:/benchmark/benchmark.csv", "Benchmark data with USL curve")

   3:  # CSV file must have two columns with a header of "p, x"

   4:  # Example:

   5:  #     p, x

   6:  #    1, 1

   7:  #    2, 1.5

   8:  #    3, 2

9:

  10:  plotUSL <- function(dataFile, graphTitle) {

  11:    uslData <- read.csv(dataFile, header=TRUE);

  12:    uslData$c <- uslData$x / uslData$x[1];

  13:    usl <- nls(c ~ p/(1+sigma*(p-1)+kappa*p*(p-1)),

  14:               uslData,

  15:               algorithm="port",

  16:               start=c(sigma=0.0, kappa=0.0),

  17:               lower=c(0,0));

  18:    sigma <- coef(usl)["sigma"];

  19:    kappa <- coef(usl)["kappa"];

  20:    p <- 1:round(1.75 * max(uslData$p));

  21:    Relative_Capacity <- p/(1+sigma*(p-1)+kappa*p*(p-1));

  22:    plot(p, Relative_Capacity, type="l", ylim=c(0, round(max(Relative_Capacity, uslData$c)*1.1)));

  23:    points(uslData$p, uslData$c, pch=20);

24:

  25:    indexValue <- 1;

  26:    testValue  <- Relative_Capacity[1];

  27:    maxValue   <- max(Relative_Capacity);

28:

  29:    while(testValue != maxValue) {

  30:      indexValue = indexValue + 1;

  31:      testValue = Relative_Capacity[indexValue];

  32:    };

33:

  34:    points(p[indexValue], Relative_Capacity[indexValue], col=2, pch=13, cex=2.0);

35:

  36:    title(main=graphTitle, col.main="black", font.main=4, cex=1.5);

  37:    title(sub=sprintf("USL max at p = %d with Relative_Capacity = %f",

  38:          indexValue,

  39:          Relative_Capacity[indexValue]),

  40:          col.main="black",

  41:          font.main=4,

  42:          cex=1.5);

  43:  };

And viola, we can quick and dirty results after invoking the function!

> plotUSL("g:/usl/benchmark.csv", "Benchmark data with USL curve")

This function could easily be modified to accept a third parameter and save off the image to a JPG, PNG or even PDF. But I leave that as an exercise for the reader.

11001001

2009-12-20T22:33:00.013-06:00

What a strange trip it has been.

Two weeks ago I got a call from an IT Mercenary outfit and my former employer wanted to bring me in for a face-to-face. This isn't the first time that my former employer and I have sniffed out each other but this is the first time that something came of it.

It appears that on 28 Dec I will be going back to my former employer doing the exact same gig (for now) as a contractor via the above mentioned IT Mercenary firm. It's not a bad rate and it is W2 with a benefits option. The only loss is vacation/sick time. However, if I work 60 hours in a week I get paid for 60 hours. I'll have to wait and see how that goes.

So, I've been away for roughly 19 months. What have I really done in that time?

Agile Project Management

Does Agile work? Yes and no from what I've seen first hand. It can work if management allows it to work. I've also seen Agile as an abstraction for sales. I worked for a firm that shouted Agile from the rooftops but did not allow it's own projects to run in an agile fashion. Agile was squashed for tyrannical project management with tacit upper management approval, developers opinions and experience be damned.

Alternative Load Testing Frameworks

I got a lot of good work in with Microsoft's load testing tool. Microsoft really needs to have a better naming scheme for it's testing tools. When you say, "LoadRunner" you don't have to explain it too much. Most people know that LoadRunner is for performance testing. Sure, they might not know how it works, but they know it has something to do with performance testing. The Microsoft solution? Visual Studio 2008 Team System Test Edition. That's a tadbit verbose. Sure, you know it's Visual Studio and that it is for the Team Edition and hey, you know it's with the Testing Edition. But still, hardly anybody really knows what the hell it does or how it does it. Visual Studio 2008 Team System Test Edition Coded Web Test is just too verbose. Try doing some job searches on indeed.com for "Visual Studio 2008 Team System Test Edition Coded Web Test" and see how many hits you get versus the generic "LoadRunner."

I also got to mess around a lot with JMeter. I found JMeter to be handy little tool. I felt a little constrained by the GUI nature of JMeter but still, it gets the job done and has more protocols available than the Microsoft Tool. I preferred the Microsoft tool as I could develop virtual users in C# versus JMeter use of GUI elements and beanshell coding for extraction and correlation.

While not a performance testing tool, I did get to play with QTP a bit. I'd like to get to know QTP better. It seems to be a real handy tool. I'm also excited by the UI testing tool that will be part of VS2010, as well.

Deployment Tools

I got to mess around with tools like NAnt, MSBuild and WiX. Of the three I prefer NAnt but generally the XML files get very tiresome. I'm still a perl kind of guy at heart but of the three I prefer NAnt. I got to extend NAnt with objects created for MSBuild via a wrapper which was nifty and solved a problem.

Project Planning

For a project that I worked on for the Veterans Administration I gutted and re-wrote the performance engineering plan. The end document was over 90 pages of generic performance testing documentation.

Capacity Analysis Tools

Again for the Veterans Administration I did a comparison of different capacity analysis tools and techniques and the way of combining the efforts of performance testing and capacity analysis and trending. I found the books by Neil Gunther to be most excellent along with the PDQ API. While TeamQuest Model is popular I found the interface to be extremely clunky. It reminded me of VB4/Powerbuilder era interfaces.

In the past I've done plenty of empirical capacity analysis based upon my load testing results and observations of production systems but this was the first time that I learned the math behind heuristic queuing analysis and the various tools related to queuing theory.

Of all the things that I learned on my little adventure I think that this ranks near the top in terms of importance and application to future endeavors. It really is quite handy and goes hand-in-hand with performance testing. For my last position I recommended the Python versis of PDQ, PyDQ. I selected PyDQ because I like the clean duck typing syntax of Python coupled with the available IDEs for Windows and Linux.

The Joys of Virtualization

I used Hyper-V and VirtualBox on a lot of projects. Virtualization just plain rocks. Prototyping complete systems is just incredibly cool. I found the snapshot system of Hyper-V to be better performing than VirtualBox but VirtualBox is much more flexible as VirtualBox is not bound to Windows 2008. When I was testing deployment scripts with NAnt, Hyper-V was invaluable. Test, check results, fix, rollback and test again. Fabulous! Both Hyper-V R2 and VirtualBox 3.1 support live motion of VMs which is pretty cool. I really hope that Oracle doesn't kill off VirtualBox. VirtualBox just rocks.

I used VirtualBox to build a JMS prototype system with multiple clients processing messages. For another learning experience I had built an OpenSolaris server exporting an iSCSI target. I learned that ZFS is one of the greatest file systems that I have ever used. Really cool stuff.

Python

For some of the capacity analysis tools I needed to learn Python and I found that I liked what I found. I've done some Python development in the past (early 2000s) but never did follow up as I used perl for what I needed. Python led to learning Django and my desire to further learn MVC with different frameworks. I also played around a lot with different Python development platforms like Eclipse with PyDev and NetBeans with support for Python. Pretty darn cool.

Java

For the VA project I did a bit of JMS and Swing development in Java. I've never done any Java development and I started to learn Java development. I'm now continuing learning Java in continuing education courses at the local community college. While I may not be utilizing Java right now it's a good thing to learn. I would have liked to have done more JMS development but alas, that was put on the back burner when my project was coming to an end in favor of the Swing development. I've never been much of a GUI developer and still don't have any major desire to do front ends. In my heart I'm a command line kinda guy.

I covered metrics collection and tuning of J2EE apps but I've only truly done J2SE development. The VA project was supposed to expose me to WebLogic but unfortunately that never came to fruition. I've already forgotten all the Oracle training that I crammed into my brain the first few weeks of being on the project.

R

I had the opportunity to do some R development to aid in number crunching and I found R to be very handy. If I do a lot of statistical number crunching I'm more likely to do it in R than I am do write perl routines to get the same results. I especially like the built in graphing capabilities of R both in quality and ability to generate jpg and PDF results. I started out learning R as part of PDQ-R but ultimately for the project I was working on I selected PyDQ, the Python implementation of PDQ for the robustness and the widely available IDEs for Python both under Windows and Linux.

Skill Development

In the November time frame I had a job interview and while I did good on the CS portion of the interview (math, algorithms analysis, etc) I bombed the actual coding portion. The coding portion of the interview was using C# to solve a problem using interfaces. I've implemented interfaces in the past but this solution was using interfaces in a way that I had never used them. That told me that it was time to start developing my coding skill set. I've been taking some Java classes at the local community college and have plans to continue with the advanced Java class along with Hibernate and Spring. Also covered will be Ruby on Rails.

Working from Home

I worked from home for 11 months and I've decided that working from home sucks. I am not a homebody and get cabin fever easily. Face to face communication is definitely better than phone conference meetings with 15 people. I found Skype to be real handy for voice conferencing with 1-800 numbers. Working from home wasn't good for my waist line either. I'm looking to getting back into a regular work routine and making the use of the facility gym with my old trainer and utilizing the running track again once I've dropped a few pounds. Sure, I got up to a 405 pound machine bench press (useful for throwing developers over cube walls) but I've let the old BF% get up too high. Bah!

Working on Government Projects

I've worked on government projects before, specifically USAF and DoD stuff but this was the first time that I worked on a VA project. Oy! What a mess. Remember the messed up logistics systems from when you were in the US Army and twist the entropy knob to eleven. Yeah, it was really that bad. The sad thing was that the project I was working on was supposed to help those veterans that put their lives on the line for their country and this was the best that could be done? I shudder to think of Guvment Health Care being ran like the project that I was on.

Books

Here is a short list of books that I found to be handy:

Analyzing Computer System Performance with Perl::PDQ
Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services
Performance by Design: Computer Capacity Planning By Example
The Art of Capacity Planning: Scaling Web Resources
The Art of Application Performance Testing: Help for Programmers and Quality Assurance
Pro Java EE 5 Performance Management and Optimization

What's next? Who knows? Can't wait to find out.

Time to map out all the soup kitchens.

2009-11-30T16:29:00.002-06:00

Well, I finally packed up my work laptop and sent it back to my employer. As of tomorrow I will be unemployed. But I do have some phone interviews setup so that isn't too bad for a December. Need to file for unemployment tomorrow.

VirtualBox is amazing

2009-11-05T18:07:00.003-06:00

So I went to the Microsoft event yesterday and got a Win 7 Ultimate license.

A few years back I had done the same and got a Vista Ultimate key and got a copy of the Vista Ultimate 64-bit ISO. When I tried to use the key that was distributed with the 32-bit DVD from the event it didn't work. As a student I was able to get a discount license for Vista Ultimate and for some reason, the key that I got from the Microsoft event worked with the ISO that I had purchased. Weird.

I thought maybe that would happen again as the handout DVD was only for the 32-bit Win 7 Ultimate. I got a copy of a Win 7 Ultimate 64-bit ISO and decided that I would test out the install and see if the key would work correctly.

My main rig, an i7 920 machine with 12 gigs of RAM was busy performing a backup so I decided to fire up VirtualBox on my now aging Acer Ferrari 5000 machine and did a VM install of Win 7 Ultimate 64-bit to a VM and verified that the key worked just fine (yay!).

It wasn't until this morning that I realized that I had installed Win 7 Ultimate 64-bit virtual machine on a laptop running Vista 32-bit. And it worked.

Wow.

That just rocks.

I'm dreaming of a laid off Chrismas...

2009-10-09T20:01:00.005-05:00

Got a call. Looks like the program that I am on is not going to be funded past mid-November. I suspect that if laid off in mid-November two weeks before Thanksgiving and the Holiday season not much is going to happen job wise, just like last year.

At least this time I have some heads up warning and can plan accordingly unlike my previous stint at VT (may JC and RZ rot in the lower bowels of Hell). So, I don't think it'll be so bad this time around.

During this hiatus I think I'll give The Grinder a closer look as I haven't had a chance to use it yet on anything and I've been doing a lot more Python coding as of late (PyDQ and SimPy stuff). An updated Grinder plug-in has been created for Eclipse and hopefully it works out better than my previous attempt at using Grinderstone.

Good times!

Look, everybody! It's a SimPy three tier eBiz simulation! But now with extra statistics! Wowee!

2009-08-31T12:23:00.004-05:00

Added some more statistics to the output such as waiting queue length and component utilization by using SimPy Monitors. I couldn't easily use the monitors for the page response times due to the split up nature calls to multiple nodes but for .waitQ and node utilizations they worked out perfectly.


#!/usr/bin/env python

from SimPy.Simulation import *
from random import Random, expovariate, uniform

class Metrics:

  metrics = dict()

  def Add(self, metricName, frameNumber, value):
    if self.metrics.has_key(metricName):
      if self.metrics[metricName].has_key(frameNumber):
        self.metrics[metricName][frameNumber].append(value)
      else:
        self.metrics[metricName][frameNumber] = list()
        self.metrics[metricName][frameNumber].append(value)
    else:
      self.metrics[metricName] = dict()
      self.metrics[metricName][frameNumber] = list()
      self.metrics[metricName][frameNumber].append(value)

  def Keys(self):
    return self.metrics.keys()

  def Mean(self, metricName):
    valueArray = list()
    if self.metrics.has_key(metricName):
      for frame in self.metrics[metricName].keys():
        for values in range(len(self.metrics[metricName][frame])):
          valueArray.append(self.metrics[metricName][frame][values])

      sum = 0.0
      for i in range(len(valueArray)):
        sum += valueArray[i]
          
      if len(self.metrics[metricName][frame]) != 0:      
        return sum/len(self.metrics[metricName])
      else:
        return 0 # Need to learn python throwing exceptions
    else:
      return 0

class G: 

  numWS = 1
  numAS = 1
  numDS = 1

  Rnd         = random.Random(12345)

  PageNames   = ["Entry", "Home", "Search", "View", "Login", "Create", "Bid", "Exit" ]

  Entry       = 0
  Home        = 1
  Search      = 2
  View        = 3
  Login       = 4
  Create      = 5
  Bid         = 6
  Exit        = 7

  WS          = 0
  AS          = 1
  DS          = 2

  CPU         = 0
  DISK        = 1

  WS_CPU      = 0
  WS_DISK     = 1
  AS_CPU      = 2
  AS_DISK     = 3
  DS_CPU      = 4
  DS_DISK     = 5

  metrics     = Metrics()

  #             e  h  s  v  l  c  b  e
  HitCount   = [0, 0, 0, 0, 0, 0, 0, 0]   

  Resources = [[ Resource(1), Resource(1) ], # WS CPU and DISK
               [ Resource(1), Resource(1) ], # AS CPU and DISK
               [ Resource(1), Resource(1) ]] # DS CPU and DISK

  QMon      = [[ Monitor(1), Monitor(1)], # WS CPU and DISK
               [ Monitor(1), Monitor(1)], # AS CPU and DISK
               [ Monitor(1), Monitor(1)]] # DS CPU and DISK

  SMon      = [[ Monitor(1), Monitor(1)], # WS CPU and DISK
               [ Monitor(1), Monitor(1)], # AS CPU and DISK
               [ Monitor(1), Monitor(1)]] # DS CPU and DISK

  #                  Enter  Home   Search View   Login  Create Bid    Exit
  ServiceDemand = [ [0.000, 0.008, 0.009, 0.011, 0.060, 0.012, 0.015, 0.000],  # WS_CPU
                    [0.000, 0.030, 0.010, 0.010, 0.010, 0.010, 0.010, 0.000],  # WS_DISK
                    [0.000, 0.000, 0.030, 0.035, 0.025, 0.045, 0.040, 0.000],  # AS_CPU
                    [0.000, 0.000, 0.008, 0.080, 0.009, 0.011, 0.012, 0.000],  # AS_DISK
                    [0.000, 0.000, 0.010, 0.009, 0.015, 0.070, 0.045, 0.000],  # DS_CPU
                    [0.000, 0.000, 0.035, 0.018, 0.050, 0.080, 0.090, 0.000] ] # DS_DISK

  # Type B shopper
  #                     0     1     2     3     4     5     6     7
  TransitionMatrix = [ [0.00, 1.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],  # 0
                       [0.00, 0.00, 0.70, 0.00, 0.10, 0.00, 0.00, 0.20],  # 1
                       [0.00, 0.00, 0.45, 0.15, 0.10, 0.00, 0.00, 0.30],  # 2
                       [0.00, 0.00, 0.00, 0.00, 0.40, 0.00, 0.00, 0.60],  # 3
                       [0.00, 0.00, 0.00, 0.00, 0.00, 0.30, 0.55, 0.15],  # 4
                       [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 1.00],  # 5
                       [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 1.00],  # 6
                       [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00] ] # 7

class RandomPath:

  def RowSum(self, Vector):
    rowSum = 0.0
    for i in range(len(Vector)):
      rowSum += Vector[i]
    return rowSum

  def NextPage(self, T, i):
    rowSum = self.RowSum(T[i])
    randomValue = G.Rnd.uniform(0, rowSum)

    sumT = 0.0
    
    for j in range(len(T[i])):
      sumT += T[i][j]
      if randomValue < sumT:
        break
    return j

class ExecuteFrame(Process):

  def __init__(self, frameNumber, resource, QMon, SMon, serviceDemand, nodeName, pageName):
    Process.__init__(self)
    self.frame         = frameNumber 
    self.resource      = resource
    self.serviceDemand = serviceDemand
    self.nodeName      = nodeName
    self.pageName      = pageName
    self.QMon          = QMon
    self.SMon          = SMon

  def execute(self):
    StartUpTime = now()

    yield request, self, self.resource
    yield hold,    self, self.serviceDemand

    self.QMon.observe(len(self.resource.waitQ))
    self.SMon.observe(self.serviceDemand)

    yield release, self, self.resource

    R = now() -  StartUpTime

    G.metrics.Add(self.pageName, self.frame, R)

class CallPage(Process):

  def __init__(self, frameNumber, node, pageName):
    Process.__init__(self)
    self.frame          = frameNumber
    self.StartUpTime    = 0.0
    self.currentPage    = node
    self.pageName       = pageName

  def execute(self):

    if self.currentPage != G.Exit:

      print >> sys.stderr, "Working on Frame # ", self.frame, " @ ", now() , " for page ", self.pageName

      self.StartUpTime = now()

      if G.ServiceDemand[G.WS_CPU][self.currentPage] > 0.0:
        wsCPU = ExecuteFrame(self.frame,                                           \
                             G.Resources[G.WS][G.CPU],                             \
                             G.QMon[G.WS][G.CPU],                                  \
                             G.SMon[G.WS][G.CPU],                                  \
                             G.ServiceDemand[G.WS_CPU][self.currentPage]/G.numWS,  \
                             "wsCPU", self.pageName) 
        activate(wsCPU, wsCPU.execute())

      if G.ServiceDemand[G.WS_DISK][self.currentPage] > 0.0:
        wsDISK = ExecuteFrame(self.frame,                                           \
                              G.Resources[G.WS][G.DISK],                            \
                              G.QMon[G.WS][G.DISK],                                 \
                              G.SMon[G.WS][G.DISK],                                 \
                              G.ServiceDemand[G.WS_DISK][self.currentPage]/G.numWS, \
                              "wsDISK", self.pageName)
        activate(wsDISK, wsDISK.execute())
 
      if G.ServiceDemand[G.AS_CPU][self.currentPage] > 0.0:
        asCPU = ExecuteFrame(self.frame,                                            \
                             G.Resources[G.AS][G.CPU],                              \
                             G.QMon[G.AS][G.CPU],                                   \
                             G.SMon[G.AS][G.CPU],                                   \
                             G.ServiceDemand[G.AS_CPU][self.currentPage]/G.numAS,   \
                             "asCPU",                                               \
                             self.pageName)
        activate(asCPU, asCPU.execute())

      if G.ServiceDemand[G.AS_DISK][self.currentPage] > 0.0:
        asDISK = ExecuteFrame(self.frame,                                           \
                              G.Resources[G.AS][G.DISK],                            \
                              G.QMon[G.AS][G.DISK],                                 \
                              G.SMon[G.AS][G.DISK],                                 \
                              G.ServiceDemand[G.AS_DISK][self.currentPage]/G.numAS, \
                              "asDISK",                                             \
                              self.pageName)
        activate(asDISK, asDISK.execute())

      if G.ServiceDemand[G.DS_CPU][self.currentPage] > 0.0:
        dsCPU = ExecuteFrame(self.frame,                                            \
                             G.Resources[G.DS][G.CPU],                              \
                             G.QMon[G.DS][G.CPU],                                   \
                             G.SMon[G.DS][G.CPU],                                   \
                             G.ServiceDemand[G.DS_CPU][self.currentPage]/G.numDS,   \
                             "dsCPU",                                               \
                             self.pageName)
        activate(dsCPU, dsCPU.execute())

      if G.ServiceDemand[G.DS_DISK][self.currentPage] > 0.0:
        dsDISK = ExecuteFrame(self.frame,                                           \
                              G.Resources[G.DS][G.DISK],                            \
                              G.QMon[G.DS][G.DISK],                                 \
                              G.SMon[G.DS][G.DISK],                                 \
                              G.ServiceDemand[G.DS_DISK][self.currentPage]/G.numDS, \
                              "dsDISK",                                             \
                              self.pageName)
        activate(dsDISK, dsDISK.execute())

      G.HitCount[self.currentPage] += 1

      yield hold, self, 0.00001 

class Generator(Process):
 def __init__(self, rate, maxT):
     Process.__init__(self)
     self.name = "Generator"
     self.rate = rate
     self.maxT = maxT
     self.g = Random(11335577)
     self.i = 0
     self.currentPage = G.Home

 def execute(self):
   while (now() < self.maxT): 
     self.i+=1
     p = CallPage(self.i,self.currentPage,G.PageNames[self.currentPage])
     activate(p,p.execute())
     yield hold,self,self.g.expovariate(self.rate)
     randomPath = RandomPath()

     if self.currentPage == G.Exit:
       self.currentPage = G.Home
     else:
       self.currentPage = randomPath.NextPage(G.TransitionMatrix, self.currentPage)

def main():

    Lambda      = 4.026*float(sys.argv[1])
    maxSimTime  = float(sys.argv[2])

    initialize()
    g = Generator(Lambda, maxSimTime)
    activate(g,g.execute())

    simulate(until=maxSimTime)

    print >> sys.stderr, "Simulated Seconds    : ", maxSimTime

    print >> sys.stderr, "Page Hits            :"
    for i in range(len(G.PageNames)):
      print >> sys.stderr, "\t", G.PageNames[i], " = ", G.HitCount[i]

    print >> sys.stderr, "Throughput           : "
    for i in range(len(G.PageNames)):
      print >> sys.stderr, "\t", G.PageNames[i], " = ", G.HitCount[i]/maxSimTime

    print >> sys.stderr, "Mean Response Times:"

    for i in G.metrics.Keys():
      print >> sys.stderr, "\t", i, " = ", G.metrics.Mean(i)

    print >> sys.stderr, "Component Waiting Queues:"
    print >> sys.stderr, "\tWeb Server CPU          : ", G.QMon[G.WS][G.CPU].mean()
    print >> sys.stderr, "\tWeb Server DISK         : ", G.QMon[G.WS][G.DISK].mean()
    print >> sys.stderr, "\tApplication Server CPU  : ", G.QMon[G.AS][G.CPU].mean()
    print >> sys.stderr, "\tApplication Server DISK : ", G.QMon[G.AS][G.DISK].mean()
    print >> sys.stderr, "\tDatabase Server CPU     : ", G.QMon[G.DS][G.CPU].mean()
    print >> sys.stderr, "\tDatabase Server DISK    : ", G.QMon[G.DS][G.DISK].mean()

    print >> sys.stderr, "Component Utilization:"
    print >> sys.stderr, "\tWeb Server CPU          : ", ((G.SMon[G.WS][G.CPU].mean()*len(G.QMon[G.WS][G.CPU]))/maxSimTime)*100
    print >> sys.stderr, "\tWeb Server DISK         : ", ((G.SMon[G.WS][G.DISK].mean()*len(G.QMon[G.WS][G.DISK]))/maxSimTime)*100
    print >> sys.stderr, "\tApplication Server CPU  : ", ((G.SMon[G.AS][G.CPU].mean()*len(G.QMon[G.AS][G.CPU]))/maxSimTime)*100
    print >> sys.stderr, "\tApplication Server DISK : ", ((G.SMon[G.AS][G.DISK].mean()*len(G.QMon[G.AS][G.DISK]))/maxSimTime)*100
    print >> sys.stderr, "\tDatabase Server CPU     : ", ((G.SMon[G.DS][G.CPU].mean()*len(G.QMon[G.DS][G.CPU]))/maxSimTime)*100
    print >> sys.stderr, "\tDatabase Server DISK    : ", ((G.SMon[G.DS][G.DISK].mean()*len(G.QMon[G.DS][G.DISK]))/maxSimTime)*100

    print >> sys.stderr, "Total Component Hits      :" 
    print >> sys.stderr, "\tWeb Server CPU          : ", len(G.QMon[G.WS][G.CPU])
    print >> sys.stderr, "\tWeb Server DISK         : ", len(G.QMon[G.WS][G.DISK])
    print >> sys.stderr, "\tApplication Server CPU  : ", len(G.QMon[G.AS][G.CPU])
    print >> sys.stderr, "\tApplication Server DISK : ", len(G.QMon[G.AS][G.DISK])
    print >> sys.stderr, "\tDatabase Server CPU     : ", len(G.QMon[G.DS][G.CPU])
    print >> sys.stderr, "\tDatabase Server DISK    : ", len(G.QMon[G.DS][G.DISK])

    print >> sys.stderr, "Total Component Thoughput :" 
    print >> sys.stderr, "\tWeb Server CPU          : ", len(G.QMon[G.WS][G.CPU])/maxSimTime
    print >> sys.stderr, "\tWeb Server DISK         : ", len(G.QMon[G.WS][G.DISK])/maxSimTime
    print >> sys.stderr, "\tApplication Server CPU  : ", len(G.QMon[G.AS][G.CPU])/maxSimTime
    print >> sys.stderr, "\tApplication Server DISK : ", len(G.QMon[G.AS][G.DISK])/maxSimTime
    print >> sys.stderr, "\tDatabase Server CPU     : ", len(G.QMon[G.DS][G.CPU])/maxSimTime
    print >> sys.stderr, "\tDatabase Server DISK    : ", len(G.QMon[G.DS][G.DISK])/maxSimTime

    print >> sys.stderr, "Mean Component Svc Demand :"
    print >> sys.stderr, "\tWeb Server CPU          : ", G.SMon[G.WS][G.CPU].mean()
    print >> sys.stderr, "\tWeb Server DISK         : ", G.SMon[G.WS][G.DISK].mean()
    print >> sys.stderr, "\tApplication Server CPU  : ", G.SMon[G.AS][G.CPU].mean()
    print >> sys.stderr, "\tApplication Server DISK : ", G.SMon[G.AS][G.DISK].mean()
    print >> sys.stderr, "\tDatabase Server CPU     : ", G.SMon[G.DS][G.CPU].mean()
    print >> sys.stderr, "\tDatabase Server DISK    : ", G.SMon[G.DS][G.DISK].mean()

    print G.HitCount[G.Home]/maxSimTime, ",",  G.metrics.Mean("Home"), ",",  G.metrics.Mean("View"), ",",  G.metrics.Mean("Search"), ",",  G.metrics.Mean("Login"), ",",  G.metrics.Mean("Create"), ",",  G.metrics.Mean("Bid")
 
if __name__ == '__main__': 
  main()

Improved single queue SimPy code

2009-08-30T00:10:00.003-05:00

In this blog entry I had a comparison of PDQ and SimPy for a single queue. The code that I wrote was pretty cheezy and based upon a very early example from Norm Matloff's Introduction to Discrete-Event Simulation and the SimPy Language.

After writing the SimPy code to solve for the three tier eBiz solution I wanted to go back and correct some of my initial code with the single queue.

Below is what I feel is better code than my original single queue solution.


#!/usr/bin/env python

from SimPy.Simulation import *
from random import Random, expovariate, uniform
import time

class G:
  # Rnd         = random.Random(time.mktime(time.localtime()))
  Rnd            = random.Random(12345)
  MyQueue        = Resource(1)
  QMon           = Monitor()
  ServiceTime    = 0.50 
  TotalCalls     = 0L
  TotalResidence = 0L 
  TotalWait      = 0L
  TotalService   = 0L

class WorkLoad(Process):

  def __init__(self):
    Process.__init__(self)
    self.StartUpTime    = 0.0

  def Run(self):
    self.StartUpTime = now()
    yield request, self, G.MyQueue                     
    G.TotalWait  += now() - self.StartUpTime
    yield hold, self, G.ServiceTime
    G.QMon.observe(len(G.MyQueue.waitQ));
    G.TotalResidence += now() - self.StartUpTime
    yield release, self, G.MyQueue
    G.TotalCalls += 1

class Generator(Process):

  def __init__(self, Lambda):
    Process.__init__(self)
    self.Lambda = Lambda

  def execute(self):

    while 1:
      yield hold, self, G.Rnd.expovariate(self.Lambda)
      W = WorkLoad()
      activate(W, W.Run())

def main():

    Lambda      = float(sys.argv[1])
    MaxSimTime = 10000.00

    G.TotalCalls       = 0L
    G.TotalResidence   = 0L
    G.TotalWait        = 0L
    G.TotalService     = 0L

    initialize()

    print >> sys.stderr, "MaxSimTime  = ", MaxSimTime

    g = Generator(Lambda)
    activate(g, g.execute())

    simulate(until=MaxSimTime)

    print Lambda, ",", MaxSimTime, ",", G.TotalCalls, ",", G.TotalCalls/MaxSimTime, ",", G.TotalWait, ",", G.TotalResidence, ",", G.TotalResidence/G.TotalCalls, ",", G.TotalWait/G.TotalCalls, ",", (G.TotalResidence/G.TotalCalls) - (G.TotalWait/G.TotalCalls) , ",", ((G.ServiceTime * G.TotalCalls) / MaxSimTime) * 100 , "%",",",G.QMon.mean()
 
    print >> sys.stderr, "Ideal Throughput     : ", Lambda
    print >> sys.stderr, "Simulated Seconds    : ", MaxSimTime
    print >> sys.stderr, "Number of calls      : ", G.TotalCalls
    print >> sys.stderr, "Throughput           : ", G.TotalCalls/MaxSimTime
    print >> sys.stderr, "Total Wait Time      : ", G.TotalWait
    print >> sys.stderr, "Total Residence Time : ", G.TotalResidence
    print >> sys.stderr, "Mean Residence Time  : ", G.TotalResidence/G.TotalCalls
    print >> sys.stderr, "Mean Wait Time       : ", G.TotalWait/G.TotalCalls
    print >> sys.stderr, "Mean Service Time    : ", (G.TotalResidence/G.TotalCalls) - (G.TotalWait/G.TotalCalls) 
    print >> sys.stderr, "Total Utilization    : ", ((G.ServiceTime * G.TotalCalls) / MaxSimTime) * 100 , " %"
    print >> sys.stderr, "Mean WaitQ           : ", G.QMon.mean()

if __name__ == '__main__': main()

The results are the same as before, but now instead of creating a gajillion objects to activate only one "thread" is created that in turns makes the calls to the queue. Much faster than my original code and laid out better.

Hey, everybody! Let's use SimPy to simulate performance of a three tier eBiz solution!

2009-08-28T18:55:00.007-05:00

Back in this blog post, I wrote about using PDQ-R to analyze the performance of a three tier eBiz solution.

I had been getting into learning how to simulate queuing networks with SimPy and this week I got my three tier SimPy solution up and running and crunching numbers.

The SimPy solution was MUCH more involved than the PDQ solution and it took me a bit to wrap my head around all the goodies I needed to know before I started getting good results.

I started by looking at the jackson.py code and got an idea of how write the code that was needed to get the job done.

Like the jackson.py code, I also had three computers but unlike the jackson.py code I had two nodes per computer that represented the CPU and Disk. Get that setup was the real tricky part and I had to bang my wall against the SimPy brick wall for a while until I broke through and figured the solution out.

Like the jackson.py code, I use a Process to control the rate that I hit web pages by invoking another process that sets up the hits to the individual components that are also processes. One of the big lessons that I learned is that if the service demand for a component is zero, don't make a yield request call for the component resource.

You'd think that I would have figured that out earlier but it totally slipped past me and it was really kicking my buttocks. Most of the page response times were looking good but in the PDQ analysis, the HomePage didn't have a long response time due to there only being service demand on the Web CPU and Disk and I wasn't seeing that in my SimPy solution.

Here is what I was seeing in my SimPy solution for page response time for the Home page:

I had some other chores to do and as I was walking past my machine I noticed out of the corner of my eye some code and it hit me like a ton of bricks. I was making a yield request for each component even if there was no service demand for the component and that yield request was a blocking operation. Well, duh! A few if statements fixed that problem and a quick test showed that the Home Page response time was no diverging from the PDQ analysis like in the above graph!

Wow, what a difference!

The biggest things that I got out of doing the SimPy solution was that all component requests need to be in their own Processes and if there is no service demand for the component then it should be called at all.

Perhaps next week I'll add the queuing output and component utilization into my SimPy code for more left/right comparisons against PDQ. But over all, I'm fairly happy with my solution but there is some code that needs to be cleaned up.

Down below is my code and the comparisons of my SimPy solution versus my PDQ solution.


#!/usr/bin/env python

from SimPy.Simulation import *
from random import Random, expovariate, uniform

class Metrics:

  metrics = dict()

  def Add(self, metricName, frameNumber, value):
    if self.metrics.has_key(metricName):
      if self.metrics[metricName].has_key(frameNumber):
        self.metrics[metricName][frameNumber].append(value)
      else:
        self.metrics[metricName][frameNumber] = list()
        self.metrics[metricName][frameNumber].append(value)
    else:
      self.metrics[metricName] = dict()
      self.metrics[metricName][frameNumber] = list()
      self.metrics[metricName][frameNumber].append(value)

  def Keys(self):
    return self.metrics.keys()

  def Mean(self, metricName):
    valueArray = list()
    if self.metrics.has_key(metricName):
      for frame in self.metrics[metricName].keys():
        for values in range(len(self.metrics[metricName][frame])):
          valueArray.append(self.metrics[metricName][frame][values])

      sum = 0.0
      for i in range(len(valueArray)):
        sum += valueArray[i]
          
      if len(self.metrics[metricName][frame]) != 0:      
        return sum/len(self.metrics[metricName])
      else:
        return 0 # Need to learn python throwing exceptions
    else:
      return 0

class RandomPath:

  def RowSum(self, Vector):
    rowSum = 0.0
    for i in range(len(Vector)):
      rowSum += Vector[i]
    return rowSum

  def NextPage(self, T, i):
    rowSum = self.RowSum(T[i])
    randomValue = G.Rnd.uniform(0, rowSum)

    sumT = 0.0
    
    for j in range(len(T[i])):
      sumT += T[i][j]
      if randomValue < sumT:
        break

    return j

class G: 

  numWS = 1
  numAS = 1
  numDS = 2

  Rnd         = random.Random(12345)

  PageNames   = ["Entry", "Home", "Search", "View", "Login", "Create", "Bid", "Exit" ]

  Entry       = 0
  Home        = 1
  Search      = 2
  View        = 3
  Login       = 4
  Create      = 5
  Bid         = 6
  Exit        = 7

  WS          = 0
  AS          = 1
  DS          = 2

  CPU         = 0
  DISK        = 1

  WS_CPU      = 0
  WS_DISK     = 1
  AS_CPU      = 2
  AS_DISK     = 3
  DS_CPU      = 4
  DS_DISK     = 5

  metrics     = Metrics()

  #             e  h  s  v  l  c  b  e
  HitCount   = [0, 0, 0, 0, 0, 0, 0, 0]   

  Resources = [[ Resource(1), Resource(1) ], # WS CPU and DISK
               [ Resource(1), Resource(1) ], # AS CPU and DISK
               [ Resource(1), Resource(1) ]] # DS CPU and DISK

  #                  Enter  Home   Search View   Login  Create Bid    Exit
  ServiceDemand = [ [0.000, 0.008, 0.009, 0.011, 0.060, 0.012, 0.015, 0.000],  # WS_CPU
                    [0.000, 0.030, 0.010, 0.010, 0.010, 0.010, 0.010, 0.000],  # WS_DISK
                    [0.000, 0.000, 0.030, 0.035, 0.025, 0.045, 0.040, 0.000],  # AS_CPU
                    [0.000, 0.000, 0.008, 0.080, 0.009, 0.011, 0.012, 0.000],  # AS_DISK
                    [0.000, 0.000, 0.010, 0.009, 0.015, 0.070, 0.045, 0.000],  # DS_CPU
                    [0.000, 0.000, 0.035, 0.018, 0.050, 0.080, 0.090, 0.000] ] # DS_DISK

  # Type B shopper
  #                     0     1     2     3     4     5     6     7
  TransitionMatrix = [ [0.00, 1.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],  # 0
                       [0.00, 0.00, 0.70, 0.00, 0.10, 0.00, 0.00, 0.20],  # 1
                       [0.00, 0.00, 0.45, 0.15, 0.10, 0.00, 0.00, 0.30],  # 2
                       [0.00, 0.00, 0.00, 0.00, 0.40, 0.00, 0.00, 0.60],  # 3
                       [0.00, 0.00, 0.00, 0.00, 0.00, 0.30, 0.55, 0.15],  # 4
                       [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 1.00],  # 5
                       [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 1.00],  # 6
                       [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00] ] # 7

class DoWork(Process):

  def __init__(self, i, resource, serviceDemand, nodeName, pageName):
    Process.__init__(self)
    self.frame         = i
    self.resource      = resource
    self.serviceDemand = serviceDemand
    self.nodeName      = nodeName
    self.pageName      = pageName

  def execute(self):
    StartUpTime = now()
    yield request, self, self.resource
    yield hold, self, self.serviceDemand
    yield release, self, self.resource
    R = now() -  StartUpTime

    G.metrics.Add(self.pageName, self.frame, R)

class CallPage(Process):

  def __init__(self, i, node, pageName):
    Process.__init__(self)
    self.frame          = i
    self.StartUpTime    = 0.0
    self.currentPage    = node
    self.pageName       = pageName

  def execute(self):

    if self.currentPage != G.Exit:

      print >> sys.stderr, "Working on Frame # ", self.frame, " @ ", now() , " for page ", self.pageName

      self.StartUpTime = now()

      if G.ServiceDemand[G.WS_CPU][self.currentPage] > 0.0:
        wsCPU = DoWork(self.frame, G.Resources[G.WS][G.CPU],  G.ServiceDemand[G.WS_CPU][self.currentPage]/G.numWS, "wsCPU", self.pageName) 
        activate(wsCPU,  wsCPU.execute())

      if G.ServiceDemand[G.WS_DISK][self.currentPage] > 0.0:
        wsDISK = DoWork(self.frame, G.Resources[G.WS][G.DISK], G.ServiceDemand[G.WS_DISK][self.currentPage]/G.numWS, "wsDISK", self.pageName)
        activate(wsDISK, wsDISK.execute())
 
      if G.ServiceDemand[G.AS_CPU][self.currentPage] > 0.0:
        asCPU = DoWork(self.frame, G.Resources[G.AS][G.CPU],  G.ServiceDemand[G.AS_CPU][self.currentPage]/G.numAS, "asCPU", self.pageName)
        activate(asCPU,  asCPU.execute())

      if G.ServiceDemand[G.AS_DISK][self.currentPage] > 0.0:
        asDISK = DoWork(self.frame, G.Resources[G.AS][G.DISK], G.ServiceDemand[G.AS_DISK][self.currentPage]/G.numAS, "asDISK", self.pageName)
        activate(asDISK, asDISK.execute())

      if G.ServiceDemand[G.DS_CPU][self.currentPage] > 0.0:
        dsCPU = DoWork(self.frame, G.Resources[G.DS][G.CPU],  G.ServiceDemand[G.DS_CPU][self.currentPage]/G.numDS, "dsCPU", self.pageName)
        activate(dsCPU,  dsCPU.execute())

      if G.ServiceDemand[G.DS_DISK][self.currentPage] > 0.0:
        dsDISK = DoWork(self.frame, G.Resources[G.DS][G.DISK], G.ServiceDemand[G.DS_DISK][self.currentPage]/G.numDS, "dsDISK", self.pageName)
        activate(dsDISK, dsDISK.execute())

      G.HitCount[self.currentPage] += 1

      yield hold, self, 0.00001 # Needed to prevent an error. Doesn't add any blocking to the six queues above


class Generator(Process):
 def __init__(self, rate, maxT, maxN):
     Process.__init__(self)
     self.name = "Generator"
     self.rate = rate
     self.maxN = maxN
     self.maxT = maxT
     self.g = Random(11335577)
     self.i = 0
     self.currentPage = G.Home

 def execute(self):
   while (now() < self.maxT): 
     self.i+=1
     p = CallPage(self.i,self.currentPage,G.PageNames[self.currentPage])
     activate(p,p.execute())
     yield hold,self,self.g.expovariate(self.rate)
     randomPath = RandomPath()

     if self.currentPage == G.Exit:
       self.currentPage = G.Home
     else:
       self.currentPage = randomPath.NextPage(G.TransitionMatrix, self.currentPage)

def main():

    maxWorkLoad = 10000
    Lambda      = 4.026*float(sys.argv[1])
    maxSimTime  = float(sys.argv[2])

    initialize()
    g = Generator(Lambda, maxSimTime, maxWorkLoad)
    activate(g,g.execute())

    simulate(until=maxSimTime)

    print >> sys.stderr, "Simulated Seconds    : ", maxSimTime

    print >> sys.stderr, "Page Hits            :"
    for i in range(len(G.PageNames)):
      print >> sys.stderr, "\t", G.PageNames[i], " = ", G.HitCount[i]

    print >> sys.stderr, "Throughput           : "
    for i in range(len(G.PageNames)):
      print >> sys.stderr, "\t", G.PageNames[i], " = ", G.HitCount[i]/maxSimTime

    print >> sys.stderr, "Mean Response Times:"

    for i in G.metrics.Keys():
      print >> sys.stderr, "\t", i, " = ", G.metrics.Mean(i)

    print G.HitCount[G.Home]/maxSimTime, ",",  G.metrics.Mean("Home"), ",",  G.metrics.Mean("View"), ",",  G.metrics.Mean("Search"), ",",  G.metrics.Mean("Login"), ",",  G.metrics.Mean("Create"), ",",  G.metrics.Mean("Bid")
 
if __name__ == '__main__': main()

Like my previous comparison of SimPy versus PDQ for a single queue, the PDQ results are slightly higher, so no major surprises there.

Hey, everybody! Let's do a quick comparison of SimPy and PDQ for a single queue!

2009-08-24T13:44:00.007-05:00

On the side I have slowly been looking into SimPy for queue simulation and I finally got around to doing a comparison of a single queue written up with SimPy and with PDQ.

The scenario is very simple. A single queue with a service time of 0.5 seconds with one visit for a service demand total of 0.5 seconds.

The PDQ solution is very simple to write up and only took a few minutes:


library("pdq");

lambda <- seq(1, 20, 1);
lambda <- lambda / 10;

for (rate in lambda) {
  Init("Single Queue");

  CreateOpen("WorkLoad", rate);

  CreateNode("MyQueue",   CEN, FCFS);

  SetDemand("MyQueue",  "WorkLoad" , 0.5);

  SetWUnit("Trans");
  SetTUnit("Second");

  Solve(CANON);
  #Report();
  print(sprintf("%f, %f, %f", 
                rate, 
                GetResidenceTime("MyQueue","WorkLoad", TRANS), 
                GetUtilization("MyQueue", "WorkLoad", TRANS)));
};

The SimPy solution took a bit longer for me to write up and to make sure I was doing things correctly. I thought it was neat that I was able to setup a lambda into the queue that was greater than the queue was capable of processing, in this case, the lambda max is 2 requests per second as is given by the reciprocal of 0.5 seconds, 1/0.5 == 2.

PDQ gives a response time of "Inf" for 2.0 requests per second where as the SimPy solution will crunch away, but never get above 2 requests per second and will result with unbounded queueing as we'd expect.

I still need to work on my SimPy solution to add in more metrics collection, but it's a start. After I wrap my brain around metrics collection I'm going to write a solution of one of the previous problems that I worked with related to Jackson networks and the three tier E-biz solution.

One of the interesting things about SimPy is that I had to create a large number of worker "threads" to hit the queue over a long period of time to get to the correct lambda that I had specified. If I only created one worker "thread" there was no queueing taking place and wait time was always zero, as you'd normally expect. I played around a lot with the number of worker "threads" and found that for my open network that between 10,000 to 100,000 workers worked out just fine and that 10,000 seconds was an adequate time frame for the solution.

One of the first things I had to handle with the large number of worker "threads" was to control the inter arrival rate so that the requests per second of all the workers would be the correct value. After that, it was just playing around with the code to find a good number of simulation time and workers.


#!/usr/bin/env python

from SimPy.Simulation import *
from random import Random, expovariate, uniform
import time

class G:
  Rnd         = random.Random(12345)
  MyQueue     = Resource(1)
  QMon        = Monitor()
  ServiceTime = 0.50

class WorkLoad(Process):

  TotalCalls       = 0L
  TotalResidence   = 0L 
  TotalWait        = 0L
  TotalService     = 0L

  def __init__(self):
    Process.__init__(self) 
    self.StartUpTime    = 0.0

  def Run(self, MaxWorkLoad, Lambda):
    InterArrivalRate = (1.0/Lambda)
    while 1:
      WaitTime = G.Rnd.expovariate(1.0/(InterArrivalRate*MaxWorkLoad))  
      yield hold, self, WaitTime
      self.StartUpTime = now()
      yield request, self, G.MyQueue                     
      WorkLoad.TotalWait  += now() - self.StartUpTime      # W
      yield hold, self, G.ServiceTime                      # S
      G.QMon.observe(len(G.MyQueue.waitQ));
      WorkLoad.TotalResidence += now() - self.StartUpTime  # R = W + S
      WorkLoad.TotalCalls += 1
      yield release, self, G.MyQueue

def main():

    MaxWorkLoad = 1000
    Lambda      = float(sys.argv[1])
    MaxSimTime = 1000.00

    WorkLoad.TotalCalls       = 0L
    WorkLoad.TotalResidence   = 0L
    WorkLoad.TotalWait        = 0L
    WorkLoad.TotalService     = 0L

    initialize()

    print "MaxWorkLoad = ", MaxWorkLoad
    print "MaxSimTime  = ", MaxSimTime

    for I in range(MaxWorkLoad):
      W = WorkLoad()
      activate(W, W.Run(MaxWorkLoad, Lambda))

    TotalWorkLoad = I + 1

    simulate(until=MaxSimTime)

    print "Ideal Throughput     : ", Lambda
    print "Simulated Seconds    : ", MaxSimTime
    print "TotalWorkLoads       : ", TotalWorkLoad
    print "Number of calls      : ", WorkLoad.TotalCalls
    print "Throughput           : ", WorkLoad.TotalCalls/MaxSimTime
    print "Total Wait Time      : ", WorkLoad.TotalWait
    print "Total Residence Time : ", WorkLoad.TotalResidence
    print "Mean Residence Time  : ", WorkLoad.TotalResidence/WorkLoad.TotalCalls
    print "Mean Wait Time       : ", WorkLoad.TotalWait/WorkLoad.TotalCalls
    print "Mean Service Time    : ", (WorkLoad.TotalResidence/WorkLoad.TotalCalls) - 
                                     (WorkLoad.TotalWait/WorkLoad.TotalCalls) 
    print "Total Utilization    : ", ((G.ServiceTime * WorkLoad.TotalCalls) / MaxSimTime) * 100 , " %"
    print "Mean waitQ           : ", G.QMon.mean()

if __name__ == '__main__': main()

I still have a lot of learning to do with SimPy, that's for sure. It's been almost a decade since I've messed with Python and I am really rusty. Time to take care of that.

Here is the comparison of metrics for utilization and response time for the single queue with both solutions in handy, dandy line graph form:

Utilization:

No surprises here as both lines are almost the same.

Response Time:

This is the big surprise for me. As utilization increases the response times between SimPy and PDQ start to diverge. At low utilizations there almost the same but look at around 30% utilization (0.6 requests per sec) and the divergence starts. I knew that there would be some differences between SimPy and PDQ but I didn't expect this much of a difference. I need to go over the SimPy solution and make sure I didn't fat finger anything as this is my first SimPy solution for a queue.

Queue Waiting:

Like the Response Time graph, we show that PDQ is gonkulating more queuing as a result of higher expected response times than the SimPy model.

While PDQ will respond with "Inf" for 2.0 requests per second and error out with anything greater than 2 req/sec the SimPy solution response time is asymptotic to 2 requests per second. I can push the SimPy solution is 1.9999 requests per second but never hit exactly 2.

Here is an example of the SimPy solution with 2.1 requests per second:


auswipe@auswipe-desktop:~/Desktop/pbe/simpy$ ./q.py 2.1
MaxWorkLoad =  10000
MaxSimTime  =  10000.0
Ideal Throughput     :  2.1
Simulated Seconds    :  10000.0
TotalWorkLoads       :  10000
Number of calls      :  19976
Throughput           :  1.9976
Total Wait Time      :  1996958.71394
Total Residence Time :  2006728.44111
Mean Residence Time  :  100.45697042
Mean Wait Time       :  99.9678971738
Mean Service Time    :  0.489073246233
Total Utilization    :  99.88  %
Mean WaitQ           :  18.6269640142

I've still got a long way to go with SimPy but I thought this was a decent first attempt with a simple problem.

Manual model building TeamQuest Model gotcha

2009-08-08T11:27:00.012-05:00

A while back I started to play around with Perl::PDQ and decided in order to help me learn the R language that I'd just stick with PDQ-R for now. At work I don't have a system yet in place where I can perform measurements and comparisons of actual response times versus PDQ-R results.

I did the best next thing for me, which was to compare the results in a case study. As with some of my previous blog entries I was using case studies from Performance By Design.

For my first attempt of PDQ-R coding I was using the baseline from Chapter 5: Case Study I: A Database Service. This chapter is pretty nifty because it covers the topic of cluster analysis for workload analysis with three classes of workload. That led me to learning the basics of cluster analysis with R.

Here was my solution for the baseline found in 5.4 of Performance By Design.


library("pdq");

# Total req/sec into the open queue model
lambda_into_system <- 1.33;

# Split up the req/sec into each class based upon
# previously supplied ratios

coeff <- c(0.33/1.33, 0.53/1.33, 0.47/1.33);

lambda <- coeff * lambda_into_system;

#                 Class1,Class2,Class3
cpu_demand   <- c(0.096, 0.615, 0.193);
disk1_demand <- c(0.088, 0.683, 0.763);
disk2_demand <- c(0.119, 0.795, 0.400);

workStreamName <- 1:3;


Init("Chapter 5.4");

for (n in 1:3) {
  workStreamName[n] <- sprintf("class_%d", n);
  CreateOpen(workStreamName[n], lambda[n]);
};

CreateNode("CPU",   CEN, FCFS);
CreateNode("Disk1", CEN, FCFS);
CreateNode("Disk2", CEN, FCFS);

for (n in 1:3) {
  SetDemand("CPU",   workStreamName[n], cpu_demand[n]  );
  SetDemand("Disk1", workStreamName[n], disk1_demand[n]);
  SetDemand("Disk2", workStreamName[n], disk2_demand[n]);
};

SetWUnit("Trans");
SetTUnit("Second");

Solve(CANON);
#Report();

response <- 1:3;

for (n in 1:3) {
  response[n] <- GetResponse(TRANS, workStreamName[n]       );
};

for (n in 1:3) {
  print(sprintf("  PDQ-R computation shows response time for class %d is %f seconds", n, response[n]));
};

Which yields the results of:


> source('baseline.R')
[1] "  PDQ-R computation shows response time for class 1 is 0.864179 seconds"
[1] "  PDQ-R computation shows response time for class 2 is 6.105397 seconds"
[1] "  PDQ-R computation shows response time for class 3 is 4.535833 seconds"

These values match the results in the book and the available Excel spreadsheet the authors developed in support of the book.

After I coded my PDQ-R solution I started to look into TeamQuest Model for capacity analysis and decided to do the same case study and see if the numbers agreed between all three sources.

I developed my TeamQuest Model and setup the visits and service time based upon the computed service demand for all three classes of workload:

However, the results did not match at all!

I contacted the TeamQuest folks with what I had done and showed my initial work both with PDQ-R and computations by hand and figured that I was doing something wrong with TeamQuest Model. I wanted to know what was up with the difference in values.

TeamQuest tech support finally got back to me with the solution. It appears that TeamQuest Model expects the service time to be set to 0.001 seconds and the number of visits modified to meet the service demand desired. I must've overlooked that in the TeamQuest Model tutorial but sure enough, it works. After sending some e-mails back and forth with TeamQuest it appears that the 0.001 second service time limit is only for CPUs. I am told that it is OK to use the actual visits and service times with AR IO's, but I haven't tested it out yet.

Results with:

Apparently if the TeamQuest agent is used to automatically create models based upon hardware and system utilization this is automatically set and is a non-issue. It's only when building a model from scratch from the ground up that it shows up. It's an easy fix but an unexpected issue.

Hey, everybody! Let's analyze the performance of a three tier system with PDQ-R!

2009-08-02T21:37:00.010-05:00

This post is a continuation of this post that I did earlier this week.

Continuing the solution for Case Study IV: An E-Business Service from the book, Performance by Design: Computer Capacity Planning By Example.

In the previous post I discuss how to take the transition probability matrix and work backwards toward the original series of linear equations for solving the number of visits to a series of web pages. In this case study there are actually two types of visitors that result in two transition probability matrices that must be utilized. There are 25% of Type A visitors and 75% of Type B visitors.

Each tier of the hypothetical e-biz service is made up by a single CPU and a single disk drive. A matrix is supplied with the total service demand for each component by each page that is hit by visitors.

While it is simple to write some code to analyze web logs to generate the transition probability matrix based upon customer traffic it is very difficult to isolate the total demand at each component with chaotic customer traffic. But that is why we have load testing tools that are available to us. In a pseudo-production environment we are capable of simulating customer traffic to one page at a time and calculating the total demand for components. In this particular case only the CPU and disk drives are being modeled but for a real service we'd want to model the CPU, disk drives, memory system, network system, etc.

After running simulated customer traffic against isolated page hits we could generate a similar demand matrix for components and use it for what-if analysis.

I went ahead and kept my solution in R even though I saw where I thought that a perl solution would be more elegant (did I just use perl and elegant in the same sentence?) I designed my solution with two separate programs, one to spit out numbers and another to generate graphs of page response times and another showing component utilization. Both pieces of code make use of PDQ-R and allow for a variable number of web servers, application servers and database servers.


# Solution parameters 

gamma  <- 10.96;   # Rate into system
numWS  <- 1;       # Number of Web Servers
numAS  <- 1;       # Number of Application Servers
numDS  <- 1;       # Number of Database Servers

# external library
library("pdq");

# Constants #

E <- 1;
H <- 2;
S <- 3;
V <- 4;
G <- 5;
C <- 6;
B <- 7;
X <- 8;

PAGE_NAMES <- c("Enter", "HomePage", "Search", "ViewBids", "Login", "CreateAuction", "PlaceBid", "Exit");
COMPONENTS <- c("CPU", "Disk");
SERVER_TYPES <- c("WS", "AS", "DS");

WS_CPU  <- 1;
WS_DISK <- 2;
AS_CPU  <- 3;
AS_DISK <- 4;
DS_CPU  <- 5;
DS_DISK <- 6;

# Functions used in solution

VisitsByTransitionMatrix <- function(M, B) {
   A <- t(M);
   A <- -1 * A;
   for (i in 1:sqrt(length(A))) {
     j <- i;
     A[i,j] <- A[i,j] + 1;
   };
   return(solve(A,B));
};

CalculateLambda <- function(gamma, f_a, f_b, V_a, V_b, index) {
  return (
           gamma*((f_a*V_a[index]) + (f_b*V_b[index]))
         );
};


f_a    <- 0.25;    # Fraction of TypeA users
f_b    <- 1 - f_a; # Fraction of TypeB users

lambda <- 1:X;     # Array of lambda for each page 

SystemInput <- matrix(c(1,0,0,0,0,0,0,0),nrow=8,ncol=1)                             # 8.3, Figure 8.2, page 208
TypeA       <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,
                        0.2,0,0,0.4,0.2,0.15,0,0,0.25,0,0,
                        0,0,0.65,0,0,0.35,0,0,0,0,0,0.3,0.6,
                        0.1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,
                        0,0,0,0,0,0,0,0), ncol=8, nrow=8, byrow=TRUE);              # 8.4, Table 8.1, page 209
TypeB       <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,
                        0.2,0,0,0.45,0.15,0.1,0,0,0.3,0,0,
                        0,0,0.4,0,0,0.6,0,0,0,0,0,0.3,0.55,
                        0.15,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,
                        1,0,0,0,0,0,0,0,0), nrow=8, ncol=8, byrow=TRUE);            # 8.4, Table 8.2, page 210
DemandTable <- matrix(c(0,0.008,0.009,0.011,0.06,0.012,0.015,
                        0,0,0.03,0.01,0.01,0.01,0.01,0.01,0,
                        0,0,0.03,0.035,0.025,0.045,0.04,0,0,
                        0,0.008,0.08,0.009,0.011,0.012,0,0,
                        0,0.01,0.009,0.015,0.07,0.045,0,0,0,
                        0.035,0.018,0.05,0.08,0.09,0), ncol=8, nrow=6, byrow=TRUE); # 8.4, Table 8.4, page 212 (with modifications)

VisitsA <- VisitsByTransitionMatrix(TypeA, SystemInput);
VisitsB <- VisitsByTransitionMatrix(TypeB, SystemInput);

lambda[E] <- 0; # Not used in calculations
lambda[H] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, H);
lambda[S] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, S);
lambda[V] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, V);
lambda[G] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, G);
lambda[C] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, C);
lambda[B] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, B);
lambda[X] <- 0 # Not used in calculations

Init("e_biz_service");

# Define workstreams 

for (n in H:B) {
  workStreamName <- sprintf("%s", PAGE_NAMES[n]);
  CreateOpen(workStreamName, lambda[n]);
};

# Define Web Server Queues

for (i in 1:numWS) {
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("WS_%d_%s", i, COMPONENTS[j]);
    CreateNode(nodeName, CEN, FCFS);
  };
};

# Define Application Server Queues

for (i in 1:numAS) {
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("AS_%d_%s", i, COMPONENTS[j]);
    CreateNode(nodeName, CEN, FCFS);
  };
};

# Define Database Server Queues

for (i in 1:numDS) {
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("DS_%d_%s", i, COMPONENTS[j]);
    CreateNode(nodeName, CEN, FCFS);
  };
};

# Set Demand for the Web Servers

for (i in 1:numWS) {
  demandIndex <- WS_CPU;
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("WS_%d_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numWS);
    };
  };
};

# Set Demand for the App Servers

for (i in 1:numAS) {
  demandIndex <- AS_CPU;
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("AS_%d_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numAS);
    };
  };
};

# Set Demand for the Database Servers

for (i in 1:numDS) {
  demandIndex <- DS_CPU;
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("DS_%d_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numDS);
    };
  };
};

SetWUnit("Trans");
SetTUnit("Second");

Solve(CANON);

print("Arrival Rates for each page:");

for (i in H:B) {
  print(sprintf("%s = %f", PAGE_NAMES[i], lambda[i]));
};

print("[-------------------------------------------------]");

print("Page Response Times");

for (i in H:B) {
  workStreamName <- sprintf("%s", PAGE_NAMES[i]);
  print(sprintf("%s = %f seconds.", PAGE_NAMES[i], GetResponse(TRANS, workStreamName)));
};

print("[-------------------------------------------------]");

print("Component Utilizations");

for (i in 1:numWS) {
  for (j in 1:length(COMPONENTS)) {
    totalUtilization <- 0;
    nodeName <- sprintf("WS_%s_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
    };
    print(sprintf("%s = %3.2f %%", nodeName, totalUtilization * 100));
  };
};

for (i in 1:numAS) {
  for (j in 1:length(COMPONENTS)) {
    totalUtilization <- 0;
    nodeName <- sprintf("AS_%s_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
    };
    print(sprintf("%s = %3.2f %%", nodeName, totalUtilization * 100));
  };
};

for (i in 1:numDS) {
  for (j in 1:length(COMPONENTS)) {
    totalUtilization <- 0;
    nodeName <- sprintf("DS_%s_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
    };
    print(sprintf("%s = %3.2f %%", nodeName, totalUtilization * 100));
  };
};

Here is a bit of sample output with 10.96 users entering the system per second:


[1] "Arrival Rates for each page:"
[1] "HomePage = 10.960000"
[1] "Search = 13.658485"
[1] "ViewBids = 2.208606"
[1] "Login = 3.664958"
[1] "CreateAuction = 1.099487"
[1] "PlaceBid = 2.074180"
[1] "[-------------------------------------------------]"
[1] "Page Response Times"
[1] "HomePage = 0.083517 seconds."
[1] "Search = 1.612366 seconds."
[1] "ViewBids = 1.044683 seconds."
[1] "Login = 2.323417 seconds."
[1] "CreateAuction = 3.622690 seconds."
[1] "PlaceBid = 3.983755 seconds."
[1] "[-------------------------------------------------]"
[1] "Component Utilizations"
[1] "WS_1_CPU = 49.91 %"
[1] "WS_1_Disk = 55.59 %"
[1] "AS_1_CPU = 71.11 %"
[1] "AS_1_Disk = 35.59 %"
[1] "DS_1_CPU = 38.17 %"
[1] "DS_1_Disk = 97.57 %"

Take a look at that database server disk utilization. Almost 100%! That isn't any good. Let's run the model with two database servers just to ease up on the poor drives!

I modify the line that reads "numDS <- 1; # Number of Database Servers" to read "numDS <- 2; # Number of Database Servers" and let 'er rip:


[1] "Arrival Rates for each page:"
[1] "HomePage = 10.960000"
[1] "Search = 13.658485"
[1] "ViewBids = 2.208606"
[1] "Login = 3.664958"
[1] "CreateAuction = 1.099487"
[1] "PlaceBid = 2.074180"
[1] "[-------------------------------------------------]"
[1] "Page Response Times"
[1] "HomePage = 0.083517 seconds."
[1] "Search = 0.237452 seconds."
[1] "ViewBids = 0.336113 seconds."
[1] "Login = 0.358981 seconds."
[1] "CreateAuction = 0.462042 seconds."
[1] "PlaceBid = 0.440903 seconds."
[1] "[-------------------------------------------------]"
[1] "Component Utilizations"
[1] "WS_1_CPU = 49.91 %"
[1] "WS_1_Disk = 55.59 %"
[1] "AS_1_CPU = 71.11 %"
[1] "AS_1_Disk = 35.59 %"
[1] "DS_1_CPU = 19.09 %"
[1] "DS_1_Disk = 48.78 %"
[1] "DS_2_CPU = 19.09 %"
[1] "DS_2_Disk = 48.78 %"

That's better. There is a significant reduction in page response time to boot:


Page            1 DS            2 DS            Diff
HomePage        0.083517        0.083517        0
Search          1.612366        0.237452       -1.374914
ViewBids        1.044683        0.336113       -0.70857
Login           2.323417        0.358981       -1.964436
CreateAuction   3.62269         0.462042       -3.160648
PlaceBid        3.983755        0.440903       -3.542852

Holy smoke! Adding that second database server makes a heck of a difference, doesn't it?

Being able to pull up numbers is great, but it doesn't have the impact that a good graph does. I love graphs! They are so great for conveying information to non-technical folks.

So, to do that I took my original solution and did a code spin, fold and mutilation to allow the generation of graphs. Here is the code that I wrote to generate the graphs.


# Solution parameters 

maxGamma      <- 11.2 # Maximum rate into system
steppingValue <- 0.1;

numWS  <- 1;       # Number of Web Servers
numAS  <- 1;       # Number of Application Servers
numDS  <- 1;       # Number of Database Servers

# external library
library("pdq");

# Constants

E <- 1;
H <- 2;
S <- 3;
V <- 4;
G <- 5;
C <- 6;
B <- 7;
X <- 8;

PAGE_NAMES <- c("Enter", "HomePage", "Search", "ViewBids", "Login", "CreateAuction", "PlaceBid", "Exit");
COMPONENTS <- c("CPU", "Disk");
SERVER_TYPES <- c("WS", "AS", "DS");

WS_CPU  <- 1;
WS_DISK <- 2;
AS_CPU  <- 3;
AS_DISK <- 4;
DS_CPU  <- 5;
DS_DISK <- 6;

# Functions used in solution

VisitsByTransitionMatrix <- function(M, B) {
   A <- t(M);
   A <- -1 * A;
   for (i in 1:sqrt(length(A))) {
     j <- i;
     A[i,j] <- A[i,j] + 1;
   };
   return(solve(A,B));
};

CalculateLambda <- function(gamma, f_a, f_b, V_a, V_b, index) {
  return (
           gamma*((f_a*V_a[index]) + (f_b*V_b[index]))
         );
};


f_a    <- 0.25;    # Fraction of TypeA users
f_b    <- 1 - f_a; # Fraction of TypeB users

lambda <- 1:X;     # Array of lambda for each page 

SystemInput <- matrix(c(1,0,0,0,0,0,0,0),nrow=8,ncol=1)                             # 8.3, Figure 8.2, page 208
TypeA       <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,
                        0.2,0,0,0.4,0.2,0.15,0,0,0.25,0,0,
                        0,0,0.65,0,0,0.35,0,0,0,0,0,0.3,0.6,
                        0.1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,
                        0,0,0,0,0,0,0,0), ncol=8, nrow=8, byrow=TRUE);              # 8.4, Table 8.1, page 209
TypeB       <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,
                        0.2,0,0,0.45,0.15,0.1,0,0,0.3,0,0,
                        0,0,0.4,0,0,0.6,0,0,0,0,0,0.3,0.55,
                        0.15,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,
                        1,0,0,0,0,0,0,0,0), nrow=8, ncol=8, byrow=TRUE);            # 8.4, Table 8.2, page 210
DemandTable <- matrix(c(0,0.008,0.009,0.011,0.06,0.012,0.015,
                        0,0,0.03,0.01,0.01,0.01,0.01,0.01,0,
                        0,0,0.03,0.035,0.025,0.045,0.04,0,0,
                        0,0.008,0.08,0.009,0.011,0.012,0,0,
                        0,0.01,0.009,0.015,0.07,0.045,0,0,0,
                        0.035,0.018,0.05,0.08,0.09,0), ncol=8, nrow=6, byrow=TRUE); # 8.4, Table 8.4, page 212 (with modifications)

VisitsA <- VisitsByTransitionMatrix(TypeA, SystemInput);
VisitsB <- VisitsByTransitionMatrix(TypeB, SystemInput);

numSteps     <- (maxGamma/steppingValue)+1;
#numSteps    <- (maxGamma/steppingValue);
numElements  <- numSteps*X;
numUElements <- numSteps*(3*length(COMPONENTS));

steppingArray <- 1:numSteps;
responseArray <- 1:numElements;
utilArray     <- 1:numUElements;

responseArray <- responseArray * 0;
utilArray     <- utilArray * 0;

componentList <- length(COMPONENTS)*length(SERVER_TYPES);

entryNumber <- 1;
for (serverType in SERVER_TYPES) {
  for (serverComponent in COMPONENTS) {
    componentList[entryNumber] <- sprintf("%s %s", serverType, serverComponent); 
    entryNumber <- entryNumber + 1;
  };
}; 

dim(responseArray) = c(X, round(numElements/X));
dim(utilArray) = c(3*length(COMPONENTS), round(numUElements/(3*length(COMPONENTS))));

loopCount <- 1;

for (gamma in seq(0, maxGamma, steppingValue)) {

  steppingArray[loopCount] <- gamma;

  lambda[E] <- 0; # Not used in calculations
  lambda[H] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, H);
  lambda[S] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, S);
  lambda[V] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, V);
  lambda[G] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, G);
  lambda[C] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, C);
  lambda[B] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, B);
  lambda[X] <- 0 # Not used in calculations

  Init("e_biz_service");

  # Define workstreams 

  for (n in H:B) {
    workStreamName <- sprintf("%s", PAGE_NAMES[n]);
    CreateOpen(workStreamName, lambda[n]);
  };

  # Define Web Server Queues

  for (i in 1:numWS) {
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("WS_%d_%s", i, COMPONENTS[j]);
      CreateNode(nodeName, CEN, FCFS);
    };
  };

  # Define Application Server Queues

  for (i in 1:numAS) {
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("AS_%d_%s", i, COMPONENTS[j]);
      CreateNode(nodeName, CEN, FCFS);
    };
  };

  # Define Database Server Queues

  for (i in 1:numDS) {
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("DS_%d_%s", i, COMPONENTS[j]);
      CreateNode(nodeName, CEN, FCFS);
    };
  };

  # Set Demand for the Web Servers

  for (i in 1:numWS) {
    demandIndex <- WS_CPU;
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("WS_%d_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numWS);
      };
    };
  };

  # Set Demand for the App Servers

  for (i in 1:numAS) {
    demandIndex <- AS_CPU;
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("AS_%d_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numAS);
      };
    };
  };

  # Set Demand for the Database Servers

  for (i in 1:numDS) {
    demandIndex <- DS_CPU;
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("DS_%d_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numDS);
      };
    };
  };

  Solve(CANON);

  for (i in H:B) {
    workStreamName <- sprintf("%s", PAGE_NAMES[i]);
    responseArray[i, loopCount] <- GetResponse(TRANS, workStreamName);
  };

  uArrayEntries <- 0;
  for (i in 1:numWS) {
    for (j in 1:length(COMPONENTS)) {
      totalUtilization <- 0;
      nodeName <- sprintf("WS_%s_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
        if (i == 1) {
          utilArray[uArrayEntries+j, loopCount] <- totalUtilization*100;
        };
      };
    };
  };

  uArrayEntries <- 2;
  for (i in 1:numAS) {
    for (j in 1:length(COMPONENTS)) {
      totalUtilization <- 0;
      nodeName <- sprintf("AS_%s_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
        if (i == 1) {
          utilArray[uArrayEntries+j, loopCount] <- totalUtilization * 100;
        };
      };
    };
  };

  uArrayEntries <- 4;
  for (i in 1:numDS) {
    for (j in 1:length(COMPONENTS)) {
      totalUtilization <- 0;
      nodeName <- sprintf("DS_%s_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
        if (i == 1) {
          utilArray[uArrayEntries+j, loopCount] <- totalUtilization * 100;
        };
      };
    };
  };

  loopCount <- loopCount + 1;

};

# Generate Response Time Graph

loopCount <- 0;
for (i in H:B) {
  arrayLength = numElements/X;
  if (loopCount == 0) {
    jpeg(file=sprintf("response_time_%d_WS_%d_AS_%d_DS.jpg", numWS, numAS, numDS), height=768, width=1024, quality=100);
    plot(steppingArray, 
         responseArray[i, 1:arrayLength], 
         xlab="Hits Per Second into System", 
         ylab="Response Time", 
         col=i,
         ylim=c(0,ceiling(max(responseArray))),
         type="l",
         pch=i,
         lwd=4);
    title(main=sprintf("Page Response Times for E-Biz with %d WS, %d AS and %d DS", numWS, numAS, numDS), font.main=2);
  } else {
    lines(steppingArray, responseArray[i, 1:arrayLength], col=i, pch=i, lwd=4);
  };
    
  loopCount <- loopCount + 1;
};
legend(1, ceiling(max(responseArray)), PAGE_NAMES[H:B], col=H:B, lty=1, lwd=4, cex=1.2);  
dev.off()

# Graph component utilization

loopCount <- 0;
for (i in 1:length(componentList)) {
  arrayLength = numSteps;
  if (loopCount == 0) {
    jpeg(file=sprintf("resource_utilization_%d_WS_%d_AS_%d_DS.jpg", numWS, numAS, numDS), height=768, width=1024, quality=100);
    plot(steppingArray, 
         utilArray[i, 1:arrayLength], 
         xlab="Hits Per Second into System", 
         ylab="Resource Utilization, %", 
         col=i,
         ylim=c(0,100),
         type="l",
         pch=i,
         lwd=4);
    title(main=sprintf("Resource Utilization for E-Biz with %d WS, %d AS and %d DS", numWS, numAS, numDS), font.main=2);
  } else {
    lines(steppingArray, utilArray[i, 1:arrayLength], col=i, pch=i, lwd=4);
  };
  loopCount <- loopCount + 1;
};
legend(0, 100, componentList, col=1:length(componentList), lty=1, lwd=4, cex=1.2);  
dev.off()

The code isn't what I would call elegant at this time as a lot of it is still hardcoded for the specific solution but it is a step in the right direction.

And let's take a look at response time between the single database server and dual load bearing database servers:

Take a look at the effect that the overworked database disk drive has on the response time. At 11.2 customers per second into the site and the response time has shot up to the 30 second mark. Definitely not what you want your customers to have to suffer through to be sure.

Here is a graph generated with the dual load bearing database server solution:

Holy guacamole is that a heck of an improvement or what? Any management type can look at these two graphs and immediately realize the impact to the system and the need for extra equipment.

And just for good measure, here is the resource utilization with both single and dual database servers:

Even a MBA grad can look at those graphs and realize that there is a problem that needs to be solved. w00t!

Ain't heuristic analysis fun?

Applying PDQ-R to this case study was a great exercise and I'm glad I undertook it. I can't wait to apply this to a real system and compare the results to see how well it works "in the real world." One thing that this type of heuristic analysis won't show is the interaction between pages with negative performance. In some circumstances I have seen performance programs between page that were thought to not be related. When Page A is put under duress Page B slows down even though it is thought that Page A is not related to Page B. Often it has been found that in a spaghetti line of object dependency that in some way the two pages were related. With the analysis of service demand of pages we can also find pages that have a lot of high service demand as well.

In the future along with page response times and normal metric reporting I think I'll add in service demand as well so that when pages do start to perform poorly hopefully the change in service demand will give an area to look into at the start of the analysis of the root cause of the performance problem to assist the developers.

Dr. Gunther has some more examples of applying Perl::PDQ to a variety of systems in his book, Analyzing Computer System Performance with Perl::PDQ. The source to the solutions in the book can be found in the PDQ download. But developing this solution from the ground up really drove some points home for me that will no doubt be useful in my future endeavors.

Hey! Let's take a transistion matrix and convert it into a series of linear equations to gonkulate the number of visits to various pages!

2009-07-30T12:23:00.006-05:00

I've been working on learning the wonderful world of queuing theory. Two of the books that I've been using are Analyzing Computer Systems Performance: With Perl: PDQ and Performance by Design: Computer Capacity Planning By Example.

I've been learning how to apply Dr. Gunther's PDQ API, specifically, PDQ-R which is PDQ for the R programming language. Normally I'd use the Perl::PDQ module but I've found R to be quite handy for crunching numbers. If anything, I'd use perl to munge the data in preparation for crunching by routines in R.

I decided that I would apply PDQ-R to several of the case studies found in Performance by Design: Computer Capacity Planning By Example. In my first attempt to do so I found some numbers that didn't match between the results of PDQ-R and the Excel spreadsheet available as part of Performance by Design: Computer Capacity Planning By Example. I contacted Dr. Gunther and he found that there was a bug in PDQ 5.0.1 that he blogged about here. I guess that is my 15 mS of internet fame!

I got side tracked with some work related stuff and went back to the task of applying PDQ-R to some of the case studies found in Performance by Design: Computer Capacity Planning By Example.

The case study that I have started working on is in Chapter 8: Case Study IV: An E-Business Service.

The author provides us with a Customer Behavior Model Graph of a website:

From this graph we can create a set of linear equations that describes the number of visits to each page of the system presented in the CBMG (seen above). Taking a closer look at the graph we can see that there are probabilities of transitioning between one page to another. For example, the probability of moving from the Home Page (h) to the Login (g) is Phg. The probability of moving from the Login (g) page to the Search (s) page is Psg.

Using these probabilities we can create a set of linear equations to describe the number of visits to each page in the system based upon the number of visits into the entry of the system. In this case, there is only one way into the system and that is via the Entry (e) page.

We can look at the graph and manually derive the linear equations. For example, we can easily see that Vh = Peh*Ve. We can also see that Vg = Phg*Vh + Psg*Vs + Pvg*Vv. Doing this we eventually come up with eight linear equations that we can then solve. (The equations are listed on page 208 of Performance by Design: Computer Capacity Planning By Example).

The authors of the book provide us with all the probabilities in the form of a Matrix of Transition Probabilities on page 209. Here is that matrix:


                        (e)     (h)   (s)   (v)   (g)   (c)   (b)   (x)
Entry (e)               0       1     0     0     0     0     0     0
Home (h)                0       0     0.7   0     0.1   0     0     0.2
Search (s)              0       0     0.4   0.2   0.15  0     0     0.25
View Bids (v)           0       0     0     0     0.65  0     0     0.35
Login (g)               0       0     0     0     0     0.3   0.6   0.1
Create Auction (c)      0       0     0     0     0     0     0     1
Place Bid (b)           0       0     0     0     0     0     0     1
Exit (x)                0       0     0     0     0     0     0     0

So we can take those values found in the transition matrix and eventually gonkulate all the visits to each page. Although the transition matrix is provided by the author it is easy enough to re-create this matrix from web logs for a site.

A number of years ago I did the same thing for a major e-Commerce site. At the time my goal was to use the transition matrix to create a universal virtual user that would emulate what real customers did in production. At the time we were using a number of different virtual users and it was a real PITA to increase the number of vusers in a scenario. I figured if I could generate the transition matrix and generate random walks via the transition matrix that my vusers would better represent the traffic of real customers. Unfortunately, I never got to finish the implementation of the UberVuser but I did get to the point of generating a transition matrix. And that sucker was huge! It wasn't a nice 8x8 matrix but rather a 538x538 matrix. Yeah... That's a lot of elements!

So, thinking of my previous experience and looking at the author's provided transition matrix got me to thinking. There's bound to be a way of backing the transition matrix into the set of linear equations that can easily be solved in R (or any other method that can solve linear equations).

I worked it out by hand (more about this below) and everything looked good but to really make this useful I needed to automate the routine. I sat down looked at the problem and thought that I was going to have to do a bunch of element manipulation when the solution hit me. It turned out to be so damned simple! I love it when a complicated problem can be easily solved. It just amuses the hell out of me.

This is how I backed out the equations by hand. I know that the transition matrix has from "from" pages down the rows and the "to" along the columns. So for example, if I wanted to figure out the equation for the Vs I would work down the column and use the coefficients of the transition matrix as multipliers for the V variables.

For example, with the 3rd column with solving for Vs:

The third column of the transition matrix is:


    (s)
(e) 0
(h) 0.7
(s) 0.4
(v) 0
(g) 0
(c) 0
(b) 0
(x) 0

So to solve for Vs:

Vs = 0*Ve + 0.7*Vh + 0.4Vs + 0*Vv + 0*Vg + 0*Vc + 0*Vb + 0*Vx

Notice that we have a Vs on both sides of the equal sign. This corresponds to the loop on the Search node.

We can drop out the variables that are multiplied by zero.

Vs = 0.7*Vh + 0.4Vs

Repeat the steps for all the columns of the transition matrix and you end up with the equations to solve for. But, most packages want the equations setup in matrix form. So, let's get all the variables on the left side of the equal sign.

Vs - 0.4Vs - 0.7*Vh

Simplifies into:

0.6Vs - 0.7*Vh

If we do this with all the equations we can finally put it into matrix form. But we can't solve it yet. We need another vector to solve with. This vector will contain the input into the system. In this case the entry into the system is into the Entry page so the vertical vector is:

(e) 1
(h) 0
(s) 0
(v) 0
(g) 0
(c) 0
(b) 0
(x) 0

If we call our first matrix that we created from the transition matrix A and our input vertical vector B, we have the classic A*B = 0 that we can solve for.

Now we get into the R solution for the above. As I was working out the algorithm to setup the problem for calling the R solve() routine it hit me how simple the solution was!

Let's call our transition matrix supplied by the authors as M.

We can take transpose of M and let's call it A.

Take A and multiply by the scalar value of -1. This is the act of moving all the variables to the left side of the equals sign.

Now we combine variables we are solving for in each row of A by adding 1 to the trace of matrix A.

And there we have our final form of A. We can now solve for A*B = 0!

Here is my solution in R


# Define the column information for easy access by name

e <- 1;
h <- 2;
s <- 3;
v <- 4;
g <- 5;
c <- 6;
b <- 7;
x <- 8;

# M will be the supplied transition matrix

M <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,0.2,0,0,0.4,0.2,0.15,0,0,0.25,0,0,0,0,0.65,0,0,0.35,0,0,0,0,0,0.3,0.6,0.1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0), ncol=8, nrow=8, byrow=TRUE);

# B is the vector that describes entry into the system
B <- matrix(c(1,0,0,0,0,0,0,0),nrow=8,ncol=1);

VisitsByTransitionMatrix <- function(M, B) {
   A <- t(M);
   A <- -1 * A;
   for (i in 1:sqrt(length(A))) {
     j <- i;
     A[i,j] <- A[i,j] + 1;
   };
   return(solve(A,B));
};

Visits <- VisitsByTransitionMatrix(M, B);

> Visits
          [,1]
[1,] 1.0000000
[2,] 1.0000000
[3,] 1.1666667
[4,] 0.2333333
[5,] 0.4266667
[6,] 0.1280000
[7,] 0.2560000
[8,] 1.0000000

> Visits[e]
[1] 1
> Visits[h]
[1] 1
> Visits[s]
[1] 1.166667
> Visits[v]
[1] 0.2333333
> Visits[g]
[1] 0.4266667
> Visits[c]
[1] 0.128
> Visits[b]
[1] 0.256
> Visits[x]
[1] 1

And there we go! Using this I could have taken that monster 538x538 matrix and easily figured out the number of hits for each page based upon the derived transition matrix and used the R routine for playing a lot of what-if scenarios. For example, what if we want to change some of the probabilities between nodes, how will that effect the visit to other pages? Now it is simple to crunch the numbers. Huzzah!

My next task is to apply PDQ-R to solving the rest of the case study. Later in the chapter there are service times associated with each page and that will determine the total service demand per page and ultimately the page response time for each page and the effect on internal components that will be modeled in my PDQ-R model.

I can really see where I could have used this modeling at my previous job at a major e-commerce site. It would have come in really stinkin' handy to be sure.

Automating the finding of coefficients for the USL

2009-06-01T15:14:00.005-05:00

I got to playing around with R more and I've always found that to learn a language I need to solve problems with the language. I'm sure most everybody else does the same thing. My goal was to write a R function that imported a CSV with performance information to gonkulate against.

In my case, I'm using the basic performance information from "Guerrilla Capacity Planning." I've created a CSV file with the number of procs and the resultant ray trace benchmark from Table 5.1:

   1:  C:\Users\auswipe\Desktop>cat raw_throughput.csv

   2:  p,x

   3:  1,20

   4:  4,78

   5:  8,130

   6:  12,170

   7:  16,190

   8:  20,200

   9:  24,210

  10:  28,230

  11:  32,260

  12:  48,280

  13:  64,310

Then I wrote an R function to crunch the numbers:

   1:  uslCoefficients <- function(dataFile) {

   2:    uslData <- read.csv(dataFile, header=TRUE);

   3:    uslData$c <- uslData$x / uslData$x[1];

   4:    usl <- nls(c ~ p/(1+sigma*(p-1)+kappa*p*(p-1)),

   5:               uslData,

   6:               algorithm="port",

   7:               start=c(sigma=0.0, kappa=0.0),

   8:               lower=c(0,0));

   9:    sigma <- coef(usl)["sigma"];

  10:    kappa <- coef(usl)["kappa"];

  11:    return(list(sigma=sigma, kappa=kappa));

  12:  };

The function uslCoefficient returns a list where I can reference the "sigma" and "kappa" by named index:

   1:  > uslCoef <- uslCoefficients("c:\\Users\\auswipe\\Desktop\\raw_throughput.csv")

   2:  > uslCoef["sigma"]

   3:  $sigma

   4:      sigma

   5:  0.0497973

6:

   7:  > uslCoef["kappa"]

   8:  $kappa

   9:         kappa

  10:  1.143404e-05

11:

  12:  > uslCoef

  13:  $sigma

  14:      sigma

  15:  0.0497973

16:

  17:  $kappa

  18:         kappa

  19:  1.143404e-05

R is pretty nifty. I doubt I'll ever make use of all the power that is available but it'll be better than writing my own stat routines.

Using R to calculate coefficients of the Universal Scaling Law with Non-Linear Regression

2009-06-01T08:51:00.007-05:00

Ooh! Doesn't that sound fancy?

Several weeks ago I purchased the eBook from O'Reilly called "The Art of Capacity Planning." I've always thought that load testing and capacity planning went hand-in-hand. One is not a replacement for the other but one can assist with the other. Load test helps out capacity planning by applying load to psuedo-production systems and capacity planning helps load testing by verifying results in load test against real world systems.

I finished "The Art of Capacity Planning" and wanted to read more on the subject and picked up a copy of "Guerrilla Capacity Planning" which has a lot more math than "The Art of Capacity Planning." One of the concepts is the Universal Scaling Law based on Amdhal's Law. Dr. Neil J Gunther is a smart cookie. He even has a Ph.D in Theoretical Physics which makes him closer to Gordon Freeman than I'll ever be! (Side question: Do Ph.D's in Theoretical Physics get crowbars at graduation?)

Anyhoo, in section 5.6.1 one of the methods in the book is to use Excel to do second degree polynomial regression for the calculation of two necessary coefficients, sigma and kappa. But when I tried to use Excel I was getting a negative value for sigma and one of the rules of the Universal Scaling Law is that the coefficients can never, ever, ever be negative. I just figured that I fat fingered something and tried it again and once again, got mismatching results.

I scratched my noggin, tried to figure out where I err'd and did some Googling and came across this entry of Dr. Gunther's blog:

Negative Scalability Coefficients in Excel

Because in Excel (and some other packages, like my TI-89) you can't put a constraint on the lower limits of the coefficient, you might from time to time get negative coefficients. But from reading the blog entry I see that other people are using R with success.

This is the first time that I've ever messed around with R for statistical purposes. In the past I've written some stat routines (years ago!) in C# for comparing before/after load testing results.

Here is how I used R from start to finish to gonkulate the coefficients.

Using the data from Section 5.3 I did the following in R:

First I defined my p array, which in the book is the number of processors used for ray tracing:

   1:  p <- c(1, 4, 8, 12, 16, 20, 24, 28, 32, 48, 64)

Then I defined my c array, which is the relative capacity for the number of processors used for ray tracing:

   1:  c <- c(1.0, 3.9, 6.5, 8.5, 9.5, 10.0, 10.5, 11.5, 13.0, 14.0, 15.5)

I combined both arrays into a data frame for later use.

   1:  df <- data.frame(p, c)

And when I check out the contents of df I get:

   1:  df

   2:      p    c

   3:  1   1  1.0

   4:  2   4  3.9

   5:  3   8  6.5

   6:  4  12  8.5

   7:  5  16  9.5

   8:  6  20 10.0

   9:  7  24 10.5

  10:  8  28 11.5

  11:  9  32 13.0

  12:  10 48 14.0

  13:  11 64 15.5

I can now use a non-linear regression routine with my data frame that I entered above.

   1:  usl <- nls(c ~ p/(1+sigma*(p-1)+kappa*p*(p-1)), df, algorithm="port", start=c(sigma=0.0, kappa=0.0), lower=c(0,0))

I can then access the coefficients by named index:

   1:  sigma <- coef(usl)["sigma"]

   2:  kappa <- coef(usl)["kappa"]

3:

   4:  sigma

   5:      sigma

   6:  0.0497973

7:

   8:  kappa

   9:         kappa

  10:  1.143404e-05

11:

Huzzah!

I can now interpolate the relative capacity based upon the USL and the coefficients that were previously gonkulated and add that to my current data frame, df, that I defined earlier. I do have to note that I was a slackard and did not apply the significant digits rules as outlined in Chapter 3 of "Guerrilla Capacity Planning."

   1:  df$proj_c <- p/(1 + sigma * (p - 1) + kappa * p * (p - 1))

There are the projected relative capacities. Yay!

   1:  df

   2:      p    c    proj_c

   3:  1   1  1.0  1.000000

   4:  2   4  3.9  3.479686

   5:  3   8  6.5  5.929346

   6:  4  12  8.5  7.745536

   7:  5  16  9.5  9.144406

   8:  6  20 10.0 10.253815

   9:  7  24 10.5 11.154233

  10:  8  28 11.5 11.898837

  11:  9  32 13.0 12.524174

  12:  10 48 14.0 14.259114

  13:  11 64 15.5 15.298811

And here I will make a simple little graph of the actual versus projected relative capacity:

   1:  plot(p, c)

   2:  lines(p, proj_c)

And here is the graph that is generated:

Kinda nifty, eh?

I can see myself using R more in the future. I'd rather write routines for automagic analysis of data with R than write my own routines from the ground up.

Push-To-Test

2009-05-02T14:28:00.002-05:00

I went to a four hour presentation on Push-To-Test yesterday. It seems pretty nifty, the idea of wrapping an automated testing framework around a bunch of open source projects such as Selenium, soapUI and other goodies.

The presentation was a little hectic but it got the general idea across. I would have preferred to get to the meat of the subject quicker but the presenter did have 20 people to deal with and we had to go with the common denominator. No biggie.

The only thing that I wasn't too big was the idea of converting the Selenium tests for web tests from XUL into Java or Jython for more programmable control. After the conversion there is no going back to the original SeleniumIDE tool from what I saw. But, I guess that isn't much different from what I've done with LoadRunner and VSTS a bajillion times before when I had a bunch of custom code from the tree view into code. They did have an IDE so it's not too bad in hind site. I didn't get a chance to play around with the IDE to see how it compares to Eclipse or Visual Studio.

But, it is free for download to use the unsupported version, so I say thumbs up.

The idea of using Selenium for the web tests is pretty nifty as you get good control with AJAX controls that are a bit of a pain to control in HTTP Virtual user. I hadn't thought of doing that in the past.

I was told that transactions and nested transactions were supported but didn't get a chance to see them in action.

Learning QTP

2009-03-26T10:20:00.003-05:00

It's been a long time between blog posts. Since this blog is mainly for myself, that isn't a problem. :-)

I'm starting to learn QTP and finding that it looks like a real handy tool for functional and integration testing. I'm still trying to get myself to not look at it as an alternate load testing tool and trying to use the interface and not look at each problem as more code to be written. I had the same problem initially with JMeter as well having come from a background where I would normally add a bunch of custom code after the initial script creation.

I'm not a big fan of VBScript but for the purpose at hand I have to say it is probably a better choice than using C like LoadRunner. Easier to create a COM object to be invoked with CreateObject for slicing and dicing of data.

I'm not sure if raw HTML can be pulled in with QTP, yet. I wouldn't be surprised if it can. QTP seems to be a pretty nifty tool. No doubt it will help with future job searches.

Got a job!

2009-01-08T20:12:00.003-06:00

I accepted an informal job offer today at the same pay as my previous job (which wasn't too shabby). I will be a performance engineer on a rather large project involving Red Hat RHEL 4.5, Orace and Java/WebLogic. Here is a really good opportunity to bone up on Java performance analysis along with WebLogic and Java. I'm not much of a Java dude so I will finally be getting up to speed.

I'm not to worried about the RHEL as I've been using various flavors of Unices (primarily FreeBSD) over the years. I won't get into the whole debate of genetic UNIX versus copied UNIX. Just not worth wasting my breath.

Another opportunity to learn Oracle performance tuning as well. I've done a lot with SQL Server over the years and I hope that some of it applies to the Oracle side of the house.

It looks like I'll be back to using Load Runner again. Looks like either Winsock/SOA or HTTP vusers. I've heard some bad things about the SOA Vusers with respect to generated script complexity and ease of correlation of data. All I know at this point is that the client is a swing UI running a Java application that will connect to the middle tier. Hopefully it is all done via SOA/WebServies. I have an interesting idea of utilizing WireShark and chaosreader.pl and some custom perl scripts for generationg clean vusers for the probject. We'll just have to wait and see how it goes.

Another interesting thing will be working out of the house full time. I have two previous coworkers that do this and they love it but I am a very social creature. We'll see how it goes. I will get a laptop and an aircard from my new employer so that means I will be able to work from any location so I should be able to get out of the house often and see friends for lunch, which has always been important to me.

So, my adventure continues, just not on the original vector. :-)

Taking some time to keep on learning.

2008-12-11T16:14:00.005-06:00

Since I am unemployed and not having anything better to do with my time I decided I would check out "Apache JMeter" by Emily H. Halili (Packt, 2008).

It's a light read and I found that Chapter 7 (Advanced Features) was the most helpful to myself as I am not a n00b to JMeter. However, I felt that Chapter 7 would have been better served if BeanShell processing would have been tackled. I've found from my load testing experience in the past that at some point the return HTML is gonna have to be sliced and diced and data extracted that cannot be done with a simple RegEx extraction (like what is covered in Chapter 7 of the book).

Other than that, I think that if you are a total n00b to JMeter it isn't half bad as a simple introduction to using JMeter for performance/load testing. There's a lot more to performance/load testing than the book covers such as metrics collection, number crunching, et cetera but the book doesn't purport itself to be the end all be all of explaining performance/load testing so I can't complain.

Three stars.

And that job is toast!

2008-12-03T15:23:00.003-06:00

Well. Got laid off today with 20 other folks from the Dallas area. Cannot say that I am surprised. Good thing I've been saving up for this possibility. Didn't get to keep the Uberlaptop of Powah (and I wasn't gonna pay the $3700 to keep it).

I don't expect to get another job in December but I have some possibilities lined up in January. What am I gonna do with that time? Hmmmm. Wasn't GTA IV just released? Too bad I don't have a machine capable of playing the PC version and I'm not going to pay for a console while laid off just to play GTA IV.

I have no doubt that my load testing adventure will continue in January.

One thing for sure. The past seven months has been a waste. Thanks a lot, VT. Same back atcha.

Hey. What's going on in this blog?

2008-11-13T17:22:00.003-06:00

Hey. What's going on in this blog?

Not a damn thing.

*sigh*

Not even working on WiX stuff right now. Working on manually deploying with NAnt, which is fine with me as it is proven that it works time after time. I suspect that MercScum is gonna wanna go back to WiX after initial deployment prototyping. I mean, why stick with something that is proven and works, right?

Almost there... Stay on target...

2008-10-28T16:52:00.002-05:00

Getting closer to getting the first general WiX task done.

I now have WiX creating my user, adding that user to the Administrators group (No comments from the peanut gallery, please!) and then creating a custom AppPool and creating a Web Application that is bound to that custom AppPool with a created VDir. Pretty nifty.

Here is what it looks like so far:

   1:  <Component Id="MySoftwareRequirements" Guid="{SOME-VALID-GUID}">

   2:    <CreateFolder/>

   3:    <util:User Id="MySoftwareUser" Name="MySoftwareUser" Password="supersekrit">

   4:      <util:GroupRef Id="Administrators"/>

   5:    </util:User>

   6:    <iis:WebSite Id          = "DefaultWebSite"

   7:                 Description = "DefaultWebSite"

   8:                 Directory   = "MySoftwareSubDir">

   9:      <iis:WebAddress Id = "AllUnassigned" Port="80" />

  10:        <iis:WebDirProperties Id                    = "WebVirtualDirProperties"

  11:                              Execute               = "yes"

  12:                              Script                = "yes"

  13:                              Read                  = "yes"

  14:                              WindowsAuthentication = "no"

  15:                              AnonymousAccess       = "yes"

  16:                              AnonymousUser         = "MySoftwareUser"

  17:                              IIsControlledPassword = "no" />

  18:    </iis:WebSite>

  19:    <iis:WebAppPool Id="MySoftware" Identity="other" Name="MySoftware" User="MySoftwareUser"/>

  20:    <iis:WebVirtualDir Id            = "MySoftwareVDir"

  21:                       Alias         = "MySoftwareVDir"

  22:                       Directory     = "MySoftwareSubDir"

  23:                       WebSite       = "DefaultWebSite"

  24:                       DirProperties = "WebVirtualDirProperties">

  25:      <iis:WebApplication Id         = "WebApplication"

  26:                          Name       = "MySoftware"

  27:                          WebAppPool = "MySoftware" />

  28:    </iis:WebVirtualDir>

  29:  </Component>

Of course, you can't see the reference to the Administrators group that is up at the top of the .wxs file under the Package element (see previous post). It is starting to come together.

More WiX stuff: Creating a user and adding them to a group.

2008-10-28T12:55:00.002-05:00

Now that I am able to successfully create a new user I need to add that user to the Administrators group. Yeah, I hear you already, "That's a major security issue!" and I would tend to agree but this has been "mandated" by MercScum so there isn't much I can do about it.

Anyhoo. To do it I had do the following.

After the Product and Package element declarations I had to add a Group element, like this:

   1:  <Product Id="{F0C1692F-2D79-4D46-9BDF-DCBAD1DBCDDB}" Language="1033" Manufacturer="SomeManufacturer" Name="SomeProduct" UpgradeCode="{SOME-VALID-GUID}" Version="1.0.0.0">

   2:    <Package Compressed="yes" InstallerVersion="200" />

   3:      <util:Group Id="Administrators" Name="Administrators" />

It took some trial and error to find out where to put the util:Group element. The documentation says that it is a child of the Product element but I couldn't put it directly after. It had to go under the Package element. Good to know!

In my previously declared util:User element I had to nest a GroupRef element so the entire bit of XML reads like this:

   1:  <Component Id="RequiredUser" Guid="{SOME-VALID-GUID}">

   2:    <util:User Id="NewUser" Name="NewUserName" Password="supersekrit">

   3:      <util:GroupRef Id="Administrators"/>

   4:    </util:User>

   5:    <CreateFolder/>

   6:  </Component>

I was able to create the .msi and installed it on a VM and it worked like a charm. Luckily I only banged my head against the WiX brick wall for a few hours this time, unlike the create a new user head-banging. I suspect everything else will start to fall into place now that I am getting more comfortable with WiX and XML in general. Yes, I've managed to be in industry for almost 20 years without having to bow down to the great XML monster but now I am paying the price for being behind the curve. Back in my day we had CSVs and we liked 'em!

Can I has user created by WiX?

2008-10-27T17:26:00.003-05:00

Ok. I'm starting to get this stuff figured out.

The User tag is part of WixUtilExtension.dll but it is not enough to simply reference WixUtilExtension on the command line.

You must also add an XML name space reference in the .wxs file. Also, another gotcha. The namespace spelling is different from the WixUtilExtension spelling. D'oh!

I had to add the following:

   1:  <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"

   2:       xmlns:util="http://schemas.microsoft.com/wix/UtilExtension">

Now, the User tag that I created has to have the "util" prefix to let the linker know that the User tag is being referenced for the WixUtilExtension:

   1:  <Component Id="RequiredUser" Guid="{SUPER-DUPER-GUID}">

   2:    <util:User Id="UserStuff" Name="UserStuff" Password="supersekrit" />

   3:    <CreateFolder/>

   4:  </Component>

Notice that I had to add the CreateFolder tag in my User tag to prevent error ICE18. Otherwise, it'll fail and make me cry.

I was able to link with candle with a "-ext WixUtilExtension.dll" reference and the wixobj was created and with the CreateFolder I avoided the dreaded ICE18 error. I was also able to create my .msi and I will test that tomorrow on my VM.

Wasn't that all intuitively obvious?

My thanks go out to the wix-users listserv that pointed out where I was banging my head against the brick wall of wix. Now with this information I should be able to go forward with the other tasks required and be a tad bit more efficient with WiX (Finally!).

My head hurts...

2008-10-27T14:39:00.004-05:00

*bangs head against desk*

Search for creating users with WiX and you see the handy dandy User tag.

Go ahead and try that with WiX v3. Doesn't work?

Google the documentation. Aha! I need the sca.wixlib and associated sca*.dll files?

Search for files.

Not shipped with v3?

Hmmmm...

Download WiX v2.

Try using with WiX v3.

"The extension 'sca.wixlib' could not be loaded."

*continue to bang head against desk*

Now off to try this again with WiX v2. Can't use heat to harvest files? Guess I need to re-learn the process with tallow.

*continue to bang head against desk*

Do some more Googling. Hey, automated tasks are in the Wix*Extension.dlls. That's what I thought but couldn't get to load with -ext. Hmmmmm?

Some more Googling and reading of blogs. Oh, you can't just use the Wix*Extension.dll by name. Needs full path. Ok. That's not so bad.

Which Wix*Extension.dll has the task for using with the User tag? Dunno. I'll just copy all Wix*Extension.dll to my subdir and try 'em all.

Oops. WinDifxAppExtension.dll can't be loaded? Remove it from command line.

Oops. WixIsolatedAppExtension.dll can't be loaded. Remove it from command line.

Will by User tag be recognized now? *crosses fingers* Let's give it a try!

8<--------------------------------------------------
C:\SomeProduct\SP\Dist>candle -o SP.wixobj SP.wxs -ext WixDifxAppExtension.dll -ext WixDirectXExtension.dll -ext WixFirewallExtension.dl
l -ext WixGamingExtension.dll -ext WixIIsExtension.dll -ext WixMsmqExtension.dll -ext WixNetFxExtension.dll -ext WixPSExtension.dll
-ext WixSqlExtension.dll -ext WixUIExtension.dll -ext WixUtilExtension.dll -ext WixVSExtension.dll
Microsoft (R) Windows Installer Xml Compiler version 3.0.4318.0
Copyright (C) Microsoft Corporation. All rights reserved.

SP.wxs
C:\SomeProduct\SP\Dist\SP.wxs(461) : error CNDL0005 : The Component element contains an unexpected child element 'User'.
-------------------------------------------------->8

Fsck.

*sigh*

Hey. Wasn't I hired to be the performance SME? Hmmmm. What happened to that?

The pain of the WiX

2008-10-24T15:57:00.005-05:00

Much like the use of candles in BDSM play, I am feeling the pain of the WiX wax motif windows installer. But like those candles, I think it'll pay off in the long run.

Once again, into the depths of XML fun! At least it's not XML scripting languages at this point but I am definitely feeling the pain of heat, candle and not seeing the light at the moment. WiX has a very steep learning curve.

So far I've learned how to harvest a bunch of files with heat, link with candle and assemble a .msi with light. With a simple and straightforward project it's actually pretty simple but unfortunately I am not on a simple project.

The first thing that I've found that has burned me is that we have a bunch of duplicated DLLs nested in different subdirs that I want to package as a product and apparently that is a no-no and can't be done. I've had to separate the subdirs generated by MSBuild into two seperate trees with parallel structures and create two .msi packages that when executed create the desired tree.

I banged my head up against that for long enough.

Now I'm trying to figure out how to create a user and that is pretty painful. There is a most excellent older tutorial located here but it doesn't document all the gotchas. I'm trying to find the extra DLL I have to reference via -ext on the command line for candle.exe that recognizes the user tag.

Unlike the nant to MSBuild conversion (which I don't think it was necessary) I figure that learning this WiX stuff will pay off. I mean, if WiX was good enough for MS to deploy SQL Server 2005 and Office 2007 it should do everything I ever want. It's just the learning curve that has to be climbed.

I can really see why deployment/build engineering can be a full time position on a team. I don't want to be a full time build/deployment engineer but I appreciate the knowledge and suspect that it'll pay off in the long run.

From MSBuild to WiX

2008-10-21T23:04:00.001-05:00

Now that I've converted from NAnt to MSBuild scripts I am now learning the wonderful world of WiX to create .msi install packages. What does this have to do with my performance testing duties? Not a damn thing.

The hired mercenary scum that is running the project now is having do all sorts of things that are not part of our normal activities. I was hired to be a performanc SME and to build up a testing framework which is about 80% complete. My last work on that was over a month ago.

I suspect I shall not see another performance test on this project.

NAnt versus MSBuild

2008-10-11T03:09:00.007-05:00

At work we are currently converting from NAnt to MSBuild scripts as we convert from CruiseControl.NET to TeamCity. Why are we changing from NAnt to MSBuild as we change CI servers? Nobody really knows for sure besides the guy that is running the project basically says, "I've mandated it." That's the only reason. Seems like a silly reason to be to go through all the trouble to rewrite all the scripts.

After working with NAnt and MSBuild I have come to one conclusion: I don't like XML based scripting languages. That's right, you heard it here. They are both a pain. Way too verbose and too much of a PITA. My personal belief is that we should just wrap all the functionality with a perl script and be done with it. :-) Just think about it: Simple if/then/else statements and simple loops without all the XML filler junque and all the dynamic data structures you want without a bunch more XML filler stuff. It's beautiful! Back in my day, you darn whipper snappers, we had .CSV files and liked 'em! And like Ant/NAnt, perl works under both Unix and Win32 (I've been using Win32 for a gazillion years but now that I run a 64-bit OS should I start saying Win32/64? That's a discussion for another time! Now back to my MSBuild and NAnt rant!) so you have multiple platform functionality.

MSBuild is taking me longer to wrap my head around. I don't like that I can create a property inside of a target, but that property is not available until the target where it was created as finished. Makes it a pain to attempt to modularize common calls for database work. And why doesn't MSBuild have a foreach construct? Sure, you can use the metabase accessing `%` to invoke multiple calls but how non-intuitive is that? Argh! I shouldn't have to write a helper function to make multiple calls with the % operator.

Yeah. I stand by my convictions on this one. Junque both of 'em and write a perl wrapper and be happy.

Asynchronous Invocation of a Load Test from a Continuous Integration System

2008-09-30T15:13:00.008-05:00

Why the fancy pants title? A Microsoft recruiter asked for my resume for some load testing related stuff and my blog is on my resume so I figured I needed some high falutin' titles. Hopefully they don't read anything but the titles. :-)

Anyway...

I'm getting to the point where I need to be able to invoke a load test after the nightly build and automagically run a load test. Now, I know, right now you're saying to yourself, "Hey, a nightly build doesn't constitute a continuous integration system!" And I agree, however, "Continuous Integration System" sounds much more impressive than just "nightly build." ;-)

Back to the important stuff: How am I going to invoke a Load Test on another machine after the nightly build is done? Right now I am working everything out with that super-duper handy utility, psexec from the UberBrains at SysInternals (now part of Microsoft).

I have recently build a stand alone server that I had installed SQL Server 2005, VS2008 TSTE and my custom LoadTestResults database where I store summary information from runs that I will be reporting from. On that server which I will refer to as my Load Test Controller, I setup a machine account user account by the name of TestRunner. During my trial and errors I found that I had to add TestRunner to the Administrators group but I need to go back and play around with that some to see if there is a more secure way to invoke the nightly automagic load test.

On the Load Test Controller I have two batch files: trigger.bat and loadtest.bat.

trigger.bat:

8<----------------------------
@echo off
echo Cry havoc and let slip loose the dogs of loadtest!
start C:\LoadTest\Trigger\loadtest.bat
---------------------------->8

The start command allows the async execution of the second batch file loadtest.bat, where the load test is really started.

loadtest.bat:

8<----------------------------
call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat"
cd "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\"
MSTest.exe /testcontainer:c:\LoadTest\LoadTests\MyLoadTest\MyLoadTest\MyReallyCoolLoadTest.loadtest
---------------------------->8

I can call trigger.bat from psexec with any automated process that can shell out to the command line or execute a batch file. Here is an example of starting a load test on the Load Test Controller using psexec from the command line:

8<----------------------------
C:\>psexec \\loadtestbox -u testrunner -p testrunner cmd /c c:\loadtest\trigger\trigger.bat

PsExec v1.92 - Execute processes remotely
Copyright (C) 2001-2007 Mark Russinovich
Sysinternals - www.sysinternals.com

Cry havoc and let slip loose the dogs of loadtest!
c:\loadtest\trigger\trigger.bat exited on loadtestbox with error code 0.
---------------------------->8

Even though psexec exit back to the command line, the load test is actually running on my Load Test Controller:

I was having problems getting results to be automagically saved to the Load Test Results Repository. I figured the SQL connection string was stored in the VS2008 solution but I had to open up VS2008 on my Load Test Controller machine and manually add the connection string for my TestRunner profile. After that, the LoadTest database is updated with each execution. w00t!

Now it's off to add the automagic reporting SP calls via osql.

DHCP problems with Win2k8

2008-09-06T14:46:00.004-05:00

Last weekend I brought home the Portable Server of Powah! home to do some work over the weekend and when I fired things up I wasn't getting an IP addressed assigned by the DHCP server on my FreeBSD box that is the gateway to the intarweb here at the house.

I thought that very odd and when I checked /var/log/dhcpd.log it appeared that a request was going out for an IP address and being sent back but was never accepted by Win2k8. How odd...

Here is what I saw in the logs:

8<---------------------------------
Sep 6 14:42:09 firewall dhcpd: DHCPDISCOVER from 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:42:10 firewall dhcpd: DHCPOFFER on 10.0.0.204 to 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:42:13 firewall dhcpd: DHCPDISCOVER from 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:42:13 firewall dhcpd: DHCPOFFER on 10.0.0.204 to 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:42:22 firewall dhcpd: DHCPDISCOVER from 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:42:22 firewall dhcpd: DHCPOFFER on 10.0.0.204 to 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:42:39 firewall dhcpd: DHCPDISCOVER from 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:42:39 firewall dhcpd: DHCPOFFER on 10.0.0.204 to 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
--------------------------------->8

There should have been an DHCPACK after the first two messages. Now this really confused me as I have had the Portable Server of Powah! home many times and had never had any problems getting an IP address before.

But, through playing around I found out that the wireless interface could get an IP address lickitysplit and being the slacker that I am, I kept on trucking with the wifi so that I could get my work done over the holiday.

I once again brought home the Portable Server of Powah! and once again, no IP address. This made me very sad as I wanted to have a wired connection to use the Gigabit goodness, that and it was just odd that I couldn't get an IP address.

I bowed down before the Great Google and queried it's great knowledge and came across this KB. Here is the problem in a nutshell: "Windows Vista cannot obtain an IP address from certain routers or from certain non-Microsoft DHCP servers"

If you follow the directions of the KB the problem can be fixed but they leave out a very important part of the puzzle. Basically, you drill down into the registry to where the various TCP/IP settings are stored and you have to find the GUID of the interface. Since you don't have "friendly" interface names like xl0, xl1 (FreeBSD interface drivers for 3Com NICs) or even friendlier Linux names like eth0, eth1, etc this becomes quite a chore.

Here is what I was looking at:

At this point I'm doing a Kelly-esque "What the hell?" Which one of these GUIDs represents my physical NIC and which one represents my virtual NIC that Hyper-V uses? Do I modify the entry for the physical NIC or the Hyper-V NIC? Confusion abounds!

I manually assigned an IP address and found the GUID entry that represented the Hyper-V virtual NIC and made the changes in the Registry that the above mentioned KB required.

I closed regedit, attempted to do a ipconfig /release && ipconfig /renew and watched the /var/log/dhcpd.log as Win2K8 tried to get an IP address with no love. Then I remembered, this is Windows...

I reboot the machine and sure enough, as it is firing up I see this entry come across /var/log/dhcpd.log:

8<-------------------------------
Sep 6 14:44:48 firewall dhcpd: DHCPDISCOVER from 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:44:49 firewall dhcpd: DHCPOFFER on 10.0.0.204 to 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:44:49 firewall dhcpd: DHCPREQUEST for 10.0.0.204 (10.0.0.1) from 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
Sep 6 14:44:49 firewall dhcpd: DHCPACK on 10.0.0.204 to 10:80:f4:6f:0c:23 (auswipelaptop) via xl0
------------------------------->8

Hazzah! It works!

Apparently during an update something got tweaked and I could no longer pull an IP address from the FreeBSD dhcpd and had to apply this KB hack to get things to work. Or it could have been when I found how to get the Client for Microsoft Networks working with Hyper-V again. No matter, something got tweaked and now it is fixed which is good 'nuff for me. I'm easy like that.

Extracting Perfmon data from the VS2008 Load Test Repository

2008-09-05T22:31:00.004-05:00

Ok. You are performing load tests with VS2008 TSTE and you've enabled the logging of perfmon metrics in your test by tweaking the "Timing Details Storage" property of the Run Settings in your load test. Every 5 seconds the perfmon metrics you specified are being stored to the load test repository and after the run you get some metrics. That's great and all but what if you want more detailed statistics such as average AND standard deviation.

Or better yet! You want at that data to generate graphs of the metrics over the duration of the test. Sure, you can do this by firing up good 'ol Perfmon and logging the values or you can just get them from your load test repository with some SQL. Why have Perfmon logging when VS2008 already does this for you?

I have a need for just such an action and I've been working on the SQL statements to extract the data I need. It's not complete, but this is a major step forward to allowing me to finish up the SPs I want to collect the data dynamically for consumption by managers and other QA folks.

Looking at the handy dandy schema of the load test repository (located here for all to gander) I have created some SQL statements that take care of the leg work for me.

   1:  SELECT LoadTestPerformanceCounterInstance.InstanceId,

   2:         LoadTestPerformanceCounterCategory.CategoryName,

   3:         LoadTestPerformanceCounter.CounterName,

   4:         LoadTestPerformanceCounterInstance.InstanceName, LoadTestPerformanceCounterCategory.MachineName,

   5:         AVG(LoadTestPerformanceCounterSample.ComputedValue) AS AverageValue,

   6:         stdev(LoadTestPerformanceCounterSample.ComputedValue) as StdDev,

   7:         count(LoadTestPerformanceCounterSample.ComputedValue) as Count

   8:  FROM  LoadTestPerformanceCounterCategory INNER JOIN

   9:                 LoadTestPerformanceCounter INNER JOIN

  10:                 LoadTestPerformanceCounterInstance ON LoadTestPerformanceCounter.LoadTestRunId = LoadTestPerformanceCounterInstance.LoadTestRunId AND

  11:                 LoadTestPerformanceCounter.CounterId = LoadTestPerformanceCounterInstance.CounterId ON

  12:                 LoadTestPerformanceCounterCategory.CounterCategoryId = LoadTestPerformanceCounter.CounterCategoryId AND

  13:                 LoadTestPerformanceCounterCategory.LoadTestRunId = LoadTestPerformanceCounter.LoadTestRunId INNER JOIN

  14:                 LoadTestPerformanceCounterSample ON LoadTestPerformanceCounterInstance.LoadTestRunId = LoadTestPerformanceCounterSample.LoadTestRunId AND

  15:                 LoadTestPerformanceCounterInstance.InstanceId = LoadTestPerformanceCounterSample.InstanceId

  16:  WHERE (LoadTestPerformanceCounter.LoadTestRunId = 83)

  17:  GROUP BY LoadTestPerformanceCounterCategory.CategoryName, LoadTestPerformanceCounter.CounterName, LoadTestPerformanceCounterInstance.InstanceName,

  18:                 LoadTestPerformanceCounterInstance.InstanceId, LoadTestPerformanceCounterCategory.MachineName

  19:  ORDER BY LoadTestPerformanceCounterInstance.InstanceId

In the above example, the reference to LoadTestRunId is hard coded to a previous run that I have in my results repository. For all my other SPs I use the GUID of the test and will convert the above SQL statement to reference GUID all in good time. The query above returns results like this:

   1:  0     .NET CLR Memory     Gen 0 heap size                             w3wp     10.0.0.11     944824777.209431     185400315.203616     721

   2:  1     .NET CLR Memory     Large Object Heap size                      w3wp     10.0.0.11     86994000.5298197     29625345.9051784     721

   3:  2     .NET CLR Memory     # Gen 0 Collections                         w3wp     10.0.0.11     132.608876560333     69.5129454689118     721

   4:  3     .NET CLR Memory     Gen 2 heap size                             w3wp     10.0.0.11     197119053.001387     78824416.0154383     721

   5:  4     .NET CLR Memory     Allocated Bytes/sec                         w3wp     10.0.0.11     63521791.4160888     91335026.9218956     721

   6:  5     .NET CLR Memory     # Gen 2 Collections                         w3wp     10.0.0.11     29.5104022191401     12.1351287581502     721

   7:  6     .NET CLR Memory     Promoted Memory from Gen 0                  w3wp     10.0.0.11     42540355.2815534     13356693.0175676     721

   8:  7     .NET CLR Memory     # Induced GC                                w3wp     10.0.0.11     0                    0                    721

   9:  8     .NET CLR Memory     Gen 0 Promoted Bytes/Sec                    w3wp     10.0.0.11     418557.084366331     1120132.98732717     721

  10:  9     .NET CLR Memory     Promoted Memory from Gen 1                  w3wp     10.0.0.11     21740368.0721221     25126733.8606902     721

  11:  10    .NET CLR Memory     # GC Handles                                w3wp     10.0.0.11     3750.73370319001     216.191892870829     721

  12:  11    .NET CLR Memory     # Gen 1 Collections                         w3wp     10.0.0.11     70.0693481276005     35.4698332819129     721

  13:  12    .NET CLR Memory     Gen 1 heap size                             w3wp     10.0.0.11     62394980.9237171     31034211.4233198     721

  14:  13    .NET CLR Memory     Finalization Survivors                      w3wp     10.0.0.11     8217.1040221914      3024.02383909951     721

  15:  14    .NET CLR Memory     Promoted Finalization-Memory from Gen 0     w3wp     10.0.0.11     11415345.6768377     11434973.2174913     721

  16:  15    .NET CLR Memory     % Time in GC                                w3wp     10.0.0.11     3.48545611573799     6.73310861817717     721

The first column is the InstanceId of the perfmon counter and the information for the counter is split up into three separate columns (CategoryName, CounterName and InstanceName) but the column that ties them all together is the InstanceId.

Ok. That's fine and dandy but what about graphing the metrics over the duration of the load test?

Well, that's another SQL query I wrote that is in the alpha stage of development. In the above example I displayed 15 out of 466 metrics that were logged. Let's choose to graph the .NET CLR Memory Gen 2 heap size for w3wp. This particular metric happens to be InstanceId #3 this time around and we can use it in the SQL statement below to extract the data that can eventually be used to graph the results.

   1:  SELECT LoadTestPerformanceCounterInstance.InstanceName,

   2:         LoadTestPerformanceCounterSample.ComputedValue

   3:  FROM  LoadTestPerformanceCounterInstance INNER JOIN

   4:        LoadTestPerformanceCounterSample ON LoadTestPerformanceCounterInstance.LoadTestRunId = LoadTestPerformanceCounterSample.LoadTestRunId AND

   5:        LoadTestPerformanceCounterInstance.InstanceId = LoadTestPerformanceCounterSample.InstanceId

   6:  WHERE (LoadTestPerformanceCounterInstance.LoadTestRunId = 83) AND (LoadTestPerformanceCounterInstance.InstanceId = 3)

   7:  ORDER BY LoadTestPerformanceCounterSample.TestRunIntervalId

This query yields the results of:

   1:  w3wp    0

   2:  w3wp    0

   3:  w3wp    96

   4:  w3wp    3231264

   5:  w3wp    2.170597E+07

   6:  w3wp    2.170597E+07

   7:  w3wp    2.170597E+07

   8:  w3wp    2.976022E+07

   9:  w3wp    2.976022E+07

  10:  w3wp    2.976022E+07

  11:  w3wp    2.976022E+07

  12:  w3wp    4.127677E+07

  13:  w3wp    4.127677E+07

  14:  w3wp    4.127677E+07

and so on and so on until all 722 metrics are displayed. We can take those 722 metrics and copypasta into Excel and generate a quick and dirty graph:

Viola! Quick and dirty graph from Excel!

I'll need to change the column output to show more details on the metrics to display all three columns to better identify the perfmon metric and also display the time that the metric was taken.

After I do all that I'll need to write some code to generate graphs on the fly. I'm thinking that this project on SourceForge will fit the bill nicely and there seems to be a bunch of examples out on the 'net how to use ZedGraph to it's fullest.

Oh yeah! Almost forgot. I need to double check the results between the first query and the individual information for InstanceId 3 metric (the .NET CLR Gen 2 heap size for w3wp). The first query output says the average value is 197119053.001387. Take the average of output that I did the copypasta into Excel gonkulates to an average of 197119053.4. The difference between the two? A measly 0.355. I'm thinking that is good 'nuff for government work.

Now I just need to finish up the queries and get them into SPs and integrate them with the other code that I've written that does automagic before/after comparison of transaction response times (more on that at a later date).

Using NAnt custom task to wrap SDC Task to create VDir

2008-09-02T13:00:00.006-05:00

I got my VDir code working, but there are some caveats that have to be watched, I've found.

   1:  using System;

   2:  using System.Text;

   3:  using NAnt.Core;

   4:  using NAnt.Core.Attributes;

   5:  using Microsoft.Sdc.Tasks;

   6:  using Microsoft.Build.Utilities;

   7:  using Microsoft.Build.Framework;

   8:  namespace Auswipe.NAntTasks.VDirTasks {

   9:    # region Delete VDir

  10:    [TaskName("deletevdir")]

  11:    public class DeleteVDir : NAnt.Core.Task {

  12:      private string vDirToDelete;

  13:      private string machineName;

  14:      private string webSiteName;

  15:      [TaskAttribute("vdirtodelete", Required = true)]

  16:      [StringValidator(AllowEmpty = false)]

  17:      public string VDirToDelete {

  18:        get { return vDirToDelete; }

  19:        set { vDirToDelete = value; }

  20:      }

  21:      [TaskAttribute("machinename", Required = true)]

  22:      [StringValidator(AllowEmpty = false)]

  23:      public string MachineName {

  24:        get { return machineName; }

  25:        set { machineName = value; }

  26:      }

  27:      [TaskAttribute("websitename", Required = true)]

  28:      [StringValidator(AllowEmpty = false)]

  29:      public string WebSiteName {

  30:        get { return webSiteName; }

  31:        set { webSiteName = value; }

  32:      }

  33:      protected override void ExecuteTask() {

  34:        Project.Log(Level.Info, "Auswipe.NAntTasks.VDirTasks : The following AppPool will be deleted: '" + vDirToDelete + "'");

  35:        try {

  36:          Microsoft.Sdc.Tasks.Web.WebSite.DeleteVirtualDirectory delVDirObject = new Microsoft.Sdc.Tasks.Web.WebSite.DeleteVirtualDirectory();

  37:          delVDirObject.MachineName          = machineName;

  38:          delVDirObject.VirtualDirectoryName = vDirToDelete;

  39:          delVDirObject.WebSiteName          = webSiteName;

  40:          if (delVDirObject.Execute()) {

  41:            Project.Log(Level.Info, "Auswipe.NAntTasks.VDirTasks : The VDir '" + vDirToDelete + "' was successful deleted.");

  42:          } else {

  43:            Project.Log(Level.Error, "Auswipe.NAntTasks.VDirTasks : ERROR: The VDir '" + vDirToDelete + "' was NOT deleted but an exception was not thrown.");

  44:          };

  45:        } catch (Exception e) {

  46:          Project.Log(Level.Error, "Auswipe.NAntTasks.VDirTasks : VDir deletion failed with the resulting exception:");

  47:          Project.Log(Level.Error, e.ToString());

  48:        };

  49:      }

  50:    }

  51:    #endregion

  52:    # region Create VDir

  53:    [TaskName("createvdir")]

  54:    public class CreateVDir : NAnt.Core.Task {

  55:      private string vDirName;

  56:      private string webSiteName;

  57:      private string appPoolName;

  58:      private string machineName;

  59:      private string fullPath;

  60:      [TaskAttribute("apppoolname", Required = true)]

  61:      [StringValidator(AllowEmpty = false)]

  62:      public string AppPoolName {

  63:        get { return appPoolName; }

  64:        set { appPoolName = value; }

  65:      }

  66:      [TaskAttribute("vdirname", Required = true)]

  67:      [StringValidator(AllowEmpty = false)]

  68:      public string VDirName {

  69:        get { return vDirName; }

  70:        set { vDirName = value; }

  71:      }

  72:      [TaskAttribute("website", Required = true)]

  73:      [StringValidator(AllowEmpty = false)]

  74:      public string WebSite {

  75:        get { return webSiteName; }

  76:        set { webSiteName = value; }

  77:      }

  78:      [TaskAttribute("machinename", Required = true)]

  79:      [StringValidator(AllowEmpty = false)]

  80:      public string MachineName {

  81:        get { return machineName; }

  82:        set { machineName = value; }

  83:      }

  84:      [TaskAttribute("path", Required = true)]

  85:      [StringValidator(AllowEmpty = false)]

  86:      public string FullPath {

  87:        get { return fullPath; }

  88:        set { fullPath = value; }

  89:      }

  90:      protected override void ExecuteTask() {

  91:        Project.Log(Level.Info, "Auswipe.NAntTasks.VDirTasks : The following VDir will be created: '" + vDirName + "'");

  92:        try {

  93:          Microsoft.Sdc.Tasks.Web.WebSite.CreateVirtualDirectory createVDirObject = new Microsoft.Sdc.Tasks.Web.WebSite.CreateVirtualDirectory();

  94:          createVDirObject.AppCreate            = true;

  95:          createVDirObject.AppPoolId            = appPoolName;

  96:          createVDirObject.MachineName          = machineName;

  97:          createVDirObject.VirtualDirectoryName = vDirName;

  98:          createVDirObject.WebSiteName          = webSiteName;

  99:          createVDirObject.Path                 = fullPath;

 100:          if (createVDirObject.Execute()) {

 101:            Project.Log(Level.Info, "Auswipe.NAntTasks.VDirTasks : The VDir '" + vDirName + "' was successfuly created.");

 102:          } else {

 103:            Project.Log(Level.Error, "Auswipe.NAntTasks.VDirTasks : ERROR: The VDir '" + vDirName + "' was NOT created but an exception was not thrown.");

 104:          };

 105:        } catch (Exception e) {

 106:          Project.Log(Level.Error, "Auswipe.NAntTasks.VDirTasks : VDir creation failed with the resulting exception:");

 107:          Project.Log(Level.Error, e.ToString());

 108:        };

 109:      }

 110:    }

 111:    #endregion

 112:  }

A caveat that I've found though is that even though the invoked SDC Task might run into a problem during the .Execute call, .Execute still returns a true so my NAnt task doesn't know if the process has ran into a problem.

In the example below, I'm trying to create a VDir with an AppPool that doesn't exist:

8<-------------------------------------------
NAnt 0.85 (Build 0.85.2478.0; release; 10/14/2006)
Copyright (C) 2001-2006 Gerry Shaw
http://nant.sourceforge.net

Buildfile: file:///C:/customtask/default.build
Target framework: Microsoft .NET Framework 2.0
Target(s) specified: test-task

[loadtasks] Scanning assembly "Auswipe.NAntTasks.VDirTasks" for extensions.
[loadtasks] Scanning assembly "Auswipe.NAntTasks.AppPoolTasks" for extensions.

test-task:

[echo] Creating VDir
Auswipe.NAntTasks.VDirTasks : The following VDir will be created: 'someVDir'
Creating virtual directory "someVDir".
A task error has occured.
Message = Exception has been thrown by the target of an invocation.
VirtualDirectoryName = someVDir
Path = c:\somewebsite
MachineName = Auswipel-rfs2we
WebSiteName = Default Web Site
WebAppName =
AppPoolId = AppPoolDoesNotExist
AppCreate = True
AnonymousUserName =
AnonymousUserPassword =
UncUserName =
UncPassword =
AuthFlags =
AccessFlags =

at System.DirectoryServices.DirectoryEntry.Invoke(String methodName, Object[] args)
at Microsoft.Sdc.Tasks.Configuration.Web.VirtualDirectory.AppCreate3() in c:\projects\codeplex\sdctasks\Solutions\Main\Tasks\Configuration\Web\VirtualDirectory.cs:line 324
at Microsoft.Sdc.Tasks.Web.WebSite.CreateVirtualDirectory.InternalExecute() in c:\projects\codeplex\sdctasks\Solutions\Main\Tasks\Web\WebSite\CreateVirtualDirectory.cs:line 315
at Microsoft.Sdc.Tasks.TaskBase.Execute() in c:\projects\codeplex\sdctasks\Solutions\Main\Tasks\TaskBase.cs:line 66
Element not found. (Exception from HRESULT: 0x80070490)

PseudoEngineException
at Microsoft.Sdc.Tasks.PseudoBuildEngine.LogErrorEvent(BuildErrorEventArgs eventArgs) in c:\projects\codeplex\sdctasks\Solutions\Main\Tasks\PseudoBuildEngine.cs:line 77
at Microsoft.Build.Utilities.TaskLoggingHelper.LogError(String subcategory, String errorCode, String helpKeyword, String file, Int32 lineNumber, Int32 columnNumber, Int32 endLineNumber, Int32 endColumnNumber, String message, Object[] messageArgs)
at Microsoft.Build.Utilities.TaskLoggingHelper.LogError(String message, Object[] messageArgs)
at Microsoft.Sdc.Tasks.TaskBase.Execute() in c:\projects\codeplex\sdctasks\Solutions\Main\Tasks\TaskBase.cs:line 95
Auswipe.NAntTasks.VDirTasks : VDir creation failed with the resulting exception:
System.ApplicationException: PseudoEngineException
at Microsoft.Sdc.Tasks.PseudoBuildEngine.LogErrorEvent(BuildErrorEventArgs eventArgs) in c:\projects\codeplex\sdctasks\Solutions\Main\Tasks\PseudoBuildEngine.cs:line 77
at Microsoft.Build.Utilities.TaskLoggingHelper.LogError(String subcategory, String errorCode, String helpKeyword, String file, Int32 lineNumber, Int32 columnNumber, Int32 endLineNumber, Int32 endColumnNumber, String message, Object[] messageArgs)
at Microsoft.Build.Utilities.TaskLoggingHelper.LogErrorFromException(Exception exception, Boolean showStackTrace, Boolean showDetail, String file)
at Microsoft.Build.Utilities.TaskLoggingHelper.LogErrorFromException(Exception exception, Boolean showStackTrace)
at Microsoft.Sdc.Tasks.TaskBase.Execute() in c:\projects\codeplex\sdctasks\Solutions\Main\Tasks\TaskBase.cs:line 99
at Auswipe.NAntTasks.VDirTasks.CreateVDir.ExecuteTask()

BUILD SUCCEEDED - 2 non-fatal error(s), 0 warning(s)

Total time: 0.2 seconds.
------------------------------------------->8

You would think that .Execute should return a False for the call but it doesn't. Notice the verbage of "2 non-fatal error(s)" so it appears that the SDC Task does not treat the non-existant Applicaiton Pool as a fatal error. Good to know!

Going back and looking at the IIS Manager we see this for the create VDir:

Notice how the Application Pool is set to "<Invalid Application Pool>."

So, the VDir was created, despite the lack of a valid Application Pool. That's something to be aware of when testing out NAnt tasks that wrap SDC Task functionality.

Hey, whaddya know? It works!

2008-08-31T03:18:00.006-05:00

I finished the first part of my task to create a custom NAnt task that utilizes the SDC Tasks that were originally built for MSBuild. What I am interested in is deleting and creating custom AppPools for IIS as well as deleting and creating vdirs bound to custom AppPools and not DefaultAppPool.

I did some quick and dirty coding today and came up with this first pass:

   1:  using System;

   2:  using System.Text;

   3:  using NAnt.Core;

   4:  using NAnt.Core.Attributes;

   5:  using Microsoft.Sdc.Tasks;

   6:  using Microsoft.Build.Utilities;

   7:  using Microsoft.Build.Framework;

   8:  namespace Auswipe.NAntTasks.AppPoolTasks {

   9:    # region DeleteAppPool

  10:    [TaskName("deleteapppool")]

  11:    public class DeleteAppPool : NAnt.Core.Task {

  12:      private string appPoolName;

  13:      private string machineName;

  14:      [TaskAttribute("apppoolname", Required = true)]

  15:      [StringValidator(AllowEmpty = false)]

  16:      public string AppPoolName {

  17:        get { return appPoolName; }

  18:        set { appPoolName = value; }

  19:      }

  20:      [TaskAttribute("machinename", Required = true)]

  21:      [StringValidator(AllowEmpty = false)]

  22:      public string MachineName {

  23:        get { return machineName; }

  24:        set { machineName = value; }

  25:      }

  26:      protected override void ExecuteTask() {

  27:        Project.Log(Level.Info, "Auswipe.NAntTasks.AppPoolTasks : The following AppPool will be deleted: '" + appPoolName + "'");

  28:        try {

  29:          Microsoft.Sdc.Tasks.Web.DeleteAppPool.DeleteAppPool delAppPoolObject = new Microsoft.Sdc.Tasks.Web.DeleteAppPool.DeleteAppPool();

  30:          delAppPoolObject.AppPoolName = appPoolName;

  31:          delAppPoolObject.MachineName = machineName;

  32:          if (delAppPoolObject.Execute()) {

  33:            Project.Log(Level.Info, "Auswipe.NAntTasks.AppPoolTasks : The AppPool '" + appPoolName + "' was successful deleted.");

  34:          } else {

  35:            Project.Log(Level.Error, "Auswipe.NAntTasks.AppPoolTasks : ERROR: The AppPool '" + appPoolName + "' was NOT deleted but an exception was not thrown.");

  36:          };

  37:        } catch (Exception e) {

  38:          Project.Log(Level.Error, "Auswipe.NAntTasks.AppPoolTasks : AppPool deletion failed with the resulting exception:");

  39:          Project.Log(Level.Error, e.ToString());

  40:        };

  41:      }

  42:    }

  43:    #endregion

  44:    # region Create AppPool

  45:    [TaskName("createapppool")]

  46:    public class CreateAppPool : NAnt.Core.Task {

  47:      private string appPoolName;

  48:      private string userName;

  49:      private string passWord;

  50:      private string userDomain;

  51:      private string machineName;

  52:      [TaskAttribute("apppoolname", Required = true)]

  53:      [StringValidator(AllowEmpty = false)]

  54:      public string AppPoolName {

  55:        get { return appPoolName; }

  56:        set { appPoolName = value; }

  57:      }

  58:      [TaskAttribute("username", Required = true)]

  59:      [StringValidator(AllowEmpty = false)]

  60:      public string UserName {

  61:        get { return userName; }

  62:        set { userName = value; }

  63:      }

  64:      [TaskAttribute("password", Required = true)]

  65:      [StringValidator(AllowEmpty = false)]

  66:      public string PassWord {

  67:        get { return passWord; }

  68:        set { passWord = value; }

  69:      }

  70:      [TaskAttribute("machinename", Required = true)]

  71:      [StringValidator(AllowEmpty = false)]

  72:      public string MachineName {

  73:        get { return machineName; }

  74:        set { machineName = value; }

  75:      }

  76:      protected override void ExecuteTask() {

  77:        Project.Log(Level.Info, "Auswipe.NAntTasks.AppPoolTasks : The following AppPool will be created: '" + appPoolName + "'");

  78:        try {

  79:          Microsoft.Sdc.Tasks.Web.AppPool.Create createAppPoolObject = new Microsoft.Sdc.Tasks.Web.AppPool.Create();

  80:          createAppPoolObject.AppPoolName  = appPoolName;

  81:          createAppPoolObject.IdentityType = "SpecifiedUserAccount";

  82:          createAppPoolObject.Identity     = userName;

  83:          createAppPoolObject.Password     = passWord;

  84:          createAppPoolObject.MachineName  = machineName;

  85:          if (createAppPoolObject.Execute()) {

  86:            Project.Log(Level.Info, "Auswipe.NAntTasks.AppPoolTasks : The AppPool '" + appPoolName + "' was successfuly created.");

  87:          } else {

  88:            Project.Log(Level.Error, "Auswipe.NAntTasks.AppPoolTasks : ERROR: The AppPool '" + appPoolName + "' was NOT created but an exception was not thrown.");

  89:          };

  90:        } catch (Exception e) {

  91:          Project.Log(Level.Error, "Auswipe.NAntTasks.AppPoolTasks : AppPool creation failed with the resulting exception:");

  92:          Project.Log(Level.Error, e.ToString());

  93:        };

  94:      }

  95:    }

  96:    #endregion

  97:  }

After I created my Auswipe.NAntTasks.AppPoolTasks.dll I had to copy both Auswipe.NAntTasks.AppPoolTasks.dll and Microsoft.Sdc.Tasks.dll to the subdir where I put my default.build that I've been playing with so that the code could properly execute and create my AppPool.

Here is my default.build script for NAnt:

8<-----------------------------------------

1:

   2:  <?xml version="1.0"?>

   3:  <project name="Task Test" default="test-task" basedir="." xmlns="http://nant.sf.net/release/0.86-beta1/nant.xsd">

4:

   5:    <loadtasks assembly="Auswipe.NAntTasks.AppPoolTasks.dll" />

6:

   7:    <target name="test-task">

   8:      <echo message="Deleting AppPool."/>

   9:      <deleteapppool apppoolname = "AuswipeAppPool"

  10:                     machinename = "Auswipel-rfs2we"

  11:       />

12:

  13:      <echo message="Create AppPool."/>

  14:      <createapppool apppoolname = "AuswipeAppPool"

  15:                     username    = "AuswipeAppPoolUser"

  16:                     password    = "supersekrit"

  17:                     machinename = "Auswipel-rfs2we"

  18:      />

  19:    </target>

20:

  21:  </project>

----------------------------------------->8

And here is the output from the execution:

8<-----------------------------------------

   1:  C:\customtask>nant /f:default.build test-task

   2:  NAnt 0.85 (Build 0.85.2478.0; release; 10/14/2006)

   3:  Copyright (C) 2001-2006 Gerry Shaw

   4:  http://nant.sourceforge.net

5:

   6:  Buildfile: file:///C:/customtask/default.build

   7:  Target framework: Microsoft .NET Framework 2.0

   8:  Target(s) specified: test-task

9:

  10:  [loadtasks] Scanning assembly "Auswipe.NAntTasks.AppPoolTasks" for extensions.

11:

  12:  test-task:

13:

  14:       [echo] Deleting AppPool.

  15:  Auswipe.NAntTasks.AppPoolTasks : The following AppPool will be deleted: 'AuswipeAppPool'

  16:  Auswipe.NAntTasks.AppPoolTasks : The AppPool 'AuswipeAppPool' was successful deleted.

  17:       [echo] Create AppPool.

  18:  Auswipe.NAntTasks.AppPoolTasks : The following AppPool will be created: 'AuswipeAppPool'

  19:  Creating app pool "AuswipeAppPool".

  20:  Auswipe.NAntTasks.AppPoolTasks : The AppPool 'AuswipeAppPool' was successfuly created.

21:

  22:  BUILD SUCCEEDED

23:

  24:  Total time: 0.1 seconds.

----------------------------------------->8

Here we see a screen shot of the system before NAnt execution:

Now I execute the NAnt script and re-fresh the IIS manager view of AppPools and lo-and-behold! It worked!

I checked the properties of the newly created AppPool and we see that it has the credentials I specified in the default.build. Hazzah!

Now the next step is to go forth and code the deletion and creation of vdirs with the SDC Tasks.

Nifty!

Creating my own custom NAnt tasks in C#

2008-08-29T23:48:00.001-05:00

One of the things that I find myself doing when not crunching numbers as a performance enginerd is learning the ways of the Buildmiester. We use NAnt for deploying our code from the build server to a target test machine and UAT test environment.

During the process we use NAnt to delete and create vdirs under IIS with mkiisdir which is handy dandy and all but it doesn't fully get the job done as some of the vdirs we need should not belong to DefaultAppPool. And also unfortunately it appears that NAntContrib doesn't have all the tools that I need to delete and create a custom AppPool that I need to create with a special local username/password.

I found the command line code to create the AppPool that I want and I was just going to wrap the execution of the command line stuff with an <exec> task but I still couldn't find how to create a vdir on the command line and have it binding to my custom AppPool that I wanted.

This morning after Scrum another coder working on a sister project that uses MSBuild instead of NAnt suggested that I take a gander at the SDC Tasks for MSBuild up on CodePlex. After pulling down the latest and greatest SDC Tasks it appears that I can delete, then create my custom AppPool and then delete/create a customer vdir bound to my custom AppPool. How great is that?

Except that SDC Tasks was designed for MSBuild and we are already heavily into NAnt. Bummer, right? Well. I know that NAnt has a <msbuild> task but going from NAnt into MSBuild that would reference a custom MSBuild script and pass the params around seemed like a real PITA, not to mention a documentation nightmare. But then again, I really want this functionality of the SDC Tasks.

What do I do? What do I do?

After consulting the Great Google, I decided that it was time to "Man up, Nancy Boy!" as one of the female trainers at my gym would say (actually shout during a middle of a grueling spin class with the grunting and sweating and the pain!) and learn how to code my own custom NAnt task and build a custom wrapper for the SDC stuff so that I could invoke the functionality from within my NAnt script.

After consulting some blogs and some pain and suffering I've been able to create a basic useless NAnt task. I haven't written the wrapper code yet, but I think that'll be simple compared to actually getting a custom NAnt task up and running.

This is what I did to create my very own useless NAnt task.

First, I created a .NET Class project by the name of "NAntCustomAuswipeTasks." It seems to be very important that the assembly name postfix be "Tasks.dll." The second thing is that it is important to use class decorators for specifying various things that NAnt needs to get the job done. Two of those being "TaskName" and "TaskAttribute."

Another important thing that I learned is that when calling the task from inside the NAnt script that the task name is case sensitive to the decorator string.

Below is my code for my useless NAnt task:

   1:  using System;

   2:  using NAnt.Core;

   3:  using NAnt.Core.Attributes;

   4:  namespace NAntCustomAuswipeTasks {

   5:    [TaskName("auswipetask")]

   6:    public class AuswipeTask : Task {

   7:      private string customData;

   8:      [TaskAttribute("customdata", Required = true)]

   9:      [StringValidator(AllowEmpty = false)]

  10:      public string CustomData {

  11:        get {

  12:          return customData;

  13:        }

  14:        set {

  15:          customData = value;

  16:        }

  17:      }

  18:      protected override void ExecuteTask() {

  19:        Project.Log(Level.Info, "[AuwipeTask] The parameter passed is '" + customData + "'.");

  20:      }

  21:    }

  22:  }

Take notice of the TaskName, TaskAttribute and StringValidator decorators. Very important or otherwise your attempts to call your custom NAnt task will fail. The blog entry that I was learning on didn't have the TaskName decorator and so my call to my task was failing. Very frustrating.

After compiling into my assembly of "NAntCustomAuswipeTasks.dll" I copied the DLL into the same subdir where my NAnt build file resides. In my case, this is a folder off of my Desktop subdir where I am playing around checking to see if it really works. I'm using a default.build file right now just to make things simple:

8<--------------------------------
C:\Users\Auswipe\Desktop\NAntTask>dir
Volume in drive C has no label.
Volume Serial Number is A463-3DC5

Directory of C:\Users\Auswipe\Desktop\NAntTask

08/29/2008 11:44 PM <DIR> .
08/29/2008 11:44 PM <DIR> ..
08/29/2008 11:44 PM 404 default.build
08/29/2008 11:45 PM 4,608 NAntCustomAuswipeTasks.dll
2 File(s) 5,012 bytes
2 Dir(s) 332,392,656,896 bytes free
-------------------------------->8

8<--------------------------------

   1:  <?xml version="1.0"?>

   2:  <project name="Custom NAnt Task Test" default="test-task" basedir="." xmlns="http://nant.sf.net/release/0.86-beta1/nant.xsd">

   3:    <loadtasks assembly="NAntCustomAuswipeTasks.dll" />

   4:    <target name="test-task">

   5:      <echo message="Here goes nothing...." />

   6:      <auswipetask customdata="Killroy was here." />

   7:    </target>

   8:  </project>

-------------------------------->8

And here I am testing out my useless NAnt task:

8<--------------------------------
C:\Users\auswipe\Desktop\NAntTask>nant /f:default.build test-task
NAnt 0.85 (Build 0.85.2478.0; release; 10/14/2006)
Copyright (C) 2001-2006 Gerry Shaw
http://nant.sourceforge.net

Buildfile: file:///C:/Users/auswipe/Desktop/NAntTask/default.build
Target framework: Microsoft .NET Framework 2.0
Target(s) specified: test-task

[loadtasks] Scanning assembly "NAntCustomAuswipeTasks" for extensions.

test-task:

[echo] Here goes nothing....
[AuwipeTask] The parameter passed is 'Killroy was here.'.

BUILD SUCCEEDED

Total time: 0 seconds.

-------------------------------->8

Notice that I go ahead and specify the default.build file on the command line along with the target despite having the test-task as the defacto target of default.build and knowing full well that NAnt looks for default.build. Why'd I do that? Just because I hate seeing demonstrations of some technology where the author assumes that the readers know all the ins and out of the subject. Sure, I can get the same results with `nant` as I did with `nant /f:default.build test-task` but having been in the RTFM mode before, I can gleam more information from `nant /f:default.build test-task` and learn the rest later. But that is just my own opinion. YMMV, IANAL, ROFLWTFBBQ.

So, now that I've created my useless NAnt task successfully I can work on wrapping the SDC Tasks that I need to make my world (or at least my deploy) a better place. And the less time I'm spending cleaning up after a deploy changing settings is more time I can be sitting on my butt surfing the web, or something like that.

Interesting blog post about Apdex

2008-08-28T16:22:00.001-05:00

Here is an interesting post from all around smart dude, Tim Koopmans at 90kts.com concerning reporting response time to managers and other non-techy folks and it is interesting.

Check it out here.

Basically, it is a way to gonkulate a value and color code scheme to show if the response time for a transaction falls within acceptable limits for the type of transaction.

Checking out http://www.apdex.org/using.html you can see some of the target response times for the Apdex coefficient gonkulations.

This might be a better a cool way to supplement the way that I am planning on comparing before/after tests with Z-score values and a lot easier for management folk.

w00t! I can keep Hyper-V after all!

2008-08-14T13:17:00.000-05:00

I can keep Hyper-V after all! How did I get back Client for Microsoft Networking do you ask?

In my intarweb searchings I came across this blog entry.

Here is the solution in a nutshell.

Blow away any pre-existing virtual NICs with the Hyper-V manager. With your physical NIC go ahead and un-bind all protocols. Once that is done, go back into Hyper-V manager and re-create a virtual NIC and viola, you'll have a virtual NIC with Client for Microsoft Networks bound. Just reboot your machine and it should be good to go.

Back to using Hyper-V for prototyping! Yay!

Thanks, Jeff! You rock!

Coding like Ike and Tina...

2008-08-07T20:23:00.002-05:00

I'm working on a WebTestRequest that changes an address and in the post back the POST data contains a bunch of information that is displayed in the HTML and I need to extract data from text boxes, check boxes and drop down lists. Extracting the textboxes and checkboxes are pretty simple, it is the drop down boxes that are a pain.

The .NET HTML parser treats the HtmlTags that reside inside the SELECT tag separately so I cannot just get the attributes for the select and then slice and dice and get the default selected value for the drop in.

I tried to be "elegant" in my solution and use the HTML Agility Pack and a XPath pulled from FireBug but alas, there was no love to be found. I'm not sure what the exact problem is other than me being a total XPath n00b (I normally use an open source Java project to extract XPaths from XML docs) but something was amiss. After wasting several hours trying to get the elegant solution I decided to be like Ike and Tina and just brute force the solution of identifying the default selected value for the drop down lists. I populate the Context of the WebTestRequest with all the values that are extracted and just reference them by element name in the address change POST ala the HIDDEN1$ that VS2008 does for hidden field extraction and it seems to work pretty good.

Here is the extraction code that I wrote to pull the fields that I needed to reference in the Context:

   1:  void ExtractCustomerDataToContext(object sender, ExtractionEventArgs e) {

   2:    Dictionary<string, string> customerData = new Dictionary<string, string>();

   3:    string lastDropDown = null;

   4:    foreach (HtmlTag tag in e.Response.HtmlDocument.HtmlTags) {

   5:      string type = tag.GetAttributeValueAsString("type");

   6:      if (type != null) {

   7:        lastDropDown = null ;

   8:        if (type == "text" || type == "checkbox") {

   9:          if (tag.GetAttributeValueAsString("checked") != null) {

  10:            if (tag.GetAttributeValueAsString("checked") == "checked") {

  11:              customerData.Add(tag.GetAttributeValueAsString("name"), "on");

  12:            } else {

  13:              customerData.Add(tag.GetAttributeValueAsString("name"), "off");

  14:            };

  15:          } else {

  16:            customerData.Add(tag.GetAttributeValueAsString("name"), tag.GetAttributeValueAsString("value"));

  17:          };

  18:        };

  19:      } else {

  20:        if (tag.GetAttributeValueAsString("class") != null) {

  21:          if (tag.GetAttributeValueAsString("class") == "dropDownListDefault") {

  22:            lastDropDown = tag.GetAttributeValueAsString("name");

  23:          };

  24:        } else {

  25:          if (tag.GetAttributeValueAsString("selected") != null) {

  26:            if (lastDropDown != null) {

  27:              customerData.Add(lastDropDown, tag.GetAttributeValueAsString("value"));

  28:            };

  29:          };

  30:        };

  31:      };

  32:    };

  33:    foreach (KeyValuePair<string, string> kvp in customerData) {

  34:      if (this.Context.ContainsKey("CUSTOMER_UPDATE_DATA." + kvp.Key)) {

  35:        this.Context.Remove("CUSTOMER_UPDATE_DATA." + kvp.Key);

  36:      };

  37:      if (kvp.Value == null) {

  38:        this.Context.Add("CUSTOMER_UPDATE_DATA." + kvp.Key, "");

  39:      } else {

  40:        this.Context.Add("CUSTOMER_UPDATE_DATA." + kvp.Key, kvp.Value);

  41:      };

  42:    };

  43:  }

So, if the previous page HTML has a text field by the name of "txtCustomerAccountNumber" I can simply add that information to the post with:

   1:  changeAddressRequestBody.FormPostParameters.Add("txtCustomerAccountNumber", this.Context["CUSTOMER_UPDATE_DATA.txtCustomerAccountNumber"].ToString());

As long as the information was displayed via HTML in the prior hit, I should be good to go.

Revisiting CSV Data Binding in VS2008 Coded Web Tests

2008-08-06T11:10:00.001-05:00

I've been going through the WinDBG labs located here. Tess has done a great job with the labs and done all the hard work, so w00t for her!

I needed to add another .CSV data file for my main VUser that I am currently working on so I did a copypasta of a pre-existing DataBinding and DataSource. When I first created the coded web test I had added so much custom code that I deleted the web test and left only the coded web test. The fear was that I might accidently re-gen the code from the web test and over write my work. That in and of itself shouldn't be too bad since we all know to use a code repository, right? Of course we do.

Anyhoo, I made the code changes and was greeted by an error when my file couldn't be loaded. Huh? How rude!

I finally got it to work, but I wanted to note the way .CSV data binding works in a coded web test in greater detail just to remind myself next time this happens. There are plenty of examples of binding a .CSV file to a web test, but few on a pre-existing coded web test. So, here goes!

First we have to have a DataSource declaration that takes five parameters for a .CSV file:

1: dataSourceName
This is the name we are giving the DSN for our .CSV file.
2: providerName
In my case for a .CSV file I am using "Microsoft.VisualStudio.TestTools.DataSource.CSV"
3: connectionString.
For my use, this is the path with escaped back-whacks to the .CSV file. In my case, it is "c:\\data\\streetNames.csv"
4: DataBindingAccessMethod
For my use, I want a random selection of data from the .CSV file so I am using "Microsoft.VisualStudio.TestTools.WebTesting.DataBindingAccessMethod.Random"
5: tableName
I thought that I could provide my own table name to reference the .CSV file but that didn't work and was forced to use the form of #csv, like this: "streetNames#csv". When I tried to use the name "streetNameTable" I would get back an error of "cannot find streetTableName.txt" from VS2008. But, the #csv form works, so no biggie.

Here is what I am using for my entire Data Source:

   1:    [DataSource("streetNameDataSource",

   2:              "Microsoft.VisualStudio.TestTools.DataSource.CSV",

   3:              "C:\\Data\\streetNames.csv",

   4:              Microsoft.VisualStudio.TestTools.WebTesting.DataBindingAccessMethod.Random,

   5:              "streetNames#csv")]

Next, we have to define what column we are pulling from the .CSV file. In my case, it is a single column CSV file but we still have to have a column name defined. For my CSV file I am using the form of:

8<---------------------------
streetNames
"ABBINGTON"
"ALDRICH"
"ALEXANDER"
"ALICE"
"ALLISON"
"ALMOND"
"ALTA"
"AMHERST"
--------------------------->8

These are all the street names in Grover's Mill, NJ as reported by Melissa Data.

The DataBinding declaration that I am using takes four parameters.

1: dataSourceName
This is the DSN that we previously defined.
2: tableName
This is the tableName that we defined in the form of #csv.
3: columnName
This is the column name of the CSV that we are extracting. I've found this to be case insensitive.
4: contextVariableName
This is the key name of the entry that will be created in the Context for the iteration.

Here is what I am using for my DataBinding:

   1:    [DataBinding("streetNameDataSource", "streetNames#csv", "streetNames", "streetNames")]

Now with each iteration of the vuser I can reference the Context for the randomized street name, ala:

   1:  Debug.WriteLine("Street name is " + this.Context["streetNames"].ToString());

Works like a champ!

Tasty memory! Om Nom Nom!

2008-08-02T02:06:00.001-05:00

So, I created a scenario and had 45 virtual users doing nothing but browsing the application that I am testing. After one hour I noticed that the w3wp private bytes are running around 1.5 gigs of memory. That's right, just browsing. I was hitting it at a rate about 26 hits per second and running about 90% CPU utilization and 1.5 gigs of private bytes being consumed with almost 700 megs of hit in Gen 0.

Yeah, something ain't right.

But, that gives me an excuse to brush up on my WinDBG skills. I fired up adplus.vbs and took a -hang dump that resulted in a 1.5 gig dump. Woo! So tonight I've been re-learning the art of poking around in a dump file with a debugger.

Along the way I found a most excellent blog for a MS dudette in Sweden that really knows her stuff and has some great tutorials for debuggery:

http://blogs.msdn.com/tess/default.aspx

Check out the most excellent labs located here.

Let's be deviant with our think times!

2008-08-01T11:45:00.001-05:00

I mentioned in another post that I stopped using the WebTestRequest.ThinkTime as the think time is reported in the Load Test Repository and that annoys me, especially because the default option for think time is for the MS tools to use a normally distributed variable think time on the time requested.

Now, I think the normally distributed think time is awesome to help prevent VUsers from getting into a lock step situation and most of the time we want the VUsers to be in route step.

I created a ThinkTime routine the other day that just used a slightly randomized think time but of course, it was not normally distributed and I really like the idea of a normally distributed think time. So, off to the Internets and I found a simple to implement algorithm by the name of Box-Muller that I was able to use for my VUsers think times.

Below is a graph with an example of a 5 second think time with multiple values of deviation, s.

Below is the main code that I used to write the class to be invoked from inside the coded web test and of course, think time must be invoked outside of any named transactions otherwise you end up in the same situation before with variable think time being reported in the transaction time.

Because of my slackard like nature, I went ahead and made all the methods static so that I don't have in instantiate any objects for later referencing.

I simply call the code like this:

   1:        string transactionName = "Login.PageHit";

   2:        this.BeginTransaction(transactionName);

   3:        WebTestRequest loginPageHit = new WebTestRequest("http://" + targetWebServer.ToString() + "/SomeWebApp/login.aspx");

   4:        ExtractHiddenFields loginExtractionRule = new ExtractHiddenFields();

   5:        loginExtractionRule.Required = true;

   6:        loginExtractionRule.HtmlDecode = true;

   7:        loginExtractionRule.ContextParameterName = "1";

   8:        loginPageHit.ExtractValues += new EventHandler<ExtractionEventArgs>(loginExtractionRule.Extract);

   9:        yield return loginPageHit;

  10:        loginPageHit = null;

  11:        this.EndTransaction(transactionName);

  12:        #if THINKTIME

  13:          LoadTestThinkTime.ThinkTime(5, 0.1);

  14:        #endif

Below is the actual code that I wrote:

   1:  using System;

   2:  using System.Threading;

   3:  namespace LoadTestThinkTime {

   4:    public class LoadTestThinkTime {

   5:      /// <summary>

   6:      /// This routine will generate a randomized variable that is in a standard distribution with a deviation of 0

   7:      /// ± s/2 for given s and then invoke Thread.Sleep for the given amount of seconds converted to milliSeconds.

   8:      /// </summary>

   9:      /// <param name="secondsToSleep"></param>

  10:      /// <param name="s"></param>

  11:      static private void RandomizedSleep(int secondsToSleep, double s) {

  12:        if (s > 0) {

  13:          if (s <= 1.5) {

  14:            if (secondsToSleep > 0) {

  15:              if (secondsToSleep < (int.MaxValue / 1000)) {

  16:                double u  = 0;

  17:                double v  = 0;

  18:                double w  = 0;

  19:                double z0 = 0;

  20:                double z1 = 0;

  21:                Random randomizer = new Random();

  22:                do {

  23:                  u = 2.0 * randomizer.NextDouble() - 1.0;

  24:                  v = 2.0 * randomizer.NextDouble() - 1.0;

  25:                  w = u * u + v * v;

  26:                } while (w >= 1.0);

  27:                w = Math.Sqrt(((-2.0 * Math.Log(w)) / w));

  28:                z0 = u * w;

  29:                z1 = v * w; /* I don't use z1, but it is part of the original algorithm so here it is. */

  30:                /* When using small time values, i.e, 1 second, large values of s can generate negative values. In that

  31:                 * situation go ahead and take the absolute value and invoke the call with that to prevent errors during

  32:                 * loadtests. */

  33:                int milliSecondSleepTime = (int)((secondsToSleep + z0 * s) * 1000.0);

  34:                if (milliSecondSleepTime < 0) {

  35:                  milliSecondSleepTime = Math.Abs(milliSecondSleepTime);

  36:                };

  37:                Thread.Sleep(milliSecondSleepTime);

  38:              } else {

  39:                throw new Exception("LoadTestThinkTime: Maximum sleep time is " + ((int)int.MaxValue / 1000).ToString() + " seconds.");

  40:              };

  41:            } else {

  42:              throw new Exception("LoadTestThinkTime: Sleep time cannot be negative.");

  43:            };

  44:          } else {

  45:            throw new Exception("LoadTestThinkTime: Deviation cannot be larger than 1.5");

  46:          };

  47:        } else {

  48:          throw new Exception("LoadTestThinkTime: Deviation cannot be negative");

  49:        }

  50:      }

  51:      /// <summary>

  52:      /// Public method to invoke variable sleep time with a given deviation of s where s is in the domain of [0, 1.5].

  53:      /// </summary>

  54:      /// <param name="secondsToSleep"></param>

  55:      /// <param name="s"></param>

  56:      static public void ThinkTime(int secondsToSleep, double s) {

  57:        RandomizedSleep(secondsToSleep, s);

  58:      }

  59:      /// <summary>

  60:      /// Public method to invoke variable sleep time with a default deviation of 0.25.

  61:      /// </summary>

  62:      /// <param name="secondsToSleep"></param>

  63:      static public void ThinkTime(int secondsToSleep) {

  64:        RandomizedSleep(secondsToSleep, 0.25);

  65:      }

  66:    }

  67:  }

Extracting Transaction Response Time by Transaction Name per Load Test Run

2008-07-29T22:33:00.002-05:00

Despite being annoyed that think time is included in the VS2008 web test response time I would like to be able to graph the response times for a given transaction name. I load my Load Test History database by the unique GUID and forgo the integer test ID to prevent possible duplicates in the future.

I created a query that takes transaction name and test GUID to pull all the transaction times.

   1:  SELECT LoadTestTransactionDetail.TimeStamp, LoadTestTransactionDetail.ElapsedTime

   2:  FROM  LoadTest.dbo.LoadTestRun INNER JOIN

   3:                 LoadTest.dbo.WebLoadTestTransaction ON LoadTestRun.LoadTestRunId = WebLoadTestTransaction.LoadTestRunId INNER JOIN

   4:                 LoadTest.dbo.LoadTestTransactionDetail ON WebLoadTestTransaction.TransactionId = LoadTestTransactionDetail.TransactionId AND

   5:                 LoadTest.dbo.LoadTestRun.LoadTestRunId = LoadTestTransactionDetail.LoadTestRunId

   6:  WHERE (WebLoadTestTransaction.TransactionName = 'Some_Transaction_Name') AND (LoadTestRun.RunId = '987cce52-7608-40a7-ac85-96714af1ac6d')

   7:  ORDER BY LoadTestTransactionDetail.TimeStamp

8:

9:

There we go! All the transaction times for "Some_Transaction_Name:"

   1:  2008-07-29 15:58:04.467    7.944

   2:  2008-07-29 15:58:37.633    6.138

   3:  2008-07-29 15:58:55.930    5.243

   4:  2008-07-29 15:59:04.977    4.615

   5:  2008-07-29 15:59:25.607    6.461

   6:  2008-07-29 15:59:36.493    5.026

   7:  2008-07-29 15:59:40.820    5.256

   8:  2008-07-29 16:00:00.507    5.573

   9:  2008-07-29 16:00:04.940    5.457

  10:  2008-07-29 16:00:08.977    5.577

  11:  2008-07-29 16:00:16.547    5.506

  12:  2008-07-29 16:00:34.303    5.758

Uh-Oh. This isn't good.

2008-07-29T20:51:00.001-05:00

I found out something today that I really don't like with VS2008 for load testing. I created my virtual user and setup think time ala WebTestRequest.ThinkTime with reasonable think times. After the testing was done I moved the results over to my LoadTestHistory database with a SP that I wrote about earlier in my blog. When I examined the results it turned out that the think time was included in the transaction response time. Nope, I don't like that. I dislike that about as much as I don't like having a failed counter for transactions.

I guess I can't be really surprised as the Begin/End transaction is wrapped around the WebTestRequest object and the think time is handled by the WebTestRequest object and there is no communication between the transaction begin/end and the WTR object (AFAIK).

I looked in the schema for the Load Test Repository and didn't see any fields where response time alone (without think time) would be recorded but there is definitely response time without sleep time recorded during the execution of the VS2008 loadtest. I figured there would be a field in the database with this value but I didn't see it.

Luckily, though, this is pretty simple to get around. If you aren't going to be using variable think times you can simple perform a Thread.Sleep(new TimeSpan(0, 0, 0, 5, 0)) which is similar to the LoadRunner idea behind lr_think_time(). Don't forget to add the "using System.Threading;" to your code to reference the Thread object since it is not referenced by default.

If you want to use variable think time then you'll have to come up with your own mechanism for varying the randomness of the think time. It's not that difficult, it is just a pain.

Clearing cookies in VS2008

2008-07-28T12:19:00.000-05:00

In a coded web test that I am working on I have a need to clear all pre-existing cookies for a login page. I don't want the page identifying my vuser with a cookie that is assigned later on in the page hits.

While VS2008 has a CookieContainer that is a container for cookies, there is not a .Clear() method to clear out all the cookies like can be done with LoadRunner with the web_cleanup_cookies() API call.

I have found the way to do this is to use assign a new CookieContainer during the PreWebTest event to clear out all pre-existing cookies like this:

   1:  public Some_Coded_Web_Test() {

   2:    this.PreAuthenticate = true;

   3:    this.PreWebTest += new EventHandler<PreWebTestEventArgs>(ClearCookies);

   4:  }

5:

   6:  void ClearCookies(object sender, PreWebTestEventArgs e) {

   7:    Debug.WriteLine("Ha!Ha! I am clearing cookies!");

   8:    this.Context.CookieContainer = new System.Net.CookieContainer();

   9:  }

No more tasty, tasty cookies on script iterations.

Thinkin' about ditching Hyper-V

2008-07-27T23:36:00.001-05:00

Seriously thinking about ditching Hyper-V for it's smaller brother, Virtual PC 2007. Sure, Virtual PC doesn't give the VM as much horsepower or as much contact with the metal of the host machine but by golly gosh the networking doesn't try to kill off Client for Microsoft Network.

I'm sure Hyper-V works great in the intended role of hosting guest VMs while racked in a server room someplace but the network issue that I am having is really irking me. Perhaps Hyper-V isn't ready for the type of prototyping work I want to do and Virtual PC 2007 will gimp along for my prototyping work.

Of course, the only issue that I have so far is that I have four VMs created and I'd have to go through all that trouble of reloading the OS. Well, I'm just about done with my load balancing prototyping so blowing away the VMs and removing the Hyper-V role wouldn't be that painful...

Of course, the biggest thing that I'd miss is the snapshot feature of Hyper-V. For prototyping and messin' around with a VM that is the greatest thing since sliced bread. I don't suspect we'll see that in a free Virtual PC anytime soon...

Hyper-V Networking Problems

2008-07-26T00:04:00.000-05:00

I encountered some "interesting" situations with Hyper-V networking last night. For some reason I've lost the ability to connect to network shares on my machine, not good. I poke around and find for some reason that "Client for Microsoft Networks" is disabled on the Hyper-V virtual NIC. WTF?

I try to re-enable it and get the strange error message of "Your current selection will also disable the following features: Client for Microsoft Networks"

Uh, no? I'm trying to re-enable.

It appears that I am not the only person that has had this issue:

http://sqlsolace.blogspot.com/2008_04_01_archive.html

I disabled the Hyper-V virtual NIC and re-established my physical NIC and am able to truck on down the road. I find that if I delete the Hyper-V virtual NIC (external network connection) and create an internal virtual network (no physical connection) that the internal network has Client for Microsoft Network without issue. It's only the external virtual connections that are bjorked.

In the blog post above they said that if they un-installed all protocols and clients, reboot, and then re-installed that it was a work around. What a PITA. I haven't tried that yet as I got my Ubuntu 8.04 TLS installation up and running with HAProxy load balancing to two Win2k3 VMs which for prototyping works just fine on an internal virtual network just fine.

I like HAProxy. It is pretty full featured for a free, open source L7 load balance tool. It has a lot of features that remind me of NetScaler boxes (HAProxy even has a FreeBSD port so you too can run your balancer under FreeBSD just like NetScaler, 'cept of course it isn't NetSclaer).

I fired up my Win2k3 running NLB and the NLB manager shows both client machines to be up and running but it will only direct traffic to one of the Win2k3 VMs. Nice. And even niftier, if I ping the NLB cluster (in my case, it's 192.168.0.100) I will get back dupe pings as NLB creates the 192.168.0.100 VIP on both the NLB machine and all the client machines. Yeah, NLB is over kill for the stuff I want to do and HAProxy has nifty persistence support via cookies since it is a L7 balancer.

The only issue is that IIS logs the IP address of the HAProxy machine and not the actual client IP address. The HAProxy, much like the NetScaler product, sends the true IP in an X HTTP request header but you must use an ISAPI filter with IIS to log the true IP address to the IIS log. Apache has built in support for logging X-Header entries to the apache log. That is kinda handy but we aren't using apache. I've found some sample Visual C++ 6.0 ISAPI code for handling this task. Heck, I don't even have Visual Studio 6.0 installed or even Visual C++ 2008 installed on my machine.

Meh, not going to worry about it right now. Just suffice it to say that Hyper-V networking is irking me and NLB is ooky. HAProxy is pretty nifty and didn't take long to get a sample balancer up and running.

Getting back to load testing

2008-07-23T21:34:00.000-05:00

I actually got to do some load testing today! Heck, I needed to figure out how many combinations for a test and even got to whip out some discrete math! Woooooo! Look at me go!

One of the things that I like about load testing is the "WTF?" factor. Finding those things that could never be found in "onesie-twosie" testing. In my current testing I am using JMeter to chunk 1500 e-mails and FTPs through a file routing system and I found that after the xfer is done that the w3wp.exe process would go high, spike a processor and the large object heap of the w3wp would go high. On top of that, during the same period where the w3wp goes whizzing off into la-la land the private byte usage of SQL Server climbs into the exosphere. We don't know what the problem is just yet but I like weeding stuff like that out with the initial raise of the eyebrow and a muffled "WTF?" Something tells me that I'll be brushing off my memory dump skills and using Son of Strike to figure out what is being dumped into the LOH.

I've been looking at load balancers and in my Google journeys I came across HAProxy and it seems to be quite a piece of art and an industrial load balancing solution. It runs natively under Linux but there is a FreeBSD version out in the ports tree. I haven't done anything with it yet as I've been busy with other tasks but I want to give it a test drive. It should be well more than what we need for the task at hand. 9 GB/s throughput? Yeah. We're never gonna see that. :-)

http://haproxy.1wt.eu/

NLB is ooky

2008-07-21T21:58:00.001-05:00

Am I the only one that gets an oooky feeling from Microsoft's Network Load Balancer? Seems to be a little bit of over kill for just port 80 stuff. I built a virtual network in Hyper-V with three Win2k3 installations and played around with getting NLB up and running.

Sure, it gets up and running pretty easily but boy howdy if you ever want to make any changes like changing the NICs from a 10.0.0.0/8 space to 192.168.0.0/16 space. Yikes! Errors a plenty and that is after breaking down the cluster and deleting the cluster and then un-installing NLB from the target NICs and the balancer installation only to rebuild. Helllooo "unable to bind" errors!

A bit overkill for what I'm doing. I'm thinking maybe a nice balancer like Virtual Linux Server project or something similar would get the job done. Or perhaps Pure Load Balance (runs under FreeBSD, w00t!).

Maybe I just mucked up something in the process and need to play around more (this is the most likely scenario). Thank goodness for Hyper-V snapshots. They really save the day when prototyping like I have been doing.

I did run into an interesting situation. I created the three above mentioned VMs and Win2k3 didn't have a driver for my notebooks gigabit NIC. For some reason it took me a while to realize I could add a legacy NIC that Win2k3 could see.

I've enjoyed showing off the snapshot functionality by writing one line command lines that delete every file off the file system and then restoring with the snapshot and seeing the machine up and running 10 seconds later. Greatness for prototyping and fooling around!

Portable Server Drive Performance

2008-07-19T03:56:00.000-05:00

I was wondering what the performance of the RAID5 setup in my new portable server and I found this nifty little HD benchmark tool at http://www.datamarck.com/.

I ran it and it showed that the average throughput of my RAID5 array was about 77 MB/s. In contrast, my older desktop machine ran about 46 MB/s and my older laptop was a measly 36 MB/s. Not too shabby.

I haven't done any video or CPU benchmarking yet but I wonder how the video performs with Win2k8 server and the nVidia 8800M single card.

I'm currently installing two instances of Win2k3 standard under Hyper-V to play around with xfering IIS metabase and some other automated tasks and the beauty of this is that I can take snapshots and restore those snapshots while playing around.

But I did find a bummer about Hyper-V. You can bind it to a NIC as long as it is not a wireless NIC. I found that rather strange that there would be a differentiation of interfaces as both are ethernet devices. Go figure.

Another thing I noticed is that the network card is reported by Win2k8 as a 10 GB/s card which I find hard to believe. It's prolly just a reporting err. No biggie as I don't have 10 GB infrastructure in the house.

All in all, pretty nifty for a portable device.

Hyper-V is nifty!

2008-07-17T22:15:00.001-05:00

I haven't had much of a chance to play with Hyper-V but I did a little bit of fiddling tonight. I loaded up a VM with the latest Ubuntu and got it up and running and all is well with the world.

I made a snapshot and then did the classic n00b mistake of:

`cd / && rm -rf *`

Yeah. Not a good thing to do and yes, it destroyed the installation as expected.

I applied the snapshot that I had taken a few minutes before and within a few seconds my Ubuntu was up and running exactly the way it was when I took the snapshot. How stinkin' cool is that?

I have a need to build out an installation process that builds installs and configures a Win2k3 machine with users, applications, IIS, vdirs, etc and this will be the perfect environment to test. Build up a base machine, get it configured and snap shot.

Then I can try the process and tweak as much as a I need by restoring the snapshot. Too stinkin' cool!

The Beast!

2008-07-16T00:10:00.000-05:00

I got my new laptop in today. It is a rebadged Clevo 901 sporting a Quad Xeon 2.83 GHz proc, RAID5 with 3 200 gig spindles at 7200 RPM and 4 gig of RAM. I can upgrade to 8 gig of RAM when the memory becomes cheap enough. It also has a nVidia 8800M video card.

The sucker is huge with a 17 inch wide screen display (1680x1050) and is almost three inches thick. It's definately not a laptop but a desktop replacement. In my case, it is a mobile server.

I was able to order it without an OS as I can get a license through my employer. Today I loaded up Win2k8 64-bit Standard (no need for Enterprise as I am not clustering my "laptop." I also loaded SQL Server 2005 64-bit and then VS2008 Team Suite Testing Edition. On top of that I threw in Office 2007 and spent a lot of the day twiddling with installations and settings.

After using the Win2k8 interface for a while it seems that I am going to just have to bite the bullet and get used to the Vista UI as Win2k8 uses it as well. *sigh* It was good knowing ya, XP Pro.

But, for what it's worth, Win2k8 did seem pretty damn snappy. Of course, it might have something to do with the quad core Xeon and 4 gig of RAM and a 512 meg video card. :)

I also enabled IIS and Hyper-V after I enabled VMM and NX in the machine's BIOS configuration. I'm stoked about Hyper-V but I've already found out one bummer with Hyper-V and that is enabling the Hyper-V role disables the ability to hibernate the machine. I guess I can understand why. If you are hosting a bunch of VMs with Hyper-V you really don't need the ability to hibernate the server. So, it's a minor irritation that but is all.

I did enable some eye-candy such as smoothing fonts and thumbnails instead of icons. That is one Vista thing that I really liked. That and the use of c:\users instead of the iritating "documents and settings." If you are a keyboard commando like me, those spaces get to be damned annoying. Of course, Microsoft still uses spaces in a lot of their subdirs so the issue isn't totally settled.

And I learned something from my previous Vista Ultimate experience on my older laptop (15.4 inch more conventional laptop, not a UberPowah machine like my current big rig) and that is to avoid the Microsoft Unix tools and just run with the GNU Unix Tools for Win32. I found a lot of unix commands lacking with the MS implementation and it is easier to unzip the GNU tools and add the subdir to the path. Of course, I can't compile any X code like I supposedly could with the MS unix services but I didn't need to compile any anyway so it is no great loss.

Another side effect is that I had to buy a new laptop bag and this time instead of a backpack I went with a Swiss Army gear rolling laptop bag that let's me put both my new MobileServer and Acer Ferrari in the bag at the same time and roll it along. With both machines and accessories (power bricks, extra battery, typical junque) that bag feels like it weighs in at 45 - 50 pounds. That would have been a bit tough to carry around an airport. I spent more than I wanted to but I think it'll be worth it.

On other good news fronts last week work gave me two physical boxen to actually build up my LT environment and I think that I'll actually be doing some dry runs tomorrow measuring up initial testing capacity with JMeter testing against FTP chunking my GPG encrypted files for the service to munch on. Finally!

I've been doing a lot of hopping around at work but my goal is still the same, to get daily automagic load tests running and in place with our continuous integration work flow.

Transaction Transaction, what's your action?

2008-07-07T09:55:00.001-05:00

I noticed something this morning as I was going over my SP that I wrote on Thursday night. I realized that I needed to also gonkulate how many transactions had failed. Seems simple enough, right? If a page (or dependent request) fails inside a transaction the whole transaction gets marked as a failure, right? That's the way that LoadRunner handles the transaction issue.

In LoadRunner I would normally leave the determination of the status of the transaction by using LR_AUTO. There were a few times that I would explicitly call EndTransaction with a LR_FAIL.

I realized looking over the schema of the Load Test Repository Store that there does not appear to be any logging of failures on the transaction level. The page level? Sure. Not a problem. LoadTestMessage table contains those goodies for us but I cannot find a way to link LoadTestMessage with WebLoadTestRequestMap. Going back over some of the online documentation for running VS2008 load tests it appears that the tool itself does not count transaction failures and only counts failed pages.

Yeah... I'm not too happy about that. With my analysis I want to see what transactions failed as I might have multiple page hits to a single transaction. Hopefully this gets fixed in a future version of the MS LT tool.

Collecting daily LT Metrics

2008-07-04T11:21:00.001-05:00

Last night I wrote a stored procedure to automagically collect the latest LT results and insert the values into a database that I created on my local SQL Server 2005 Express.

This SP finds the latest LT that has been run, crunches the numbers and inserts them into two tables, LoadTestHistory and LoadTestHistoryDetails. I used the GUID generated by VS2008 in the Load Test Repository Store as the primary/foreign key between the two tables.

It's a bit rough and could probably use some more polish:

   1:  set ANSI_NULLS ON

   2:  set QUOTED_IDENTIFIER ON

   3:  GO

   4:  ALTER PROCEDURE [dbo].[UpdateLoadTestHistory]

   5:  AS

   6:  BEGIN

7:

   8:    SET NOCOUNT ON;

9:

  10:    declare @AlreadyExists    as int

11:

  12:    declare @latestLoadTest   as int

  13:    declare @loadTestGUID     as nvarchar(36)

  14:    declare @StartTime        as datetime

  15:    declare @EndTime          as datetime

  16:    declare @Duration         as int

17:

  18:    select @latestLoadTest = max(LoadTestRunId)

  19:    from LoadTest.dbo.LoadTestRun

  20:    where EndTime is not null

21:

  22:    select @loadTestGUID   = RunId,

  23:           @StartTime      = StartTime,

  24:           @EndTime        = EndTime,

  25:           @Duration       = RunDuration

  26:    from LoadTest.dbo.LoadTestRun

  27:    where LoadTestRunId = @latestLoadTest

28:

  29:    select @AlreadyExists = count(*)

  30:    from dbo.LoadTestHistory

  31:    where LoadTestGUID = @loadTestGUID

32:

  33:    if (@AlreadyExists = 0)

  34:      begin

35:

  36:        begin transaction

37:

  38:        begin try

39:

  40:          insert into dbo.LoadTestHistory

  41:          values (@LoadTestGUID,

  42:                  @StartTime,

  43:                  @EndTime,

  44:                  @Duration)

45:

  46:          SELECT @loadTestGUID                                as LoadTestGUID,

  47:                 WebLoadTestTransaction.TransactionName,

  48:                 LoadTestTransactionSummaryData.Average,

  49:                 STDEV(LoadTestTransactionDetail.ElapsedTime) as StdDev,

  50:                 LoadTestTransactionSummaryData.Minimum,

  51:                 LoadTestTransactionSummaryData.Maximum,

  52:                 LoadTestTransactionSummaryData.Percentile90,

  53:                 LoadTestTransactionSummaryData.Percentile95,

  54:                 LoadTestTransactionSummaryData.TransactionCount

  55:          INTO   #tmpLoadTestHistory

  56:          FROM   LoadTest.dbo.LoadTestTransactionDetail INNER JOIN

  57:                 LoadTest.dbo.WebLoadTestTransaction ON LoadTest.dbo.LoadTestTransactionDetail.LoadTestRunId = LoadTest.dbo.WebLoadTestTransaction.LoadTestRunId AND

  58:                 LoadTest.dbo.LoadTestTransactionDetail.TransactionId = LoadTest.dbo.WebLoadTestTransaction.TransactionId INNER JOIN

  59:                 LoadTest.dbo.LoadTestTransactionSummaryData ON LoadTest.dbo.WebLoadTestTransaction.LoadTestRunId = LoadTest.dbo.LoadTestTransactionSummaryData.LoadTestRunId AND

  60:                 LoadTest.dbo.WebLoadTestTransaction.TransactionId = LoadTest.dbo.LoadTestTransactionSummaryData.TransactionId

  61:          WHERE  (LoadTestTransactionDetail.LoadTestRunId = @latestLoadTest)

  62:          GROUP BY WebLoadTestTransaction.TransactionName, LoadTestTransactionSummaryData.Average, LoadTestTransactionSummaryData.Minimum,

  63:                   LoadTestTransactionSummaryData.Maximum, LoadTestTransactionSummaryData.Percentile90, LoadTestTransactionSummaryData.Percentile95,

  64:                   LoadTestTransactionSummaryData.TransactionCount

  65:          ORDER BY WebLoadTestTransaction.TransactionName

66:

  67:          insert dbo.LoadTestHistoryDetails

  68:          select LoadTestGUID, TransactionName, Average, StdDev, Minimum, Maximum, Percentile90, Percentile95, TransactionCount

  69:          from #tmpLoadTestHistory

70:

  71:          drop table #tmpLoadTestHistory

72:

  73:          commit transaction

  74:        end try

75:

  76:        begin catch

  77:          rollback transaction

  78:        end catch

  79:      end

  80:  END

And it works like a champ!

I tested it with this bit of SQL:

   1:  use LoadTestResults

   2:  go

3:

   4:  truncate table dbo.LoadTestHistory

   5:  truncate table dbo.LoadTestHistoryDetails

   6:  go

7:

   8:  dbo.UpdateLoadTestHistory

   9:  go

10:

  11:  select dbo.LoadTestHistory.LoadTestGUID,

  12:         StartTime,

  13:         EndTime,

  14:         Duration,

  15:         TransactionName,

  16:         Average,

  17:         StdDev,

  18:         Minimum,

  19:         Maximum,

  20:         [90th],

  21:         [95th],

  22:         TransactionCount

  23:  from dbo.LoadTestHistory, dbo.LoadTestHistoryDetails

  24:  where dbo.LoadTestHistory.LoadTestGUID = dbo.LoadTestHistoryDetails.LoadTestGUID

And the results come out for all to enjoy:

   1:  LoadTestGUID                         StartTime               EndTime                 Duration TransactionName            Average          StdDev            Minimum Maximum 90th  95th  TransactionCount

   2:  0ebdf821-454f-4c50-8e3a-a82a291adb97 2008-07-03 16:21:33.280 2008-07-03 16:31:33.280 600      someTransaction.Details    3.4155221238938  0.583565372475793 1.526   4.952   4.161 4.35  226

   3:  0ebdf821-454f-4c50-8e3a-a82a291adb97 2008-07-03 16:21:33.280 2008-07-03 16:31:33.280 600      someTransaction.FirstHit   3.42468584070796 0.79565610855916  1.636   11.888  4.053 4.285 226

   4:  0ebdf821-454f-4c50-8e3a-a82a291adb97 2008-07-03 16:21:33.280 2008-07-03 16:31:33.280 600      someTransaction.LookupName 3.46306194690266 0.606631255863035 2.087   5.15    4.285 4.411 226

Retrieving Standard Deviation for Transaction Response times with VS2008

2008-07-03T13:28:00.001-05:00

One of my plans for the firm for which I work is to integrate a nightly automagic LT with the Continuous Integration effort. I want to be able to automagically compare results of the previous days results and see if anything is amiss. One of the tests I would like to apply is the two population mean test which requires having the average response time, standard deviation and the number of transactions.

VS2008 doesn't provide all of these statistics by default in their LT summary. LoadRunner includes this information by default in their summary results and extracting the values are quite simple if you publish the results to an Excel Spreadsheet.

It's not so quite as simple as that with VS2008 but it can be done with a little effort.

I have configured my LT to record individual metrics to the LoadTest Data Store so that I can query the tables and extract the information that I want. In the examples below I have identified the LoadTestRunId for a given run where I have plenty of metrics to number crunch. In my nightly build and test scenario I envision a simple query to pull the max(LoadTestRunId) to get the latest run to run queries to extract the required information.

I wrote this query today to get the information that I wanted:

   1:  SELECT WebLoadTestTransaction.TransactionName,

   2:         LoadTestTransactionSummaryData.Average,

   3:         STDEV(LoadTestTransactionDetail.ElapsedTime) AS StdDev,

   4:         LoadTestTransactionSummaryData.Minimum,

   5:         LoadTestTransactionSummaryData.Maximum,

   6:         LoadTestTransactionSummaryData.Percentile90,

   7:         LoadTestTransactionSummaryData.Percentile95,

   8:         LoadTestTransactionSummaryData.TransactionCount

   9:  FROM   LoadTestTransactionDetail INNER JOIN

  10:                    WebLoadTestTransaction ON LoadTestTransactionDetail.LoadTestRunId = WebLoadTestTransaction.LoadTestRunId AND

  11:                    LoadTestTransactionDetail.TransactionId = WebLoadTestTransaction.TransactionId INNER JOIN

  12:                    LoadTestTransactionSummaryData ON WebLoadTestTransaction.LoadTestRunId = LoadTestTransactionSummaryData.LoadTestRunId AND

  13:                    WebLoadTestTransaction.TransactionId = LoadTestTransactionSummaryData.TransactionId

  14:  WHERE  (LoadTestTransactionDetail.LoadTestRunId = 33)

  15:  GROUP BY WebLoadTestTransaction.TransactionName, LoadTestTransactionSummaryData.Average, LoadTestTransactionSummaryData.Minimum,

  16:                    LoadTestTransactionSummaryData.Maximum, LoadTestTransactionSummaryData.Percentile90, LoadTestTransactionSummaryData.Percentile95,

  17:                    LoadTestTransactionSummaryData.TransactionCount

  18:  ORDER BY WebLoadTestTransaction.TransactionName

The QBE table entries look this this:

And I get the results I want:

   1:  TransactionName            Average             StdDev               Minimum Maximum 90th  95th  TransactionCount

   2:  someTransaction.Details    0.47510353043101411 0.090671477518885088 0.416   3.695   0.522 0.552 4617

   3:  someTransaction.FirstHit   0.44759085986571417 0.077896506427873255 0.384   3.956   0.502 0.526 4617

   4:  someTransaction.LookupName 0.51917002382499466 0.082360597958219192 0.45    2.844   0.577 0.605 4617

Huzzah!

I'll end up setting up an automagic comparison routine. A number of years ago I wrote a C# class for doing statistical tests so I'll probably end up using that for my left/right comparisons and number crunching.

It's finally been ordered!

2008-07-01T21:19:00.001-05:00

I got news today that my new laptop has finally been official ordered. Here is the basic specs as it stands right now:

Quad Core Xeon 2.83 GHz proc
4 gig RAM
17 inch display at 1680x1080
RAID5 with 3 drives spinning 7200 RPM with 200 gig each for 400 gig of filespace
nVidia 8800M 512 meg video card (only 1 but it can support 2)

I was able to order it without an operating system so I will be installing either Win2k3 or Win2k8 for my load testing needs.

Hopefully tomorrow I'll find out when it is arriving. It's like Christmas in July!

Another neat use for PostRequest

2008-06-30T14:50:00.000-05:00

I have a page that I am testing against that comes back as a failure because a .js file hasn't been migrated yet so I don't want my coded web test to hit that page and fail. I took care of the problem with PostRequest for the WebTestRequest call like this:

   1:  void preventCallsToJS(object sender, PostRequestEventArgs e) {

   2:    List<WebTestRequest> requestsToRemove = new List<WebTestRequest>()

   3:    foreach (WebTestRequest linkToRemove in e.Request.DependentRequest

   4:      if (linkToRemove.Url.EndsWith("someJavaScriptFile.js")) {

   5:        requestsToRemove.Add(linkToRemove);

   6:      }

   7:    };

   8:    foreach (WebTestRequest linkToRemove in requestsToRemove) {

   9:      e.Request.DependentRequests.Remove(linkToRemove);

  10:    };

  11:  }

For each request I have added the call to the above PostRequest like this:

   1:  request4.PostRequest += new EventHandler<PostRequestEventArgs>(preventCallsToJS);

No more problems hitting those pesky not yet deployed .js files. w00t!

Saving sessions as webtests with Fiddler2

2008-06-30T10:49:00.001-05:00

Today I was trying to save some saved sessions with Fiddler2 to a webtest since I cannot record webtests with VS2008 with my common user profile that I use on my laptop.

When I tried to save the file I got a very annoying error that a specified assembly could not be found. Fiddler2 is trying to load version 8.0.0.0 of Microsoft.VisualStudio.QualityTools.WebTestFramework.dll but alas, I do not have VS2005 install, I have VS2008. What to do? I'm already bummed that I cannot record webtests as I should be able to. I find this to be really annoying as recording webtests shouldn't be this annoying.

At this point you're probably asking, "So, Mr. Auswipe-Load-Tester-Dude, does this shake your faith in the VS2008 framework for load testing?" Not yet is my reply. Why you ask when these two issues are clearly impediments? Well, with Load Runner I was running into problems recording virtual users and had the problem for over three weeks where I could not record vusers on my work machine. When I tried to record vusers both FireFox and IE would detonate and crash. I worked with Level I and Level II techs for over three weeks trying to correct the problem and the issues was never solved. It turns out I got my current job offer and accepted it. I told my former coworkers that I figured the solution was to take the fdisk quiz and reload the OS and start again. I certainly hope the same solution is not required for the MS solution.

Anyway, back to the issue at hand...

I did some Googling and I found this entry on CodeProf.com with a solution by Ed Glas. Ed basically says to modify the fiddler.exe.config and use the tag but as documented here but did not include a full fledged example, just the link to the .NET config file documentation. So, I'm not a .NET expert by any stretch of the imagination (but am working on it slowly but surely) so I wasn't quite sure what all config tags were required for the redirect. Here is what I used and it appears to be working *cross fingers*.

   1:  <configuration>

   2:    <runtime>

   3:      <legacyUnhandledExceptionPolicy enabled="1" />

   4:      <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">

   5:        <dependentAssembly>

   6:          <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.WebTestFramework"

   7:                                  publicKeyToken="b03f5f7f11d50a3a"

   8:                                  culture="neutral" />

   9:              <bindingRedirect oldVersion="8.0.0.0"

  10:                               newVersion="9.0.0.0"/>

  11:           </dependentAssembly>

  12:        </assemblyBinding>

  13:    </runtime>

  14:  </configuration>

After doing some recording it appears that the code that is generated from the converted Fiddler2 to .webtest to coded web test looks good and I see the automagic correlation of data extracted from form fields and the ever dreaded VIEWSTATE.

This oughta be good enough for me right now until I resolve the situation with VS2008. I have no personal problems with recording a session, saving as a .webtest and then added the .webtest to a project and converting to a coded web test so it's all good with me.

The Good, The Bad and The Ugly.

2008-06-28T22:58:00.001-05:00

I've been real busy with non load testing stuff at work and haven't had much of a chance to continue playing around with VS2008 for LT purposes but I have found out some interesting stuff.

The Good:

You can specify the data store (either SQL Server or SQL Server Express) for storing all the individual hits from web sites and then query that information which renders that I was doing in the earlier blog post as moot. Yay! By default, VS2008 TSTE keeps these results in SQL Server Express that is installed when Visual Studio is installed. It's pretty cool that you can choose to store results in another location for historical reasons.

Looking around the data in the tables I'm not sure if you can group page hits together by transaction ID. But, I haven't had much of a chance to play around with querying data and am not for sure if this is the case. If that is the case, I can still use a method similar to a previous blog entry for collecting the metrics myself for catching page response time as it relates to individual transactions. A bit of a PITA but fantastic that I can do it if I want. Yay for a flexible framework that allows me as a test with development background to run around and do this kind of stuff.

One thing I know for sure though is with the ability to query the databases I should now have the ability to crunch the numbers for transactions and now get average, standard deviation, hit counts, etc. The idea is that in a nicely controlled environment I should be able to generate an automagic report for day to day automated load test execution and be able to do something like a mean population comparison of transactions times from one test to the next and pick out statistically significant deviations for alpha = 0.05. That'll be kinda neat if it works out.

Now that I think about it, the VS2008 tool should have been reporting these metrics all along. LoadRunner defaults with basic statistical information and VS2008 should report the same as well. It is kind of annoying, but I like the fact that I have a published schema that I can use to extract the information that I want (see link to schema information for VS2008 further down below). So, another plus in the LoadRunner column. [I might really like the VS2008 product, but I do have to be fair in my comparisons of Pro/Con of VS2008 versus LoadRunner.]

I wasn't able to use the statistical modeling at my previous employer because our systems were just too large with too many variables involved. We had communication from the web server to the database server (which was on the same switch so not much of a problem) but we also had a bunch of communications to the Host. The Host was required for testing the web sites and was actually hitting production systems that merely disregarded the requests and sent back bogus results so that we wouldn't kill the mainframe.

Many years ago I tried to get rid of this variability by writing what I called a "parroting proxy server." It was a bit of C# code that was essentially a protocol agnostic proxy server that would pass messages from point A to point C where the intermediate Point B was the proxy. It would store responses with a in memory hash and if the same request was caught being requested it would go ahead and send back the hashed response to eliminate the variance that we would see. It worked out pretty good but we never got around to implementing it. Unfortunately in a Fortune 50 company the bureaucracy can be too great and with Empire Building as it is the other managers didn't see what was in this for them and I suppose I didn't do a very good sales job. So be it.

To flip the switch to log all data so that you can query the data for fun and profit is to modify the "Timing Details Storage Property" in the Run Setting node of the load test editor to "All individual details."

And unlike the LoadRunner database (AFAIK, I could be wrong!), this schema is published with all sorts of handy information located at http://blogs.msdn.com/billbar/articles/529874.aspx. How great is that?

The Bad:

I haven't had enough time to continue my little load testing adventure. I'm finding that my new employer needs to update their processes as it relates to building production final machines. The current steps are extremely painful. But I suspect it'll be just fine in the long run as I've had some experience in this in the past and now that this process can be fully automated and integrated into our build process to be painless.

The Ugly:

Ok, this is ugly. I tried to write a simple script to augment some stuff that I did last week only to find that I couldn't bring up an instance of IE with the Web Recorder active and it is frustrating as hell. I found several posts that essentially say that the "solution" is to delete your login profile and start again. Yeah, that really sucks, Microsoft. I'd like to have a solution that doesn't require me to blow away my profile and have me spend hours getting my desktop back into the shape that I like it. That is a real PITA for sure.

I have three profiles on my machine and I can only generate recorded web tests with one of these profiles. I blew away one of the smaller profiles (not my main profile where I have 2.5+ gig of files) and gave it a try and it was no-go. I still could not record web tests. Yeah, that is bad. Very bad.

I'm not sure what the solution on this is going to be. In the short term I'm going to have to use my Administrator login to create scripts and then xfer that over to my main login script. How craptacular is that? Once I get some free time I'll have to look into this further but I am dreading it. No fun!

Here are some MS blog entries related to the problem. Perhaps they could help somebody else but I haven't seen any love for me, yet.

http://blogs.msdn.com/mtaute/archive/2007/11/09/diagnosing-and-fixing-web-test-recorder-bar-issues.aspx
http://blogs.msdn.com/edglas/archive/2007/11/01/web-test-recorder-bar-not-showing-up.aspx

It appears that Fiddler2 might be an interim solution from what I read here in this MS blog post located at http://blogs.msdn.com/slumley/pages/enhanced-web-test-support-in-fiddler.aspx.

A PITA? Sure, but it beats not having any solution at all. Fiddler is purt near cool and I've "fiddled" with it a little bit and it is nifty.