Manual model building TeamQuest Model gotcha

A while back I started to play around with Perl::PDQ and decided in order to help me learn the R language that I'd just stick with PDQ-R for now. At work I don't have a system yet in place where I can perform measurements and comparisons of actual response times versus PDQ-R results.

I did the best next thing for me, which was to compare the results in a case study. As with some of my previous blog entries I was using case studies from Performance By Design.

For my first attempt of PDQ-R coding I was using the baseline from Chapter 5: Case Study I: A Database Service. This chapter is pretty nifty because it covers the topic of cluster analysis for workload analysis with three classes of workload. That led me to learning the basics of cluster analysis with R.

Here was my solution for the baseline found in 5.4 of Performance By Design.

library("pdq");

# Total req/sec into the open queue model
lambda_into_system <- 1.33;

# Split up the req/sec into each class based upon
# previously supplied ratios

coeff <- c(0.33/1.33, 0.53/1.33, 0.47/1.33);

lambda <- coeff * lambda_into_system;

#                 Class1,Class2,Class3
cpu_demand   <- c(0.096, 0.615, 0.193);
disk1_demand <- c(0.088, 0.683, 0.763);
disk2_demand <- c(0.119, 0.795, 0.400);

workStreamName <- 1:3;


Init("Chapter 5.4");

for (n in 1:3) {
  workStreamName[n] <- sprintf("class_%d", n);
  CreateOpen(workStreamName[n], lambda[n]);
};

CreateNode("CPU",   CEN, FCFS);
CreateNode("Disk1", CEN, FCFS);
CreateNode("Disk2", CEN, FCFS);

for (n in 1:3) {
  SetDemand("CPU",   workStreamName[n], cpu_demand[n]  );
  SetDemand("Disk1", workStreamName[n], disk1_demand[n]);
  SetDemand("Disk2", workStreamName[n], disk2_demand[n]);
};

SetWUnit("Trans");
SetTUnit("Second");

Solve(CANON);
#Report();

response <- 1:3;

for (n in 1:3) {
  response[n] <- GetResponse(TRANS, workStreamName[n]       );
};

for (n in 1:3) {
  print(sprintf("  PDQ-R computation shows response time for class %d is %f seconds", n, response[n]));
};

Which yields the results of:

> source('baseline.R')
[1] "  PDQ-R computation shows response time for class 1 is 0.864179 seconds"
[1] "  PDQ-R computation shows response time for class 2 is 6.105397 seconds"
[1] "  PDQ-R computation shows response time for class 3 is 4.535833 seconds"

These values match the results in the book and the available Excel spreadsheet the authors developed in support of the book.

After I coded my PDQ-R solution I started to look into TeamQuest Model for capacity analysis and decided to do the same case study and see if the numbers agreed between all three sources.

I developed my TeamQuest Model and setup the visits and service time based upon the computed service demand for all three classes of workload:



However, the results did not match at all!



I contacted the TeamQuest folks with what I had done and showed my initial work both with PDQ-R and computations by hand and figured that I was doing something wrong with TeamQuest Model. I wanted to know what was up with the difference in values.

TeamQuest tech support finally got back to me with the solution. It appears that TeamQuest Model expects the service time to be set to 0.001 seconds and the number of visits modified to meet the service demand desired. I must've overlooked that in the TeamQuest Model tutorial but sure enough, it works. After sending some e-mails back and forth with TeamQuest it appears that the 0.001 second service time limit is only for CPUs. I am told that it is OK to use the actual visits and service times with AR IO's, but I haven't tested it out yet.



Results with:



Apparently if the TeamQuest agent is used to automatically create models based upon hardware and system utilization this is automatically set and is a non-issue. It's only when building a model from scratch from the ground up that it shows up. It's an easy fix but an unexpected issue.

Hey, everybody! Let's analyze the performance of a three tier system with PDQ-R!

This post is a continuation of this post that I did earlier this week.

Continuing the solution for Case Study IV: An E-Business Service from the book, Performance by Design: Computer Capacity Planning By Example.

In the previous post I discuss how to take the transition probability matrix and work backwards toward the original series of linear equations for solving the number of visits to a series of web pages. In this case study there are actually two types of visitors that result in two transition probability matrices that must be utilized. There are 25% of Type A visitors and 75% of Type B visitors.

Each tier of the hypothetical e-biz service is made up by a single CPU and a single disk drive. A matrix is supplied with the total service demand for each component by each page that is hit by visitors.

While it is simple to write some code to analyze web logs to generate the transition probability matrix based upon customer traffic it is very difficult to isolate the total demand at each component with chaotic customer traffic. But that is why we have load testing tools that are available to us. In a pseudo-production environment we are capable of simulating customer traffic to one page at a time and calculating the total demand for components. In this particular case only the CPU and disk drives are being modeled but for a real service we'd want to model the CPU, disk drives, memory system, network system, etc.

After running simulated customer traffic against isolated page hits we could generate a similar demand matrix for components and use it for what-if analysis.

I went ahead and kept my solution in R even though I saw where I thought that a perl solution would be more elegant (did I just use perl and elegant in the same sentence?) I designed my solution with two separate programs, one to spit out numbers and another to generate graphs of page response times and another showing component utilization. Both pieces of code make use of PDQ-R and allow for a variable number of web servers, application servers and database servers.

# Solution parameters 

gamma  <- 10.96;   # Rate into system
numWS  <- 1;       # Number of Web Servers
numAS  <- 1;       # Number of Application Servers
numDS  <- 1;       # Number of Database Servers

# external library
library("pdq");

# Constants #

E <- 1;
H <- 2;
S <- 3;
V <- 4;
G <- 5;
C <- 6;
B <- 7;
X <- 8;

PAGE_NAMES <- c("Enter", "HomePage", "Search", "ViewBids", "Login", "CreateAuction", "PlaceBid", "Exit");
COMPONENTS <- c("CPU", "Disk");
SERVER_TYPES <- c("WS", "AS", "DS");

WS_CPU  <- 1;
WS_DISK <- 2;
AS_CPU  <- 3;
AS_DISK <- 4;
DS_CPU  <- 5;
DS_DISK <- 6;

# Functions used in solution

VisitsByTransitionMatrix <- function(M, B) {
   A <- t(M);
   A <- -1 * A;
   for (i in 1:sqrt(length(A))) {
     j <- i;
     A[i,j] <- A[i,j] + 1;
   };
   return(solve(A,B));
};

CalculateLambda <- function(gamma, f_a, f_b, V_a, V_b, index) {
  return (
           gamma*((f_a*V_a[index]) + (f_b*V_b[index]))
         );
};


f_a    <- 0.25;    # Fraction of TypeA users
f_b    <- 1 - f_a; # Fraction of TypeB users

lambda <- 1:X;     # Array of lambda for each page 

SystemInput <- matrix(c(1,0,0,0,0,0,0,0),nrow=8,ncol=1)                             # 8.3, Figure 8.2, page 208
TypeA       <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,
                        0.2,0,0,0.4,0.2,0.15,0,0,0.25,0,0,
                        0,0,0.65,0,0,0.35,0,0,0,0,0,0.3,0.6,
                        0.1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,
                        0,0,0,0,0,0,0,0), ncol=8, nrow=8, byrow=TRUE);              # 8.4, Table 8.1, page 209
TypeB       <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,
                        0.2,0,0,0.45,0.15,0.1,0,0,0.3,0,0,
                        0,0,0.4,0,0,0.6,0,0,0,0,0,0.3,0.55,
                        0.15,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,
                        1,0,0,0,0,0,0,0,0), nrow=8, ncol=8, byrow=TRUE);            # 8.4, Table 8.2, page 210
DemandTable <- matrix(c(0,0.008,0.009,0.011,0.06,0.012,0.015,
                        0,0,0.03,0.01,0.01,0.01,0.01,0.01,0,
                        0,0,0.03,0.035,0.025,0.045,0.04,0,0,
                        0,0.008,0.08,0.009,0.011,0.012,0,0,
                        0,0.01,0.009,0.015,0.07,0.045,0,0,0,
                        0.035,0.018,0.05,0.08,0.09,0), ncol=8, nrow=6, byrow=TRUE); # 8.4, Table 8.4, page 212 (with modifications)

VisitsA <- VisitsByTransitionMatrix(TypeA, SystemInput);
VisitsB <- VisitsByTransitionMatrix(TypeB, SystemInput);

lambda[E] <- 0; # Not used in calculations
lambda[H] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, H);
lambda[S] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, S);
lambda[V] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, V);
lambda[G] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, G);
lambda[C] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, C);
lambda[B] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, B);
lambda[X] <- 0 # Not used in calculations

Init("e_biz_service");

# Define workstreams 

for (n in H:B) {
  workStreamName <- sprintf("%s", PAGE_NAMES[n]);
  CreateOpen(workStreamName, lambda[n]);
};

# Define Web Server Queues

for (i in 1:numWS) {
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("WS_%d_%s", i, COMPONENTS[j]);
    CreateNode(nodeName, CEN, FCFS);
  };
};

# Define Application Server Queues

for (i in 1:numAS) {
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("AS_%d_%s", i, COMPONENTS[j]);
    CreateNode(nodeName, CEN, FCFS);
  };
};

# Define Database Server Queues

for (i in 1:numDS) {
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("DS_%d_%s", i, COMPONENTS[j]);
    CreateNode(nodeName, CEN, FCFS);
  };
};

# Set Demand for the Web Servers

for (i in 1:numWS) {
  demandIndex <- WS_CPU;
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("WS_%d_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numWS);
    };
  };
};

# Set Demand for the App Servers

for (i in 1:numAS) {
  demandIndex <- AS_CPU;
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("AS_%d_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numAS);
    };
  };
};

# Set Demand for the Database Servers

for (i in 1:numDS) {
  demandIndex <- DS_CPU;
  for (j in 1:length(COMPONENTS)) {
    nodeName <- sprintf("DS_%d_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numDS);
    };
  };
};

SetWUnit("Trans");
SetTUnit("Second");

Solve(CANON);

print("Arrival Rates for each page:");

for (i in H:B) {
  print(sprintf("%s = %f", PAGE_NAMES[i], lambda[i]));
};

print("[-------------------------------------------------]");

print("Page Response Times");

for (i in H:B) {
  workStreamName <- sprintf("%s", PAGE_NAMES[i]);
  print(sprintf("%s = %f seconds.", PAGE_NAMES[i], GetResponse(TRANS, workStreamName)));
};

print("[-------------------------------------------------]");

print("Component Utilizations");

for (i in 1:numWS) {
  for (j in 1:length(COMPONENTS)) {
    totalUtilization <- 0;
    nodeName <- sprintf("WS_%s_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
    };
    print(sprintf("%s = %3.2f %%", nodeName, totalUtilization * 100));
  };
};

for (i in 1:numAS) {
  for (j in 1:length(COMPONENTS)) {
    totalUtilization <- 0;
    nodeName <- sprintf("AS_%s_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
    };
    print(sprintf("%s = %3.2f %%", nodeName, totalUtilization * 100));
  };
};

for (i in 1:numDS) {
  for (j in 1:length(COMPONENTS)) {
    totalUtilization <- 0;
    nodeName <- sprintf("DS_%s_%s", i, COMPONENTS[j]);
    for (k in H:B) {
      workStreamName <- sprintf("%s", PAGE_NAMES[k]);
      totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
    };
    print(sprintf("%s = %3.2f %%", nodeName, totalUtilization * 100));
  };
};

Here is a bit of sample output with 10.96 users entering the system per second:

[1] "Arrival Rates for each page:"
[1] "HomePage = 10.960000"
[1] "Search = 13.658485"
[1] "ViewBids = 2.208606"
[1] "Login = 3.664958"
[1] "CreateAuction = 1.099487"
[1] "PlaceBid = 2.074180"
[1] "[-------------------------------------------------]"
[1] "Page Response Times"
[1] "HomePage = 0.083517 seconds."
[1] "Search = 1.612366 seconds."
[1] "ViewBids = 1.044683 seconds."
[1] "Login = 2.323417 seconds."
[1] "CreateAuction = 3.622690 seconds."
[1] "PlaceBid = 3.983755 seconds."
[1] "[-------------------------------------------------]"
[1] "Component Utilizations"
[1] "WS_1_CPU = 49.91 %"
[1] "WS_1_Disk = 55.59 %"
[1] "AS_1_CPU = 71.11 %"
[1] "AS_1_Disk = 35.59 %"
[1] "DS_1_CPU = 38.17 %"
[1] "DS_1_Disk = 97.57 %"

Take a look at that database server disk utilization. Almost 100%! That isn't any good. Let's run the model with two database servers just to ease up on the poor drives!

I modify the line that reads "numDS <- 1; # Number of Database Servers" to read "numDS <- 2; # Number of Database Servers" and let 'er rip:

[1] "Arrival Rates for each page:"
[1] "HomePage = 10.960000"
[1] "Search = 13.658485"
[1] "ViewBids = 2.208606"
[1] "Login = 3.664958"
[1] "CreateAuction = 1.099487"
[1] "PlaceBid = 2.074180"
[1] "[-------------------------------------------------]"
[1] "Page Response Times"
[1] "HomePage = 0.083517 seconds."
[1] "Search = 0.237452 seconds."
[1] "ViewBids = 0.336113 seconds."
[1] "Login = 0.358981 seconds."
[1] "CreateAuction = 0.462042 seconds."
[1] "PlaceBid = 0.440903 seconds."
[1] "[-------------------------------------------------]"
[1] "Component Utilizations"
[1] "WS_1_CPU = 49.91 %"
[1] "WS_1_Disk = 55.59 %"
[1] "AS_1_CPU = 71.11 %"
[1] "AS_1_Disk = 35.59 %"
[1] "DS_1_CPU = 19.09 %"
[1] "DS_1_Disk = 48.78 %"
[1] "DS_2_CPU = 19.09 %"
[1] "DS_2_Disk = 48.78 %"

That's better. There is a significant reduction in page response time to boot:

Page            1 DS            2 DS            Diff
HomePage        0.083517        0.083517        0
Search          1.612366        0.237452       -1.374914
ViewBids        1.044683        0.336113       -0.70857
Login           2.323417        0.358981       -1.964436
CreateAuction   3.62269         0.462042       -3.160648
PlaceBid        3.983755        0.440903       -3.542852

Holy smoke! Adding that second database server makes a heck of a difference, doesn't it?

Being able to pull up numbers is great, but it doesn't have the impact that a good graph does. I love graphs! They are so great for conveying information to non-technical folks.

So, to do that I took my original solution and did a code spin, fold and mutilation to allow the generation of graphs. Here is the code that I wrote to generate the graphs.

# Solution parameters 

maxGamma      <- 11.2 # Maximum rate into system
steppingValue <- 0.1;

numWS  <- 1;       # Number of Web Servers
numAS  <- 1;       # Number of Application Servers
numDS  <- 1;       # Number of Database Servers

# external library
library("pdq");

# Constants

E <- 1;
H <- 2;
S <- 3;
V <- 4;
G <- 5;
C <- 6;
B <- 7;
X <- 8;

PAGE_NAMES <- c("Enter", "HomePage", "Search", "ViewBids", "Login", "CreateAuction", "PlaceBid", "Exit");
COMPONENTS <- c("CPU", "Disk");
SERVER_TYPES <- c("WS", "AS", "DS");

WS_CPU  <- 1;
WS_DISK <- 2;
AS_CPU  <- 3;
AS_DISK <- 4;
DS_CPU  <- 5;
DS_DISK <- 6;

# Functions used in solution

VisitsByTransitionMatrix <- function(M, B) {
   A <- t(M);
   A <- -1 * A;
   for (i in 1:sqrt(length(A))) {
     j <- i;
     A[i,j] <- A[i,j] + 1;
   };
   return(solve(A,B));
};

CalculateLambda <- function(gamma, f_a, f_b, V_a, V_b, index) {
  return (
           gamma*((f_a*V_a[index]) + (f_b*V_b[index]))
         );
};


f_a    <- 0.25;    # Fraction of TypeA users
f_b    <- 1 - f_a; # Fraction of TypeB users

lambda <- 1:X;     # Array of lambda for each page 

SystemInput <- matrix(c(1,0,0,0,0,0,0,0),nrow=8,ncol=1)                             # 8.3, Figure 8.2, page 208
TypeA       <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,
                        0.2,0,0,0.4,0.2,0.15,0,0,0.25,0,0,
                        0,0,0.65,0,0,0.35,0,0,0,0,0,0.3,0.6,
                        0.1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,
                        0,0,0,0,0,0,0,0), ncol=8, nrow=8, byrow=TRUE);              # 8.4, Table 8.1, page 209
TypeB       <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,
                        0.2,0,0,0.45,0.15,0.1,0,0,0.3,0,0,
                        0,0,0.4,0,0,0.6,0,0,0,0,0,0.3,0.55,
                        0.15,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,
                        1,0,0,0,0,0,0,0,0), nrow=8, ncol=8, byrow=TRUE);            # 8.4, Table 8.2, page 210
DemandTable <- matrix(c(0,0.008,0.009,0.011,0.06,0.012,0.015,
                        0,0,0.03,0.01,0.01,0.01,0.01,0.01,0,
                        0,0,0.03,0.035,0.025,0.045,0.04,0,0,
                        0,0.008,0.08,0.009,0.011,0.012,0,0,
                        0,0.01,0.009,0.015,0.07,0.045,0,0,0,
                        0.035,0.018,0.05,0.08,0.09,0), ncol=8, nrow=6, byrow=TRUE); # 8.4, Table 8.4, page 212 (with modifications)

VisitsA <- VisitsByTransitionMatrix(TypeA, SystemInput);
VisitsB <- VisitsByTransitionMatrix(TypeB, SystemInput);

numSteps     <- (maxGamma/steppingValue)+1;
#numSteps    <- (maxGamma/steppingValue);
numElements  <- numSteps*X;
numUElements <- numSteps*(3*length(COMPONENTS));

steppingArray <- 1:numSteps;
responseArray <- 1:numElements;
utilArray     <- 1:numUElements;

responseArray <- responseArray * 0;
utilArray     <- utilArray * 0;

componentList <- length(COMPONENTS)*length(SERVER_TYPES);

entryNumber <- 1;
for (serverType in SERVER_TYPES) {
  for (serverComponent in COMPONENTS) {
    componentList[entryNumber] <- sprintf("%s %s", serverType, serverComponent); 
    entryNumber <- entryNumber + 1;
  };
}; 

dim(responseArray) = c(X, round(numElements/X));
dim(utilArray) = c(3*length(COMPONENTS), round(numUElements/(3*length(COMPONENTS))));

loopCount <- 1;

for (gamma in seq(0, maxGamma, steppingValue)) {

  steppingArray[loopCount] <- gamma;

  lambda[E] <- 0; # Not used in calculations
  lambda[H] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, H);
  lambda[S] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, S);
  lambda[V] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, V);
  lambda[G] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, G);
  lambda[C] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, C);
  lambda[B] <- CalculateLambda(gamma, f_a, f_b, VisitsA, VisitsB, B);
  lambda[X] <- 0 # Not used in calculations

  Init("e_biz_service");

  # Define workstreams 

  for (n in H:B) {
    workStreamName <- sprintf("%s", PAGE_NAMES[n]);
    CreateOpen(workStreamName, lambda[n]);
  };

  # Define Web Server Queues

  for (i in 1:numWS) {
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("WS_%d_%s", i, COMPONENTS[j]);
      CreateNode(nodeName, CEN, FCFS);
    };
  };

  # Define Application Server Queues

  for (i in 1:numAS) {
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("AS_%d_%s", i, COMPONENTS[j]);
      CreateNode(nodeName, CEN, FCFS);
    };
  };

  # Define Database Server Queues

  for (i in 1:numDS) {
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("DS_%d_%s", i, COMPONENTS[j]);
      CreateNode(nodeName, CEN, FCFS);
    };
  };

  # Set Demand for the Web Servers

  for (i in 1:numWS) {
    demandIndex <- WS_CPU;
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("WS_%d_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numWS);
      };
    };
  };

  # Set Demand for the App Servers

  for (i in 1:numAS) {
    demandIndex <- AS_CPU;
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("AS_%d_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numAS);
      };
    };
  };

  # Set Demand for the Database Servers

  for (i in 1:numDS) {
    demandIndex <- DS_CPU;
    for (j in 1:length(COMPONENTS)) {
      nodeName <- sprintf("DS_%d_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        SetDemand(nodeName, workStreamName, (DemandTable[demandIndex + (j-1), k])/numDS);
      };
    };
  };

  Solve(CANON);

  for (i in H:B) {
    workStreamName <- sprintf("%s", PAGE_NAMES[i]);
    responseArray[i, loopCount] <- GetResponse(TRANS, workStreamName);
  };

  uArrayEntries <- 0;
  for (i in 1:numWS) {
    for (j in 1:length(COMPONENTS)) {
      totalUtilization <- 0;
      nodeName <- sprintf("WS_%s_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
        if (i == 1) {
          utilArray[uArrayEntries+j, loopCount] <- totalUtilization*100;
        };
      };
    };
  };

  uArrayEntries <- 2;
  for (i in 1:numAS) {
    for (j in 1:length(COMPONENTS)) {
      totalUtilization <- 0;
      nodeName <- sprintf("AS_%s_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
        if (i == 1) {
          utilArray[uArrayEntries+j, loopCount] <- totalUtilization * 100;
        };
      };
    };
  };

  uArrayEntries <- 4;
  for (i in 1:numDS) {
    for (j in 1:length(COMPONENTS)) {
      totalUtilization <- 0;
      nodeName <- sprintf("DS_%s_%s", i, COMPONENTS[j]);
      for (k in H:B) {
        workStreamName <- sprintf("%s", PAGE_NAMES[k]);
        totalUtilization <- totalUtilization + GetUtilization(nodeName, workStreamName, TRANS);
        if (i == 1) {
          utilArray[uArrayEntries+j, loopCount] <- totalUtilization * 100;
        };
      };
    };
  };

  loopCount <- loopCount + 1;

};

# Generate Response Time Graph

loopCount <- 0;
for (i in H:B) {
  arrayLength = numElements/X;
  if (loopCount == 0) {
    jpeg(file=sprintf("response_time_%d_WS_%d_AS_%d_DS.jpg", numWS, numAS, numDS), height=768, width=1024, quality=100);
    plot(steppingArray, 
         responseArray[i, 1:arrayLength], 
         xlab="Hits Per Second into System", 
         ylab="Response Time", 
         col=i,
         ylim=c(0,ceiling(max(responseArray))),
         type="l",
         pch=i,
         lwd=4);
    title(main=sprintf("Page Response Times for E-Biz with %d WS, %d AS and %d DS", numWS, numAS, numDS), font.main=2);
  } else {
    lines(steppingArray, responseArray[i, 1:arrayLength], col=i, pch=i, lwd=4);
  };
    
  loopCount <- loopCount + 1;
};
legend(1, ceiling(max(responseArray)), PAGE_NAMES[H:B], col=H:B, lty=1, lwd=4, cex=1.2);  
dev.off()

# Graph component utilization

loopCount <- 0;
for (i in 1:length(componentList)) {
  arrayLength = numSteps;
  if (loopCount == 0) {
    jpeg(file=sprintf("resource_utilization_%d_WS_%d_AS_%d_DS.jpg", numWS, numAS, numDS), height=768, width=1024, quality=100);
    plot(steppingArray, 
         utilArray[i, 1:arrayLength], 
         xlab="Hits Per Second into System", 
         ylab="Resource Utilization, %", 
         col=i,
         ylim=c(0,100),
         type="l",
         pch=i,
         lwd=4);
    title(main=sprintf("Resource Utilization for E-Biz with %d WS, %d AS and %d DS", numWS, numAS, numDS), font.main=2);
  } else {
    lines(steppingArray, utilArray[i, 1:arrayLength], col=i, pch=i, lwd=4);
  };
  loopCount <- loopCount + 1;
};
legend(0, 100, componentList, col=1:length(componentList), lty=1, lwd=4, cex=1.2);  
dev.off()

The code isn't what I would call elegant at this time as a lot of it is still hardcoded for the specific solution but it is a step in the right direction.

And let's take a look at response time between the single database server and dual load bearing database servers:



Take a look at the effect that the overworked database disk drive has on the response time. At 11.2 customers per second into the site and the response time has shot up to the 30 second mark. Definitely not what you want your customers to have to suffer through to be sure.

Here is a graph generated with the dual load bearing database server solution:



Holy guacamole is that a heck of an improvement or what? Any management type can look at these two graphs and immediately realize the impact to the system and the need for extra equipment.

And just for good measure, here is the resource utilization with both single and dual database servers:





Even a MBA grad can look at those graphs and realize that there is a problem that needs to be solved. w00t!

Ain't heuristic analysis fun?

Applying PDQ-R to this case study was a great exercise and I'm glad I undertook it. I can't wait to apply this to a real system and compare the results to see how well it works "in the real world." One thing that this type of heuristic analysis won't show is the interaction between pages with negative performance. In some circumstances I have seen performance programs between page that were thought to not be related. When Page A is put under duress Page B slows down even though it is thought that Page A is not related to Page B. Often it has been found that in a spaghetti line of object dependency that in some way the two pages were related. With the analysis of service demand of pages we can also find pages that have a lot of high service demand as well.

In the future along with page response times and normal metric reporting I think I'll add in service demand as well so that when pages do start to perform poorly hopefully the change in service demand will give an area to look into at the start of the analysis of the root cause of the performance problem to assist the developers.

Dr. Gunther has some more examples of applying Perl::PDQ to a variety of systems in his book, Analyzing Computer System Performance with Perl::PDQ. The source to the solutions in the book can be found in the PDQ download. But developing this solution from the ground up really drove some points home for me that will no doubt be useful in my future endeavors.

Hey! Let's take a transistion matrix and convert it into a series of linear equations to gonkulate the number of visits to various pages!

I've been working on learning the wonderful world of queuing theory. Two of the books that I've been using are Analyzing Computer Systems Performance: With Perl: PDQ and Performance by Design: Computer Capacity Planning By Example.

I've been learning how to apply Dr. Gunther's PDQ API, specifically, PDQ-R which is PDQ for the R programming language. Normally I'd use the Perl::PDQ module but I've found R to be quite handy for crunching numbers. If anything, I'd use perl to munge the data in preparation for crunching by routines in R.

I decided that I would apply PDQ-R to several of the case studies found in Performance by Design: Computer Capacity Planning By Example. In my first attempt to do so I found some numbers that didn't match between the results of PDQ-R and the Excel spreadsheet available as part of Performance by Design: Computer Capacity Planning By Example. I contacted Dr. Gunther and he found that there was a bug in PDQ 5.0.1 that he blogged about here. I guess that is my 15 mS of internet fame!

I got side tracked with some work related stuff and went back to the task of applying PDQ-R to some of the case studies found in Performance by Design: Computer Capacity Planning By Example.

The case study that I have started working on is in Chapter 8: Case Study IV: An E-Business Service.

The author provides us with a Customer Behavior Model Graph of a website:



From this graph we can create a set of linear equations that describes the number of visits to each page of the system presented in the CBMG (seen above). Taking a closer look at the graph we can see that there are probabilities of transitioning between one page to another. For example, the probability of moving from the Home Page (h) to the Login (g) is Phg. The probability of moving from the Login (g) page to the Search (s) page is Psg.

Using these probabilities we can create a set of linear equations to describe the number of visits to each page in the system based upon the number of visits into the entry of the system. In this case, there is only one way into the system and that is via the Entry (e) page.

We can look at the graph and manually derive the linear equations. For example, we can easily see that Vh = Peh*Ve. We can also see that Vg = Phg*Vh + Psg*Vs + Pvg*Vv. Doing this we eventually come up with eight linear equations that we can then solve. (The equations are listed on page 208 of Performance by Design: Computer Capacity Planning By Example).

The authors of the book provide us with all the probabilities in the form of a Matrix of Transition Probabilities on page 209. Here is that matrix:

                        (e)     (h)   (s)   (v)   (g)   (c)   (b)   (x)
Entry (e)               0       1     0     0     0     0     0     0
Home (h)                0       0     0.7   0     0.1   0     0     0.2
Search (s)              0       0     0.4   0.2   0.15  0     0     0.25
View Bids (v)           0       0     0     0     0.65  0     0     0.35
Login (g)               0       0     0     0     0     0.3   0.6   0.1
Create Auction (c)      0       0     0     0     0     0     0     1
Place Bid (b)           0       0     0     0     0     0     0     1
Exit (x)                0       0     0     0     0     0     0     0

So we can take those values found in the transition matrix and eventually gonkulate all the visits to each page. Although the transition matrix is provided by the author it is easy enough to re-create this matrix from web logs for a site.

A number of years ago I did the same thing for a major e-Commerce site. At the time my goal was to use the transition matrix to create a universal virtual user that would emulate what real customers did in production. At the time we were using a number of different virtual users and it was a real PITA to increase the number of vusers in a scenario. I figured if I could generate the transition matrix and generate random walks via the transition matrix that my vusers would better represent the traffic of real customers. Unfortunately, I never got to finish the implementation of the UberVuser but I did get to the point of generating a transition matrix. And that sucker was huge! It wasn't a nice 8x8 matrix but rather a 538x538 matrix. Yeah... That's a lot of elements!

So, thinking of my previous experience and looking at the author's provided transition matrix got me to thinking. There's bound to be a way of backing the transition matrix into the set of linear equations that can easily be solved in R (or any other method that can solve linear equations).

I worked it out by hand (more about this below) and everything looked good but to really make this useful I needed to automate the routine. I sat down looked at the problem and thought that I was going to have to do a bunch of element manipulation when the solution hit me. It turned out to be so damned simple! I love it when a complicated problem can be easily solved. It just amuses the hell out of me.

This is how I backed out the equations by hand. I know that the transition matrix has from "from" pages down the rows and the "to" along the columns. So for example, if I wanted to figure out the equation for the Vs I would work down the column and use the coefficients of the transition matrix as multipliers for the V variables.

For example, with the 3rd column with solving for Vs:

The third column of the transition matrix is:

    (s)
(e) 0
(h) 0.7
(s) 0.4
(v) 0
(g) 0
(c) 0
(b) 0
(x) 0

So to solve for Vs:

Vs = 0*Ve + 0.7*Vh + 0.4Vs + 0*Vv + 0*Vg + 0*Vc + 0*Vb + 0*Vx

Notice that we have a Vs on both sides of the equal sign. This corresponds to the loop on the Search node.

We can drop out the variables that are multiplied by zero.

Vs = 0.7*Vh + 0.4Vs

Repeat the steps for all the columns of the transition matrix and you end up with the equations to solve for. But, most packages want the equations setup in matrix form. So, let's get all the variables on the left side of the equal sign.

Vs - 0.4Vs - 0.7*Vh

Simplifies into:

0.6Vs - 0.7*Vh

If we do this with all the equations we can finally put it into matrix form. But we can't solve it yet. We need another vector to solve with. This vector will contain the input into the system. In this case the entry into the system is into the Entry page so the vertical vector is:

(e) 1
(h) 0
(s) 0
(v) 0
(g) 0
(c) 0
(b) 0
(x) 0

If we call our first matrix that we created from the transition matrix A and our input vertical vector B, we have the classic A*B = 0 that we can solve for.

Now we get into the R solution for the above. As I was working out the algorithm to setup the problem for calling the R solve() routine it hit me how simple the solution was!

Let's call our transition matrix supplied by the authors as M.

We can take transpose of M and let's call it A.

Take A and multiply by the scalar value of -1. This is the act of moving all the variables to the left side of the equals sign.

Now we combine variables we are solving for in each row of A by adding 1 to the trace of matrix A.

And there we have our final form of A. We can now solve for A*B = 0!

Here is my solution in R

# Define the column information for easy access by name

e <- 1;
h <- 2;
s <- 3;
v <- 4;
g <- 5;
c <- 6;
b <- 7;
x <- 8;

# M will be the supplied transition matrix

M <- matrix(c(0,1,0,0,0,0,0,0,0,0,0.7,0,0.1,0,0,0.2,0,0,0.4,0.2,0.15,0,0,0.25,0,0,0,0,0.65,0,0,0.35,0,0,0,0,0,0.3,0.6,0.1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0), ncol=8, nrow=8, byrow=TRUE);

# B is the vector that describes entry into the system
B <- matrix(c(1,0,0,0,0,0,0,0),nrow=8,ncol=1);

VisitsByTransitionMatrix <- function(M, B) {
   A <- t(M);
   A <- -1 * A;
   for (i in 1:sqrt(length(A))) {
     j <- i;
     A[i,j] <- A[i,j] + 1;
   };
   return(solve(A,B));
};

Visits <- VisitsByTransitionMatrix(M, B);

> Visits
          [,1]
[1,] 1.0000000
[2,] 1.0000000
[3,] 1.1666667
[4,] 0.2333333
[5,] 0.4266667
[6,] 0.1280000
[7,] 0.2560000
[8,] 1.0000000

> Visits[e]
[1] 1
> Visits[h]
[1] 1
> Visits[s]
[1] 1.166667
> Visits[v]
[1] 0.2333333
> Visits[g]
[1] 0.4266667
> Visits[c]
[1] 0.128
> Visits[b]
[1] 0.256
> Visits[x]
[1] 1

And there we go! Using this I could have taken that monster 538x538 matrix and easily figured out the number of hits for each page based upon the derived transition matrix and used the R routine for playing a lot of what-if scenarios. For example, what if we want to change some of the probabilities between nodes, how will that effect the visit to other pages? Now it is simple to crunch the numbers. Huzzah!

My next task is to apply PDQ-R to solving the rest of the case study. Later in the chapter there are service times associated with each page and that will determine the total service demand per page and ultimately the page response time for each page and the effect on internal components that will be modeled in my PDQ-R model.

I can really see where I could have used this modeling at my previous job at a major e-commerce site. It would have come in really stinkin' handy to be sure.

Automating the finding of coefficients for the USL

I got to playing around with R more and I've always found that to learn a language I need to solve problems with the language. I'm sure most everybody else does the same thing. My goal was to write a R function that imported a CSV with performance information to gonkulate against.

In my case, I'm using the basic performance information from "Guerrilla Capacity Planning." I've created a CSV file with the number of procs and the resultant ray trace benchmark from Table 5.1:

   1:  C:\Users\auswipe\Desktop>cat raw_throughput.csv

   2:  p,x

   3:  1,20

   4:  4,78

   5:  8,130

   6:  12,170

   7:  16,190

   8:  20,200

   9:  24,210

  10:  28,230

  11:  32,260

  12:  48,280

  13:  64,310

Then I wrote an R function to crunch the numbers:

   1:  uslCoefficients <- function(dataFile) {

   2:    uslData <- read.csv(dataFile, header=TRUE);

   3:    uslData$c <- uslData$x / uslData$x[1];

   4:    usl <- nls(c ~ p/(1+sigma*(p-1)+kappa*p*(p-1)),

   5:               uslData,

   6:               algorithm="port",

   7:               start=c(sigma=0.0, kappa=0.0),

   8:               lower=c(0,0));

   9:    sigma <- coef(usl)["sigma"];

  10:    kappa <- coef(usl)["kappa"];

  11:    return(list(sigma=sigma, kappa=kappa));

  12:  };

The function uslCoefficient returns a list where I can reference the "sigma" and "kappa" by named index:

   1:  > uslCoef <- uslCoefficients("c:\\Users\\auswipe\\Desktop\\raw_throughput.csv")

   2:  > uslCoef["sigma"]

   3:  $sigma

   4:      sigma 

   5:  0.0497973 

   6:   

   7:  > uslCoef["kappa"]

   8:  $kappa

   9:         kappa 

  10:  1.143404e-05 

  11:   

  12:  > uslCoef

  13:  $sigma

  14:      sigma 

  15:  0.0497973 

  16:   

  17:  $kappa

  18:         kappa 

  19:  1.143404e-05 

R is pretty nifty. I doubt I'll ever make use of all the power that is available but it'll be better than writing my own stat routines.

Using R to calculate coefficients of the Universal Scaling Law with Non-Linear Regression

Ooh! Doesn't that sound fancy?

Several weeks ago I purchased the eBook from O'Reilly called "The Art of Capacity Planning." I've always thought that load testing and capacity planning went hand-in-hand. One is not a replacement for the other but one can assist with the other. Load test helps out capacity planning by applying load to psuedo-production systems and capacity planning helps load testing by verifying results in load test against real world systems.

I finished "The Art of Capacity Planning" and wanted to read more on the subject and picked up a copy of "Guerrilla Capacity Planning" which has a lot more math than "The Art of Capacity Planning." One of the concepts is the Universal Scaling Law based on Amdhal's Law. Dr. Neil J Gunther is a smart cookie. He even has a Ph.D in Theoretical Physics which makes him closer to Gordon Freeman than I'll ever be! (Side question: Do Ph.D's in Theoretical Physics get crowbars at graduation?)

Anyhoo, in section 5.6.1 one of the methods in the book is to use Excel to do second degree polynomial regression for the calculation of two necessary coefficients, sigma and kappa. But when I tried to use Excel I was getting a negative value for sigma and one of the rules of the Universal Scaling Law is that the coefficients can never, ever, ever be negative. I just figured that I fat fingered something and tried it again and once again, got mismatching results.

I scratched my noggin, tried to figure out where I err'd and did some Googling and came across this entry of Dr. Gunther's blog:

Negative Scalability Coefficients in Excel

Because in Excel (and some other packages, like my TI-89) you can't put a constraint on the lower limits of the coefficient, you might from time to time get negative coefficients. But from reading the blog entry I see that other people are using R with success.

This is the first time that I've ever messed around with R for statistical purposes. In the past I've written some stat routines (years ago!) in C# for comparing before/after load testing results.

Here is how I used R from start to finish to gonkulate the coefficients.

Using the data from Section 5.3 I did the following in R:

First I defined my p array, which in the book is the number of processors used for ray tracing:

   1:  p <- c(1, 4, 8, 12, 16, 20, 24, 28, 32, 48, 64)

Then I defined my c array, which is the relative capacity for the number of processors used for ray tracing:

   1:  c <- c(1.0, 3.9, 6.5, 8.5, 9.5, 10.0, 10.5, 11.5, 13.0, 14.0, 15.5)

I combined both arrays into a data frame for later use.

   1:  df <- data.frame(p, c)

And when I check out the contents of df I get:

   1:  df

   2:      p    c

   3:  1   1  1.0

   4:  2   4  3.9

   5:  3   8  6.5

   6:  4  12  8.5

   7:  5  16  9.5

   8:  6  20 10.0

   9:  7  24 10.5

  10:  8  28 11.5

  11:  9  32 13.0

  12:  10 48 14.0

  13:  11 64 15.5

I can now use a non-linear regression routine with my data frame that I entered above.

   1:  usl <- nls(c ~ p/(1+sigma*(p-1)+kappa*p*(p-1)), df, algorithm="port", start=c(sigma=0.0, kappa=0.0), lower=c(0,0))

I can then access the coefficients by named index:

   1:  sigma <- coef(usl)["sigma"]

   2:  kappa <- coef(usl)["kappa"]

   3:   

   4:  sigma

   5:      sigma 

   6:  0.0497973 

   7:   

   8:  kappa

   9:         kappa 

  10:  1.143404e-05 

  11:   

Huzzah!

I can now interpolate the relative capacity based upon the USL and the coefficients that were previously gonkulated and add that to my current data frame, df, that I defined earlier. I do have to note that I was a slackard and did not apply the significant digits rules as outlined in Chapter 3 of "Guerrilla Capacity Planning."

   1:  df$proj_c <- p/(1 + sigma * (p - 1) + kappa * p * (p - 1))

There are the projected relative capacities. Yay!

   1:  df

   2:      p    c    proj_c

   3:  1   1  1.0  1.000000

   4:  2   4  3.9  3.479686

   5:  3   8  6.5  5.929346

   6:  4  12  8.5  7.745536

   7:  5  16  9.5  9.144406

   8:  6  20 10.0 10.253815

   9:  7  24 10.5 11.154233

  10:  8  28 11.5 11.898837

  11:  9  32 13.0 12.524174

  12:  10 48 14.0 14.259114

  13:  11 64 15.5 15.298811

And here I will make a simple little graph of the actual versus projected relative capacity:

   1:  plot(p, c)

   2:  lines(p, proj_c)

And here is the graph that is generated:



Kinda nifty, eh?

I can see myself using R more in the future. I'd rather write routines for automagic analysis of data with R than write my own routines from the ground up.

Push-To-Test

I went to a four hour presentation on Push-To-Test yesterday. It seems pretty nifty, the idea of wrapping an automated testing framework around a bunch of open source projects such as Selenium, soapUI and other goodies.

The presentation was a little hectic but it got the general idea across. I would have preferred to get to the meat of the subject quicker but the presenter did have 20 people to deal with and we had to go with the common denominator. No biggie.

The only thing that I wasn't too big was the idea of converting the Selenium tests for web tests from XUL into Java or Jython for more programmable control. After the conversion there is no going back to the original SeleniumIDE tool from what I saw. But, I guess that isn't much different from what I've done with LoadRunner and VSTS a bajillion times before when I had a bunch of custom code from the tree view into code. They did have an IDE so it's not too bad in hind site. I didn't get a chance to play around with the IDE to see how it compares to Eclipse or Visual Studio.

But, it is free for download to use the unsupported version, so I say thumbs up.

The idea of using Selenium for the web tests is pretty nifty as you get good control with AJAX controls that are a bit of a pain to control in HTTP Virtual user. I hadn't thought of doing that in the past.

I was told that transactions and nested transactions were supported but didn't get a chance to see them in action.

Learning QTP

It's been a long time between blog posts. Since this blog is mainly for myself, that isn't a problem. :-)

I'm starting to learn QTP and finding that it looks like a real handy tool for functional and integration testing. I'm still trying to get myself to not look at it as an alternate load testing tool and trying to use the interface and not look at each problem as more code to be written. I had the same problem initially with JMeter as well having come from a background where I would normally add a bunch of custom code after the initial script creation.

I'm not a big fan of VBScript but for the purpose at hand I have to say it is probably a better choice than using C like LoadRunner. Easier to create a COM object to be invoked with CreateObject for slicing and dicing of data.

I'm not sure if raw HTML can be pulled in with QTP, yet. I wouldn't be surprised if it can. QTP seems to be a pretty nifty tool. No doubt it will help with future job searches.