Retrieving Standard Deviation for Transaction Response times with VS2008

One of my plans for the firm for which I work is to integrate a nightly automagic LT with the Continuous Integration effort. I want to be able to automagically compare results of the previous days results and see if anything is amiss. One of the tests I would like to apply is the two population mean test which requires having the average response time, standard deviation and the number of transactions.

VS2008 doesn't provide all of these statistics by default in their LT summary. LoadRunner includes this information by default in their summary results and extracting the values are quite simple if you publish the results to an Excel Spreadsheet.

It's not so quite as simple as that with VS2008 but it can be done with a little effort.

I have configured my LT to record individual metrics to the LoadTest Data Store so that I can query the tables and extract the information that I want. In the examples below I have identified the LoadTestRunId for a given run where I have plenty of metrics to number crunch. In my nightly build and test scenario I envision a simple query to pull the max(LoadTestRunId) to get the latest run to run queries to extract the required information.

I wrote this query today to get the information that I wanted:

   1:  SELECT WebLoadTestTransaction.TransactionName, 

   2:         LoadTestTransactionSummaryData.Average, 

   3:         STDEV(LoadTestTransactionDetail.ElapsedTime) AS StdDev, 

   4:         LoadTestTransactionSummaryData.Minimum, 

   5:         LoadTestTransactionSummaryData.Maximum, 

   6:         LoadTestTransactionSummaryData.Percentile90, 

   7:         LoadTestTransactionSummaryData.Percentile95, 

   8:         LoadTestTransactionSummaryData.TransactionCount

   9:  FROM   LoadTestTransactionDetail INNER JOIN

  10:                    WebLoadTestTransaction ON LoadTestTransactionDetail.LoadTestRunId = WebLoadTestTransaction.LoadTestRunId AND 

  11:                    LoadTestTransactionDetail.TransactionId = WebLoadTestTransaction.TransactionId INNER JOIN

  12:                    LoadTestTransactionSummaryData ON WebLoadTestTransaction.LoadTestRunId = LoadTestTransactionSummaryData.LoadTestRunId AND 

  13:                    WebLoadTestTransaction.TransactionId = LoadTestTransactionSummaryData.TransactionId

  14:  WHERE  (LoadTestTransactionDetail.LoadTestRunId = 33)

  15:  GROUP BY WebLoadTestTransaction.TransactionName, LoadTestTransactionSummaryData.Average, LoadTestTransactionSummaryData.Minimum, 

  16:                    LoadTestTransactionSummaryData.Maximum, LoadTestTransactionSummaryData.Percentile90, LoadTestTransactionSummaryData.Percentile95, 

  17:                    LoadTestTransactionSummaryData.TransactionCount

  18:  ORDER BY WebLoadTestTransaction.TransactionName

The QBE table entries look this this:


And I get the results I want:

   1:  TransactionName            Average             StdDev               Minimum Maximum 90th  95th  TransactionCount

   2:  someTransaction.Details    0.47510353043101411 0.090671477518885088 0.416   3.695   0.522 0.552 4617

   3:  someTransaction.FirstHit   0.44759085986571417 0.077896506427873255 0.384   3.956   0.502 0.526 4617

   4:  someTransaction.LookupName 0.51917002382499466 0.082360597958219192 0.45    2.844   0.577 0.605 4617

Huzzah!

I'll end up setting up an automagic comparison routine. A number of years ago I wrote a C# class for doing statistical tests so I'll probably end up using that for my left/right comparisons and number crunching.