Extracting Transaction Response Time by Transaction Name per Load Test Run

Despite being annoyed that think time is included in the VS2008 web test response time I would like to be able to graph the response times for a given transaction name. I load my Load Test History database by the unique GUID and forgo the integer test ID to prevent possible duplicates in the future.

I created a query that takes transaction name and test GUID to pull all the transaction times.

   1:  SELECT LoadTestTransactionDetail.TimeStamp, LoadTestTransactionDetail.ElapsedTime

   2:  FROM  LoadTest.dbo.LoadTestRun INNER JOIN

   3:                 LoadTest.dbo.WebLoadTestTransaction ON LoadTestRun.LoadTestRunId = WebLoadTestTransaction.LoadTestRunId INNER JOIN

   4:                 LoadTest.dbo.LoadTestTransactionDetail ON WebLoadTestTransaction.TransactionId = LoadTestTransactionDetail.TransactionId AND 

   5:                 LoadTest.dbo.LoadTestRun.LoadTestRunId = LoadTestTransactionDetail.LoadTestRunId

   6:  WHERE (WebLoadTestTransaction.TransactionName = 'Some_Transaction_Name') AND (LoadTestRun.RunId = '987cce52-7608-40a7-ac85-96714af1ac6d')

   7:  ORDER BY LoadTestTransactionDetail.TimeStamp

   8:   

   9:   

There we go! All the transaction times for "Some_Transaction_Name:"

   1:  2008-07-29 15:58:04.467    7.944

   2:  2008-07-29 15:58:37.633    6.138

   3:  2008-07-29 15:58:55.930    5.243

   4:  2008-07-29 15:59:04.977    4.615

   5:  2008-07-29 15:59:25.607    6.461

   6:  2008-07-29 15:59:36.493    5.026

   7:  2008-07-29 15:59:40.820    5.256

   8:  2008-07-29 16:00:00.507    5.573

   9:  2008-07-29 16:00:04.940    5.457

  10:  2008-07-29 16:00:08.977    5.577

  11:  2008-07-29 16:00:16.547    5.506

  12:  2008-07-29 16:00:34.303    5.758

Uh-Oh. This isn't good.

I found out something today that I really don't like with VS2008 for load testing. I created my virtual user and setup think time ala WebTestRequest.ThinkTime with reasonable think times. After the testing was done I moved the results over to my LoadTestHistory database with a SP that I wrote about earlier in my blog. When I examined the results it turned out that the think time was included in the transaction response time. Nope, I don't like that. I dislike that about as much as I don't like having a failed counter for transactions.

I guess I can't be really surprised as the Begin/End transaction is wrapped around the WebTestRequest object and the think time is handled by the WebTestRequest object and there is no communication between the transaction begin/end and the WTR object (AFAIK).

I looked in the schema for the Load Test Repository and didn't see any fields where response time alone (without think time) would be recorded but there is definitely response time without sleep time recorded during the execution of the VS2008 loadtest. I figured there would be a field in the database with this value but I didn't see it.

Luckily, though, this is pretty simple to get around. If you aren't going to be using variable think times you can simple perform a Thread.Sleep(new TimeSpan(0, 0, 0, 5, 0)) which is similar to the LoadRunner idea behind lr_think_time(). Don't forget to add the "using System.Threading;" to your code to reference the Thread object since it is not referenced by default.

If you want to use variable think time then you'll have to come up with your own mechanism for varying the randomness of the think time. It's not that difficult, it is just a pain.

Clearing cookies in VS2008

In a coded web test that I am working on I have a need to clear all pre-existing cookies for a login page. I don't want the page identifying my vuser with a cookie that is assigned later on in the page hits.

While VS2008 has a CookieContainer that is a container for cookies, there is not a .Clear() method to clear out all the cookies like can be done with LoadRunner with the web_cleanup_cookies() API call.

I have found the way to do this is to use assign a new CookieContainer during the PreWebTest event to clear out all pre-existing cookies like this:

   1:  public Some_Coded_Web_Test() {

   2:    this.PreAuthenticate = true;

   3:    this.PreWebTest += new EventHandler<PreWebTestEventArgs>(ClearCookies);

   4:  }

   5:   

   6:  void ClearCookies(object sender, PreWebTestEventArgs e) {

   7:    Debug.WriteLine("Ha!Ha! I am clearing cookies!");

   8:    this.Context.CookieContainer = new System.Net.CookieContainer();

   9:  }

No more tasty, tasty cookies on script iterations.

Thinkin' about ditching Hyper-V

Seriously thinking about ditching Hyper-V for it's smaller brother, Virtual PC 2007. Sure, Virtual PC doesn't give the VM as much horsepower or as much contact with the metal of the host machine but by golly gosh the networking doesn't try to kill off Client for Microsoft Network.

I'm sure Hyper-V works great in the intended role of hosting guest VMs while racked in a server room someplace but the network issue that I am having is really irking me. Perhaps Hyper-V isn't ready for the type of prototyping work I want to do and Virtual PC 2007 will gimp along for my prototyping work.

Of course, the only issue that I have so far is that I have four VMs created and I'd have to go through all that trouble of reloading the OS. Well, I'm just about done with my load balancing prototyping so blowing away the VMs and removing the Hyper-V role wouldn't be that painful...

Of course, the biggest thing that I'd miss is the snapshot feature of Hyper-V. For prototyping and messin' around with a VM that is the greatest thing since sliced bread. I don't suspect we'll see that in a free Virtual PC anytime soon...

Hyper-V Networking Problems

I encountered some "interesting" situations with Hyper-V networking last night. For some reason I've lost the ability to connect to network shares on my machine, not good. I poke around and find for some reason that "Client for Microsoft Networks" is disabled on the Hyper-V virtual NIC. WTF?

I try to re-enable it and get the strange error message of "Your current selection will also disable the following features: Client for Microsoft Networks"

Uh, no? I'm trying to re-enable.

It appears that I am not the only person that has had this issue:

http://sqlsolace.blogspot.com/2008_04_01_archive.html

I disabled the Hyper-V virtual NIC and re-established my physical NIC and am able to truck on down the road. I find that if I delete the Hyper-V virtual NIC (external network connection) and create an internal virtual network (no physical connection) that the internal network has Client for Microsoft Network without issue. It's only the external virtual connections that are bjorked.

In the blog post above they said that if they un-installed all protocols and clients, reboot, and then re-installed that it was a work around. What a PITA. I haven't tried that yet as I got my Ubuntu 8.04 TLS installation up and running with HAProxy load balancing to two Win2k3 VMs which for prototyping works just fine on an internal virtual network just fine.

I like HAProxy. It is pretty full featured for a free, open source L7 load balance tool. It has a lot of features that remind me of NetScaler boxes (HAProxy even has a FreeBSD port so you too can run your balancer under FreeBSD just like NetScaler, 'cept of course it isn't NetSclaer).

I fired up my Win2k3 running NLB and the NLB manager shows both client machines to be up and running but it will only direct traffic to one of the Win2k3 VMs. Nice. And even niftier, if I ping the NLB cluster (in my case, it's 192.168.0.100) I will get back dupe pings as NLB creates the 192.168.0.100 VIP on both the NLB machine and all the client machines. Yeah, NLB is over kill for the stuff I want to do and HAProxy has nifty persistence support via cookies since it is a L7 balancer.

The only issue is that IIS logs the IP address of the HAProxy machine and not the actual client IP address. The HAProxy, much like the NetScaler product, sends the true IP in an X HTTP request header but you must use an ISAPI filter with IIS to log the true IP address to the IIS log. Apache has built in support for logging X-Header entries to the apache log. That is kinda handy but we aren't using apache. I've found some sample Visual C++ 6.0 ISAPI code for handling this task. Heck, I don't even have Visual Studio 6.0 installed or even Visual C++ 2008 installed on my machine.

Meh, not going to worry about it right now. Just suffice it to say that Hyper-V networking is irking me and NLB is ooky. HAProxy is pretty nifty and didn't take long to get a sample balancer up and running.

Getting back to load testing

I actually got to do some load testing today! Heck, I needed to figure out how many combinations for a test and even got to whip out some discrete math! Woooooo! Look at me go!

One of the things that I like about load testing is the "WTF?" factor. Finding those things that could never be found in "onesie-twosie" testing. In my current testing I am using JMeter to chunk 1500 e-mails and FTPs through a file routing system and I found that after the xfer is done that the w3wp.exe process would go high, spike a processor and the large object heap of the w3wp would go high. On top of that, during the same period where the w3wp goes whizzing off into la-la land the private byte usage of SQL Server climbs into the exosphere. We don't know what the problem is just yet but I like weeding stuff like that out with the initial raise of the eyebrow and a muffled "WTF?" Something tells me that I'll be brushing off my memory dump skills and using Son of Strike to figure out what is being dumped into the LOH.

I've been looking at load balancers and in my Google journeys I came across HAProxy and it seems to be quite a piece of art and an industrial load balancing solution. It runs natively under Linux but there is a FreeBSD version out in the ports tree. I haven't done anything with it yet as I've been busy with other tasks but I want to give it a test drive. It should be well more than what we need for the task at hand. 9 GB/s throughput? Yeah. We're never gonna see that. :-)

http://haproxy.1wt.eu/

NLB is ooky

Am I the only one that gets an oooky feeling from Microsoft's Network Load Balancer? Seems to be a little bit of over kill for just port 80 stuff. I built a virtual network in Hyper-V with three Win2k3 installations and played around with getting NLB up and running.

Sure, it gets up and running pretty easily but boy howdy if you ever want to make any changes like changing the NICs from a 10.0.0.0/8 space to 192.168.0.0/16 space. Yikes! Errors a plenty and that is after breaking down the cluster and deleting the cluster and then un-installing NLB from the target NICs and the balancer installation only to rebuild. Helllooo "unable to bind" errors!

A bit overkill for what I'm doing. I'm thinking maybe a nice balancer like Virtual Linux Server project or something similar would get the job done. Or perhaps Pure Load Balance (runs under FreeBSD, w00t!).

Maybe I just mucked up something in the process and need to play around more (this is the most likely scenario). Thank goodness for Hyper-V snapshots. They really save the day when prototyping like I have been doing.

I did run into an interesting situation. I created the three above mentioned VMs and Win2k3 didn't have a driver for my notebooks gigabit NIC. For some reason it took me a while to realize I could add a legacy NIC that Win2k3 could see.

I've enjoyed showing off the snapshot functionality by writing one line command lines that delete every file off the file system and then restoring with the snapshot and seeing the machine up and running 10 seconds later. Greatness for prototyping and fooling around!