Federico Silva Armas

Did you know? – NuGetGallery Build (part 6)

2012-01-19T01:28:00.001-08:00

Last post left off where NuGet.exe was used to download all packages before compiling the project. Let’s talk a little bit about conditions. If you remember from NuGet.targets, here is the target that restores the packages:

<Target Name="RestorePackages" DependsOnTargets="CheckPrerequisites">
    <Exec Command="$(RestoreCommand)"
          LogStandardErrorAsError="true"
          Condition="Exists('$(PackagesConfig)')" />
</Target>

I skipped over its DependsOnTargets=”CheckPrerequisites” attribute, this target is defined on the companion NuGet.settings.targets (in retrospect, I don’t know why there are two .target files where it seems that one would have sufficed):

<Target Name="CheckPrerequisites">
    <!-- Raise an error if we're unable to locate nuget.exe  -->
    <Error Condition="!Exists('$(NuGetExePath)')" Text="Unable to locate '$(NuGetExePath)'" />
</Target>

This is self explanatory, it will error if Nuget.exe was not found. Furthermore, the “RestorePackages” target will error if the packages.config file is not found.

PS: I did left off one piece of the puzzle: precompiling the razor files before building. This is defined in the RazorGenerator.targets which is also imported by Website.csproj.

Did you know? – NuGetGallery Build (part 5)

2012-01-18T03:12:00.001-08:00

In the previous post we left off trying to figure out how NuGetGallery downloads its required packages during the build process. We arrived at the file that contains the targets to do this NuGet.targets (simplified):

<Project ToolsVersion="4.0">
    <Import Project="NuGet.settings.targets"/>
    <Target Name="RestorePackages" DependsOnTargets="CheckPrerequisites">
        <Exec Command="$(RestoreCommand)"
              LogStandardErrorAsError="true"
              Condition="Exists('$(PackagesConfig)')" />
    </Target>
</Project>

Excellent, there’s the definition of a “RestorePackages” target that runs an Exec task which presumably will take care of downloading the packages. The “RestoreCommand” is defined in yet another targets file which is imported at the beginning, NuGet.settings.targets (simplified):

<Project ToolsVersion="4.0">
    <PropertyGroup>
        <NuGetToolsPath>$(SolutionDir).nuget</NuGetToolsPath>
        <NuGetExePath>$(NuGetToolsPath)\nuget.exe</NuGetExePath>
        <PackagesConfig>$(ProjectDir)packages.config</PackagesConfig>
        <PackagesDir>$(SolutionDir)packages</PackagesDir>

        <!-- Commands -->
        <RestoreCommand>"$(NuGetExePath)" install "$(PackagesConfig)" -o "$(PackagesDir)"</RestoreCommand>

        <!-- Make the build depend on restore packages -->
        <BuildDependsOn Condition="$(RestorePackages) == 'true'">
            RestorePackages;
            $(BuildDependsOn);
        </BuildDependsOn>
    </PropertyGroup>   
 </Project>

A lot of things happening here:
- There is a property definition to the location of NuGet.exe (which is included in the source)
- There is a property definition to a packages.config, which contains the list of packages that it needs to download.
- There is a property definition to a directory where the packages will be downloaded into.
- Then there is a property definition for the actual restore command, which is a call to Nuget.exe with the “install” parameter.

The next line is probably the most important one, it’s where the target defined previously is glued into the build process. “BuildDependsOn” is a special MSBuild property used to override the sequence of steps during build. In this case it adds the “RestorePackages” target at the beginning.

Don’t get tripped up by the condition included in the “BuildDependsOn” which uses a property with the same name as the target “RestorePackages” (this confused me for a while). Basically we are only going to download the references if the “RestorePackages” property is set to true, which if you remember it was set at the beginning of Website.csproj.

Did you know? – NuGetGallery Build (part 4)

2012-01-16T23:37:00.001-08:00

In the last post we left at the point where all previous build output from all projects was deleted. Let’s continue with the build from NuGetGallery.msbuild:

<Target Name="Build" DependsOnTargets="Clean">
    <MSBuild Projects="..\NuGetGallery.sln" Targets="Build" Properties="Configuration=$(Configuration);CodeAnalysis=true;" />
  </Target>

The msbuild task is invoked again, to run the Build target on the NuGetGallery.sln. This file references the “Website\Website.csproj”, MSBuild runs the Build target on this project.

Now, before it can compile anything it needs to download all the dependencies. Notice that NuGetGallery includes very few assemblies as part of the source, take a look at the “3rd Party” folder to see an unimpressive list of references. The trick is that it uses NuGet.exe to download all the extra references before the build. So how does it accomplish this magic? Let’s take a look at WebSite.csproj (simplified):

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Build">
  <PropertyGroup>
    <RestorePackages>true</RestorePackages>
  </PropertyGroup>

  <Import Project="$(SolutionDir)\.nuget\NuGet.targets" />
</Project>

Mh, there’s a property called “RestorePackages” that sounds like it’s used to, I don’t know, restoring nuget packages? Remember this one. And at the bottom there is an import of NuGet.targets. That’s where the answers must be!

Did you know? – NuGetGallery Build (part 3)

2012-01-14T23:11:00.001-08:00

In the previous posts we saw how the NuGetGallery build got kicked off by Build-Solution.ps1, which internally went and grabbed the connection string from the website’s web.config using Get-ConnectionString.ps1. At the end of the file it finally calls msbuild.exe passing the NuGetGallery.msbuild file and the connection string (pay special attention to the /t:FullBuild at the end which defines which target to run):

$projFile = join-path $scriptPath Scripts\NuGetGallery.msbuild
& "$(get-content env:windir)\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" $projFile /p:DbConnection=$connectionString /t:FullBuild

Let’s peel another piece of the onion and look at this msbuild file (simplified):

<Project DefaultTargets="Build">
  <Target Name="Clean">
    <MSBuild Projects="..\NuGetGallery.sln" Targets="Clean" Properties="Configuration=$(Configuration)"/>
  </Target>
  
  <Target Name="Build" DependsOnTargets="Clean">
    <MSBuild Projects="..\NuGetGallery.sln" Targets="Build" Properties="Configuration=$(Configuration);CodeAnalysis=true;" />
  </Target>

  <Target Name="RunFacts" DependsOnTargets="Build"> ... </Target>
  <Target Name="FullBuild" DependsOnTargets="RunFacts;UpdateDatabase" />
</Project>

This is where it starts to wire up a list of targets to run during the build. As you can see, the “FullBuild” target is empty, but depends on the “RunFacts” and “UpdateDatabase” targets. Working our way back, the “RunFacts” depends on “Build”, which in turn depends on “Clean”.

Phew, maybe we can now do some work? All right, at this point it calls msbuild to run the “Clean” target on the NuGetGallery.sln file. I am going to skip how this part works, the important thing is that all the projects included in the solution will delete all their intermediate and final build outputs. Note: the “Clean” target is defined in “Microsoft.Common.targets” file which lives in your framework directory.

Did you know? – NuGetGallery RequestModels

2012-01-13T13:26:00.001-08:00

A while back, while working on ASP.NET MVC, I learned to not pass the business models directly into my Views. Especially if they were Entity Framework entities. So I got in the habit of creating specialized ViewModels for this purpose.

But last week, while reading NuGetGallery’s source, I saw how they took this concept to the next step: RequestModels. You see, for the MVC applications that I wrote I was accustomed to re-using the ViewModel to capture data from the requests, my thinking was “well, I used the ViewModel to generate the response, it’s fitting that it should also be used to receive requests from the page that it created”.

I like the idea of further separating the concerns between the object that is used to create response and the one used to bind to the request. You’ll notice how the RequestModels have the validations from DataAnnotations that are to be used when binding to the request and only have the fields necessary to process each request.

For some reason, the application doesn’t use this extra separation all the time and some times receives the ViewModel (for example in the PackagesController.cs). An oversight?

I am going to play around with this way of organization in my own projects and see how it feels.

Did you know? – XUnit Theories

2012-01-06T00:03:00.001-08:00

Another discovery while reading through the NuGet Gallery source. As I was taking a look at their tests I bumped into RequireRemoteHttpsAttributeFacts.cs and saw this attribute:

[Theory]
[InlineData(new object[] { "POST" })]
[InlineData(new object[] { "DELETE" })]
[InlineData(new object[] { "PUT" })]
[InlineData(new object[] { "head" })]
[InlineData(new object[] { "trace" })]
public void RequireHttpsAttributeReturns403IfNonGetRequest(string method)
{
    ...
}

Now, I realise this may be old news since XUnit Extensions has been around since at least 2008, but it was new to me. This works very similar to NUnit Testcase attribute, each of the InlineData will be passed to the test as an argument.

There are different data providers including in the xunit.extensions assembly:
1. ExcelData
2. InlineData
3. OleDbData
4. PropertyData
5. SqlServerData

An introduction blog post about xunit extensions can be found here (by Ben Hall).

Did you know? – NuGetGallery Build (part 2)

2012-01-05T23:40:00.001-08:00

To continue my previous post, let’s take a peek at Get-ConnectionString.ps1, because there is something in there that I had no idea you could do:

function Get-ConnectionString($configPath, $connectionStringName) 
{
  $config = [xml](get-content $configPath)
  
  $connectionString = ($config.configuration.connectionStrings.add | where { $_.name -eq $connectionStringName }).connectionString
  
  $connectionString = $connectionString.Replace("=", "%3D")
  $connectionString = $connectionString.Replace(";", "%3B")

  return $connectionString
}

Look at line #3, what’s up with that “[xml]” statement? Is it trying to cast the output of get-content as xml? That’s ok, I guess. But then the next line hits:
($config.configuration.connectionStrings.add | where { $_.name -eq $connectionStringName }).connectionString

It can traverse XML nodes and attributes like property objects?!? Kudos to you Mr. PowerShell. I have no idea how that [xml] works or what others are supported but I did a small experiment in the PowerShell REPL:

$foo = [xml] "<span></span>"
$foo.getType()

And it says that it is of type System.Xml.XmlDocument. The fact that you can traverse it and the way the author uses the where command-let to filter is awesome (was that you Drew? Kudos to you too man).

So now you know, maybe it will come handy some day.

Did you know? – NuGetGallery Build

2012-01-05T01:37:00.001-08:00

Today I read over the NuGetGallery source and naturally, started with the first entry point according to the documentation: their build script.
Build-Solution.ps1
There is a lot of PowerShell stuff that I didn’t know about in this short file, so let's look at it line by line:
1. The param() function is used to process named parameters, so the connection string can be passed when calling into this script.

param($connectionString = "")

2. Next it gets the path to the directory that contains this script file and uses it to invoke another .ps1 file. Note that the “.” is used to set PowerShell to parse the line in command mode. We’ll look at what Get-ConnectionString.ps1 does some other time, for now just be aware that it is going to load a new function to be used.

$scriptPath = Split-Path $MyInvocation.MyCommand.Path
. (join-path $scriptPath Scripts\Get-ConnectionString.ps1)

3. This is where the function is used, note that the web.config file is passed to it, so apparently that script is capable of parsing the xml to retrieve the connection string.

if ($connectionString.Trim() -eq "") 
{
  $connectionString = Get-ConnectionString -configPath (join-path $scriptPath Website\web.config) -connectionStringName NuGetGallery
}

4. Finally msbuild is invoked. Three things to note:
- it is not building the .sln file directly but instead a custom .msbuild file is used (take a look at that for some nice usage of targets).
- it uses the get-content command let to retrieve the value of an environment variable for the windows directory
- note the “&” at the begnning of the line? This turns PowerShell into expression mode so that the string is evaluated as an expression. First time I ever saw that.

$projFile = join-path $scriptPath Scripts\NuGetGallery.msbuild
 
& "$(get-content env:windir)\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" $projFile /p:DbConnection=$connectionString /t:FullBuild

Pretty cool stuff, source file here.

Did you know? - WCF and ClientMessageInspector

2012-01-03T23:35:00.001-08:00

Last month I got a feature request to add time logging to the Atlassian .NET SDK. I thought it would be simple until I ran into a peculiar JIRA bug: it turns out that the SOAP response to any call involving time tracking blows up miserably. Curse you JIRA!

After searching the inter webs, I found what the error is on the response and how to fix it. Now what I needed was a way to intercept the response from the server and fix it up before it got passed to the internals of WCF

ClientMessageInspector to the rescue!

First create one of these guys that can update the content of the message

class RemoteWorklogMessageInspector : IClientMessageInspector
{
    public void AfterReceiveReply(ref System.ServiceModel.Channels.Message reply, object correlationState)
    {
        // update the contents of the response and return a new one with the changes
    }

    public object BeforeSendRequest(ref Message request, IClientChannel channel)
    {
        return null;
    }
}

Then you have to register it, which wasn’t totally intuitive. Next you need to create an EndpointBehaviour that adds the inspector to the ClientRuntime.

class RemoteWorklogPatchBehavior : IEndpointBehavior
{    
    public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
    {
        clientRuntime.MessageInspectors.Add(new RemoteWorklogMessageInspector());
    }
    
    // other methods from the interface
}

Finally, add the behaviour to the endpoint of the service:

var jiraSoapServiceClient = new JiraSoapServiceClient(binding, endpoint);
jiraSoapServiceClient.Endpoint.Behaviors.Add(new RemoteWorklogPatchBehavior());

Now, whenever a SOAP request goes to JIRA, the response will be inspected and updated if necessary to fix that nasty bug.

Look here for the full source.

“Tags” plugin now available in plugins.atlassian.com

2012-01-03T23:02:00.001-08:00

For those interested, the entry is available here.

Now I need to convince the admins to install it in our internal instance so that I can start using it for my test sessions.

- Federico

LeoMejor

2011-12-30T13:44:00.000-08:00

Awhile ago, I blogged about a website that my sister and I were building using backbone.js. The site is now finished and all that remains is for my sister to periodically add more content.

http://leomejor.apphb.com

“Tags” plugin for Confluence

2011-12-04T01:19:00.001-08:00

Exploratory Testing is an approach that I use a lot while testing a feature, and part of it is taking good notes while doing it. For years I used to take notes with pen and paper but eventually I evolved to do it directly on the computer (mainly because it's way easier if I am taking lots of screen shots and video).

My preferred tool is OneNote, primarily because it does a good job for staying out of my way and it is very flexible (the sync feature is also nice for collaboration). When I joined Atlassian, everything is written on an internal wiki using Confluence. For the most part, it does a good job for test note taking except that I really missed the “tags” feature of OneNote. I used to rely heavily on it for my own organization of test tasks, so I decided to take some 20% time and write a Confluence plug-in to add this.

Overview
Confluence 4 plug-in that creates a status list created from "tags" added to the content of pages.
- Bitbucket project
- Download (.jar)

Features

Tag Content
Use any of the built in emoticons to "tag" a line while typing in Confluence. For fast editing, you can use short cuts:
- (/): check mark
- (x): error
- (?): question mark
- (!): exclamation mark
- (on): light bulb
"Tags" can have any meaning that you want, it is a way of marking content for follow up.

Tag summary
On a parent page, add the "Tag Summary" macro to view a tree that summarizes all the tags available on child pages.

Page hierarchy
The tag summary becomes really useful when there are many hierarchical pages of content and the author needs to keep track of action items across them.

Filter tags
Filter which tags to show on the tree to quickly identify the items that you care about.

Preview
Click on a tag icon on the tree to open a preview window that loads the content of the child page, scrolled to the section where the tag is located.

Clear tags
When an item is resolved you can clear the tag directly from the tree. This will remove the icon from the target page. Right click on a tree node to bring up the context dialog.

- Federico

Projects page

2011-11-18T20:46:00.001-08:00

Small announcemnt that there is now a page on my blog where I’ll store information of the projects that I’ve worked on.

- Federico

Quickly create issues with the JIRA Tray App

2011-11-18T13:51:00.001-08:00

I wrote the Jira Tray App to solve two problems that I had in my daily workflow:

I practice exploratory testing and divide my work into focused sessions. I don’t open bugs as I am testing, instead I take notes of the problem and keep testing. At the end of the session, I go over my notes, perform deeper investigation of the issues I found and THEN open bugs (in a batch). I wanted a tool that would speed up creation of consecutive bugs.
At Atlassian, every team has its own JIRA instance to track their work with their own conventions of which fields are required. I wanted a tool that could keep track of all the servers for me and expose a common interface to create issues in all them.

Windows Tray

This tool is a windows tray application that maintains a list of templates to quickly open JIRA issues. The idea is that you create “issue templates” with pre-populated values for each of your JIRA projects.

Issue Form

The issue template form is used to create new issues based on the template or manipulating the list of templates.

Template: The name of the template (what appears on the tray menu).
The summary of the issue.
The description of the issue.
Fields of the issue to create (also includes the data of the server). For fields that support it, a picker is available with valid values from server.
List of attachments.
Clears the list of attachments.
Adds an attachment from disk to the list.
Pastes the contents of the clipboard as an attachment (either as an image or text file). Really handy if you use a screen capturing tool like Jing.
Template: Saves the values of current template.
Template: Clones this template.
Template: Deletes this template.
Automatically closes the form when an issue is created.
Creates an issue with the current values.
Closes the form without saving template or creating an issue

Handy Features

The “Paste” from clipboard for adding attachments to an issue (images or text files)
The app automatically saves the last issue created as a template, if you need to open multiple issues with some similar data between them.

Up Coming Features

Ability to rename attachment files
Ability to save/restore templates (so you can pass them around)
Querying capabilities.

Resources

Source
Binaries
Built on top of the Atlassian .NET SDK

- Federico

Upgrade testing is rotting my brain

2011-11-16T02:27:00.001-08:00

Installer and upgrade testing is probably the worst activity that I have to do as a tester. To put it bluntly: I hate it.

I like creating the strategy and the plan to get good coverage with smart test design, but when one gets to executing these tests, it tends to fall into the repetitive, slow grind that quickly spirals into tester fatigue land.

For the past week, I have been reimaging a set of machines, preparing them with test data, performing an upgrade and verifying the state of the application and trying to find if anything unexpected happened. If I find a failure, instead of the usual rush of adrenaline that every tester gets, I groan when I think of all the extra reimaging I’ll have to do to narrow down the repro steps. Plus, trying the scenario on the previous version and the “other” platform to gather more information for developers.

The sad part about this is that I feel like I am not learning anything during these periods. And I get sloppy after a while. A part of me thinks that this salt-mine work is necessary, but another wonders if there is a better way to do this…

Anybody has good suggestions for a fellow tester?

- Federico

A simple WebDriver runner for QUnit

2011-11-12T13:16:00.001-08:00

I have been working on a Confluence plugin during my 20% time and, as part of it, writing a number of qunit tests. I was looking for a way to run them as part of CI and found this qunit-maven-plugin out there. After trying to use it unsuccessfully for a couple of minutes, I confess to falling into the well-known “reinvent the wheel” mentality trap.

I’ve been working with WebDriver quite a lot and the idea came to me that it would not be hard to create a simple qunit runner using it. Well, here it is.

How to use it?

Download and include it in your pom.xml (in this case I downloaded the .jar to the /lib folder of my project):

<dependency>
    <groupId>org.farmas</groupId>
    <artifactId>qunit-webdriver-runner</artifactId>
    <version>0.1</version>
    <scope>system</scope>
    <systemPath>${project.basedir}/lib/qunit-webdriver-runner-0.1.jar</systemPath>
</dependency>
<dependency>
    <groupId>com.atlassian.selenium</groupId>
    <artifactId>atlassian-pageobjects-elements</artifactId>
    <version>2.1.0-m8</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>jetty</artifactId>
    <version>6.1.24</version>
    <scope>test</scope>
</dependency>

Then you can use the QunitWebDriver from your test (in this case I am using JUnit):

public class QunitTest {

    public static QunitWebDriver driver;

    @BeforeClass
    public static void start() throws Exception
    {
        // these are the class loaders particular to my project
        driver = QunitWebDriver.start(
                        QunitTest.class.getClassLoader(), 
                        TagParser.class.getClassLoader()); 
    }

    @Test
    public void test()
    {
        driver.runTest("/spec.htm");
    }

    @AfterClass
    public static void stopServer() throws Exception
    {
        driver.stop();
    }
}

How does it work?

The module assumes that all your javascript, html and css files (product and test) are included as resources.
At the beginning of your test, start a QunitWebDriver passing the class loaders that are capable of locating all your resources.
QunitWebDriver will start a local webserver (using Jetty) that can serve requests from the resource files.
QunitWebDriver will then download, configure and start an instance of FireFox browser using WebDriver.
Lastly, you call QunitWebDriver.runTest() passing the html page that will load qunit and run your tests.
QunitWebDriver checks the UI of the page to verify that no test has failed.

atlassian-pageobjects-elements?

You might be wondering what’s up with this dependency. This is the Atlassian WebDriver PageObject library, another project that I worked on at Atlassian. One of the things I get for free by using it is auto-download and configuration of browsers which was very useful to get this thing going quickly

Watch it live

Unable to display content. Adobe Flash is required.

qunit-webdriver-runner

- Federico

What are they “really” asking?

2011-11-07T02:58:00.001-08:00

My previous manager taught me an important principle for navigating the work place: whenever somebody asks a question, think about what they are really asking. This came to mind because I recently moved to a new team full-time at work and one of the ceremonies that they have is a weekly status/progress/management type of meeting where I get asked this question:

“What is the feeling from QA?” [this from product manager regarding the project that we are working on]

I normally create a quality report that tells the story of the product state, but in these meetings nobody cared about any of that. I was asked for a some form of statement. My first reaction to these types of ambiguous questions is to feel the heat of being put on the spot and quickly attempt to synthesize the vast testing space into a judgment in seconds. This valiant effort is quickly followed by confusion as my mind races to evaluate the thousands of ways that I can tackle the answer.

My natural solution at this stage is to simplify the terms of the problem by asking clarification of what the person means. Is there something specific that they want me to focus on? Naming examples of what I can elaborate on helps. However, what I discovered is that most product managers are as confused as I am and don’t really know what they are after with these questions!

This triggers my third response: suspicion. Are they just asking for the sake of asking? Are they really going to do anything with this information? This bias comes from years of cases where no matter what is the assessment of QA, the plan remains unchanged, leaving the suspicion that the decision has already been made and voicing my opinion is just a formality.

On good days, serenity brings me to my final reaction: humbleness. Product managers have a very difficult job, they never have complete information and have to balance many things to make a responsible call. Most of the times all they have to go with is their emotional reaction to something I say. That’s a tough position.

What I do now is make sure that I can always summarize my test report into these 3 parts: a) top two important issues that are active, b) top test activity that we have not done yet, and c) top test activity that we have had to cut. Don’t fret too much, in my experience, product managers have a difficult time reacting to these (unless the risks are emotionally charged). Instead, the most important part is that I follow these statements by asking the product manager if there is anything else he/she wants to know. I think that this helps form clarity of what is important in the eyes of the manager. Remember that they are the ones that are going to make the ship/no-ship decision at the end and the more I can help this person be aware and articulate what information he/she feels is important the better. Make sure to address these in the written test report that you send later (where it can be read in calmness).

Lastly, I never do the job for them. I do not answer questions related to ship-readiness, if we’ll be able to ship on time or if we should block the release. This is the product manager’s job (or if not them, someone higher up). If he/she wants my opinion I’ll be happy to give it over a cup of coffee. I like lattes.

- Federico

Learn Spanish while I learn Backbone.js

2011-09-24T21:06:00.001-07:00

My sister is a bilingual teacher in the US and after seeing all the cool work books and educational software that exist for children to learn English, she decided to create something similar for Spanish. After designing some of the exercises, she asked me if I could create a program so that children could practice.

I decided it would be a good opportunity to learn Backbone.js. After watching some tutorials and screen casts I jumped in. I had to rewrite it two times because I got stuck and I feel like I am still learning it, but I thought I’d share it anyway.

Head over to LeoMejor.apphb.com to try it out and learn some Spanish in the process!

If you are interested, head over to the BitBucket project to checkout the source. (Be mindful that all the art is my sister’s, if you want to use any of it please let me know.)

I’ll follow this up with a write up about my experience with Backbone and some pitfalls that I fell into.

- Federico

My promise as a (future) developer

2011-08-27T21:53:00.001-07:00

With slightly different choices, in a different life I could have been a developer. In fact, I am somewhat convinced that in my next gig, I’ll shoot for becoming a developer with a strong test mentality.

If and when that happens, I might land on a team where there are dedicated QA folks, and in such an event I want to promise something to any tester that happens to work with me. That’s right, I am talking to you tester, the one who is going to help me create awesome features. This post is dedicated to you:

1. I promise to ping you, send you an IM or shout at you when my fellow programmers and I are having a discussion about the design of a feature that you are going to test. I will constantly use this phrase: “Could you guys hold on a minute, I am going to grab our tester, he/she needs to be here”. If you are not there, I will do my best to bring you up-to-speed with any decisions made.

2. I promise that when I think I have a first implementation done, I will go grab you and walk through the code and modules with you, including automated tests. I will ask for some of your time to brainstorm additional test scenarios that we should take into account. I will welcome your feedback.

3. I promise to include you in any code reviews that I create.

4. I promise to sit with you and make sure that you know how to build and install latest versions of the product, as well as how to setup a debugging environment and run any automated tests.

5. I promise to sit with you and create a test mission plan for our feature and divide the exploration between us. I understand that sometimes you will be overwhelmed with other work, and in this case I promise to do my best and create the test missions and run exploratory sessions on my own (despite my obvious bias). I will keep test notes of what I have tried, any assumptions that I have made and areas where I am unsure about some implementation.

6. I promise that whenever I resolve a bug I will include detailed explanation of what was the problem as well as how I fixed it. I will include notes of what I did to verify my changes including manual tests performed.

If I ever do make the jump, and you feel I am not treating you this way, please print this blog post and bring it to me and shove it in my face.

- Federico

Mother nature always wins

2011-08-15T01:23:00.001-07:00

This weekend I spent some time viewing Paul Irish’s 10 Things I Learned from the JQuery Source and its follow up. I was amazed at some of the code patterns that Paul shows and at one point the wife (who I will call “La Jefa”) popped up her head from behind her computer as I exclaimed “Whoa!” when I saw a particular piece of code.

When asked what was so amazing, I pointed at the screen, burping out some incoherency in the lines of: “look how this crazy guy coded up that stuff”.

She turned to me and said, “You want to look at something REALLY amazing?” And she turned her laptop to show me this:

Where’s the Octopus?

No matter how amazing the pieces of software we create are, a look at this puts things into perspective.

- Federico

Tester as an identity, have we gone too far? – Part 2

2011-07-31T04:53:00.001-07:00

Summary

The need to maintain the tester’s identity is preventing us from investing in testing skills for other team members, especially programmers. In this series of blog posts, I will focus on this simple idea and how a development team might benefit from taking responsibility of more test approaches than just automation.

Part 1: Rise of the Tester Identity

Part 2: A world divided, but collaborating

Regardless of how and why, the fact is that internal test organizations exist and they are usually separate from programmers. The focus of managers and leaders then is to figure out a way to have these two groups collaborate with each other, with the assumption that each brings a different but equally important skill to the table, which if balanced correctly can produce better software.

The point here is to be aware and accept that organizations spend time creating processes to satisfy two conditions: keep testers as a separate group to programmers, and keep the two groups working together as a unit. This is a noble cause, and it plays to our desires of “team work” and “better together”. I don’t doubt that there are organizations that have been able to create a co-habitat that produces good results for them, however, we should not lose track of the risks.

What follows are some examples of how testers and programmers interact with each other that can be problematic. Note: I like to over simplify and show the behavior at its extreme, perhaps you would be able to identify a grain of truth in real organizations.

Creator - Destroyer

The basis of this setup is to have each group apply force at opposite sides which hopefully results in healthy compromise. The groups are kept separate by focusing on different software dimensions. Programmers are encouraged and motivated to write as many features as possible, testers are encouraged and motivated to break the software by any means possible.

I have heard this mentality put into words when describing the roles of these groups. For example: “Programmers have a creative mentality, testers have a destructive mentality” and the more ingenious “Programmers try to put as much code and features out; testers try to stop them by showing how buggy they are.”

The risk of this approach is that the groups may to become too polarized. A group might be pushing too hard that over-powers the other or a group might start caring more about “winning” the battle against the other group than delivering value to customers. Actually, one symptom of this arrangement is when people start talking about “winning fights” against developers over some bug.

Creator - Verifier

Collaboration in this model has testers functioning as a checker for programmer’s work. The programmer claims that he/she is done and passes the work for a tester to verify that the claims are true and that the requirements are met. Bugs are mistakes the developer commits over a known space of functionality.

This setup tends to be more harmonious than the polarizing structure, the tester is merely checking that the software does what it's supposed to do. The down-side is that it devalues the work of testing as a non-skilled activity. Programmers do not include testers in any design activity and testers feel unappreciated.

Creator - Investigator

This model of collaboration rests on the assumption that the tester will question the unwritten requirements--all assumptions made by programmers are fair game to be evaluated against their impact to customers. Testers are free to use whatever approach best fits the context of the project and provide an assessment to the project owners over more dimensions than just “defects”.

In my opinion, this is one of the healthier models of collaboration, the skills of the tester are recognized and the team is thankful for the increase in awareness that the tester’s findings bring to attention. Besides the difficulty of implementing this structure, the risk is that programmers can begin to take design less seriously, because the tester is there to work out all the details of the impact of the current implementation.

Creator - Gate Keeper

Under this model, testers have veto power. If they are so intimately aware of the state of the product and not under the direct reporting structure as programmers, they have the most objective perspective and are best capable to decide whether the project is ready to be released or not.

At its best, it gives testers a way to have equal say in the decision to release software and “stand up for quality” if the need arises. At its worst, it turns into law enforcement where testers refuse to sign off on a release until they are satisfied with the quality.

In my opinion, this is a facade of the intentional polarized groups presented above. Its implications are mostly negative: apparently programmers would run wild releasing bad software because there would be no one to stop them and programmers are simply not good at effectively considering other dimensions of software.

Other Possibilities

What these collaboration strategies have in common is the belief that there must be two groups. It’s almost a requirement and we invest time and effort on creating processes, team visions, role descriptions and training so that these two groups of people learn how to work well together.

I believe that two things are happening here: a) the testers identity as a profession demands that it holds a place of value, and b) too little credit is given to programmers, the assumption is that they lack testing skills and don't care to learn them.

So what are we to do? Destroy the test team and move the testing responsibility to the programmers? Isn’t that what we mean by “moving quality upstream” after all?

Not necessarily. Stay tuned.

- Federico

CAST 2011 keynotes on UStream.TV

2011-07-31T03:42:00.001-07:00

For those of you that, like me, won’t be attending CAST 2011 but still want in some of the action. The keynotes and the “Emerging Topics” tack will be streamed like on UStream . August 8th and 9th.

UStream: http://ustream.com/channel/CASTLive

Don’t miss it!

- Federico

Tester as an identity, have we gone too far? – Part 1

2011-07-16T21:46:00.001-07:00

Summary

The need to maintain the tester’s identity is preventing us from investing in testing skills for other team members, specially programmers. In this series of blog posts I focus on this simple idea and how a development team might benefit from taking responsibility of more test approaches than just automation.

Rise of the Tester Identity

I am too young to know how it started. It’s a bit like the Big Bang really, we all have theories about how it all began, but nobody actually knows. My guess is that at some point, really smart persons realized that software was too complex that they might benefit from double checking if the thing worked before they sold it. For good conscience sake.

Then I suspect people realized that making sure that things work was boring, hard and time consuming (depending on who you asked). Some bright fellow had the idea, “Hey, let’s pay somebody to do it” And BAM! The dedicated tester came into being.

From the beginning I think the tester was born into a world of struggle. I can imagine that when presented with this new person, programmer’s reactions fell into one of these:

“They are useless, we write perfect code!”
“They are dumb, they don’t know as much as we do.”
“They are slaves, go do the things that we dislike.”
“They are evil, they just make us look bad.”
“They are a necessary evil, we ship better software thanks to them”
“They are ok, but without us there would be no business, so don’t get ideas into your head.”
“Who are these guys again? What do they do? I am confused”

Fortunately, there was a bit of a Renaissance, testers united and created our own national identity. We weren’t completely on our own, companies (for reasons that I won’t go into) promoted the rise of the tester as a separate identity. There were great luminaries and thought leaders that defined, and continue to define, the role of a tester in the development process.

I believe that, in some part, the testing discipline needed to justify its existence. And so we created our membership group which has the side effect of creating an out-group of pretty much everyone else. This is not without merit, there are differences between a “tester” and a “non-tester” as well as concrete skills like any other profession. But the creation of a group of specialization is arbitrary. Some companies make a distinction between “developer” and “front-end developer”, others have a different set of specialized groups. Testers are no different, we exist because organizations feel that a separate specialized group is needed to best fills its needs.

Over time, testers developed a very strong pride in their own identity. It is a group that is constantly reasserting its existence, like most other groups really, “developers” being a possible exception. There is a lot invested in the continued presence of this group, for example: the livelihood of many persons, whole test organizations, test managers and architects, conferences specifically tailored for testers, products for testers, plenty of books and even some certifications.

All this yells at me: “we are testers and this is how we are different from the rest of the team”.

Now, I work as a tester. I have been a tester for 9 years, but I ask myself, what if instead of focusing on promoting a separate testing identity, we focus on transferring our unique skills to our cousins the programmers?

That is the basis of this series of blog posts. And I should remind you, dear reader, that these are just theories, all I have in terms of evidence is observation and experience having worked as a QA member in three teams at Microsoft and three teams at Atlassian. However, I do have a plan to experiment with this approach on my current team that I will share on this medium.

- Federico

Once upon a time…

2011-07-04T03:44:00.001-07:00

This link was shared on the internal QA chat today: ftp://dc1.dawsoncollege.qc.ca/VSHelp/SAMPLES/VC98/SDK/WINUI/COMCTL/REITP/REITP.C

I don’t know where the source code comes from, or if its even real. But there is a part that was funny, I copy it here:

static TCHAR szText[] = "Once upon a time there was an edit control. "
    "This was a happy edit control until some mean evil tester "
    "(who shall remain nameless) came along and started beating on "
    "the poor little helpless edit control. The edit control bravely "
    "faced it's unprovoked attacker, but an edit control can only take "
    "so much. One day when it was being ferociously tortured it thought "
    "to itself \"What have I ever done to this poor homely creature to "
    "deserve this type of punishment?\". It stirred up it's self esteem "
    "and stood up and said to the nasty creature \"I am an edit control, "
    "hear me roar! I deserve to be treated with dignity and respect!\". "
    "The evil little tester was taken aback by the edit control's sudden "
    "show of courage, but being the spawn of hell, he dismissed the edit "
    "control's declaration and continued his ghastly attacks. The edit "
    "control decided that nothing would ever reform this hideous beast "
    "imparting terror upon it so he decided to do something about it. "
    "The next day when the heathen was preparing to nuke the edit control "
    "the edit control jumped out at the beast and ripped it's heart out. "
    "The tester stood shocked for a millisecond and then dropped over dead. "
    "The end.\r\n\r\n";

Have a good testing week!

- Federico

Atlassian .NET SDK

2011-06-11T02:48:00.001-07:00

Shortly after I joined, I participated on my first Fedex Day at Atlassian. This is 36 hour coding competition, where everyone in the Engineering team puts together something cool (usually with a high density of hacks) related to our products and Atlassian. There are semi-finals, finals, a big trophy and bragging rights involved.

My project was to build a LINQ provider for JIRA. I got like 6 votes in the first round, but it wasn’t enough to go forward on the road to fame and glory. However, it gave me a fun project to work on my 20% time in .NET in an almost 100% Java shop.

WIth only 173 downloads so far, the library has a tiny mini niche of users, but I have enjoyed learning about query expression trees.

The source is available in bitbucket: https://bitbucket.org/farmas/atlassian.net-sdk

Example:

var issues = from i in jira.Issues
              where i.Assignee == "admin" && i.Priority == "Major"
              orderby i.Created
              select i;

- Federico

Speaking at Monash University in Victoria, Australia

2011-05-21T16:21:00.001-07:00

Next week I’ll be flying to Monash University to give a couple of talks about testing. This is the first time that the University offers a testing class and Dr. Yuang-Fang Li, who is in charge of the unit, invited me as a guest lecturer.

I am looking forward to visit the University and meet the students.

- Federico

Learning Javascript with Douglas Crockford

2011-05-18T01:56:00.001-07:00

This set of videos by Douglas Crockford was originally shared by Damian Edwards. I recently re-watched them on YUI Theater and had forgotten how packed with good info they are. Good resource to keep around:

JavaScript: The Good Parts (download)
The JavaScript Programming Language
1. Part 1 (download)
2. Part 2 (download)
3. Part 3 (download)
4. Part 4 (download)
Advanced JavaScript
1. Part 1 (download)
2. Part 2 (download)
3. Part 3 (download)
An Inconvenient API: The Theory of the DOM
1. Part 1 (download)
2. Part 2 (download)
3. Part 3 (download)
The State and Future of JavaScript (download)
Crockford on JavaScript

Good idea to get his book too.

- Federico

Parallel Universes

2011-01-08T19:37:00.001-08:00

Note: This is a refurbished old blog post I found on my laptop.

Many months back (while I was at Microsoft) I had a conversation with a Dev on my team that went like this:

Me: “All right, let me write the regression test and I can include it in our next test pass”.
Dev: “Perfect, so I’ll see results tomorrow?”
Me: “Well no, the next test pass is until next week”
Dev: “You mean you don’t run all tests every day?”

This simple conversation started a deluge of reality shocks between us. It turned out the majority of the Dev team had no clue how any of our processes worked. Among the mysteries where:

No understanding of what tests we ran.
No understanding of how we run tests.
No understanding of what tools we used to run tests.
No understanding of platforms or configurations.
The belief that we just “pushed a button” and all tests where done. (I love this one, because I hear it very often from higher up managers).

Here we were, on the same team working on the same product in close proximity with each other. Yet, most of what the test team did was a complete mystery to the Dev team. It was interesting to see this total disconnect between our activities, I called it “The Abyss”. There are many reasons why our team worked in this way. One could argue that this was a side effect of specialization, having different groups of persons focus on different things will create area expertise that is unreasonable to share with everyone. But in the case of software development, working like this made us loose the following strengths:

Cooperation: Having a basic understanding of the work of other team members (what’s hard, how they tackle problems) increases the chance members want to help each other.
Improve the process: Having an understanding of the test process is the first step to attempt to change and improve it.
Learn from each other: I like learning how a developer writes code, because it makes me a better software engineer, the more a developer learns how I test code would make him a better software engineer.
Quality assistance instead of quality assurance: A test team’s existence should be to support the group build great products, a smart engineer would aim to understand how to leverage the test assets to make a better product. This is in contrast with quality assurance which has the developer treat the test team as a mere checker or verifier of his/her work.

We were taking steps to improve this state, but culture is very hard to change and at the point I left the team the bridge over the Abyss was still under construction.

Dear reader, does this story sound familiar to you? What’s the distance of awarenes in terms of tools and processes between your QA and Dev team? A gopher hole, a ditch, a canyon… an abyss?

- Federico

New Horizons

2010-11-15T02:27:00.000-08:00

It’s been a while since my last post. After 8 years working on the ASP.NET team at Microsoft, the time came to move on to something else. I am excited to announce that I have recently relocated to Australia to work for Atlassian. This is a change that was long in the making, but I finally had my first day on my new job today.

I am really thankful for all my time at Microsoft and I will miss the team, but I look forward to the new challenges at Atlassian. The experience will be very different, here is an example:

From USA to Australia.
From .NET to Java.
From VisualStudio to Eclipse.
From Windows to whatever OS you want.
From Office to Google Docs.
From individual offices to open space.
From individual phones to no phones.
From thousands of employees to hundreds.
From mainly closed source to open source.
From internal bug database to public bug database.

I look forward to do my part to craft a QA team that is effective on an agile environment. :-)

- Federico

Testing Toasters

2010-08-15T20:07:00.001-07:00

Mark Berryman, a tester on my team that I have worked with for years, started his own blog recently! You can follow him here:

http://www.testingtoasters.com

Nice to read his perspective on the ASP.NET team. Good posts Mark.

- Federico

… or should I say, “I do testing”?

2010-06-20T18:00:00.001-07:00

I received a piece of Jedi wisdom from my friend Eduardo regarding my previous post: “It is incredible how we base the identity of a person on what we do.”

I love talking with people that can put my ideas into perspective, and Eduardo does it in a very gentle way. Lately I’ve been so focused on testing, that is easy to loose track that this is just what I do for work.

There was this exercise that Sallie was telling me about, where in some motivational conference they would ask people to identify themselves. I thought about it, and “I am a tester” won’t really appear on that description.

A small point, but I thought necessary to bring up. If this post was totally obvious for you reader, no worries, move along, these aren’t the droids you are looking for. :P

- Federico

“I am a tester”

2010-06-13T18:46:00.001-07:00

A couple of weeks ago I followed Sallie to her 15th college reunion at Brown University. It was a very different experience than my graduation in Mexico (I still think that those togas that they wear here are kind of silly), but what I am here to talk to you, dear reader, is a personal realization that I made on this trip.

I met a lot of people and inevitably, they would ask me what I do, and I would answer: "I am a tester". So what, you may ask, after introspection I noted that I know enough about my discipline as a highly skilled profession that I feel proud of telling it as it is.

I remember when I first started at Microsoft, 8 years ago, I was prejudiced and was interested in making it clear that I wasn't just a button clicker. When people asked I used to say "I am a Software Developer in Test". It was my actual title, but I still thought that a "developer" had a higher place in the totem pole.

After a year or two, I learned more and I decided that I actually love to test. I began telling people "I am a Quality Engineer". Yeah, sounds pompous, but I believed that the word tester had a social stigma for folks not involved in software development. Besides, I thought that what I was doing was so much more than "just testing".

Finally, two or three years ago it clicked to me. I am a service to my development team, a big part of my job is to support our programmers, I bring unique skills to the process and those skills take effort to master. I am still learning. Yes, I can also code, but that's just one of my skills (that also needs improvement, by the way). Others may not know this, and it is in my power to dispel the word.

Reader, if you are into testing, I encourage you to think how you present yourself to others, it can reveal things about our conceptions (or miss conceptions) about our craft.

What is your title in your business card?

- Federico

My first experience with MSpec

2010-06-09T20:04:00.001-07:00

In my spare time I like to implement small sample applications to keep in touch with web development. Lately, I have been working on a WishList application implemented in MVC to get better at using Moq.

If you are interested you can download the sample in progress:

Source available here: https://farmas@bitbucket.org/farmas/mvcwishlist (using mercurial)
See it in action: http://farmas.net/wishlist (user=fede, pass=123)

However, today I decided to give BDD a try. I chose MSpec (by Aaron Jensen) over SpecFlow because it looked more interesting. Right out of the bat, one of the disadvantages is the lack of documentation (T_T) but after searching the web and watching some TekPub with Rob Conery I finally got it working. I want to paste here the transformation of my unit tests from MSTest to MSpec for those of you that are looking at getting started with this tool and as a reference. You can also download the source from bitbucket to see both test styles.

MSTest

This test is for the login page of the site, and it verifies that if valid credentials are provided, the user will be signed in and redirected to the the lists page.
The CreateController method is separate since I use it for all the other tests for this controller.
Notice the use of Moq to setup membership and verify formsauth.

[TestMethod]
public void WhenLoginPost_IfUserAndPasswordAreValid_VerifyRedirectAndFormsAuth()
{
    //Arrange
    var controller = CreateAccountController();
    _mockMembership.Setup(m => m.ValidateUser("User1", "ValidPassword")).Returns(true);
    var loginData = new LoginViewModel() { UserName = "User1", Password = "ValidPassword" };

    //Act
    var result = controller.LogOn(loginData) as RedirectToRouteResult;

    //Assert
    _mockFormsAuth.Verify(m => m.SignIn("User1", false));
    Assert.AreEqual("Lists", result.RouteName);
    Assert.AreEqual("User1", result.RouteValues["username"]);
}
private AccountController CreateAccountController(MockBehavior behavior = MockBehavior.Default)
{
    _mockWishListRepo = new Mock<IWishListRepository>(behavior);
    _mockFormsAuth = new Mock<IFormsAuthenticationService>(behavior);
    _mockMembership = new Mock<IMembershipService>(behavior);

    return new AccountController()
    {
        FormsAuthenticationService = _mockFormsAuth.Object,
        MembershipService = _mockMembership.Object,
        WishListRepository = _mockWishListRepo.Object,
    };
}

MSpec (original specification)

Of course I cheated from BDD because I basically translated the UnitTest into MSpec :P, however, you get the idea of the readable form that specs take.

[Subject("Login Page")]
public class when_posting_with_valid_username_and_password
{
    It should_sign_in_the_user;
    It should_redirect_to_Lists_page;
    It should_place_username_in_the_url;
}

MSpec (the real deal)

I could have left the Establish call on the test method, but since I ended up retyping it over and over I decided to remove it to a common base class.

[Subject("Login Page")]
public class when_posting_with_valid_username_and_password: with_account_controller
{
    static RedirectToRouteResult result;

    Establish context = () => membership.Setup(m => m.ValidateUser("User1", "ValidPassword")).Returns(true);

    Because of = () =>
    {
        var loginData = new LoginViewModel() { UserName = "User1", Password = "ValidPassword" };
        result = controller.LogOn(loginData) as RedirectToRouteResult;
    };

    It should_sign_in_the_user = () => formsAuth.Verify(f => f.SignIn("User1", false));
    It should_redirect_to_Lists_page = () => result.RouteName.ShouldEqual("Lists");
    It should_place_username_in_the_url = () => result.RouteValues["username"].ShouldEqual("User1");
}

public abstract class with_account_controller
{
    protected static AccountController controller;
    protected static Moq.Mock<IFormsAuthenticationService> formsAuth;
    protected static Moq.Mock<IMembershipService> membership;
    protected static Moq.Mock<IWishListRepository> repo;

    Establish context = () =>
    {
        formsAuth = new Moq.Mock<IFormsAuthenticationService>();
        membership = new Moq.Mock<IMembershipService>();
        repo = new Moq.Mock<IWishListRepository>();

        controller = new AccountController()
        {                    
            FormsAuthenticationService = formsAuth.Object,
            MembershipService = membership.Object,
            WishListRepository = repo.Object
        };
    };
}

Execution

I use TestDriven.NET to execute the tests and the output looks really cool:

------ Test started: Assembly: MvcWishList.Tests.dll ------

Login Page, when posting with valid username and password

» should sign in the user

» should redirect to Lists page

» should place username in the url

3 passed, 0 failed, 0 skipped, took 4.27 seconds (MSpec).

Conclusion

Even with this small app I found myself doing time consuming refactoring of code as I “discovered” hidden facts that I had not taken into account when writting the first tests for each controller. I believe that if I had written the specs first in this way, I would have ironed out parts of the design even before implementation. I’ll continue to translate my unit tests into MSpec to see how more complex controller actions look and eventually write the parts that are missing using BDD. I’ll upload all source to bitbucket.

- Federico

LTAF Screen Cast #1: Getting Started

2010-03-14T12:23:00.001-07:00

I am starting to record screen casts for LTAF. This is the first in the series where I go over what is LTAF and how to write your first test. If there are any topics that you want me to cover, let me know!

Resources:

Download LTAF from CodePlex
Download LTAF Demos (from a talk I did at DevConnections)

- Federico

The Boat Anchor Odyssey (part 3)

2010-03-07T12:50:00.001-08:00

This is the third and last part in these series. Part 1 can be found here, and part 2 can be found here.

The Details

Ok, enough of planning, let’s get down to the details. The work to get 2,000 tests to a robust and trusted state was:

Nightly runs. The concept is to be able to run 2,000 tests every day in a some what predictable manner. This is important because we need a way to validate the changes that we were about to make.
Remove low value tests. This idea was to define what we thought our new test bed should cover, and then to go back and review the 2,000 tests and make a decision whether to keep it or remove them.
Increase robustness. Nothing kills automation faster than flaky tests. This idea was to track test failures very explicitly and to make it a scheduled work item to resolve them.
Port old tests. This is very specific to our team, but across the 10 years of the product, there are tests written in 3 different automation frameworks.
Manual tests. The idea was to create a formal manual test bed that can complement our automation.

So there you have it, these activities went on in parallel with a team of 4 persons. What follows are the details of each activity.

Nightly Runs

This is one of the first thing that we tackled. The system that we ended up with monitors every new build of asp.net, and kicks off an automated run every night so that we get results the next morning. This helped because we started to get a lot more history of each test and we could compare results with past results, it also allowed us to discover patterns across time frames.

The first challenge was the need to make runs under many different platforms. Clay, who was in charge of this project, came up with a clever rotation schedule. We have 4 “templates”:

Up level x64
Up level x86
Down level x64
Down level x86

Up level means operating systems that are Vista or greater (which come with IIS 7), down level means system previous to Vista. Each day, one template would be used in sequence, and for each template an OS would be used in sequence. This gave us coverage and some degree of predictability. For example, if we saw a test fail on XP x86, we could look at previous runs and see if this was something particular to this platform or generic. And after we made a fix we could either reset it for this week run, or wait until the next scheduled XP x86 to see what happened. (As I am writing this, we are investigating including Localized builds and framework only builds into our mix)

The second challenge was the reimaging of machines. The problem was that our tests need the machine to be set up in specific ways. For example, some may need some SDK tools, SSE to be installed, smtp server to be configured, an SSL certificate, etc. It was very important that the machines setup be predictable between runs, so Clay had to spend a fair amount of time making sure the install steps work correctly for each of our 4 templates. You would be surprised by the complexity of just being able set up test machines in a consistent and automated way.

Remove Low Value Tests

This idea came after sampling some tests in our proposed test bed. There was such a high difference in level of coverage per feature. For example, some controls would test setting each property to 30 different colors and verify rendering, while others assumed that the color was handled by the base class and didn’t verify it at all. Some features considered error reporting from edge cases to be the most important test, while others assumed that the most important test was a positive case. There was also a big disparity in the number of tests between features, but I assumed that the number of tests wasn’t that relevant (we would not try to level them).

Reviewing 2,000 tests was not trivial, but we thought it was worth it. So little by little we chipped at them. The challenge here was coming up with a new bar for what the tests should be covering, and then apply it across each feature. This was also a good opportunity to document our test bed. As we went along, we took note of the automated coverage that each feature in the product has.

Part of this review was to remove something that we call contexts, which is like a variation of a test. For example, a dynamic data test would be run on IE, FireFox and Chrome, producing 3 results in total. Even though there were 2,000 tests, when executed we actually got 9,100 results because of contexts.

The result of this review was a reduction from 2,000 to 1,800 tests, and from 9,100 results per nightly to 4,000 results, with out a big impact to coverage. As you can see, the biggest gain by far was due to contexts. This is because we found out that there was little value in running some of the tests in multiple contexts. We also came up with a product “heat map” that shows the distribution of tests and contexts across the product. We plan to use this later when deciding what tests are missing.

Increase Robustness

All right, we had a process in place to trim down our test bed to remove duplication and low value tests and a way to run tests every day. Now we needed to make our tests robust and maintainable. We identified 4 reasons why a test would fail (assuming there is no product bug):

The machine was set up incorrectly.
A defect in the automation framework.
A defect in the test code due wrong expectation check.
A fragile test code (race conditions, bad recovery, focus timing).

The process that we created was that we would spend one or two hours every morning to analyze the nightly results and open test defects for any test failure in these 4 buckets. We divided the tests between us so that we each started to get familiar with a segment of tests. Every week, we would triage all the bugs and assign the highest priority (usually the ones that are causing the most failures) to be fixed next week. Each of us is committed to get the bug fixed.

This method allowed us to keep a very concrete count of work left. If I see 50 bugs in the database, that tells me that there are 50 problems that are preventing us for having a clean passing run, and I can some what estimate effort.

Collectively, we have fixed a lot of bugs, more than 150. The result is we now have 98% pass rate in two of our nightly templates. This is 80 failures each morning, which are easily handled. We still have 20 active bugs, which we continue to push for every week.

Port Old Tests

I won’t spend too much time here. At the beginning of this project we still had 250 tests written in the old frameworks that none of our team has expertise in analyzing. We found out that they constituted at least 1/3 of each night’s failures. The decision was that maintaining those tests was more expensive than to port them to the new framework. At this point, we have gotten rid of those, which reduced our test bed to ~1,500 tests.

Manual Tests

This is a touchy subject in DevDiv, where there is a strong culture of automation. But look, there are features in ASP.NET that are no longer being developed that are costly to maintain automation for. Is it good enough to run a check for these features once per milestone? This is roughly equivalent to run them once every 3 months. We would still catch the regression in time for the iteration, but we would not be checking it every single night. If frequency can be this low, the value of automation is greatly diminished.

We decided to embrace manual tests where it made sense and create a set of them that are cheaper and easy to run by a person. We have clear guidelines of what should be part of our manual test bed, as well as how to run and track results for them. Some examples of good candidates for this:

WebParts drag and drop. This is a feature that is fairly atomic that we haven’t changed since .NET 2.0. However, it is difficult to automate for all supported browsers, but very simple for a person to run.
Web Admin Tool. Our team is no longer investing in this tool, and was released since .NET 2.0 as well. Many tests were written for it, but for several reasons they are very flaky. This is another area where running by hand could makes sense.

Conclusion

This project is scheduled to end at the end of March 2010. We are in the home stretch. The real trial for the new test bed is yet to come, as we use it for servicing after Dev10 RTM, as well as regression coverage during the next product releases. Hopefully I will post a post mortem after a couple of months to recap how things went.

What I am most excited about is the opportunity that our team has to impact the product going forward. If a regression check takes us only a couple of hours each week. Imagine all the new tests that we could be designing with the rest of the time. The plans for this next stage will have to wait for another post.

I want to thank Anton, Alex, Clay and Drew for their hard work on making this happen. As I mentioned before, this was not our only deliverable during this time frame but they kept in it for all this time.

- Federico

The Boat Anchor Odyssey (part 2)

2010-02-10T21:18:00.001-08:00

Part 1 can be found here.

Keeping it within constraints

I experimented with a ton of approaches to reach my objective. I knew that going and fix 20,000 tests would not scale, so there had to be some form of evaluation. Here are some ideas I tried:

Fix the failed tests.
Fix the ones that have found defects in the past.
Remove tests that have not found defect in the past.
Remove tests for features that are not high priority.
Replace tests with a new designed tests.

I reviewed my findings with some team members to get feedback. The problem with these techniques is the effort of evaluating this many tests. It could work, but they depended on my resources. This was tricky, the more resources I would need, the more I ran the risk of getting into time consuming negotiations with management. I decided to create my own constraints that would seem “reasonable” to my product unit. After some not-entirely-scientific calculations I came up with:

Due date: End of March 2010. This coincides with the release of Dev10, which made it easy to justify.
Resources: 3 persons. This decision fit within a bigger initiative that we were driving in the team. The basic idea was that this group would provide key services that will allow the rest of the team to really focus on product development.

Around this time, I decided to become a manager lead. My group, named Engineering Services, would consist of 3 team members and one of our charters was the Test Bed goals that I previously described. This is not our only responsibility, but it is up to us to manage our time properly so that this project is a success.

It may sound backwards for some, but by arbitrarily choosing constraints that were easily “agreeable”, I got no push back and now had clear constraints that I could work with to come up with an actual strategy. Of course, it helps that I have a manager that believes in me.

Deciding on a Strategy

It was November 2009, which meant I had 5 months to turn things around and I had to juggle this deliverable with at least 3 others (won’t go into them): hand off our test bed to the servicing team, driving test passes, take ownership of tests.

I concluded that my first priority was to create a “base line”. The idea was to create a small test bed that was well understood and robust that we could use to make intelligent decisions about our coverage going forward. I entertained the idea of creating it from scratch, a compact super integrated test bed, but I abandoned it in December after a costing exercise. Instead, we would use some of our current assets to create the base line.

To understand this next part I need to talk about numbers. Our test bed is divided into 3 priorities, what each priority meant was never clearly defined, which means that it is totally subjective to what the tester thought what Pri 0, 1 or 2 meant. We knew that a human at some point made the decision of how to prioritize their tests, but we didn’t really understand how that decision was made. The break out was:

2,000 Pri 0
10,000 Pri 1
8,000 Pri 2

2,000 looked manageable, so the strategy became: the Pri 0’s will become our base line, once we get these tests to be stable and robust, we can analyze whether more tests are needed. This was a touchy subject, it read like we were going to start fresh by only keeping Pri 0 tests and discarding 18,000 tests. That made people nervous. However, this is only step #1, once we got to a stable state we would evaluate if more tests are needed in any area and surgically move tests from the 18,000 to our new test bed. The point was to protect our test bed, from now on, the only thing that goes in there are things that product development absolutely needs and that we approve.

Thanks to Leslie for trusting in my methods and to Drew who helped me bounce ideas with him.

Next: Details and Making it Happen.

The Boat Anchor Odyssey

2010-02-06T20:46:00.001-08:00

Disclaimer: These posts will go into a lot of specifics of my work, the problems that I faced along the way and how I chose to solve them. I will try to give as much context as possible, but I warn you reader that to give a complete picture of the situation would take more time than I have. Any feedback is appreciated.

The ASP.NET test bed is a franken-monster that was built over the course of 10 years, with no clear direction by over 60 different testers that came and went in short periods. We called it the “boat anchor” and you can read more about it on this post.

Last year, me and others in my team began to seriously tackle the problem, and today I am here to report our progress and what I learned during this process.

Clear Leadership

We “officially” started in June 2009, we divided the test bed into 3 parts: design time (the experience of ASP.NET controls in VS); runtime controls (the page framework and controls); runtime core (the http pipeline and core app services). All in all 20,000 tests. Each of the test leads was given one of these pieces (I got design time).

The goal was loosely defined as “reduce the test bed”. Each lead went out to tackle their own section, and we each came up with a different approach:

Design time: I decided that it was more cost effective to toss the current test bed and start anew (none of the tests even ran, and in my opinion where low value).
Run time core: The lead decided to jump in and start porting the tests that were written with an old automation framework (a major cause of failures, and he evaluated the worth of each test along the way).
Run time controls: The lead decided to create a tool so that he could query tests for each control and update its priority.

After 3 months of work, our status was:

Design time: I had devised a methodology for evaluating the product to identify what tests needed to be written and how to measure the effectiveness of such tests. I applied it to a sample area which I thought validated the approach. Tangible results: nada.
Run time core: The lead was 15% complete with porting the tests (there were 5,000 to begin with). Tangible results: ~300 tests were reviewed, and I believe we decreased the number of tests by ~200.
Run time controls: He delegated the task to another team member, the tool was not finished yet. Tangible results: zilch.

3 months of work, and we were almost where we started. What the hell happened? In my opinion it was the lack of strong leadership. For each of us, the priority of the work was different, the goal of the work was different and the approach was different.

I strongly believe that the cost of our test bed was a major problem of my team, so I decided to grab the whole problem and invest myself in it. Take away: Good intentions are just that. You need a person that will truly own the effort and be held accountable for the result (which needs to be crystal clear). I decided to be that person, my review this year will be partly based on the success of this project.

Clear Expectations

By October 2009, I was now in charge of the whole test bed, and I was committed to fix it. First, I worked on creating the goals that will drive my effort. Here they are:

Detect regressions in the product as close to the moment of injection.
Increase the capacity of the test team by treating regression testing as a service.

That’s what I started with. Not perfect, but workable. Notice that test bed is nowhere in my goals. Also, the way I phrased them made it very easy to “sell it” to our managers. It is very simple to imagine how these goals will actually help our business. The sooner we catch regressions and the more time testers have to perform quality exploratory testing the better for our product.

Ok, next I created some tactical objectives that would help me achieve my goals. Here they are:

Have a clear understanding of what our test bed is covering.
Capable of running and reporting status of builds every day.
Have a team that owns the test bed.

In my mind we needed to be able to run our automated checks more often and we needed to understand what the runs where really telling us about the product. Additionally, a team of specialists would own the test bed and be responsible for it. This change has a broader cultural goal that I will not go into in this post, we will focus on the test bed aspect of the work.

Two things that I would like to point out. First, it is important to take a step back and put some thinking to the bigger picture. Your efforts have to be easily mapped to your business. It helps to write it down.

Second, I have seen a lot of initiatives in Microsoft die due to “paralysis analysis”. It is good to plan and take time to consider options with pros and cons, but at some point you have to bite the bullet and take action. If you find yourself going to meetings and meetings to review the vision/plan with a bunch of persons that are not directly accountable to the project, you may find that politics and processes will grind you to a halt.

Next: strategy, resources and planning.

- Federico

Lost in Metrics. A tale of process overhead.

2010-01-31T10:22:00.001-08:00

How do you report status in your work? Do you have to give metrics or some form of cost/value analysis? Maybe some conjured up risk analysis? I am assuming most of us have to report status some way or another. In this post I will narrate a true story that shows status reporting process overhead in action.

This occurred last year, when in some random conversation with a tester in my team I asked what he was working on. He told me he was working on the “test pass status report”. Apparently he had been working on this on or off for two days. I was intrigued, I was hearing that a tester that could be finding defects was spending a lot of time doing some form of status reporting. After probing some more, I learned that there is this mandate from the test leaders in DevDiv that test pass status has to be reported in a certain cumbersome way, that ends up consuming a person 3 to 4 days to gather the data, organize it and input it in a special tool so that management can see the status of our team.

This seemed excessive to me until I actually tried it myself. There is this tool created for the sole purpose of reporting on test pass status (I moan internally when I imagine the resources of other testers in the company spent creating this thing). This tool is extremely confusing. That’s the first time waste: the time to understand it. Each team in the DevDiv are tracked in there, ASP.NET is one of them, and each team has to fill metric for about 40 items. An example of one item is “Full Redist on Win2k3 x64”, which means have you run your tests against this platform install. And the metrics are things like how many tests did you run and how many failed. Just numbers.

You may think that this doesn’t sound too bad and that maybe once we get accustomed it will go faster (that’s what the leaders at DevDiv told me anyway). However, whatever the cost, this activity was providing absolutely no value for my team, and I wanted to know what was the point of reporting this metrics. So I went and asked the person in charge of driving the test pass. I told her of the pain and waste that was reporting this metrics, and wondered if there was any other way that my team could get the information that management was after. The response was: “Look, I don’t care how you get the data, just get me the data in the format that I need it so that I can report it up the chain”. Great.

I obviously asked who the report went to, because I wanted to learn what it was used for and how. After a couple of steps up the test leaders I arrived at the person that actually consumed the report. “I look at trends”, that’s the answer I got when I asked what was the point of this report. I didn’t totally buy it, but I could see that by seeing some red indicators in this report leaders could go and ask questions about why a team was doing bad here or there. Possibly preventing bigger issues from happening later by addressing them early. I thought it was quite sensible to ask if my team could report our areas of risk ourselves instead of this metrics based reporting. “Sorry, it has to be this way”. You see, having all teams report the same way in this repository makes it easier for our manager’s to read the report. In a way, all teams are paying a price so that managers can get a nice report, to spot what teams might be in problems.

My team knows what are our problems with our product, where our risks are, and whether we are on track or not. Unfortunately we can’t simply communicate that, we have to spend our tester’s time to fill out this report.

So, what did I learn from this?

Direct challenge of management does not work. I feel kind of foolish now for even trying, indirect change through influence works better.
Even though we keep reporting how we are told to, having understood the purpose of the report, by following the “spirit” of the report I could create a way to get management what they want without too much cost. (I won’t tell how :P )

- Federico

The vicious cycle of QA

2010-01-02T08:40:00.000-08:00

One of the challenges that a tester must overcome is to create and sustain a reputation. This is particularly hard for testers not only because our problem space is vast and ambiguous, but because everybody expects something different from us. In this post I will narrate how having a loosely defined role for QA lead our team down a path that eventually created a culture of QA as a burden.

Step 1. Unclear expectations.

It probably started way before I joined 8 years ago. Nobody in the team clearly understood what was expected of the QA team. The general idea existed that testers where meant to “find bugs”. But additionally each person had preconceptions of the role for QA, that included:

· QA signs off on the product.

· QA writes automation so that testing can be performed by just clicking a button.

· QA should say exactly the time it will take to test the product at the beginning.

· QA will test everything and whatever they are asked.

Furthermore, nobody could explain any of the specifics. Like, what it means for a product to be “ready” to ship? What does “sign off” mean? What does each “priority” of defects mean? What does “automation” mean? What is “test complete”?

It was an error from our part to not seek to fully define and declare what our role was. Hell, it would have been better if someone had come and defined what our role was for us. Instead, each person had its own story and expectation of what testers were supposed to do and deliver to the team.

You can guess what happened, we failed all expectations!

Step 2. Fail and get blamed.

I mean, we did our best, but it was impossible to meet all the expectations from each Dev and PM, especially when we didn’t even know what they were. And we got the blame. It was not uncommon to hear things like

“why did you miss this bug?”
“what are you going to do to not let it happen again?”
“why didn’t you run tests on this or that weird combination?”
“create more automated tests to patch the holes”
“automation is cheap, run more tests”.

If any bug was missed it was because QA didn’t do its job. The message, as our leaders at the time heard it was, “you must catch all bugs pronto, and if you don’t you failed and it’s your fault”.

What did we do when faced with this? We created more processes!

Step 3. Cover your back with processes

I am sure it was not the stated goal, but I have a feeling that the idea was that as long as we showed all the processes and we followed them, it was not our fault if a bug was missed. The process was not designed to do better testing, but to distribute responsibility in case of problems.

For example: at the very beginning of a feature, and I am talking about when the spec was not even complete, the tester would create a “test plan” which tried to capture beforehand ALL of the tests that we would execute. Never mind that we didn’t even had a product yet. Then we would go into a review with Dev and PM and the plan was treated as a contract. The unsaid conversion would be: “this is everything that we are going to test, if it is not here it won’t be tested. This is your last chance to tell us if we need to test anything else, after this you will be partly responsible for any holes”. You can imagine that the result was a ridiculous amount of tests. There was little thought of value. If constraints are ignored and another person is doing the work, what comes out is a brute force brain dump. Furthermore, testing meant automation, so we ended up automation a ton of tests L

We had similar processes for practically every test activity, although not many involved Dev and PM like the example above. Most were checklists and checklists of things to do. At some point, most testers didn’t understand why we had to do all of the activities but they were required to “sign off”.

What did we gain by so much process overhead? We became the bottle neck!

Step 4. Push back when process doesn’t fit the plan.

Now enter the turf wars. It went like this: Dev and PM want to create a feature. They ask QA to cost and we came back with a huge cost, I am talking months even for a small feature. We had become victims of our own brute force process and now it was impacting the team because we would not sign off on anything unless it passed through our test method juggernaut.

Typical questions would go:

“why does it take 2 months to test this small control?”
“everybody else is done, why are you so inefficient?”
“can’t you get more people to test? Like in China or something?”
“this is unacceptable, can you finish in half the time?”

Interestingly, I can’t recall anyone that wanted to know just how are we spending our time. We were just “doing our thing” to sign off, that just how long it takes.

It might have been bearable, except for one fact: our testing was low quality!

Step 5. QA as a burden, high cost for low value.

Put simply, we were taking more time and still missing a lot of bugs. Our shield was that it “had to be done this way”.

A person external to QA would see a group of persons that are constantly pushing back against new features because of test cost, but that give little observable value to the team. We would be busy filling status and reports, automating like crazy, debugging and fixing crummy old tests, going through our checklists with little questioning. And then, a manager would grab the latest build, find an obvious bug, and wonder why the test team didn’t find it. Just what are those guys doing with all that time?

And so we go back to step #1, rinse and repeat, and you settle at a culture that views QA as a burden.

Breaking the habit

This is what I named the vicious cycle of QA. Unclear expectations lead to blame which leads to more process which leads to bottle necks which leads to more un met expectations. This is what I think happened in my team and probably others in DevDiv, if you identify with this story or a piece of it, then I should say that we managed to break out of it. The road was long and the work is certainly not over, and the story of rehabilitation will have to wait for another day.

Federico

An LTAF post in Russian… translated.

2009-12-23T09:37:00.001-08:00

Matthew pointed me to a Russian blog that talks about LTAF here. And then tried the google translate service which I hadn’t used before. Check out the translation here.

- Federico

Job opportunity in the ASP.NET QA Team

2009-12-13T18:15:00.001-08:00

There is an opening for a tester in my team. If you are interested in help building a test team that is strongly based on the context driven testing principles, then shoot your resume to Mark Berryman, we would love to talk to you.

You can see the external job post here.

- Federico

DevConnections slides and demos

2009-11-16T19:40:00.001-08:00

Last week I presented at the DevConnections conference in Las Vegas. The topic was the Lightweight Test Automation Framework (LTAF) and the links to the powerpoint presentation and the demos are at the bottom of this post.

The conference is mostly for developers, so I tried to mold the talk towards practical applications of a test framework to enable product teams to catch regressions sooner. I asked at the beginning how many persons in the audience considered themselves as “testers” and only 4 persons raised their hands. Based on the comments that I got after the talk, there is still the stigma that testers are second class citizens below developers, which was not too surprising. However, I did find high interest from attendees to have more test oriented talks in future conferences which was a surprise and something to look forward to.

I am trying to improve my skills as a speaker, and this opportunity definitely helped (translating spanish to english on stage is harder than I thought). To my knowledge, this was the first time that a Microsoft technical conference had a session dedicated to QA, and seems like people want more. I’ll start preparing for next time. ^_^

- Federico

What I learned from James Bach?

2009-11-15T18:07:00.001-08:00

This Friday I got to meet and interview James Bach (episode should be available soon-ish at codingqa). I really want to thank James for meeting with Matthew and I on a Friday afternoon, and taken the time to teach us some ropes of the trade. Besides learning some very useful testing exercises, there were some points that I got while talking with him about being a professional tester.

Create your own definitions. I thought I had taken some time to internally define what “testing” and “quality” mean, but James took it to another level. He challenged me to define and describe all my behavior so that I could explain with determinism what I was doing. I find this extremely important: to develop the capacity to synthesize behavior into its essential nature and to be able to classify it against other behaviors. In other words, to develop my own vocabulary. Why? Because once I have this framework I will be able to evaluate and standardize test work as a profession.
Train your intellect. Testing is a highly intellectual activity that needs to be developed. I already knew this, or more accurately, I knew it by intuition. But to be more serious and even systematic about training my mind to be a test ninja is work that I need to start doing.
Learn by experiment. The reason why I respect the people that I respect is because they lead by example. This is the type of the lead that I want to become in my team. One cannot preach without having tried it over and over, and of the best ways to do that is to experiment a lot, observe, learn and try it again.
Build your reputation. A professional works hard to build and guard his/her reputation, it is this standing among peers and others that grants you executing power and it translates to money at the end of the day. My first step is to work on point #1 and #2 to increase my confidence, that is the key ingredient.
Leverage the community. I care deeply about testing and the product that we are building (ASP.NET), I go to great lengths to envision new ways to test, try them out and roll them out to the team. However one thing that I haven’t done so much is reach to the community of other testers for more ideas and to help me learn. There are a lot of thought leaders out there, and I need to put myself with them to share what I got and take in what they got.

Lastly, something that was more of an experience rather than a learning, was the motivation that I got by talking with James. There is so much more out there that is exciting, just thinking of how me and my team could be better testers makes me want to get in the car and floor the gas and just go. Recklessly.

- Federico

Working with numbers and common sense

2009-10-19T19:34:00.001-07:00

The Asp.Net QA team is going through some changes to better focus our activities and avoid randomization, I’ll talk about it in some later post. For now, one of the roles that I am assuming is to oversee the “test pass”. This is basically a set of product wide runs that all QA teams in the division must do to ascertain the state of the product against team expectations.

The moral of the story comes as we were ready to start a test run and I asked when it would be ready for testers to analyze it: “two days”. Wait, what? Two days? It takes 48 HOURS to execute our tests? I was told that’s what it always takes. I took a calculator and went to my office.

Turns out the type of run that we wanted to run generated 9,000 results. Each test takes an average of 3 minutes to run, don’t ask me why right now, just work with me. That is 27,000 minutes. That’s 450 hours, or 18 days. So, if I wanted my results to be done in one night, or 12 hours of execution, I need roughly 35 machines. And how many machines do we use in our runs? Eight. I asked why? It’s not as if we are in a super shortage of machines in our lab. There didn’t seem to be a good reason, eight just seemed to be a “good” number and we had “learned” to live with it.

We are now running with ~30 machines and I expect to see the results tomorrow.

- Federico

Has anybody used Concordion?

2009-10-17T19:31:00.001-07:00

Leonid pointed me to this framework/tool back in August but I hadn’t had time to check it out. It is called Concordion, and it looks like a very interesting way of writing acceptance test that read pretty much like a spec. You basically generate a set of linked HTML pages that read as a functional specification, but by putting special syntax in them you can call into your test fixture code that can interact with the system under test.

At first glance it sounds like a ton of work, but then again I don’t have a lot of experience writing end user applications in a team with agile methods (writing and testing a framework like ASP.NET is not quite the same as writing a testing an Amazon).

If you have experience using this kind of spec-to-code approach I would be very interested in hearing your experience. Drop me a line!

- Federico

Presenting at DevConnections

2009-10-17T19:18:00.001-07:00

Hey everyone, I got a talk lined up for November in DevConnections about Testing ASP.NET. Here is the abstract:

AMS09: Testing in ASP.NET
Federico Silva Armas
As more people in the community and organizations recognize the value of testing, developers are sometimes left with the challenge of testing ASP.NET. With so many tools and approaches, it is difficult to decide what to invest in. In this introductory sessions for developers and testers alike, we'll explore different tools and strategies for testing an ASP.NET Web application. We'll focus on how to write functional tests using a variety of tools and how you can leverage them in your next project. We will also discuss some of the lessons that the ASP.NET QA team has learned while testing ASP.NET.

I am still unsure about the audience (I have never been to DevConnections before), so I hope I don’t shoot wildly off the mark. I will be talking mostly about tools and strategies to write functional automated tests. I plan to show off LTAF, Selenium and WatiN. Hopefully get the audience excited and interested in test automation.

- Federico

Using a unit testing harness to execute functional tests

2009-10-14T20:08:00.001-07:00

As I am preparing for my talk at DevConnections, one of the things that I was advised to demonstrate is how to incorporate LTAF tests into MSTest or NUnit. The idea is not new as certainly that’s one of the preferred ways to author Selenium or WatiN tests (if I am totally wrong, please correct me).

Personally, the idea of using a tool that was designed for unit testing to write and run functional tests is a little awkward. My reasons for this are:

Functional tests need a better logging mechanism than simple asserts and exceptions. Our harness produces super cool logs that contains the HTML at each step of a test, with visualizers to see what rules failed, debug them and fix them. It is definitely leaps forward than the exception reporting of a unit test harness.
Functional tests need a better test manager. What I have seen of unit test tools is some support for creating list of tests, but a formal test manager system will include additional test metadata like requirements, description, abstract, list of scenarios, contexts, several ways to organize them and dependencies.
Functional tests need more advanced analysis support. A unit test tool basically "runs and forgets” tests, it shows you the results of the current run and some support storing data of past runs. But a good functional testing harness will provide tools like comparing results of many runs, can track what was the reason for failure of each result, it can group failures that seem to be related to make it easier to analyze, it can check the bug database to see if there are similar problems to the one you are analyzing, it can correlate how much of the product code each test touches, it can organize and display the results when there are thousands of tests into views that are useful for a test team.

These are all things that are possible with the tools that we use internally in DevDiv. Now, you can argue that you can accomplish all of these with MSTest or NUnit or whatever, and that may be true. What I am saying is that functional tests have a specific set of requirements that having a specialized harness for functional tests makes more sense to me than trying to coerce a set of tools that were designed for dev unit tests.

What do you guys think? You like using a tool like MSTest or NUnit to write and execute functional tests?

- Federico

The role of QA

2009-10-13T23:25:00.001-07:00

I was in a meeting recently with some higher ups in our team, we were reviewing some bugs to make a decision on whether to fix them in this milestone, and there was a comment from one of the managers that went like: “QA’s job is to measure quality, not to improve it”.

That was interesting for me and gave a little insight on how our discipline is sometimes perceived by others. The tester that had opened the bug had written comments in there that specified what he thought the correct behavior of the product should be, giving a very specific way that the product should be changed. Regardless whether his recommendation was workable or totally unrealistic, I thought it was great that our testers were beginning to contribute with design suggestions. However, the rest of the persons in this meeting were very quick to dismiss it because “he didn’t know what he was talking about”.

I don’t doubt that there was some reason why the recommendation was not feasible, but to discourage a tester from opening suggestions and just stick to reporting what doesn’t work according to design felt contradictory to what I am trying to achieve in our team. I felt bad for the tester that spent time thinking of what a better experience should be, describing it in full with repro steps and scenarios, only to be shot down with an implied message of “don’t waste your time with these”.

We have a long ways to go as a discipline, and I hope that as we demonstrate that QA can bring more value than just measuring quality we can start turning the corner on perception.

- Federico

Selenium testers, help me!

2009-09-13T20:46:00.001-07:00

As I was preparing for my talk, I ran into the problem described here. I’ll be the first to admit that I have not used Selenium in over 2 years (yikes!), so hopefully what I am running into is known and has an easy work around:

“I have a GridView control with sorting enabled inside an UpdatePanel. I used Selenium IDE to record a test that clicks on the sort link of the table, but when I try to execute the test it get's stuck on the click command. Looking at the log I see:

[info] Executing: |click | link=Name | | 
[error] Timed out after 30000ms

I haven't tried it with Selenium-RC yet, I don't know if it will be any different. I don't want Selenium to wait for anything. Any ideas of how to work around it?”

Thanks !

Presenting at DevConnections

2009-09-13T20:27:00.001-07:00

Whoa, it’s been a long time since my last post. I have to pick up the slack.

First I want to announce that I will be presenting at DevConnections in Las Vegas this year. Even though it is not the first time that I do a talk at a conference, it is the first time that is a regular break out session.

The topic of my presentation is testing in ASP.NET. The description is listed here, but I copy it below:

AMS09: Testing in ASP.NET
Federico Silva Armas
As more people in the community and organizations recognize the value of testing, developers are sometimes left with the challenge of testing ASP.NET. With so many tools and approaches, it is difficult to decide what to invest in. In this introductory sessions for developers and testers alike, we'll explore different tools and strategies for testing an ASP.NET Web application. We'll focus on how to write functional tests using a variety of tools and how you can leverage them in your next project. We will also discuss some of the lessons that the ASP.NET QA team has learned while testing ASP.NET.

Mmh, I need to figure out how to put my Bio in the DevConnections site…

Nice post from Telerik's QA dept.

2009-08-23T10:39:00.001-07:00

Today I found myself quoted in this nice post by Konstantine Petkov, I can't remember if this was always the case, but the post is now part of the Telerik's QA blog. I am now following it since I am very interested in their new automation tool which looks awesome.

- Federico

Another LTAF blog post

2009-08-22T14:20:00.001-07:00

Matthew recently pointed me to this blog post by John Anderson that talks about Lightweight Test Automation Framework.

I understand that there is some confusion around how LTAF positions itself against Selenium and other automation tools. I am working on a blog post to try to clarify this point. For now you can learn some more about it in CodingQA Podcast Episode #1.

- Federico

Change the process or change the team?

2009-08-21T19:15:00.000-07:00

Lately I have been reading about Scrum (book called “Agile Project Management with Scrum”). I figured that I had never actually read a formal book about this methodology, and it was a book that we got for free in our team (although to my knowledge we have never done Scrum). One of the things that I found interesting is the way that it talks about change.

According to the majority of this book, Scrum is a set of practices and rules that an organization can follow to achieve great results, and it goes to lengthy explanations of how the organization had to be changed to adopt Scrum. There is very little said about how to change Scrum to fit an organization. There are even certification seminars that one can attend to be a ScrumMaster.

The reason why this is interesting to me is because the approach that I used to transmit test methodology within my team is so different. Scrum presents itself as a process that is already a proven best practice, and as long as you can change the team to follow it you’ll see the goodness. I operate under the assumption that the process that I come up with hasn’t been validated until it can be successfully used by my team. In other words, I fully expect to change the process so that it adapts to the circumstances and particularities of my team.

Am I miss reading this? Maybe the book that I got isn’t very good, and I’ll just assume that successful implementations of Scrum actually change the process to adapt it to the circumstance. Anybody has experience with it that they want to share?

- Federico

QA Myth #1: “I can’t work without a spec!”

2009-08-03T20:45:00.000-07:00

One of the complaints that some times I hear in my team is the bad state of the design specs, or how late we get them. This is rooted in the original way of doing development in MS, where the PM is responsible for writing a very detailed specification that the Dev then uses to come up with an implementation that matches its requirements, and that QA uses to create test cases. You can image how under this model, if a spec is not produced it is cause for much grief on the tester that otherwise has no way of knowing what the feature is going to do.

Don’t get me wrong, an agile team needs a why to document design, be it email, notes on a board, or hallway conversations (whatever works), but when a tester is “blocked” because the lack of a spec is sign of bad QA culture. I have two problems with this:

The belief that a full blown specification is required to successfully test a product.
The belief that a specification is something that happens externally from QA.

The first one I consider a skill of a good tester. So what if you don’t have a written specification? You have the code and you know the customer’s problems that the feature is meant to solve. And if you don’t, you have the rest of the team (Dev, QA, PM, UE) to go and figure it out. All you need is to figure out one way to call into the feature, and from there put yourself in the shoes of your customer and use the feature in all ways that you can come up with. Is it doing what you expect? No? Raise an issue. Are there problems that it doesn’t address? Yes? Raise an issue. Is it too hard to figure out how to use it? Yes? Raise an issue.

The key is to constantly talk to your team. Is there anything else that the feature can do that you haven’t discovered? Are there any known cases where it won’t work? Are you ok with that? This is another reason why I love exploratory testing, it can be adapted very well to discover functionality.

The second problem I believe is cultural. When a QA person starts waiting and demanding that a spec be written so that he/she can do their job, they are implicitly promoting a return to a waterfall model. It can be very subtle, but what they are saying is “look, coming up with the design is not my job, you just tell me what the thing is supposed to do, and I’ll figure out how to test it. And don’t leave any details out, otherwise I am going to miss scenarios”. In contrast, a good QA person wants to be involved in the design, and puts him/herself where the design happens and gives input. He knows exactly what is being implemented because he/she was part of designing it.

Whenever I start hearing that a QA team got in problems or is blocked because of lack of good specs, its a sign that some change in the way they develop software and they think of themselves might be needed.

- Federico

Evolution of the ASP.NET Test Process (Part 2)

2009-07-31T20:50:00.000-07:00

This series was originally seen on the ASP.NET QA team blog and never finished. I decided to continue it on my personal blog as its my own opinions of our processes.

This is the second part of a short narration that describes how the testing process has evolved in the ASP.NET QA Team. First part can be found here.

The Feature Crew Model

After the release of .NET 2.0, all of Developer Division underwent a general change in the way we built software. This change came from top-down, and it was called the "Feature Crew Model'. I remember that all DevDiv made a lot of preparations for this new methodology, taking months out of development time to build a lot of infrastructure and tools to make it happen. Soma must be recognized for taking this bold step across the division. The basics are:

Each Product Unit (PU) inside DevDiv would divide their release into features. A "feature" meant a logical sub component of the product, that in theory could be developed independently.
A feature crew would be created from resources from the PM, Dev and QA disciplines and they would be given some degree of autonomy to shape the feature and manage their milestones.
A feature crew would develop everything on a separate independent branch.
A series of Quality Gates were put in place and only after a feature passed all of them, would the code be permitted to integrate into the main product branch.

The key part of this process was that it enforced a high level of quality before each individual piece made it into the build, and that each piece was finished. This was a big improvement because in the past there was a lot code churn happening on the main branch from all over the division that generated a lot of regressions. The second benefit was that this way of working encouraged better communication between Dev and Test, and allowed for teams that wanted to experiment with SCRUM or other agile methodologies.

Our First Feature Crews (ASP.NET Ajax 1.0 Release)

Our team's first encounter with this process was during a project code named Atlas (later to be known as ASP.NET AJAX). Back then, nobody was really sure how this would work out (and there wasn't a lot of guidelines coming from the division either), so it was understood that we would experiment and modify as we went.

Atlas was divided into the following feature crews (they were kicked off in this order, since subsequent crews relied on the previous):

Core: which included the type system and BCL.
Networking: everything under Sys.Net
Partial Rendering: UpdatePanel (this is the feature crew that I worked on, along side Eilon Lipton)
UpdateProgress/Timer: targeted crews for only these 2 controls.
Services: script callable Membership, Profile and Roles services.

ASP.NET QA Process in a Feature Crew

Looking back, the QA team didn't made any substantial changes from our old process, we were too busy preparing for having our test sources in different branches. So the QA process ended up being:

QA resources would be assigned to a feature crew.
Dev/PM/Test would start working on the feature at the same time, on an independent branch.
Dev was now responsible for writing UnitTests
Each feature crew would meet at least 3 times a week to design the feature and provide status updates. (The crew was in charge of costing all work, and arranging it in milestones. At the end of each milestone, the crew would integrate the progress into main).
As the design was being developed, the PM worked on the spec, Dev on prototypes and QA on the test plan.
QA would have a test plan review as before.
QA would start testing the feature as soon as code was checked in by Dev, and would start writing automation.
Progress was still tracked by automation complete numbers.
QA was responsible of signing off a number of items on the Quality Gates check list. Of importance, is the fact a quality gate was to be automation complete, and to have run all test in a number of configuration runs.
After all quality gates were met, feature was integrated to Main.

How did it Work?

Benefits
- Better collaboration between disciplines (we were all on the same boat to get our stuff integrated).
- QA was involved in the design.
- Code that entered the build was of high quality already.
- Dev's wrote unit tests which caught bugs earlier.
Disadvantages
- Probably the biggest problem was that QA still tried to do everything that we used to do on a 2-3 year release, but now on a compressed 3-4 months iterations. Because we didn't change anything in our process, we now really became the bottle neck.
- QA was still creating full featured test plans and focused heavily on automation, for features that now changed even more often.
- Overlap between dev unit tests and QA functional tests, the old mentality of "test everything" did not go away.
- Dependencies between crews created version problems that sometimes got crews blocked.

In the end, the team pushed quality up stream, but the QA team bled and struggled a lot. And some of the problems that we had identified before began happening again, in particular, finding bugs to late. My opinion is that after Atlas was shipped, the team had a better development process that put quality first and helped the QA team, but our own QA processes still needed to adapt to this new way of doing things.

In the next part I'll narrate how the QA team reinvented its methodology for the next ASP.NET release, and how it did it from the bottom up.

Evolution of the ASP.NET Test Process (part 1)

2009-07-26T18:17:00.001-07:00

This series was originally seen on the ASP.NET QA team blog and never finished. I decided to continue it on my personal blog as its my own opinions of our processes.

For those of you interested in testing methodologies, in these series of posts I'll go over how the QA team has changed the way we test ASP.NET over the years. I have been in the team for over 6 years now and have witnessed several transitions (most of which I think are for the better).

The Early Days (.NET 1.0, 1.1 and 2.0)

I joined the team when .NET 1.1 was about to be released and the team was getting ready to work on 2.0 (which turned out to be a major and lengthy release). Back then the general QA process was:

By the time the QA team finished testing and singing off on a release, the next release was well under development. There is usually a "stabilization" time at the end of a release where the QA team is busy writing automated tests, running them under all sorts of configurations and going through a list of checks called "Exit Criteria". This meant that Dev/PM were off prototyping and designing the features of the next release while we were fully engaged testing the current release.
The QA team divided the features between its members, taking into account interest and affinity. This was called "owning" a feature. When I very first started I was the owner of the new GridView and Login controls. At that point, the spec was pretty much finished, and the controls were already checked into the build.
The first thing to do was to write a test plan. What this meant was to define the totality of tests that you planned to verify and automate. These documents were very descriptive and usually amounted to many pages. We entered the tests into an internal tool that DevDiv uses called "Maddog".
This test plan got reviewed by the PM and the Dev of the feature and by fellow testers to try and uncover anything missing or worthy of expansion.
Next came the main testing phase where the tester divided his/her time between performing "ad-hoc" (which meant playing with the features trying out all sorts of scenarios to uncover defects), and writing automation.
Progress was tracked by the number of tests automated that were defined in the test plan. For example, a test plan defined 200 tests (with different priorities) and we would report status of how many tests were left to automate and that's how we scheduled the remaining work.
One thing to note is that the Developers were not writing any UnitTests back then.
At some point, the team would start to kick off runs. This means grabbing all tests in the repository and running them on many different platforms and configurations to uncover any possible defects. In this phase the time of a tester was divided between writing automation and analyzing the inevitable failures of each run.
This is were we entered the "stabilization" time for the release that I mentioned at the beginning. Making lab runs on different SKU's, languages, side by side, x32 and x64, stress, perf, etc.
After many months, at last we would have finished automating all the tests and have them executing on all the runs. Any bugs found were logged, and either fixed or postponed. At this point each feature owner would sign off, and the product would be shipped. (A similar process was repeated for every beta release as well).

Problems with this model

Over-Automation
- By placing too much focus on automation, the tester spent more time writing and debugging a huge amount of regression tests rather than trying scenarios on the product.
- There was a philosophy of leaving no stone unturned and to automate everything that you could think of.
High cost of change
- ASP.NET is a framework, and frameworks go through many revisions and changes before they are released. However, our QA process was not designed to accommodate this. The QA methodology consistently tried to lock down details too early.
- We spent a lot of time writing fully detailed test plans based on the spec for things that were bound to change a lot, and immediately after test plan creation we launched ourselves into an automation frenzy.
- What ended up happening is that changes in the design of a feature where disastrous, the cost of updating all test plans and all the previously written automation was very high.
No UnitTests
- The QA team was responsible for testing everything, from passing every possible value to each public method of a class to end to end integration tests. This amounted to a TON of tests written by QA.
- Since the only automated mechanism to find regressions was the QA tests it was common for the builds to be broken regularly.
QA team found bugs to late
- A side effect of our automation story was that scenarios were not verified until it came time to automate the test, which usually meant months after the feature was coded.
- All this miss placed effort meant that bugs where found later rather than sooner.
QA team did not start at the same time as Dev/PM.
- In some ways this fact set us up to be the long pole of the team. There were occasions where the Dev had completed all his/her work, but the QA team still had months and months of work to go through.
- It was also common practice for the QA members to have little input on the design of the product. By the time we became fully engaged, most of the features were already designed and implemented.

How we survived

By reading this post you may be disappointed by our team, but keep in mind that I only focused on the "bad" things of the way we used to do things 6 years ago. I believe that regardless of the limitations of our approach the release had a high quality bar. The problems that I mention here, plus the problems with the Dev's process and with our automation infrastructure, all these together are one of the reasons why the .NET 2.0 release took such a long time. Luckily, most of these have been addressed during subsequent releases, and I'll go through the changes that we made and how well they worked in following posts. Maybe you can learn something from our experience...

I am a “subjective” person…

2009-07-26T18:12:00.001-07:00

Last week I was reminded by a coworker what the agile manifesto statement of “People over Process” means. We were having a hallway discussion whether the QA team had a process for analyzing the root cause of test failures over time. For me the point wasn’t whether we had a process like that or not, but rather whether we needed to impose a team wide process like that or not.

I have a bit of a problem with processes that do not give enough value, and I try to always ask 3 simple questions:

What is the problem that this is trying to solve?
Is fixing the problem really worth it?
If it is, is this the only way to fix it?

I should mention that the individuals in our QA team are divided into cross-discipline “feature crews” that are in charge of developing several features within ASP.NET, and you can imagine that each crew has a different problem space, which is why context-driven testing is so valuable to me.

In this particular case, the tester wanted a process that he could use to know when it made sense to invest in re-writing tests that failed often. A worthy cause, but was it high enough to go and pay the overhead tax of creating, documenting and transmitting this new “process” for the whole team?

Even though I oversee test methodologies across the QA team, I work more closely with the MVC crew (what I consider my crew) and we have put such processes in place that we didn’t really have a need for the new process. I mean, we run all our tests very often (twice day), they pass all the time, and if they don’t we fix them relatively quickly. We run them so frequently that if a test begins failing often, we would notice and take action. At least I didn’t feel my crew needed to follow and be monitored by some team wide process to find this out. For this I was called a “subjective” person, because I didn’t need tabulated data and some process to measure it and make a call.

And that is really the key: in the context of our crew, and the way that we work, we don’t need this process. Maybe in the context of some other crew, that makes sense, and hard historical data analysis is required. That is what “people over process” means to me: that people are empowered to create whatever processes makes sense for their project to be successful. What we don’t want is to impose and maintain a barrage of processes for the whole team unless it has a high enough value for everyone.

Another thought: I suppose that at the end there are some persons that need more structured and highly defined processes than others, and this adds an interesting dynamic to a team. How much should we standardize and how much should we leave flexible for each feature crew? I am still not quite sure what the right balance is.

How to learn exploratory testing?

2009-07-20T19:46:00.001-07:00

One of the process changes that I have been in charge of in the team is to use exploratory testing as a technique to discover bugs earlier. Not only am I a huge a fan of exploratory testing, I am of those who believe that it takes skill and training to be truly good at it. And that’s what this post is about, how does someone “learn” something as subtle as exploratory testing.

I’ll start by saying that ET can’t be learned, is more of an acquired taste, and as such, there are some people that can really identify with it, and others that could not care less. If I think of myself, the reasons why I got convinced that this testing approach works were:

I could very easily identify the problems with the way we did testing in our team.
I was willing to try something new.
I practiced and practiced and practiced.
I made the methodology my own by tweaking it.

I had the luck of working with another tester that was equally passionate about agile testing that we could experiment together. We would sit together, testing whatever we could find, taking notes and reviewing our results. I don’t think I will ever claim that I am a master at it, since there is always ways to improve, but now I am very proficient at taking good notes, coming up with good scenarios and fishing out bugs (Note: being a lead has severely decreased the time that I have to keep practicing, which is definitely a downside for me. I am already feeling that I am getting rusty at it).

Once we were convinced that this methodology would work for our product, the real challenge was how to transmit this to the rest of the QA team. In particular:

Even though we had made a fair amount of research, you can’t really teach ET by theory alone, just having a presentation wasn’t enough.
Most of the team could not relate with the problems that we used to have because they weren’t here when we had them.
Some people were not willing or did not see the need to try something new.

The best thing that I could come up with was to slowly introduce the concept in some “pilot” feature crews. Where I would act as the “expert” and coach other testers using the methodology. This worked fine in most cases, because I would spend a lot of time in ET sessions with other testers, and we would practice and discuss together. The problem was that I could not do this for all the team members for lack of time.

Eventually, the value of ET was obvious to the team, that we decided to roll out the methodology for the VS2010 release cycle. The idea was that I would do some hands on training with everyone (in groups of 4) and then a group of senior testers would act as coaches overseeing how this worked with all crews. However, one problem that we faced was that due to resources all coaches ended being sucked into feature crews of their own, and so where not available as true coach resources.

Following my belief that ET is about skills and practice, I established a bi-weekly optional session where testers interested would get together and practice testing software. Unfortunately, most team members don’t attend, they either don’t have time or they think this is not useful at all.

The end result, in my opinion, is that overall our team took a step in the right direction but we are still missing that last mile where we practice ET as a highly disciplined skill instead of just a process. And this takes a personal commitment and lots of practice.

I am interested in hearing your comments: Do you use ET in your work? How do you “teach” it to other team members? How do you manage a team of exploratory testers?

Meeting Frenzy

2009-07-15T22:39:00.001-07:00

Today I was reminded of why some times it feels like we move so slowly in my team and in Microsoft. Here is how my day looked like:

9am: Mail/Work
10am: Meeting
11am: Meeting
12:30: Lunch
1pm: Work
2:30pm: Meeting
3pm: Meeting
4pm: Meeting
5pm: Meeting

I don't know if it's being a lead, but with all these meetings its a wonder I get to test software at all. What few hours I get in front of my computer I have to work as a fast madman to get stuff done. It's this the same in other companies? Or do we have a case of Meetinguitis in my team?

- Federico

Finally tackling the ASP.NET QA “Boat Anchor”

2009-07-14T20:43:00.001-07:00

We have mentioned several times in the CodingQA podcast the problem of our oversized test bed. I am happy to tell that we are now seriously looking at addressing it. In this series of blog posts I’ll describe our progress, as well as our approach.

First thing, what is the problem? I thought that the facts would illustrate our current situation better than anything:

We currently have ~ 20,000 functional tests that have been accumulating since .NET 1.0
In average, each test has at least 3 contexts that it runs with (for example, a test might run with IE and FireFox browsers, but it is not unlikely to see tests with 5 or 10 contexts). Roughly, that produces ~60,000 results on each “full test run”.
We’ll ignore how much time it takes to even execute the tests, let’s assume that we can get an infinite number of machines in a time warp so that we can run all of this in one night.
We’ll ignore the fact that we have to make multiple runs to sign off on a release. Let’s assume we only do one (otherwise, we would be closer to 300,000 results).
Even if 95% of the tests passed that leaves us with 3,000 failures.
Even if you are a genius and it only takes you a minute to analyze a failure (meaning, you investigate if the failure was due to a test issue or a product bug), that is 50 hours of effort, equivalent to 6 full work days (assuming that you don’t take bathroom breaks at all).

Now, even though the tests are distributed amongst a 12 person team, let’s consider that our pass rates are rarely as high as 95% and nobody takes a minute to analyze a test. Why? Because we are in a vicious cycle of:

It is too expensive to run the tests.
As a result, we don’t run them as often.
As a result, they get stale and never get robust and the tester never gains enough experience with them.
As a result, is too expensive to run the tests…

Me and two other of my team members are tasked with correcting the course of this ship, and it is a challenge which I always like. So far, the strategies that we have come up with sound good on paper, but start looking overwhelming if we consider the magnitude of the problem (20,000 tests!)

I’ll keep you posted with our progress.

- Federico

Just another tech blog...

2009-07-13T23:18:00.001-07:00

This past month I had the good luck of talking with Scott Bellware over the phone, and he re-sparked my interest in sharing and learning more about testing from others. I have been doing a lot of experimenting in the ASP.NET QA team with different ways of testing (mostly ideas spawned from James Back), but I don't have an easy way to share these experiences nor to get feedback from others testers/devs out there.

So I decided to start yet another tech blog where I can bore whoever is willing to listen with some lessons learned from a tester in the trenches. And hopefully, listen to other points of view and learn in the process.

- Federico Silva Armas
SDET, ASP.NET QA Team