Don’t be afraid to change the tests! As soon as you’ve verified that you’ve transcribed the code correctly, don’t be afraid to just nuke things and start again. Especially if you have integration level tests that confirm that your features actually work, your unit tests are expendable. If they’re not useful, kill them!

A few people on Twitter found this odd, and I’d have included myself in this statement a while back.

@isaacsanders @avdi seeing @KentBeck delete unit tests during his screencasts changed my TDD process forever.—
Shaun Finglas (@F1nglas) January 21, 2012

Kent Beck’s TDD screencasts changed my view on deleting unit tests however. During the later videos, he actually deleted some tests. Pretty much all TDD resources don’t really mention this. One of the key points beginners learn is that if you break any tests, you’ve introduced a regression. This is not always the case. If you follow the rule of never deleting ANY tests you encounter you are going to be stuck with someone else’s implementation forever. Likewise unit tests are there to drive design, not enforce how something works. I remember discussing deleting unit tests with my work colleagues and finding Kent’s videos pretty shocking at the time. I mean deleting unit tests!?

The more I do TDD, the less this statement becomes so jarring. For example.

public Result DoSomething()
{
    // Implementation here.
    // Pretty straightforward.
    return result;
}

Consider a test for the above behavior, such as we get the result back in a particular state. Pretend the logic is rather simple, and it does not warrant a separate object. Any other developer should be free to come along and change the internals of this method. As long as we get a result back in the correct state, the test should be valid. The test should not care that we are using strings, lists or whatever internally.

Occasionally I find tests like this hard to pass. In other words, I feel like the logic is correct yet the test fails. Maybe I’m using a new language feature, or a language feature that seems to be not working as I expected. If so I’ll break out a new unit test that tests the implementation. Such tests are often refereed to as learning tests. Here with a smaller focus I often become aware of what I’m doing wrong. Following Kent Becks example, I ditch the test after and move on.

I feel this sums up my feelings nicely.

@elliottcable @F1nglas @isaacsanders @kentbeck let them guide you to a design but don't let them hold you to a design.—
Avdi Grimm (@avdi) January 21, 2012

I and others are not saying bin every unit test you have that is covered by end to end tests. Unit tests are great, you can run hundreds in a matter of seconds. They have their place as part of the development process, but do not find yourself working against them. However I am saying you should delete any test which relies on implementation details. I am saying bin any test which does not make sense. I am also saying bin tests as part of a refactoring session as long as you have test coverage higher up. If you don’t have test coverage such as acceptance tests, you cannot be sure you have not broke anything after the refactor.

Any developer worth their salt will know what logging is. You could argue there are two types of logging, either developer logging or auditing. Developer logging would be what we log when something goes wrong. Using the results of this logging we can track down what went wrong, and put fixes in place to stop this event from occurring again. If this logging fails, or logs the incorrect thing it is not the end of the world. Due to this, I generally do not care for testing such scenarios. The code should behave the same with our without this logging.

Auditing would come under logging which as part of the application needs to be carried out at all times. Consider visiting a cash machine. When withdrawing fifty pounds, you want to make sure your bank logs this in case anything goes wrong. This sort of logging is crucial, and must work and must log the correct data. This is considered a feature, therefore this should be tested as it is part of the behavior of the application.

When I think back to my first few years of programming my code was littered with logging. In the early days after each statement, variable and function I would print out what happened, along with any errors that happened. In fact I’d say that everyone starts out like this. The strange thing is as we get better, the logging becomes less and less. Rather than the first class citizen we relied on in the early days, logging is seen as boring. The problem with treating logging code as a second class citizen is that when things go wrong, it can be very difficult or near impossible to track down what has happened. When you realise you need logging, its often too late. You will need to redeploy the application and wait for the problem to expose itself again.

Back in 2011 we had a difficult problem to track down. The dreaded “OutOfMemoryException“. Being the cocky developers we were, we decided to add the logging last. After all, it was there for when things went wrong. We never planned things would go wrong, after all it “worked on my machine“. After redeploying the application with logging we managed to track down roughly what was going wrong, and in turn began to resolve the problem. Had we added this logging initially, we could have resolved this problem in half the time.

The lesson I learned here was simple. Any time you have an error, log it. If the logging is not in place, we add it. Creating a new application? In the first iteration(s) make sure some form of logging is in place. I believe by following this simple rule any future issues can be handled easier. Logging should be a first class citizen regardless of purpose.

Around the same day I had this revelation, the image above popped into my Twitter feed. Image via @olafurw

1. Turn off automatic updates for your browser/plugins
2. Set your IIS (or equivalent) app timeouts to zero
3. Create a base Selenium Fixture for use in your tests
4. Update to the latest version of Selenium
5. Warm up your apps prior to testing
6. Ditch Selenium – test at the API level

Turning off automatic updates seems like a no brainer, but after a fresh install we forgot to do this once and spent some time figuring out why Firefox would not load on the CI server. It turns out that the “You’ve just updated” window was blocking the test from continuing as it had stole focus.

The second point is with regards caching and the general responsiveness of your application. We have a few applications that take about thirty seconds to fully warm up due to the huge data set they rely on. If we can build this cache once, then store it for as long as possible, subsequent hits to the app should be instant. In other words, we try to mirror our live environment as much as possible.

Our custom test fixture attribute enables the ability to modify all Selenium tests in one go. We found that from 3am to 5am our databases undergo maintenance, therefore we do not run our regression tests during this time. All this took was one change within the attribute to apply to all tests. For example:

public class RegressionTestFixture : TestFixtureAttribute
{
    public RegressionTestFixture()
    {
        Category = "Regression";        
        OnlyRunTestsWhenDatabaseIsActive();
        // Etc...
    }
}

We simply inherit from NUnit’s TestFixtureAttribute and use this custom attribute rather than the standard TestFixture attribute. The inheritance is required to ensure that third party tools such as test runners still work as expected.

Previously we were using Selenium 1.x with Sauce RC. Having ditched this and upgraded to Selenium 2.x we’ve been able to update our browsers to the latest versions, in turn this allows improved speed and stability when runing the tests.

On our local development machines the application you are working on is often in memory, meaning subsequent hits should be much faster after all dependencies are loaded and cached. The issue we discovered on our CI server was that after a fresh build of the whole codebase, the initial hits to the applications would be very slow. To combat this we added a warm up step to our build. Just before the tests are run we would perform a HTTP GET to invoke our applications start up processes. This added somewhere in the region of thirty seconds to the build, but the increase in stability is staggering. No longer will Selenium report timeouts.

Finally the fastest end to end tests come from not using Selenium. Ditching the browser completely and testing as high up in your API is the quickest, and most stable solution. Combining this thinking, with a handful of dumb Selenium tests that just check for the likes of 404s seems to be the most optimal solution currently.

Having done these at some point over the past few months we’re starting to get to a more stable point with our Selenium tests. We’ll be looking to take this forward with future tests and hope to enjoy continued stability.

Recently myself and another colleague wrote an acceptance test for a feature that had yet to be implemented. For this end to end test we used Selenium, after all we wanted to test the whole feature so this made sense. Our test performed some simple user input, performed a calculation and checked the response. The problem with the test was it was very brittle. If the application had not recently been used, the massive data set the application relied on would not be cached.

To get around this we added a few Thread.Sleep() statements into the test. This worked rather well for the majority of test runs, but sometimes these pauses were not long enough. On the other hand sometimes the data was cached, meaning these sleeps would be unnecessary. One resource which has recently done the rounds was regarding useful advice about using WaitForPageLoad() and WaitForCondition(). WaitForCondition will only execute once a condition has been met, such as a element becoming visible. This meant that for the times when the dataset was in memory the test would be executed immediately, while the times when the data was being loaded, the test would simply wait until the test was ready to move on. This was a very simple, yet highly effective tweak to our tests. The execution time went from roughly thirty seconds, to just less than ten seconds in one case.

This was not the end of the battle to achieve more stable Selenium tests. Some of our tests were still rather flaky. Some mornings we would enter work, notice the red build and discover that the several failed tests were down to Selenium timeouts. During the daytime however, we rarely had these issues. In order to fix these problems I increased the frequency of builds. The idea being the more we run our builds the more chance we have of spotting the errors. After all, if something was to fail at 2am, I am unlikely to care. 2pm however, and the team will be all over it. By making the problem more visible, we would be forced to fix the outstanding issues.

The aim was to make the tests as fast as possible, while maintaining stability. One thing the excellent Growing Object-Oriented Software (Goos) touches on is the aspect of not needing to perform end to end testing at the GUI all the time. The benefit of not touching the UI is huge. Your tests are faster, they’re more stable and a heck of lot easier to write. The other nice benefit of testing from an API point of view, rather than the browser is it forces you to decouple your app from the views. If you’re not writing fat models and skinny controllers, you’ll have adapt in order to test as much of your application as possible without hitting the UI.

What about the remaining part of your feature that is not covered by the application? I like to imagine this part as the tip of an iceberg. As this area is small enough the actual UI testing you need should be minimal. So here we can let Selenium do what it is good at. Click things. Selenium is great at this. All you need to do at this level is check for 404s, incorrect page titles and a few other mundane aspects of the UI. There should be no need to check if your actual application is correct at this level. For correctness, you should have a large suite of fast, isolated, unit tests.

Another point to consider is how often your view actually changes, in comparison to your actual underlying API. A designer should be free to move things, rename content, add images and so forth without breaking tests. As long as there is a calculate button somewhere on the page, and said button takes you to a result page, who cares about everything else? Likewise the code underneath can be consistently changing behind the scenes, as long as the API remains constant, our tests should always be valid.

For the technical low down on some of the ways we are achieving more stable end to end tests, check out six tips to speed up Selenium tests.

Image courtesty of Natalie Lucier

To get to this point we decided to auto generate a C# dictionary (hashtable) of postcode districts that would be used during compilation and stored in memory. The file we were provided came from a Word document that in turn went through Google docs, and finally Open Office. The end result was something similar to the snippet below. Please note the inconsistent tabs and spaces, as well as upper/lower case conditions.

The task is simple. Write some code that will parse a file containing lines of text in the format of district, condition and value. The code should accept a string representing each individual line and return a string representation of a C# dictionary. For example given the first line in the input file below, the output would correspond to the first dictionary underneath.

• Column 1 contains the district, either one or two characters.
• Column 2 contains the condition, either “All” or “Ex Below”. Exclusions apply to the district that was last processed. Exclusions should be stored as areas e.g. in the form {DISTRICT}{AREA} while conditions that apply to all should be stored in the form {DISTRICT}.
• Values that are linked to a district/area are in the range one to five

Rules

1. Copy & paste the input file to maintain the “formatting”
2. Handle any number of tabs between columns.
3. Handle any number of spaces between columns.
4. Handle any number of tabs and spaces between columns.
5. Handle any case of conditions, eg. Ex below or EX below
6. Handle empty lines – no tabs/spaces
7. Must be able to read the whole input file in one go.

Input File

AB		All				1

AB		4				2

ST		3-5			5

ST		ex below			4
		26				3

S	Ex below	        4
	26/30			3

B		Ex below			2
		45-47				3

C		EX BELOW			4
		26/28-31			3

DY		1/2/4				5

DY    1/2/4                             5
      5-6                               3

DY		1/2/4				5
		5-6				3
		10-12				2

Output Per Line

{{"AB", 1},}

{{"AB4", 2},} 

{{"ST3", 5}, {"ST4", 5}, {"ST5", 5},} 

{{"ST", 4}, {"ST26", 3},}

{{"S", 4}, {"S26", 3}, {"S30", 3},}

{{"B", 2}, {"B45", 3}, {"B46", 3}, {"B47", 3},} 

{{"C", 4}, {"C26", 3}, {"C28", 3}, {"C29", 3}, {"C30", 3}, {"C31", 3},} 

{{"DY1" 5}, {"DY2", 5}, {"DY4", 5},}

{{"DY1" 5}, {"DY2", 5}, {"DY4", 5}, {"DY5", 3}, {"DY6", 3},}

{{"DY1" 5}, {"DY2", 5}, {"DY4", 5}, {"DY5", 3}, {"DY6", 3}, {"DY10", 2}, {"DY11", 2}, {"D12", 2},}

File Output

The final test will be to process the whole file in one go. In the programming language of your choice this would be a case of reading all lines within the input file and invoking the code you wrote for each line. The end result should be:

{{"AB",1},{"AB4",2},{"ST3",5},{"ST4",5},{"ST5",5},{"ST",4},{"ST26",3},{"S",4},{"S26",3},{"S30",3},{"B",2},{"B45",3},{"B46",3},{"B47",3},{"C",4},{"C26",3},{"C28",3},{"C29",3},{"C30",3},{"C31",3},{"DY1",5},{"DY2",5},{"DY4",5},{"DY1",5},{"DY2",5},{"DY4",5},{"DY5",3},{"DY6",3},{"DY1",5},{"DY2",5},{"DY4",5},{"DY5",3},{"DY6",3},{"DY10",2},{"DY11",2},{"DY12",2},}

Our current VS theme

There’s always lots of buzz around whether or not like or dark themes are better. Some people claim that darker backgrounds are easier on the eyes, and I personally agree with this statement.

That being said, if you know of a better looking theme let us know in the comments.

Download the updated theme  (minus all other VS settings)

Our reward, even with optional fruit

Having just begun the process of moving our CI builds to use Team City instead of Cruise Control.NET we decided on the the following rule.

For every twenty four hours of green we have, the team gets a beer on Friday during the weekly retrospective.

This rule is not only to enable the team to have a cheeky beer every now and then but to encourage more responsibility with regards the build being broken for too long. Hopefully the times for broken builds will begin to decrease over the coming weeks.

Regardless of the future, today we hit our target of the build being green for over twenty four hours, will we hit the target of forty eight hours for the following retro? Only time will tell.

We decided to rewrite the feature again, but applying all the lessons we had learned from the first attempt. A recent blog post by royvanrijn on this very topic made me appreciate what we had done. He points out that the best code occurs from several attempts, and unlike what people may expect, the repeat attempts need not take the same amount of time to deliver as the initial attempt.

“The second time you write the code, it’ll only take a fraction of the time it took initially.”

This principle of repeating a task made me think of when I was decorating my old bedroom. I helped partake in the difficult task of wallpapering the ceiling. Prior to this I had experience wallpapering before, and would have no trouble repeating this exercise again. However, wallpapering a ceiling was something completely new. Me and my dad were reluctant to start, until I had a rather devious plan. We would decorate my brothers room first, followed by mine. That way, if our first attempt was a disaster I would not be the one living with the dodgy ceiling.

It turned out that our first efforts were not too bad. Granted it took a while, there was the odd rough patch and several obscenities were used, but we got the job done. For the second room we completed the task much quicker and with practically no problems.

The process of wallpapering the second ceiling was the DIY equivalent of scrapping our feature and rewriting the code. We never stripped the first ceiling afterwards, we just took everything we learned from the first round and used it to make the process of papering the second ceiling much easier. The interesting point to bare in mind with scrapping code and rewriting is the rewrite will not take the same amount of time to get back up to speed. Just because it takes n to implement a feature, the second time around you can often complete the feature in less time, at much higher quality.

I’m not suggesting all code should be rewritten multiple times. Spike solutions are often a more suitable process to ease the development process, but in certain cases practice makes perfect, even if it means you wait that little bit longer for the perfect ceiling.

Image courtesy of Nils Geylen

Regardless, one area that remains troublesome in my opinion is object mapping (or the correct term of conversion) code. While I’ve not personally been involved with this reworking of the codebase, I have recently just finished reading Kent Beck’s – Smalltalk Best Practice Patterns. Many of the developers I follow on Twitter have been blogging about this book and I figured it was time to give it a go. After all it gets massive praise whether or not you use Smalltalk. While reading this book a few key points regarding object conversion are discussed and I found them incredibly relevant.

So should you read the book? I would say yes. I don’t program in Smalltalk. I don’t plan on programming Smalltalk. Nor had I read a line of Smalltalk before. But you should still read this book. The first half is incredibly relevant to any OO programming language. Granted I found the second half is less useful, but the gems I’ve picked up in the first half more than make up for this. In fact, pages 28 to 30 are so good I figured it would be worth sharing.

I’ve been convinced for a while that creating separate objects to convert objects is unnecessary, and in fact adds to the amount of code you need to write and maintain, thus increasing resistance for change. So if we remove this unecesary, intermediate object, how do we create a new object from another object? The answer is conversion. This answer strangely comes from a book all about Smalltalk. The answer also strangely comes from a book over ten years old. Pages 28 – 30 cover the topic of conversion. The following is quoted heavily from the book, but I recommend reading the pages in full.

Conversion

Question - How do you convert information from one object’s format to another’s?

Answer - Convert from one ojbect to another rather than overwhelm any one object’s protocol.

What this is getting at is we could using C# extension methods do the following to the String class.

"B74 BH2".ToPostcode();
"01215435643".ToTelephoneNumber();
"AD3KDK1-2121K2A".QuoteReference();
"Hello".MoreConversion();

This would be abusing the String class. If we want a postcode from a string, we should have the Postcode object create us a Postcode from a string, not the other way around. There could be hundreds of conversions from strings to a new object, but we would violate the string class if we did this. In turn, Kent goes on to say “Conversions that return similar responsibilities should use a Convert Method. To convert to an object with different protocol use a Converter Constructor Method“.

Converter Method

Question – How do you represent simple conversion of an object to another object with the same protocol but different format?

Answer – Provide a method in the object to be converted that converts to the new object. Name the method by pre-appending “as” to the class of the object returned.

Smalltalk examples

Collection<<asSet
Number<<asFloat

In C# these would be:

Collection.ToSet()
Number.ToFloat();

In C# the convention is to use To rather than As for converter methods. For example we could do quotes.ToArray() on a List of Quotes. We still have the same protocol, a collection of quotes, we are just storing them in a different format. The rule for adding such methods is that there should only be one sensible way to perform this conversion, and the source and destination share the same protocol.

Converter Constructor Method

Question – How do you represent the conversion of an object to another with different protocol?

Answer – Make a constructor method that takes the object to be converted as an argument

Smalltalk example:

Date class<<fromString

In C# this would be:

class Date
{
    public Date(string date) { ... }
}

In our codebase we have a RegistrationDate object. We have a constructor that takes a string representation of the date (from the outside world) and constructs a RegistrationDate . This very same principle can be applied to other, more complex objects. For example consider an active record style approach below. Here QuoteRecord represents our database object, with Quote representing a domain object. The following would be the converter constructor method. In other words, we create (or convert) our quote from the quote record. No separate mapper. No intermediate object. Less resistance for change.

class Quote
{
    public Quote(QuoteRecord record)
    {
          // Code...
    }
}

The benefit here is that we have minimised friction. If the requirements for this code changes we will need up to update at worst, the record and the domain object. Had we used a separate object to perform the mapping we would end up with a third place to maintain if we decided to add a new property to our QuoteRecord.

I’ll admit to having only used this technique for a week or so, though so far it has worked a treat and I expect it to continue working considering these techniques have stood the test of time.

We often use a test fixture per behavior, so our tests are often grouped by behavior, for example “GivenASuccessfulQuote” or “GivenAUnsuccessfulQuote” would be test fixtures for testing the quotes behavior. These fixtures would test the happy path of a quote and erroneous behavior respectively. The issue comes when we wish to add a new feature. Imagine we wanted the unsuccessful quotes to return a result with a message explaining why the quote was unsuccessful. We’d often start by writing a new fixture, adding the required set up and relevant asserts/verifies. You might argue that we should look for similar fixtures to add to at this point, but I prefer duplicating code (intentionally or unintentionally) first to get the feature working.

Recently having carried out a similar process mentioned above myself and another developer noticed that there was duplication, albeit slightly different between our new fixture and the unsuccessful quote fixture. We never wrote the other fixture, so we never planned for our new fixture to become so similar. I tried rather unsuccessfully removing the common setup code between the two classes. The revelation came when we took the assert from our new fixture and pushed it up into the unsuccessful quote scenario. This assert failed as we were missing the expected message that was not being set up correctly. After moving this expected message from the new fixture up into the unsuccessful fixture the assert passed. Once this went green, we pushed the assert out of the test we had just migrated to and all was well. The new message fixture could be deleted, after all it was behavior of an unsuccessful quote. Having done this we removed in the region of about twenty lines of code. I was stunned at how well this worked. We had essentially tested our test code worked.

The process was as follows – Remember, it’s only worth doing this if the behavior you are testing makes sense in the fixture you wish to migrate to. Putting all your tests into one fixture defeats the purpose of grouping behavior via test fixtures.

Steps:

1. Write new fixture – make its test(s) pass
2. Identify duplication between fixtures
3. Migrate if it makes sense for the behavior you are testing to belong on a new fixture – otherwise try and remove duplication another way
4. Push assert into an existing test method
5. Run tests
6. If the tests pass you’re done, or if they fail move the required setup from the new fixture into the current fixture
7. Run tests
8. If the tests pass – move assert(s) into separate test method(s) otherwise go to step 6
9. Delete new fixture
10. Run tests

I have to thank Paul Shannon for this gem as he was the one who explained how to do this. He plays the process down but the fact that this is a safe way to remove duplication between tests is great.

Pseudo code worked example

Part 1:

class GivenAUnSuccessfulQuote
   Setup()
   ThenItShouldBeInValid()
   ThenItShouldDoOtherStuff()
end

class GivenAUnSuccessfulQuotesMessage
   Setup()                             // This is near duplication above
   ThenItShouldReturnAInfoMessage()
   ThenTheMessageShouldDoOtherStuff()
end

Part 2:

class GivenAUnSuccessfulQuote
   Setup()
   ThenItShouldBeInValid()
   ThenItShouldDoOtherStuff()
   ThenItShouldReturnAInfoMessage()    // Moved this from the fixture below
   ThenTheMessageShouldDoOtherStuff()  // Moved this from the fixture below
end

class GivenAUnSuccessfulQuotesMessage
   Setup() // This setup is near duplication of the fixture above
end

Part 3:

Run tests

Part 4:

Move required setup from the GivenAUnSuccessfulQuotesMessage until the GivenAUnSuccessfulQuote fixture is passing.

Part 5

Delete the GivenAUnSuccessfulQuotesMessage fixture.