Often a quick search on usages of a method or class is sufficient. You look at the tests briefly to see what is going on, and then you make a new test to assert the new behavior that is going to be added.

Sometimes, when there are a lot of tests, I just prefer to remove the code I’m going to modify, and see what tests fail.

This technique provides some quick and useful information. For instance:

1. What tests were written that test the code? Are there any applicable tests?
2. Are there other tests that implicitly test the code and should not?
3. Are the current tests testing all the conditions?
4. Is there a test that no longer applies?
5. Is there a test who’s expectation are changing? That test can be modified instead of writing a new test.

This approach tends to be practical in the case of unit tests, and perhaps a few integration tests that you may know are related. Otherwise, it becomes too expensive to run a long running suite.

An additional benefit is that if you already know of some tests that should be testing some code, you may find a test that meant to test the logic, but for some reason was always passing.

This made me think about a general organizational “smell” in software development. For lack of a better label, I’ll call it fragmentation of goals. A manager might be juggling multiple projects, of which yours is not the highest priority. QA’s may be measured based on how many bugs they find, or how many test cases the document. A developer might see a better chance at promotion by going to all the meetings he or she is invited to, moreso than the delivery of features. Entire teams within a project may even be motivated by goals that are not complimentary to the success of the project.

When I was in college, I liked to compare the team projects I was on to games. My goal was making sure that we were winning the game (in that situation, getting an “A”). Later in my education, I read Agile Software Development by Alistair Cockburn, and he made the analogy that software development is like a cooperative game that we are trying to win. I’ve always loved that analogy, and I think that it’s a useful tool in conceptualizing an effective team or project. Much like a team game, software projects must be a coordinated and unified effort amongst individuals to achieve a common goal.

There are three important factors that are core to a team being able to deliver software successfully, and each easily becomes fragmented (I’m generally using team and project synonimously).

1. Knowing how to win the game
2. Being motivated to win the game
3. Having the ability to win the game

I’ll discuss each inline below.

Knowing how to win the game

A team or project must know how to “win the game” - there has to be acceptance criteria on the results of the teams efforts. Not having this clearly established makes it impossible for people to unite towards winning (I am not implying having all the “requirements” - I am talking about a clear understanding of how to accomplish the business goal). Without a clear goal, a team can only work towards something that they hope is right, and everyone may not agree on what that is.

An important aspect of this strategy is to have a single dedicated product owner - someone who can provide a vision of the end goal, and be the final decision maker for what aspects of the product will achieve that goal. This will help not only knowing what to build, but have the ability to prioritize when to build something.

Being motivated to win the game

The second part is ensuring that the team is motivated to win the game. The team as a whole will suffer if the individuals are engaging in activities meant to promote their own goals over the goals of the team. There are many, many parts to this. My basic opinion is that measuring an individuals success should be based on their contribution to the team, and the teams success.

It is difficult to make sure that everyone has the incentives to be motivated. It is not enough to hire people and then put them to work on something. You must make sure that they are happy, and personally interested in succeeding. It’s especially important to make sure that the personal goals and desires of the individuals aligned with the success of the team.

This doesn’t just apply to individuals. Often large projects involves dividing the project into smaller teams. It’s not beneficial to set up teams with goals that allows them to succeed while the project fails. The concept that this works is often an attempt to suboptimize a system - the idea that if the individual components are succeeding then the whole will succeed. This approach is flawed and often leads to a team achieving it’s goals, even at the expense of the project.

Having the ability to win the game

The team must have the ability to be effective and able to win the game. Further, the team must be empowered to make changes to increase their ability to win the game. The easy example of a team not being able to win is that when the team is formed with people who’s skill-sets are not suited to success. For example, building a team of people that have never done web development, and asking them to build a sophisticated web application. Another easy example is creating unachievable goals for the team, such as having the scope be too great for the team to ever achieve.

More complicated scenarios involve putting a restriction on a team that they must use a certain language or technology, or to disallow some tools and technologies, without consideration as to whether it actually helps the team succeed. It could even be as simple as the developers not being able to get licenses for software that they need, or computers to support the development efforts.

This point is part of the reason that it is critical for teams to be responsive, continually improving, and honest about what will make them more effective and better able to succeed. It could be that 1 month into a project, the team already knows that it’s failing and it’s likely that nothing can be done to help. It could be that the team is producing features, but is losing a lot of time trying to communicate to disparate groups that they rely upon.

Designing the game

A final point here is that all of these factors are part of setting up the game in the first place. Often the decisions that are made early on in a projects life sets it up to succeed or to fail. In my experience, the structure of a software project tends to be an extension of the organizational behavior of the company. That structure may not be an ideal structure for a software project.

Regardless of how the project begins, it’s important to change how the game is played if it is necessary to win the game.

It was a fun application to write - It had a Java Swing front end with JNI to use portions of the win32 API and X-Windows API, and we wrote a video and audio server to distribute the view of the application and allow the users to talk to each other in on online group phone call while they were using the application.

There was one desired feature that we could not implement. We wanted to allow users to simultaneously enter text, and simultaneously control their own mouse cursor to interact with the application. We were completely blocked on this feature because it goes against a major assumption of all major operating systems - that one user will use the system at a time. For instance, it’s assumed that one window will have focus at a time. It is also assumed that there is only one mouse cursor at a time. There are even more subtle issues that come up, such as there being only one copy and paste buffer.

At the time, I was disappointed, but I didn’t dwell on it too much. I figured that that level of multi-user collaboration would only service a very small niche market, and that it wasn’t so bad that we didn’t have the possibility of better collaboration.

Fast forward to today. I pair program most of the time, and I find myself longing for an environment that would allow me and my pair to collaborate in different ways. It’s standard pair programming, where we both work on the same window, and have to share the same keyboard and mouse input (preferably with 2 mice and 2 keyboards). So what do you do when one person has to check email, or one person decides to research an issue online, or otherwise engage in a task that is not the same as the other pair member? Typically, each person will have a laptop that they can use for these situations, but that causes the person to separate themselves from the primary work environment. Sharing information will have to done by sending an email or putting a file on a shared drive, or having the other person look at information on the laptop’s screen. What about spiking a problem side by side, where each pair tries two similar but different approaches to solving a programming problem? I would prefer to collaborate with someone in the same environment, where we can do some different tasks in parallel, and can share windows, applications, and data in the same desktop environment.

I think that it would be interesting to expand this idea even further. What if an entire team worked in the same environment, on the same desktop? Each person could have a portion of a huge desktop, and people in a team could choose to collaborate with each other in a very fluid fashion by putting their desktop spaces close together. It would be an interesting experiment in collaboration applied to software development.

Even if multiple simultaneous mouse and keyboard input were generally possible in operating systems, there would still likely be a lot of complex concurrency issues for developers, such as making sure that servers bind to different ports and write out to different files. It’s likely that once popular operating systems generally allow the multi-user features I mentioned before, that it would still be preferable to have isolated machines that interact through collaborative software rather then have everyone actually use the same machine.

I’ve seen software that allows sharing of a the mouse and keyboard across machines, and sharing the copy and paste buffer, but the purpose was to allows one person to connect 2 machines and use them like an extended desktop. Perhaps software like this could be tailored for better collaboration for pair programming or developers on a team not pairing.

Basically, what I want is an inspection tool that also has a bundle in textmate, so that I can see the history of objects that are created, as well as interactions in a form that is searchable and that quickly shows what really happened. It would be like having debugging on all the time.

The goal of this tool would be to allow developers to view what really happens with the code when it executes. Sometimes finding the source of bugs is difficult because statically analyzing code is not the easiest way to predict the future state of an object, or to predict the interactions between objects. You don’t really know what happens until the code executes.

Nor is traditional debugging an efficient way to get the information that you need. You have to set a break-point, add in a line to call the a debugger, or even print out some information in order to answer a question about the actual execution of the code.

What I want is a tool that records a wealth of data about the actual execution of some code, such as when I run a test, or initiate an action from a controller. Then, I would like to be able to filter/query that information to get more precise information.

For the inspection tool, I’d like to know things like:

- See the actual methods and fields of a class or object instance at the beginning and end of code execution
- See all calls made to an object, or a specific instance of an object
- See all callers of a specific method of a specific object type or object instance
- Detect changes in the structure of an individual object that happens, such as methods being added or redefined after the object is initialized
- View all changes to fields of an object
- I would like to see the order that objects are created in. I would like to see when all objects of a certain type were created, or objects that respond to a specific method
- I would like to see all interactions between 2 objects
- I would like to be able to further scope the search, so that I only see information within the scope of a specific method call

Preferably, I would want to be able to view all of this information after running the code one time. And then be able to enter some search criteria into a form and have it reduce the amount of information to match my search criteria. Then, I can tweak some starting data in a test, and then rerun the test and get a new set of information to query.

I think that most of this information could be obtained by creating a Module to intercept all method sends to every object, and save some data about the context of the call when it’s made. Then, a client tool could parse the output, and allow a developer to query it.

I tried to do some googling to find something like this, but all I found were standard debugging tools, which I don’t think have the features that I’m looking for. If anyone knows of something that can do what I want, let me know.

In Ruby, you can do this in one line.

It’s lovely how much more succinct that operation becomes. It can actually get more succinct by using Symbol#to_proc:

There is still the question of whether to move this to a method. Now, moving it to a method doesn’t reduce multi-line duplication the way it would in the Java example; there is only a minor amount of single line duplication that can be reduced.

The resistance I often get to refactoring a single line to a method is that it creates more total lines in the file with the def/end lines and spacing. Also, it takes a slight amount of effort to do the refactoring, and some developers won’t refactor unless the need is glaring and obvious.

There are still important reasons to move this to a method, and those reasons are better readability of code and reducing duplication of expressions. Without the refactoring, you’ll see code that looks like this:

After the refactoring, this is what it would look like.

It’s a minor point perhaps, and I’m sure that I could come up with much more excessive examples, but I just wanted to focus on a simple example. By replacing the expression with a symbolic reference, a method in this case, you express in English something that is a programmatic operation. This improves readability quite a bit.

Also, there is the additional benefit of abstracting on the concept of expired claims. By having multiple places using “all_claims.select(&:expired?)” to express expired claims, you duplicate the implementation detail that expired claims are derived from a larger collection of claims. This may not always be true, and a change in that derivation results in a change in many places.

Perhaps the question here is how DRY should you make your code. I’m still undecided on this on this point, but I think that the amount of work that it takes to reduce duplication to this level is minimal, and the result will be an important part of a pristine codebase.

Ironically, I often find that the people that make these claims do it out of ignorance. Many of these personality types make these statements and then have people explain to them why it’s not as stupid as it seems. Even then, a subset of these people will continue to assert that it’s all stupid.

Strictly speaking, it’s not a bad technique; I’ve met many people with this approach who earn validation in their claims from people who trust that they are, in fact, very smart. I don’t want to discredit some of these people; I’ve met some that were incredibly intelligent and could successfully back up their claims with both facts and a quick wit.

For many, it’s an argumentative technique, where they begin an argument basically saying “I’m smart, and most of you are stupid, now try to tell me that I’m wrong.” When I was a teenager, I started a lot of debates this way. I thought that I was smarter than most, and I felt empowered by this approach.

The problem that I find is that I meet too many developers are not good at it. They are clearly ignorant of the context of their statements, and it shows. They tend to look a lot like the naive teenager that thinks that they know more than they really do. Nor are they often the type of people that you can easily inform of the downside to their approach; Many of them won’t consider that they are wrong long enough to accept feedback.

Perhaps this bothers me as much as it does because I’m a consultant. Either it occurs with other consultants, and it demonstrates a lack of consulting skills, or it happens with client employees, where I’m not in a position to give them completely honest feedback.

Aside from this being a rant, I guess the purpose of this post is to cause some amount of introspection from people out there. Are you putting your ideas out there as best as you can? If you just read this blog post, and you feel like it could apply to you, then think for a while whether you can improve your approach. If you really want your ideas and opinions to have value, then try to articulate them in way that shows that you are cognizant of the factors involved, and are trying to make things better. Otherwise, you run the risk of discrediting yourself, and decreasing your ability to affect change.

For instance, a public facing application generally requires a higher bar of quality than an internal application. Often, small defects in an internal app can be tolerated, while they can become hugely embarrassing mistakes in the public facing application.

An example for a public facing application: If amazon.com has a bug in whether users can add items to the cart at certain points, it may not be the only way to add items to the cart, but a small bug could result in many missed sales, and an expensive loss to amazon (not to mention the risk of losing shoppers from a poor user experience). An acceptance testing suite that tests on specific small features like that can be very valuable - offsetting the cost of maintenance.

However, some projects may find that they can get away with a reduced mitigation of defects. These projects may also find that they can run acceptably with a small team of QA even without those tests.

For instance, I worked on an internal workflow application. It was a large application in terms of the number of screens and tables (more than 150 tables). It was implemented as a Java Swing thick client that communicated with an apache tomcat server. There were less than 500 users using the application, and all of them were employees of the company. The important consideration was not to have no bugs occur, but to not have any occur that would block a users ability to complete some portion of a business critical workflow. If any defects like that were discovered in production after a release (released every 1 to 2 weeks), we would be able to push out a branched hotfix release to get it working again.

We had 2 - 3 QA’s and about 12 devs. They were unable to manually do a full regression test run since they had over 10,000 test cases (which would take months to test manually), so they focused on testing what they knew changed, and doing a quick happy path test of the workflow to make sure.

An additional aspect to this was that developers often helped out in QA. We thoroughly (manually) tested our changes when we were done with a story or bug fix, and often put developers on QA tasks in order to increase testing capacity when needed. One last thing of critical importance; when a developer made changes to the codebase, there was communication to the QA people on how risky the change might be. Low risk changes could get by with a QA testing a single screen briefly, whereas high risk changes would merit greater regression testing of the application. Developers worked with QA to give them better information to help them discover bugs more often.

The project ran well with this approach. There were defects, but almost none of them were critical. In the case of a critical bug occurring, they were quickly resolve with little cost.

I’m sure that you have seen code like this:

And then somewhere in the code, you might see:

This is a common smell in a code base where client code has logic that should be better encapsulated. Now, whenever someone adds a friend, they are going to have to search for references on Friends and add a new entry, or they are going to get a RuntimeException from the switch statement. In a well tested codebase, there is probably going to be a unit test that asserts that all Friends have best friends. In any case, the switch statement in the client object code is not great from an OO standpoint and it creates a maintainability issue.

The first refactoring is to move all Friend related logic to the Friend enum where it belongs.

Now, we can just ask the Friend who their best friend is.

Nice. Still, I’m not wild about that switch statement. Developers still have to know to update it, and really, I’d rather not even have to throw an exception because it was misused. It would be better if the structure of the code did not allow misuse.

Here is an example of how to do this:

Then,

Great, now we know that when someone adds a new friend, they will immediately be confronted with having to supply a best friend. My only issue with this approach is that all the method definitions become verbose when you introduce many methods like this. I tried different approaches to solving this problem, but due to the enums referencing each other, I was not able to do a different approach.

Here is an example of what you can’t do:

This won’t work because you can’t reference Chandler in the enum definition for Joey. The Chandler Enum hasn’t been defined yet, so this won’t even compile. However, you can “trick” the compiler by fully referencing Chandler using Friend.Chandler;

However, the result is not what we want:

Joey.bestFriend(); –> null
Chandler.bestFriend(); –> Joey

Even though I can reference the other enum instance this way, it resolves to null. The reason lies in the fact that when the Enum is compiled, each instance is a static final field, and initialized in a static block. Here is a snippet of the generated code:

Interestingly, I can write a program that tries the fully qualified name for explicit static constants, and it works:

But if I use a static initialization, it doesn’t:

Interesting. The compiler is smart enough to resolve the correct value when you don’t use a static initialization block. Back to my original example with Friends. The next attempt was to create an anonymous constructor (actually, an instance initializer) for the enum instances and see if I could get what I want:

I had to remove the final from bestFriend, since I’m inializing the value when the object is instantiated using an instance initializer. This compiles, and seems like an okay approach. My hope was that the references to other enum types would get resolved in much the same manner as in the case of creating a method that returns each one. Interestingly, this doesn’t happen.

The reason is that even though I am using an instance initializer, it’s being called from a static block since the instances are created in a static block. Turns out to be a naive attempt. Here is what it ends up looking like when the enum gets generated as a class:

Oh well. That’s as far as my experimentation went. I’m satisfied that I can at least create an anonymous subtype of an enum that returns the correct value, but if anyone has any ideas on how to do this in a cleaner way, let me know.

Developer Machines

I was once on a project where the local update and build process became an hour long. I won’t get into the details, but it was largely an IO bound delay, with portions with the processor as the bottleneck. We were using Dell 610 laptops. When some developers started gettting Dell 620’s (dual core laptops), we discovered that it reduced the local build time on the machines by 33% to 50%. Whoa.

Think about that. A 60 minute build cut down to 30 minutes. Let’s assume that developers only build once per day and that each developer has an average cost of $100 per hour (total cost to the organization, not just wages). With those savings, getting every developer a Dell 620 instead of a Dell 610 pays for itself in couple weeks. This is just considered cutting a long build in half. There are many other situations where having a slow machine causes lost developer time.

We lobbied for getting the developers better machines, and were mostly denied. I discovered that organizations measure the cost of people separate from the cost of hardware. In fact, they may be accounted for by different departments entirely, where an arbitrary budget is given to the department that issues employee computers.

I’ve seen this on every project I’ve been on. We are given slow machines, and time is lost. It may be lost because I’m running grep over a lot of files, it may be because when I have my all my development tools open and the machine slows down.

I think that it’s fine that organizations begin developers with cheap machines, but they should be quick to spend money at the first sign that it is needed by the developers. I believe that it is an aspect of agility that many organizations fall short on, where the ability to respond to constraints in the software development system is hindered by the structure and policies of the organization.

In fact, I think that IT organizations should do a few tests against their technology stack and see what kind of performance difference exist, and use those numbers to decide what kind of machines that developers can use that will result in the best performance while being reasonable on cost. This is especially true of Java J2EE projects, as most of the tools and applications are intensive, and the time it takes to build an entire application can be intensive.

Build Machines

If your project has any kind of continuous integration (and it should!) then you have probably felt the pain of long builds at some point. I’ve seen this on every project I’ve been on. There are two areas in particular that I’ve found to be painful: Long running regression or acceptance tests, and long compilation and deployment cycles due to heavyweight tools.

Often builds and tests are segmented into builds that are run locally on developer machines, and builds that are run by the build server. A typical approach is to have developers run unit tests and fast running integration tests locally while developing, but to have long running integration tests and acceptance tests run by a build server, where failures will be fixed later by developers when the build completes.

Many projects will find over time that the time it takes to run these large integration tests and acceptance tests becomes so long that the value is reduced. The time it takes to get feedback might be hours or even days. Often, these tests are failing, as by the time they complete, multiple developers or teams may made changes that break a portion of the test suite.

I’ve seen or read of different approaches to this problem, from using in-memory databases, to manually splitting the regression suites into separate builds or “pipelines”, to distributed computing, to transparent parallelization of tests.

An example of a new tool for transparently running tests in parallel is the Selenium Grid which attempts to run selenium tests in parallel. While I think there is merit in exploring these tools, they are non-trivial to setup and maintain, and while it may result in the build/test time being cut down to a fraction of it’s original time, it increases the complexity of the infrastructure that developers will have to maintain. There tends to be surprise issues with parallization as well. You have to make sure that you can have tests that are writing to the filesystem, querying a database, or calling other services in parrallel.

One day, I hope to try a different approach. I would rather spend the money trying to use hardware to solve the problem instead of using some complicated tool. From a previous experience of dealing with an incredibly IO bound build, I’ve long dreamt of building a hard drive out of RAM. I’m not talking about using flash memory; I’m talking about using DDR RAM instead of a traditional hard drive.

I recently looked into this concept, and I found that there are a few manufacturers out there that provide devices to do this very thing, such as the Hyper Drive 4. There are a few other devices out there that can achieve this, but I liked the information/propaganda on the Hyper Drive page the best.

The stats claimed by using a RAM based hard disk are nothing short of sexy.

I won’t reiterate the numbers here, but depending on the usage, it ranges from an order of magnitude to several orders of magnitude in increased performance. Even in builds and test suites that are not predominately IO bound, I am willing to bet that the performance boost to the operating system will translate into large gains for the performance of the tests. My favorite statistic was that Windows XP booted in 2 seconds with their test configuration, and that was only because of device polling.

I don’t think that the cost of such as system is unreasonable. One 16 gigabyte drive using the Hyper Drive system would probably cost around $5000. Assuming an average developer cost of $100 dollars an hour, it pays for itself if one week of 1 developers time is saved, let alone considering the benefits to an entire team. Come to think of it, I would argue that developers should have similar setups for their local machines. For instance, grep would probably be instantaneous with a DDR based drive.

In this example, you could test calculateClaimTotal by creating 3 claims with amounts = 1, and assert that the total is 3.

This may seem sufficient for a simple summation, but it becomes more flawed when the summation becomes based on business rules. Now, if the values to be summed are all 1, you can’t differentiate which rules were enforced by looking at the resulting number directly.

Now, we want to be able to differentiate that the active claims are the ones that get summed. This can become a bloated assertion to make. For instance, you can extend the test objects and count invocations on getAmount(). Then, you can assert that only the objects you expected where used to calculate the total.

A much simpler way of asserting that the objects that you expected were the ones that were used is to use amount values that, when summed, always have a distinct result. This is achieved by using numbers that map to distinct bitwise values:

00001 = 1
00010 = 2
00100 = 4
01000 = 8
10000 = 16

There is no way to add any of these numbers and have them be equal to any other summation of a subset of the numbers. This is because addition of 2 of these numbers is equivalent to an “or” operation on the bits of the numbers:

00001 = 1
00010 = 2
+__________
00011 = 3

This characteristic allows me to identify exactly what claims where used in the summation, because the resulting value could only occur from the values I expect:

Claim 1 is active with value = 1
Claim 2 is inactive with value = 2
Claim 3 is active with value = 4

I can assert that calculateClaimTotal will return 5 here, and that 5 will occur if and only if Claim 1 and Claim 3 are the ones used for the summation. Essentially, a bit location set to 1 instead of 0 will uniquely identify the object that the value came from.

I call this an old school testing technique because these days, there is not a lot of bitwise math that occurs in the course of programming. Back in the day, high performance C programmers would use bitwise operation often, and api’s such as the win32 api used bitwise math as part of api.

Even in today’s fancy world of high level abstraction, sometimes it takes a smooth low level trick like this one to achieve the most succinct code.