It’s housecleaning time again, and like last time, I stumbled across an article I wrote back in 2006 that I don’t believe ever reached publication (at least, I don’t think it did…how am I expected to remember what I did in 2006?). For the most part, I’ve left it in its original state, except that I removed the Agile Manifesto and 12 supporting principles. There are easily enough found on the Agile Manifesto website, and the article is long enough without this duplication. The wordle at right shows the most common words used in this document (click to enlarge). Here, in it’s otherwise unadulterated glory, is Agile: A New Era of Software Development.

Agile: A New Era of Software Development

Embrace Change

Writing code is easy, but developing software is hard. While syntax errors are common, their severity pales in comparison to the logic flaws inherent in many software systems. The difficulty in software development is not writing code or applying a certain technology stack. Instead, the challenge lies in the specification and design of the software itself.  Therein lies the essential complexity of software development, an idea introduced by Frederick Brooks in his 1987 article titled, “No Silver Bullet” [Brooks]. The most difficult aspect of software development is deciding what, exactly, needs to be built.

There is certainly evidence backing this claim. The original Chaos Report shows the top three impediments to a successful development effort are lack of user input, incomplete requirements and specifications, and changing requirements and specifications [CHAOS]. No other activity, if done incorrectly, stands to compromise the system more than incorrect requirement specifications.

It might not be so difficult were software a concrete entity, existing in a world where we could easily visualize it’s structure and behavior, allowing us to more reliably assess and share the impact of change. But software is a highly conceptual, invisible construct. It is considered infinitely malleable by those not intimately familiar with the conceptual complexity of it’s structure and behavior. The contractor building your home would look at you with incredulous disbelief if you suggested that the house he has 90% complete no longer met your needs, and you asked that he move walls. Or imagine how ridiculous it would sound to suggest that a new third floor be inserted to a 100 story skyscraper. Physicists labor on with firm belief that there exist an underlying set of unifying principles to serve as guidance. Or at least, there are laws of physics that we hold to be true. There are no such rules or principles that guide software development. We are left with the imagination and dreams of our clients, and they demand and deserve rapid response to change.

We have made valiant attempts at conformity. Ceremonial processes attempting to define standardized activities that guide the development process have failed, however. We cannot define detailed up-front requirements specifications and expect them to survive the development lifecycle intact. We cannot establish an initial design of the conceptual construct and expect the structure to go unscathed throughout the process of construction. Software development is an error prone human activity involving experts with varying backgrounds and skills who must come together and attempt to communicate uniformly, working as a team toward a common goal. While tools and process do help, we must also accept that change is expected. We cannot treat change as evil. Instead, the tools and process used must allow us to accommodate change, treating it as an inherent part of software development. Changing requirements is a rule of our game. The software we develop must be malleable and adaptive to change, and the process we use must embrace change.

We often draw comparisons between software development and various manufacturing processes. As Larman points out, however, manufacturing is a predictive process [Larman]. Herein lies one of the greatest differences between software development and the manufacturing processes to which we often draw comparisons. Manufacturing is a repeatable activity, with high rates of near-identical creation where a consistent product is produced in assembly line fashion. Little change is expected, making it possible to reliably estimate and predict the outcome. Software development is much more like new product development, where evolutionary specifications and adaptive planning is necessary to deal with the many unknowns that lie ahead.

Agile Principles

In early 2001, a small group of industry experts converged in Utah to discuss alternatives to heavy, document driven software development methods. Emerging from this meeting was the Agile Manifesto, a symbolic proclamation endorsing the virtues of a lighter, more flexible, people-oriented approach to software development, giving birth to the agile software development movement. (Since this is already a long article, I’ve snipped the manifesto and principles, which were included in the original version. If you’re interested, you can view the manifesto and its 12 principles on the Agile Manifesto website.)

<Snipped the Manifesto for Agile Software Development and 12 Principles>

The ideas behind these 12 principles are simple, and contain no hidden messages. Of course, there are different techniques embodied within various agile processes that support these principles. The one certainty is that agile teams definitely work differently from their less agile peers. They recognize there is one end goal - to create a working, functional software product. With that in mind, they work very closely with project stakeholders throughout the development lifecycle, knowing it is the stakeholders who possess the knowledge the system must embody. Agile teams work very hard to deliver working software iteratively and incrementally, and they adopt techniques representative of that ideal.

Agile project managers tend to favor intense communication and collaboration over heavy documentation. Empowering team members to make decisions enables responsiveness to change. Facilitating and negotiating requirements scope provides important feedback, helping plan future iterations, where each iteration produces a deliverable that can be shared with clients and stakeholders. Instead of forcing the team to follow a predictive project plan, agile project managers are more opportunistic. They prioritize features based on stakeholder feedback, and make adjustments as the iterations progress. Concurrent and parallel activities are favored over a phased approach. Agile project managers tend to guide the team instead of manage the team, and strongly discourage unnecessary overhead.

Agile developers work with a similar set of goals, knowing functional software must be delivered early and often. They work judiciously to grow a code base built upon a solid foundation, where each day represents a step forward. They integrate frequently, and do not tolerate failed builds. A rich suite of tests provide the courage necessary to respond to change when the need arises. They avoid the notion of code ownership, empowering other developers to make improvements to any component of the software product.

A common misconception is that agile processes discourage all documentation. This is untrue. Agile processes discourage unnecessary documentation, favoring collaboration as the preferred technique. Instead of using documentation to drive communication, agile processes favor face-to-face communication. Documents are encouraged by agile processes, so long as the need is immediate and significant.

Transitioning to Agile

Agile software development is based upon the fundamental premise that we must drive and respond to change quickly. The Agile Manifesto and 12 supporting principles serve this premise well. Advocates of agility claim speedier delivery of software, software with more business value, increased team productivity, higher quality systems, and a more enjoyable development experience. I believe each of these to hold true. Agile teams not only welcome change, they are able to respond to change at all levels of development. A project manager might discuss a changing requirement with a business client, empower a business analyst to schedule a meeting with the client to discuss further details, while a developer assesses the impact of change knowing she has the courage to accommodate the request because of the rich suite of unit tests in place.

Saying you’ll be more responsive to change and creating an environment that embraces change are separate beasts, however. Practicing agility is hard work, especially if your team is accustomed to more traditional approaches. As with many things new and unfamiliar, some resistance will no doubt arise by those who aren’t fully convinced. Agile projects differ greatly from their less agile counterparts, and skeptics will have many opportunities to express their discontent. As someone experimenting with agility, you may even have doubts. But don’t be discouraged, and give your agile transition the time it deserves.

One of the most significant changes you may experience is a feeling that you’ve been thrust into a chaotic nightmare. I doubt it’s unusual to feel this way. You’ve lost the security of detailed requirements specification and user sign-off. You are writing code without the comfort of knowing exactly what your stakeholders want. The detailed plans that have served as your security blanket on past projects no longer exist. And the celebrations accompanying completion of your various phase milestones are gone. Of course, these were all false comforts anyway. Stakeholders always changed their minds. Your detailed requirements and plans were outdated as quickly as they were completed.

Instead, you’re now working in shorter iterations with vague requirements. Initial releases early in the lifecycle may be completely thrown away. Your first few weeks may seem wasted, with little or no valuable artifacts produced. Naysayers will immediately come forward and cite the lack of progress. Previously, those first few weeks or months were spent producing very detailed requirement specifications and beautiful design models. But don’t give up yet. In that previous world, you were only delaying risk and postponing integration, avoiding the most difficult aspect of software development until the end of the lifecycle. Now you’re attacking risk early, prioritizing features, and working hard to develop a functional piece of software as early as possible. Progress may not be at breakneck speeds, but you are learning a tremendous amount about the requirements of the system, and your velocity is sustainable. Additionally, you are also performing a number of other important lifecycle activities.

Depending on the level of ceremony and bureaucracy within your organization, you will experience varying degrees of success when adopting agile techniques. As with any new technology adoption, it’s best to phase the transition. Some agile techniques are easier to adopt than others, and some serve as valuable catalysts to adopting additional techniques in the future. Don’t attempt to completely redefine how you work. It’s relatively easy to phase the agile transition, and you’ll want to adopt those principles that offer you the greatest initial reward.

For instance, if you’re struggling to produce quality software at a consistent rate, implementing a continuous integration strategy will help you frequently verify the quality of your work. In addition to the comfort of knowing you have a product always in a functional state, the ability to share the product with clients using functional demos and prototypes will tighten the feedback loop and offer valuable insight to the client’s perception of the software. In a number of cases, I’ve found this to be valuable in identifying subtle requirements that can be difficult to identify in other requirements elicitation venues.

Empirical Evidence

In recent years, there has been a significant amount of research comparing agile development methods to their waterfall counterpart. In Agile and Iterative Development: A Manager’s Guide, Craig Larman illustrates the advantage of agile development through detailed analysis of multiple studies[Larman]. The compilation of his results are illustrated below.

A study by Alan MacCormack at Harvard Business School explored whether evolutionary development techniques yielded better results than the waterfall model. The study included applications ranging from application software to embedded systems, with median values of nine developers spanning a 14 month development cycle.

A key conclusion of the study, in which 75% of participants used agile techniques compared to 25% using waterfall, explained releasing software earlier, rather than later, contributed to a lower defect rate and higher productivity. There was little evidence showing that a detailed design specification resulted in a lower defect rate, however, reviews with peers did help in reducing the rate of defects. The study found that iterative and agile practices have a significant impact on defect and productivity factors, as indicated by the following points.

• Releasing a system with 20% of the functionality complete is associated with a decrease in the defect rate of 10 defects per month per million lines of code as compared to waiting to release a product until 40% of the functionality is complete, and an increase in productivity of eight more lines of source code per person-day.
• Continuous Integration, the idea of integrating and testing code as it is released to your source code repository, resulted in a decrease in the defect rate of 13 defects per month per million lines of code, and an increase in productivity of 17 lines of source code per person-day.

The study also found four practices that were consistently used by the most successful development teams. The first two are deeply embedded in the ideals of agile software development.

• Releasing early and often to project stakeholders, using an iterative lifecycle.
• Continuous integration, with daily builds including regression testing.
• Teams with broad experience delivering multiple projects.
• Careful attention to modular and loosely coupled, componentized architectures.

In a separate study [Shine], 88% of organizations cited improved productivity when using agile methods, and 84% cited improved productivity. 49% stated that the cost of development was less when using agile methods. Additionally, 83% claimed increased business satisfaction and 26% claimed significantly better satisfaction. Another study by Boehm and Papaccio [Boehm] discovered that a typical project experiences a 25% change in requirements, while yet another [Johnson] showed  that 45% of features were never used.

There have also been many research efforts devoted exclusively to the analysis of waterfall methods. Below is a summary of these findings, taken from a variety of studies.

• Scope management related to detailed up-front requirements was a significant contributing factor of failure [Thomas].
• The U.S. Department of Defense (DoD), when following a waterfall lifecycle, experienced a 75% failure rate [Jarzombek]. This resulted in the DoD adopting a more iterative and agile approach.
• On a study including 400 waterfall projects, only 10% of the code was deployed. Only 20% of code deployed was used. The main factors included changing and misunderstood requirements [Cohen].

As these studies clearly illustrate, there is significant evidence showing that agile and iterative techniques offer significant advantages over the waterfall model of development. In fact, for larger projects, the statistics supporting agility were even more pronounced.

Conclusion

There are a variety of agile processes available to choose from, and each abide by the spirit of the manifesto and it’s 12 supporting principles. The agile movement and it’s supporters recognize that software development is a human (though not always humane) activity. Instead of forcing process on people, agile methods allow process conformance to people. Good people, working toward a common goal, can achieve great things will little ceremonial process, assuming you give them an environment that empowers them. Solid empirical evidence backs this claim. And if the quality of people is in question, it’s doubtful that any process will produce success.

References

• [Alliance]. The Agile Alliance. Manifesto for Agile Software Development. 2001. http://www.agilemanifesto.org
• [Boehm]. Boehm, B, and Papaccio, P. Understanding and Controlling Software Costs. IEEE Transaction on Software Engineering. October 1988.
• [Brooks]. Brooks, Frederick. No Silver Bullet: Essence and Accidents of Software Engineering. 1987.
• [CHAOS]. The Standish Group International, Inc. The CHAOS Report. 1995.
• [Cohen]. Cohen, D., Larson, G., and Ware, B. Improving Software Investments through Requirements Validation. IEEE 26th Software Engineering Workshop. 2001.
• [Jarzombek]. Jarzombek, J. The 5th Annual JAWS S3 Proceedings. 1999.
• [Johnson]. Johnson, J. Keynote speech, XP 2002, Sardinia, Italy. 2002.
• [Larman]. Larman, Craig. Agile and Iterative Development: A Manager’s Guide. Addison-Wesley, 2004.
• [MacCormack]. MacCormack, A. Product-Development Practices That Work. MIT Sloan Management Review. 2001.
• [Shine]. Corporate Report. Agile Methodologies Survey Results. Shine Technologies Pty Ltd. Victoria, Australia. 2003.
• [Thomas]. Thomas, M. IT Projects Sink or Swim. British Computer Society Review. 2001.

Story of the Concorde

On July 25th, 2000, flight 4590 crashed. It was the first, and only, crash of the famed Concorde. Eventually, it would lead to retirement for the amazing aircraft. Investigators spent countless hours poring over the wreckage, and placed blame on a piece of runway debris that slashed the tire. A piece of that tire struck one of the fuel tanks, causing it to rupture and the plane caught fire. Case closed, right? Not so fast.

Surely a small piece of runway debris shouldn’t bring down a commercial airliner! As it turns out, there is quite a bit of contention among experts surrounding other factors that may have contributed to the crash. Some argue that it was a complex chain of events, all coming together, that brought down the aircraft. The plane was missing a spacer between the two wheels, had too much fuel in the tank, attempted to takeoff in unstable conditions, and was overweight. The flight was also delayed, causing angst among the flight crew. Finally, a required daily runway inspection was not performed.

Perhaps, if the runway inspection had been performed, the piece of debris would have been spotted. Or had the aircraft not been overfueled, the piece of tire may not have caused an increase in fuel tank pressure that some say caused the tank to burst. Had the spacer been installed, it’s possible the tire would have never burst. A series of small events, each contributing in their own way, to the fatal crash.

No Matter How Small

The story reminds me of the importance of attention to detail in software development. Of how the aggregate of all of the small, seemingly insignificant decisions we make on a continuous basis can have long-term consequences on the future of the software system. Possibly even your organization. Every time you design a class, define a variable, write a test, create a package, build a module, modify a method, or make a design decision, you are affecting the future of your system in some unsuspecting way.

What may seem insignificant today, can have a detrimental affect tomorrow. Really, there are no small, insignificant decisions in software development. I’m reminded of how important it is to make conscious decisions that are given careful thought, no matter how small. It also reminded me of a poem on build automation and continuous integration that I read a while back on the Test Early blog.

FOR WANT OF A BUILD

For want of a build, a test case was not executed
For want of test case, a defect was not detected
For want of a defect report, a bad release was promoted
For want of a good release, a strategic customer was lost
For want of a customer, a development team was reduced
For want of developers, a product stagnated
For want of a product, a company was lost
And all for the want of a build…

I’ve been thinking a bit more about the list of disruptive technologies on Richard’s list, and then I watched Dan Pink’s TED talk on the surprising science of motivation. There must be some relationship between the list, which is comprised of almost all open source software products, and Dan’s assertion that the 20th century reward system won’t work for the cognitive tasks performed by workers in the 21st century. What is it, though?

We develop open source software to scratch a personal itch, ease the pain in performing a certain type of task, or create a more compelling alternative to a commercial product. While the professional open source model has emerged the last few years as a way for the open source community to create a sustainable business model, none of the open source technologies on the list were initially developed that way. They were developed in response to need. Ant because Java lacked a good build system. Spring because Java EE was cumbersome and bloated. JUnit to help increase quality. In many cases, these tools have grown to become defacto standard technologies widely used by enterprise development teams.

In Dan’s mind, the new incentive program within organizations must revolve around three things - autonomy, opportunity, and purpose. We must be given autonomy, or the empowerment to make our own decisions. We must be given the opportunity to master something that matters to us. And we must be given purpose, which is the desire to do what we do in the service of something larger than ourselves. Dan notes that financial incentive is also important, but is not the decisive factor in what motivates us.

There are a lot of ways to connect the dots between the open source products so prevalent on Richard’s list and Dan’s point about incentives. I’ll allow you the opportunity to connect these dots any way you wish. A few that immediately come to mind for me include the way commercial software is sold, corporate incentive programs, empowering developers, and flaws in corporate culture. But here’s something else to chew on.

In most of the cases, it was the developer who spurred adoption of the most disruptive application development technologies of the last decade. We weren’t motivated to develop or adopt these great products because of financial incentive or a reward system. Sure, in some cases that’s a positive side affect, but it is not the force that motivates. Instead, we were motivated because they make our jobs a little bit easier and our software a little bit better.

Developers are fighting like hell to create better software. While a lot of commercial vendors are selling shelfware (they aren’t selling it to the developer, mind you), developers are driving adoption of the technologies that are making a difference. Developers seek autonomy,  opportunity, and purpose. Given corporate culture, it’s not always easy to find. But if the last decade is any indication, we are finding it, developers do have a voice, and that voice is being heard.

Tight coupling between modules is a bad idea, and the worst form of coupling is cyclic dependencies between modules. Fortunately, there are a few techniques we can use to break the cycles. They are Callback, Escalation, and Demotion, and I’m going to walk through some examples that show each of them in action.

Then, once the dependencies are broken, we’ll look at two more techniques that allow us to invert and eliminate the relationship altogether. The code for each of the samples can be found in the edcie project on my Google code repository. Each example includes a build script and a simple test case. To execute them though, you’ll need GraphViz if you want to use JarAnalyzer. To invoke the build scripts without invoking JarAnalyzer, you can simply type:

ant compile

Keep in mind that each variation of the system has the exact same behavior!

The Example

The example we’re going to use to drive the remainder of our discussion is incredibly simple. We’ve got a Customer and a Bill class that we’re going to bundle into two separate modules - cust.jar and bill.jar. There’s also a test case called PaymentTest that serves as a sample client to drive the interactions between the two classes. The test case is bundled into the billtest.jar module. The initial class diagram is seen at right. Note the bi-directional relationship between the two classes.

As we progress, we’ll add more classes and abstractions to the system to help increase the modularity. Additionally, we’re going to use JarAnalyzer to illustrate the relationships between the modules and also help us assess the quality of our design. The module structure is below, as generated by JarAnalyzer. You can see how to use JarAnalyzer in the build by reviewing the build file. Again, our goal is to break the cyclic dependency between cust.jar and bill.jar, and we’re going to look at three different ways to do this before moving on to examine different ways to massage acyclic module relationships.

Initial Module Structure with Cyclic Dependencies

Escalation

The first technique we’re going to apply is called Escalation. With Escalation, we break the cyclic dependencies by escalating the cause of the dependency to a higher level entity. Before we do that, we need to more fully understand why a cyclic dependency exists in this example. This reason follows:

A Customer has a list of Bill instances. When the pay method on Bill is invoked, the Bill needs to determine if a discount should be applied. The discount is a product of the Customer the bill belongs to, not necessarily the Bill. Therefore, the Bill class calls a method on Customer to determine the appropriate discount amount. Think of it this way…The Customer represents a payee and we negotiate a discount with each payee. The calculation of this discounted amount is encapsulated within the Customer.

To break this dependency, we want to escalate the cause of the dependency up to a higher level class - the CustomerMediator. The mediator now encapsulates calculation of the discount and passes that to the bill class. The best way to see this change is to look at the modified PaymentTest class. Now, I’ve modified the build script and have bundled the mediator into it’s own module, as shown below. If you dig a bit more deeply into the class structure, you’ll wonder why I didn’t just pass the discount amount from Customer into Bill. Don’t worry about that. This example is slightly contrived because escalation isn’t the best way to solve this type of problem. The key takeaway is that we’ve escalated the dependency up to the mediator.jar bundle, breaking the cyclic dependency.

Escalating the Cause of the Cyclic Dependency

Demotion

A slightly better way to solve this particular type of cyclic dependency (where we have a true composite relationship between Customer and Bill) is to use demotion. With demotion, we push the cause of the dependency to a lower level module. Exactly the opposite of escalation. We do this by introducing a DiscountCalculator class that will be passed into the Bill class. Our modified PaymentTest class will create the calculator and pass it in for us. The Customer class will serve as the factory for the DiscountCalculator, since it’s the Customer that knows the discount that must be applied. The new class structure can be seen at right.

Now we’ll modify our build script to create an additional calc.jar bundle which will contain the DiscountCalculator class. Our resulting module structure is shown below.

Demoting the Cause of the Cyclic Dependency

Already you can see how this is a more natural solution than escalation for this particular type of cyclic dependency problem.  What’s the key difference you might ask? With escalation, notice how I would be able to deploy the cust.jar and bill.jar modules independently. While demotion is a more natural solution in this situation, it also means that to deploy bill.jar or cust.jar, I must also deploy calc.jar. The right solution is always going to be contextual and the ideal solution is likely to shift throughout the development lifecycle.

Callback

Using a Callback is similar to the Observer pattern. With this approach, we’ll refactor our DiscountCalculator class to an interface, and then modify the Customer class to implement this interface. This new class structure can be seen at right.

As it happens in this specific situation, using a Callback represents a combination of demotion and our initial solution. We’ll go back to passing the Customer into the Bill, but will pass it in as a DiscountCalculator type. Whereas in the Demotion example we bundled the DiscountCalculator in a separate module, we’ll now just include it in our bill.jar module. Note that putting the DiscountCalculator in the cust.jar module would introduce the cyclic dependency we’re trying to get rid of. The new module structure, which resembles our original version minus the cyclic dependency, is shown below.

Using a Callback to Eliminate the Cyclic Dependency

Inverting Relationships

Now we’re going to play around a bit with the module relationships. While Callback seems like the most logical solution, what if we wanted to use the cust.jar module without the bill.jar module? Callback, as it’s implemented, doesn’t allow us to do this. But with a bit of trickery, I can actually invert the relationship between the cust.jar and bill.jar modules.

I start by refactoring the Bill class to an interface. Then, to avoid split packages (where classes in the same package are bundled into separate modules), I move the Bill class into the same package as the Customer class. The new class diagram is shown at right, and the inverted module structure is shown below.

Inverted Module Structure

Eliminating Relationships

Inverting the relationships allows us to deploy the cust.jar module independent of the bill.jar module. Again, it’s all about need. But I’d like to explore another option based on another important need - the ability to test modules independently. Before inverting the relationships, I am able to test the bill.jar module independently. After inverting the relationships, I can test the cust.jar module independently. But what if I want to test (or deploy) both modules independently? To do this, I need to completely eliminate the relationship altogether.

As it turns out, because I’ve got a pretty flexible class structure after I inverted the relationships (lot’s of abstract coupling), I can do this by simply bundling the two interfaces - Bill and DiscountCalculator - into a separate module. No other coding changes required. I start by moving them to a new package called base. Then, I modify my build script to bundle these two interfaces into a separate base.jar module, and we have successfully eliminated the relationship between the bill.jar and cust.jar modules, as shown below.

Eliminating Relationships Between Modules

Wrapping Up

We’ve come a long ways from our original version of the system. Two modules with a cyclic relationship to a module structure where the original modules don’t have any relationship to each other. This means these modules can be tested and deployed independently. If you follow this blog, you know I’ve written plenty about the tradeoffs between flexibility and complexity, use and reuse, and many other architectural and design challenges. I hope this little exercise has helped drive some of those concepts home.

As a final note, to explore some of these design decisions a bit more deeply, and to examine the tradeoffs a bit more objectively, I encourage you to run the builds for each of the projects and examine the dependencies.html file in the stats directory. To do this, you’ll need to make sure JarAnalyzer is executed, which requires GraphViz. As you’ll see when comparing the initial version to the final version, we made considerable progress in improving the quality of the design.

In Turtles and Architecture, I talked about how important it is that we “architect all the way down”. It helps increase transparency and understanding between developers and architects by emphasizing a lot of the middle ground that noone ever seems to focus on. It’s just another reason why modularity is so important. I used the diagram at right to illustrate the point (click to enlarge). Look at the huge gap that exists if we don’t focus on the stuff highlighted by the gray bar in the middle.

One reason I like this visual is that it illustrates the social aspect of software architecture. Yet, there are other significant advantages to architecture all the way down that we haven’t explored yet. Another is structural flexibility.

Structural Flexibility - Different Entities, Different Purpose

Another benefit of module design in filling that middle ground is that modules can offer different benefits than classes and services. The diagram at right (click to enlarge) illustrates some of the capabilities of different types of entities.

For example, classes are pretty easily reused within an application, but because classes aren’t a unit of deployment, it’s difficult to use them across applications. Of course, intra-application reuse is a sweet spot of services, since they’re method of invocation is through some distributed protocol (SOAP, HTTP, even RMI/IIOP). Yet because services are invoked remotely, we typically like to make them coarser-grained because of the performance implications of distributed requests. So this begs the question - If a service is too coarse-grained to reuse (ie. it does more than what we want), how do I reuse some desirable behavior across applications? Well, without modules, our only other choice is the class. Since a class can’t be reused intra-process, we do one of either two things. Expose it as a service or copy the class (ie. the code). Given the circumstances, neither of these may be ideal. Another option is desirable.

Modules represent that other option. They are a finer level unit of granularity than services, and are a unit of deployment. Since each of these different types of entities are units of composition, we have tremendous flexibility in how we assemble applications. Possibly more important though is our increased ability to accommodate the architectural shifts that naturally occur as requirements evolve. Let’s look at a more concrete example.

A Bit More Concretely

Let’s say I have a business function called Pay Bill for which I develop a web service that can be invoked by many different consumers. That service happens to be relatively coarse-grained, and performs all the steps involved in paying the bill. These happen to include the following:

• audit bill - apply a discount to the bill based on payee
• check for duplicate - ensure the bill hasn’t already been paid
• remit payment - cut the check
• reconcile payment - reconcile with accounts payable financials

This seems reasonable. We have a nice little service that we can reuse any time we want to pay a bill. Unfortunately, the real world often gets in the way of our idealistic solutions. In fact, there are two problems that will eventually surface, and modularity benefits both scenarios. Let’s start by looking at the first scenario.

What should I do when a different scenario arises that demands I follow a slightly modified Pay Bill function?

As part of the remit step, I have a new payee that demands electronic payment. This is pretty easy actually. I simply modify the service to support electronic payments and then configure the service to context for that specific payee. So how does modularity help here?

If the service is composed of modules, it’s going to be much easier for me to understand the structure of the service, assess the impact of what’s it’s going to cost to change the service, and then introduce a new module (or modify the existing module) to provide the new capability. Without modules, I’m simply wading through the code trying to figure out all of these things. Now, the 2nd scenario.

What should I do when I want to reuse just one step of the Pay Bill function?

Let’s say another new requirement emerges. Whereas traditionally bills were entered by data entry personnel, we now have to support electronic delivery of bills. We also know that bills delivered electronically are often duplicates. It’s just one of those business things, you know? If we don’t pay the bill on the day it’s received, the billing party sends us the bill again, asking for payment. So we need to check for duplicates before we save the bill to the database and prepare it for processing. What do we do?

We could take the same approach as before and modify the Pay Bill service so that the duplicate check could be invoked separately from the higher level pay bill function. But that’s a bastardized design solution. Why? We are exposing behavior of the Pay Bill service that shouldn’t be exposed. The API is coarse-grained in some areas and fine-grained in others.

Maybe exposing the finer-grained capabilities of the Pay Bill function isn’t a severe compromise, but it is a compromise nonetheless. And we are forced to compromise because of architectural inflexibility. We don’t have architecture all the way down, and are therefore left with limited choice as to how we support the new requirement. We can either modify the service to reuse what we know is already encapsulated within the service, or copy the code to create something new. But those are the two options we have, and neither may be ideal.

As we continue to hack the system in this manner, with a multitude of other similar changes, the design will rot. Eventually, our Pay Bill service is transformed into a utility that performs all general-purpose bill functions. It’s API is a mix of coarse-grained and fine-grained operations. It’s become the monolith that we’re trying to avoid. While the Pay Bill service is still pretty reusable (it does everything now), it isn’t that usable. That tension between reuse and use surfaces again.

Our decision to modify the Pay Bill service to expose the duplicate check was driven by one thing - ease. It was our easiest option. Really, it was our only option. But it isn’t the best option.

If we architect all the way down, we have another option. A Pay Bill service composed of modules that audit the bill, check for duplicates, remit payment, and reconcile the payment means we can choose the desirable solution over the easiest short term choice. We can avoid the long term compromises that degrade architectural integrity.

Shown at right (click to enlarge), we see the service composed of modules. If we have a check for duplicates module, we can simply reuse that module in the new service or application that’s going to process electronic bills. Or we might expose the capabilities of the check for duplicates module as a separate service. We have multiple reuse entry points, as well as different levels of granularity through which our software entities can be managed, deployed, built, and much more.

To Conclude

My point here isn’t to debate the original design nor the decisions made to reuse the check for duplicates functionality in another service or application. Certainly, debating architectural and design decisions is a favorite past-time of developers. There are a variety of different ways that we can support new requirements as they emerge. Some are better than others and all are contextual.

The gist of my point is that architecture all the way down gives us options, and these options help maintain architectural integrity. They increase our options when making decisions and allow our system to accommodate unforeseen architectural shifts. I can modify an existing application or service to give me what I need. Or I can reuse an existing module that’s already available for me. Or I can compose a new service from an existing set of modules. Or I can break apart existing modules to create new modules that result in new services. And I can do a lot of this refactoring without significant impact on the existing code. This was an important point illustrated in the series of posts on Applied Modularity.

In other words, architecture all the way down helps increase architectural agility, and modularity is a key ingredient.

All too often, software process improvement initiatives fail. In a recent post discussing SEMAT, Ralph Johnson provided some words of wisdom that serve as a wonderful guide to any team about to embark on that much vaunted software process improvement initiative.

The state of the practice in software development is pretty dismal. Some groups do a great job, but most do not.  As I tell the students in my software engineering course, if you manage requirements, make sure the developers talk to each other, release working code regularly, have some sort of a systematic testing process, use build and version control tools, and periodically stop and see how you are doing and how you can improve, you will be better than 90% of the groups out there. Of course, I could be exaggerating.  Maybe it is only better than 75%.

I suppose that pretty much sums it up! Amazing how difficult we tend to make things though, heh?

Here’s my Agile Architecture - Patterns & Technology slide deck (bottom of this post) that I presented at SpringOne2GX and OOPSLA. Thank you to all who attended, and for providing such positive feedback. I’ve also uploaded the code samples used to show the modularity pattern examples found toward the end of the presentation. The code can be found under the billpayevolution project in my Google code repository.

A Bit More on the Code

The project is named billpay, and is a very simple system. It starts out in life as a project that’s bundled and deployed using a single WAR file (InitialVersion). Through a number of refactorings (7 in total), the project is broken out into separate modules, and the final version of the system (Refactoring7ImplementationFactory) has a number of modules (JAR files) that are included in the WEB-INF/lib of the WAR.

I intentionally avoided using OSGi for this exercise, primarily to show how we can modularize an application without a runtime module system. I do have an OSGi version of the system that can be deployed to an OSGi runtime (Tomcat using the Equinox Servlet Bridge is what I used), but I haven’t uploaded that code to the repository yet because I have a few things I need to change (like using Spring DM to remove the code dependencies on the OSGi API). That’ll come later, after I’ve had a chance to do a bit more work with it.

Each project along the path toward refactoring the application into modules comes with it’s own build script. For each project, except for the final version (Refactoring7ImplementationFactory), all you’ll need to execute the build is Ant and JUnit.

A Note on JarAnalyzer

Because I used JarAnalyzer in the final refactoring, you’ll need to make sure you have GraphViz installed and then modify the build script to point to the dot executable. Of course, you can also comment out the JarAnalyzer target, too. But JarAnalyzer was very helpful in identifying and managing the dependencies between modules, so I think you’ll find it useful to keep it. If you’re not familar with JarAnalyzer, it produces visual output showing the dependencies among JAR files (shown at right, click to enlarge), as well as an HTML report with some of the Martin metrics. Here’s the JarAnalyzer HTML report for the final version of billpay.

Like with any project, there are a few things I’d like to change. For example, in Refactoring7ImplementationFactory, I’d like to use Spring to inject the implementation classes instead of using the factory. Sometime soon, I hope to walk through these samples in a series of blog entries that shows how modularity can have an amazingly positive influence on the architecture of a software system. Until I do this, or you attend one of my presentations (btw, I’m speaking next week at Agile Development Practices), it may be a tad difficult to understand the forces behind my decisions (hint: they were driven by various business requirements). So please be patient, and stay tuned.

And Now the Presentation

Here’s the presentation, hosted on SlideShare. I look forward to your questions and/or any feedback you might have!

The story of software architecture reminds me of the following story.

“A well-known scientist (some say it was Bertrand Russell) once gave a public lecture on astronomy. He described how the earth orbits around the sun and how the sun, in turn, orbits around the center of a vast collection of stars called our galaxy. At the end of the lecture, a little old lady at the back of the room got up and said: “What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise.” The scientist gave a superior smile before replying, “What is the tortoise standing on?” “You’re very clever, young man, very clever”, said the old lady. “But it’s turtles all the way down!”

- A Brief History of Time by Stephen Hawking

Software architecture is “‘turtles all the way down.”

The Ivory Tower

Many of us can relate. In dysfunctional organizations, architects and developers fail to communicate effectively. The result is a lack of transparency and a lack of understanding by both sides, as shown in the diagrm (click to enlarge). The failure often occurs (though I recognize there are other causes) because architecture is about breadth and development is about depth.  Each group has disparate views of software architecture, and while both are warranted, a gap between these views exists. The architect might focus on applications and services while the developer focuses on the code. Sadly, there is a lot in between that nobody is focused on. It is this gap between breadth and depth that contributes to ivory tower architecture.

Turtles and The Tower

Without question, the ivory tower is dysfunctional (regardless of cause), and systems lacking architectural integrity are a symptom of ivory tower architecture. So assuming good intent on the part of the architect and the developer, how can we bridge the gap between breadth and depth? How can we communicate more effectively? How do we increase understanding and transparency?

My favorite definition of software architecture was offered by Ralph Johnson in an article by Martin Fowler. He states:

In most successful software projects, the expert developers working on that project have a shared understanding of the system design. This shared understanding is called ‘architecture.’ This understanding includes how the system is divided into components and how the components interact through interfaces. These components are usually composed of smaller components, but the architecture only includes the components and interfaces that are understood by all the developers…Architecture is about the important stuff. Whatever that is.

The key aspect of this definition that differentiates it from many other definitions of architecture is that of “shared understanding.” We must have a shared understanding of how the system is divided into components, and how they interact. Architecture isn’t just some technical concept, but also a social construct. And it is through this that we can break down the divide between architects and developers.

To ensure shared understanding, we have to architect “all the way down.” Architects cannot worry only about services and developers cannot worry only about code. There is a huge middle ground that each must also focus on, as illustrated by the diagram at right (click to enlarge).

Focusing exclusively on top level abstractions is not enough. Emphasizing only code quality is not enough either. We must bridge the gap through other means, including module and package design. Often times, when I speak, I ask the audience to raise their hands if they spend time on service design. Most do. I also ask them to raise their hand if they spend time on class design and code quality. Again, most do. But then I ask if they also spend time on package and module design. Usually, only a small percentage leave their hands raised.

This is unfortunate, because module and package design are equally as important as service and class design. But somewhere along the way, with our emphasis on services and code quality, we’ve lost sight of what lies in between. Within each application or service awaits a rotting design, and atop even the most flexible code sits a suite of applications or services riddled with duplication and lack of understanding. A resilient package structure and corresponding software modules help bridge the divide between services and code.

Certainly, if I’ve said it once, I’ve said it at least a few times, we need to start focusing on modularity to ensure a consistent architecture story is told. It is the glue that binds. It’s the piece that helps bridge low level class design with higher level service design. It’s the piece that helps bring down the ivory tower, enhance communication, increase transparency, ensure understanding, and verify consistency at multiple levels. It is the piece that allows us to “architect all the way down.”

For those of you interested in Lean and Kanban, but not much knowledge yet, it can be tough filter the noise and find information explaining what they are and how they might be different from what you’re already doing (agile, Scrum, etc.). Since top 10 lists are rather passé, I’ve gone the whole nine yards and scoured the blogosphere to come up with the best nine posts I could find that capture the essence of Lean and Kanban.

The list is in no particular order, and there’s no doubt that I’ve missed some great posts that offer insight. The blogosphere is a pretty big place, after all. Naturally, I’ve cheated a bit by pointing to related posts from within the top nine. Enjoy!

• The Essence of Lean and Kanban - Post by @alshalloway that talks about Flow.
  
• Patterns and Practices of Lean Software Development - Post by @corey_ladas discussing principles such as One Piece Flow, Real Options, and Build Quality In.
• Noodling on Kanban - The first of a four-part series written by @mcottmeyer. Also be sure to check out Part 2, Part 3, and Part 4.
• How Is Kanban Different From Other Approaches - Post by Karl Scotland that examines Kanban’s primary practices. Also check out his follow-up post where he adds The Fifth Primary Practice of Kanban. And his posts on Kanban, Flow, and Cadence and XP as a Kanban System are great, as well.
• Kanban Applied to Software Development: From Agile to Lean - A detailed article on InfoQ that begins by exploring the origins of Kanban in the Toyota Production System, before launching into a discussion on Kanban in software development.
• Kanban in Action - An early post by @agilemanager (David Anderson) where he describes his initial forays into Kanban.
• I Have Heard of Kanban - A response to the critical post, Have you heard of Kanban, by Matthew Huesser. There were actually a few posts that were part of this conversation. It started with @chris_mcmahon calling into question Kanban with his post, against kanban. That led to Matthew’s post, and ultimately the response. Matthew has since posted a Kanban Redux and Chris has posted a couple of responses in against kanban part 2 and against kanban part 3.
• Bumping into Scrum’s Boundaries - Part 1 of @damonpoole’s Transitioning from Scrum to Kanban Series. Part 2 can be found in his post on One Piece Flow.
• The Poppendieck’s on Lean - An interview with Mary and Tom Poppendieck on Lean.

If you’re interested in learning more about Kanban, consider joining the Kanban Development Yahoo Group. Additionally, the Limited WIP Society is the home of Kanban Development, and contains some good definitions of what Kanban means within software development. And of course, the best place to learn more about Lean is to start with the Poppendiecks.

Note: Graphic courtesy of InfoQ. Used without permission.

Next Page →