Java With a Bit of OSGi – The Book

I’m dancing. By god I’m dancing on the walls. I’m dancing on the ceiling. I’m ecstatic. I’m overjoyed. I’m really, really pleased.
– An excerpt from the Foreword by Uncle Bob (aka. Robert C. Martin)

My book, Java Application Architecture: Modularity Patterns With Examples Using OSGi is now available. Uncle Bob and Peter Kriens each contributed Forewords to the book. The book itself is part of the Robert C. Martin series. The book is intended for all software developers interested in designing better software using modularity. Though the examples use Java, the techniques can be applied to other languages and platforms, such as .NET, with relative ease.

Even if you’re not using OSGi (or perhaps not even familiar with OSGi), I’m confident you’ll find the book valuable. The book (and patterns) has been designed to allow you to realize the benefits of modularity whether you’re using a module framework, or not. As Uncle Bob says in the Foreword, “This is how you build a Java application, people.” Peter sums it up nicely too, in saying “This book…will give you a view into the magic of modularity.

You can order it online at Amazon (print edition & Kindle edition), iBooks, InformIT, or a number of other publishers. For more details on the book, please see the book’s website. Over the next couple of weeks, I plan to post a sample chapter or two that will give you a feel for the book’s contents.

Here’s what a few people have to say:

  • “The fundamentals never go out of style, and in this book Kirk returns us to the fundamentals of architecting economically interesting software-intensive systems of quality. You’ll find this work to be well-written, timely, and full of pragmatic ideas.” Grady Booch, IBM Fellow
  • “Along with GOF ‘Design Patterns’ – ‘Java Application Architecture’ is a must own for every enterprise developer and architect, and on the required reading list for all Paremus engineers.” — Richard Nicholson, Paremus CEO & President of the OSGi Alliance
  • “In writing this book, Kirk has done the software community a great service: he’s captured much of the received wisdom about modularity in a form which can be understood by newcomers, taught in Computer Science courses, and referred to by experienced programmers. I hope this book finds the wide audience it deserves.” Glyn Normington, Eclipse Virgo Project Lead
  • “Our industry needs to start thinking in terms of modules – it needs this book!” Chris Chedgey, Founder and CEO of Structure 101
  • “In this book Kirk Knoernschild provides us with the design patterns we need to make modular software development work in the real world. While it’s true that modularity can help us manage complexity and create more maintainable software, there’s no free lunch. If you want to achieve the benefits modularity has to offer, buy this book.” Patrick Paulin, Consultant and Trainer at Modular Mind
  • “Kirk has expertly documented the best practices for using OSGi and Eclipse runtime technology.  A book any senior Java developer needs to read to better understand how to create great software.’ Mike Milinkovich, Executive Director Eclipse Foundation
I’d like to thank all of you who helped me along this journey. I hope you enjoy the book.

Farewell Blog, Hello POMA

As the saying goes…all good things must end. As of today, my blog is shutting down!

I do intend to leave all content on-line so you’ll always be able to see a list of all 134 posts, but there won’t be any new posts for the foreseeable future. I’d like to thank everyone that has taken the time to read my often times long-winded posts. I hope you’ve found them useful.

For the past few years, I’ve written almost exclusively about modularity and OSGi (with a few other topics sprinkled in on occasion). To that end, I’ve decided to pursue a long overdue book project, tentatively titled Patterns of Modular Architecture (POMA). I hope you’ll decide to join me in my journey. Also, be sure to follow the book updates on Twitter.

So as to not disappoint those who have come to expect something a bit more verbose, I have managed to cobble together a few words explaining my decision in more detail. So if you want to know more, read on!

Story Behind the Story

Over the past few years of blogging, I’ve focused considerable energy on espousing the virtues of modularity and OSGi. From conceptual posts that discuss the important architectural benefits of modularity to simple examples that illustrate the benefit. From the beginning to the end, I’ve written about the benefits of modular architecture at a conceptual level, practical level, and have even provided some concrete examples. To the chagrin of some, I’ve even discussed the challenges that lie ahead. Of course, modularity and OSGi aren’t the only topics I write about. I’ve also written about agility, IT labor, metrics, and more. At some point though, the topics always came back to modularity and OSGi.

In the time that I’ve been focused on modularity and OSGi, the number of folks that access the content on this site have gone from less than 1000 visitors per month to more than 10,000, culminating in more than 100,000 pages served up and over a quarter million hits to the site each month. The blog was listed as one of the Top 200 Blogs for Developers, a Top Analyst Blog, and many of my posts are syndicated on JavaLobby. While still small by many standards, I recognized progress. And in general, the feedback I’ve received from the community is positive, though there is no doubt that at times, I’ve struck some nerves.

For those that read my posts, you already know that most tend to span many paragraphs. They require extensive writing time, considerable editing time, and careful review. It’s taken me hours to author many of them. And all of the content on this site has always been licensed under a Creative Commons Attribution-Noncommercial-Share Alike license. I vowed to never include any noise on the site, such as Google AdSense. It’s always been an earnest attempt to help developers improve the systems they create. A labor of love, you might say.

A while back, I introduced the modularity patterns, but didn’t elaborate much on the patterns themselves. Alas, that time has come. I’ve quickly realized that I do not have the bandwidth to continue with my long-winded posts and write a book. I’ve decided to pursue the latter and am hopeful that a book on modular architecture will have a far greater, longer lasting impact that serves as a capable ambassador for modular architecture. So while my blog may be silent, I have every intent to continue my advocacy of modular architecture, albeit in a way that I hope is more impactful.

I’ve decided to adopt an open and transparent approach to writing the book. The book’s website illustrates the book in it’s current form – a rough and incomplete draft to be sure. But I hope you’ll take the time to check it out, and offer any feedback you might have. I’ve created a Reviewers page that provides some guidance on the type of feedback I hope to receive. Also, be sure to follow the updates on Twitter.

By the way, this does not mean that I’ll necessarily cease my blogging activities altogether. In the past, I’ve published some articles to the APS Blog. Soon, the APS blog will also be shut down, and we’ll begin to blog on the Gartner Blog Network (GBN). So be sure to checkout GBN on occasion!

Thank you!

The Exciting Enterprise

Developing enterprise software is hard work. Bureaucracy. Large teams. Legacy systems. The list of challenges is immense. But enterprise development can also offer tremendous opportunity and reward. It can even be fun! I hope that at some point, you have the opportunity to experience this feeling.

The Exciting Enterprise

I worked here once. In the exciting enterprise. Working here was different. It was actually exciting. What made it different? Well, it was the process. Exciting process? What’s that? It’s the process that makes software development fun again, productive, meaningful, and satisfying.

So what kind of process is this, you ask? Iterative? Agile? RUP? XP? Scrum? Kanban? Hell no! We never talked process lingo. We didn’t care how agile people thought we were. We didn’t want to get caught up in the bureaucracy and political mayhem surrounding software process improvement. There was work to be done. That’s what we talked about. And then we did what we needed to do to get it done.

Not Always Easy

Working in the exciting enterprise wasn’t always easy though. It required stamina, determination, and discipline. There were lots of people who didn’t appreciate how we went about our business. We didn’t submit weekly status reports. We didn’t have the Gantt chart. Actually, we didn’t have a lot of things. But we had what we needed. To us, it all seemed like common sense. Of course, we had a very experienced team, and we knew what worked and what didn’t.

Plan driven? Predictive? Estimates? Models? Oh yeah…we had it all. We had a 18 month project road map. It was quite a plan. Took a small group of us about 1.5 days to develop. All on a single spreadsheet. Showed all the major systems that we’d retire right along with the new functionality that was going to come online. Probably could have gotten another three months of project time if we wanted to do more planning. But we had code to write.

Oh That Build

Code? But what about the requirements? Yep. Gathered them as we went. And when developers had questions, they asked them. Right to the customers face too. Blasphemy! The business analyst working with the developers. Those BAs were awesome too. They worked real hard to clear any confusion. Made sure developers always had what they needed. And we got ’em as quickly as we could handle them. A steady stream of requirements flowing in and right back out as an executable piece of software.

Oh sure, it wasn’t always easy. We had lots of important checkpoints along the way to make sure we were on the right track. Weekly checkpoints. Daily checkpoints. Hourly checkpoints. Developers and customers sitting in a room together to see what we’d gotten done the past week. Continuous deployment to an environment where folks could actually use the software. Hourly builds that made sure we never strayed too far from a working system.

I still think about that build. Actually, it was more than a build. It was a piece of frickin’ art. That build was the coolest piece of process I’ve ever seen. It was staged. It was fast. It did pretty much anything we asked it to do. It never got in our way. It just went on it’s merry way, building our system. Hourly. Every hour. Automatically. It was the glue that held the team together as we grew in size from a fledgling crew of six developers to upwards of 100 at times.

We protected that build. When somebody cause it to fail, they’d feel the wrath. Eventually, they all grew to love what it could do. Honestly, what choice did we have if we wanted to ensure we could have weekly checkpoints throughout the process? The only way to pull it off was to emphasize software that works. It always had to work. ALWAYS! Guard the source. The prime directive – software that works!

The Source

Now don’t get me wrong. We had documentation. We had lots of documentation. Some of it was pretty nice looking stuff too. But we weren’t afraid to let it go. Let it serve it’s purpose and then move onto what was real important…the source code. What else really matters, anyway? All the pretty documents, models and plans don’t amount to a hill of beans if the source code isn’t provably correct. That was our focus. It was the teams focus. And we guarded that source code very closely.

Improving the Right Process

I look back now. An 18 month project. A team that clicked. Not just a development team. A team. Including the customers. Honestly, we didn’t just build a piece of software, though. We improved the process. We automated it. We optimized it. No, I’m not talking about the software process. I mean the business process! That’s what made it so cool…

Granularity: Architecture’s Nemesis

Granularity is the extent to which a system is broken down into it’s behavioral entities. Coarse-grained entities tend to be richer in behavior than fine-grained entities. Because coarse-grained entities do more, they tend to be larger than fine-grained entities, and are easier to use. In general, a fine-grained entity is more reusable than a coarse-grained entity because it does a little bit less. This is pretty natural. If it does less, then it’s more likely to apply across a wider variety of usage scenarios. I spent some time discussing this tension in Reuse: Is the Dream Dead?

Unfortunately, while there are a lot of patterns, principles, and idioms that offer really good guidance surrounding many aspects of architecture and design, there isn’t a lot of guidance surrounding granularity. Let’s start with a really simple example that illustrates the challenge.

A Really Simple Example

Consider the following save method:

public class Person {
   private String firstName;
   private String lastName;
   private String ssNumber;
   public void save() {
      ValidationErrors errors = this.validate();
      if (errors != null) {
         //handle the errors somehow
      } else {
         //save to the database.
   private ValidationErrors validate() {
      //perform validation

As can be seen, the save method invokes the validate method before saving the information to the database. Generally speaking, this makes saving the information easier (and possibly safer) since whatever invokes save doesn’t have to worry about validating. And this seems to be a good thing.

Yet, what happens when a new consumer of this method wants to save the information but apply a slightly different set of rules than the current validation method provides? Here’s where the existing save method is simply too coarse-grained. It does just a little bit too much. We’ve made it easier to use, but also less reusable.

There are quite a few variations I could go with at this point to improve the design. One would be to make the validate method public, and have clients invoke the validate method before invoking save. Then, if validate didn’t do what it was supposed to, clients could always opt out of invoking the validate method and apply their own validation before invoking save. Certainly less safe though because validation is optional.

Yet another alternative might be the following:

public class Person {
   private String firstName;
   private String lastName;
   private String ssNumber;
   public void save(Validator validator) {
      ValidationErrors errors = validator.validate();
      if (errors != null) {
         //handle the errors somehow
      } else {
         //save to the database.

This seems to offer the best of both worlds. I now require a Validator be passed in before I can save, and if the Validator is an interface, I can certainly allow clients to pass in the Validator they need. Of course, I could easily pass in a NOP validator.

Whatever…there are lots of different options, and each has their own set of tradeoffs. I’d like to avoid extensive debate surrounding which approach to save and validation is best and instead focus on the main point here, which is that achieving the right level of granularity for an entity can be very challenging. It requires more than just looking at the problem from a code level viewpoint and demands that we possess contextual information that will help us answer the question, “What’s the right level of granularity?”

Bring It Up a Level

The code above was a pretty simple example that illustrates the challenges of granularity at the method level. But the same challenges exist when developing classes, packages, modules and services. I discussed another, much higher level example, in the post on architecture all the way down. So what I really want to know is, “How do I determine the appropriate level of granularity?

This is the million dollar question. Because granularity is a significant inhibitor to creating reusable software that’s easy to use. (Oh, and managing dependencies too.) If something does too much, it’s less reusable. If something does too little, it’s more difficult to reuse.

In the past, I’ve used the diagram at right (click to enlarge) to help illustrate one view of granularity. As can be seen, services are more coarse-grained than modules which are more coarse-grained than packages which in turn are slightly more coarse-grained than classes. This begins to help answer the question, “What’s the right level of granularity?”

If I’m concerned that a service I create is simply too coarse-grained and fails to maximize its reuse potential, I can break the behaviors of the service out into modules that are a bit finer grained and more reusable (of course, one might consider doing this in general). Then I can compose the services from the modules and reuse the modules across services. The result is different entities at different levels of granularity that lends tremendous flexibility to how I compose, use, and reuse software entities.

This provides some guidance on the level of granularity at which different types of software entities should be defined. However, it still doesn’t offer enough guidance to determine the right level of granularity for the save method in our example above.

Another Dimension

There’s another dimension that is relevant, and that ties in nicely (though in a rather subtle way) to the initial coding example. It has to do with the way we traditionally layer our software. The diagram at right illustrates this (click to enlarge). As we can see, as we move down the layered hierarchy, entities become finer grained.

While this isn’t absolute, and depends on your architectural principles and style, entities in higher level layers tend to be more coarse-grained than entities in lower level layers because they are an amalgamation of their own behavior and the behavior of entities in the lower level layers.

Armed with this information, we can determine the right level of granularity for the save method as long as we understand in which layer the code containing the save method lives. If it’s in the data access layer, and we place architectural constraints on business rules living in the data access layer, then we shouldn’t have any validation code that lives in the data access layer. If the save method lives in a class in the domain layer, however, it may be suitable for the save method to contain validation code in the domain layer and invoke another method in the data access layer that actually performs the save operation.

The Complete Picture

At this point we can make the following general statements:

  • Entities in higher level layers are more coarse-grained than entities in lower level layers.
  • Services are coarser than modules which are coarser than packages which are coarser than classes.

And if we combine these two ideas into a single thought, and overlay the two diagrams, we can begin to visualize the level of granularity for different types of software entities.

Certainly this doesn’t offer hard guidance. Realistically, there are very few architecture and design principles and patterns that are universally applicable. In fact, I’m not certain you’d ever create a “presentation service.” But certainly you might create a data service that is composed of multiple data access modules. And you might have separate data services that reuse a data access module. And you might have a business service that uses a data service and is also composed of multiple domain modules. But you definitely do not want a data access module that invokes a business service, nor a business service that references a presentation module.

So this general guidance can serve as useful information to help answer our question, “How do I determine the appropriate level of granularity for a software entity?” And it can also serve as guidance when establishing architectural principles and constraints that help determine where software entities with specific behaviors should reside. Because without some scheme that will help determine the appropriate level of granularity, it’s hopeless to imagine that we’ll be able to design reusable software entities that are also easy to use.

Complexity: Essence and Accidents

I’d like to take a short moment to offer an additional perspective to my discussion on OSGi: Complexity is NOT the Problem. I believe this perspective adds clarity to that previous discussion, as well. All initiated thanks to a tweet, which summed up the situation in much less than 140 characters.

So over the weekend, I turned to the essay that said tweet refers, and reviewed the essence and the accidents. From the article, I quote Mr. Brooks:

“…to see what rate of progress we can expect in software technology, let us examine its difficulties. Following Aristotle, I divide them into essence – the difficulties inherent in the nature of the software – and accidents – those difficulties that today attend its production but that are not inherent.”

This leads me, pretty clearly, to the following simple conclusion.

Today, development teams leveraging OSGi to build server-side applications are burdened by accidental complexity. As platforms and tools mature, the accidental complexity of OSGi will be reduced to near zero. Modularity though, enabled and enforced by OSGi, attacks the essence.

If the difference between essential and accidental complexity isn’t quite clear, I highly recommend you take a few moments and read Mr. Brook’s essay (linked above). And then, if you can find a copy of the book’s 2nd edition, take a look at Chapter 17, “No Silver Bullet” Refired (Note: If anyone can find the Refired essay online, please post the link in comments).

Throughout these two discussions, you’ll find subtle hints extolling the virtues of modularity. But only do this if you’re willing to exercise your brain with thought, because the connection you’ll discover might just be transformational!

OSGi: Complexity is NOT the Problem

The editors at SD Times have proclaimed that OSGi is too complex. Unfortunately, they miss the mark, and the following statement taken from the article is misleading.

We believe that OSGi, rather than simplifying server application development, has actually made it more complex, and there aren’t sufficient benefits to justify the added complexity.

This statement should have been tempered with context. It is not universally applicable, though is made to sound as if it were. OSGi may face challenges, but complexity of the OSGi framework is not one of the long-term inhibitors to adoption.

Technology Adoption Lifecycle

All technology goes through various phases of adoption. A lack of platform support and tooling may be inhibiting adoption today, but that is today’s problem, and one that is being dealt with. And there is a stark difference between complexity brought on by lack of tooling and platform support versus complexity associated with an ill-conceived idea. OSGi suffers today from lack of enterprise-class tooling and platform support. OSGi is not an ill-conceived idea.

Whenever an organization adopts a new technology, they’ll undoubtedly face some degree of initial complexity. New technology takes time to learn. As the innovators and early adopters continue to use OSGi, tooling is going to get better and developing modular applications using OSGi is going to get easier.

The question you must answer is whether the pain of adoption today is worth the gain of a modular architecture. There are many factors to consider when answering this question, but as the technology matures, the question will be easier to answer. As I’ve alluded to, other factors that have little to do with OSGi’s technical benefits will likely determine if it’s is able to cross the chasm.

Long Term Value

But the article’s biggest flaw is in stating:

And what’s the benefit again? Enterprise developers have written many, many server-side Java applications without using OSGi.

It’s important to understand its value over time. Let me pose some simple questions.

  • Is Spring too complex? It is far easier to create an application with concrete class references than it is to use dependency injection and abstractions.
  • Is Hibernate too complex? It is far easier to use JDBC than it is an ORM framework.
  • Is unit testing too costly? It is far easier to forego unit testing than it is to create a robust suite of tests.

Yet each contribute to more flexible software systems, and in that, there is significant benefit. So it is with OSGi and modularity, as well.

Leveraging OSGi to design modular applications will require us to learn new concepts, tools, and design paradigms, but there is no doubt that modularity has a tremendous upside. This makes the SD Times article paradoxical. OSGi does not increase complexity; the very benefit of modularity is that it helps us manage complexity.

More on Benefits

If you’re interested in exploring some of the benefits of modularity, you can start with the following:

OSGi – Feast or Famine?

I want to be careful here, but also very clear – I advocate OSGi and modularity, though am concerned about its future. In some circles, OSGi is hailed as a disruptive technology; in others, it lives in relative obscurity. There are some indications that adoption is growing, and possibly at some point, it’ll catch like wildfire. But right now, OSGi is the most widely acclaimed technology that nobody is using.

Maybe I’m too harsh. Or too impatient. Maybe I need to give it more time. I know some will argue that people are using OSGi, and they’re right. Some are. But of the more than 6 million Java developers worldwide, those using OSGi and designing modular applications is only a very small fraction.

So what’s the problem? What’s needed to drive adoption? How will OSGi and modularity succeed?

What’s In a Name?

One of the greatest inhibitors to adoption has been platform support. Until recently, few platforms exposed the virtues of OSGi to the enterprise developer. That’s beginning to change, and should help drive adoption. Another obstacle is tooling, which I discussed a while back when lamenting that one of the biggest barriers to adoption is no migration path. With better tools, we’re beginning to overcome that challenge.

But will it be enough? Will platform support and great tools help drive adoption? Sadly, possibly not. And part of the problem, I believe, has very little to do with the technology, and much more to do with its image.

So, what’s in a name?

OSGi is the dynamic module system for Java!

Modularity – Old and Stodgy

Recently, I commented on a recent conversation I had with a conference organizer and his reluctance to accept my speaking proposal on modularity. The general sentiment was that it’s an old, boring concept that can be traced back 40 years! He was interested in my talk on agility, but I explained that given the choice, I’d prefer to speak on modularity.

Currently, I’m participating in a similar conversation with the organizers of the Agile 2010 conference on a few sessions I’ve proposed on modularity. And now, Peter writes about a similar experience. This sends an important message to the OSGi community, and highlights an important challenge surrounding modularity. The message: Modularity is boring! The problem: The importance of modularity, and its support as a first class citizen on the Java platform, is not well communicated nor understood.

Not a Technology Problem

This is a serious problem that requires attention. In his post, Peter suggested that micro-services may be a more apt term for OSGi services, so as not to confuse the term with web services. He even hints at proposing a new JSR that would introduce the idea of micro-services as a new primitive on the Java platform. This is a good start, and micro-services isn’t a bad idea, but more needs to be done. The problem is not that OSGi isn’t a great technology with real benefits, the problem is that nobody cares about modularity.

Possibly it isn’t that they don’t care, but instead that modularity won’t sell. Nobody wants to buy a 40 year old technology, they want to buy the next great thing. And without question, developing modular software is not a new idea. But in general, the ideas surrounding SOA were not new either. Yet, without question, SOA has garnered significantly more attention than OSGi over the past decade. SOA has had its day in the sun; OSGi and modularity have not.

This is not a technology problem. The real problem is that OSGi doesn’t have a clearly articulated identity that explains why an organization must be using it, and organizations will adopt OSGi only if its adoption can be tied to real business benefits. Just like object-orientation in the 1990’s and SOA in the 2000’s, each were sold with a wonderful accompaniment of benefits that would reduce cost, speed delivery, model the real world, blah, blah, and blah. Whether they did or did not help organizations realize these benefits is arguable. Whether each were widely adopted is not!

Crossing the Chasm

OSGi has the potential to be this decade’s most disruptive technology. Unfortunately, the virtues of OSGi run the very real risk of never coming to fruition if it’s unable to cross the chasm. And sadly, it can never cross the chasm unless we find an effective way to sell, or reinvent, modularity.

Foremost, OSGi, modularity, and micro-services do not represent the paradigm shift that’s necessary. Too often, we talk about OSGi in terms of its technical merits alone. Unfortunately, that’s not enough. If OSGi is to cross the chasm, it is going to require something different. Something that people can really latch onto. Something that gets them excited. Modularity isn’t enough. Nor is plug-in architecture. Nor any other technically superior thing that OSGi is.

Naturally, this is only my perspective. My opinion. But OSGi has been a widely acclaimed technology for several years, and it has achieved very little enterprise adoption. While the move by SpringSource to donate dm Server to the Eclipse Foundation was hailed as a great contribution to open source, it’s also a clear indication that adoption of dm Server was gaining little traction.

Now, it is possible that an 800 pound gorilla could throw their weight behind OSGi and modularity that will help drive enterprise adoption. Maybe IBM’s move to expose the virtues of OSGi to the enterprise will be enough. With other vendors following suit, and up and coming vendors such as Paremus on the rise, that just might be enough. But if it is enough, I still believe the enterprise isn’t going to buy OSGi and modularity for the reasons we believe they should, they’ll buy something else very trendy in which OSGi and modularity plays a central role. Something new. Something fashionable. Because as I recall Ivar Jacobson alluding to at a keynote a couple of years ago,

The software industry is more driven by fashion than the fashion industry.

Sad. But true! And hopefully when it happens it won’t dilute the value of OSGi and modularity.

Ignore my Rant if You Want

OSGi is a great technology solution. But we also recognize that technologically superior solutions often fail to win. One of the most popular stories is of the video tape format wars. So my little pontification here should in no way be interpreted as questioning whether OSGi is capable, but instead whether OSGi will.

You can choose to ignore me if you’d like, scoff at me, or call this a rant. Maybe I’m just impatient. Either way, I’ll still continue to proclaim the virtues of OSGi and modularity. I believe in it…firmly! But the nagging feeling I have surrounding its ability to cement its place in the world of enterprise software development will continue to tug at me. And I wonder…will it ever succeed? And if so, how? And…when?

What do you think?

A Badge of Honor

Often times, I hear folks exclaim that they’ve been on “40 development projects over the past 10 years.” Or, “70 projects spanning a 20 year career.” They say this as if it’s some badge of honor. But I’m not so sure that it is.

Of course, there is value in widespread project exposure. Gaining experience with new technologies. Experiencing the dynamics of different teams. Understanding the challenges surrounding different organizational cultures. Each project is unique, bringing its own set of experiences, and these experiences are all incredibly valuable.

But when I see that someone has jumped from one project to another, it also leaves me wondering! Have they ever stuck around long enough on any single project to see their decisions through to the end? Have they ever had to live with the decisions they’ve made? An amazing compilation of knowledge is obtained by not only participating in the early stages of development, but also in maintaining the software system after it’s been released. To name just a few:

  • Dealing with the challenges of Phase 2, while also maintaining Phase 1.
  • Managing multiple branches of the software system. Merging those branches back together. Creating new branches.
  • A critical production issue arising about the same time you need to start heading out for that important customer demo.
  • Attempting to change a piece of code that hasn’t been touched in 10 months.
  • Watching the software system grow from nothing to a system that’s more than 500,000 lines of code. Keeping the build performant. Keeping the build working!
  • Being forced to live with seemingly meaningless early design decisions that haunt you over time.
  • Trying to upgrade versions of third party libraries while development is ongoing. Simply recognizing the right time to upgrade.

And there is so much more. Priorities tugging you in five different directions at one time. If you’ve had the luxury of living with a system (and the mistakes) you created, you’ll realize that there are very few, if any, decisions that shouldn’t be given conscious thought.

When an individual sticks with a project for a long time, they realize the importance of maintaining a clean design, a robust suite of unit tests, and how they package their software system. There is significant knowledge gained by sticking with a project for a long period of time. Perhaps the next time you hire a developer, it might be wise to ask them, “What’s the longest you’ve spent on any given project?” That may be more important than the number of projects they’ve been on.

Somewhere along the way, I picked up a small piece of advice that has helped serve as a valuable guide. There is a big difference between ten years of experience and one year of experience ten times. When you jump from project to project, never living with the decisions you’ve made, you have lost an opportunity to learn what works well and what does not.

Agile: The New Era

It’s housecleaning time again, and like last time, I stumbled across an article I wrote back in 2006 that I don’t believe ever reached publication (at least, I don’t think it did…how am I expected to remember what I did in 2006?). For the most part, I’ve left it in its original state, except that I removed the Agile Manifesto and 12 supporting principles. There are easily enough found on the Agile Manifesto website, and the article is long enough without this duplication. The wordle at right shows the most common words used in this document (click to enlarge). Here, in it’s otherwise unadulterated glory, is Agile: A New Era of Software Development.

Agile: A New Era of Software Development

Embrace Change

Writing code is easy, but developing software is hard. While syntax errors are common, their severity pales in comparison to the logic flaws inherent in many software systems. The difficulty in software development is not writing code or applying a certain technology stack. Instead, the challenge lies in the specification and design of the software itself.  Therein lies the essential complexity of software development, an idea introduced by Frederick Brooks in his 1987 article titled, “No Silver Bullet” [Brooks]. The most difficult aspect of software development is deciding what, exactly, needs to be built.

There is certainly evidence backing this claim. The original Chaos Report shows the top three impediments to a successful development effort are lack of user input, incomplete requirements and specifications, and changing requirements and specifications [CHAOS]. No other activity, if done incorrectly, stands to compromise the system more than incorrect requirement specifications.

It might not be so difficult were software a concrete entity, existing in a world where we could easily visualize it’s structure and behavior, allowing us to more reliably assess and share the impact of change. But software is a highly conceptual, invisible construct. It is considered infinitely malleable by those not intimately familiar with the conceptual complexity of it’s structure and behavior. The contractor building your home would look at you with incredulous disbelief if you suggested that the house he has 90% complete no longer met your needs, and you asked that he move walls. Or imagine how ridiculous it would sound to suggest that a new third floor be inserted to a 100 story skyscraper. Physicists labor on with firm belief that there exist an underlying set of unifying principles to serve as guidance. Or at least, there are laws of physics that we hold to be true. There are no such rules or principles that guide software development. We are left with the imagination and dreams of our clients, and they demand and deserve rapid response to change.

We have made valiant attempts at conformity. Ceremonial processes attempting to define standardized activities that guide the development process have failed, however. We cannot define detailed up-front requirements specifications and expect them to survive the development lifecycle intact. We cannot establish an initial design of the conceptual construct and expect the structure to go unscathed throughout the process of construction. Software development is an error prone human activity involving experts with varying backgrounds and skills who must come together and attempt to communicate uniformly, working as a team toward a common goal. While tools and process do help, we must also accept that change is expected. We cannot treat change as evil. Instead, the tools and process used must allow us to accommodate change, treating it as an inherent part of software development. Changing requirements is a rule of our game. The software we develop must be malleable and adaptive to change, and the process we use must embrace change.

We often draw comparisons between software development and various manufacturing processes. As Larman points out, however, manufacturing is a predictive process [Larman]. Herein lies one of the greatest differences between software development and the manufacturing processes to which we often draw comparisons. Manufacturing is a repeatable activity, with high rates of near-identical creation where a consistent product is produced in assembly line fashion. Little change is expected, making it possible to reliably estimate and predict the outcome. Software development is much more like new product development, where evolutionary specifications and adaptive planning is necessary to deal with the many unknowns that lie ahead.

Agile Principles

In early 2001, a small group of industry experts converged in Utah to discuss alternatives to heavy, document driven software development methods. Emerging from this meeting was the Agile Manifesto, a symbolic proclamation endorsing the virtues of a lighter, more flexible, people-oriented approach to software development, giving birth to the agile software development movement. (Since this is already a long article, I’ve snipped the manifesto and principles, which were included in the original version. If you’re interested, you can view the manifesto and its 12 principles on the Agile Manifesto website.)

<Snipped the Manifesto for Agile Software Development and 12 Principles>

The ideas behind these 12 principles are simple, and contain no hidden messages. Of course, there are different techniques embodied within various agile processes that support these principles. The one certainty is that agile teams definitely work differently from their less agile peers. They recognize there is one end goal – to create a working, functional software product. With that in mind, they work very closely with project stakeholders throughout the development lifecycle, knowing it is the stakeholders who possess the knowledge the system must embody. Agile teams work very hard to deliver working software iteratively and incrementally, and they adopt techniques representative of that ideal.

Agile project managers tend to favor intense communication and collaboration over heavy documentation. Empowering team members to make decisions enables responsiveness to change. Facilitating and negotiating requirements scope provides important feedback, helping plan future iterations, where each iteration produces a deliverable that can be shared with clients and stakeholders. Instead of forcing the team to follow a predictive project plan, agile project managers are more opportunistic. They prioritize features based on stakeholder feedback, and make adjustments as the iterations progress. Concurrent and parallel activities are favored over a phased approach. Agile project managers tend to guide the team instead of manage the team, and strongly discourage unnecessary overhead.

Agile developers work with a similar set of goals, knowing functional software must be delivered early and often. They work judiciously to grow a code base built upon a solid foundation, where each day represents a step forward. They integrate frequently, and do not tolerate failed builds. A rich suite of tests provide the courage necessary to respond to change when the need arises. They avoid the notion of code ownership, empowering other developers to make improvements to any component of the software product.

A common misconception is that agile processes discourage all documentation. This is untrue. Agile processes discourage unnecessary documentation, favoring collaboration as the preferred technique. Instead of using documentation to drive communication, agile processes favor face-to-face communication. Documents are encouraged by agile processes, so long as the need is immediate and significant.

Transitioning to Agile

Agile software development is based upon the fundamental premise that we must drive and respond to change quickly. The Agile Manifesto and 12 supporting principles serve this premise well. Advocates of agility claim speedier delivery of software, software with more business value, increased team productivity, higher quality systems, and a more enjoyable development experience. I believe each of these to hold true. Agile teams not only welcome change, they are able to respond to change at all levels of development. A project manager might discuss a changing requirement with a business client, empower a business analyst to schedule a meeting with the client to discuss further details, while a developer assesses the impact of change knowing she has the courage to accommodate the request because of the rich suite of unit tests in place.

Saying you’ll be more responsive to change and creating an environment that embraces change are separate beasts, however. Practicing agility is hard work, especially if your team is accustomed to more traditional approaches. As with many things new and unfamiliar, some resistance will no doubt arise by those who aren’t fully convinced. Agile projects differ greatly from their less agile counterparts, and skeptics will have many opportunities to express their discontent. As someone experimenting with agility, you may even have doubts. But don’t be discouraged, and give your agile transition the time it deserves.

One of the most significant changes you may experience is a feeling that you’ve been thrust into a chaotic nightmare. I doubt it’s unusual to feel this way. You’ve lost the security of detailed requirements specification and user sign-off. You are writing code without the comfort of knowing exactly what your stakeholders want. The detailed plans that have served as your security blanket on past projects no longer exist. And the celebrations accompanying completion of your various phase milestones are gone. Of course, these were all false comforts anyway. Stakeholders always changed their minds. Your detailed requirements and plans were outdated as quickly as they were completed.

Instead, you’re now working in shorter iterations with vague requirements. Initial releases early in the lifecycle may be completely thrown away. Your first few weeks may seem wasted, with little or no valuable artifacts produced. Naysayers will immediately come forward and cite the lack of progress. Previously, those first few weeks or months were spent producing very detailed requirement specifications and beautiful design models. But don’t give up yet. In that previous world, you were only delaying risk and postponing integration, avoiding the most difficult aspect of software development until the end of the lifecycle. Now you’re attacking risk early, prioritizing features, and working hard to develop a functional piece of software as early as possible. Progress may not be at breakneck speeds, but you are learning a tremendous amount about the requirements of the system, and your velocity is sustainable. Additionally, you are also performing a number of other important lifecycle activities.

Depending on the level of ceremony and bureaucracy within your organization, you will experience varying degrees of success when adopting agile techniques. As with any new technology adoption, it’s best to phase the transition. Some agile techniques are easier to adopt than others, and some serve as valuable catalysts to adopting additional techniques in the future. Don’t attempt to completely redefine how you work. It’s relatively easy to phase the agile transition, and you’ll want to adopt those principles that offer you the greatest initial reward.

For instance, if you’re struggling to produce quality software at a consistent rate, implementing a continuous integration strategy will help you frequently verify the quality of your work. In addition to the comfort of knowing you have a product always in a functional state, the ability to share the product with clients using functional demos and prototypes will tighten the feedback loop and offer valuable insight to the client’s perception of the software. In a number of cases, I’ve found this to be valuable in identifying subtle requirements that can be difficult to identify in other requirements elicitation venues.

Empirical Evidence

In recent years, there has been a significant amount of research comparing agile development methods to their waterfall counterpart. In Agile and Iterative Development: A Manager’s Guide, Craig Larman illustrates the advantage of agile development through detailed analysis of multiple studies[Larman]. The compilation of his results are illustrated below.

A study by Alan MacCormack at Harvard Business School explored whether evolutionary development techniques yielded better results than the waterfall model. The study included applications ranging from application software to embedded systems, with median values of nine developers spanning a 14 month development cycle.

A key conclusion of the study, in which 75% of participants used agile techniques compared to 25% using waterfall, explained releasing software earlier, rather than later, contributed to a lower defect rate and higher productivity. There was little evidence showing that a detailed design specification resulted in a lower defect rate, however, reviews with peers did help in reducing the rate of defects. The study found that iterative and agile practices have a significant impact on defect and productivity factors, as indicated by the following points.

  • Releasing a system with 20% of the functionality complete is associated with a decrease in the defect rate of 10 defects per month per million lines of code as compared to waiting to release a product until 40% of the functionality is complete, and an increase in productivity of eight more lines of source code per person-day.
  • Continuous Integration, the idea of integrating and testing code as it is released to your source code repository, resulted in a decrease in the defect rate of 13 defects per month per million lines of code, and an increase in productivity of 17 lines of source code per person-day.

The study also found four practices that were consistently used by the most successful development teams. The first two are deeply embedded in the ideals of agile software development.

  • Releasing early and often to project stakeholders, using an iterative lifecycle.
  • Continuous integration, with daily builds including regression testing.
  • Teams with broad experience delivering multiple projects.
  • Careful attention to modular and loosely coupled, componentized architectures.

In a separate study [Shine], 88% of organizations cited improved productivity when using agile methods, and 84% cited improved productivity. 49% stated that the cost of development was less when using agile methods. Additionally, 83% claimed increased business satisfaction and 26% claimed significantly better satisfaction. Another study by Boehm and Papaccio [Boehm] discovered that a typical project experiences a 25% change in requirements, while yet another [Johnson] showed  that 45% of features were never used.

There have also been many research efforts devoted exclusively to the analysis of waterfall methods. Below is a summary of these findings, taken from a variety of studies.

  • Scope management related to detailed up-front requirements was a significant contributing factor of failure [Thomas].
  • The U.S. Department of Defense (DoD), when following a waterfall lifecycle, experienced a 75% failure rate [Jarzombek]. This resulted in the DoD adopting a more iterative and agile approach.
  • On a study including 400 waterfall projects, only 10% of the code was deployed. Only 20% of code deployed was used. The main factors included changing and misunderstood requirements [Cohen].

As these studies clearly illustrate, there is significant evidence showing that agile and iterative techniques offer significant advantages over the waterfall model of development. In fact, for larger projects, the statistics supporting agility were even more pronounced.


There are a variety of agile processes available to choose from, and each abide by the spirit of the manifesto and it’s 12 supporting principles. The agile movement and it’s supporters recognize that software development is a human (though not always humane) activity. Instead of forcing process on people, agile methods allow process conformance to people. Good people, working toward a common goal, can achieve great things will little ceremonial process, assuming you give them an environment that empowers them. Solid empirical evidence backs this claim. And if the quality of people is in question, it’s doubtful that any process will produce success.


  • [Alliance]. The Agile Alliance. Manifesto for Agile Software Development. 2001.
  • [Boehm]. Boehm, B, and Papaccio, P. Understanding and Controlling Software Costs. IEEE Transaction on Software Engineering. October 1988.
  • [Brooks]. Brooks, Frederick. No Silver Bullet: Essence and Accidents of Software Engineering. 1987.
  • [CHAOS]. The Standish Group International, Inc. The CHAOS Report. 1995.
  • [Cohen]. Cohen, D., Larson, G., and Ware, B. Improving Software Investments through Requirements Validation. IEEE 26th Software Engineering Workshop. 2001.
  • [Jarzombek]. Jarzombek, J. The 5th Annual JAWS S3 Proceedings. 1999.
  • [Johnson]. Johnson, J. Keynote speech, XP 2002, Sardinia, Italy. 2002.
  • [Larman]. Larman, Craig. Agile and Iterative Development: A Manager’s Guide. Addison-Wesley, 2004.
  • [MacCormack]. MacCormack, A. Product-Development Practices That Work. MIT Sloan Management Review. 2001.
  • [Shine]. Corporate Report. Agile Methodologies Survey Results. Shine Technologies Pty Ltd. Victoria, Australia. 2003.
  • [Thomas]. Thomas, M. IT Projects Sink or Swim. British Computer Society Review. 2001.