February 23 - Keynote titled OSGi in the Enterprise: Agility, Modularity, and Architecture’s Paradox
March 22 - 25 - Tutorial on Modular Architecture
June 14 - 17 - Sessions titled Turtles and Architecture and Patterns of Modular Architecture
July 26 - 30 - Two sessions on rich mobile applications and one on agile development. Half day tutorial on software process improvement.
Right on. I just won a Best Buy drawing worth $1000. Either that or I won a shiny new virus by clicking the link. Hmm...what to do. 2012-08-29
The opinions expressed on this site are my own, and not necessarily those of my employer.
Joel tells the story of The Duct Tape Programmer, and Uncle Bob offers his response. Now these are two pretty smart guys who know a lot about software development. But when we receive fundamentally different messages from a couple of industry luminaries like Joel and Uncle Bob, we’re left wondering - “Who is right? Who should we listen to?”
Joel makes some valid points, but it’s strange that immediately after discussing the dangers of overengineering and the importance of shipping the product, he points out the following:
Zawinski didn’t do many unit tests. They “sound great in principle. Given a leisurely development pace, that’s certainly the way to go. But when you’re looking at, ‘We’ve got to go from zero to done in six weeks,’ well, I can’t do that unless I cut something out. And what I’m going to cut out is the stuff that’s not absolutely critical. And unit tests are not critical. If there’s no unit test the customer isn’t going to complain about that.”
I’m not quite sure how unit testing encourages overengineering nor how unit testing impedes shipping the product. Unit testing does neither. In fact, unit testing inspires a simpler design, not a more complex one. And because unit testing helps developers build quality into the product, I’m certain it doesn’t inhibit shipping the product - or at least a product that works. The correlation makes no sense, and I can’t help but wonder if the message wasn’t an intentional jab based on some past conflicts.
Of course, I certainly agree that favoring simplicity and getting the job done are righteous goals. Sometimes you have to cut a few corners, take a few shortcuts, and implement a solution that isn’t perfect. But there still exist fundamental practices of professionalism that must serve as our guide. And as Uncle Bob points out, unit testing is one of these practices. Bottom line! As a humorous post on twitter suggests:
never take software advice from a bug tracking system salesman
Unit testing is important, has significant long-term benefits, and developers should use them. However, I do take slight issue with Uncle Bob’s following statement:
I found myself annoyed at Joel’s notion that most programmers aren’t smart enough to use templates, design patterns, multi-threading, COM, etc. I don’t think that’s the case. I think that any programmer that’s not smart enough to use tools like that is probably not smart enough to be a programmer period.
In this respect, Joel is right. There might be a lot of people out there who aren’t smart enough to be programmers, but the reality is that only 50% of us can be above average. Of course, we all think we’re in the upper 50 percentile and when we’re looking at code written by someone else, it’s apparent to us it’s the other guy or gal who’s in the lower half. Without arguing who belongs where, the reality is that if we were all smart enough to use these advanced constructs, we wouldn’t be left with the mess we have today.
But really this only reinforces the importance of unit testing, which is why statements that denounce unit testing are so surprising. Think about it…Would you rather maintain a codebase with near 100% test coverage or a codebase with near 0% test coverage? Maybe those who can’t answer this question are the folks that shouldn’t be programmers!
Most of my discussions surrounding agile architecture have been focused on exploring how modularity helps increase architectural agility. I claim that modularity is a required (and to this point, missing) aspect of agile architecture. The basis for this claim follows:
Without modularity, we can’t identify the joints so it’s more difficult to understand where we need the flexibility. My posts titled Modularity & Architecture and Modularity by Example show visual examples comparing designs that are isolated and insulated to designs that span the joints of a system. I’ve also devoted extensive discussion to these ideas in a number of other posts, which I summarize in my Agile Architecture Presentation post.
Recently, I’ve been spending more time exploring lean software development principles and their relationship to agile architecture, and the best place to start when examining lean is with the Poppendiecks. I’m fascinated by the synergy that exists between the lean principles and agile architecture. In fact, when reading the second chapter of their book, “Implementing Lean Software Development: From Concept to Cash“, I was pleasantly surprised by an interesting discovery.
It seems that a study of software development practices by Harvard Business School professor Alan MacCormack revealed four fundamental practices that lead to successful software development. These include releasing early, continuous integration, experience and instinct, and a modular architecture. So it seems I’m not alone in feeling modularity is a critical component of agile architecture. But the thrust of the discussion comes later on in Chapter Two, when speaking of deferring commitment.
Deferring commitment focuses on two fundamental factors - reversibility and irreversibility. In general, reversible decisions are those that can be changed while irreversible decisions are those that cannot be changed. We should strive to make irreversible decisions at the last responsible moment. For it is at this moment when we possess the most knowledge that will allow us to choose the most viable option. But we are also advised that, and I quote:
“First and foremost, we should try to make most decisions reversible, so they can be made and then easily changed.”
For me, this captures the essence of eliminating architecture. If we are able to take a seemingly architecturally significant challenge and make it reversible, then we have effectively minimized the impact and cost of change to a point where change is no longer architecturally significant.
Going forward, I intend to more fully explore additional synergies between lean software development principles and agile architecture.
I recently wrote about eliminating architecture, and there were a few comments, especially by folks on JavaLobby, who thought I had my head in the clouds. Too much theory. Too many abstract concepts. Not achievable in a real world development scenario. That I’m making a play on words. Let’s take another angle.
There are numerous definitions of architecture. But within each lies a common theme, and some key phrases. Here are a few of the definitions. From Booch, Rumbaugh, and Jacobson in UML User Guide (Addison-Wesley, 1999):
An architecture is the set of significant decisions about the organization of a software system, the selection of the structural elements and their interfaces by which the system is composed, together with their behavior as specified in the collaborations among those elements, the composition of these structural elements and behavioral elements into progressively larger subsystems, and the architecture style that guides this organization — these elements and their interfaces, their collaborations, and their composition.
From the ANSI/IEEE Std 1471-2000:
The fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.
In TOGAF, architecture has two meanings depending on context:
1.) A formal description of a system, or a detailed plan of the system at component level to guide its implementation
2.) The structure of components, their inter-relationships, and the principles and guidelines governing their design and evolution over time.
And in a QCon presentation by James Coplien and Kevlin Henney, architecture is defined as:
1.) Architecture embodies the critical design decisions that typify a system. Relates to cost of change, organizational structure, structure of code, capabilities of a system, etc.
2.) The significance of decisions needs to be understood and assessed. A heavy-weight approach is likely to reduce understanding and our ability to assess
Certainly, we see a pattern here as some key phrases and terms recur. These include “significant/critical decisions”, “components”, “structure”, and “cost of change”. And the definition from the Fowler article (which is actually part of a statement made by Ralph Johnson) introduced in the Eliminate Architecture post uses similar terminology and phrases.
It is these definitions that led us to the goal of architecture - eliminate the cost and impact of change. If we are able to eliminate the cost and impact of change, then change is no longer architecturally significant. If something isn’t architecturally significant, then we don’t consider it architecture. Essentially, we’ve eliminated architecture because the cost of change is no longer significant. That must be what we strive to achieve. And the way to achieve this is by increasing flexibility while taming complexity. Let’s take an analogy.
The goal of object-oriented design can be summed pretty effectively using the open-closed principle, which states:
“a system should be open for extension but closed to modification.”
This is done through abstraction and inheritance. If we have abstractions in the right place, we can extend the system by introducing new classes without modifying existing system classes. The key is that we have to recognize which areas of the system require this added flexibility.
Certainly not all areas of the system can possess this flexibility. It’s unrealistic and entirely too complex. But it should still be our goal. It’s the same with architecture. Obviously we cannot eliminate the architectural significance of all change, but it must still be our goal. Otherwise, what goal are we striving to achieve?
Two of the key elements of the architectural definitions are component and composition. Yet there is no standard and agreed upon definition of component (reminding me of architecture, actually), and most use the term pretty loosely to mean just “a chunk of code”. But that doesn’t work, and in the context of OSGi, it’s clear that a module is a software component. That’s excellent fodder for my claim that agile architecture requires modularity because agile architecture demands that we design a flexible system that allows us to make temporal decisions based on shifts that occur throughout development. Modularity has been a missing piece that allows us to more easily accommodate these shifts.
I illustrate this using the simple diagram at right, and it’s what I was referring to in the Eliminate Architecture post when I stated that modularity, in conjunction with design patterns and SOLID principles, represent our best hope to minimize the impact and cost of change. It’s easier to change a design embedded within a module than it is a design that spans modules. I provided a similar example in Modularity by Example.
In other words, it’s easier to isolate change by insulating a design within a module. Designs that span modules are the joints of our system, and changes in these areas are more complex and costly. This is where we need the flexibility. It’s where we need stability. But if we don’t have modularity, we don’t have joints. Or maybe everything becomes a joint? Either way, without modularity we can’t identify the joints so it’s more difficult to identify where we need the flexibility. If we are able to avoid changes that span joints, that change is isolated…insulated…encapsulated within a module, and the impact of change is minimized..
Can we do this in all cases? Of course not! Just like we cannot always design software that’s open for extension but closed to modification. But it must be what we strive to achieve. Again, modularity has been the missing ingredient. And now is a perfect time to start understanding how to design more modular software.
The slide deck for my upcoming presentations at SpringOne 2GX, Agile Development Practices, and OOPSLA is complete. This presentation doesn’t spend much time talking about the “softer” side of agile architecture. I don’t make silly statements about avoiding the ivory tower, delivering solutions that work, or embracing change. These are given! I intentionally avoid the buzzwords, like Emergent Architecture, so you won’t find a session that’s full of fluff and short on substance.
I won’t present earth shattering principles of agile architecture. If you’re not already convinced that ivory tower architecture doesn’t work, that you need to find a way to embrace change throughout the lifecycle, or that delivering working solutions is the best way to move, then this may not be the session for you. The general thesis of the presentation follows.
The goal of architecture is to minimize the impact and cost of change. To do this requires we create flexible systems with the ability to adapt and evolve. But with flexibility comes complexity, presenting architects with a paradox. As we design more adaptable systems, the additional complexity reduces their ability to survive. Eventually, the software rots, the architecture crumbles, and the system fails. Therefore, we must find a way to increase adaptability without decreasing survivability.
The presentation begins by examining this goal of software architecture, helping explain why it’s so critically important that we accommodate the natural shifts in architecture that inevitably take place as a software system grows. In other words, change impacts architecture and this demands that we be able to accommodate architectural shifts throughout the development lifecycle. The meat of the presentation is spent exploring how to do this, with lots of real world examples interwoven throughout the discussion. In fact, the presentation is a partial case study that’s gleaned from about a decade of experience practicing what I’m preaching. For more details on these ideas, and the type of discussions we’ll have as part of the presentation, see a few of the following blog posts:
Admittedly, the material I’m presenting isn’t all of my own creation. I view the session as a consolidation of the tried and proven. For example, Ralph Johnson helped backed my thesis when providing the following quote:
…making everything easy to change makes the entire system very complex…
I cite the SOLID principles. I spend some a great deal of time examining the role of modularity in agile architecture, which in my opinion, is a requirement (not the only one, but one that has been missing). I might even talk a bit about services. In the end, I pull all of this information together into what is a clear and cohesive strategy for increasing architectural agility. If you’re attending one of these conferences, I hope you’ll stop by.
The best way to deal with architecture is to eliminate it.
Let’s start at the beginning. First, by defining architecture. Second, by understanding the goal of architecture. I’ll offer two perspectives, one from Fowler and the other from Booch. First, the Fowler statement.
In most successful software projects, the expert developers working on that project have a shared understanding of the system design. This shared understanding is called ‘architecture.’ This understanding includes how the system is divided into components and how the components interact through interfaces. These components are usually composed of smaller components, but the architecture only includes the components and interfaces that are understood by all the developers…Architecture is about the important stuff. Whatever that is.
Now Booch says:
All architecture is design but not all design is architecture. Architecture represents the significant design decisions that shape a system, where significant is measured by cost of change.
Examining these two statements is very revealing. Fowler makes it’s apparent that architecture is about the important stuff, while Booch is clear that something is architecturally significant if it’s difficult to change. Therefore, I’ll conclude that the goal of software architecture must be to eliminate the impact and cost of change, thereby eliminating architectural significance. And if we can do that, we have eliminated the architecture.
The idea behind eliminating architecture isn’t new. In fact, Fowler mentions “getting rid of software architecture” in the article linked to above. And the way to eliminate architecture by minimizing the impact of cost and change is through flexibility. The more flexible the system, the more likely that the system can adapt and evolve as necessary. But herein lies the paradox, and I’ll use a statement by Ralph Johnson to present and support the idea.
…making everything easy to change makes the entire system very complex…
So as flexibility increases, so too does the complexity. And complexity is the beast we are trying to tame because complex things are more difficult to deal with than simple things. It’s a battle for which there is no clear path to victory, for sure. But what if we were able to tame complexity while increasing flexibility? Is it even possible? In other words, how do we eliminate architecture?
As the Johnson quote clearly points out, it’s not feasible to design (or should I say architect?) an infinitely flexible system. Therefore, it’s imperative that we recognize where flexibility is necessary to reduce the impact and cost of change. The challenge is that we don’t always know early in the project what might eventually change, so it’s impossible to create a flexible solution to something we can’t know about. This is the problem with Big Architecture Up Front (BAUF), and it’s why we must make architectural decisions temporally. It’s also why we must take great care in insulating and isolating decisions we’re unsure of, and ensuring that these initial decisions are easy to change as answers to the unknown emerge. For this, modularity is a missing component that helps minimize the impact and cost of change, and it’s why agile architecture requires modularity.
In the UML User Guide (P. 163), Booch talks about “modeling the seams in a system.” He states:
Identifying the seams in a system involves identifying clear lines of demarcation in your architecture. On either side of those lines, you’ll find components that may change independently, without affecting the components on the other side, as long as the components on both sides conform to the contract specified by that interface.
Where Booch talks about components, today we talk about modules. Where Booch talks about seams, I talk about joints. Modularity combined with design patterns and SOLID principles, represent our best hope to minimize the impact and cost of change, thereby eliminating architecture.
Recently, I posted an entry introducing 19 patterns for modularity. The first of these patterns was ManageRelationships. ManageRelationships is a simple pattern stating that we should design module relationships. I categorize this as a base pattern, meaning it’s a prerequisite pattern for many of the other patterns in the list. For example, the Dependency Patterns listed demand that we manage the relationships between modules. It’s a simple pattern, but important. The discussion below isn’t exhaustive, but it does offer a bit more insight to ManageRelationships.
A relationship between two modules exists when a class within one module depends upon a class within another module. In other words,
If changing the contents of a module, M2, may impact the contents of another module, M1, we can say that M1 has a dependency on M2.
Dependencies can manifest themselves in different ways. The most straightforward is a direct dependency, where a client module depends directly on a service module (shown at left in the diagram). Indirect, or transitive, dependencies involve at least three modules, where one service module is also a client module. Also shown at left, the service module is a client module of the subsystem module and a service module to the client module. Here, the client module has an indirect relationship to the subsystem module, and if something changes in the subsystem module, it may ripple through the other modules.
Understanding the relationships between modules makes it easier to isolate the impact of change to a specific set of modules, which is not something easily done at the class level. I provided an example of this in a previous post. In the example above, changes to the client module clearly indicate that the impact of change is isolated only to the client module. Likewise, changes to the service module indicate the impact of change could spread to other classes within the service module as well as classes within the client module. Without designing and understanding module relationships, it’s difficult to understand the impact of change.
There are a lot of things to consider when designing module relationships. In general, a module has incoming dependencies, outgoing dependencies, or a combination of each. Different forces affect modules depending on the types of dependencies they possess.
Modules with a lot of incoming dependencies are more difficult to change because they are being reused by more client modules. Because of this, it’s imperative that they undergo more thorough and rigid testing, and steps should be taken to minimize changes to these modules (like using AbstractModules).
Modules with a lot of outgoing dependencies are easier to change because they are reused by fewer modules. Unfortunately, these modules are more difficult to test in isolation because of their dependencies on other modules.
There are ways to alleviate each of these challenges. Designing abstract modules, ensuring module relationships are acyclic, and separating abstractions from the classes that realize them are each examples.
When designing module relationships, there are some important implementation details to consider. For example, a service module must be included in the build classpath of the client module. Failing to do so will result in a compile error. If the client module references a class in the service module at runtime, then the service module must also be included in the runtime classpath, as well. Failing to do so will result in a ClassNotFoundException.
Modules with excessive incoming and outgoing dependencies are the most difficult to manage because they are widely reused but also difficult to change and test. Because of this, it’s ideal if modules are either heavily depended upon or heavily depend upon another module. Unfortunately, this isn’t always possible. In these cases, other modularity patterns can help ease the tension when modules have a lot of incoming and outgoing dependencies.
Identifying the modules is the first step in designing modular software systems. Shortly following, however, is managing the relationships between those modules. It’s these relationships between modules that I refer to as the joints of the system, and it’s these areas of the system that demand the most nurturing to ensure a flexible system design. If care isn’t given to managing the relationships between modules, separating out the system into a set of modules isn’t going to provide significant advantages.
More on some of the other modularity patterns coming. In the meantime, your feedback is appreciated.