Last week at OSGi DevCon, I attended Peter Krien’s Advanced OSGi tutorial. The tutorial focused on OSGi services, and I had just come off a rather lengthy e-mail discussion with Peter on a similar topic. So this was pretty good timing. I want to share with you one aspect of the discussion, and its relationship to architecture. It’s quite simple, though makes an important point.

In the past, I’ve talked about the importance of designing the joints of a system. OSGi services represent these joints, since services are essentially a module’s published interface. In general, if the implementation details of a module are well encapsulated, then implementation details can change without impacting the rest of the system. In other words, we minimize that nasty ripple affect of change if we confine change to a single module. But changes that span modules is nasty. Let’s take a few visuals to illustrate the point here.

Where’s the Architecture

Commonly, we visualize the module structure of a software system using a simple diagram similar to what’s shown at right (click to enlarge). We see the modules and the relationships between them. Of course, the relationships between modules represent the joints. Pretty simple stuff! Unfortunately, our attention is immediately drawn to the modules themselves, but not so much the joints.

And we really must be much more concerned about the joints than we are the implementation details because changes in the joints of the system are more complex and costly. This is where we need flexibility and it’s also where we need stability. Emphasizing module implementation is important, but flexibility in the joints is more important.

Illustrating the Joints

In the session, Peter annotated the diagram using a new convention, which I’ve depicted in the diagram at right (click to enlarge). The red triangles more clearly illustrate the joints. Here, our attention is drawn more toward the design constructs that span module boundaries. No longer is emphasis placed on the implementation details, but instead has moved to focus on the relationships between the modules. And the direction of the red triangle also helps illustrate the direction of the dependency relationship between the modules. In other words, these red triangles are the services.

The Real Architecture

Now, because the module itself represents behavior that is encapsulated, we can remove the modules from the diagram altogether and show only the joints, as shown at right (click to enlarge). This is where the impact and cost of change is most significant. And this is where we really need to ensure we have the greatest flexibility. These are the decisions that are most architecturally significant.

A Short Digression

Now I don’t think it’s necessarily important to start drawing architecture diagrams this way. In fact, I’ve never felt that it’s critically important to draw these pretty diagrams in the first place. While diagrams (and documentation) can be useful, they are little more than a pimple on software development’s ass. There are much more important things to worry about than diagrams. But I digress, and in general, this simple exercise helps recognize the importance of designing the joints of the software system.

The Real Point

Of course, while this example illustrates the point using modules, it’s not just the joints between modules that are important, but the joints at many different levels of the system, including applications, services, packages, and classes. In other words, the point where two entities connect is where the real architecture lives.

After the OSGi DevCon keynote, I had a few folks stop by and ask me for the slides. They are now available on slideshare. Usually, when browsing the slides without discussion, the lack of context can cause some confusion. So if you have any questions regarding the content, please feel free to reach out. Enjoy!

I’ll be at OSGi DevCon next week, having been given the opportunity to present the keynote session on Tuesday. The conference is co-hosted with JAX London. I thought I’d take a moment before jumping on the flight to offer a preview of what I’ll be speaking about.

The Cost of Complexity

The title of the session is OSGi in the Enterprise - Agility, Modularity, and Architecture’s Paradox. We all know that OSGi isn’t a new technology. It’s been around for over a decade. But the recent surge in interest in modularity on the Java platform is overdue. The complexity of software is increasing exponentially. Did you know:

• In 1990, there were 120 billion lines of code
• in 2000, there were 250 billion lines of code
• The number of lines of code doubles every 7 years
• 50% of development time is spent understanding code
• 90% of software cost is maintenance and evolution

Let’s put this in perspective using the visual at right (click to enlarge), where we can see that the amount of code doubles every seven years. Pretty staggering! But let’s put this in perspective for what it means over the course of the next seven years, as well. It means that between 2010 and 2017, we’ll produce more code than the total amount of code ever written combined! That’s staggering!

I decided to do a bit of research to see if I could find examples that supported these claims. I did:

• Since the Spring framework was released in 2002, the number of lines of code has grown more than 500% through the release of Spring framework 2.5 in 2008.
• FreeBSD contained roughly 8 million lines of code in 2002. In 2009, it was close to 16 million.
• In 2004, the Linux Kernel contained approximately 6 million lines of code. In 2009, it was around 12 million.

I’m sure many of us have experienced this phenomenon. Systems tend not to get smaller over time. And we also recognize that in almost every way, larger software systems are inherently more difficult to design, build, manage, and maintain than are smaller software systems.

Yet, such phenomenal growth is desirable. We can only hope that the systems we create are in such high demand that we have the opportunity to grow them over time. A software system that doesn’t change, dies. Evolution is a big deal! As Lehman’s law suggests:

As a system evolves, its complexity increases unless work is done to maintain or reduce it.

Add it all up, and there are some key takeaways here. We need something that will help us understand complex systems. We need something that help manage the complexity. We need something that will help ease maintenance. We need something that will help us deal with the natural evolution of software systems. We need something that will allow us deal with the natural architectural shifts that occur as a system grows to accommodate demand. For a long time, a central ingredient has been missing. But not for much longer, because the enterprise will get its OSGi!

For the rest of the story, see you in London.

Source of statistical information above: http://users.jyu.fi/~koskinen/smcosts.htm

I’ve spent some time thinking a bit more deeply about a few of my recent posts on software architecture, and have come to the following revelation.

Architects architect architecture!

Don’t let the triviality of this statement undermine its depth. While each of the three words are variations of the same thing, each have different contextual meaning. Let’s tease the statement apart to see what I mean.

• Architects - Humans create software architecture, and for architecture to be effective, we must also be able to understand the architecture. In Eliminate Architecture, I cited a definition of architecture that introduces the social dimension. Architects signify the social pillar.
• Architect - The way we arrive at architecture is through some process or series of steps. We might create diagrams or software architecture documents. We might write a little code (proofs, spikes, prototypes) to determine the viability of architecture. There are many different activities we perform as we create the architecture. Architect signifies the process pillar.
• Architecture - In Modularity and Architecture, I offered up a few industry definitions of architecture. Common keywords that span definitions include components, composition, interfaces, subsystems, and structure. Architecture signifies the technology pillar.

To ensure balance, we must give attention to each of the three pillars. Additionally, each pillar is related to the other. For instance, ignoring the social pillar impacts the other two in unexpected ways.

The Social Pillar

Turtles and Architecture generally focused on the social pillar of software architecture, but also talked a bit about how the technology pillar can increase understanding, visibility, and transparency. The general sentiment is that architects must focus on more than just concepts and developers must focus on more than just code. There is a middle ground that demands attention, as well.

I used a visual similar to what’s at right (click to enlarge) to illustrate the middle ground. It also illustrates how aspects of the technology pillar can help increase understanding and transparency of architecture. Increased understanding of the architecture hopefully leads to improved structural quality (technology pillar) and transparency of the process (process pillar).

The Technology Pillar

Architecture All the Way Down primarily focused on the technology pillar. The visual at right (click to enlarge) illustrates this view. Again, we see the huge gap that exists if we fail to emphasize the importance of modularity. The rightmost portion illustrates how modularity fills the gap - architecture all the way down. Of course, other gaps are also created if we ignore any of the other aspects, such as class design. Note that as we move from services to modules to packages and classes, we increase the granularity along the way. Our classes should not be as fine-grained as our modules, nor our modules as fine-grained as our services.

Additionally, each entity solves a different type of problem (or provides a different type of advantage), as illustrated by the bars at the bottom. All are units of composition, but only services and modules are units of deployment. Services are reused through distributed computing, whereas module reuse is constrained by process boundaries. The technology pillar affects the other pillars. An inflexible rigid architecture makes it difficult for people to understand and communicate (social pillar) and inhibits how quickly we’re able to respond to change (process pillar).

The Process Pillar

The process pillar is one that I’ve not spent much time discussing. Certainly, it’s important though, and includes various aspects like deferring commitment to significant architectural decisions, evolutionary and emergent architecture, and reversibility. The visual at right illustrates the process pillar (click to enlarge). It’s not as descriptive as the other diagrams, I admit.  Anyone have something better that illustrates the process pillar?

I did talk a little about these ideas in Agile Architecture, Lean Principles. But certainly more needs to be fleshed out surrounding the process pillar. This tends to be where most spend their time when discussing agile architecture, but the other pillars are certainly important. The process pillar affects the other pillars. A bad process accompanied by bad practices results in an inflexible architecture (technology pillar) that noone is able to understand (social pillar).

For those that follow this blog, but not the APS blog, I’ve just published a synopsis of modularity on the Java platform. With a lot of momentum continuing to build around OSGi, we’re still in a state of limbo surrounding a standard module system on the Java platform. Some might argue that OSGi has already won - it is the defacto standard module system. Possibly.

Oracle has offered us very little direction on their future plans. There was little fanfare surrounding WebLogic dm Server, with only sparse references and no official word from Oracle. I’m hopeful we can expect to hear some news of the direction they intend to take in their upcoming webcast, though I’m not expecting them to make any significant statements at such a fine level of technical detail. Coincidentally, Apple is holding their own event that very same day, and is expected to announce their new tablet. Let your imagination run wild. Those two events should make for a very exciting day.

Regardless, I’ve had quite a few conversations over the past several months surrounding modularity and OSGi. I have my suspicions on Oracle’s intent, though am reticent to share at this time. Read the APS post titled, Java Modularity - Time to Set Sail.

Tight coupling between modules is a bad idea, and the worst form of coupling is cyclic dependencies between modules. Fortunately, there are a few techniques we can use to break the cycles. They are Callback, Escalation, and Demotion, and I’m going to walk through some examples that show each of them in action.

Then, once the dependencies are broken, we’ll look at two more techniques that allow us to invert and eliminate the relationship altogether. The code for each of the samples can be found in the edcie project on my Google code repository. Each example includes a build script and a simple test case. To execute them though, you’ll need GraphViz if you want to use JarAnalyzer. To invoke the build scripts without invoking JarAnalyzer, you can simply type:

ant compile

Keep in mind that each variation of the system has the exact same behavior!

The Example

The example we’re going to use to drive the remainder of our discussion is incredibly simple. We’ve got a Customer and a Bill class that we’re going to bundle into two separate modules - cust.jar and bill.jar. There’s also a test case called PaymentTest that serves as a sample client to drive the interactions between the two classes. The test case is bundled into the billtest.jar module. The initial class diagram is seen at right. Note the bi-directional relationship between the two classes.

As we progress, we’ll add more classes and abstractions to the system to help increase the modularity. Additionally, we’re going to use JarAnalyzer to illustrate the relationships between the modules and also help us assess the quality of our design. The module structure is below, as generated by JarAnalyzer. You can see how to use JarAnalyzer in the build by reviewing the build file. Again, our goal is to break the cyclic dependency between cust.jar and bill.jar, and we’re going to look at three different ways to do this before moving on to examine different ways to massage acyclic module relationships.

Initial Module Structure with Cyclic Dependencies

Escalation

The first technique we’re going to apply is called Escalation. With Escalation, we break the cyclic dependencies by escalating the cause of the dependency to a higher level entity. Before we do that, we need to more fully understand why a cyclic dependency exists in this example. This reason follows:

A Customer has a list of Bill instances. When the pay method on Bill is invoked, the Bill needs to determine if a discount should be applied. The discount is a product of the Customer the bill belongs to, not necessarily the Bill. Therefore, the Bill class calls a method on Customer to determine the appropriate discount amount. Think of it this way…The Customer represents a payee and we negotiate a discount with each payee. The calculation of this discounted amount is encapsulated within the Customer.

To break this dependency, we want to escalate the cause of the dependency up to a higher level class - the CustomerMediator. The mediator now encapsulates calculation of the discount and passes that to the bill class. The best way to see this change is to look at the modified PaymentTest class. Now, I’ve modified the build script and have bundled the mediator into it’s own module, as shown below. If you dig a bit more deeply into the class structure, you’ll wonder why I didn’t just pass the discount amount from Customer into Bill. Don’t worry about that. This example is slightly contrived because escalation isn’t the best way to solve this type of problem. The key takeaway is that we’ve escalated the dependency up to the mediator.jar bundle, breaking the cyclic dependency.

Escalating the Cause of the Cyclic Dependency

Demotion

A slightly better way to solve this particular type of cyclic dependency (where we have a true composite relationship between Customer and Bill) is to use demotion. With demotion, we push the cause of the dependency to a lower level module. Exactly the opposite of escalation. We do this by introducing a DiscountCalculator class that will be passed into the Bill class. Our modified PaymentTest class will create the calculator and pass it in for us. The Customer class will serve as the factory for the DiscountCalculator, since it’s the Customer that knows the discount that must be applied. The new class structure can be seen at right.

Now we’ll modify our build script to create an additional calc.jar bundle which will contain the DiscountCalculator class. Our resulting module structure is shown below.

Demoting the Cause of the Cyclic Dependency

Already you can see how this is a more natural solution than escalation for this particular type of cyclic dependency problem.  What’s the key difference you might ask? With escalation, notice how I would be able to deploy the cust.jar and bill.jar modules independently. While demotion is a more natural solution in this situation, it also means that to deploy bill.jar or cust.jar, I must also deploy calc.jar. The right solution is always going to be contextual and the ideal solution is likely to shift throughout the development lifecycle.

Callback

Using a Callback is similar to the Observer pattern. With this approach, we’ll refactor our DiscountCalculator class to an interface, and then modify the Customer class to implement this interface. This new class structure can be seen at right.

As it happens in this specific situation, using a Callback represents a combination of demotion and our initial solution. We’ll go back to passing the Customer into the Bill, but will pass it in as a DiscountCalculator type. Whereas in the Demotion example we bundled the DiscountCalculator in a separate module, we’ll now just include it in our bill.jar module. Note that putting the DiscountCalculator in the cust.jar module would introduce the cyclic dependency we’re trying to get rid of. The new module structure, which resembles our original version minus the cyclic dependency, is shown below.

Using a Callback to Eliminate the Cyclic Dependency

Inverting Relationships

Now we’re going to play around a bit with the module relationships. While Callback seems like the most logical solution, what if we wanted to use the cust.jar module without the bill.jar module? Callback, as it’s implemented, doesn’t allow us to do this. But with a bit of trickery, I can actually invert the relationship between the cust.jar and bill.jar modules.

I start by refactoring the Bill class to an interface. Then, to avoid split packages (where classes in the same package are bundled into separate modules), I move the Bill class into the same package as the Customer class. The new class diagram is shown at right, and the inverted module structure is shown below.

Inverted Module Structure

Eliminating Relationships

Inverting the relationships allows us to deploy the cust.jar module independent of the bill.jar module. Again, it’s all about need. But I’d like to explore another option based on another important need - the ability to test modules independently. Before inverting the relationships, I am able to test the bill.jar module independently. After inverting the relationships, I can test the cust.jar module independently. But what if I want to test (or deploy) both modules independently? To do this, I need to completely eliminate the relationship altogether.

As it turns out, because I’ve got a pretty flexible class structure after I inverted the relationships (lot’s of abstract coupling), I can do this by simply bundling the two interfaces - Bill and DiscountCalculator - into a separate module. No other coding changes required. I start by moving them to a new package called base. Then, I modify my build script to bundle these two interfaces into a separate base.jar module, and we have successfully eliminated the relationship between the bill.jar and cust.jar modules, as shown below.

Eliminating Relationships Between Modules

Wrapping Up

We’ve come a long ways from our original version of the system. Two modules with a cyclic relationship to a module structure where the original modules don’t have any relationship to each other. This means these modules can be tested and deployed independently. If you follow this blog, you know I’ve written plenty about the tradeoffs between flexibility and complexity, use and reuse, and many other architectural and design challenges. I hope this little exercise has helped drive some of those concepts home.

As a final note, to explore some of these design decisions a bit more deeply, and to examine the tradeoffs a bit more objectively, I encourage you to run the builds for each of the projects and examine the dependencies.html file in the stats directory. To do this, you’ll need to make sure JarAnalyzer is executed, which requires GraphViz. As you’ll see when comparing the initial version to the final version, we made considerable progress in improving the quality of the design.

2010 is going to be a big year for modularity on the Java platform. So based on recent industry trends, and as momentum continues to build around modularity on the Java platform:

I hereby officially declare 2010 as the year of modularity on the Java platform.

So that’s a relatively silly statement, and I really don’t have any authority to make such sweeping declarations. But the statement is backed by some pretty serious trends. Here are a few nuggets to chew on that has led me to believe modularity is going to be bigger in 2010 than ever before!

Between 2007 and 2009, we saw amazing momentum build around OSGi. GlassFish, Jetty, Spring DM, SpringSource dm Server, Paremus Service Fabric, Nimble, PAX, Sigil, Aries, WebSphere, JBoss, Maven, and of course Eclipse are just a few products that leverage OSGi. The list goes on. The number of 3rd party frameworks that have been “osgi-ified” has also grown substantially. And whether you love it or hate it, let’s not forget about Jigsaw and JSR-294.

But there has been a nagging problem…an elephant in the room you might say…that’s hindered widespread adoption. Without gluing together a platform, assembling it yourself, or adopting an entirely new platform, aside from the vendors who were already exposing the virtues of OSGi in their products, the majority of enterprise developers were unable to leverage any of it to build modular enterprise applications! But in 2010, that’s all going to change.

Modularity is coming to the enterprise and you’re going to be able to use it to build modular applications. IBM has already made this announcement. I suspect others will follow suit. For those using SpringSource dm Server or Infiniflow, you’re probably already doing it. With the major platform vendors showing their support for a technology that’s already embedded in products such as dm Server and Infiniflow, I suspect these platforms will see an increase in market penetration through 2010 and beyond. And I suspect you can expect some big announcements surrounding these products soon. Finally, let’s not forget, if it can avoid another delay, JDK 7 (with support for modularity) will be released in September.

Whereas prior to 2010, enterprise developers could only dream about building modular applications (or work very hard to do so), modularity is going to take center stage. And since modularity lies at the heart of the platform, it doesn’t matter what language you use. Neil showed us how to do it with Scala way back in 2007. And Groovy does OSGi, as does Clojure. Yep, so does Scala. So no matter which language you use, you’ll be able to take advantage of modularity on the Java platform.

In 2010 modularity on the Java platform is going to be a game changer…a disruptor…that reaches from the developer to the data center. As 2010 progresses, more and more developers will be touched by modularity. It’s not something we should ignore. Really, it’s not something we can ignore anymore.

Thanks

This is my last post of the year, and I want to take a moment to thank everyone that spends their precious time reading my long-winded entries. In 2009, I saw a four fold increase in blog traffic over 2008, and December 2009 saw a five fold increase in traffic over January 2009. Out of a total of 74 posts this year, more than half were related to modularity. I’d like to think the increase in traffic is related to the topics I’ve been writing about. Modularity is something that folks are interested in. But we still have a ways to go.

Future of Modularity

Interestingly, I was speaking to a conference organizer yesterday, and I sensed his mild surprise when I began to talk about modularity. He noted that modularity is certainly not a new concept. I agreed. It’s an idea that has been around since at least the early 1970’s when David Parnas published his essay. But things are different now. Until recently, modularity was not a first class concept on the Java platform. If we wanted to develop software with a modular architecture, we were required to do so without much framework or platform support. In fact, we hadn’t even identified any standard definition of module on the platform.

That’s changing. As the application platform vendors continue to bake OSGi into their products, it’s clear that a module is a JAR file and there’s a framework to back it up. In 2010 modularity on the Java platform will gain considerable visibility in our industry. The stackless stack is coming to fruition, and it’s a game changer…a disruptor…that reaches from the developer to the data center. It’s not something we should ignore. Really, it’s not something we can ignore anymore.  So, as we close out one decade and usher in another, modularity will celebrate at least its 40th birthday. Maybe we’re finally starting to get it.

All Posts Page

A final note before closing out 2009. For those interested, I’ve created a custom Wordpress page that shows a summary of all posts published on this blog. Enjoy!

Note: This is a re-post, with slight modifications, from an entry in October 2007. And now, two years later, we’re just about there! For the abridged version of this post, focus on the text in bold. You can see the original version here.

Design Rot

We’ve all experienced that sinking feeling when maintaining a piece of crappy software. Has my change broken the system in some unintended way? What is the ramification of my change on other parts of the system? If you’re lucky, and the system has a robust suite of unit tests, they can offer some support in proving your work. In practice, however, few systems have thorough automated test coverage. Mostly we’re alone, left to verify our changes as best as possible. We might privately criticize the original developers for creating such garbage. It certainly lends a plausble excuse in explaining why the maintenance effort is so costly or time-consuming. Or it might serve as the basis upon which we recommend a re-write. But mostly, we should wonder how it happened.

For sure, most software doesn’t start out this way. Most software starts out clean, with a clear design strategy. But as the system grows over time, strange things begin to happen. Business rules change. Deadline pressures mount. Test coverage slips. Refactoring is a forgotten luxury. And the inherent flaws present in every initial design begin to surface. Reality has proven that few enterprise development teams have the time or resources to fix a broken design. More often, we are left to work within the constraints of the original design. As change continues, our compromises exacerbate the problem. The consequence of rotting design is seen throughout the enterprise on a daily basis. Most apparent is the affect on software maintenance. But rotting design leads to buggy software and performance degradation, as well. Over time, at least a portion of every enterprise software system experiences the problem of rotting design. A quote from Brook’s sums it well:

All repairs tend to destroy the structure, to increase the entropy and disorder of the system. Less and less effort is spent on fixing the original design flaws; more and more is spent on fixing flaws introduced by earlier fixes. As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one. Although in principle usable forever, the system has worn out as a base for progress.

The most obvious question is, “How do we prevent rotting design?” Unfortunately, rotting design is not preventable, only reducable. Of the past ten years, the design patterns movement has provided insight to the qualities of good design. Dissecting design patterns reveals many important design principles that contribute to more resilient software design. Favor object composition over class inheritance, and program to an interface, not an implementation, are two examples. Of the 23 patterns in the GOF book, all adhere to these fundamental statements. Alone however, design patterns that emphasize class structure are not enough to help reduce rotting design.

Reducing Rot

Most patterns emphasize class design, and present techniques that can be used in specific contexts to minimize dependencies between classes. Teasing apart the underlying goal of most patterns shows us that each aim to manage the dependencies between classes through abstract coupling. Conceptually, classes with the fewest dependencies are highly reusable, extensible, and testable. The greatest influence in reducing design rot is minimizing unnecessary dependencies. Yet enterprise development involves creating many more entities beyond only classes. Teams must define the package structure in which those classes live, and the module structure in which they are deployed. Increasing the survivability of your design involves managing dependencies between all software entities - classes, packages, and modules.

But if minimal dependencies were the only traits of great design, developers would lean towards creating very heavy, self-contained software entities with a rich API. While these entities might have minimal dependencies, extreme attempts to minimize dependencies results in excessive redundancy across entities with each providing its own built-in implementation of common behavior. Ironically, avoiding redundant implementations, thereby maximizing reuse, requires that we delegate to external entities, increasing dependencies. Attempts to maximize reuse results in excessive dependencies and attempts to minimize dependencies results in excessive redundancy. Neither is ideal, and a gentle balance must be sought when defining the behavior, or granularity, of all software entities - classes, packages, and modules. For more on the use/reuse paradox, see Reuse: Is the Dream Dead?

Software design is in a constant quandary. Any single element key to crafting great designs, if taken to its individual extreme, results in directly the opposite - a brittle design. The essential complexity surrounding design is different for every software development effort. The ideal design for a software system is always the product of it’s current set of behavioral specifications. As behavior changes, so too must the granularity of the software entities and the dependencies between them. The most successful designs are not characterized by their initial brilliance, but instead through their ability to withstand the test, and evolve over the course, of time. As the complexity of software design is an essential complexity surrounding software development, our hopes lie with technologies and principles that help increase the ability of your design to survive. Such is the reason why agile architecture is so important.

A Promising Future

I’m hopeful that all software developers have experienced the pleasure of a design that, through the course of time, has withstood the test of time. Unfortunately, many enterprise development teams have too few of these experiences. Likewise, few enterprise development teams devote adequate effort to package and module design. It’s unreasonable to believe that even the most flexible class structure can survive should the higher level software entities containing those classes not exhibit similarily flexible qualities. The problems are rampant. Increased dependencies between packages and modules inhibit reusability, hinder maintenance, prevent extensibility, restrict testability, and limit a developer’s ability to understand the ramification of change.

Services offer some promise to remedy our failures with object-oriented development. Yet, while services may offer tangible business value, within each awaits a rotting design. There exists a world between class design and web services that deserves more exploration, and as an industry, we are beginning to notice. OSGi is a proven module system for the Java platform, while Jigsaw aims to modularize the JDK. JSR-294 aims to improve modularity on the Java platform. While some friction might exist between the constituencies involved, it’s only because they too recognize that something has been missing, and are passionate about fixing the problem. Of course, it doesn’t stop there. A plethora of application servers and tools are also including support for modularity using OSGi, which has grown into the defacto standard module system on the Java platform.

All aim to help manage the complexity, from design through deployment, of enterprise software development. With each, new practices, heuristics, and patterns will emerge that increase the ability of a design to grow and adapt.

In Turtles and Architecture, I talked about how important it is that we “architect all the way down”. It helps increase transparency and understanding between developers and architects by emphasizing a lot of the middle ground that noone ever seems to focus on. It’s just another reason why modularity is so important. I used the diagram at right to illustrate the point (click to enlarge). Look at the huge gap that exists if we don’t focus on the stuff highlighted by the gray bar in the middle.

One reason I like this visual is that it illustrates the social aspect of software architecture. Yet, there are other significant advantages to architecture all the way down that we haven’t explored yet. Another is structural flexibility.

Structural Flexibility - Different Entities, Different Purpose

Another benefit of module design in filling that middle ground is that modules can offer different benefits than classes and services. The diagram at right (click to enlarge) illustrates some of the capabilities of different types of entities.

For example, classes are pretty easily reused within an application, but because classes aren’t a unit of deployment, it’s difficult to use them across applications. Of course, intra-application reuse is a sweet spot of services, since they’re method of invocation is through some distributed protocol (SOAP, HTTP, even RMI/IIOP). Yet because services are invoked remotely, we typically like to make them coarser-grained because of the performance implications of distributed requests. So this begs the question - If a service is too coarse-grained to reuse (ie. it does more than what we want), how do I reuse some desirable behavior across applications? Well, without modules, our only other choice is the class. Since a class can’t be reused intra-process, we do one of either two things. Expose it as a service or copy the class (ie. the code). Given the circumstances, neither of these may be ideal. Another option is desirable.

Modules represent that other option. They are a finer level unit of granularity than services, and are a unit of deployment. Since each of these different types of entities are units of composition, we have tremendous flexibility in how we assemble applications. Possibly more important though is our increased ability to accommodate the architectural shifts that naturally occur as requirements evolve. Let’s look at a more concrete example.

A Bit More Concretely

Let’s say I have a business function called Pay Bill for which I develop a web service that can be invoked by many different consumers. That service happens to be relatively coarse-grained, and performs all the steps involved in paying the bill. These happen to include the following:

• audit bill - apply a discount to the bill based on payee
• check for duplicate - ensure the bill hasn’t already been paid
• remit payment - cut the check
• reconcile payment - reconcile with accounts payable financials

This seems reasonable. We have a nice little service that we can reuse any time we want to pay a bill. Unfortunately, the real world often gets in the way of our idealistic solutions. In fact, there are two problems that will eventually surface, and modularity benefits both scenarios. Let’s start by looking at the first scenario.

What should I do when a different scenario arises that demands I follow a slightly modified Pay Bill function?

As part of the remit step, I have a new payee that demands electronic payment. This is pretty easy actually. I simply modify the service to support electronic payments and then configure the service to context for that specific payee. So how does modularity help here?

If the service is composed of modules, it’s going to be much easier for me to understand the structure of the service, assess the impact of what’s it’s going to cost to change the service, and then introduce a new module (or modify the existing module) to provide the new capability. Without modules, I’m simply wading through the code trying to figure out all of these things. Now, the 2nd scenario.

What should I do when I want to reuse just one step of the Pay Bill function?

Let’s say another new requirement emerges. Whereas traditionally bills were entered by data entry personnel, we now have to support electronic delivery of bills. We also know that bills delivered electronically are often duplicates. It’s just one of those business things, you know? If we don’t pay the bill on the day it’s received, the billing party sends us the bill again, asking for payment. So we need to check for duplicates before we save the bill to the database and prepare it for processing. What do we do?

We could take the same approach as before and modify the Pay Bill service so that the duplicate check could be invoked separately from the higher level pay bill function. But that’s a bastardized design solution. Why? We are exposing behavior of the Pay Bill service that shouldn’t be exposed. The API is coarse-grained in some areas and fine-grained in others.

Maybe exposing the finer-grained capabilities of the Pay Bill function isn’t a severe compromise, but it is a compromise nonetheless. And we are forced to compromise because of architectural inflexibility. We don’t have architecture all the way down, and are therefore left with limited choice as to how we support the new requirement. We can either modify the service to reuse what we know is already encapsulated within the service, or copy the code to create something new. But those are the two options we have, and neither may be ideal.

As we continue to hack the system in this manner, with a multitude of other similar changes, the design will rot. Eventually, our Pay Bill service is transformed into a utility that performs all general-purpose bill functions. It’s API is a mix of coarse-grained and fine-grained operations. It’s become the monolith that we’re trying to avoid. While the Pay Bill service is still pretty reusable (it does everything now), it isn’t that usable. That tension between reuse and use surfaces again.

Our decision to modify the Pay Bill service to expose the duplicate check was driven by one thing - ease. It was our easiest option. Really, it was our only option. But it isn’t the best option.

If we architect all the way down, we have another option. A Pay Bill service composed of modules that audit the bill, check for duplicates, remit payment, and reconcile the payment means we can choose the desirable solution over the easiest short term choice. We can avoid the long term compromises that degrade architectural integrity.

Shown at right (click to enlarge), we see the service composed of modules. If we have a check for duplicates module, we can simply reuse that module in the new service or application that’s going to process electronic bills. Or we might expose the capabilities of the check for duplicates module as a separate service. We have multiple reuse entry points, as well as different levels of granularity through which our software entities can be managed, deployed, built, and much more.

To Conclude

My point here isn’t to debate the original design nor the decisions made to reuse the check for duplicates functionality in another service or application. Certainly, debating architectural and design decisions is a favorite past-time of developers. There are a variety of different ways that we can support new requirements as they emerge. Some are better than others and all are contextual.

The gist of my point is that architecture all the way down gives us options, and these options help maintain architectural integrity. They increase our options when making decisions and allow our system to accommodate unforeseen architectural shifts. I can modify an existing application or service to give me what I need. Or I can reuse an existing module that’s already available for me. Or I can compose a new service from an existing set of modules. Or I can break apart existing modules to create new modules that result in new services. And I can do a lot of this refactoring without significant impact on the existing code. This was an important point illustrated in the series of posts on Applied Modularity.

In other words, architecture all the way down helps increase architectural agility, and modularity is a key ingredient.

Next Page →