In part 3, we applied the SeparateAbstractions pattern so that we could independently manage the two different AuditFacade interface implementations. The result was the module relationships shown at right (click to enlarge). The code can for this version of the system can be found in the SeparateAbstractions project. Also, be sure to check out Part 1, where we introduced our initial version of the system, and Part 2 where we applied a couple of the modularity patterns. Now, we’re going to focus on the financial.jar module and try to decouple bill.jar from financial.jar so the two are completely independent of each other.
I think you’ll really like this one…But first, some additional housekeeping.
Fifth Refactoring
Before we decouple the bill.jar module from the financial.jar module, we’re going to talk about exceptions. In general, we need to answer the question.
Where do exceptions belong?
In the CollocateExceptions project, we introduce an AuditException that is thrown if an error is encountered. The CollocateExceptions pattern says that exceptions should be close to the classes (or interfaces) that throw them. Since the AuditFacade interface throws the exception (shown below), we should put the AuditException in the same module as the AuditFacade interface, which is the auditspec.jar module. Note that if we put the exception anywhere else, we’d create an unwanted dependency between modules.
Sixth Refactoring
Ok, the previous refactoring was pretty simple, and it’s time to move on. Here’s the need, and I think you’ll find the solution quite interesting.
A new requirement has emerged, and we need to use the bill.jar module in another system. In this other system, we don’t need the financial.jar module to make the payment. We just want to use the bill.jar functionality. So, what do we do?
First, recall our initial version of the class structure of the system. In the third refactoring in Part 2, we introduced a Payable interface, and then created the financial.jar module. After we finished Part 3, we were left with the module structure shown at right (click to enlarge). Again, our goal here is to eliminate the dependency between the bill.jar and financial.jar modules so that we can deploy the bill.jar module without the financial.jar module. We’re going to apply a little trick called escalation. Essentially, we’re going to escalate the dependency to a higher level module. How? First, let’s take a look at the new class structure that’s going to allow us to remove the module dependency.
As shown at right (click to enlarge), we see the new class structure that’s going to allow us to decouple the bill.jar and financial.jar modules. Things are starting to get a bit tricky here, so before we allocate these classes to their respective modules, let’s walk through the steps to illustrate how these classes will callaborate to give us what we need.
We start by passing a BillPayAdapter instance to the Bill as a BillPayer type. The BillPayAdapter is going to manage the relationship between Bill and Payment. When the pay method on Bill is called, it will turn around and invoke the generateDraft method on BillPayAdapter. The BillPayAdapter invokes the generateDraft method on Payment, passing itself in as a Payable. This allows the Payment to callback on the Payable and invoke the appropriate method (either getAmount or getAuditedAmount) on BillPayAdapter. The BillPayAdapter can now query the Bill to obtain the audited amount and pass it back to the payment. Once the Payment has this amount, it can make the payment. Whew! A fairly complex callaboration, but enough to ensure we can decouple the bill.jar module from the financial.jar module. The code for the new BillPayAdapter class, which mediates (hmm..possibly naming it a mediator would have been better, huh?) the relationship between Bill and Payment, is shown below.
Now it’s time to allocate these classes to the appropriate modules. The key decision here is where we place the BillPayAdapter. Because this controls the Bill and Payment callaboration, we can’t put it in either of those two modules. In fact, we’re going to create a new module that contains the BillPayAdapter. The new module, which we’ll call billpay.jar, will depend on both the bill.jar and financial.jar modules, as shown at right (click to enlarge). The code for this version can be found in the IndependentDeployment project.
For the most part, our system is complete now. But we do have one nasty little problem that we haven’t solved yet. We have two different AuditFacade implementations, but the current system hardcodes creation of AuditFacade1 within the AuditAction class. All this flexibility compromised by a single line of code. Unnerving how that happens! We really do have to architect all the way down. Anyway, let’s fix this problem.
Seventh Refactoring
The problem of creating instances is pretty common, so the solution is relative easy. There are a lot of different ways to do this, but we’re going to take the easiest route (and not the most flexibile mind you) by creating an AuditFacadeFactory class, which can be found in the ImplementationFactory project. Now, instead of the AuditAction creating the AuditFacade implementation, it invokes the factory and is returned an instance, as shown below.
Wrapping Up and Getting Ready for the Postmortem
We’ve made considerable progress. Amazing really. From a single monolithic application to a fully modularized architecture with considerable flexibility. But there are a lot of very interesting takeaways that we haven’t talked about yet. A number of positive side affects have resulted that aren’t immediately obvious. In our final post in this series, we’ll take a more in-depth look at the impact of modularity.
In Part 1, we introduced the system, and broke each layer out into separate modules. In Part 2, we applied two refactorings using two different modularity patterns - AbstractModules and AcyclicRelationships. But we were still left with a problem, which we’ll solve here.
Fourth Refactoring
After the third refactoring, we were left with the modules shown at right (click to enlarge). But this doesn’t entirely solve our problem of how we deploy a new AuditFacade implementation, especially if we want to avoid redeploying the audit.jar module and everything in it. Additionally, because the two implementations of AuditFacade live in the same module, we can’t rip one out (ie. uninstall) when we add the other. In other words, because they are in the same module, they can only be managed together.
To solve this tricky little challenge, we’re going to apply the SeparateAbstractions pattern, which states that we should separate abstractions from the classes that realize them. If you’ll recall from part 2, we applied this pattern when allocating the Auditable interface to the audit.jar module, but we didn’t apply it to the AuditFacade1 class. So we’ll apply it to the AuditFacade1 implementation, as well as a new AuditFacade2 implementation. First, we have to answer the following question.
Where do we put the AuditFacade interface so that new implementations of the interface can be managed separately from the interface and other implementations?
Our answer can be found by looking at the module structure shown at right (click to enlarge). Note that I’ve also included the financial.jar module in this diagram, which was actually introduced in Part 2 when we introduced the Payable interface. We’ll use this soon, but I digress.
Anyway, we separate the AuditFacade interface out into its own module - auditspec.jar. Each of the implementations (AuditFacade1 and AuditFacade2) are also placed in their own modules. The code for this solution can be found in the SeparateAbstractions project. No real coding changes are necessary, other than we’ve added a new class - the AuditFacade2 implementation. Note that we’ve also separated each implementation out into different packages to avoid splitting packages across modules. After adding this new class, we modified our build script to allocate the classes to the appropriate modules.
A Note on the Benefit of OSGi
Now, we have an overall increase in architectural flexibility. Our AuditFacade implementation classes are allocated to separate modules, allowing us to manage the two independently. From a maintenance perspective, we can work on each of the implementations separately, resting assured that changes to one implementation won’t negatively impact changes to the other. We can also test each independently and reuse each independent of the other.
This change also increases the resiliency of the bill.jar module since it’s no longer tightly coupled to any AuditFacade implementation. We can test the bill.jar module using only the auditspec.jar module by creating a mock implementation of AuditFacade for testing purposes. While the billpay.jar module is still dependent on the financial.jar module, we’re going to solve that problem in the next refactoring. In general, we have completely eliminated the dependencies between the bill.jar module and the AuditFacade implementations in the audit1.jar and audit2.jar modules. Remember, at the beginning of Part 2, these modules containing the bill and audit behavior had a cyclic relationship.
While this example doesn’t leverage OSGi, it is important to point out the benefit that OSGi can provide here. Because we don’t have a runtime module system we still need to deploy these modules within the WAR file. The presence of OSGi, however, brings the same degree of flexibility to the runtime that we have at development time (for more information, see The Two Faces of Modularity & OSGi). With OSGi, I would be able to install and uninstall the audit1.jar and audit2.jar modules without redeploying the entire system. When we’re finished with all of our refactorings, we’ll “osgi-ify” the system, deploy it to Tomcat, and realize this benefit.
Wrapping Up and Getting Ready for Part 4
So we’ve made quite a bit of progress from our initial version of the system that completely lacked modularity. By breaking the system out into modules, we ease the maintenance effort and increase overall system flexibility. In part 4, we’re going to to turn our attention to the financial.jar module that focuses on payment. Recall that in Part 2 (when we applied the AcyclicRelationships pattern to the AcyclicRelationships version of the project) , we separated the Payment class out into a separate module and decoupled Bill and Payment through the Payable interface. Part 4 will examine how we can decouple the bill.jar module from the financial.jar module so bill.jar can be managed, tested, and deployed independent of financial.jar. Stay tuned!
See Part 1 for an overview of how we got started. In this post, we’re going to take the modularization of our application a step further. We’re going to apply two refactorings using two different modularity patterns - AbstractModules and AcyclicRelationships. First, we’re going to separate the bill and audit functionality out into separate modules so we can manage (develop, deploy, etc.) them independently. Second, we’re going to remove the cyclic dependency between these two modules.
A Quick Review
Before we get started with this second installment, let’s review what we did in Part 1. Initially, our billing application was all packaged and deployed in a single WAR file. Since we had a conceptually layered application, it was pretty easy to separate the UI layer from the business object and data access layer. Not sure we explicitly pointed it out in Part 1, but that refactoring essentially applied the PhysicalLayers pattern. After doing this, we were left with the system at right (click to enlarge).
Second Refactoring
Recall the initial class diagram, where we had a Bill class with a bi-directional relationship to the AuditFacade class. This design has two fundamental flaws. The Bill is tightly coupled to the concrete AuditFacade class and the relationship is bi-directional. Bad all around! This can be seen in the following code snippet illustrating the Bill’s audit method.
Notice that the audit method actually creates the AuditFacade, calls the audit method, and passes a reference to Bill. Ugly. Let’s clean this up a little bit. While there are obvious technology reasons why we need to clean this up, there is also a motivating business force.
The system needs to go live with the current vendor’s auditing system, but the business has indicated that they aren’t renewing the contract with the vendor and are in ongoing negotiation with another vendor. The contract expires in 6 months, but we deliver the initial version of the system in 3 months.
So, 3 months after deployment, we know we’ll need to swap out auditing systems. What to do?
The first half of this solution can be found in the AbstractComponents project, where we apply the AbstractModules pattern which states that we should depend upon the abstract elements of a module. So we’re going to refactor the AuditFacade to an interface and create a separate AuditFacade1 implementation. This solves the first half of our problem - the tight coupling between the Bill and AuditFacade implementation. The result is the class diagram shown at right (click to enlarge). Take a look at the refactored Bill class, and you’ll see that the AuditFacade interface is now passed into the Bill, allowing us to swap out AuditFacade implementations. Now we need to decide how to modularize the system.
If we take this flexible class structure and continue to deploy in the single bill.jar module, we have the flexibility to swap out AuditFacade implementations at the class level, but we’re still required to package everything up into a single bundle and deploy it together. So what we really must do is separate the audit functionality out into separate modules from the bill functionality. Separating the AuditFacade interface and AuditFacade1 implementation out into separate modules results in the diagram at right (click to enlarge). We can also see this change reflected in the build file that packages up these modules. Here’s where the bi-directional relationship between Bill and the AuditFacade interface (and AuditFacade1 implementation) rears it’s ugly head. We have a cyclic dependency between our bill.jar and audit.jar modules. We need to fix this problem.
Third Refactoring
The second half of this solution can be found in the AcyclicRelationships project. Now, we need to remove the cyclic dependency between the modules, and we’ll apply the AcyclicRelationships pattern which states that module relationships must be acyclic. To do this, we’ll introduce an additional abstraction, called Auditable, that our Bill class implements. Upon applying this little trick, we can see that we have now removed the bi-directional relationship between Bill and the AuditFacade interface, as shown at right (click to enlarge).
Whereas previously the AuditFacade accepted a Bill to the audit method, it now accepts an Auditable type. The new AuditFacade interface can be seen in the code snippet below.
The Bill will pass itself to the AuditFacade interface as an Auditable type. The key element at this point is how we allocate these classes to their respective modules. Here’s a simple rule to abide by when determining how to allocate classes and interfaces to modules.
Interfaces should be closer to the classes that use them, and farther away from the classes that implement them.
We’ll talk more about this in the next post, because we build on this idea to address another problem that has surfaced. But applying this rule now, it’s clear that Auditable should be bundled with the AuditFacade interface, not in the same module as the Bill class.
While we haven’t talked about the new financial.jar module, we applied a similar refactoring to the Payment class by implementing a Payable interface, and breaking it out into a separate module. Allocation of classes to modules is done at build time, so after modifying our build script, the result is the modules shown at right (click to enlarge).
As I’m sure you’ve noticed at this point, we broke our rule above and bundled the AuditFacade1 implementation close to the interface it implements (btw, this is the other problem that has surfaced). If we apply our rule in this situation, we’d bundle the AuditFacade interface and Bill class in the same bundle, which certainly wouldn’t work because it would result in the cyclic dependency between the bill.jar and audit.jar modules that we’ve worked so hard to remove. Alas, this is the focus of our next refactoring, so we’re going to wait to tackle this challenge.
Wrapping Up and Getting Ready for Part 3
This post involved two refactorings. The first was to separate the bill and audit functionality out into separate modules. The second was to remove the cyclic dependency between the bill and audit modules.This offers us decent flexibility. We can introduce a new AuditFacade implementation, knowing that we’d only need to rebuild the audit.jar module. We can test the audit.jar module independent of any other module because it doesn’t have any outgoing dependencies. And if we wanted to, we could deploy the audit functionality separately (ie. we can reuse it elsewhere). So overall, some decent progress.
But some problems remain. If we really want the ability to swap out Audit systems, bundling the AuditFacade interface and AuditFacade1 implementation into the same module doesn’t give us the flexbility we need. While we can easily create a new AuditFacade implementation and allocate it to the audit.jar, this requires us to redeploy audit.jar unnecessarily. In the next post, we’ll explore how we can change the module structure to allow a new AuditFacade implementation while also allowing the existing AuditFacade1 implementation to be removed from the system.
This is my first in a series of posts that shows how we can modularize Java applications today. A few notes before we get started. First, the examples are all core Java, with no OSGi. I’ll explain why in just a moment. Second, I’ve applied these techniques on real projects, so I know that they work.
Why No OSGi?
So why didn’t I use OSGi? Good question. One would think that if I’m going to modularize my system, I’d want to use a module framework. There are a couple of reason I chose not to.
- While OSGi is a module system, OSGi will not help you design more modular software. Among other things, designing modular software requires that we understand the weight and granularity of individual modules, and use the right techniques to manage the coupling between modules. In other words, designing good software is our job. Tools and technologies may help, but make no guarantee. I spend more time talking about this in The Two Faces of Modularity & OSGi.
- Most of us aren’t able to leverage OSGi today because the platforms and langauges we use don’t support it. I want to use the same tools and techniques that we can leverage in the enterprise right now. This is where the modularity patterns, as well as a few other tools, will help.
In general, I really want to focus on the design paradigm, not the tools and technologies. So instead of answering, “How do I use OSGi?”, I want to focus on “How do I modularize my system?”. Hopefully, that resonates with you. For the curious, if you do want to see the difference that OSGi makes, I have OSGi-ified the final version. But you’ll have to wait.
Let’s Get Started - The System
Interestingly, I’ve found that when designing modular software, it’s tough to identify the modules early in the lifecycle. Instead, shifts typically occur, and as things unfold, the modules become more apparent as development progresses. With a SOLID OO design, it’ll make it much easier to move things around and create new modules. So to start, while the system is small, I favor larger (coarser-grained and heavier-weight) modules.
As specific needs emerge, we’ll break larger modules out into a bunch of smaller (finer-grained and lighter-weight) modules that address specific needs (both functional and non-functional requirements). If you’re interested in the abstract essence of what I’m referring to here, you should read some of my prior blog posts (Agile Architecture might be a good place to start). It’s a deep and very interesting topic, and impacts how we understand software, maintain the system, reuse software entities, and more. Or, you can wait and see what I’m talking about, as we’ll experience this phenomenon as we move through the exercise.
Here’s a simple, high-level description of the system we’ll develop. It’s the common bill payment sample system often used.
We’ve been asked to develop a system to handle payment of of bills. Prior to paying the bill, the system should apply a discount to the bill in an amount that has been negotiated with the payee (we call this the process of auditing the bill). Applying this discount is a fairly complex process, and a 3rd party vendor has been commissioned that will apply this discount. Additionally, we must integrate with a legacy financials system that must be fed payment information for reconciliation.
We’ll flesh out additional details as development progresses.
Version One
The initial version for this system uses Struts as the web framework, and packages everything into a single WAR file. The initial class diagram can be seen at right (click to enlarge). It’s not a complete class diagram, but it does show the main abstractions. I’ve greatly simplified the system for purposes of example.
As you can see, there are some Action and ActionForm classes that leverage Struts. There are also a couple of JSP you don’t see here, but will if you take a look at the project. A Customer has a list of Bills, and each Bill has a reference to an AuditFacade and Payment class. The AuditFacade integrates with the 3rd party vendor software that applies the discount, and the Payment class integrates with the legacy financials system. Of particular interest, note the bi-directional relationship between Bill and these classes - a sure sign of a problem that will haunt us later. But don’t worry, we’ll fix it.
For the sake of simplicity, I’ve hardcoded the database into the data access layer, which is represented by the Loader interfaces. This way, you can experiment with the system without running any DDL scripts to create the database. If you really want to, feel free to add a real database backend. It wouldn’t be very difficult, but it’s not what I want to focus on here. The code for this initial version can be found in my Google Code Repository.
First Refactoring
So packaging everything up into a single WAR file for deployment certainly isn’t modular. In most systems we develop, we try to design layers that encapsulate specific behaviors and isolate certain types of change. Typical layers include a UI layer, a business or domain object layer, and a data access layer. In this system, we have these three layers. The Struts action and Form classes represent a part of the UI layer, which is shown in red. The Customer, Bill, Name, AuditFacade, and Payment form the business object layer, and the Loader classes form the data access layer. These classes are shown in blue. Now, here’s a key statement that you need to take with you.
If I truly have a layered system, then I should be able to break out each layer into a separate module where modules in the upper layers depend on modules in lower layers, but not vice versa.
If you try this, it’s likely you’ll find it’s not so easy. The takeaway here? Most development teams feel they have a layered architecture, but in reality, they don’t because somewhere deep within the bowels of the system lies an import or reference to a class higher up in the food chain that we aren’t aware of.
In fact, if I really do have a layered system, then I shouldn’t have to change anything other than my build script to break the layers out into separate JAR files. If I do have to change more than a build script, then I didn’t have a layered system to begin with, and I should perform some architectural refactoring to clean things up. Anyway, the end result is relatively simple to understand. No code changes. Only a build script change. And the structure shown at right is the result (click to enlarge). You can view the refactored project in the PhysicalLayers project, which includes the change to the build script on line 41.
Wrapping Up and Getting Ready for Part 2
This first refactoring was quite simple, but has significant implications. Foremost, it proves that my class level architecture was pretty decent. I was able to break the system out into modules for the various layers without having to change a bunch of code. Really, that’s the reason why it was so simple…because the design was decent. Had the design been shoddy, it would have been significantly more difficult pulling off this refactoring. Trust me!
Yet, as we’ll see, the existing design may meet the needs of today, but it’s going to have to evolve as change emerges. In part 2 of this series, we’ll take a look at what we need to do to integrate with another auditing system, and how modularity can help us do this. And as we progress, the amazing transformation of a system lacking modularity to a highly modularized version will unfold. Stay tuned!
Here’s my Agile Architecture - Patterns & Technology slide deck (bottom of this post) that I presented at SpringOne2GX and OOPSLA. Thank you to all who attended, and for providing such positive feedback. I’ve also uploaded the code samples used to show the modularity pattern examples found toward the end of the presentation. The code can be found under the billpayevolution project in my Google code repository.
A Bit More on the Code
The project is named billpay, and is a very simple system. It starts out in life as a project that’s bundled and deployed using a single WAR file (InitialVersion). Through a number of refactorings (7 in total), the project is broken out into separate modules, and the final version of the system (Refactoring7ImplementationFactory) has a number of modules (JAR files) that are included in the WEB-INF/lib of the WAR.
I intentionally avoided using OSGi for this exercise, primarily to show how we can modularize an application without a runtime module system. I do have an OSGi version of the system that can be deployed to an OSGi runtime (Tomcat using the Equinox Servlet Bridge is what I used), but I haven’t uploaded that code to the repository yet because I have a few things I need to change (like using Spring DM to remove the code dependencies on the OSGi API). That’ll come later, after I’ve had a chance to do a bit more work with it.
Each project along the path toward refactoring the application into modules comes with it’s own build script. For each project, except for the final version (Refactoring7ImplementationFactory), all you’ll need to execute the build is Ant and JUnit.
A Note on JarAnalyzer
Because I used JarAnalyzer in the final refactoring, you’ll need to make sure you have GraphViz installed and then modify the build script to point to the dot executable. Of course, you can also comment out the JarAnalyzer target, too. But JarAnalyzer was very helpful in identifying and managing the dependencies between modules, so I think you’ll find it useful to keep it. If you’re not familar with JarAnalyzer, it produces visual output showing the dependencies among JAR files (shown at right, click to enlarge), as well as an HTML report with some of the Martin metrics. Here’s the JarAnalyzer HTML report for the final version of billpay.
Like with any project, there are a few things I’d like to change. For example, in Refactoring7ImplementationFactory, I’d like to use Spring to inject the implementation classes instead of using the factory. Sometime soon, I hope to walk through these samples in a series of blog entries that shows how modularity can have an amazingly positive influence on the architecture of a software system. Until I do this, or you attend one of my presentations (btw, I’m speaking next week at Agile Development Practices), it may be a tad difficult to understand the forces behind my decisions (hint: they were driven by various business requirements). So please be patient, and stay tuned.
And Now the Presentation
Here’s the presentation, hosted on SlideShare. I look forward to your questions and/or any feedback you might have!
There are certain paradoxes that create conflict when designing software systems. The paradoxes result in opposing forces that are counterintuitive, and require further examination to more fully understand how the tension can be resolved. Here, I explore the tension between use and reuse. Certainly there are others, too. What software development paradoxes have you encountered?
Use and Reuse
I have a simple question. What’s the difference between “use” and “reuse”? Dirk Riehle broaches the subject in suggesting that using a component is when you embed that component in a collective work and reusing a component is when you create a derivative of that component. I use the term “component” above, but it could just as easily be replaced by “method”, “object”, “module”, “service”, or anything else that you want to use. Or should I say reuse? Funny!
A few responses to the same question I posted to Twitter garnered some responses from @RSessions, @bigballofmud (Brian Foote), and @IsaGoksu, though not enough to offer perfect clarity.
The topic can be quite confusing. For instance, is invoking a once deployed service from multiple consumers use or reuse? And how does this differ from including a component in multiple services? Which is use and which is reuse? The differences between these two scenarios can be seen in the diagram at right (click to enlarge).
A Trick Question
Instead of trying to distinguish between reuse and use, let’s consider an alternative perspective. If we adopt a canonical definition of reuse, we can state that it means to simply “leverage an existing asset”. Now, let’s define use as “the ability to leverage an asset”. If we’re willing to accept these definitions, then the relationship is opposing - as one goes up, the other goes down. A module or service might be highly reusable, but very difficult to use. Likewise, a module or service might be very easy to use, but difficult to reuse. And it’s incredibly difficult to offer both.
If we make a software entity highly reusable then it’s likely a lightweight and fine-grained entity. This allows for environmental configuration driven by context and extensibility through well-defined interfaces and extension points. But with this flexibility is additional complexity that makes the entity inherently more difficult to use (flexibility and complexity are another paradox?). I explore these ideas further in Reuse: Is the Dream Dead, and draw the conclusion that
Maximizing reuse complicates use.
Dealing With It
Understanding the forces at play here is important because they are consequential to architecture, and resolving the tension is an aspect of architectural agility. It’s virtually impossible to design a reusable software entity until we have a better understanding of it’s usage scenarios (I recall a “rule of 3″, but can’t seem to place it. Anyone know?). And since the unit of reuse is the unit of release, it figures that modularity plays a prominent role here.
Understanding principles, patterns and practices (like SOLID and modularity patterns) that increase architectural agility help resolve the tension between use and reuse, and are certainly a step in the right direction. But so too is understanding that certain decisions must be deferred until we have the requisite knowledge to make the most informed decision possible. Because of this, we should strive to minimize the architectural significance (impact and cost) of change by making our designs as reversible as possible. Reversibility doesn’t always mean great flexibility, though. Sometimes it means we make something as simple as possible so that it’s easy to change later. Either way, it’s imperative to accommodate the natural shifts that occur throughout development, and modularity plays a central role in making this happen.
Moving On
For those that follow this blog, you’ll know it’s not the first time I’ve written about this topic. For others, if you’re clicking on any of the links in this post, you’re quickly discovering that, as well. Going forward, I intend to explore many of these concepts using some concrete examples that should offer a bit more insight to the discussions. I’ve put together some sample exercises for some upcoming conferences, and I intend to walk through those samples in a series of future posts. The result will be roughly seven or eight separate posts that show the evolution of a system. There’ll be code, builds, tests, and of course, modularity. Along with a lot of other stuff, too.
For now, if you’re interested in this topic, as well as ways to increase architectural agility, you might consider checking out some of my following entries (some of which are linked to above) related to this topic. You can bet there will be more coming, too!
Oct '09
2
Filed Under Agile, Development | 2 Comments
For those of you interested in Lean and Kanban, but not much knowledge yet, it can be tough filter the noise and find information explaining what they are and how they might be different from what you’re already doing (agile, Scrum, etc.). Since top 10 lists are rather passé, I’ve gone the whole nine yards and scoured the blogosphere to come up with the best nine posts I could find that capture the essence of Lean and Kanban.
The list is in no particular order, and there’s no doubt that I’ve missed some great posts that offer insight. The blogosphere is a pretty big place, after all. Naturally, I’ve cheated a bit by pointing to related posts from within the top nine. Enjoy!
- The Essence of Lean and Kanban - Post by that talks about Flow.
- Patterns and Practices of Lean Software Development - Post by @corey_ladas discussing principles such as One Piece Flow, Real Options, and Build Quality In.
- Noodling on Kanban - The first of a four-part series written by @mcottmeyer. Also be sure to check out Part 2, Part 3, and Part 4.
- How Is Kanban Different From Other Approaches - Post by Karl Scotland that examines Kanban’s primary practices. Also check out his follow-up post where he adds The Fifth Primary Practice of Kanban. And his posts on Kanban, Flow, and Cadence and XP as a Kanban System are great, as well.
- Kanban Applied to Software Development: From Agile to Lean - A detailed article on InfoQ that begins by exploring the origins of Kanban in the Toyota Production System, before launching into a discussion on Kanban in software development.
- Kanban in Action - An early post by @agilemanager (David Anderson) where he describes his initial forays into Kanban.
- I Have Heard of Kanban - A response to the critical post, Have you heard of Kanban, by Matthew Huesser. There were actually a few posts that were part of this conversation. It started with @chris_mcmahon calling into question Kanban with his post, against kanban. That led to Matthew’s post, and ultimately the response. Matthew has since posted a Kanban Redux and Chris has posted a couple of responses in against kanban part 2 and against kanban part 3.
- Bumping into Scrum’s Boundaries - Part 1 of @damonpoole’s Transitioning from Scrum to Kanban Series. Part 2 can be found in his post on One Piece Flow.
- The Poppendieck’s on Lean - An interview with Mary and Tom Poppendieck on Lean.
If you’re interested in learning more about Kanban, consider joining the Kanban Development Yahoo Group. Additionally, the Limited WIP Society is the home of Kanban Development, and contains some good definitions of what Kanban means within software development. And of course, the best place to learn more about Lean is to start with the Poppendiecks.
Note: Graphic courtesy of InfoQ. Used without permission.
Sep '09
25
Filed Under Agile, Development | 9 Comments
Joel tells the story of The Duct Tape Programmer, and Uncle Bob offers his response. Now these are two pretty smart guys who know a lot about software development. But when we receive fundamentally different messages from a couple of industry luminaries like Joel and Uncle Bob, we’re left wondering - “Who is right? Who should we listen to?”
Joel makes some valid points, but it’s strange that immediately after discussing the dangers of overengineering and the importance of shipping the product, he points out the following:
Zawinski didn’t do many unit tests. They “sound great in principle. Given a leisurely development pace, that’s certainly the way to go. But when you’re looking at, ‘We’ve got to go from zero to done in six weeks,’ well, I can’t do that unless I cut something out. And what I’m going to cut out is the stuff that’s not absolutely critical. And unit tests are not critical. If there’s no unit test the customer isn’t going to complain about that.”
I’m not quite sure how unit testing encourages overengineering nor how unit testing impedes shipping the product. Unit testing does neither. In fact, unit testing inspires a simpler design, not a more complex one. And because unit testing helps developers build quality into the product, I’m certain it doesn’t inhibit shipping the product - or at least a product that works. The correlation makes no sense, and I can’t help but wonder if the message wasn’t an intentional jab based on some past conflicts.
Of course, I certainly agree that favoring simplicity and getting the job done are righteous goals. Sometimes you have to cut a few corners, take a few shortcuts, and implement a solution that isn’t perfect. But there still exist fundamental practices of professionalism that must serve as our guide. And as Uncle Bob points out, unit testing is one of these practices. Bottom line! As a humorous post on twitter suggests:
never take software advice from a bug tracking system salesman
Unit testing is important, has significant long-term benefits, and developers should use them. However, I do take slight issue with Uncle Bob’s following statement:
I found myself annoyed at Joel’s notion that most programmers aren’t smart enough to use templates, design patterns, multi-threading, COM, etc. I don’t think that’s the case. I think that any programmer that’s not smart enough to use tools like that is probably not smart enough to be a programmer period.
In this respect, Joel is right. There might be a lot of people out there who aren’t smart enough to be programmers, but the reality is that only 50% of us can be above average. Of course, we all think we’re in the upper 50 percentile and when we’re looking at code written by someone else, it’s apparent to us it’s the other guy or gal who’s in the lower half. Without arguing who belongs where, the reality is that if we were all smart enough to use these advanced constructs, we wouldn’t be left with the mess we have today.
But really this only reinforces the importance of unit testing, which is why statements that denounce unit testing are so surprising. Think about it…Would you rather maintain a codebase with near 100% test coverage or a codebase with near 0% test coverage? Maybe those who can’t answer this question are the folks that shouldn’t be programmers!
Most of my discussions surrounding agile architecture have been focused on exploring how modularity helps increase architectural agility. I claim that modularity is a required (and to this point, missing) aspect of agile architecture. The basis for this claim follows:
- Architecture is design, but not all design is architecture (ala Booch).
- Design is architecture if the design is architecturally significant. That is, if it’s hard to change.
- The goal of architecture is to eliminate the impact and cost of change.
- The way to eliminate the impact and cost of change is through flexibility.
- With flexibility comes complexity. We must therefore strive to increase flexibility while taming complexity.
- Modularity helps us identify where we need flexibility by understanding the joints of the system where flexibility is necessary.
Without modularity, we can’t identify the joints so it’s more difficult to understand where we need the flexibility. My posts titled Modularity & Architecture and Modularity by Example show visual examples comparing designs that are isolated and insulated to designs that span the joints of a system. I’ve also devoted extensive discussion to these ideas in a number of other posts, which I summarize in my Agile Architecture Presentation post.
Lean Principles
Recently, I’ve been spending more time exploring lean software development principles and their relationship to agile architecture, and the best place to start when examining lean is with the Poppendiecks. I’m fascinated by the synergy that exists between the lean principles and agile architecture. In fact, when reading the second chapter of their book, “Implementing Lean Software Development: From Concept to Cash“, I was pleasantly surprised by an interesting discovery.
It seems that a study of software development practices by Harvard Business School professor Alan MacCormack revealed four fundamental practices that lead to successful software development. These include releasing early, continuous integration, experience and instinct, and a modular architecture. So it seems I’m not alone in feeling modularity is a critical component of agile architecture. But the thrust of the discussion comes later on in Chapter Two, when speaking of deferring commitment.
Deferring Committment, Reversibility, and Eliminating Architecture
Deferring commitment focuses on two fundamental factors - reversibility and irreversibility. In general, reversible decisions are those that can be changed while irreversible decisions are those that cannot be changed. We should strive to make irreversible decisions at the last responsible moment. For it is at this moment when we possess the most knowledge that will allow us to choose the most viable option. But we are also advised that, and I quote:
“First and foremost, we should try to make most decisions reversible, so they can be made and then easily changed.”
For me, this captures the essence of eliminating architecture. If we are able to take a seemingly architecturally significant challenge and make it reversible, then we have effectively minimized the impact and cost of change to a point where change is no longer architecturally significant.
Going forward, I intend to more fully explore additional synergies between lean software development principles and agile architecture.
I recently wrote about eliminating architecture, and there were a few comments, especially by folks on JavaLobby, who thought I had my head in the clouds. Too much theory. Too many abstract concepts. Not achievable in a real world development scenario. That I’m making a play on words. Let’s take another angle.
Defining Architecture
There are numerous definitions of architecture. But within each lies a common theme, and some key phrases. Here are a few of the definitions. From Booch, Rumbaugh, and Jacobson in UML User Guide (Addison-Wesley, 1999):
An architecture is the set of significant decisions about the organization of a software system, the selection of the structural elements and their interfaces by which the system is composed, together with their behavior as specified in the collaborations among those elements, the composition of these structural elements and behavioral elements into progressively larger subsystems, and the architecture style that guides this organization — these elements and their interfaces, their collaborations, and their composition.
From the ANSI/IEEE Std 1471-2000:
The fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.
In TOGAF, architecture has two meanings depending on context:
1.) A formal description of a system, or a detailed plan of the system at component level to guide its implementation
2.) The structure of components, their inter-relationships, and the principles and guidelines governing their design and evolution over time.
And in a QCon presentation by James Coplien and Kevlin Henney, architecture is defined as:
1.) Architecture embodies the critical design decisions that typify a system. Relates to cost of change, organizational structure, structure of code, capabilities of a system, etc.
2.) The significance of decisions needs to be understood and assessed. A heavy-weight approach is likely to reduce understanding and our ability to assess
Certainly, we see a pattern here as some key phrases and terms recur. These include “significant/critical decisions”, “components”, “structure”, and “cost of change”. And the definition from the Fowler article (which is actually part of a statement made by Ralph Johnson) introduced in the Eliminate Architecture post uses similar terminology and phrases.
It is these definitions that led us to the goal of architecture - eliminate the cost and impact of change. If we are able to eliminate the cost and impact of change, then change is no longer architecturally significant. If something isn’t architecturally significant, then we don’t consider it architecture. Essentially, we’ve eliminated architecture because the cost of change is no longer significant. That must be what we strive to achieve. And the way to achieve this is by increasing flexibility while taming complexity. Let’s take an analogy.
An OO Analogy
The goal of object-oriented design can be summed pretty effectively using the open-closed principle, which states:
“a system should be open for extension but closed to modification.”
This is done through abstraction and inheritance. If we have abstractions in the right place, we can extend the system by introducing new classes without modifying existing system classes. The key is that we have to recognize which areas of the system require this added flexibility.
Certainly not all areas of the system can possess this flexibility. It’s unrealistic and entirely too complex. But it should still be our goal. It’s the same with architecture. Obviously we cannot eliminate the architectural significance of all change, but it must still be our goal. Otherwise, what goal are we striving to achieve?
Modularity - The Missing Ingredient
Two of the key elements of the architectural definitions are component and composition. Yet there is no standard and agreed upon definition of component (reminding me of architecture, actually), and most use the term pretty loosely to mean just “a chunk of code”. But that doesn’t work, and in the context of OSGi, it’s clear that a module is a software component. That’s excellent fodder for my claim that agile architecture requires modularity because agile architecture demands that we design a flexible system that allows us to make temporal decisions based on shifts that occur throughout development. Modularity has been a missing piece that allows us to more easily accommodate these shifts.
I illustrate this using the simple diagram at right, and it’s what I was referring to in the Eliminate Architecture post when I stated that modularity, in conjunction with design patterns and SOLID principles, represent our best hope to minimize the impact and cost of change. It’s easier to change a design embedded within a module than it is a design that spans modules. I provided a similar example in Modularity by Example.
In other words, it’s easier to isolate change by insulating a design within a module. Designs that span modules are the joints of our system, and changes in these areas are more complex and costly. This is where we need the flexibility. It’s where we need stability. But if we don’t have modularity, we don’t have joints. Or maybe everything becomes a joint? Either way, without modularity we can’t identify the joints so it’s more difficult to identify where we need the flexibility. If we are able to avoid changes that span joints, that change is isolated…insulated…encapsulated within a module, and the impact of change is minimized..
Can we do this in all cases? Of course not! Just like we cannot always design software that’s open for extension but closed to modification. But it must be what we strive to achieve. Again, modularity has been the missing ingredient. And now is a perfect time to start understanding how to design more modular software.
← Previous Page — Next Page →
2010 Conferences
Tweets @ Twitter
LinkedIn Profile
Modularity Patterns
Recently Written
Categories
Archives