February 23 - Keynote titled OSGi in the Enterprise: Agility, Modularity, and Architecture’s Paradox
March 22 - 25 - Tutorial on Modular Architecture
June 14 - 17 - Sessions titled Turtles and Architecture and Patterns of Modular Architecture
July 26 - 30 - Two sessions on rich mobile applications and one on agile development. Half day tutorial on software process improvement.
Right on. I just won a Best Buy drawing worth $1000. Either that or I won a shiny new virus by clicking the link. Hmm...what to do. 2012-08-29
The opinions expressed on this site are my own, and not necessarily those of my employer.
After the OSGi DevCon keynote, I had a few folks stop by and ask me for the slides. They are now available on slideshare. Usually, when browsing the slides without discussion, the lack of context can cause some confusion. So if you have any questions regarding the content, please feel free to reach out. Enjoy!
I’ll be at OSGi DevCon next week, having been given the opportunity to present the keynote session on Tuesday. The conference is co-hosted with JAX London. I thought I’d take a moment before jumping on the flight to offer a preview of what I’ll be speaking about.
The title of the session is OSGi in the Enterprise - Agility, Modularity, and Architecture’s Paradox. We all know that OSGi isn’t a new technology. It’s been around for over a decade. But the recent surge in interest in modularity on the Java platform is overdue. The complexity of software is increasing exponentially. Did you know:
Let’s put this in perspective using the visual at right (click to enlarge), where we can see that the amount of code doubles every seven years. Pretty staggering! But let’s put this in perspective for what it means over the course of the next seven years, as well. It means that between 2010 and 2017, we’ll produce more code than the total amount of code ever written combined! That’s staggering!
I decided to do a bit of research to see if I could find examples that supported these claims. I did:
I’m sure many of us have experienced this phenomenon. Systems tend not to get smaller over time. And we also recognize that in almost every way, larger software systems are inherently more difficult to design, build, manage, and maintain than are smaller software systems.
Yet, such phenomenal growth is desirable. We can only hope that the systems we create are in such high demand that we have the opportunity to grow them over time. A software system that doesn’t change, dies. Evolution is a big deal! As Lehman’s law suggests:
As a system evolves, its complexity increases unless work is done to maintain or reduce it.
Add it all up, and there are some key takeaways here. We need something that will help us understand complex systems. We need something that help manage the complexity. We need something that will help ease maintenance. We need something that will help us deal with the natural evolution of software systems. We need something that will allow us deal with the natural architectural shifts that occur as a system grows to accommodate demand. For a long time, a central ingredient has been missing. But not for much longer, because the enterprise will get its OSGi!
For the rest of the story, see you in London.
Source of statistical information above: http://users.jyu.fi/~koskinen/smcosts.htm
It’s housecleaning time again, and like last time, I stumbled across an article I wrote back in 2006 that I don’t believe ever reached publication (at least, I don’t think it did…how am I expected to remember what I did in 2006?). For the most part, I’ve left it in its original state, except that I removed the Agile Manifesto and 12 supporting principles. There are easily enough found on the Agile Manifesto website, and the article is long enough without this duplication. The wordle at right shows the most common words used in this document (click to enlarge). Here, in it’s otherwise unadulterated glory, is Agile: A New Era of Software Development.
Writing code is easy, but developing software is hard. While syntax errors are common, their severity pales in comparison to the logic flaws inherent in many software systems. The difficulty in software development is not writing code or applying a certain technology stack. Instead, the challenge lies in the specification and design of the software itself. Therein lies the essential complexity of software development, an idea introduced by Frederick Brooks in his 1987 article titled, “No Silver Bullet” [Brooks]. The most difficult aspect of software development is deciding what, exactly, needs to be built.
There is certainly evidence backing this claim. The original Chaos Report shows the top three impediments to a successful development effort are lack of user input, incomplete requirements and specifications, and changing requirements and specifications [CHAOS]. No other activity, if done incorrectly, stands to compromise the system more than incorrect requirement specifications.
It might not be so difficult were software a concrete entity, existing in a world where we could easily visualize it’s structure and behavior, allowing us to more reliably assess and share the impact of change. But software is a highly conceptual, invisible construct. It is considered infinitely malleable by those not intimately familiar with the conceptual complexity of it’s structure and behavior. The contractor building your home would look at you with incredulous disbelief if you suggested that the house he has 90% complete no longer met your needs, and you asked that he move walls. Or imagine how ridiculous it would sound to suggest that a new third floor be inserted to a 100 story skyscraper. Physicists labor on with firm belief that there exist an underlying set of unifying principles to serve as guidance. Or at least, there are laws of physics that we hold to be true. There are no such rules or principles that guide software development. We are left with the imagination and dreams of our clients, and they demand and deserve rapid response to change.
We have made valiant attempts at conformity. Ceremonial processes attempting to define standardized activities that guide the development process have failed, however. We cannot define detailed up-front requirements specifications and expect them to survive the development lifecycle intact. We cannot establish an initial design of the conceptual construct and expect the structure to go unscathed throughout the process of construction. Software development is an error prone human activity involving experts with varying backgrounds and skills who must come together and attempt to communicate uniformly, working as a team toward a common goal. While tools and process do help, we must also accept that change is expected. We cannot treat change as evil. Instead, the tools and process used must allow us to accommodate change, treating it as an inherent part of software development. Changing requirements is a rule of our game. The software we develop must be malleable and adaptive to change, and the process we use must embrace change.
We often draw comparisons between software development and various manufacturing processes. As Larman points out, however, manufacturing is a predictive process [Larman]. Herein lies one of the greatest differences between software development and the manufacturing processes to which we often draw comparisons. Manufacturing is a repeatable activity, with high rates of near-identical creation where a consistent product is produced in assembly line fashion. Little change is expected, making it possible to reliably estimate and predict the outcome. Software development is much more like new product development, where evolutionary specifications and adaptive planning is necessary to deal with the many unknowns that lie ahead.
In early 2001, a small group of industry experts converged in Utah to discuss alternatives to heavy, document driven software development methods. Emerging from this meeting was the Agile Manifesto, a symbolic proclamation endorsing the virtues of a lighter, more flexible, people-oriented approach to software development, giving birth to the agile software development movement. (Since this is already a long article, I’ve snipped the manifesto and principles, which were included in the original version. If you’re interested, you can view the manifesto and its 12 principles on the Agile Manifesto website.)
The ideas behind these 12 principles are simple, and contain no hidden messages. Of course, there are different techniques embodied within various agile processes that support these principles. The one certainty is that agile teams definitely work differently from their less agile peers. They recognize there is one end goal - to create a working, functional software product. With that in mind, they work very closely with project stakeholders throughout the development lifecycle, knowing it is the stakeholders who possess the knowledge the system must embody. Agile teams work very hard to deliver working software iteratively and incrementally, and they adopt techniques representative of that ideal.
Agile project managers tend to favor intense communication and collaboration over heavy documentation. Empowering team members to make decisions enables responsiveness to change. Facilitating and negotiating requirements scope provides important feedback, helping plan future iterations, where each iteration produces a deliverable that can be shared with clients and stakeholders. Instead of forcing the team to follow a predictive project plan, agile project managers are more opportunistic. They prioritize features based on stakeholder feedback, and make adjustments as the iterations progress. Concurrent and parallel activities are favored over a phased approach. Agile project managers tend to guide the team instead of manage the team, and strongly discourage unnecessary overhead.
Agile developers work with a similar set of goals, knowing functional software must be delivered early and often. They work judiciously to grow a code base built upon a solid foundation, where each day represents a step forward. They integrate frequently, and do not tolerate failed builds. A rich suite of tests provide the courage necessary to respond to change when the need arises. They avoid the notion of code ownership, empowering other developers to make improvements to any component of the software product.
A common misconception is that agile processes discourage all documentation. This is untrue. Agile processes discourage unnecessary documentation, favoring collaboration as the preferred technique. Instead of using documentation to drive communication, agile processes favor face-to-face communication. Documents are encouraged by agile processes, so long as the need is immediate and significant.
Agile software development is based upon the fundamental premise that we must drive and respond to change quickly. The Agile Manifesto and 12 supporting principles serve this premise well. Advocates of agility claim speedier delivery of software, software with more business value, increased team productivity, higher quality systems, and a more enjoyable development experience. I believe each of these to hold true. Agile teams not only welcome change, they are able to respond to change at all levels of development. A project manager might discuss a changing requirement with a business client, empower a business analyst to schedule a meeting with the client to discuss further details, while a developer assesses the impact of change knowing she has the courage to accommodate the request because of the rich suite of unit tests in place.
Saying you’ll be more responsive to change and creating an environment that embraces change are separate beasts, however. Practicing agility is hard work, especially if your team is accustomed to more traditional approaches. As with many things new and unfamiliar, some resistance will no doubt arise by those who aren’t fully convinced. Agile projects differ greatly from their less agile counterparts, and skeptics will have many opportunities to express their discontent. As someone experimenting with agility, you may even have doubts. But don’t be discouraged, and give your agile transition the time it deserves.
One of the most significant changes you may experience is a feeling that you’ve been thrust into a chaotic nightmare. I doubt it’s unusual to feel this way. You’ve lost the security of detailed requirements specification and user sign-off. You are writing code without the comfort of knowing exactly what your stakeholders want. The detailed plans that have served as your security blanket on past projects no longer exist. And the celebrations accompanying completion of your various phase milestones are gone. Of course, these were all false comforts anyway. Stakeholders always changed their minds. Your detailed requirements and plans were outdated as quickly as they were completed.
Instead, you’re now working in shorter iterations with vague requirements. Initial releases early in the lifecycle may be completely thrown away. Your first few weeks may seem wasted, with little or no valuable artifacts produced. Naysayers will immediately come forward and cite the lack of progress. Previously, those first few weeks or months were spent producing very detailed requirement specifications and beautiful design models. But don’t give up yet. In that previous world, you were only delaying risk and postponing integration, avoiding the most difficult aspect of software development until the end of the lifecycle. Now you’re attacking risk early, prioritizing features, and working hard to develop a functional piece of software as early as possible. Progress may not be at breakneck speeds, but you are learning a tremendous amount about the requirements of the system, and your velocity is sustainable. Additionally, you are also performing a number of other important lifecycle activities.
Depending on the level of ceremony and bureaucracy within your organization, you will experience varying degrees of success when adopting agile techniques. As with any new technology adoption, it’s best to phase the transition. Some agile techniques are easier to adopt than others, and some serve as valuable catalysts to adopting additional techniques in the future. Don’t attempt to completely redefine how you work. It’s relatively easy to phase the agile transition, and you’ll want to adopt those principles that offer you the greatest initial reward.
For instance, if you’re struggling to produce quality software at a consistent rate, implementing a continuous integration strategy will help you frequently verify the quality of your work. In addition to the comfort of knowing you have a product always in a functional state, the ability to share the product with clients using functional demos and prototypes will tighten the feedback loop and offer valuable insight to the client’s perception of the software. In a number of cases, I’ve found this to be valuable in identifying subtle requirements that can be difficult to identify in other requirements elicitation venues.
In recent years, there has been a significant amount of research comparing agile development methods to their waterfall counterpart. In Agile and Iterative Development: A Manager’s Guide, Craig Larman illustrates the advantage of agile development through detailed analysis of multiple studies[Larman]. The compilation of his results are illustrated below.
A study by Alan MacCormack at Harvard Business School explored whether evolutionary development techniques yielded better results than the waterfall model. The study included applications ranging from application software to embedded systems, with median values of nine developers spanning a 14 month development cycle.
A key conclusion of the study, in which 75% of participants used agile techniques compared to 25% using waterfall, explained releasing software earlier, rather than later, contributed to a lower defect rate and higher productivity. There was little evidence showing that a detailed design specification resulted in a lower defect rate, however, reviews with peers did help in reducing the rate of defects. The study found that iterative and agile practices have a significant impact on defect and productivity factors, as indicated by the following points.
The study also found four practices that were consistently used by the most successful development teams. The first two are deeply embedded in the ideals of agile software development.
In a separate study [Shine], 88% of organizations cited improved productivity when using agile methods, and 84% cited improved productivity. 49% stated that the cost of development was less when using agile methods. Additionally, 83% claimed increased business satisfaction and 26% claimed significantly better satisfaction. Another study by Boehm and Papaccio [Boehm] discovered that a typical project experiences a 25% change in requirements, while yet another [Johnson] showed that 45% of features were never used.
There have also been many research efforts devoted exclusively to the analysis of waterfall methods. Below is a summary of these findings, taken from a variety of studies.
As these studies clearly illustrate, there is significant evidence showing that agile and iterative techniques offer significant advantages over the waterfall model of development. In fact, for larger projects, the statistics supporting agility were even more pronounced.
There are a variety of agile processes available to choose from, and each abide by the spirit of the manifesto and it’s 12 supporting principles. The agile movement and it’s supporters recognize that software development is a human (though not always humane) activity. Instead of forcing process on people, agile methods allow process conformance to people. Good people, working toward a common goal, can achieve great things will little ceremonial process, assuming you give them an environment that empowers them. Solid empirical evidence backs this claim. And if the quality of people is in question, it’s doubtful that any process will produce success.
On July 25th, 2000, flight 4590 crashed. It was the first, and only, crash of the famed Concorde. Eventually, it would lead to retirement for the amazing aircraft. Investigators spent countless hours poring over the wreckage, and placed blame on a piece of runway debris that slashed the tire. A piece of that tire struck one of the fuel tanks, causing it to rupture and the plane caught fire. Case closed, right? Not so fast.
Surely a small piece of runway debris shouldn’t bring down a commercial airliner! As it turns out, there is quite a bit of contention among experts surrounding other factors that may have contributed to the crash. Some argue that it was a complex chain of events, all coming together, that brought down the aircraft. The plane was missing a spacer between the two wheels, had too much fuel in the tank, attempted to takeoff in unstable conditions, and was overweight. The flight was also delayed, causing angst among the flight crew. Finally, a required daily runway inspection was not performed.
Perhaps, if the runway inspection had been performed, the piece of debris would have been spotted. Or had the aircraft not been overfueled, the piece of tire may not have caused an increase in fuel tank pressure that some say caused the tank to burst. Had the spacer been installed, it’s possible the tire would have never burst. A series of small events, each contributing in their own way, to the fatal crash.
The story reminds me of the importance of attention to detail in software development. Of how the aggregate of all of the small, seemingly insignificant decisions we make on a continuous basis can have long-term consequences on the future of the software system. Possibly even your organization. Every time you design a class, define a variable, write a test, create a package, build a module, modify a method, or make a design decision, you are affecting the future of your system in some unsuspecting way.
What may seem insignificant today, can have a detrimental affect tomorrow. Really, there are no small, insignificant decisions in software development. I’m reminded of how important it is to make conscious decisions that are given careful thought, no matter how small. It also reminded me of a poem on build automation and continuous integration that I read a while back on the Test Early blog.
FOR WANT OF A BUILD
For want of a build, a test case was not executed
For want of test case, a defect was not detected
For want of a defect report, a bad release was promoted
For want of a good release, a strategic customer was lost
For want of a customer, a development team was reduced
For want of developers, a product stagnated
For want of a product, a company was lost
And all for the want of a build…