February 23 - Keynote titled OSGi in the Enterprise: Agility, Modularity, and Architecture’s Paradox
March 22 - 25 - Tutorial on Modular Architecture
June 14 - 17 - Sessions titled Turtles and Architecture and Patterns of Modular Architecture
July 26 - 30 - Two sessions on rich mobile applications and one on agile development. Half day tutorial on software process improvement.
Right on. I just won a Best Buy drawing worth $1000. Either that or I won a shiny new virus by clicking the link. Hmm...what to do. 2012-08-29
The opinions expressed on this site are my own, and not necessarily those of my employer.
Below is a table that shows some popular and emerging programming languages classified according to the following:
Is this a relevant classification scheme? Are languages classified correctly? Are certain qualified languages missing from the list? How would you modify this list?
For the first part of the story, see my blog post titled OSGi Discontent - No Migration Path.
I’m a bit surprised by the response I’ve gotten about that post. There has been healthy discussion on Javalobby, with folks standing on each side of the debate. Eric Newcomer has posted a partial rebuttal, stating that he validates my technical analysis and concern about lack of tooling and a migration path, but that I’ve drawn an incorrect conclusion. Ian Skerrett also supports my observation with his latest tweet.
I thought I was stating something that was pretty obvious. OSGi isn’t ready for enterprise web development. Not today. The problem isn’t the OSGi specification. It’s not Equinox or Felix. The problem is the current generation of application platforms and tooling. I recognize that OSGi can be used to develop enterprise applications, but it’s cumbersome to do so. Early adopters can leverage OSGi, but they must be willing to incur the complexity and cost of doing so. Most organizations aren’t willing, nor should they be willing, to do that. That is my conclusion.
Eric does make a key point. We aren’t dealing with black and white here, but instead shades of gray. That reinforces my point. Today, organizations that want to leverage OSGi may decide it’s not feasible because of lack of tooling and platform support. As the technology and tools mature, more organizations will decide they can leverage OSGi. That’s a good thing, especially for the large and complex enterprise projects that could stand to benefit from modularity, but aren’t willing to incure the technical burden required today.
Keep in mind, I’m an ardent supporter of OSGi. I want it to succeed, and I believe it will. I’ve posted quite a few blog entries about OSGi, including a number of simple examples that illustrates it’s capabilities. The intent of my most recent post was to illuminate some deficiencies. The benefits it will bring to enterprise development, especially for large software systems, is substantial. But unfortunately, the cost to leverage OSGi today is high. High enough, in my opinion, that it’s simply not feasible for most organizations.
OSGi has emerged as the de facto standard for modularity on the Java platform. The Eclipse RCP plug-in architecture is built on Equinox. Major application server vendors, including IBM, Oracle, and JBoss are leveraging OSGi to increase the modularity of their platforms. The ability to dynamically provision server-side modules reduces footprint and decreases start-up time. Without question, OSGi will play a major role in the next generation application platform.
In my last post where I embedded OSGi (Equinox) in Tomcat, I promised to provide a synopsis of my experiences thus far with OSGi. I’ve come to a disturbing conclusion - OSGi is not ready for the enterprise. With so much industry momentum, OSGi is not ready for enterprise web application development! The reason is because enterprise developers don’t have the tools to build OSGi based server-side applications, and vendors aren’t exposing the capabilities of OSGi to enterprise developers. Neil says that I should be able to name 10 things I don’t like about OSGi. Well here’s six things I don’t like today when using OSGi to build enterprise applications. But first, some background.
Understanding the root of this problem demands we examine the two ways to build web applications using OSGi. The first is Host OSGi. The second is Embedded OSGi.
The differences may seem subtle, but have significant ramifications on how we develop and manage enterprise applications.
Why is this important? Foremost, it allows us to build more modular applications. The bane of many enterprise development efforts is the lack of modularity, the excessive (and often unknown) dependencies, the ripple effect of change, and the design rot that ultimately brings the system to it’s knees. Beyond this, a platform built atop OSGi can remove the artificial application boundaries that exist today. By separating the unit of deployment from application and process boundaries, we open up amazing flexibility in how enterprise applications can adapt, shrink, grow, and morph over time.
Host OSGi is the preferred scenario, but today, SpringSource dm Server and Paremus Infiniflow are the only products I’m aware of that provides the capability. While other major application platform vendors are beginning to build OSGi into their platform, they are not exposing the capabillities to enterprise developers. Until application platform vendors expose the capabilities of OSGi to the enteprise, our only option is Embedded OSGi, which can be tricky and difficult. Unfortunately, the enterprise is left building applications on many of the traditional application servers - WAS, WebLogic, and JBoss - and this poses challenges.
Attempting to develop enterprise web applications today using OSGi presents significant challenges. In fact, these challenges are significant enough that they are severely impeding OSGi adoption within the enterprise. While the OSGi framework implementations themselves are capable, the surrounding ecosystem is not yet mature enough.
OSGi was once heralded as the most important technology of the decade, and the majority of enterprise software developers can’t even use it yet. OSGi proper may be ready for primetime, but we have a long ways to go before the enterprise is able to develop web applications leveraging OSGi. The problem isn’t the OSGi specification, nor the popular Equinox and Felix implementations. Each of these are waiting and ready. Unfortunately, there is no easy way to build modular applications using OSGi and deploy them to the current generation application servers. There are no frameworks that help me do this. There are no tools that make this easy. There is no migration path that helps developers move away from building monolithic Java EE applications and move toward building the modular applications of tomorrow.
Ultimately, we need a migration path. As OSGi is built into the platform and exposed to enterprise developers, we need tools and patterns that allow us to build and manage more modular applications. These tools must allow us to take advantage of OSGi in building more modular applications today, and help us prepare for the platforms of tomorrow so that we are ready when they arrive.
If we are able to architect modular applications today, built using OSGi, then we are prepared for what tomorrow brings as we migrate to application platforms that Host OSGi. Just because the application platform doesn’t currently expose OSGi to enterprise developers doesn’t mean we shouldn’t be striving to create more modular applications. There is benefit to this of itself. And because OSGi bundles are backward compatible and capable of running in today’s application servers, what should prevent us from designing more modular software. Shouldn’t we be doing this today? Until there are tools available that help us do this, it’s not likely to happen. And that’s going to make leveraging the capabilities of the next generation platform more difficult, even after it’s arrived.
OSGi is an important and valuable technology that can help enterprise developers create more modular applications. The OSGi Tool Summit is being held Friday (03/27/09). Hopefully, this will result in another small step toward being able to leverage OSGi in the enterprise.
About 10 years ago I recall studying profusely so that I might pass my Java programmer’s certification exam. I purchased a copy of an exam cram book that had sample questions similar to what I might encounter on my certification test. I recall dealing with questions like the following:
I’m sure many of us are familar with these questions, as well as others of similar ilk. Of course, I’d taken similar exams prior to this for a variety of technologies, but the Java certification exam was the last one I’ve taken. I’d come to the realization that time spent studying for these exams is largely a waste of time. I was no better at developing software systems in Java after taking the exam than I was before. I simply possessed a bunch of useless knowledge that I could have easily gotten by looking at reference material.
The skill that I needed to develop enterprise applications could only be obtained by developing applications. It could be found no other way. There are no shortcuts. Making mistakes and fixing them. Discovering new ways of doing things that worked better than what I had done before. Knowing what assumptions I could make, and those I could not. The reality is that certifications exist largely to sell training courses, while the marketing that surrounds them leads organizations to believe they are relevant. A while back, the agile alliance issued their public policy denouncing agile certification. They state:
Knowledge is a wonderful thing, but businesses pay for performance. Performance requires skill.
A skill is not as simple to acquire as knowledge: the learner has to
perform the skill badly, recover from mistakes, do it a bit better, and
keep repeating the whole process. Especially for the interrelated and
interpersonal skills required of Agile software development, much of
the learning has to take place on real projects. It is that learning
that a certification should vouch for.
And unfortunately, it is that learning and skill that certifications do not provide nor account for. In fact, this is exactly why the software craftsmanship movement has begun to gain so much traction. Through trial and tribulation, a software craftsman has obtained the practiced skill required to develop great software. Of course, it’s easy to dilute the notion of craftsmanship if we are not careful.
Tom DeMarco also published an open letter to Cutter IT Journal expressing his dismay with certification. The debate on certification certainly is not new. But with the Scrum certifications, among the many others that abound today, the debate is still relevant. Where do you stand on certification? Would you rather work with a craftsman or someone with a wall plastered with certifications?
By the way, if you stand on the side favoring certification, and would like to get an agile certification, then maybe you should visit Agile Certification Now to get started.
In Big Teams & Agility, I talked about a macro process for agile development on large teams (those up to, possibly exceeding, 100 developers). The article was posted on AgileZone at JavaLobby (where I now help out as zone leader, though did not at the time the article was posted), where I was accused of having my head in the cloud. While I could construe that as a compliment given the buzzwords du jour, I probably shouldn’t make that mistake. So I want to take a moment and respond because there were some good points made, and clarification is necessary.
First, you can apply agile practices on big teams, and it does work. I was pretty clear that I’ve used the structure on teams up to 100 developers, and have done so for a good share of the work I’ve done since 2001. It’s worked marvelously. However, I never said it was easy. In many ways, it’s the most difficult approach to developing software I have ever taken, but it’s also the most successful. One question posed by a commenter follows:
Do you really expect big company to go through integration/testing/whatever every week?
Absolutely! The more often you integrate, the earlier you’ll discover problems before they’ve had an opportunity to fester within the system for prolonged periods of time. I recognize this is against the grain of conventional wisdom we’ve been taught for decades. As I mentioned in the original post:
The economies of scale lead us to believe we need longer iterations
because there is so much more to manage. But that’s flawed because it
delays risk mitigation and discovery.
At the end of the post, I stated that there were many micro process details omitted, such as how to keep the build running quickly. To address slow build times on very big systems, you may have to implement staged builds. A staged build is basically a pipeline of builds that perform different build activities.
For instance, a stage 1 build might perform a subset of the overall build steps to provide rapid feedback to the team. A stage 1 build is performed hourly or anytime new code is checked into the source code repository. A stage 2 build is a more complete build process. It might integrate all system components, or execute a complete suite of tests. The actual tasks are going to vary given the context, but the idea remains the same. It’s ok to have multiple build processses for a single software system. It’s up to the development team to identify the components of each build, and it may change throughout the life of the project. But the key element is the rapid feedback the team receives.
Another point of contention with the article centered around the misunderstood suggestion to release frequently to the customer, and the perceived lack of QA, acceptance testing, etc., etc. In fact, I stated the exact opposite, and it’s the continuous integration strategy that allows us to close the loop and perform these types of testing frequently. I stated pretty clearly the following:
We should also frequently execute a variety of tests. Not just unit and
acceptance tests, but usability tests, performance tests, load tests,
The key element here is that we increase project transparency because we have the ability to get our product in front of the customer on a frequent basis. The customer experiences the growth of the application with the development team. They see it’s evolution, and can provide valuable feedback along the way. There are fewer surprises at the end of the project.
However, I have never said that each build should be released to the customer for use as a production software system. Never! I did say the following:
Once the build executes successfully, the application can be deployed to an environment where it’s accessible by the customers.
The environment is likely a test environment that the customer has access to, and can experiment with the system to provide feedback. Or a QA environment where acceptance testing can be performed. It’s also a place that can be used for system demonstrations. I recognize the pain in delivering large enterprise software systems, and also realize the impossibility of releasing each build to production. But we should be striving for that level of quality each time we write a line of code.
Regardless, the key takeaway is that because we always have a functional system, we are able to perform various lifecycle activities at any time, and many times, throughout the development effort. This increases project transparency where the customer, management, and developers have a consistent understanding of the current state of the system. We avoid those nasty surprises late in the development effort that plague many projects.
This is not overly zealous. It is not unrealistic. It is not some abstract theory born of academia that has never been proven in the real world on a large enterprise development effort. In fact, while it adopts various practices from popular agile methods, it is not Scrum, nor is it XP. But it is agile, and it does work. It captures the essence of agile development - rapid feedback through software that works while maintaining the ability to respond quickly to change. This is, quite simply, one of the best approaches to software development I’ve ever used. But it’s not easy, nor will it ever be easy. Software development is hard work, and it will always be hard work. The comments are welcome, the questions are valid, and the discussion is important.
In Grass Roots Agile, I talked about some of the details surrounding how development teams can increase their agility, and I presented a diagram similar to what’s above that discusses how to measure and manage a system’s tested features. Here, I want to talk a bit more about the macro development process, and how I’ve used agile practices on large software development teams. I’ve used this structure with teams up to roughly 100 developers. I have no reason to believe it wouldn’t work on larger teams, though. Here’s the general mechanics illustrated by the diagram.
One of the keys to building big software projects is to break it down into a bunch of smaller projects. Each team in the diagram (Team 1, Team 2, etc.) represents a group of people working on one of these smaller projects. The smaller projects are organized around coarse grained units of business functionality and each team focuses on developing the complete unit of functionality (front to back).
Each of these teams consists of around two to five developers, a business analyst (BA), tester, UI designer, and customer. Depending on the size of the effort, some of these individuals may span teams. A project manager spans all teams. The developers on each of these teams focus on some aspect of technology expertise. Examples include a JSP expert, Hibernate expert, etc. These technology experts form alliances with experts on other teams, which is critical to ensure consistency with how the technology is applied project-wide (ie. the architecture). The technology experts will also spend some time on infrastructure code, such as system-wide error handling utilities and other important architectural aspects. Naturally, you adjust the size of the teams and the roles of team members based on various factors.
The feature board is a virtual concept that represents the stream of requirements that flows from the customer to the development teams. It is the responsibility of the BA to make sure there are an adequate set of requirements on the feature board, to resolve conflicts, and manage change. Everyone on the individual teams should have the option to participate in discussions surrounding requirements. The feature board could be your typical project room whiteboard, but for large teams, that’s probably not going to work because you have a group of geographically dispersed developers.
Instead, the feature board is simply a snapshot of requirements, and each team pulls those requirements and starts working on them when ready. These requirements might come in the form of user stories, use cases, or something else. The format used is not as important as the fact that the feature board is organized in a way that each team can pull the next set of requirements for the part of the system they are working on when they are ready, while allowing project management to see a complete view of the system requirements.
One of the challenges with a large team is that it’s virtually impossible to get everyone aligned on the same iteration schedule. There is too much management overhead. This is one of the driving forces behind why skeptics don’t feel agile is a good fit for really big projects and teams. The economies of scale lead us to believe we need longer iterations because there is so much more to manage. But that’s flawed because it delays risk mitigation and discovery.
In practice, there isn’t a need to have each project team on the same iteration schedule, and in fact, because the teams pull requirements from the feature board when necessary, there isn’t a need for iterations at all. Instead, there is a continuous stream of work that flows from the customers to the development teams. As a team completes a piece of work, they simply release the code to the version control system, build it, and feed it right back to their customer in form of a functional system.
With each team releasing code, there is a continuous stream of feature rich functionality that is added to the system. Continuous integration and the automated build process holds it all together. Because we are building on a frequent basis, we’ve got a system of checks and balances in place to make sure no breaking changes enter the main product line.
Occassionally, we’ll hit a situation where two teams release incompatible changes. That’s the purpose of the build! When this does happen, the build breaks, the problem is identified, the teams fix the problem, and we’re back on track. This is a central part of the macro process. Without the automated build, it all falls apart. Once the build executes successfully, the application can be deployed to an environment where it’s accessible by the customers.
Adopt all the agile processes and practices you want, but increased agility completely falls apart unless you close the feedback loop. I’ve seen too many teams fail because they’ve stopped just short. Feedback is the crucial piece to ensure we close the loop, and because we’ve got a continuous integration process in place due to the automated build, we are able to effectively close the loop because we’ve always got a product that works. It’s important to leverage this. Eliciting frequent feedback from customers through weekly demonstrations is one way to close the loop.
We should also frequently execute a variety of tests. Not just unit and acceptance tests, but usability tests, performance tests, load tests, and more. Because the build is an automated process, we should incorporate code analysis and inspection tools into the process and output the results to a project dashboard that is accessible by the project team. In the end, closing the loop is going to help significantly in increasing project transparency.
This is a very broad overview of how agile can work on big teams. In fact, I believe agile is more beneficial for large teams than it is for smaller teams. Small teams are inherently more nimble than bigger teams in the first place, and so agile is an obvious and natural fit. But on large teams, it’s not so obvious how agile practices can improve the team’s success. Above is one formula I’ve found that works very well.
Without question, there are numerous micro process mechanics that I haven’t discussed here. For instance, how often should the build run? (I say at least hourly). For really big systems, how do I keep the build running quickly? (You may need staged builds). What tools do I use to make this all happen? (You don’t need to buy anything, actually). And much more. But in general, the macro process above is a good way to get started.