Below is a table that shows some popular and emerging programming languages classified according to the following:

• Type system - Dynamic or Static type system
• Problem space - A General Purpose language versus a Domain Specific Language
• Runtime environment - A Managed environment (ie. garbage collection, etc.) or an Unmanaged environment.
• Paradigm - Object-Oriented, Procedural, Functional, Imperative, or Declarative

Is this a relevant classification scheme? Are languages classified correctly? Are certain qualified languages missing from the list? How would you modify this list?

For the first part of the story, see my blog post titled OSGi Discontent - No Migration Path.

I’m a bit surprised by the response I’ve gotten about that post. There has been healthy discussion on Javalobby, with folks standing on each side of the debate. Eric Newcomer has posted a partial rebuttal, stating that he validates my technical analysis and concern about lack of tooling and a migration path, but that I’ve drawn an incorrect conclusion. Ian Skerrett also supports my observation with his latest tweet.

I thought I was stating something that was pretty obvious. OSGi isn’t ready for enterprise web development. Not today. The problem isn’t the OSGi specification. It’s not Equinox or Felix. The problem is the current generation of application platforms and tooling. I recognize that OSGi can be used to develop enterprise applications, but it’s cumbersome to do so. Early adopters can leverage OSGi, but they must be willing to incur the complexity and cost of doing so. Most organizations aren’t willing, nor should they be willing, to do that. That is my conclusion.

Eric does make a key point. We aren’t dealing with black and white here, but instead shades of gray. That reinforces my point. Today, organizations that want to leverage OSGi may decide it’s not feasible because of lack of tooling and platform support. As the technology and tools mature, more organizations will decide they can leverage OSGi. That’s a good thing, especially for the large and complex enterprise projects that could stand to benefit from modularity, but aren’t willing to incure the technical burden required today.

Keep in mind, I’m an ardent supporter of OSGi. I want it to succeed, and I believe it will. I’ve posted quite a few blog entries about OSGi, including a number of simple examples that illustrates it’s capabilities. The intent of my most recent post was to illuminate some deficiencies. The benefits it will bring to enterprise development, especially for large software systems, is substantial. But unfortunately, the cost to leverage OSGi today is high. High enough, in my opinion, that it’s simply not feasible for most organizations.

OSGi has emerged as the de facto standard for modularity on the Java platform. The Eclipse RCP plug-in architecture is built on Equinox. Major application server vendors, including IBM, Oracle, and JBoss are leveraging OSGi to increase the modularity of their platforms. The ability to dynamically provision server-side modules reduces footprint and decreases start-up time. Without question, OSGi will play a major role in the next generation application platform.

In my last post where I embedded OSGi (Equinox) in Tomcat, I promised to provide a synopsis of my experiences thus far with OSGi. I’ve come to a disturbing conclusion - OSGi is not ready for the enterprise. With so much industry momentum, OSGi is not ready for enterprise web application development! The reason is because enterprise developers don’t have the tools to build OSGi based server-side applications, and vendors aren’t exposing the capabilities of OSGi to enterprise developers. Neil says that I should be able to name 10 things I don’t like about OSGi. Well here’s six things I don’t like today when using OSGi to build enterprise applications. But first, some background.

Building for the Web

Understanding the root of this problem demands we examine the two ways to build web applications using OSGi. The first is Host OSGi. The second is Embedded OSGi.

• Host OSGi - With this approach, OSGi is the host environment, and the advantages of modularity are brought to the entire application platform. For an example of this approach, see my post where I embed Jetty within Felix.
• Embedded OSGi - With this approach, OSGi is not embedded within the platform, but instead is embedded within each web application we build. For an example of this approach, see my post where I embed Equinox in Tomcat.

The differences may seem subtle, but have significant ramifications on how we develop and manage enterprise applications.

• With Host OSGi, I don’t need to embed an OSGi framework into my web application. It’s already been embedded for me by the application platform vendor. All I need to do is build atop it and take advantage of what OSGi has to offer. For instance, when embedding OSGi, I need to manage starting up and shutting down the OSGi framework. With Host OSGi, the application platform would do it for me.
• With Host OSGi, I’m no longer limited by the coarse grained WAR and EAR units of deployment. Instead, I have an option of finer grained control over how I deploy and manage enterprise applications and components.
• With Host OSGi, I’m not constrained by application boundaries that limit and prohibit class sharing between applications. The classes within a JAR file are available to the entire environment, whereas today, they are constrained based on the WAR or EAR file in which the JAR is packaged.
• With Host OSGi, the environment is adaptable, whereas with Embedded OSGi, only my application is adaptable. Vendors are leveraging OSGi to build an adaptable platform. For instance, if an EJB container isn’t needed, it’s not loaded. No longer is an EJB container loaded simply because I’m using a Java EE application server. The platform will adapt based on the needs of the application. I’d like this same level of adaptability for applications that I create.

Why is this important? Foremost, it allows us to build more modular applications. The bane of many enterprise development efforts is the lack of modularity, the excessive (and often unknown) dependencies, the ripple effect of change, and the design rot that ultimately brings the system to it’s knees. Beyond this, a platform built atop OSGi can remove the artificial application boundaries that exist today. By separating the unit of deployment from application and process boundaries, we open up amazing flexibility in how enterprise applications can adapt, shrink, grow, and morph over time.

Host OSGi is the preferred scenario, but today, SpringSource dm Server and Paremus Infiniflow are the only products I’m aware of that provides the capability. While other major application platform vendors are beginning to build OSGi into their platform, they are not exposing the capabillities to enterprise developers. Until application platform vendors expose the capabilities of OSGi to the enteprise, our only option is Embedded OSGi, which can be tricky and difficult. Unfortunately, the enterprise is left building applications on many of the traditional application servers - WAS, WebLogic, and JBoss - and this poses challenges.

The Challenge

Attempting to develop enterprise web applications today using OSGi presents significant challenges. In fact, these challenges are significant enough that they are severely impeding OSGi adoption within the enterprise. While the OSGi framework implementations themselves are capable, the surrounding ecosystem is not yet mature enough.

• Platforms not ready - Even though most application platform vendors are leveraging OSGi to modularize their platforms, they are not exposing the capabilities of OSGi to developers. Therefore, the only way to leverage OSGi is to embed it within your web application. While dm Server does provide support for Host OSGi, it’s a long way from capturing the marketshare required to be placed in the same category as the major application server vendors.  Embedding is not easy and requires developers to write a lot of infrastructure code. This is the main reason I chose Equinox instead of Felix when embedding within Tomcat - the Servlet Bridge performed two very important functions for me. It launched Equinox, and it bridged servlet requests from the container to bundles managed by OSGi that contained my servlets. Certainly I could have written this functionality myself, but I didn’t want to. And no enterprise developer should have to do this either. They should focus on business logic, not infrastructure logic.
• No Migration Path - Where are the tools that allow developers to create modular applications today such that when the platforms expose the virtues of OSGi to the enterprise developers, they’ll be able to take advantage of it? Where are the tools that allow us to build more modular applications today?
• Lack of best practices - Currently, there are few patterns that document how OSGi can be used to develop enterprise web applications. There are few practices that explain how to refactor large applications into applications with higher degrees of modularity. And because the JAR file has never been treated as the unit of modularity, tools provide no help in refactoring.
• Dearth of Frameworks - Over the past few years, enterprise Java development has made great strides in developing frameworks that allow developers to focus on writing business logic instead of infrastructure code. Developers have grown accustomed to this. Unfortunately, these frameworks don’t exist for us with OSGi today. For instance, if it weren’t for PAX WEB, I’d still be trying to figure out how to support JSP with Jetty embedded in Felix. The very fact I have to embed Jasper to compile JSP pages is a prime example of why we aren’t ready. Finding the right bundles, frameworks, and writing the glue code necessary got tricky pretty quickly. There is no enterprise application developer that has the time nor the desire to do this. Uuntil the platform or a framework provides this capability, it’s unrealistic to expect the enterprise to use OSGi.
• No Management Tools - Even if we’re able to develop more modular applications using OSGi, products do not have the capability to take advantage of modularity because most of the tooling is focused on managing the existing Java EE application modules, such as WAR and EAR files. It’s unrealistic for an enterprise development team to manage an application at the console, which makes managing OSGi applications unwieldy.
• Bundle Problem - Taking advantage of OSGi requires that JAR files are valid OSGi bundles. While I can make sure the JAR files i develop are OSGi valid, it’s not as easy to find 3rd party JAR file packaged as valid OSGi bundles. This is beginning to change, with commercial OBRs beginning to surface, such as the SpringSource Enterprise Bundle Repository, that transform many open source frameworks to valid OSGi bundles. But until framework creators package their products as OSGi bundles, the challenges will continue to linger.

OSGi was once heralded as the most important technology of the decade, and the majority of enterprise software developers can’t even use it yet. OSGi proper may be ready for primetime, but we have a long ways to go before the enterprise is able to develop web applications leveraging OSGi. The problem isn’t the OSGi specification, nor the popular Equinox and Felix implementations. Each of these are waiting and ready. Unfortunately, there is no easy way to build modular applications using OSGi and deploy them to the current generation application servers. There are no frameworks that help me do this. There are no tools that make this easy. There is no migration path that helps developers move away from building monolithic Java EE applications and move toward building the modular applications of tomorrow.

Migration Path Necessary

Ultimately, we need a migration path. As OSGi is built into the platform and exposed to enterprise developers, we need tools and patterns that allow us to build and manage more modular applications. These tools must allow us to take advantage of OSGi in building more modular applications today, and help us prepare for the platforms of tomorrow so that we are ready when they arrive.

If we are able to architect modular applications today, built using OSGi, then we are prepared for what tomorrow brings as we migrate to application platforms that Host OSGi. Just because the application platform doesn’t currently expose OSGi to enterprise developers doesn’t mean we shouldn’t be striving to create more modular applications. There is benefit to this of itself. And because OSGi bundles are backward compatible and capable of running in today’s application servers, what should prevent us from designing more modular software. Shouldn’t we be doing this today? Until there are tools available that help us do this, it’s not likely to happen. And that’s going to make leveraging the capabilities of the next generation platform more difficult, even after it’s arrived.

OSGi is an important and valuable technology that can help enterprise developers create more modular applications. The OSGi Tool Summit is being held Friday (03/27/09). Hopefully, this will result in another small step toward being able to leverage OSGi in the enterprise.

About 10 years ago I recall studying profusely so that I might pass my Java programmer’s certification exam. I purchased a copy of an exam cram book that had sample questions similar to what I might encounter on my certification test. I recall dealing with questions like the following:

• Can an abstract class be subclassed?
• What interface does TreeMap implement?
• What is the output of the following code snippet?
• Identify the compile errors in the following code snippet.
• What are the different access qualifiers?

I’m sure many of us are familar with these questions, as well as others of similar ilk. Of course, I’d taken similar exams prior to this for a variety of technologies, but the Java certification exam was the last one I’ve taken. I’d come to the realization that time spent studying for these exams is largely a waste of time. I was no better at developing software systems in Java after taking the exam than I was before. I simply possessed a bunch of useless knowledge that I could have easily gotten by looking at reference material.

The skill that I needed to develop enterprise applications could only be obtained by developing applications. It could be found no other way. There are no shortcuts. Making mistakes and fixing them. Discovering new ways of doing things that worked better than what I had done before. Knowing what assumptions I could make, and those I could not. The reality is that certifications exist largely to sell training courses, while the marketing that surrounds them leads organizations to believe they are relevant. A while back, the agile alliance issued their public policy denouncing agile certification. They state:

Knowledge is a wonderful thing, but businesses pay for performance. Performance requires skill.

A skill is not as simple to acquire as knowledge: the learner has to
perform the skill badly, recover from mistakes, do it a bit better, and
keep repeating the whole process. Especially for the interrelated and
interpersonal skills required of Agile software development, much of
the learning has to take place on real projects. It is that learning
that a certification should vouch for.

And unfortunately, it is that learning and skill that certifications do not provide nor account for. In fact, this is exactly why the software craftsmanship movement has begun to gain so much traction. Through trial and tribulation, a software craftsman has obtained the practiced skill required to develop great software. Of course, it’s easy to dilute the notion of craftsmanship if we are not careful.

Tom DeMarco also published an open letter to Cutter IT Journal expressing his dismay with certification. The debate on certification certainly is not new. But with the Scrum certifications, among the many others that abound today, the debate is still relevant. Where do you stand on certification? Would you rather work with a craftsman or someone with a wall plastered with certifications?

By the way, if you stand on the side favoring certification, and would like to get an agile certification, then maybe you should visit Agile Certification Now to get started.

In Big Teams & Agility, I talked about a macro process for agile development on large teams (those up to, possibly exceeding, 100 developers). The article was posted on AgileZone at JavaLobby (where I now help out as zone leader, though did not at the time the article was posted), where I was accused of having my head in the cloud. While I could construe that as a compliment given the buzzwords du jour, I probably shouldn’t make that mistake. So I want to take a moment and respond because there were some good points made, and clarification is necessary.

First, you can apply agile practices on big teams, and it does work. I was pretty clear that I’ve used the structure on teams up to 100 developers, and have done so for a good share of the work I’ve done since 2001. It’s worked marvelously. However, I never said it was easy. In many ways, it’s the most difficult approach to developing software I have ever taken, but it’s also the most successful. One question posed by a commenter follows:

Do you really expect big company to go through integration/testing/whatever every week?

Absolutely! The more often you integrate, the earlier you’ll discover problems before they’ve had an opportunity to fester within the system for prolonged periods of time. I recognize this is against the grain of conventional wisdom we’ve been taught for decades. As I mentioned in the original post:

The economies of scale lead us to believe we need longer iterations
because there is so much more to manage. But that’s flawed because it
delays risk mitigation and discovery.

At the end of the post, I stated that there were many micro process details omitted, such as how to keep the build running quickly. To address slow build times on very big systems, you may have to implement staged builds. A staged build is basically a pipeline of builds that perform different build activities.

For instance, a stage 1 build might perform a subset of the overall build steps to provide rapid feedback to the team. A stage 1 build is performed hourly or anytime new code is checked into the source code repository. A stage 2 build is a more complete build process. It might integrate all system components, or execute a complete suite of tests. The actual tasks are going to vary given the context, but the idea remains the same. It’s ok to have multiple build processses for a single software system. It’s up to the development team to identify the components of each build, and it may change throughout the life of the project. But the key element is the rapid feedback the team receives.

Another point of contention with the article centered around the misunderstood suggestion to release frequently to the customer, and the perceived lack of QA, acceptance testing, etc., etc. In fact, I stated the exact opposite, and it’s the continuous integration strategy that allows us to close the loop and perform these types of testing frequently. I stated pretty clearly the following:

We should also frequently execute a variety of tests. Not just unit and
acceptance tests, but usability tests, performance tests, load tests,
and more.

The key element here is that we increase project transparency because we have the ability to get our product in front of the customer on a frequent basis. The customer experiences the growth of the application with the development team. They see it’s evolution, and can provide valuable feedback along the way. There are fewer surprises at the end of the project.

However, I have never said that each build should be released to the customer for use as a production software system. Never! I did say the following:

Once the build executes successfully, the application can be deployed to an environment where it’s accessible by the customers.

The environment is likely a test environment that the customer has access to, and can experiment with the system to provide feedback. Or a QA environment where acceptance testing can be performed. It’s also a place that can be used for system demonstrations. I recognize the pain in delivering large enterprise software systems, and also realize the impossibility of releasing each build to production. But we should be striving for that level of quality each time we write a line of code.

Regardless, the key takeaway is that because we always have a functional system, we are able to perform various lifecycle activities at any time, and many times, throughout the development effort. This increases project transparency where the customer, management, and developers have a consistent understanding of the current state of the system. We avoid those nasty surprises late in the development effort that plague many projects.

This is not overly zealous. It is not unrealistic. It is not some abstract theory born of academia that has never been proven in the real world on a large enterprise development effort. In fact, while it adopts various practices from popular agile methods, it is not Scrum, nor is it XP.  But it is agile, and it does work. It captures the essence of agile development - rapid feedback through software that works while maintaining the ability to respond quickly to change. This is, quite simply, one of the best approaches to software development I’ve ever used. But it’s not easy, nor will it ever be easy. Software development is hard work, and it will always be hard work. The comments are welcome,  the questions are valid, and the discussion is important.

In Grass Roots Agile, I talked about some of the details surrounding how development teams can increase their agility, and I presented a diagram similar to what’s above that discusses how to measure and manage a system’s tested features. Here, I want to talk a bit more about the macro development process, and how I’ve used agile practices on large software development teams. I’ve used this structure with teams up to roughly 100 developers. I have no reason to believe it wouldn’t work on larger teams, though. Here’s the general mechanics illustrated by the diagram.

Team Structure

One of the keys to building big software projects is to break it down into a bunch of smaller projects. Each team in the diagram (Team 1, Team 2, etc.) represents a group of people working on one of these smaller projects. The smaller projects are organized around coarse grained units of business functionality and each team focuses on developing the complete unit of functionality (front to back).

Each of these teams consists of around two to five developers, a business analyst (BA), tester, UI designer, and customer. Depending on the size of the effort, some of these individuals may span teams. A project manager spans all teams. The developers on each of these teams focus on some aspect of technology expertise. Examples include a JSP expert, Hibernate expert, etc. These technology experts form alliances with experts on other teams, which is critical to ensure consistency with how the technology is applied project-wide (ie. the architecture). The technology experts will also spend some time on infrastructure code, such as system-wide error handling utilities and other important architectural aspects. Naturally, you adjust the size of the teams and the roles of team members based on various factors.

The Feature Board

The feature board is a virtual concept that represents the stream of requirements that flows from the customer to the development teams. It is the responsibility of the BA to make sure there are an adequate set of requirements on the feature board, to resolve conflicts, and manage change. Everyone on the individual teams should have the option to participate in discussions surrounding requirements. The feature board could be your typical project room whiteboard, but for large teams, that’s probably not going to work because you have a group of geographically dispersed developers.

Instead, the feature board is simply a snapshot of requirements, and each team pulls those requirements and starts working on them when ready. These requirements might come in the form of user stories, use cases, or something else. The format used is not as important as the fact that the feature board is organized in a way that each team can pull the next set of requirements for the part of the system they are working on when they are ready, while allowing project management to see a complete view of the system requirements.

Continuous Flow

One of the challenges with a large team is that it’s virtually impossible to get everyone aligned on the same iteration schedule. There is too much management overhead. This is one of the driving forces behind why skeptics don’t feel agile is a good fit for really big projects and teams. The economies of scale lead us to believe we need longer iterations because there is so much more to manage. But that’s flawed because it delays risk mitigation and discovery.

In practice, there isn’t a need to have each project team on the same iteration schedule, and in fact, because the teams pull requirements from the feature board when necessary, there isn’t a need for iterations at all. Instead, there is a continuous stream of work that flows from the customers to the development teams. As a team completes a piece of work, they simply release the code to the version control system, build it, and feed it right back to their customer in form of a functional system.

The Build

With each team releasing code, there is a continuous stream of feature rich functionality that is added to the system. Continuous integration and the automated build process holds it all together. Because we are building on a frequent basis, we’ve got a system of checks and balances in place to make sure no breaking changes enter the main product line.

Occassionally, we’ll hit a situation where two teams release incompatible changes. That’s the purpose of the build! When this does happen, the build breaks, the problem is identified, the teams fix the problem, and we’re back on track. This is a central part of the macro process. Without the automated build, it all falls apart. Once the build executes successfully, the application can be deployed to an environment where it’s accessible by the customers.

Closing the Loop

Adopt all the agile processes and practices you want, but increased agility completely falls apart unless you close the feedback loop. I’ve seen too many teams fail because they’ve stopped just short. Feedback is the crucial piece to ensure we close the loop, and because we’ve got a continuous integration process in place due to the automated build, we are able to effectively close the loop because we’ve always got a product that works. It’s important to leverage this. Eliciting frequent feedback from customers through weekly demonstrations is one way to close the loop.

We should also frequently execute a variety of tests. Not just unit and acceptance tests, but usability tests, performance tests, load tests, and more. Because the build is an automated process, we should incorporate code analysis and inspection tools into the process and output the results to a project dashboard that is accessible by the project team. In the end, closing the loop is going to help significantly in increasing project transparency.

In Summary

This is a very broad overview of how agile can work on big teams. In fact, I believe agile is more beneficial for large teams than it is for smaller teams. Small teams are inherently more nimble than bigger teams in the first place, and so agile is an obvious and natural fit. But on large teams, it’s not so obvious how agile practices can improve the team’s success. Above is one formula I’ve found that works very well.

Without question, there are numerous micro process mechanics that I haven’t discussed here. For instance, how often should the build run? (I say at least hourly). For really big systems, how do I keep the build running quickly? (You may need staged builds). What tools do I use to make this all happen? (You don’t need to buy anything, actually). And much more. But in general, the macro process above is a good way to get started.