Software failure statistics are abundant and serve as clear evidence that we must reform software development. While industry claims an IT labor shortage is the motivating force behind outsourcing, the greatest factor is directly related to our inability to deliver value-add software. As organizations continue to lose faith in IT as a trusted partner, the services we offer are little more than an ample commodity, and the search for cheaper labor will persist. But, there is no IT labor shortage.

Read more

I wonder if Andy has seen this.

Quite a few moons ago, I interviewed a gentleman working for a CASE tool vendor. They had just shutdown one of their development shops, and employees had two choices. Find another job, or be relocated. This chap decided to go searching, and our paths crossed. It didn’t take long for me to get way off track with my questions, and eventually I point-blank asked him if they used their own CASE tool on internal development projects. He said “no”, and then nervously suggested that we get the interview back on track. I can’t recall if we ended up hiring the guy, but that interview left an indelible impression and taught an important lesson. If the guy who developed the software doesn’t use it, then you shouldn’t use it. How many open source projects do you think are created where the authors have never used the tool or framework? I’m guessing very few. How many product companies develop products that they don’t use internally? I’m guessing quite a few.

It’s a pretty simple question to ask the next tool vendor trying to sell you their product. If they say yes, then ask them what they like and dislike about the tool. Their honesty is apparent.

The Greater Wisconsin Software Symposium marks the kick-off of the 2008 No Fluff Just Stuff Software Symposiums. The session schedule looks amazing. Traditional sessions on good ole J2EE are still aplenty, but there is also a lot of exciting content surrounding Groovy and Grails, DSLs, Ruby and Rails, Ajax, and agile development that keep things out there on the edge. I’d encourage everyone to attend Sunday’s session on OSGi, which arguably is the most important technology you’ve never heard of - possibly even technology of the decade.

And after a couple months off the tour, you can rest assured that their will be a special energy brought to the conference as it kicks off this year’s tour. Be sure to check it out.

I encountered an incredibly interesting situation recently that clearly illustrates the importance of Continuous Integration. Two separate teams were each working on separate software modules. Eventually, these teams knew they’d need to integrate the two modules into a larger whole. Unfortunately, communication wasn’t that great between the two teams, and while they each had a robust suite of unit tests (they created stubs when testing the integration points between each other), and regularly tested their individual components, they did not ever test them together. Finally, the day came (late in the project, mind you), when the two teams needed to integrate. In fact, integration even went fairly well, with the two modules able to communicate between each other without many problems. Integration testing revealed something alarming, however. A major piece of functionality was missing. Each team thought the other was providing that behavior.

The moral of the story, of course, is that had these separate teams been focused on integrating their two components early and often, this problem would have been discovered much earlier in the development lifecycle. But working in silos is easy and integration is hard. By creating a false sense of progress by avoiding integration until late in the lifecycle, they jeopardized the project.

Agile 2008, being held in Toronto August 4 to 8, seems like a long ways away. However, the submission system just went live this week. I’m excited because I have the opportunity to help organize the Developer Jam stage. There are a few changes surrounding the conference for 2008, starting with the submission system, which is wide open to the public. Any session proposed can be reviewed by any registered user, so hopefully we’ll build some great sessions with the level of community involvement we hope to achieve. Because the conference has been a sell-out the past few years, we’ve also secured a venue in Toronto that can host up to 1600+ individuals.

But what’s most exciting for me is the return of agile in the trenches…getting down and dirty with the developer again. My main issue with last years conference was that it seemed to have forgotten about the developer. There was so much “agile in the enterprise this”, and “application lifecycle management that”. So I’m very excited to help build a track (we’re calling them stages) that focuses exclusively on the developer. I know we can all still stand to learn more about unit testing, continuous integration, refactoring, and more. Please, take a look at the Developer Jam, submit a proposal or two, and provide some constructive feedback. We’ve assembled a great review committee consisting of some outstanding professional developers (listed at the bottom of the Developer Jam page). Let us know what you think would be cool to incorporate into the Developer Jam stage at Agile 2008.

Updated (11/02/07) : Please note the responses from Alberto and Bob attached to this post. They’ve offered some assurance that Crap4J does not transmit any code to their servers, and that the licensing snafu was due to a simple oversight. They also resolve to correct the licensing agreements. Thank you, Alberto and Bob! : End Update

I went to install the Crap4J Eclipse plug-in. As part of this plug-in, there are four separate features, and I happened to actually read the license agreement for each of them. In a nutshell, for three of the features (Agitair JUnit Runner, Agitair JUnit4 Suport, and Public API for Generated Tests), the license agreement states that the software is experimental and primarily for academic, research, and open source use. But that’s not the alarming part. It also says that it transmits your code over the open internet to be analyzed on non-secure Agitair computers shared by multiple users. Here’s the exact text:

THIS SOFTWARE IS INTENDED PRIMARILY FOR ACADEMIC, RESEARCH, AND OPEN SOURCE
USE. WHILE COMMERCIAL USE IS ALLOWED, PLEASE BE AWARE THAT YOUR CODE
IS TRANSMITTED OVER THE OPEN INTERNET AND ANALYZED ON NON-SECURE
COMPUTERS SHARED BY MULTIPLE USERS.

I don’t like that much, and it seems a bit sneaky to hide that rather important note in a license agreement that I doubt many folks read. There should be a more noticeable disclaimer somewhere. Also, I found no such notice in the Ant Task distribution (in fact, couldn’t find a license agreement included at all). But that’s not saying the Ant Task does or does not transmit your code.
I don’t know the internal behavior of Crap4J. Maybe it doesn’t send your code anywhere. But the license agreement indicates that Crap4J does, or at the very least, that they have the right to do so. Maybe, giving them the benefit of the doubt, they didn’t fully review the license agreement, and aren’t aware of what it says. Either way, the fact that this important note is buried within a license agreement without any other public disclaimer is very alarming and deceiving.

I saw the news item on InfoQ regarding MicroSoft Oslo. There was quite a bit marketing mumbo jumbo surrounding Oslo, SOA, BPM, composite applications and MDD, and not a lot of it resonated with me all that much. However, one quote did stand out.

We want to send the model to the server not the printer.

This got me thinking about Big Software in general, even if it isn’t about SOA, BPM, and a lot of other standard industry buzzwords. Developing large enterprise systems comes with a lot of baggage that smaller development efforts don’t have, and a larger codebase is part of it. There are a lot of techniques that teams apply in attempts to architect, maintain and downright understand large software systems. In fact, the problem understanding Big Software is one of the compelling forces that led me to create JarAnalyzer. Since no one person can understand all detailed aspects of a large system, give people high level visual insight into the component relationships and let them drill down from that point. Unfortunately, there is no “drill down” capability in JarAnalyzer - you only get the .jar relationships and a few metrics. But combine it with tools like JDepend, and UMLGraph and you can get a few different views. Either way, there are quite a few different techniques that can be used when it comes to dealing with large codebases. But it seems like most approaches have serious holes.

• UML Modeling. UML Modeling is typically associated with Big Design Up-Front. While I don’t agree with detailed design up-front, establishing some idea of how you want to approach design is a good idea. I don’t like designing individual use cases, but do find value in modeling application frameworks, for example. There are a few advantages in doing this. The entire development team understands the general direction. Nothing is ever set in stone, so if something changes, it also serves as a vehicle for communication. Creating a design model also offers you the opportunity to structure the model in a way that best fits your team and how developers will want to interact with and navigate the model. Unfortunately, the greatest problem with modeling is that keeping your model in sync with the source code is impossible, and most reverse engineering solutions suck. So, any modeling done needs to be at a sufficiently high enough level of abstraction such that detailed change doesn’t impact it, or the models must be short-lived, serve their immediate purpose, and then get filed in the trash bin. Generally, I like to do a bit of modeling early, followed by TDD to drive out the details of my designs.
• Post-Construction Software Visualization. Quite a few static structural analysis tools generate some form of diagram. JarAnalyzer, JDepend, and UMLGraph are examples. Some exist for the .Net platform, too. The obvious advantage is that you get an up-to-date accurate snapshot of your software system. That’s valuable. Unfortunately, the diagrams aren’t always laid out to give you the view into the system that you’d like. Tools like GraphViz do a great job laying out the diagram, but a diagram is just a single view into a system, and possibly not the view I’m looking for. They aren’t complete models, and since the output of these tools are usually images, they typically aren’t navigable. Static structure analysis is only half the battle…maybe less. Understanding class, package, and component relationships is important, but with frameworks like Spring, it’s virtually impossible to get a feel for run-time structure and behavior. Two packages might have no structural dependency between the two, but the placement of abstractions could result in a run-time relationship. I know of no tool that does a good job analyzing the run-time structure.
• Metrics. Many static analysis tools generate metrics and other type of feedback that provide insight into code quality. Most of these don’t necessarily help you understand a system better, but they can help you create a system that is more easily understandable. I like metrics to help guide refactoring, because the design as it’s manifest in code is not always the design as it’s envisioned in the mind of the developer. Metrics offer objective advice on the quality of your application that should be used subjectively. A great resource for analysis tools for Java/JEE development is java-source.net.
• DSM matrices. I haven’t done much work with Dependency (aka. Design) Structure Matrices, but I intend to explore. I’ve talked to a few co-workers who have found them useful.
• MDD. Model-Driven-Development has been thrown around the industry for a while. If anyone has actually had any great deal of success developing Big Software using MDD, I’d love to hear about it. I’m very skeptical.

If I had the perfect solution, it would be real-time visualization of the source, be it static or dynamic, with the ability to create different views of the system. I’d be able to start high-level, with either .jar or package relationships. Then drill down into the .jar and packages to view details. But I’d easily be able to filter out information that I didn’t want to see. For instance, I might want to see the class diagram for a particular package, but in a different diagram I might want to see the class diagram for only those classes that span packages while suppressing classes that have relationships only to other classes in a package. I’d like to capture the behavior of a running system as a sequence diagram, and I’d like to double-click on any class or object and view the source. It’d be perfect if all this would integrate with my Spring configuration files, so I can build that into a view of the system, too. Of course, I want my metrics. I’d like an API available so that I could write unit tests that validate my design. And most important, if I have all of this coolness, I want the ability to filter out the noise. Essentially, I’d like to combine the navigability and view aspects of modelling with the accuracy of post-construction visualization. I need to make sure my models are accurate, but I also want different views into the system depending on what I’m looking for. And I’d like this wonderful tool available as an Eclipse plug-in, standalone IDE, and Ant task. I want all of this, and I want it to perform well, too. Vendors might claim their tool does this for me, but I know it doesn’t.

What’s your wishlist?

JarAnalyzer has always had the ability to create a dot-compliant output file that could be used with GraphViz to generate a component diagram. In the past, this had always been done using the DOTSummary class. Unfortunately, this meant that if you wanted to generate output files in both xml and dot, you had to run JarAnalyzer twice. Now, thanks to a stylesheet that I graciously stole from JDepend and modified to work with JarAnalyzer, there’s a new way to generate a dot-compliant output file that is much nicer than what you’ll get when using DOTSummary. Plus, you only have to run JarAnalyzer once, then apply two stylesheets to the xml file generated to get both the html report and component diagram.

In addition to being a bit more efficient, it’s also cleaner. The old component diagram is shown at left on top, while the new component diagram using the stylesheet is shown at bottom left. The stylesheet avoids the confusion where DOTSummary changed the name of the .jar file and stripped off the .jar extension. As seen on the diagrams, a .jar file named bill.jar now actually appears as bill.jar on the component diagram, not bill.

The new stylesheet isn’t part of the JarAnalyzer distribution…yet, but you can download the stylesheet. To run JarAnalyzer as part of your Ant build script and get both the html and component diagram output, drop the stylesheet in the directory containing JarAnalyzer (the same directory with jaranalyzer.xsl), and modify your build script similar to the following (you need GraphViz installed to run dot):

<target name="dotanalyzerapp.new" depends="bundle">
   <taskdef name="jaranalyzer"
             classname="com.kirkk.analyzer.textui.JarAnalyzerTask">
       <classpath>
           <pathelement path="${buildlib}/jaranalyzer-1.2.jar"/>
           <pathelement path="${buildlib}/lib/bcel-5.2.jar"/>
           <pathelement path="${buildlib}/lib/jakarta-regexp-1.3.jar"/>
           <pathelement path="${buildlib}/lib"/>
       </classpath>

   </taskdef>

   <jaranalyzer srcdir="${buildstats}"
                destfile="${buildstats}/appdependencies.xml"
                summaryclass="com.kirkk.analyzer.textui.XMLUISummary"/>

   <style in="${buildstats}/appdependencies.xml"
                    out="${buildstats}/appdependencies.html"
                    style="${buildlib}/jaranalyzer.xsl">
   </style>

   <style in="${buildstats}/appdependencies.xml"
                    out="${buildstats}/appdependencies.grph"
                    style="${buildlib}/jaranalyzer2dot.xsl">
   </style>

   <exec executable="dot" >
       <arg line="-Tpng -Nshape=box -Nfontsize=30 -Nwidth=1.5
              -Nheight=1.25<br></arg> ./buildstats/appdependencies.grph
              -o ./buildstats/appdependencies.png">
   </exec>

 </target>

Here’s the test task in my rake file. Works on Windows, but not on my Mac. It just hangs. What’s the problem?

Rake::TestTask.new do |t|
  t.libs << "./app;./test"
  t.test_files = FileList['test/test*.rb']
  t.verbose = true
end

Two hours later, I tell you that I need this on my Mac.

Rake::TestTask.new do |t|
  t.libs << "./app:./test"
  t.test_files = FileList['test/test*.rb']
  t.verbose = true
end

See the difference? Semi-colon…colon…Wow…