The New Era of Programming Languages

Burton Group has just published the overview I authored titled  The New Era of Programming Languages. This is a timely document. The Tiobe Programming Index shows an increasing mix of different language types that are popular today. Also included is a graph that illustrates the shrinking gap between the use of dynamically and statically typed languages.

At JavaOne a couple of weeks ago, there were numerous sessions I attended that discussed alternative languages on the JVM. In a session hosted by Brian Goetz, he talked about the renaissance JVM, and the more than 200 languages it currently hosts. There were also sessions on Clojure, Ruby, Scala, along with a script bowl where Jython (represented by Frank Wiersbicki), Groovy (represented by Guillaume Laforge), Clojure (represented by Rick Hickey), and Scala (represented by Dick Wall) squared off against one another.

While the clear crowd favorite at the script bowl was Groovy, it was apparent that each language stood out in separate ways – Jython as a form of “executable pseudocode”, Groovy with it’s compile-time metaprogramming capabilities, Clojure as a dialect of Lisp with it’s powerful multi-threaded and concurrency capabilities, and Scala with it’s Java-like syntax and type inference capabilities. And all had great integration with the JVM along with the ability to leverage existing Java librairies.

Here’s an excerpt from the Burton Group report that walks the timeline of language evolution as depicted in the diagram above. It’s a walk down history lane, and offers a perspective on the future of languages.

The Birth of Language

The first modern programming language is hard to identify, but historians trace the profession of programming back to Ada Lovelace, a mathematician often credited with creating the world’s first computer program in the mid-1800’s. In these early days of computing, languages provided no abstraction from the computer hardware on which the programs would run, and hardware restrictions often defined the language. These earliest computing machines had fixed programs, and changing how these machines behaved was a labor intensive process that involved redesigning the hardware in conjunction with the program it would run.

In the 1940’s, general-purpose computers began to emerge that were able to store and load programs in memory. Creating programs for these computers required developers to use first-generation machine code languages. Eventually, second-generation languages, such as Assembly, emerged and provided a symbolic representation for the numeric machine code. Regardless, each language was low-level and cryptic. Developing even trivial programs was error-prone, required a great deal of intellectual ability, and generally took a long time.

The Compiler Era

The 1950’s gave way to the first modern programming languages—Fortran, LISP, and COBOL—that represent the ancestors of the languages in widespread use today. Each was a higher-level third-generation language that abstracted away the underlying complexities of the hardware environment and allowed developers to create programs using a more consumable and understandable language syntax. Though languages were greatly simplified, adoption of these languages was slow because developers wanted assurance that programs created using these languages was comparable to that of assembly language. As compilers were optimized, adoption of higher-level programming languages progressed.

Even at this early stage of language evolution, language capabilities diverged with an emphasis on solving specific types of programming problems. COBOL and Fortran dominated business and scientific computing, respectively. LISP prospered in academia. These early languages, and their emphasis on specialization for solving certain types of programming problems, were an indication of the language evolution to come.

The Paradigm Era

Language evolution experienced significant innovation in the 1960’s and 1970’s, and many of the major paradigms in use today formed in this period. Software systems were increasing in size, and development was incredibly complex. Maintaining those systems presented additional challenges. Language designers and developers were seeking ways to make complex programming tasks easier by raising the level of abstraction. Discussion of the language effect on software design began to surface, and the use of the GOTO statement and the advantages of structured programming were serious topics of debate.

A multitude of languages emerged that supported these new paradigms. Object-oriented programming was born when the Simula programming language was created, and eventually Smalltalk would surface as the first pure dynamically typed, object-oriented language. C and Pascal were created in this era, as was Structured Query Language (SQL). The first functional language, ML, was also invented. A major shift was under way to make programs easier to develop and maintain through new language enhancements and programming paradigms.

The 1980’s were a period of far less innovation while emphasis turned toward paradigm and language maturation. Languages such as C++, invented in 1979 and originally called C with Classes, emerged that brought the advantage of object-oriented programming to a language that was strong in systems programming. Modular languages, such as Modula and Ada, began to emerge that helped in developing large-scale systems. Leveraging the capabilities of advanced computer architecture led to compiler advancements that were a precursor to managed code environments.

The Productivity Era

Without question, the early 1990’s were driven by a motivation to increase developer productivity. Managed runtime environments emerged that removed the burden of memory allocation and deallocation from the developer. Advanced programming environments, such as PowerBuilder, provided an increase in productivity by combining the language and integrated development environment (IDE) into a consolidated product suite. Attempts were made to push languages down the stack, and attention shifted to advanced tooling and frameworks that increased developer productivity.

In the mid-1990’s, the World Wide Web (WWW) popularized the use of the Internet as a massively scalable hypermedia system for the easy exchange of information. Soon thereafter, a nascent technology was integrated into the omnipresent Navigator browser. Java technology, and applets specifically, had been officially unveiled to the world as a language whose programs could execute atop a managed environment in a plethora of operating environments.

Although the hype surrounding applets fizzled out shortly thereafter, the write once, run anywhere (WORA) promise of Java moved to the server-side as dynamic web applications began to increase in popularity. The portability of Java applications was realized through a Java Virtual Machine (JVM) that managed application execution. Because of this, the language was far simpler than C and C++, because the JVM provided complex memory management capabilities. What was once a complex task to create programs that ran on disparate platforms now required little effort. The simplicity of the language, combined with the portability of applications, was a force that would affect computing for the next decade.

The ecosystem surrounding Java technology flourished, and tools and frameworks emerged that made Java programming tasks much easier. Early after the turn of the millennium, .NET arrived along with its Common Language Runtime (CLR), and platform support for programs written in multiple languages was a reality. Initially, many scoffed at the need for a platform supporting multiple languages. Instead, they adopted general-purpose languages and accompanying tools, frameworks, and managed runtime environments, believing they offered the greatest productivity advantages. Even though a multitude of languages existed, organizations rejected many of them in favor of Java and C#. However, platform support for multiple languages proved to eliminate a significant barrier to language adoption.

Eventually, developers discovered that frameworks and tools were increasing in complexity and becoming difficult to work with. In many ways, these frameworks and tools were hindering productivity, and the advantages of using a single general-purpose language was called into question. Developers were growing frustrated. Though frameworks and tools might exist that aid a task, learning the framework or tool was as daunting as learning a new language altogether. While the complexity of frameworks and tools continued to increase, developers sought alternative languages that were easier and more productive.

The overview goes on to talk about the present day and the postmodern era. The Java and .NET platforms dominate enterprise development, and each support a variety of languages. Platform support has reduced the barrier to entry. Subsequently, a shift is taking place as developers are looking for alternative languages (instead of frameworks, for example) that increase productivity and make programming tasks easier.

The postmodern era recognizes that multiple languages are a fact of life. Language evolution will continue, and new languages will emerge that blend aspects from multiple paradigms. The strong distinction made between static and dynamic languages will disappear, and metaprogramming will become more mainstream. Even compilers will change, as developers gain more control, and are able to take advantage of compilers just as they do APIs today.

Without question, we live in interesting times, especially if you’re a language geek.

Software Rot – Manage those Dependencies

In Rotting Design, I spoke of how software tends to rot over time. When you establish your initial vision for the software’s design and architecture, you imagine a system that is easy to modify, extend, and maintain. Unfortunately, as time passes, changes trickle in that exercise your design in unexpected ways. Each change begins to resemble nothing more than another hack, until finally the system becomes a tangled web of code that few developers care to venture through. The most common cause of rotting software is tightly coupled code with excessive dependencies.

The Problem with Dependencies

Dependencies hinder the maintenance effort. When you’re working on a system with heavy dependencies, you typically find that changes in one area of the application trickle to many other areas of the application. In some cases, this cannot be avoided. For instance, when you add a column to a database table that must be displayed on a page, you’ll be forced to modify at least the data access and user interface layers. Such a scenario is mostly inevitable. However, applications with a well thought dependency structure should embrace this change instead of resist the change. Applications with complex dependencies do not accommodate change well. Instead, with change, the system breaks in unexpected ways and in unexpected places. For this to happen, the module you unexpectedly broke must be dependent on the module that changed.

Dependencies prevent extensibility. The goal of object-oriented systems is to create software that is open for extension but closed to modification. This idea is known as the Open-Closed Principle. The desire is to add new functionality to the system by extending existing abstractions, and plugging these extensions into the existing system without making rampant modifications. One reason for heavy dependencies is the improper use of abstraction, and those cases where abstractions are not present are areas that are difficult to extend.

Dependencies inhibit reusability.
Reuse is often touted as a fundamental advantage of well-designed object oriented software. Unfortunately, few applications realize this benefit. Too often, we emphasize class level reuse. To achieve higher levels of reuse, careful consideration must also be given to the package structure and deployable unit structure. Software with complex package and physical dependencies minimize the likelihood of achieving higher degrees of reuse.

Dependencies restrict testability. Tight coupling between classes eliminates the ability to test classes independently. Unit testing is a fundamental principle that should be employed by all developers. Tests provide you the courage to improve your designs, knowing flaws will be caught by unit tests. They also help you design proactively and discourage undesirable dependencies. Heavy dependencies do not allow you to test software modules independently.

Dependencies limit understanding. When you work on a software system, it’s important that you understand the system’s structural architecture and design constructs. A structure with complex dependencies is inherently more difficult to understand.

Dependencies of the worst kind

Excessive dependencies are bad. But cyclic dependencies are especially bad. Cyclic dependencies are manifest in various ways at different levels within a system. It’s also possible that acyclic relationships at one level cause cycles at another.

Types of Cycles

Cycles exist across a variety of entities; notably class, package and JAR. Class cycles exist when two classes, such as Customer and Bill shown here, each reference the other (assume Customer has a list of Bill instances, and Bill references the Customer to calculate a discount amount). This is also known as a bi-directional assocation. It’s a maintenance and testing issue, since you can’t do anything to either class without affecting the other.

Class cycles can be broken a few different ways, one of which is to introduce an abstraction that breaks the cycle, as shown here. Now you can test Bill with a mock DiscountCalculator. Testing Customer, of course, still requires the presence of Bill. This is not a cyclic issue, it’s a different type of coupling issue as Bill is a concrete class, and is fodder for a separate discussion. Introducing DiscountCalculator has broken the cycle between Customer and Bill…but has it broken all cycles?

Creeping cycles

We don’t intentionally create cyclic dependencies. Instead, they tend to creep into our design. They commonly surface when cyclic or acyclic relationships at one level cause cycles at another. For instance, if Customer and DiscountCalculator are placed in a cust package, and Bill is placed in a billing package, a cyclic dependency between cust and billing exists even though the class structure is acyclic, as shown here. Allocating the cust and billing packages to cust.jar and bill.jar also causes a cycle between the .jars.

To break the cycle, we should move DiscountCalculator to its own package, or the billing package. Simple heh? Well sure…but now toss in a few thousand classes, a few hundred packages, and numerous JAR files, and it’s not so simple to manage anymore.

Managing Cycles

Fortunately, there are many ways (some easier than others) to manage dependencies and eliminate cycles. Test Driven Development is a great way to manage class cycles assuming we strive to test classes in isolation. JDepend allows you to manage package cycles, either by writing package constraint tests or including JDepend reports within your Ant build script. Jar cycles can be managed using a Levelized Build, where individual jars are built, including only necessary components in the build class path. JarAnalyzer can also be included in your build script, generating a component diagram illustrating the relationship between JAR files, or a dependency report similar to that of JDepend. Maven and Ivy also provide ways to help manage dependencies. And of course, looming on the horizon is OSGi, which may not be ready for widespread enterprise use today, but hopefully will be soon.

Are Cycles Always Bad?

Generally speaking, cycles are always bad! But some cycles are worse than others. Cycles among classes are tolerable, assuming they don’t cause cycles among the packages or JAR files containing them (ie. the classes must be in the same package, essentially encapsulating the design). Cycles among packages may also be tolerable, assuming they don’t cause cycles among the JAR files containing them (again, packages are in the same JAR file). Most important is that we are aware of the relationships existing between the JAR files. In so many cases, we aren’t.

Java and .Net Communities

For the past 5 months, I?ve been working on a .Net project, my first of the sort. Until then, I’d always been a developer on some pretty large Java efforts, some small PHP efforts, and some personal experimentation with Ruby. But I had done absolutely nothing with .Net, and the opportunity to gain experience with another major platform was exciting. Now, I’m back on another Java project, and I am not sad to leave .Net behind.

I know Microsoft has always done a great job ensuring their technologies interoperate well, having experience with such interoperability going back to my days as a PowerBuilder and VB programmer using OLE automation to integrate with Microsoft Word. With Microsoft, there always seemed to be an easy way of doing things using Microsoft technology. Unlike Java, this meant less time trying to sift through the bad frameworks in search of the good one, and more time implementing a solution using Microsoft’s platform. Ironically, I didn’t appreciate this as much as I expected. I feelpretty strongly about architecture and design, and the easy way wasn’t always the right way. However, what startled me most about .Net wasn?t the platform, the C# language, or my dislike for WinForms development, but the .Net community and culture surrounding it.

Microsoft has a fledgling open source community compared to many other platforms. While a plethora of competing open source projects means you’ll make a mistake in choosing every once in a while, even the dead open source projects contribute new ideas and innovation within the Java (ok… and Ruby) communities. And many of the .Net open source projects are little more than ports from their Java peers.

While Microsoft claims they support open source, their actions speak otherwise. Look no further than the Visual Studio Express EULA limiting 3rd party extensibility, placing the TestDriven.Net plugin in violation of the license agreement. There’s been a bit of banter going on surrounding the decision by Microsoft to disallow this plug-in. The irony here is that the TestDriven.Net plugin is only free for personal users, not professional or enterprise users, further illustrating my point. It’s not just Microsoft, but the Microsoft community. You don’t have to look far to find other examples.

I recognize that there are some very good open source projects underway in the .Net world. I run Mono on my Mac. CSLA is a full-stack framework that allows you to hit the ground running. But the culture surrounding open source on the .Net platforms seriously lags behind what I’m accustomed to with Java.