Disappointed by Jigsaw delay? Java Modularity is here today!

I recently heard that some people are expressing disappointment over the lack of a built-in module system in Java 8, now that Jigsaw is delayed. Some are apparently even looking at technologies other Java because of this. Well, I'd like to point out that There is No Need To Wait! All the modularity that your application needs is already available with OSGi! OSGi pretty much supports everything that Jigsaw was planning to support to allow modularization of applications. And OSGi has more features such as a dynamic lifecycle and a very useful services model. Still OSGi frameworks can be remarkably small. Of the popular implementations Apache Felix is probably the smallest with its implementation in under 500kilobytes. But Eclipse Equinox and our own JBoss OSGi Framework inside AS7 are also very good opensource implementations.

Want to learn how to use OSGi? Like many people I like learning stuff by reading a book. Recently a number of books have been published that provide really good explanations of how to use OSGi, notably OSGi in Action, OSGi in Depth and Kirk Knoernschilds Modular Architecture book, but probably my favorite book is Enterprise OSGi in Action. Besides containing a good description of a number of Enterprise OSGi specs, it also describes really well why certain things are done in OSGi they way they are, which give you a nice and deep understanding of the technology. Additionally it contains some great chapters on how to get your build, test and development environments to work smoothly with OSGi. Enterprise OSGi in Action will be in print later this year, but you can already get it in electronic form from here.

Besides the books, the OSGi Javadoc can be very useful and if you really want to know the details of it all, the OSGi specs are a great resource. Oh, and don't forget to check out the Wikipedia page that shows where you can get implementations of the various OSGi technology.

So in short - no need to wait or turn away from Java if you need Modularity. OSGi is here today and has been used in production by numerous users for many years, just get started by downloading one of the opensource implementations and try it out!

No Jigsaw for JavaSE 8

Yesterday Mark Reinhold announced that Oracle will not pursue modularity for JavaSE 8: http://mreinhold.org/blog/late-for-the-train. Following long delays for the Modularity JSR the decision was made not to postpone JavaSE 8 or rush through the JSR but rather postpone the Modularity part of Java 8 and push ahead with the rest of it.

So what does this mean for Modularity in Java? Well, I think its an appropriate decision to not rush things through. As modularity is such a fundamental piece of functionality you don't want to cut corners with the architecture. OSGi has been providing modularity to Java for over 12 years now and there are a lot of lessons to be learnt from it.

Project Jigsaw is a prototype of a modularity system that's part of the Java runtime, but at this stage it's just one approach to this. In Project Penrose work is ongoing around improving the extensibility of Jigsaw so that it can work better with other module systems such as OSGi.

While some work is definitely needed to modularize Java itself, as an application developer you don't need to wait for this. If you need modularity in your applications, then OSGi provides all that you need today and it has been proven for years by many in the field.

It's good that the Modularity JSR will get some proper time now. I hope the process will be democratic and that the result will provide the best technology possible that keeps extensibility in mind.

Java 8: extensible module definitions with JSON

While the Java Modularity JSR [1] is still AWOL (come on, Oracle, I thought that this JSR was planned to be launched by September 2011!), I did some work in OpenJDK Penrose around changing the Java SE 8 module metadata from the .java/.class format into JSON.

JSON is quite nice because

• it's a simple text format
• easy to read by humans
• easy to parse for computers
• its highly extensible

with JSON, the SE 8 Quick Start module org.astro module definition looks like this:

{
  "module" :
  {
    "name" : "org.astro",
    "version" : "1.2",
    "exports" : [ {"name" : "org.astro"} ]
  }
}

and the same module info file can be expanded to contain OSGi metadata, for example to add a version to the exported package like this:

{
  "module" :
  {
    "name" : "org.astro",
    "version" : "1.2",
    "exports" : 
    [
      { 
        "name" : "org.astro", 
        "org.osgi.version" : "1.2.3"
      }
    ]
  }
}

Any OSGi metadata can be added to the JSON module descriptor. I've added a simple test to the Penrose codebase to give an idea what this looks like: hello-json-custom.sh.

This works today in OpenJDK project Penrose. The codebase still supports module-info.java, to allow a gradual migration to JSON-based metadata, however I'd like to see the module-info.java/.class support go away ultimately.

While Jigsaw/Penrose are two variants on a module system for Java SE, other ones are clearly possible. Once the modularity JSR is up in action, it should seriously consider OSGi as a module system for Java SE 8. Apache Harmony has clearly demonstrated in the past that modularizing the Java platform using OSGi metadata works quite well. OSGi has proven to be an excellent module system for Java for the last 12 or so years, so it would be foolish not to take that experience on board...

On the lack of Modularity JSR, I think at that at this stage we can expect that it will delay the release of Java 8, given that modularity is such an important building block for Java, and is certainly not something that can be rushed through.

[1] 'Java Platform Module System' JSR as mentioned in the Java SE 8 umbrella JSR.

The need for Modularity

The title of this blog is not new - it has been used by many other blogs and articles in the past, but Rod Johnson seems to think that Modularity not really needed any more so I decided to reuse the title and take a look at why we really need Modularity.

In this age of Cloud Computing where hardware utilization and energy consumption are being highly optimized you will find that more and more service providers are moving towards a multi-tenancy model where multiple deployments are co-located on a single platform. Whether these deployments are from different providers or from the same provider is the choice of the deployer, but the end-result is a situation where multiple deployments share a single platform. In such cases modularity is an absolute must as without it there is a reasonably big chance that the system won't work. This is because deployables might depend on incompatible versions of the same library, which means that in order to work, they need to be isolated from each other. Or, a deployable might depend on a library that also happens to be used by the platform provider, this is an even harder case which requires the platform itself to be developed in a modular fashion so that it's details don't bleed into the user-space.
Scenarios like this require modularity and in the case of Java this modularity is provided through OSGi and/or many application servers. Additionally, modularity can also be found in many non-appserver systems that allow plugins to be installed.

Further benefits of modularity relate to memory footprint and performance. A modular system can make intelligent decisions about its memory usage, where only modules that the end user actually needs are loaded. This makes it possible for a fully featured server to have a tiny runtime footprint if only a small portion of it is actually used. And all of these benefits have performance benefits too: you'll save a lot of time if you're only loading the modules you actually need rather than pre-loading the entire system as a single entity.

But even if you don't need modularity for side-by-side versioning, isolation or performance just yet, there are additional benefits associated with using a modular approach in your development. I think that even for small projects it's not a bad idea to start out by using Modularity. It's true - getting a modular project off the ground takes a larger initial effort compared to the 'traditional' approach. But that cost is quickly earned back once the project grows a little bigger. Once your project is structured to support modularity, it becomes much more maintainable, easier to understand and to extend while 'traditional' approaches tend to become much harder to maintain as they grow in size.
See the following diagram, adapted from a blog by Hal Hildebrand:

Cost of Software Maintenance

The investment in a Modular Architecture earns itself back quicker than you think.

Using a modular approach together with a service oriented architecture*) gives your system highly cohesive entities with a clear responsibility. This makes it easier to identify what parts are affected when a system needs to be enhanced or changed. Changes for a particular feature are often more localized to a specific set of modules, which enables better parallelization of development and will ultimately make your developers more productive.

So Modularity is highly desirable for a number of reasons and the best place to get modularity today is by taking an OSGi Framework, as OSGi has been providing Modularity as well as an excellent Services Framework to Java for over 12 years and is standardized in the OSGi specifications.

Part of the choice for a particular development architecture is always the consideration regarding tooling support. With OSGi the following tools are popular:

• The Eclipse PDE with Maven Tycho
• Maven Bundle Plugin
• Bnd and Bndtools

Additionally, Netbeans and IntelliJ have OSGi Support. While tools can always be improved the above toolsets are quite usable. Especially the Maven Bundle Plugin is very popular in the Open-Source world and many, many examples can be found that show its use.

To conclude, Rod Johnson says that especially on the server side, using a non-modular system is better for customers. This seems like a highly incorrect statement to me. Modularity is needed especially on the server side, as without it your deployments become fragile, especially if you have more than one deployment on that server...

*) Note that by saying 'service oriented architecture' I'm not implying a distributed SOA. Using services within a single process is highly valuable and works great in combination with modularity to avoid tight coupling. Yes, services open the door to distributing parts of your application, but that's really a different topic...

Java SE 8 Modularity Requirements

Mark Reinhold posted the requirements for the Module System in JavaSE 8 here today: http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12 . And, I won't deny it, I was part of the group that worked on getting these requirements formulated, although I don't fully agree with all of them, but more on that below.

There are many, many good requirements in this document and Java will certainly gain from having these realized - if this is done in an open and inclusive manner.

So what does this mean for OSGi? OSGi is a widely used, mature, modularity standard today, existing for 10+ years.
Well, the good news is that the new requirements are not incompatible with OSGi. There is actually a specific requirement around the necessity for OSGi to be able to interact with the JavaSE modules.

Many of the requirements mentioned in the document are already supported by OSGi today. The file format that defines the modules will most likely not be the MANIFEST.MF file that is used by OSGi today, but another file of some sort. OSGi will support this new mechanism to declare the module as well in the future; the semantics are the same, the format will be slightly different.

Another good thing is that once modularization is part of Java SE 8, moving to OSGi will be easier than it is today. The biggest stumbling block for migrating an existing system to OSGi today is often the modularization of a system that wasn't created in a modular way in the first place. It's not an issue with OSGi per se, but with modularity as a concept. If you have existing code that is non-modular, it can be difficult to modularize it. With JavaSE 8 this problem will appear earlier in the development cycle which should help people design their systems in a modular way from early on. If you need more than the basic JVM module system then you can move to OSGi taking your existing JavaSE 8 modules and they will just work as-is - so migration will be a breeze. Then you can enhance the modules to take advantage of  the OSGi benefits (see below).

Why not take OSGi as-is?
Well, this would be nice. And it should be quite possible to take a (possibly reduced) OSGi framework and make that the module system for Java SE 8. But to a certain degree it's an implementation detail. The important thing is that the Java SE 8 Module System is a well-defined standard. Whether JRE implementations take OSGi as the basis for their module system or write their own from scratch is up to them.

Does OSGi still add value? Absolutely! Here are some points:

• dynamicity and lifecycle: OSGi modules are dynamic; you can add, remove and update modules from a running framework. It allows patching applications without taking them down! (I have demo'ed this many times) This will probably not be possible in the JavaSE 8 modules.
• advanced service registry: the OSGi Service Registry is much richer than the Java SE java.util.ServiceLoader class. It allows filtering for services on additional metadata (beyond the interface name), supports multiple instantiation modes and, again, is dynamic: services can come and go.
• richer dependency metadata: OSGi has an extremely rich metadata model that supports specifying dependencies in many ways. It does support module-level dependencies (as is proposed in the Mark's requirements doc) through the Require-Bundle header. However, most people who have been working with OSGi for the last 10+ years have realized that pure module-level dependencies are inferior to declarative dependencies such as the ones provided by OSGi's Export-Package and Import-Package. Since OSGi 4.3 generic capabilities and requirements are also supported, which allows you to create your own dependency system.
• many other OSGi specifications: there are many OSGi specifications that describe useful things which can be used on top of any OSGi framework. See the compendium and enterprise specifications for the details.

Where are the disagreements?
They're actually spelled out in the requirements document.

1. The form of the module definition. Having module metadata in a file different from MANIFEST.MF is fine, in my opinion. MANIFEST.MF is a near text-file format with some strange limitations. A proper DSL would be nice for module definitions. I don't see the value, however in compiling that information into some binary format. It has been invaluable to me many times in the past to be able to open an OSGi bundle, look at the manifest and see exactly from that information what the module definition is. Compiling the information into a binary format will make it harder to see the module definition and it might also make life harder for tools that work with the module system.
2. The file extension of the module definition. Some argued that the module source definition should be in a .java file and because it's in a .java file there should be a compiled .class file in the resulting Jar file. However, I think that the structure of a module definition screams for a specific DSL that might be quite different from ordinary Java. Especially the extensibility requirement makes a DSL much more natural than a traditional .java source file. I think the module definition should be in  META-INF/module.jmod or something like that. Then it becomes natural to migrate that file as-is to the resulting jar. In a sense the existing MANIFEST.MF does cover most of the requirements. Sure, it has some issues (esp. the 76 chars-per-line limit) but it's clear, simple and extensible - and - it is readable by humans in the Jar file.

Conclusion
I congratulate Mark Reinhold and all others involved in this for providing the basic requirements and design proposal that will make OSGi work nicely with the JavaSE 8 module system. OSGi will extend the basic JavaSE 8 modules and both systems will work with each other to form a layered design.