The Spring XD team is pleased to announce that Spring XD Milestone 6 is now available for download.

This is our biggest release yet! The team has been hard at work, and Milestone 6 contains a wealth of new features that meet enterprise requirements in terms of reliability, performance, and user experience. Below is a quick Top Ten (in no particular order), but if you checkout the release notes you will realize how difficult it is to pick out 10 from the list of 299.

• Distributed and Fault Tolerant Runtime: Leader election among multiple xd-admin servers and automatic redeployment of modules to other xd-containers in the case of failure. ZooKeeper is introduced to manage the cluster and its deployment state.

• Support for running XD on YARN: Run admin and container nodes on a Hadoop YARN cluster rather than on VMs or physical servers that you need to manage. There are simple configuration and shell scripts that make this process very easy.

• Deployment Manifests: When deploying a stream you can provide a deployment manifest that describes how to transform the logical stream definition (e.g. http | hdfs) to a physical deployment on the cluster. You can specify the number of instances of each module to deploy and also a criteria expression (using SpEL) that evaluates each of the available containers in the cluster to determine the best matches for those module instances. This will be an area of active development for the next release as we extend the manifest to include support for data partitioning strategies.

• Real-Time Evaluation of Machine Learning Scoring Algorithms: Integration with the JPMML-Evaluator library that provides support for a wide range of model types and is interoperable with models exported from popular data analysis packages such as R. Integration with other libraries is supported by providing an implementation of XD's Analytic and MappedAnalytic abstractions. Give it a try with the sample application based on the classic iris data set.

• Updated UI: A redesign and rewrite of the UI that has a modern look and feel.

• Enhanced DSL Auto-completion: Tab completion now works within the value of the --definition option when creating streams and jobs. Creating streams and jobs has never been easier. Try it out via xd-shell!

• Default stream for the Batch Job lifecycle: Batch jobs are a source of event data. Add a tap to a Batch Job to receive events upon upon job success/failure, individual step actions or even item level actions. This functionality along with the support for launching jobs by sending messages to a named channel, enables your applications to have a complex event-driven flow between the batch and streaming domains.

• Improved server and module configuration options: Server and module configuration have been separated to help with the life-cycle of upgrading them independently. Module configuration has been made more modular while allowing for shared configuration between related modules.

• High performance TCP/UDP source: Based on the Reactor project - the Reactor IP source on commodity hardware can consume ~1 Million msgs/second.

• FTP to HDFS job: Out of the box support for jobs to transfer files from FTP to HDFS. Those jobs take advantage of another new feature: partitioning of steps across multiple XD Containers.

Wrapping up

You can also install Spring XD on OSX using homebrew and on RHEL/CentOs using yum.

The Spring XD project home is the central hub for learning more about Spring XD. Some useful links are the reference docs, sample applications, and QCon SF 2013 Session Replay: Introducing Spring XD.

We look forward to your comments and feedback:

• StackOverflow spring-xd tag
• Spring Forum
• Spring JIRA

Our plan is to have a one month cadence of releases leading up to a GA release in July.

SpringOne 2GX 2014 is around the corner

Book your place at SpringOne in Dallas, TX for Sept 8-11 soon. It's simply the best opportunity to find out first hand all that's going on and to provide direct feedback. There will be deep dive sessions on Spring XD along with general Big Data talks to provide an introduction to the landscape and challenges in developing Big Data applications.