SpringOne2GX 2014 Replay: Spring XD for Real-time Hadoop Workload Analysis

News | Pieter Humphrey | December 09, 2014 | ...

Recorded at SpringOne2GX 2014.

Speakers: Vineet Goel, Girish Lingappa, Rodrigo Meneses

Slides: http://www.slideshare.net/SpringCentral/spring-one2gx-2014springxdhadoopworkloadanalysis

Big Data Track

As Hadoop goes mainstream in enterprise big data deployments, IT organizations expect and demand enhanced operational management of their Hadoop clusters in production. Admins require more than just cluster health monitoring; they need the ability to do real time workload analysis for performance tuning and troubleshooting. Real-time log analysis of jobs at a user or application level can allow admins to manage and tune workloads better, especially in multi-tenancy Hadoop cluster services. Join us to learn how Pivotal team leveraged Spring XD data ingestion and batch processing framework, GemFire XD & other components to solve this interesting challenge on a large 1000-node (Analytics Workbench) cluster. Using Spring XD to ingest YARN service and MapReduce application logs through a real-time data pipeline into HDFS, the team leveraged familiar SQL-based queries to analyze fine-grained cluster utilization.

!{iframe width="560" height="315" src="//www.youtube.com/embed/cBcznXjL840" frameborder="0" allowfullscreen}{/iframe}

Get the Spring newsletter

Stay connected with the Spring newsletter

Subscribe

Get ahead

VMware offers training and certification to turbo-charge your progress.

Learn more

Get support

Tanzu Spring offers support and binaries for OpenJDK™, Spring, and Apache Tomcat® in one simple subscription.

Learn more

Upcoming events

Check out all the upcoming events in the Spring community.

View all