Spark

Transformations and Caching – Spark Screencast #3

Andy — Wed, 17 Apr 2013 00:06:22 +0000

In this third Spark screencast, we demonstrate more advanced use of RDD actions and transformations, as well as caching RDDs in memory.

For more information, check out the Spark documentation page.

Spark screencasts published

Andy — Tue, 16 Apr 2013 23:59:19 +0000

We have released the first two screencasts in a series of short hands-on video training courses we will be publishing to help new users get up and running with Spark in minutes.

The first Spark screencast is called First Steps With Spark and walks you through downloading and building Spark, as well as using the Spark shell, all in less than 10 minutes!

The second screencast is a 2 minute overview of the Spark documentation.

We hope you find these screencasts useful.

Spark Documentation Overview – Screencast #2

Andy — Thu, 11 Apr 2013 23:40:22 +0000

This is our 2nd Spark screencast. In it, we take a tour of the documentation available for Spark users online.

And for convenience, here are links to the documentation shown in the video:

First Steps with Spark – Screencast #1

Andy — Wed, 10 Apr 2013 22:15:26 +0000

This screencast marks the beginning of a series of hands-on screencasts we will be publishing to help new users get up and running in minutes. In this screencast, we:

Download and build Spark on a local machine (running OS X, but should be a similar process for Linux or Unix).
Introduce the API using the Spark interactive shell to explore a file.

Check out the next spark screencast in the series, Spark Screencast #2 – Overview of Spark Documentation. You can also find the Spark documentation online:

Strata exercises now available online

matei — Mon, 18 Mar 2013 02:19:25 +0000

At this year’s Strata conference, the AMP Lab hosted a full day of tutorials on Spark, Shark, and Spark Streaming, including online exercises on Amazon EC2. Those exercises are now available online, letting you learn Spark and Shark at your own pace on an EC2 cluster with real data. They are a great resource for learning the systems. You can also find slides from the Strata tutorials online, as well as videos from the AMP Camp workshop we held at Berkeley in August.

Spark 0.7.0 released

matei — Wed, 27 Feb 2013 17:06:59 +0000

We’re proud to announce the release of Spark 0.7.0, a new major version of Spark that adds several key features, including a Python API for Spark and an alpha of Spark Streaming. This release is the result of the largest group of contributors yet behind a Spark release — 31 contributors from inside and outside Berkeley. Head over to the release notes to read more about the new features, or download the release today.

Spark Release 0.7.0

matei — Wed, 27 Feb 2013 16:26:19 +0000

The Spark team is proud to release version 0.7.0, a new major release that brings several new features. Most notable are a Python API for Spark and an alpha of Spark Streaming. (Details on Spark Streaming can also be found in this technical report.) The release also adds numerous other improvements across the board. Overall, this is our biggest release to date, with 31 contributors, of which 20 were external to Berkeley.

You can download Spark 0.7.0 as either a source package (4 MB tar.gz) or prebuilt package (60 MB tar.gz).

Python API

Spark 0.7 adds a Python API called PySpark that makes it possible to use Spark from Python, both in standalone programs and in interactive Python shells. It uses the standard CPython runtime, so your programs can call into native libraries like NumPy and SciPy. Like the Scala and Java APIs, PySpark will automatically ship functions from your main program, along with the variables they depend on, to the cluster. PySpark supports most Spark features, including RDDs, accumulators, broadcast variables, and HDFS input and output.

Spark Streaming Alpha

Spark Streaming is a new extension of Spark that adds near-real-time processing capability. It offers a simple and high-level API, where users can transform streams using parallel operations like map, filter, reduce, and new sliding window functions. It automatically distributes work over a cluster and provides efficient fault recovery with exactly-once semantics for transformations, without relying on costly transactions to an external system. Spark Streaming is described in more detail in these slides and our technical report. This release is our first alpha of Spark Streaming, with most of the functionality implemented and APIs in Java and Scala.

Memory Dashboard

Spark jobs now launch a web dashboard for monitoring the memory usage of each distributed dataset (RDD) in the program. Look for lines like this in your log:

15:08:44 INFO BlockManagerUI: Started BlockManager web UI at http://mbk.local:63814

You can also control which port to use through the spark.ui.port property.

Maven Build

Spark can now be built using Maven in addition to SBT. The Maven build enables easier publishing to repositories of your choice, easy selection of Hadoop versions using the Maven profile (-Phadoop1 or -Phadoop2), as well as Debian packaging using mvn -Phadoop1,deb install.

New Operations

This release adds several RDD transformations, including keys, values, keyBy, subtract, coalesce, zip. It also adds SparkContext.hadoopConfiguration to allow programs to configure Hadoop input/output settings globally across operations. Finally, it adds the RDD.toDebugString() method, which can be used to print an RDD’s lineage graph for troubleshooting.

EC2 Improvements

Spark will now read S3 credentials from the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, if set, making it easier to access Amazon S3.
This release fixes a bug with S3 access that would leave streams open when they are not fully read (e.g. when calling RDD.first() or a SQL query with a limit), causing nodes to hang.
The EC2 scripts now support both standalone and Mesos clusters, and launch Ganglia on the cluster.
Spark EC2 clusters can now be spread across multiple availability zones.

Other Improvements

Shuffle operations like groupByKey and reduceByKey now try to infer parallelism from the size of the parent RDD (unless spark.default.parallelism is set).
Several performance improvements to shuffles.
Standalone deploy cluster now spreads jobs out across machines by default, leading to better data locality.
Better error reporting when jobs aren’t being launched due to not enough resources.
Standalone deploy web UI now includes JSON endpoints for querying cluster state.
Better support for IBM JVM.
Default Hadoop version dependency updated to 1.0.4.
Improved failure handling and reporting of error messages.
Separate configuration for standalone cluster daemons and user applications.
Significant refactoring of the scheduler codebase to enable richer unit testing.
Several bug and performance fixes throughout.

Compatibility

This release is API-compatible with Spark 0.6 programs, but the following features changed slightly:

Parallel shuffle operations where you don’t specify a level of parallelism use the number of partitions of the parent RDD instead of a constant default. However, if you set spark.default.parallelism, they will use that.
SparkContext.addFile, which distributes a file to worker nodes, is no longer guaranteed to put it in the executor’s working directory—instead, you can find the directory it used using SparkFiles.getRootDirectory, or get a particular file using SparkFiles.get. This was done to avoid cluttering the local directory when running in local mode.

Credits

Spark 0.7 was the work of many contributors from Berkeley and outside—in total, 31 different contributors, of which 20 were from outside Berkeley. Here are the people who contributed, along with areas they worked on:

Mikhail Bautin — Maven build
Denny Britz — memory dashboard, streaming, bug fixes
Paul Cavallaro — error reporting
Tathagata Das — streaming (lead developer), 24/7 operation, bug fixes, docs
Thomas Dudziak — Maven build, Hadoop 2 support
Harvey Feng — bug fix
Stephen Haberman — new RDD operations, configuration, S3 improvements, code cleanup, bug fixes
Tyson Hamilton — JSON status endpoints
Mark Hamstra — API improvements, docs
Michael Heuer — docs
Shane Huang — shuffle performance fixes
Andy Konwinski — docs
Ryan LeCompte — streaming
Haoyuan Li — streaming
Richard McKinley — build
Sean McNamara — streaming
Lee Moon Soo — bug fix
Fernand Pajot — bug fix
Nick Pentreath — Python API, examples
Andrew Psaltis — bug fixes
Imran Rashid — memory dashboard, bug fixes
Charles Reiss — fault recovery fixes, code cleanup, testability, error reporting
Josh Rosen — Python API (lead developer), EC2 scripts, bug fixes
Peter Sankauskas — EC2 scripts
Prashant Sharma — streaming
Shivaram Venkataraman — EC2 scripts, optimizations
Patrick Wendell — streaming, bug fixes, examples, docs
Reynold Xin — optimizations, UI
Haitao Yao — run scripts
Matei Zaharia — streaming, fault recovery, Python API, code cleanup, bug fixes, docs
Eric Zhang — examples

Thanks to everyone who contributed!

Spark/Shark Tutorial for Amazon EMR

matei — Sun, 24 Feb 2013 22:43:50 +0000

This weekend, Amazon posted an article and code that make it easy to launch Spark and Shark on Elastic MapReduce. The article includes examples of how to run both interactive Scala commands and SQL queries from Shark on data in S3. Head over to the Amazon article for details. We’re very excited because, to our knowledge, this makes Spark the first non-Hadoop engine that you can launch with EMR.

Spark 0.6.2 released

matei — Fri, 08 Feb 2013 03:16:53 +0000

We recently released Spark 0.6.2, a new version of Spark. This is a maintenance release that includes several bug fixes and usability improvements (see the release notes). We recommend that all users upgrade to this release.

Spark Release 0.6.2

matei — Fri, 08 Feb 2013 03:05:16 +0000

Spark 0.6.2 is a maintenance release that contains several bug fixes and usability improvements. You can download it as a source package (2.5 MB tar.gz) or prebuilt package (48 MB tar.gz).

We recommend that all Spark 0.6 users update to this maintenance release.

The fixes and improvements in this version include:

A number of fault tolerance fixes regarding detecting dead nodes, handling missing map output fetches, and allowing failed nodes to rejoin the cluster
Documentation fixes that clarify the configuration for the standalone mode and improve the quick start instructions
A connection reuse bug fix that improves shuffle performance
Support for launching a cluster across multiple availability zones in the EC2 scripts
Support for deleting security groups when an EC2 cluster is terminated
Improved memory configuration for the standalone deploy cluster daemons: instead of using SPARK_MEM for their memory, which often led people to give them much more memory than they intended, they now use a separate variable, SPARK_DAEMON_MEMORY, with a reasonable default of 512 MB
Fixes to the Windows run scripts for Spark
Better detection of a machine’s external IP address
Several small optimizations and bug fixes

In total, eleven people contributed to this release:

Stephen Haberman (bug fix)
Shane Huang (shuffle fix)
Fernand Pajot (bug fix)
Andrew Psaltis (bug fix)
Imran Rashid (standalone cluster, bug fix)
Charles Reiss (fault recovery fixes, node re-registration, tests)
Josh Rosen (fault recovery, Java API fixes, deploy scripts)
Peter Sankauskas (EC2 scripts)
Lee Moon Soo (bug fix)
Patrick Wendell (bugs, docs)
Matei Zaharia (fault recovery, UI, docs, bug fixes)