A tool to record and visualize metrics captured from Cascading (Scalding) workflows at runtime.
Designed to target the pain points of analysts and end users of Cascading, Sahale provides insight into a workflow's runtime resource usage and makes job debugging and locating relevant Hadoop logs easy. The tool reveals optimization opportunities by exposing inefficient MapReduce jobs in a larger workflow, and enables users to track the execution history of their workflows.
Sahale has been verified to work with several Cascading DSLs, but the example projects and several features are tailored to Twitter's Scalding DSL, which is the DSL we use at Etsy. Pull requests for better support for other flavors of Cascading are welcome!
There are two components to Sahale, a flowtracker
JAR that contains the code that tracks running jobs and a Node.JS dashboard for visualizing the tracked jobs.
The flowtracker
JAR is published to Maven Central for easy inclusion in your projects:
<dependency>
<groupId>com.etsy.sahale</groupId>
<artifactId>flowtracker_2.11</artifactId>
<version>2.0.0</version>
</dependency>
or, for the Google-Auth-enabled variant:
<dependency>
<groupId>com.etsy.sahale</groupId>
<artifactId>flowtracker-gcp_2.11</artifactId>
<version>2.0.0</version>
<exclusions>
<exclusion>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
</exclusion>
</exclusions>
</dependency>
Note that we recommend excluding the transitive dependency google-api-client
when running on Google Compute Engine / Dataproc, because the Dataproc clusters already have versions of google-api-client
preinstalled. We have run into errors when including our own copy of google-api-client
, despite our attempts to match the packaged version with the preinstalled version.
You will need add some configuration for the flow tracker. Create the file src/main/resources/flow-tracker.properties
in your project with the following contents:
host=http://host.where.nodejs.server.is.running
port=5735
The host
setting will be the hostname of the server where you install the dashboard (see below)
You can also add the setting sahale.flow.selected.configs
in this file. This is a comma-delimited list of Hadoop configuration properties that will be tracked along with the other job information.
If this configuration step is not performed job tracking will fail.
-
A MySQL instance that Sahale can use.
-
A server with Node.JS and
npm
installed.
-
Set up the MySQL database. Follow the instructions in
create_db_tables.sql
to do this. -
Clone the Sahale repo onto the server where you would like to run the dashboard.
-
Modify
db-config.json
to point to your database. -
Run
npm install
from the root directory of the Sahale repo checkout.
Now you can run node app
to start the Sahale dashboard. It will be running on port 5735, so go to <hostname>:5735
in your browser to see the dashboard.
Sahale FlowTracker can be configured to send Google-Auth bearer tokens along with its requests to the Sahale server. This is used when the Sahale server is running in Google Cloud Platform, for example under Google App Engine with authentication enabled. To use this feature, include flowtracker-gcp
in your Cascading project, and use the com.etsy.sahale.GoogleAuthFlowTracker class instead of the ordinary com.etsy.sahale.FlowTracker. GoogleAuthFlowTracker
works on Hadoop clusters running in Google Compute Engine or Dataproc without any configuration, generating tokens using the application default service account credentials. For clusters not running in GCE/Dataproc, it also accepts a service account JSON file as an optiona argument.
Sahale server does not yet support Google authentication directly; our approach has been to run Sahale server behind an authenticating proxy server, for which there are several open source options.
Sahale server (NodeJS app) and FlowTracker (Scala client jar) must always maintain parity between clients and server versions. An upgrade deployment must coordinate the distribution of the new client jar and restart of the updated server. In rare cases (tagged in the Git repo) Sahale will make breaking changes that will require addition steps. The two notable cases are listed below:
In addition to some minor fixes, we upgrade to scalding 0.17.4. We are alo publishing scala 2.11 and 2.12 artifacts for the first time. We will drop scala 2.11 support in a future version.
We added a new FlowTracker module, flowtracker-gcp
, which adds a new FlowTracker subclass, GoogleAuthFlowTracker
. This subclass adds Authorization
headers to its requests to Sahale server, passing OAuth bearer tokens generated from Google service accounts. This class is meant to be used with a Sahale server running in an environment employing service-to-service authentication via, for example, Google App Engine.
There were no data model or other incompatibility changes in this release.
There were no data model or other incompatibility changes, but Sahale jar is now built with Scala 2.11 and published as flowtracker_2.11
.
There were no data model or other incompatibility changes, but the flowtracker
JAR is now being published to Maven Central. It is no longer required to manually build this JAR; see the above instructions for its use and configuration.
The data model has changed. Please recreate your backing MySQL tables using the script in src/main/sql
before using the new client jar or restarting the NodeJS app. The old tables and data will remain, the new tables will be suffixed with _new
unless you opt to alias them. No additional change is required.
Mark incompatible changes between older and newer versions of Scala/Scalding. If your org still uses older versions of Scala/Scalding, please see this commit. All other changes and feature upgrades in the 0.6 line will work as expected with this commit reverted and your own choice of versions applied to the pom.xml
.
Sahale was handmade at Etsy.com and is named for Sahale Mountain, which is a wonderful vantage point from which to view the Cascades