An sbt plugin for launching and scheduling Oozie applications.
- sbt (0.13.5+ or 1.0.0+)
- An installation of Hadoop (>= 2.6.x) to target
- An installation of Oozie (>= 4.0.x) to target
Add the following line to project/plugins.sbt
. See the Using plugins section of the sbt documentation for more information.
addSbtPlugin("com.tapad.sbt" % "sbt-oozie" % "0.1.0")
By default, sbt-oozie assumes the OOZIE_URL
environment variable is set and is pointing to the Oozie server's request endpoint (e.g. http://my-oozie-server:11000/oozie
.
If the OOZIE_URL
environment variable cannot be set, the Oozie server URL can be provided to the oozieUrl
setting key in your build definition.
Remember to enable the OoziePlugin
in your build definition, or if using a multi-project setup, on each subproject that you wish to integrate with Oozie.
hadoopClasspath := hadoopClasspathFromExecutable.value // used by sbt-hadoop, see next section of README.md
oozieUrl := "http://my-oozie-server:11000/oozie" // required iff OOZIE_URL env variable is not set
enablePlugins(OoziePlugin)
sbt-oozie will transitively bring sbt-hadoop into your project once enabled.
sbt-hadoop is used for uploading necessary artifacts (Oozie workflow and coordinator definitions, and if necessary, application JAR files) to HDFS, or any file system or storage backend that implements the Hadoop FileSystem
abstraction (e.g. Amazon S3, Google Cloud Storage).
The oozieUpload
task is used to upload a given Oozie application to HDFS for subsequent submittal, execution, or scheduling.
From the sbt project which has enabled the OoziePlugin
, issue:
> oozieUpload <name of application>
Tab-completion is supported:
> oozieUpload <TAB>
baz/quux baz/qux foo
> oozieUpload foo
[info] Uploading foo to /oozie/applications/foo
Your application will be uploaded to the path defined by the application's oozie.wf.application.path
or oozie.coord.application.path
property.
To run an Oozie workflow that has been uploaded to HDFS, invoke the oozieRun
task.
oozieRun
takes, as a parameter, the name of an application that exists in your project.
An Oozie application is any directory that exists in your project's oozieLocalApplications
directory that contains a workflow.xml
file.
Tab-completion is supported:
> oozieRun <TAB>
baz/quux baz/qux foo
> oozieRun foo
[info] 0044773-170912171845818-oozie-oozi-W
The workflow ID of your application will be printed to the log.
Scheduling an Oozie coordinator that has been uploaded to HDFS also uses the oozieRun
task.
An Oozie coordinator application has the same requirements as a workflow application with the addition of also including a coordinator.xml
file.
Again, oozieRun
takes the name of your Oozie application as a parameter.
Tab-completion is supported:
> oozieRun <TAB>
baz/quux baz/qux foo
> oozieRun baz/quux
[info] 0044790-170912171845818-oozie-oozi-C
The coordinator ID of your application will be printed to the log.
Kill a running coordinator by application name or by coordinator ID using oozieKill
:
> oozieKill baz/quux
> oozieKill 0044790-170912171845818-oozie-oozi-C
The twirl templating engine can be leveraged to help author Oozie applications.
NOTE: If using sbt 1.0.x, sbt-twirl requires that you use sbt version 1.0.1 or greater due to the Append
instance backwards compatibility issue addressed in the release notes.
Add the following lines to project/plugins.sbt
.
addSbtPlugin("com.tapad.sbt" % "sbt-oozie" % "0.1.0")
addSbtPlugin("com.tapad.sbt" % "sbt-oozie-templating" % "0.1.0")
Also be sure to enable both the sbt-oozie and the sbt-oozie-templating plugins in your build.sbt
file:
enablePlugins(OoziePlugin, OozieTemplatingPlugin)
Add your twirl templates to the location specifed by oozieTemplating:sourceDirectory
. By default, this location will be the templates
subdirectory inside of your project's resources directory (e.g. src/main/resources/templates
).
For the following example, a template with the file name workflow.scala.xml
has been added to src/main/resources/templates/my_oozie_application/workflow.scala.xml
. Its contents are:
@()
<workflow>@helper("bar")</workflow>
The helper template is placed alongside workflow.scala.xml
, in the same directory, in a file called helper.scala.xml
:
@(foo: String)
<foo>@{foo}</foo>
Generate your Oozie application by invoking oozieEvaluateTemplates
. The results from evaluating these templates will be placed in oozieTemplating:target
, which by default, will be the generated
subdirectory of your project's resources directory (e.g. src/main/resources/generated
).
The values for the settings provided by sbt-oozie-templating, given a default build definition, can be found in the table below:
Setting key (scope:name) | Default value |
---|---|
templating:sourceDirectory | src/main/resources/templates |
templating:target | src/main/resources/generated |
These can be customized to suit your project's structure.
Non-templated, static resources will be copied to oozieTemplating:target
with no modifications made to their contents.
sbt-oozie-templating need not only be used for templating Oozie XML files. It is possible to template any type of resource and evaluate it for inclusion in your application. This is a handy way of providing boilerplate and snippets that typically would have been copied from application to application.
For more information, refer to the scripted integration test found at templating-plugin/src/sbt-test/sbt-oozie/simple.
Various Oozie functionality is exposed by the sbt-oozie plugin. The following settings, tasks, and commands are available:
Setting, task, or command name | Description |
---|---|
oozieUrl | The Oozie server URL |
oozieUser | Username that will be used when interacting with Oozie |
oozieUpload | Upload a given Oozie application to HDFS |
oozieCoordinatorInfo | Retrieve information about a given coordinator (via ID or name) |
oozieWorkflowInfo | Retrieve information about a given workflow (via ID or name) |
oozieDryRun | Test a given Oozie application by executing a dryrun |
oozieRun | Run a given Oozie application |
oozieSubmit | Submit a given Oozie application |
oozieStart | Start a submitted Oozie (via ID or name) |
oozieRerun | Rerun one or more Oozie coordinator actions (via ID or name) |
oozieIgnore | Ignore one or more Oozie coordinator actions (via ID or name) |
oozieDryUpdate | Test updating an Oozie job by executing a dryrun |
oozieUpdate | Update a running Oozie application applying any changes that exist locally |
oozieChange | Change a value (endtime, concurrency, or pausetime) of an Oozie coordinator |
oozieSuspend | Suspend an Oozie job (via ID or name) |
oozieResume | Suspend an Oozie job (via ID or name) |
oozieKill | Kill an Oozie job (via ID or name) |
oozieLocalApplications | The local directory in this project that contains Oozie applications |
oozieLocalShare | A local directory that contains resources that will be included in every Oozie application |
oozieHdfsApplications | The remote path where Oozie applications will exist |
- plugin
- library
- templating-plugin
- templating-library
- util
The necessary wiring to that exposes Oozie functionality via sbt.
An underlying service used to interface with the Oozie Java client.
An sbt plugin that provides the capability to template Oozie applications using twirl.
Supporting library code leveraged by the templating plugin.
Supporting library code and common abstractions.
Although unit tests exist, the main features and functionality of sbt-oozie
and sbt-oozie-templating
are tested using sbt's scripted-plugin
. scripted
tests exist in the src/sbt-test
directories of the sbt-oozie
and sbt-oozie-templating
subprojects.
To run these tests, issue scripted
from an sbt session after targeting either of these subprojects:
$ sbt
> project plugin
> scripted
> project templatingPlugin
> scripted
To selectively run a single scripted
test suite, issue scripted <name of plugin>/<name of test project>
. e.g. scripted sbt-oozie/simple
.
Please note that publishLocal
will be invoked when running scripted
. scripted
tests take longer to run than unit tests and will log myriad output to stdout. Also note that any output written to stderr during the execution of a scripted
test will result in ERROR
level log entries. These log entries will not effect the resulting status of the actual test.
sbt-oozie
uses https://github.com/sbt/sbt-release. Simply invoke release
from the root project to release all artifacts.