aws2scala: a Scala-friendly API for AWS

Caution

This is pre-1.0 software, interfaces are subject to change. Please refer to the documentation for each individual service API to find out more.

This library wraps the AWS SDK for Java to make it much more Scala-friendly and simpler to consume. It provides asynchronous and streaming APIs that are easier to use, perform paging automatically, and more idiomatic.

Table of Contents

Features
Supported clients
Getting started
- Add the resolver and dependencies

Features

Fully asynchronous

The AWS SDK for Java provides both synchronous and asynchronous APIs for its clients. Using the synchronous APIs in an asynchronous context, such as a Spray or Akka HTTP route, is problematic. Unfortunately, using the asynchronous APIs is problematic since they use Java futures rather than Scala futures. All of the non-streaming aws2scala APIs are fully asynchronous and return Scala futures, meaning you can compose them and use them with for comprehensions.

The following section contains an example of using a synchronous AWS API asynchronously.

Eliminates most request/result objects

The AWS APIs make extensive use of request and result objects, this is even the case when they are only wrapping just one argument/result object.^[1] aws2scala generally provides methods that take the essential arguments for a request and return the object of interest in the result.

For example, you can compare how creating a role asynchronously differs between using the AWS IAM client and using the aws2scala IAM client.

Using the synchronous AWS API

import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient
import com.amazonaws.services.identitymanagement.model.CreateRoleRequest
import com.monsanto.arch.awsutil.identitymanagement.model.Role

val iam = new AmazonIdentityManagementClient()
val policy = // some policy string
val eventualRole =
    Future { // (1)
        val request = new CreateRoleRequest()  // (2)
                        .withRoleName("MyRole")
                        .witAssumedRolePolicyDocument(policy)
        val result = blocking { iam.createRole(request) } // (3)
        val awsRole = result.getRole // (4)
        Role.fromAws(awsRole) // (5)
    }

Execute the code asynchronously (we are using the synchronous AWS client)
Create an AWS request object
Execute the request withing a blocking context, getting the result object
Extract the mutable AWS Role bean from the result
Build an immutable aws2scala Role instance from the AWS result

Using the asynchronous aws2scala IAM client

import com.monsanto.arch.awsutil.AsyncAwsClient

val iam = AsyncAwsClient.Default.identityManagement
val policy = // some policy string
val role = iam.createRole("MyRole", policy) // (1)

This method automatically builds the request object and unpacks the result object, returning an immutable Role value

Uses immutable Scala collections, options, and values

The AWS APIs make extensive use of JavaBeans-style classes. These are mutable objects that use Java-style getters and setters. In cases where one of these properties may be optional, it is a null value. Additionally, all collections are mutable Java collections.

aws2scala opts to take a more idiomatic Scala approach, it:

Uses immutable Seq and Map objects for all arguments and return values,
Represents compound data structures using case classes, ensuring immutability and allowing for pattern matching, and
Uses an Option whenever a value is optional.

Pages automatically

Many AWS APIs that perform listings will return paged results. Unfortunately, these paging APIs suffer from a couple of drawbacks:

Preparing a new request usually requires getting data from the previous result. This means that is difficult to process pages fully asynchronously.
They are inconsistent. Some use Result.getNextToken and Request.getNextToken, others use Result.getMarker and Request.setMarker, and still others use Result.getNextMarker and Request.setMarker.

You can compare how to manually perform the work using the AWS client versus using aws2scala. Note that in both versions, the future will not complete until all pages have been retrieved from AWS. If this is undesirable, i.e. you only want to request new pages as needed, use a streaming client.

Listing all roles asynchronously using the synchronous AWS API

import com.amazonaws.services.identitymanagement.AmazonIdentityManagementClient
import com.amazonaws.services.identitymanagement.model.ListRolesRequest
import com.monsanto.arch.awsutil.identitymanagement.model.Role
import scala.collection.JavaConverters._

val iam = new AmazonIdentityManagementClient()

val eventualRoles: Future[Seq[Role]] =
    Future {
        val request = new ListRolesRequest // (1)
        var result: ListRolesResult = new ListRolesResult // (2)
        val rolesListBuilder = Seq.newBuilder[Role] // (3)

        do {
            Option(request.getMarker).foreach(m ⇒ request.setMarker(m)) // (4)
            result = blocking { iam.listRoles(request) } // (5)
            val pagedRoles = result.getRoles.asScala.map(Role.fromAws) // (6)
            rolesListBuilder ++= pagedRoles // (7)
        } while (result.isTruncated) // (8)

        rolesListBuilder.result() // (9)
    }

Create the new request
Create an empty result
Create builder to accumulate results
Set the next pages marker if it is in the result
Get results in a blocking context
Convert the Java collection of JavaBeans to a (mutable) Scala collection of case class instances
Add to the accumulated result
Repeat until there are no further pages
Get the final (immutable) Scala collection of immutable Role instances

Listing all roles asynchronously using aws2scala

import com.monsanto.arch.awsutil.AsyncAwsClient

val iam = AsyncAwsClient.Default.identityManagement
val roles = iam.listRoles() // (1)

Can it get any easier than this?

Streaming

In addition to the asynchronous APIs, all aws2scala functionality is available through streaming APIs that are built using Akka streams. For example, the following listing constructs and runs a flow that:

Gets the current IAM user,
Creates a role for that user’s account,
Attaches a policy to the new role, and
Emits the role that was created.

While the same result can be achieved using the asynchronous APIs and future composition, creating reusable graphs can make code easier to understand. Additionally, the various listing flows that process paged results will emit items as soon as they are retrieved. This allows for the construction of graphs that can process items in a listing as they are available without having to wait for the listing to complete.

Setting up a role for the current IAM user

import com.monsanto.arch.awsutil.identitymanagement.model._
import com.monsanto.arch.awsutil.StreamingAwsClient

val s3ReadOnlyPolicy = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
def createAssumRolePolicy(user: User): String =
    s"""{
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "sts:AssumeRole",
                    "Effect": "Allow",
                    "Principal": { "AWS": "arn:aws:iam::${user.account}:root" }
                }
            ]
        }"""

val iam = StreamingAwsClient.Default.identityManagement
val createdRole: Future[Role] =
    Source.single(GetUserRequest.currentUser) // (1)
        .via(iam.userGetter) // (2)
        .map(user ⇒ CreateRoleRequest("MyRole", createAssumeRolePolicy(user))) // (3)
        .via(iam.roleCreator) // (4)
        .flatMapConcat { role ⇒
            Source.single(AttachRolePolicyRequest(role.name, s3ReadOnlyPolicy)) // (5)
                .via(iam.rolePolicyAttacher)
                .map(_ ⇒ role) // (6)
        }
        .runWith(Sink.head) // (7)

Start with a single GetUserRequest to get the current user
Send it through the IAM userGetter flow, which emits a User instance
Now, transform the the user into a CreateRoleRequest
Send it through the IAM roleCreator flow, which emits a Role instance
Create a subflow that will attach the AmazonS3ReadOnlyAccess policy to the role.
Have the policy emit the role that was passed in (rolePolicyAttacher emits a role ARN)
Runs the entire flow, resulting in a future with the created role

Supported clients

The following clients are currently available in aws2scala:

CloudFormation

Create, describe, list, and delete stacks
Describe stack events
List stack resources
Validate templates

Elastic Compute Cloud (EC2)

Create, describe, and delete key pairs
Describe instances

Identity Management (IAM)

Create, list, and delete roles
Attach, list, and detach managed policies to roles
Get users

Key Management Service (KMS)

Create, describe, and list keys
Schedule and cancel deletion of keys
Generate data keys with and without plaintext keys
Encrypt and decrypt

Relational Database Service (RDS)

Create, describe, and delete DB instances

Simple Storage Service (S3)

Create, list, check existence of, empty, and delete buckets
Manage bucket policies and tagging
Upload and download strings, byte arrays, and files
Copy, list, and delete objects
Get object URLs

Security Token Service (STS)

Assume roles

Simple Notification Service (SNS)

Create, list, and delete topics
Add and remove topic permissions
Create, confirm, list, and unsubscribe subscriptions
Create, list, and delete platform applications
Create, list, and delete platform endpoints
Get and set attributes for:
- Topics
- Subscriptions
- Platform applications
- Platform endpoints
Publish

Getting started

Add the resolver and dependencies

You will need to add the following to your build.sbt:

Add the JCenter resolver to get the aws2scala dependency,
The aws2scala dependency itself, and
Any AWS SDK dependencies you may need.^[2]

Adding aws2scala with KMS support to build.sbt

resolvers += Resolver.jcenterRepo                           // (1)

libraryDependencies ++= Seq(
    "com.monsanto.arch" %% "aws2scala"         % "0.4.1"    // (2)
    "com.amazonaws"      % "aws-java-sdk-kms"  % "1.10.52"  // (3)
)

1. In all fairness, most request objects are subclasses of AmazonWebServiceRequest and allow setting things like request timeouts, which is not yet implemented in aws2scala.

2. aws2scala only transitively depends on aws-java-sdk-core. It uses the provided scope for all other dependencies, allowing consumers to only pull in the library dependencies they need

bayer-group / aws2scala 0.6.0