Geny is a small library that provides push-based versions of common standard library interfaces:

• geny.Generator[T], a push-based version of scala.Iterator[T]

• geny.Writable, a push-based version of java.io.InputStream

• geny.Readable, a pull-based subclass of Writable

More background behind the Writable and Readable interface can be found in this blog post:

• Standardizing IO Interfaces for Scala Libraries

Generator is basically the inverse of a scala.Iterator: instead of the core functionality being the pull-based hasNext and next: T methods, the core is based around the push-based generate method, which is similar to foreach with some tweaks.

Unlike a scala.Iterator, subclasses of Generator can guarantee any clean up logic is performed by placing it after the generate call is made. This is useful for using Generators to model streaming data from files or other sources that require cleanup: the most common alternative, scala.Iterator, has no way of guaranteeing that the file gets properly closed after reading. Even so called "self-closing iterators" that close the file after the iterator is exhausted fail to close the files if the developer uses .head or .take to access the first few elements of the iterator, and never exhausts it.

Although geny.Generator is not part of the normal collections hierarchy, the API is intentionally modelled after that of scala.Iterator and should be mostly drop-in, with conversion functions provided where you need to interact with APIs using the standard Scala collections.

Geny is intentionally a tiny library with one file and zero dependencies, so you can depend on it (or even copy-paste it into your project) without fear of taking on unknown heavyweight dependencies.

geny.Writable is a minimal interface that can be implemented by any data type that writes binary output to a java.io.OutputStream:

Writable allows for zero-friction zero-overhead streaming data exchange between these libraries, e.g. allowing you pass Scalatags Frags directly os.write:

Sending ujson.Values directly to requests.post

Serialize Scala data types directly to disk:

Or streaming file uploads over HTTP:

All this data exchange happens efficiently in a streaming fashion, without unnecessarily buffering data in-memory.

geny.Writable also allows an implementation to ensure cleanup code runs after all data has been written (e.g. closing file handles, free-ing managed resources) and is much easier to implement than java.io.InputStream.

Writable has implicit constructors from the following types:

• String

• Array[Byte]

• java.io.InputStream

And implemented by the following libraries:

• uPickle: implemented by ujson.Value, upack.Msg, and can be constructed from JSON-serializable data structures via upickle.default.stream or upickle.default.writableBinary

• Scalatags: implemented by scalatags.Text.Tag

• Requests-Scala: requests.get.stream(...) methods return a Readable subtype of Writable

• OS-Lib: os.read.stream returns a Readable subtype of Writable

• Cask: cask.Request returns a Readable subtype of Writable

And is accepted by the following libraries:

• Requests-Scala takes Writable in the data = field of requests.post and requests.put

• OS-Lib accepts a Writable in os.write and the stdin parameter of subprocess.call or subprocess.spawn

• Cask: supports returning a Writable from any Cask endpoint

Any data type that writes bytes out to a java.io.OutputStream, java.io.Writer, or StringBuilder can be trivially made to implement Writable, which allows it to output data in a streaming fashion without needing to buffer it in memory. You can also implement Writables in your own datatypes or accept it in your own method, if you want to inter-operate with this existing ecosystem of libraries.

Readable is a subtype of Writable that provides an additional guarantee: not only can it be written to an java.io.OutputStream, it can also be read from by providing a java.io.InputStream. Note that the InputStream is scoped and only available within the readBytesThrough callback: after that the InputStream will be closed and associated resources (HTTP connections, file handles, etc.) will be released.

Readable is supported by the following built in types:

• String

• Array[Byte]

• java.io.InputStream

Implemented by the following libraries

• Requests-Scala: requests.get.stream(...) methods return a Readable

• OS-Lib: os.read.stream returns a Readable

• Cask: cask.Request implements Readable to allow streaming of request data

And is accepted by the following libraries:

• uPickle: upickle.default.read, upickle.default.readBinary, ujson.read, and upack.read all support Readable

• FastParse: fastparse.parse accepts parsing streaming input from any Readable

Readable can be used to allow handling of streaming input, e.g. parsing JSON directly from a file or HTTP request, without needing to buffer the whole file in memory:

You can also implement Readable in your own data types, to allow them to be seamlessly passed into uPickle or FastParse to be parsed in a streaming fashion.

Note that in exchange for the reduced memory usage, parsing streaming data via Readable in uPickle or FastParse typically comes with a 20-40% CPU performance penalty over parsing data already in memory, due to the additional book-keeping necessary with streaming data. Whether it is worthwhile or not depends on your particular usage pattern.