since
0.3.1
it is compiled against scala2.12
for spark3.x
only
earlier versions compiled against both scala
2.11
and2.12
and aimed to cover spark2.4.x
When it comes to specifying options on spark readers and writers sometimes it may look inconvenient
when you need to work with effects like Option
, Either
, Try
.. Although last 2 might mot be considered as a
good candidate, the first, which is Option
often is taken into the loop.
And code becomes messy...
val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???
val reader = spark.read.format("parquet").option("foo", "bar")
extra1Option foreach { reader.option("extra-option-1", _) }
extra2Option.right foreach { reader.option("extra-option-2", _) }
val df = reader.load()
So with this simple library you can acheave this level of readiness of code
import com.github.andyglow.spark.options._
val extra1Option: Option[String] = ???
val extra2Option: Either[String, Long] = ???
val spark: SparkSession = ???
spark.read
.format("parquet")
.option("foo", "bar")
.option("extra-option-1", extra1Option)
.option("extra-option-2", extra2Option)
.load()
libraryDependencies += "com.github.andyglow" %% "spark-option-setter" % "0.0.1"