A library for reading public search results from Reddit using Spark Streaming.
Before you start using the Reddit stream for Spark, you will need to make sure you have registered a Reddit app in your account. If you have not done so, you can follow these steps:
- Log in and click on the
preferences
link on the top right.
- Click the
apps
tab.
- Click the
Create an app
button and fill in the form.
Run a demo via:
# set up all the requisite environment variables
export REDDIT_APPLICATION_ID="..."
export REDDIT_APPLICATION_TOKEN="..."
# compile scala, run tests, build fat jar
sbt assembly
# run locally
java -cp target/scala-2.11/streaming-reddit-assembly-0.0.1.jar RedditDemo standalone
# run on spark
spark-submit --class RedditDemo --master local[2] target/scala-2.11/streaming-reddit-assembly-0.0.1.jar spark
Add to your own project by adding this dependency in your build.sbt
:
libraryDependencies ++= Seq(
//...
"com.github.catalystcode" %% "streaming-reddit" % "0.0.1",
//...
)
Currently, this streaming library polls Reddit's /r/all/search.json endpoint at interval that conforms to Reddit's API guidelines (http://github.com/reddit/reddit/wiki/API). However, at some point in the near future, this will be migrated to use Reddit live using its websockets support.
- Configure your credentials via the
SONATYPE_USER
andSONATYPE_PASSWORD
environment variables. - Update
version.sbt
- Run
sbt sonatypeOpen "enter staging description here"
- Run
sbt publishSigned
- Run
sbt sonatypeRelease