Introduction

This project is our collection of Scala modules.

Requirements

• IntelliJ IDEA
• IntelliJ Scalafmt Plugin
• Scala 2.11
• Spark 2.4.0+

Spark is expected to be a provided dependency, so you should have a working Spark install somewhere, and $SPARK_HOME should be set in your environment.

You should use IntelliJ IDEA (CE is fine). We use the scalafmt IntelliJ IDEA plugin, configured to update on file save, and scalastyle

Some editor config to put in place: Case Class Definition Style

We follow the Twitter Effective Scala style guide.

Saving this here for future reference: Spark + S3

Installing scala and sbt on Mac OS X

Use homebrew:

brew install [email protected]
brew install sbt