Writing in IntelliJ, running on Databricks

So, Apache Spark is getting popular and I’m starting to get a hang of it. Our team uses Databricks, and I found it (and the whole concept of notebooks) great for small things. However, when it comes to writing more complex programs, I’m very much used to convenience of IntelliJ for writing, testing and refactoring the code. Moreover, the new Dataset[T] API allows you to rely on type safety and depend much less on constant re-runs.

Read More