Using Light Clusters in Databricks

Disclaimer: Post written in early March 2023, and as Databricks changes things constantly, this information may be outdated in the near future. We try to keep everything up to date, but sometimes we may miss some things. So please, always verify things with the official documentation.

If you have explored the different pricing options for Databricks, you may have noticed the existence of the “Job Light Compute”, a very cheap option for running our workloads in Databricks, but with no much documentation about it.

Source: https://azure.microsoft.com/en-us/pricing/details/databricks/

In order to be able to use it, you need to deep dive into the interface, and in the Runtime Version selection dropdown, you may see an option for “Light” where you can choose between the different light runtimes available.

As you can see, it works with Scala 2.11 and Spark 2.4… so you probably won’t be able to run your code without changing some things to adapt it.