This is a self-documentation of learning distributed data storage, parallel processing, and Linux OS using Apache Hadoop, Apache Spark and Raspbian OS. In this project, 3-node cluster will be setup using Raspberry Pi 4, install HDFS and run Spark processing jobs via YARN.

Developing Spark with sbt

A modal REPL protocol for Neovim, e.g. to use the Scala REPL or spark-shell.

Starting spark locally

This Big Data project consists of obtaining data on vehicle theft in the city of São Paulo and consolidating it in a counting and heat map, in order to show areas with a higher index of this type of crime. All applicable in AWS Resources.

The idea with this project is to learn the basis about how to use spark with generated ROOT files from LHC experiment.

Scripts, libraries and more utilities for spark-shell and spark-submit testing.

Docker Compose sandbox for various test environments

An example of installation Apache Spark on AWS

A collection of Spark/Scala example programs

Library for remote JVM ExecutorService with only dependency being password-less SSH -- Run clustered Hadoop/Spark jobs from IDE -- IDE-pimped Spark shell with full auto-completion!

Raspberry pi OS installation and system call demonstration