RainStorm

Introduction

Welcome to our MP4 project (the final one) for CS 425: Distributed Systems. This repository consists of all the files for RainStorm. We hope that this README will help guide you through understanding and using them.

File and Directory Contents

AutoScale

This contains scripts associated with the AutoScale variant of RainStorm. Please refer to the README.md within this directory for instructions on how to run.

ExactlyOnce

This contains scripts associated with the ExactlyOnce variant of RainStorm. Please refer to the README.md within this directory for instructions on how to run.

Spark Streaming

1. The PySparkstuff contains all the scripts associated with running Spark Streaming on the VMs
2. exp_runs refers to the log files from running Spark on our streaming applications for a fair baseline comparison

Other Important Files

Python scripts at this top-level directory were from previous MPs (PingAck from MP2 and HyDFS from MP3)

This top-level directory consists of several log files across dataset streaming applications outlined in our report.

1. dataset1_autoscale contains a log file for the autoscale experiment for streaming application 1 on dataset 1
2. dataset1_exactlyonce contains log files for the exactly-once experiments for both streaming applications 1 and 2 on dataset 1
3. dataset2_autoscale contains a log file for the autoscale experiment for streaming application 1 on dataset 2
4. dataset2_exactlyonce contains log files for the exactly-once experiments for both streaming applications 1 and 2 on dataset 2
5. stream_execs directory consists of all the .c files we use as executables within RainStorm. This includes scripts for our streaming applications for the report and also for the demo. The top-level Makefile can compile all of these scripts simply by using make.
6. mp4_report.pdf: our report!

Library installation required (and dependencies)

This project requires the aioconsole and aiofiles Python libraries. We also use a Python implementation of gRPCs. The following shell script makes sure that the libraries using RPCs are installed.

#!/bin/bash

# Range of hostnames
START=2601
END=2610
HOST_PREFIX="fa25-cs425-"
HOST_SUFFIX=".cs.illinois.edu"

# Pip packages to install
PACKAGES=("grpcio" "grpcio-tools")  # Add your packages here

# Loop through each VM
for i in $(seq $START $END); do
    HOST="${HOST_PREFIX}${i}${HOST_SUFFIX}"
    echo "Installing packages on $HOST..."
    
    ssh "gajith2@$HOST" "python3 -m pip install --upgrade pip && python3 -m pip install ${PACKAGES[*]}"
    
    if [ $? -eq 0 ]; then
        echo "Packages installed successfully on $HOST"
    else
        echo "Failed to install packages on $HOST"
    fi
done

All proto files can be generated by running the command below for a respective .proto file.

python3 -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. dfs.proto

To comply with course policy, code relating to previous MPs (distributed file system, failure detector, distributed logger) have been redacted. To reproduce, please look at the .proto files as an API to implement the relevant RPCs. Specifically, dfs.proto outlines all of the RPCs that need to be implemented for the underlying distributed file system. Code may also need to be adapted to implement a background failure detector or introducer.