Install SystemML

1Pre-requisite

Apache Spark 2.x

Set SPARK_HOME to a location where Spark 2.x has installed.

2Setup

      # Install SystemML
      pip install systemml
      
Download and extract SystemML jar (systemml-0.14.0-incubating-SNAPSHOT.jar) file from systemml-0.14.0-incubating-bin.tgz or systemml-0.14.0-incubating-bin.tgz file located on https://dist.apache.org/repos/dist/release/incubator/systemml/0.14.0-incubating/
# Install latest SystemML
pip install https://sparktc.ibmcloud.com/repo/latest/systemml-1.0.0-incubating-SNAPSHOT-python.tgz
Download SystemML jar (systemml-1.0.0-incubating-SNAPSHOT.jar) from https://sparktc.ibmcloud.com/repo/latest/

3Configure Jupyter Notebook (Optional)

# Start Jupyter Notebook Server
PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS="notebook" pyspark --master local[*] --conf "spark.driver.memory=12g" --conf spark.driver.maxResultSize=0 --conf spark.akka.frameSize=128 --conf spark.default.parallelism=100

Toree Kernel Setup (Required for Scala Kernel)

Toree installation

# For detail instructions visit https://github.com/apache/incubator-toree
pip install https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0/snapshots/dev1/toree-pip/toree-0.2.0.dev1.tar.gz

Installation of Toree component in Jupyter

# For detail instructions visit  https://toree.apache.org/docs/current/user/installation/
jupyter toree install —-replace —-interpreters=Scala,PySpark --spark_opts="--master=local --jars <SystemML JAR File>” --spark_home=${SPARK_HOME}

Start Jupyter Notebook Server

jupyter notebook

This will start a default browser with contents from current directory where above command has run. You can create your own notebook example or download sample notebooks from SystemML resository https://github.com/apache/incubator-systemml/tree/master/samples/jupyter-notebooks

Start Jupyter Notebook Server
Start Jupyter Notebook Server

4Run SystemML in batch mode

Download systemml-0.14.0-incubating-bin.tgz or systemml-0.14.0-incubating-bin.tgz file located on https://dist.apache.org/repos/dist/release/incubator/systemml/0.14.0-incubating/ and extract into a directory say SYSTEMML_HOME Once you extract zip.tgz file you will have files required to run steps outlined in instructions link: http://apache.github.io/incubator-systemml/spark-batch-mode

View Sample Notebooks