SKIL Documentation

Skymind Intelligence Layer

The community edition of the Skymind Intelligence Layer (SKIL) is free. It takes data science projects from prototype to production quickly and easily. SKIL bridges the gap between the Python ecosystem and the JVM with a cross-team platform for Data Scientists, Data Engineers, and DevOps/IT. It is an automation tool for machine-learning workflows that enables easy training on Spark-GPU clusters, experiment tracking, one-click deployment of trained models, model performance monitoring and more.

Get Started

Memory Settings

Neural networks are memory hungry and more ambitious SKIL projects will require more memory allocated to SKIL and Notebooks. Often a lack of memory appears as a java.lang.OutOfMemoryError and knowing how to change your memory settings can prevent this in the future.

Environment Variables

To fix memory issues in Zeppelin (the system behind Notebooks), increase the amount of JVM RAM available. Zeppelin's default configuration provides 8 GB of RAM on the JVM, and 4 GB of off-heap space. You can expand these by changing the DEFAULT_ZEPPELIN_JVM_ARGS environment variable and deleting the Zeppelin interpreter process in the Processes tab.

There are a couple ways to set this variable:

  • Specify when launching SKIL Docker container with -e
  • Add it to /etc/skil/skil-env.sh (or /etc/profile.d/skil-env.sh if /etc/skil-env.sh doesn't exist)

If you have a GPU such as a K80, the following lines are enough to maximize your system:

DEFAULT_ZEPPELIN_JVM_ARGS=-Xmx16g -Dorg.bytedeco.javacpp.maxbytes=16G -Dorg.bytedeco.javacpp.maxphysicalbytes=16G -Dorg.nd4j.versioncheck=false -Dorg.deeplearning4j.config.custom.enabled=false

Once you have updated your settings, you will need to restart SKIL. If you have a bare-metal SKIL installation:

sudo systemctl stop skil
sudo systemctl start skil

Otherwise you'll need to docker exec into the SKIL Docker instance and restart it from a bash instance.

Docker Memory

In certain cases you may need to ensure Docker has enough memory at runtime in addition to other adjustments. Docker on Windows and OSX have these features built-in to the user interface. If running on Linux or a server and using the Docker CLI, you can pass a value to the --memory flag. Note that this is typically used for limiting memory.

You can read more on the Docker website.

Common Errors

Often when a SKIL workspace or experiment is constrained by memory, it will appear as something similar as:

java.lang.OutOfMemoryError: Cannot allocate new DoublePointer(100000000): totalBytes = 588M, physicalBytes = 4G
	at org.bytedeco.javacpp.DoublePointer.<init>(DoublePointer.java:76)
	at org.nd4j.linalg.api.buffer.BaseDataBuffer.<init>(BaseDataBuffer.java:549)
	...

Depending on the data type of NDArray used in your code, this can also appear with FloatPointers or occur during memory deallocation.

Memory Settings