Pyplot in Jupyter inside pyenv on El Capitan

With the new root protection feature in El Capitan, it’s hard to install pip packages into the system python. Besides, it’s generally considered bad practice to clutter up your system python’s library with packages. Most developers prefer to set up their own virtualenv for python and then install packages into that local environment. I did the same for setting up Jupyter Notebook and everything worked as expected. Well, mostly everything.

When I tried to import matplotlib.pyplot, I got a rather long error:

RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are Working with Matplotlib in a virtual enviroment see ‘Working with Matplotlib in Virtual environments’ in the Matplotlib FAQ

So it seems that pyplot doesn’t work well with python inside a virtualenv, at least on OS X. The FAQ describes a way to work around this by adding a script to your virtualenv and then invoking that script as your interpreter instead of invoking python directly. That’s half the problem solved, the other half being getting IPython to use this script. A little googling provided a solution to that as well.

So here’s the combined solution, including how to create and use the virtualenv.

Install required packages

# Install pyenv

brew install pyenv

# A convenient way to create virtualenvs
brew install pyenv-virtualenv

# Install the desired version of python
pyenv install 2.7.10

# Create a virtualenv, I'm calling mine "jupyter"
pyenv virtualenv 2.7.10 jupyter

# Activate the virtualenv
# You'll have to do this everytime before running "jupyter notebook"
pyenv activate jupyter

# Install all the packages your heart desires
pip install notebook matplotlib numpy scipy sklearn

Create the wrapper around python

Find the path to your virtualenv’s python binary:

# Only run this if you're not already in the jupyter virtualenv
pyenv activate jupyter

# Where's python?
pyenv which python

This will give you a path that looks like /Users/amey/.pyenv/versions/jupyter/bin/python. Create a new file named frameworkpython (or whatever you want, this is the name suggested by the matplotlib FAQ) and put the following inside:

#!/bin/bash

# what real Python executable to use
PYVER=2.7
PATHTOPYTHON=/usr/bin/
PYTHON=${PATHTOPYTHON}python${PYVER}

# find the root of the virtualenv, it should be the parent of the dir this script is in
ENV=`$PYTHON -c "import os; print os.path.abspath(os.path.join(os.path.dirname(\"$0\"), '..'))"`

# now run Python with the virtualenv set as Python's HOME
export PYTHONHOME=$ENV
exec $PYTHON "$@"

Remember to use the correct version in the line PYVER=2.7 and the correct path to your system’s python in the line PATHTOPYTHON=/usr/bin/. The system python directory can usually be found by running which python in a new shell session. If you get a path to pyenv (like /Users/amey/.pyenv/shims/python), you’ll have to go hunting. It should be at /usr/bin/python or /usr/local/bin/python.

Mark the new file as executable with chmod +x /path/to/frameworkpython, and that’s the wrapper taken care of. Next we need to create a new kernel for jupyter and tell it to use the newly-created frameworkpython as the interpreter. For this, we need to find out where jupyter is reading its configuration files from. Run jupyter --paths inside the virtualenv, you should get output similar to the following:

config:
    /Users/amey/.jupyter
    /Users/amey/.pyenv/versions/jupyter/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /Users/amey/Library/Jupyter
    /Users/amey/.pyenv/versions/jupyter/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
/Users/amey/Library/Jupyter/runtime

The first data directory is usually the one we’d like to use. So create a directory there for the new kernel. I decided to call my new kernel matplotlib to make it easy to identify, and so I ran

mkdir -p ~/Library/Jupyter/kernels/matplotlib

And then create a configuration file named kernel.json for the new kernel inside the directory you just created. For me, this was ~/Library/Jupyter/kernels/matplotlib/kernel.json. Paste the following inside this file:

{
 "argv": [ "/Users/amey/.pyenv/versions/jupyter/bin/frameworkpython", "-m", "ipykernel",
          "-f", "{connection_file}"],
 "display_name": "matplotlib",
 "language": "python"
}

Tweak the path inside argv and the display_name variable according to where your wrapper is and the name of your new kernel respectively. And that should do it. When you activate the virtualenv and launch jupyter, you should see an option to create a notebook with your new kernel and Matplotlib and Pyplot should work fine inside it if everything went according to plan.