A Workflow for Data Science
Jupyter notebooks are a phenomenal tool for investigating and interacting with data. The interactive nature of the notebooks allows for quick and effective interrogation of the data and more generally the prototyping of an idea. By combining interaction, the documentation, the code, and the visualisations, the notebooks are also a fantastic tool for communicating ideas. However, notebooks are not the most appropriate for all the code we write, with many there being many detractors to using notebooks and there are definitely valid reasons for disliking the notebook. However, most of the issues can be addressed by recognising that the jupyter notebook is one of many tools that we have available, and knowing when to move beyond the notebook can help us make the most of it.
While I am using python to illustrate these steps and examples, the same ideas apply for any language being used in the notebook just with some slightly different specifics on the how.
The first step within the process is the exploration and development of an idea. This is where we are just getting started looking at a problem and we probably haven't written any code yet. In this phase of development we want to make full use of the interactivity of jupyter. This is where we can easily play around to make sure we are using the right options for reading in our file, trying lots of different types of analysis, while having a record of what we have done so that we can easily regenerate a previous state when we break something.
It is here where the interactive nature of the juptyer notebook really stands out. It is easy to introspect the state at each step in the process, looking at the problem in different ways to properly understand what is going on. We also have the visualisations, which allow us to generate nice figures representing the data, however also includes the pretty printed outputs, like the nicely formatted tables of dataframes, which make working with the data easier.
The way we retain state within the notebook is incredibly appealing, particularly where the first step of the process is time consuming, like loading the data from a URL, performing a complex database query, or doing some intermediate processing.
Organisation of Thoughts
As you start to get an understanding of the problem you are solving and the types of analyses and steps that you will be performing. You are going to start grouping cells together, to make it easier to run. Reading the dataset and the initial transformations in one cell, another to calculate some interesting values, and one to generate a visualisation.
This process of organising related lines of code together is the first step towards looking at creating functions. The functions allow you to abstract away the details of what is going on in a particular step, like the data loading, and focus on the important parts.
We might have the following code to read in a file
df = pd.read_csv("data/gapminder_data.csv") df["country"] = df["country"].astype("category") df["continent"] = df["continent"].astype("category")
where the datafile
gapminder_data.csv is from the software carpentry training courses
and can be downloaded here.
In loading this file we are concerned with two inputs,
firstly, the name of the file that we are loading,
and secondly the names of the fields
we want to load as categorical columns.
This results in a dataframe that is ready for us to use.
The function we might write could look something like what we have below,
where I have used type annotations to be clear about the data passed as input.
def read_csv_dataset(filename: str, categorical_columns: List[str] = ): df = pd.read_csv("data/gapminder_data.csv") for column in categorical_columns: df[column] = df[column].astype("category") return df
By transforming our code snippet into a function, we can make our intent clearer within the narrative of the notebook. Now when we load the dataset we can use the line
df = read_csv_dataset( filename="data/gapminder_data.csv", categorical_columns=["country", "continent"] )
which makes it clear to the reader that we want the
continent columns to be categorical
while everything else is less important to our analysis.
This function can then be re-used within a notebook,
serving slightly different purposes each time,
loading a different file,
or allowing us to investigate whether setting columns to categorical
changes the performance of our analysis.
Solidification of Idea
Once we have defined a function within one notebook,
it often becomes something we want to reach for within other notebooks.
Alternatively the definition of the function
could be detracting from the story we are trying to tell within the notebook.
Once we have solidified an idea and defined a function,
we can move that function to a
By creating a file
loading.py in the same directory as the jupyter notebook
read_csv_dataset function as its contents
# loading.py import pandas as pd def read_csv_dataset(filename: str, categorical_columns: List[str] = ): df = pd.read_csv("data/gapminder_data.csv") for column in categorical_columns: df[column] = df[column].astype("category") return df
we can import this function into the notebook as follows
from loading import read_csv_dataset
This works because the file loading is in the same directory as the juptyer notebook,
if it was located elsewhere, we would need to modify where jupyter looks for files to import.
I will commonly locate all my notebooks within a
notebook directory of a project,
while all the python files in the
src directory of a project,
a structure like below.
project/ ├── notebook │ └── gapminder.ipynb └── src └── loading.py
To import the
loading.py file (in this context it is called a module) from the
the simplest method is to tell python to look in that
If we change the import to
import sys sys.path.append("../src") from loading import read_csv_dataset
sys module is one of python's built in modules
sys.path is the list of places that python looks
for things to import.
By manually adding the relative path to the
we are able to import from any files defined within it.
One disadvantage of the move to importing from a file
is that it becomes harder to update the function,
requiring a restart of the kernel to reload the module.
This can be worked around by using the
built into IPython.
%load_ext autoreload %autoreload 2 import sys sys.path.append("../src") from loading import read_csv_dataset
which means that any time we edit the file
python will reload the file and its definitions
replacing the previous version of
read_csv_dataset with the new one.
Does this code actually work
Once we have moved code from a jupyter notebook to a python file we have a range of tools available to help evaluate whether the code we have written works. Within the notebook this would normally be a quick process of running the code, within a python file this can become a little harder, particularly as the size of the file gets larger.
To help with this there are a number of tools making this process easier,
blackprovides automatic code formatting so you can worry about the code rather than how it is laid out.
mypyprovides static type checking, which can be really useful for finding all the places
Nonecan show up in the code. Note that this only applies if you are using type annotations.
flake8provides a range of checks that the code works, primarily checking for common mistakes.
None of these tools however actually run the code,
for that we want to look towards
providing tools to run the code to make sure it works.
In the vast majority of data analysis projects, having the code in a separate file is as far as we need to go. All the tooling is going to work in helping us keep our code working. However, there are some cases where the functionality you are working on is more widely applicable and needed in multiple projects.
For this we can create a package for the code,
with the package becoming a dependency of the project.
This allows for anyone to install the package
and we can use the same tools
that we use to manage the rest of our external dependencies.
For information on packaging the Python Package Authority (PyPA)
has a useful user guide
and conda also has a user guide for getting started.
The Jupyter notebook is the starting place for code, it allows us to quickly iterate and try ideas out to see what works. Additionally notebooks they are a fantastic storytelling tool, however to make the most of them and to keep the story engaging, we sometimes need to look beyond the notebook. How far we go depends heavily on the type of project being undertaken and the tools we want to use.