Blog Posts

What's the time Azure Synapse

2022-02-07T00:00:00+00:00

Azure Synapse is Microsoft's fancy data analytics platform designed to handle all kinds of data analytics tasks. Unfortunately, there are significant deficiencies in how this platform handles datetime data and timezones. Making it incredibly difficult to use Synpase in conjunction with other tools within the Data Science ecosystem.

Dealing with time and timezones is an incredibly difficult problem, not at all helped by the 6 separate timezones currently in use on mainland Australia¹. However, despite all the complexity of dealing with time and the corresponding timezones, there are two main ways of storing and working with time information;

Use a naive datetime that just includes the date and the time which we are going to call Local Time, or
use a timezone aware datetime that includes the date, time, and a timezone offset which we are going to call Instantaneous Time.

It is really important to note that these two types of times, Local Time and Instantaneous Time are different representations, and converting between them is an incredibly complex process —talking to you daylight savings, Samoa, and all the other exceptions. This conversion needs to be done carefully and deliberately, that is, use a library specifically designed for handling the conversion and trust that whoever spent years handling all the edge cases knows far, far better than you.

Both of these approaches have their use cases, Local Time is simpler and suited to collecting data locally, for example most of our daily activities, or monitoring cycling patterns across a bridge. Instantaneous Time is required for data collected remotely where we need the additional context of where the time was taken allowing the comparison of the instant the event occurred. As such see plenty of use in computer systems, and also when travelling by aeroplane. The major difference between Local Time and and Instantaneous Time is we can convert the timezone of Instantaneous Time without changing the instant at which the event occured, allowing for the comparison and ordering of events across timezones.

This ability to change the timezone of Instantaneous Time also gives us the ability to convert all the times to a standard timezone, which makes for an easy comparison of events. The standard timezone to use for this comparison is Co-ordinated Universal Time (UTC), which has a timezone offset of +0. Working with this standard timezone allows for an important simplification of Instantaneous Time. Rather than storing the date, time, and timezone offset, we can store just the date, the time in the UTC timezone, and that we have an Instantaneous Time. This is how many programming languages handle timezones, performing the conversion to a local time when displaying the data. This is also the approach used by the parquet file format, which natively supports a TIMESTAMP data type. The parquet format is designed to hold columnar data, that is data where all the values within a column have the same data type. They will all be integers, strings, ...or TIMESTAMPs. The metadata of a parquet file describes the contents, including the name of each column and the type of data stored within it. When the data type is TIMESTAMP, there is a option to specify whether we are storing Local Time, or Instantaneous Time. This flag that we set has the name isAdjustedtoUTC documenting whether we are using the trick of converting everything to UTC described above.

The Parquet file format is the enabler of the Data Lake, allowing software to read and write data to disk in an efficient and portable way. Azure Synapse is Microsoft's fancy serverless database, specifically designed to work alongside a Data Lake. So it makes sense that there is built in support for working with Parquet files.

Like the Parquet file format, Synapse has different timestamp types for Local Time and Instantaneous Time. The datetime2 type is used for Local Time, while the datetimeoffset type is used for Instantaneous Time.

Unfortunately the handling of datetime data within synapse is significantly lacking. When trying to read a Local Time from a parquet file into the datetime2 type Synapse gives us this really helpful error message

Column 'datetime' of type 'DATETIME2' is not compatible with external data type
'Parquet physical type: INT64', please try with 'BIGINT'. File/External table
name: 'local.parquet'.

Under the covers, the TIMESTAMP within a Parquet file is an INT64 type, counting the number of milliseconds ¹ since 1 January 1970 (1970-01-01 00:00). Even more confusing about this error message is that the documentation for the OPENROWSET function which is how we are opening the file clearly states that TIMESTAMP (MILLIS / MICROS) are converted to the datetime2 type. You might think that there would be a function built into Synapse to allow for the manual conversion of a TIMESTAMP to the internal datetime2 format, however, there is absolutely no mention of it within the documentation. So we can't even perform this conversion ourselves without first implementing the function to perform that conversion.

So if reading in a Local Time doesn't work, how about an Instantaneous Time? When we reading the Instantaneous Time using the same approach, rather than the cryptic error message, the file reads with no problems. So we can just use Instantaneous Time and everything works right???

No.

Remember how I mentioned we want to be really careful about the conversion between Instantaneous Time and Local Time, well here Synapse is completely ignoring our Instantaneous Time and just reading the data into its datetime2 type representing Local Time. The problem here is that other tools² actually pay attention to these details, so the 'quick fix' of flicking the switch to using the Instantaneous Time is going to cause far more problems with other tools using this data. To solve this problem properly, we need changes at both sides of the data pipeline. Firstly, the Local Time needs to be converted to Instantaneous Time, assuming we know the locale in which the times were collected. We will need this locale at the end, so better hope it is documented somewhere (I know that might be asking too much). If we have no idea trying UTC and hoping for the best at least allows us to read the file, even if it causes problems down the track. With the data having a locale, one of the Apache Arrow Parquet writers will automatically convert the timezone to UTC upon writing the file. Now when reading the file within Synapse we need to do two conversions when selecting our timestamp column, the first to convert Synapse's Local Time to an Instantaneous Time in the UTC time zone, followed by a conversion to the local timezone we identified earlier.

SELECT timestamp AT TIME ZONE 'UTC' AT TIME ZONE 'Cen. Australia Standard Time'

So how about doing the conversion manually. Well it turns out this acutally isn't possible with the tools available within Synapse. There is no inbuilt method to convert from this TIMESTAMP to the inbuilt date types. Additionally, if we want to try and do this conversion manually the DATEADD function within Synapse only supports a 32 bit signed integer, of which the largest is 2^16-1 (~2.1 billion). In the most optimistic case supported by the Parquet file, that is, using milliseconds since 1970-01-01 00:00:00, the only dates we can describe are the 25 days surrounding it. Which really doesn't solve the problem here.

What I find really unusual about this whole situation is that this is definitely not a result of using a really old Parquet reader. We can know this because there has been a change in how the TIMESTAMP type is defined within the Parquet file. The previous types had no notion of Local Time and were either annotated as TIMESTAMP_MILLIS or TIMESTAMP_MICROS. For compatibility reasons these old types are still annotated on the files with the filetype specifications noting

Despite there is no exact corresponding ConvertedType for local timestamp semantic, in order to support forward compatibility with those libraries, which annotated their local timestamps with legacy TIMESTAMP_MICROS and TIMESTAMP_MILLIS annotation, Parquet writer implementation must annotate local timestamps with legacy annotations too, as shown below.

If the Parquet reader for Synapse was only looking at these legacy annotations then both Local Time and Instantaneous Time would work since they have the same type defined. It is only in the newer file format definition that there is anything differentiating Local and Instantaneous time. So the new types are supported, but not properly which in my opinion is worse than not supporting them at all. To add even more confusion to the mix, the type provided when trying to read the parquet timestamp into a datetimeoffset type annotates the type in the Parquet file as TIMESTAMP_MILLIS, that is, using the legacy annotation.

For Synapse to be a properly fulfil a role within a Data Science toolkit Microsoft really needs to properly support TIMESTAMPs from the Parquet file format. Currently, Synapse is unsuitable as part of a data pipeline that handles datetime data. The lack of both inbuilt support for TIMESTAMPs and the tools for individuals to support them severly limits the usefullness of Synapse, and makes it difficult to recommend as part of a broader data analytics platform.

During summer these are: AEST, AEDT, ACDT, ACST, AWST, and Eucla. There are also at least 4 more in use off mainland Australia.

Microseconds are also a supported option. And in newer versions of the parquet file format, Nanoseconds are also supported.

For example, the entire python Data Science ecosystem.

Installing Fedora 35 on Raspberry Pi 4

2022-01-08T00:00:00+00:00

I have found the installation of Fedora on a Raspberry Pi to be an incredibly difficult process, with many of the approaches I have tried not working for some reason. This documents the process that ended up working for me, along with many of the approaches that didn't work.

Installation Process

Unlike standard hardware like a server or desktop, there are a number of steps in the installation process. This is because the Raspberry Pi doesn't come with a standard UEFI bios we can to boot from to boot from an installer USB. Normally this would allow us to boot from a USB stick to enable the installation, however, on the Raspberry Pi we first need to install the UEFI firmware.

Standard install

The latest version of Fedora for the aarch64 architecture used by the Raspberry Pi 4 is available on the Fedora Downloads site. There are two downloads for aarch64,

an aarch64 DVD ISO, and
an aarch64 raw image.

We need second of those, the aarch64 raw image. The raw image can be installed on a MicroSD card and should provide a working installation of Fedora. I was unable to get a functional installation using this approach, however this should work.

To get the raw image onto the MicroSD card we can use the arm-image-installer utility, which is available as part of the fedora distribution through dnf. Running

dnf install arm-image-installer

will ensure it is installed locally. Once installed we can use it to configure the installation.

To find the appropriate path of the MicroSD card, we can run the command

sudo fdisk -l

and finding the drive matching the size of the MicroSD card. For me this is /dev/sda since I have an NVMe boot drive on my local computer. Make sure to double check the drive path, since you don't want to overwrite your local boot drive. With the appropriate drive path found, replace <MicroSD Card> with the path.

sudo arm-image-installer --image=Downloads/Fedora-Workstation-35-1.2.armhfp.raw.xz --target=rpi4 --media=<MicroSD Card> --resizefs

where;

the image is the file we downloaded,
the target is the device we want to use with the MicroSD card in this case a Raspberry Pi 4 (rpi4).
The media is where we are writing the data to
and --resizefs tells arm-image-installer to expand the storage to fill the media, making the entire drive usable when booted up.

Once these steps are completed, we can try running the drive on a Raspberry Pi. Insert the drive, connect up the power and monitor, and everything should boot up fine, giving the initial configuration screen for Fedora.

Unfortunately in my case, I was unable to go through the configuration, with my keyboard unresponsive upon reaching the configuration screen. Because I was unable to continue through the installation, I had to take an alternative approach.

Installing UEFI Firmware

With the raw image not working, I had to take an alternate approach for the installation. The alternative download to the RAW image was the DVD ISO, so that was my next approach. However, as mentioned earlier, the Raspberry PI doesn't ship with the UEFI bootloader we need to get the DVD ISO installed. Fortunately however, we can download one. The UEFI firmware that we need is available in the releases of the pftf/RPi4 GitHub repository. The .zip file should be downloaded onto your local machine.

The instructions listed describe creating a FAT32 partition for the firmware however, it just so happens that when we created the RAW image, we have also created all the partitions we need for this step already. Putting the SD card back into the computer that is working, you should see three partitions on the drive. A large one that close to the total capacity of the SD card, which will be the root partition and is where all the Fedora OS is installed. This will contain many folders including those named dev, etc, home, sys, and proc. There will be a drive about 1.1 GB which is where the OS Kernel is installed, this will include folders like dtb, grub2, loader. The final directory is the one we want, which appears for me as 629 MB in size; containing a number of .elf files. The zip file with the UEFI firmware can be extracted to this drive, replacing any of the files that are already there.

With the bootloader installed onto the drive, we now have a UEFI firmware. This can be checked by inserting the MicroSD card into the Raspberry Pi and booting it up. It will now provide options for how you want to boot the Pi.

While you are here, it is a good idea to remove the 3GB memory limit. This is imposed to ensure support for older kernels, however using Fedora 35 is new enough to not cause any issues.

Installing the DVD ISO

To install the DVD image onto our Pi, we first need to download the DVD ISO, which is available at the same Fedora Downloads link as above. However, this time we need the first link for the DVD ISO.

The simplest method to get this ISO onto Fedora Workstation onto a USB is to use the Fedora Media Writer utility, which can be installed using dnf

dnf install fedora-media-writer

Running Fedora Media Writer we need to select the downloaded image.

Why all this Effort?

The key reason for choosing Fedora is because I am familiar with it and the tooling and versions are consistent across the rest of my workspaces. Additionally I have been playing around with Rust and want to compile my code for the aarch64 architecture. This means that the standard Raspberry Pi OS is not appropriate, only being the 32 rather than 64 bit architecture.

A Workflow for Data Science

2021-06-29T00:00:00+00:00

Jupyter notebooks are a phenomenal tool for investigating and interacting with data. The interactive nature of the notebooks allows for quick and effective interrogation of the data and more generally the prototyping of an idea. By combining interaction, the documentation, the code, and the visualisations, the notebooks are also a fantastic tool for communicating ideas. However, notebooks are not the most appropriate for all the code we write, with many there being many detractors to using notebooks and there are definitely valid reasons for disliking the notebook. However, most of the issues can be addressed by recognising that the jupyter notebook is one of many tools that we have available, and knowing when to move beyond the notebook can help us make the most of it.

While I am using python to illustrate these steps and examples, the same ideas apply for any language being used in the notebook just with some slightly different specifics on the how.

Idea Development

The first step within the process is the exploration and development of an idea. This is where we are just getting started looking at a problem and we probably haven't written any code yet. In this phase of development we want to make full use of the interactivity of jupyter. This is where we can easily play around to make sure we are using the right options for reading in our file, trying lots of different types of analysis, while having a record of what we have done so that we can easily regenerate a previous state when we break something.

It is here where the interactive nature of the juptyer notebook really stands out. It is easy to introspect the state at each step in the process, looking at the problem in different ways to properly understand what is going on. We also have the visualisations, which allow us to generate nice figures representing the data, however also includes the pretty printed outputs, like the nicely formatted tables of dataframes, which make working with the data easier.

The way we retain state within the notebook is incredibly appealing, particularly where the first step of the process is time consuming, like loading the data from a URL, performing a complex database query, or doing some intermediate processing.

Organisation of Thoughts

As you start to get an understanding of the problem you are solving and the types of analyses and steps that you will be performing. You are going to start grouping cells together, to make it easier to run. Reading the dataset and the initial transformations in one cell, another to calculate some interesting values, and one to generate a visualisation.

This process of organising related lines of code together is the first step towards looking at creating functions. The functions allow you to abstract away the details of what is going on in a particular step, like the data loading, and focus on the important parts.

We might have the following code to read in a file

df = pd.read_csv("data/gapminder_data.csv")
df["country"] = df["country"].astype("category")
df["continent"] = df["continent"].astype("category")

where the datafile gapminder_data.csv is from the software carpentry training courses and can be downloaded here. In loading this file we are concerned with two inputs, firstly, the name of the file that we are loading, and secondly the names of the fields we want to load as categorical columns. This results in a dataframe that is ready for us to use. The function we might write could look something like what we have below, where I have used type annotations to be clear about the data passed as input.

def read_csv_dataset(filename: str, categorical_columns: List[str] = []):
  df = pd.read_csv("data/gapminder_data.csv")
  for column in categorical_columns:
    df[column] = df[column].astype("category")
  return df

By transforming our code snippet into a function, we can make our intent clearer within the narrative of the notebook. Now when we load the dataset we can use the line

df = read_csv_dataset(
  filename="data/gapminder_data.csv",
  categorical_columns=["country", "continent"]
)

which makes it clear to the reader that we want the country and continent columns to be categorical while everything else is less important to our analysis. This function can then be re-used within a notebook, serving slightly different purposes each time, loading a different file, or allowing us to investigate whether setting columns to categorical changes the performance of our analysis.

Solidification of Idea

Once we have defined a function within one notebook, it often becomes something we want to reach for within other notebooks. Alternatively the definition of the function could be detracting from the story we are trying to tell within the notebook. Once we have solidified an idea and defined a function, we can move that function to a .py file.

By creating a file loading.py in the same directory as the jupyter notebook with the read_csv_dataset function as its contents

# loading.py
import pandas as pd

def read_csv_dataset(filename: str, categorical_columns: List[str] = []):
  df = pd.read_csv("data/gapminder_data.csv")
  for column in categorical_columns:
    df[column] = df[column].astype("category")
  return df

we can import this function into the notebook as follows

from loading import read_csv_dataset

This works because the file loading is in the same directory as the juptyer notebook, if it was located elsewhere, we would need to modify where jupyter looks for files to import. I will commonly locate all my notebooks within a notebook directory of a project, while all the python files in the src directory of a project, a structure like below.

project/
├── notebook
│   └── gapminder.ipynb
└── src
    └── loading.py

To import the loading.py file (in this context it is called a module) from the gapminder.ipynb notebook, the simplest method is to tell python to look in that src folder.

If we change the import to

import sys
sys.path.append("../src")

from loading import read_csv_dataset

where the sys module is one of python's built in modules and sys.path is the list of places that python looks for things to import. By manually adding the relative path to the src directory we are able to import from any files defined within it.

One disadvantage of the move to importing from a file is that it becomes harder to update the function, requiring a restart of the kernel to reload the module. This can be worked around by using the %autoreload magic built into IPython.

%load_ext autoreload
%autoreload 2
import sys
sys.path.append("../src")

from loading import read_csv_dataset

which means that any time we edit the file ../src/loading.py, python will reload the file and its definitions replacing the previous version of read_csv_dataset with the new one.

Does this code actually work

Once we have moved code from a jupyter notebook to a python file we have a range of tools available to help evaluate whether the code we have written works. Within the notebook this would normally be a quick process of running the code, within a python file this can become a little harder, particularly as the size of the file gets larger.

To help with this there are a number of tools making this process easier,

black provides automatic code formatting so you can worry about the code rather than how it is laid out.
mypy provides static type checking, which can be really useful for finding all the places None can show up in the code. Note that this only applies if you are using type annotations.
flake8 provides a range of checks that the code works, primarily checking for common mistakes.

None of these tools however actually run the code, for that we want to look towards pytest providing tools to run the code to make sure it works.

Package Creation

In the vast majority of data analysis projects, having the code in a separate file is as far as we need to go. All the tooling is going to work in helping us keep our code working. However, there are some cases where the functionality you are working on is more widely applicable and needed in multiple projects.

For this we can create a package for the code, with the package becoming a dependency of the project. This allows for anyone to install the package and we can use the same tools pip and/or conda that we use to manage the rest of our external dependencies. For information on packaging the Python Package Authority (PyPA) has a useful user guide and conda also has a user guide for getting started.

Conclusion

The Jupyter notebook is the starting place for code, it allows us to quickly iterate and try ideas out to see what works. Additionally notebooks they are a fantastic storytelling tool, however to make the most of them and to keep the story engaging, we sometimes need to look beyond the notebook. How far we go depends heavily on the type of project being undertaken and the tools we want to use.

Updating Query with Soft Delete for SQLAlchemy 1.4

2021-03-31T00:00:00+00:00

When working on a database, there are times where we want to delete a row, but at the same time we still want to keep around the record that the particular row was referring to. There can be many reasons for this, however the basic idea is that we want the row to generally be hidden. The Soft Delete pattern allows for the 'delete' operation, to hide the row from the external interface, while it remains within the database. Miguel Grinberg has a great writeup of the pattern and implementation in SQLAlchemy on their blog.

This implementation is great, however the SQLAlchemy 1.4 release changed some of the internals meaning Miguel's implementation no longer works, giving the below error

AttributeError: 'QueryWithSoftDelete' object has no attribute '_mapper_zero'

The fix for this issue is to replace the _mapper_zero method with the _entity_from_pre_ent_zero method, giving a with_deleted method as below.

class QueryWithSoftDelete(BaseQuery):
    ...

    def with_deleted(self):
        return self.__class__(
            db.class_mapper(self._entity_from_pre_ent_zero().class_),
            session=db.session(),
            _with_deleted=True
        )

Why does this work?

With the fix out of the way, we can have a look at what the with_deleted method is doing. The entire code snippet for the QueryWithSoftDelete is reproduced below for reference.

class QueryWithSoftDelete(BaseQuery):
    def __new__(cls, *args, **kwargs):
        obj = super(QueryWithSoftDelete, cls).__new__(cls)
        with_deleted = kwargs.pop('_with_deleted', False)
        if len(args) > 0:
            super(QueryWithSoftDelete, obj).__init__(*args, **kwargs)
            return obj.filter_by(deleted=False) if not with_deleted else obj
        return obj

    def __init__(self, *args, **kwargs):
        pass

    def with_deleted(self):
        return self.__class__(
            db.class_mapper(self._entity_from_pre_ent_zero().class_),
            session=db.session(),
            _with_deleted=True
        )

The with_deleted method provides a way to transform the Query from one that filters out deleted values to one that does. Now there isn't a straightforward method for removing filters from a query, so the approach taken here is to create a new Query, this time setting _with_deleted=True. The problem now becomes, which model are we trying to query? When we call the with_deleted method, we are doing so from an instance of the QueryWithSoftDelete class, so we are unable to extract the model being used through methods like __class__, it gives us the QueryWithSoftDelete class. So instead we can look to SQLAlchemy to get this information. Within SQLAlchemy, every query from the database needs a FROM clause, the table that we are going to be using to initially select our data. This table we are selecting, is the same one that we want to query though this time instead of filtering the deleted items, we will be including them.

Previously the _mapper_zero method provided that table in the FROM clause. However the restructuring of the Query to consolidate the core and ORM components meant the method disappeared, completely fine given it is a 'hidden' method. We need the function that will give the equivalent behaviour, which happens to be the much more verbose, _entitiy_from_pre_ent_zero.

Handling Exceptions in Numerical Python

2020-10-19T00:00:00+00:00

The numerical computing capabilities of the Python ecosystem are incredibly powerful, with the Numpy and Scipy, and Pandas packages providing high performance, well tested tools. Additionally, tools like Dask and Rapids build upon these foundational packages supporting larger datasets. All these tools are excellent when the data is well formed and contains the expected values. It is when we are handling the error cases that we can run into issues.

During my PhD, I used scipy for the analysis of molecular dynamics simulations. These simulations are effectively putting things in a box and shaking it to see what happens. The novel component of my research was shaking the box for a really, really, really, long time. So long that the software I was using ran out of numbers to count. ¹ Having run the simulation, I then needed to go though the terabytes of data generated to calculate quantities of interest.

The huge benefit of using python and the suite of numerical tools is the ability to quickly prototype a solution, working through the ideas on a small scale that can then be applied to the larger dataset. One of the calculations I was performing was to fit a straight line to a region of calculated values. ² We could come up with the following solution for fitting line of best fit to a region of a curve, returning both the best parameters and the standard deviation.

import scipy

def linear(x, m, b):
    """Define the linear relationship y=mx+b."""
    return m * x + b

def fit_linear_region(x, y):
    """Fit a linear curve where the points lie within a region of 2D space."""
    # Select a region of points
    region = np.logical_and(2 < y, y < 50)
    # Perform the curve fitting only using the linear region
    popt, pcov = scipy.optimize.curve_fit(linear, x[region], y[region])
    # Calculate the standard deviation of the fit parameters
    perr = 2 * np.sqrt(np.diag(pcov))
    # Returns the optimal curve and associated error
    return popt, perr

When performing tests on small numbers of values, we typically choose those behaving as we expect. Over the course of many long simulations nearly all possible values are expressed, meaning this initial testing only covers a small selection of all the possible values. On many occasions, these unexpected values resulted in the code raising an exception, unexpectedly bringing an hours long calculation to a halt. For interactive analysis, these exceptions are not too problematic, you can easily fix the problem and re-run the analysis. However when leaving the calculation running overnight, coming back to halted execution partway through the analysis is incredibly frustrating. In these long running analyses, in moving from the prototyping to the production step of the process, we have to consider all the places where our code might fail allowing us to walk away knowing a result will be waiting for us the next morning.

Exploring the Solution Space

The problem of handling errors within code is faced by programmers in a wide range of disciplines. Accordingly, there are a range of approaches to ensure our code doesn't unexpectedly fail.

Catch-all

One of the possible approaches we can take is to capture all exceptions with a naked except block, that is

try:
    popt, pcov = scipy.optimize.curve_fit(linear, x[region], y[region])
except Exception:
    return np.nan, np.nan

While this approach handles all possible issues that could occur, it is also heavy handed in its application. The general goal is to prevent unexpected errors in a particular configuration halting the execution of our program. However, in capturing all possible errors, we also capture errors that can be problematic. For example a KeyboardInterrupt, meaning that rather than <Ctrl>-c stopping the execution of the program, it instead describes a curve that doesn't fit. This could also be problematic with a FileNotFoundError, where instead of getting an error at the start of the program we have to work out why our results file doesn't exist when trying to look through them.

While the idea of the catch-all is useful, it is primarily delaying problems identified when the program runs to subsequent steps in analysis, which become more difficult to solve, amplified by the lack of error messages and tracebacks that python provides.

Hypothesis

The main issue with the catch-all case is that we are overly broad in the types of exceptions that we catch. Rather than catching all exceptions, what if we could determine the exceptions expected from a function call, only catching that smaller subset. This is similar to the approach that you would take in an interactive type analysis where you keep adding the exceptions that arise to the list of those caught and handled. An alternative to finding these exceptions manually is through [hypothesis], a package for enhancing test suites by using a directed random search, probing for values that cause problems. The directed part of hypothesis ensures values likely to cause issues, including small values, large values, zero, NaN, and Infinities are handled appropriately. In ensuring we handle all these different values, Hypothesis is a tool that ensures that we codify the assumptions we make about the data we put into our algorithms.

Working with hypothesis requires a fair amount of setup, building upon a test framework like pytest. Additionally, as a property testing framework, rather than testing specific values we instead need to test properties, requires a rethinking of how we test functions. In the example covered in the this article, the property we are testing is ensuring our function is doesn't unexpectedly raise an exception. The testing of this property can be achieved by creating the test as shown in the code snippet below.

from hypothesis import given
from hypothesis.extra.numpy import arrays

@given(x=arrays(dtype=float, shape=10), y=arrays(dtype=float, shape=10))
def test_example(x, y):
    fit_linear_region(x, y)

Here we use hypothesis to generate the two inputs to our function, x and y. Each of these is created from a numpy array of 10 elements (shape=10) containing floating point values (dtype=float). The @given decorator allows hypothesis to generate inputs based on the values that we have provided, running each to check that it runs. In this code snippet we are making clear some assumptions that we have made about our data. Firstly that we are expecting floating point values, being the dtype that we are generating. The second is that the shape of both x and y is equal. One of the advantages of hypothesis is bringing to light these occasionally hidden assumptions about the inputs to our functions.

By using hypothesis to generate inputs to our function, we find some additional errors that can occur. When there are no points that lie within our specified range, we get a ValueError, while when there is only a single value in that range we get a TypeError. Additionally, when scipy is unable to find and appropriate line of best fit, an OptimizeWarning is raised. This results in the following fit_linear_region function to handle these three exception cases.

def fit_linear_region(x, y):
    """Fit a linear curve where the points lie within a region of 2D space."""
    # Select a region of points
    region = np.logical_and(2 < y, y < 50)
    # Perform the curve fitting only using the linear region
    try:
        popt, pcov = scipy.optimize.curve_fit(linear, x[region], y[region])
    except (ValueError, TypeError, scipy.optimize.OptimizeWarning):
        return np.nan, np.nan
    # Calculate the standard deviation of the fit parameters
    perr = 2 * np.sqrt(np.diag(pcov))
    # Returns the optimal curve and associated error
    return popt, perr

While using hypothesis allows us to easily find some of the failure modes of functions, it is another tool to learn how to use effectively. It would be nicer if there was a simpler solution to finding the errors raised by a function that made use of existing tools.

Documentation

The documentation for a function is something regularly referred to, whether that be through the online documentation or through the help messages of a function. This makes the function docstrings a fantastic place to put information about the exceptions a function could raise. The scipy docs include this information for some of the functions, including curve fit. Within the Raises section, it notes there are three different exceptions the function can raise, a ValueError, RuntimeError, or OptimizeWarning. This list of exceptions demonstrates a limitation of using hypothesis, it won't always find all the possible exceptions a function can raise, here missing the RuntimeError. However, on the flip side, the TypeError we found with hypothesis is currently missing from the documentation. This is a limitation of the documentation approach, the exception comes from a function called by curve_fit, propagating beyond the initial call site, making it difficult to notice and to keep the documentation up to date. The documentation approach is also prone to degrading over time, as the code gets updated and changed, the documentation has to be changed and updated alongside it, and for exceptions this can be in multiple unrelated locations. One of the solutions for this is representing the possible exceptions in code, rather than through the documentation allowing tooling to provide assistance, much like mypy and pyright do for type annotations.

Rust

One of the highly regarded parts of the Rust programming language, is the comprehensive type system that explicitly handles errors. In a similar way to exceptions within Python, the Result type within Rust is a way of stating that something has gone wrong. The big difference between an exception in python and the Result type in Rust, is that in Rust you must explicitly handle the error. In the simplest case this error handling is equivalent to a python exception, stopping the execution of the program and exiting. The key enhancement with the Rust approach is that there is a record of every place within the code that an error could halt the execution. When prototyping on the sample datasets within Rust, we can use the exception model of error handling, halting the execution of the program using .unwrap(). However, when moving to a more complex analysis pipeline, or when developing a library that people rely on, unexpected errors are no longer a good option. Here, all instances of .unwrap() can be converted into more appropriate methods to handle those errors. These propagation of the Result type occurs similarly to the handling of Optional values within type checked python code. Whenever we call a function that returns a Result, we have to check whether we have the Ok value, in which case we can continue on, or the Err value, where it has to be either be handled or passed on to a calling function through the Result type.

In principle, supporting a Result type within python like exists within Rust is possible, however, what makes the type so useful within Rust is that it must be handled, a guarantee that can't and won't be enforced within python.

Conclusion

One of the key principles of python is the ease of getting started. Advanced parts of the language, can be ignored until they are required. Ignoring advanced parts of the language is not just helpful for the learning process, but also for slowly building up complexity in the projects that we start. When working with long running processes, exception handling becomes an important part of the development process, however, I don't know of a clear way to find all the possible ways an exception could be raised from a function we might use. This makes the development process a continually iterative one, run the code until you find an exception, handle the exception and start again, a process that is not ideal, particularly for overnight calculations.

It was only using 32 bit unsigned integers, however that was decided to be more than anyone would need.

If you have to know, I was calculating the Diffusion constant, a measure of how fast particles move over long timescales. This is a linear function of the Mean Squared Displacement vs time.

Failing Hard: How I 'lost' two years of data

2018-11-22T00:00:00+00:00

Failure is one of those topics that is discussed far less than it should be ---It is hard to tell other people about mistakes you have made--- yet failure is usually a far better teacher than success. This is why I want to share my story of how I spent the first two years of my PhD collecting useless data because of a bug in my code.

The Problem

I perform computer simulations of toy molecular systems, using something simple and understandable to gain insights into more complicated chemical systems. This is just like in High School and Undergraduate Physics where you perform calculations for objects in a frictionless vacuum. Apart from the simplicity of a toy system, a major benefit of using one is that you can modify individual parameters of the system to test a hypothesis. I had the hypothesis that rotational motion plays an important role in the large scale translational motion---known as diffusion, so I was using a toy system which allowed me to control the effective magnitudes of these two types of motion. The small scale translational motion is related to the mass; heavy objects are more difficult to move in a straight line. While the rotational motion is related to the moment-of-inertia, a measure of how difficult something is to rotate. Though varying these two properties, the mass and the moment-of-inertia, I can establish how the translational and rotational motions contribute to the rate of diffusion. Setting the moment-of-inertia really large effectively stops the rotational motion, so what happens to the diffusion? Or the opposite, making the moment-of-inertia really small what is the resulting effect on the diffusion?

Before I start changing the parameters of the toy model, I need a reference (or control) model related to existing results. Science is always performed with reference to our current understanding, so the control model provides a point of comparison to existing results, including other toy models and real chemical systems. The molecule I am studying is comprised of three circles, one large and two small, arranged like a Mickey Mouse head.

To make calculations with these toy systems easier, I use a system of units in which most quantities have a unitless value of 1. Each particle has a point mass at the center of the circle of 1, the large circle has a radius of 1, a reasonable temperature is 1 ---It is really nice that most of the multiplication and division disappears. The smaller particles in the molecule have a radius of $0.63$ and are located on the circumference of the large particle at an angle of $120^\circ$.

I am using the above molecule in Molecular Dynamics (MD) simulations, which use Newton's equations of motion to update the positions of each particle. The translational acceleration $a$ of a molecule is related to the force $F$ acting on it divided by the mass $m$ as given by the equation $a=F/m$. While the angular acceleration $\alpha$ is related to the torque $\tau$ divided by the angular momentum $I$ as given by $\alpha = \tau/I$. Molecular Dynamics simulations have two different methods of dealing with the molecules I am using. The first is to solve these equations for each of a molecule's component particles individually, followed by a second step which restores the molecular shape. The second approach is to calculate the force on each component particle, then adding them together to solve the above equations for the molecule as a whole.

Molecular Dynamics simulations are a standard computational tool for understanding a variety of problems from crystal growth to protein folding. As a widely used tool, there are many different software packages available for performing these simulations, with each having their own strengths and weaknesses. Using these software packages involves expressing the simulation you want to perform in the appropriate manner for the software. At the start of my PhD I decided that Hoomd developed by the Glotzer group at the University of Michigan was the most suitable for the types of simulations I was going to be performing. Soon after starting to understand the software, there was a new major release, which included a significant overhaul of the software. This overhaul included changing how it handled the molecular calculations from the first method to the second method, something which I didn't realise at the time. In getting my simulations to work with the new version of Hoomd, I manually entered a moment of inertia, calculated from each particle having a point mass of 1, however the total mass of the molecule was taken as the mass of an individual particle being 1 instead of 3. By not actually checking the simulations were behaving as I intended I spent the next two years of my PhD characterising the behaviour of the wrong control experiment.

Finding the Bug

I found the bug as part of trying to add additional functionality. I was trying to randomly initialise velocities of particles in a configuration such that they matched the desired temperature.

The reason I found the bug was a result few the Bug result of setting thermodynamic quantities - temperature is a distribution of velocities - energy is evenly distributed between translational and rotational motion

Lessons Learnt

The lessons I have learnt from this experience can be grouped into two groups, the identification of bugs, and the management of computational projects.

Identifying Bugs

Even before the identification of this bug I had developed a test suite for my code, ensuring I was configuring the simulations in Hoomd with the appropriate values. What I wasn't checking, was that these values produced the simulations I thought I was running. Typically the method of doing this is to take some previous results and replicating them, although since no-one else has run simulations on the particular system I am using it is a little difficult to do this.

I had been told by my supervisor on many occasions to confirm that the simulations were physically accurate, usually with reference to conservation of energy. This was really easy to ignore because that out of scope for the code I was writing, however, the idea of checking the simulation is consistent with the laws of physics was definitely worth considering earlier.

Testing is not about ensuring exact correctness, it is also ensuring the absence of obvious errors.

A simple check of physicality is though the equi partition theorem, which describes that each of the degrees of freedom, whether translational or rotational, have the same energy. The relationships below link the mass $m$ and moment-of-inertia $I$ to the temperature $T$ providing a route to testing the simulations, where the angled brackets $\langle \rangle$ denote an averaged quantity.

\[ \begin{aligned} \langle \frac{1}{2} m v^2 \rangle &= k_B T \\ \langle \frac{1}{2} I \omega^2 \rangle &= \frac{1}{2}k_B T \end{aligned} \]

What might not be so obvious here is that the temperature $T$ in the above equations is not a static quantity, instead fluctuating about the desired temperature, a construct of the type of simulations I am performing. This makes the task of testing the temperature is correct significantly more difficult, since it is not a set value, lying in a distribution. While creating automated testing for whether a value that varies can be difficult, re-framing the test as not-wrong is just as valid. With this bug I created I was off by a factor of 3, many times larger than any reasonable variation. I didn't need to test that the temperature was exactly right, I just needed to ensure it is not obviously wrong.

Rebuilding the Dataset

While the main storyline of this article has been losing the years of my PhD research, a major part of the learning experience has been the rebuilding of all the data I had collected.

Reproducibility in science is not just for others to reproduce your work it is also so you can reproduce it when you stuff up.

I have been constantly working to improve my workflow, as I learn more about what works and what doesn't ending up with a structure based heavily on the cookiecutter-cms and cookiecutter-data-science project templates. All the new work was structured into these project templates which are well organised, with an emphasis on being able to replicate analyses. What I really noticed in rebuilding my dataset was that replicating these newer elements was simple and straightforward, while the older projects just working out what I had done was a challenge, let alone what needed to be recreated.

Further to having a well organised data analysis, having all the computational experiments, including the permutations of every variable and the scripts required to run the simulation was invaluable. The experiments where this was up to date were the simplest to deal with, I just needed to run them again.

Conclusion

Although we often don't like to admit it, we all make mistakes. Mistakes, failures, and bugs are all part of doing research. While they may be disruptive and inconvenient, they are only truly damaging when they keep re-occurring. Making a mistake once is part of the process, repeatedly making the same mistake is a problem. Best practices like automated testing are ways of making the mistakes that do occur obvious and simple to fix, as well as providing a way of preventing them from reoccurring. I am now fairly confident that should this particular bug reappear I will notice, although that doesn't mean there isn't another hiding away for me to find.

A Complete Guide to Docker on Fedora

2018-09-02T00:00:00+00:00

While there are numerous guides to installing Docker on Fedora, none of the guides leave the installation in a state that I would consider usable. This is intended to be a single complete guide for the setup and configuration of Docker, highlighting the differences that are required to get Docker running on Fedora. I will be demonstrating using Fedora 28, however, this should be the same for previous or future releases.

Installation

This is the part that all the guides include, including the fedora documentation. Docker is in the Fedora repositories enabling installation using the dnf package manager

sudo dnf install docker

Once installed, the Docker service can be started by running

sudo systemctl start docker

and should you want to start docker every time you boot your machine you can run

sudo systemctl enable docker

Note that the above command doesn't start the Docker service immediately, so you will have to run both the start and enable commands to have the Docker service running now and on following reboots.

At this point you might want to try running a Docker container

$ docker run hello-world
/usr/bin/docker-current: Got permission denied while trying to connect
to the Docker daemon socket at unix:///var/run/docker.sock ...

only you get a permission denied error. It is possible to run Docker as root, however it is probably not the best idea since it is kind of simple to make a mistake.

If instead you received the message

$ docker run hello-world
/usr/bin/docker-current: Cannot connect to the Docker daemon at
unix:///var/run/docker.sock. Is the docker daemon running?

this means you haven't started Docker and need to run

sudo systemctl start docker

Setting Permissions

This section has the potential to break things which are difficult to fix. Please be really careful, unlike me.

This follows the optional post installation steps in the Docker documentation.

For users to have permission to access the Docker socket, they either need to be root, or they can be a member of the docker group.

This group probably doesn't exist on your system yet, though you can check by running

grep docker /etc/group

If there is no output the group does not yet exist and can be created with the groupadd command

sudo groupadd docker

The group should now appear in the /etc/group file

$ grep docker /etc/group
docker:x:1001:

The final step is adding yourself and/or any other users to the Docker group. This is done with the command

Running the below command without append will remove you from the wheel group meaning you will no longer be able to run commands with sudo. If you are the only user with root access you will have to repair your install from a live image.

sudo usermod --append --groups docker $USER

You have to go through the login process to update your group membership, with the safest method being to open an ssh connection to localhost. This means that if you accidentally remove yourself from the wheel group,¹ you just have to disconnect the session to regain sudo permissions and fix things.

To check everything is as expected, the groups command will list the groups you are a part of.

$ groups
malcolm wheel docker

You should have a list of groups similar to those output above. Now you are a member of the docker group you can test Docker is working with the test image

$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

...

This indicates that Docker is working properly.

Mounting Local Directories

Now you have Docker working, you probably want to do something useful with it like access the local filesystem for processing.

$ docker run --interactive --tty --volume $(pwd):/srv ubuntu
root@aa7c5f9dcef9:/# _

The above command creates an interactive terminal (tty) running in an Ubuntu container, with the prompt for the container now showing. Additionally we have mounted the current directory to the container at the /srv folder of the container. We can try and access the contents of the current directory from within the container

root@aa7c5f9dcef9:/# ls
ls: can't open '/srv': Permission denied

On an Ubuntu install this would work, however Fedora uses SELinux for security, which requires the appropriate labelling of file objects for the processes using them. By default Docker doesn't perform this labelling, however we can tell it to with the :z or :Z suffixes for the volume. The lowercase :z allows multiple containers to access the volume and the uppercase :Z allows a single container to access the volume. The command becomes

$ docker run -it -v $(pwd):/srv:Z ubuntu
root@aa7c5f9dcef9:/# ls /srv
docker_on_fedora.md

Here I have used the more common shortened command line options, -it for the interactive terminal, and -v for the volume.

For a program that at first glance appears to simple to install, Docker is rather difficult to get set up properly on Fedora. Hopefully this

I have done this...twice

Speed Through Specificity

2018-08-07T00:00:00+00:00

A common criticism about the Python programming language is that it is slow, often with reference to a benchmark comparing a range of tasks. This criticism is widely addressed with articles by Jake van der Plass and Anthony Shaw being two excellent examples. While I don't disagree with any of the points raised in these articles, I think they miss an important aspect of performance---specificity. Python is a general purpose language, used for nearly everything from embedded devices with uPython to distributed processing of petabytes of data.

Speed comes through specificity for a task. Even when working with C or C++, which are generally regarded as the reference standard for performance, there is still an argument to be made for increased performance through hand optimised assembly. Writing assembly, which is the stream of instructions the CPU interprets to do it's work, can result in faster code than a compiler if you know what you doing. Only nearly no-one actually writes assembly because we want our applications to work on different processor architectures and to take advantage of the features of newer CPUs like out-of-order execution.¹ Instead of writing the fastest possible code for a particular processor a more common approach is to provide hints to the compiler to help it optimise performance, trading some performance for generality and simplicity.

While it is uncommon to try and eke out every last bit of performance from a CPU, offloading the work to more specialised hardware is commonplace. A CPU is a general purpose tool being adaptable to many different types of operations which make it ideal for powering our computers, though to really understand fast we need specialised hardware like GPUs, FPGAs, or ASICs. Graphics Processing Units (GPUs)² are designed for performing the same operation on large quantities of data at the same time; whether that is working out the colour of each pixel on a display, or the values at each point of a large matrix. GPUs are designed at a hardware level to perform these types of tasks, eschewing much of the capability of a CPU. Having hardware specific to a task is a key factor for performance and Field Programmable Gate Arrays (FPGAs) are one method of achieving this. Instead of providing a stream of instructions to be interpreted like a CPU or GPU, an FPGA is programmed by arranging the circuits to perform the desired processing, providing phenomenal processing capability. FPGAs are used in places like signal processing and the Mars Rovers as they allow for reprogramming for task switching or hardware updates. In some applications the programmability of a FPGA is unnecessary, so Application-Specific Integrated Circuits (ASICs) are used instead. An ASIC is a piece of silicon for processing a single specific task, with a common use case being decoding video streams enabling you to watch hours of video in a single charge on your phone. The progression of hardware from CPU to GPU to FPGA to ASIC represents an increasing specificity to a task for improved performance at that task.

The specificity of a task doesn't just refer to the low level details of processor architecture, there are also the levels most developers are more likely to encounter of type and application specificity. Python is a dynamically typed programing language, allowing variables to change type and change functionality during execution. A byproduct of this dynamic typing is types and operator functions need to be evaluated for each operation. Should we want to add the values of two lists together like in the example below, each time the code reaches the line result = i + j it has to evaluate the types of both i and j to then work out how to perform the addition operation. Most noticeable with large numbers of numerical values, the type evaluation takes far longer than the addition operation.

list1 = [1, 2, 3]
list2 = [4, 5, 6]
list_result = []
for i, j in zip(list1, list2):
    result = i + j
    list_result.append(result)

A key concept in optimising numerical python code is limiting the number of type evaluations, with the canonical method being to use NumPy arrays. Instead of having a list which is able to hold many different objects, NumPy arrays only contain a single type meaning the type only needs to be evaluated once for the entire array. Other optimisation techniques for numerical python, namely Cython and Numba, operate in a very similar way; reducing the calculation to a limited range of types evaluated once. With these tools for numerical computation it is possible to get performance on par or even exceeding that of C code.

Even with the most specific type information the approach to solving a problem is a huge factor for performance. Applications designed to solve a very specific problem are able to make significant optimisations though handling only the cases the application will come across; an algorithm for acyclic graphs doesn't have to handle cyclic graphs. Additionally an application for user interaction can optimise for responsiveness, or a data processing application can optimise for data throughput. The general purpose tool has to optimise for everything, so it is optimised for nothing. Python is a general purpose programming language; not optimised for numerical computing, for the web, for command line scripts, or any other use case. However, the adaptability of python has allowed the development of application specific tools, like NumPy, containing a small range of very specific functionality. Python also allows for the simple integration of problem specific hardware, with tools like PyCUDA, scikit-cuda, MyHDL, nifpga-python, and many others.

Where speed is important, use the specific tool for the job. In most cases this isn't the python standard library, instead it is usually just a pip install away.³

I guess you could also consider this a bug...

I would consider Tensor Processing Units (TPUs) in the same category as GPUs, with the main difference being the targeted precision of mathematical operations.

Or conda install for the tools requiring C/C++ libraries

Compiling LaTeX on Travis-CI

2018-07-16T00:00:00+00:00

One of the best parts of the current software development environment is the proliferation of Continuous Integration (CI) services like Travis-CI. These CI services plug into GitHub or other code repositories to automatically run when new code is pushed to a repository. Typically CI is used for running automated testing every time new code is added so you can be reasonably confident a change hasn't broken any functionality. The premise of CI is the automation of tedious tasks like running tests.

When writing a LaTeX document, I find compilation the most tedious task. Particularly for large documents, where it can take a long time. This means I run the compilation irregularly, invariably resulting in a whole collection of errors that have accumulated and I now have to fix. Additionally the compilation of LaTeX documents is often highly machine dependent, only working properly with a specific configuration some reason. My final issue with LaTeX documents is ensuring all the files required for compilation are included in the git repository, not just hiding in a directory somewhere on your local filesystem.

As a way of preventing these issues in the write-up of my PhD thesis, I have developed a configuration for building LaTeX documents on Travis-CI. While I have come across other methods for compiling LaTeX documents on Travis-CI, they all compromise on the workflow I would like. I want a system that is;

fast, with builds completing in a couple of minutes
adaptable, not having to manually specify every package I use
extensible, I can easily use pandoc to convert files to LaTeX before compilation
uses biber, the best practice for compiling the references

Choosing a LaTeX Distribution

There are a number of different methods to get a distribution of LaTeX installed. The main method for Linux is the TeXLive distribution which is typically installed via the package manager. The base TeXLive distribution is pre-installed on the Travis-CI image for A. However, this is only the core extensions and so this approach is either fast in that the image just boots up, or adaptable by downloading texlive-full which takes a long time.

The key issue with the TeXLive distribution is the lack of automatically downloading required packages. A LaTeX distribution with this feature is MiKTeX, having an installer that is only 200 MB compared to the ~3 GB of the complete TeXLive. MiKTeX also provides a docker container which is a great method for having exactly the same environment compiling locally and through a CI service. Unfortunately MiKTeX doesn't install the biber binary on Linux (or macOS).¹ While it is possible to create a new container which includes the biber binary, extending and maintaining a Docker container is non-trivial making this approach less favourable. That said, for other Docker-centric CI services this could be an excellent approach.

Another newer and less well known LaTeX distribution is Tectonic. Although still considered beta software, it works for most scenarios and has a lot of features that make it suitable for CI. I would recommend installation with conda although there are a range of installation methods for both Linux and macOS (currently no Windows support). Like MiKTeX, Tectonic automatically downloads the packages required to compile a document, making the same configuration adaptable to many different documents. Having conda as an installation method is also particularly useful, allowing simple installation of many other tools (like pandoc) which might be required to compile a more complex document. The only requirement not satisfied by Tectonic is the automatic installation of biber. Although not supported natively, it is possible to use biber with tectonic as long as the appropriate binary for biber 2.5 is installed, either from Sourceforge or using conda

conda install -c malramsay biber==2.5

Compiling Documents with Tectonic

Typically, compiling documents with tectonic requires a single command

tectonic document.tex

creating the file document.pdf and automatically removing all intermediate files normally associated with compiling LaTeX documents. To use biber instead of biblatex for the bibliography the process is not quite so simple. In an attempt to make it easier I have created the Makefile below which can be used to create a document.

# Makefile

# directory to put build files
build_dir := output

.PHONY: all

all: document.pdf

%.pdf: %.tex | $(build_dir)
	tectonic -o $(build_dir) --keep-intermediates -r0 $<
	# Run biber if we find a .bcf file in the output
	if [ -f $(build_dir)/$(notdir $(<:.tex=.bcf)) ]; then \
		biber --input-directory $(build_dir) $(notdir $(<:.tex=)); \
	fi
	tectonic -o $(build_dir) --keep-intermediates $<
	cp $(build_dir)/$(notdir $@) .

$(build_dir):
	mkdir -p $@

This build process runs tectonic once with the --keep-intermediates option to generate the intermediate files. I then check for the presence of a .bcf file, which is the file biber uses to to it's thing. Tectonic is then run afterwards, which runs the compilation step as many times as it needs to finalise the output. The final step is copying the output PDF from the build directory to the local working directory.

If you want to continue using your standard LaTeX build tool locally (in my case this is latexmk), you can check whether the TRAVIS environment variable is defined enabling separate build processes as in the example compilation step below.

%.pdf: %.tex | $(build_dir)
ifdef TRAVIS
	tectonic -o $(build_dir) --keep-intermediates -r0 $<
	# Run biber if we find a .bcf file in the output
	if [ -f $(build_dir)/$(notdir $(<:.tex=.bcf)) ]; then \
		biber --input-directory $(build_dir) $(notdir $(<:.tex=)); \
	fi
	tectonic -o $(build_dir) --keep-intermediates $<
else
	latexmk -outdir=$(build_dir) -pdf -xelatex $<
endif
	cp $(build_dir)/$(notdir $@) .

The TRAVIS environment variable is defined on all Travis instances and can be set locally for testing using the command

TRAVIS=true make

This sets the variable TRAVIS just for the single command. Note that you will want to run a make clean between running with the different build systems as there will be incompatibility between versions.

Configuring Travis CI

With a LaTeX distribution that is suitable for CI in Tectonic, how do I actually use Travis CI?

This is a complicated process, which took me 81 attempts to get right. I hope this guide helps you in making significantly fewer than me. The outline of the steps is below. The first three are configuring and linking your accounts on GitHub and Travis. The rest of the steps are explained in more detail in the rest of this document.

Create a public repository on GitHub. Travis CI only works with GitHub and while it does work with private repositories they requires a paid account with Travis.
Using your GitHub account, sign in to GitHub and add the Travis CI app to the repository you want to activate. You'll need Admin permissions for that repository.
Once signed in to Travis CI, go to your profile page and enable the repository you want to build.
Create a .travis.yml file in the repository which tells Travis CI what to do. What you need to put in the file is addressed below.
Commit the .travis.yml file to the repository and push to GitHub. Travis will see the commit and start the build process.
Any further commits to the repository, whether to the master branch, other branches, tags, or pull-requests will trigger a build on Travis.

Creating a .travis.ml file

The .travis.yml file is comprised of a number of sections which I have described individually below. The complete file is available for download. To get Travis to start building your repository, commit the .travis.yml file to the repository and push the commit to GitHub.

The first part of the .travis.yml file is specifying the language, which is choosing which of the base containers to use for the compilation.² Since we are using conda for installing any dependencies we can use the minimal image.

language: minimal

The next part is the cache, where I added the $HOME/.cache/Tectonic directory. This is where tectonic stores the downloaded tex packages. Rather than downloading all the packages required on every build, the state of this entire directory will be downloaded at the start of the build and updated when changed at the end of the build. This significantly speeds up the build process.

cache:
  directories:
    - $HOME/.cache/Tectonic

An additional directory that can be cached is $HOME/miniconda so the conda packages are also pre-installed on every build. This is less of a speed-up, although it can be helpful.

The next step is specifying the steps to occur before_install. Travis has a number of steps in the lifecycle of a build providing ways of breaking the build into logical steps. Each item in the list below is a bash command, which is run to update the environment of the test container. This downloads and installs both biber and conda, with conda then being used to install tectonic. Any other dependencies that are required can also be added to this step, say if you are converting Markdown to LaTeX with pandoc you could add conda install pandoc.

before_install:
  # Download and install conda
  - wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O $HOME/miniconda.sh
  - bash $HOME/miniconda.sh -b -u -p $HOME/miniconda
  - export PATH="$HOME/miniconda/bin:$PATH"
  - hash -r

    # Install tectonic
  - conda install -y -c conda-forge tectonic==0.1.8
  - conda install -y -c malramsay biber==2.5

The final step is the script, the code that is used to determine success or failure. Like the before_install section, this is a list of commands which are executed one after another. Unlike the before_install section all these commands are run, even when a command fails.

script:
  - make

The script section is also where you can add any other checks, like ensuring you haven't left in any TODOs, or spelling mistakes. You can run any code you like and if the exit code is zero it is deemed successful, while a non-zero exit code is a build failure. Note that a failing build will not go on to produce a release.

Deploying to GitHub Releases

Having configured Travis to compile our document on every commit, it would be nice to actually do something with the resulting document. Every repository on GitHub has releases, which can be accessed by clicking on the releases link which is outlined in red on the image below.

GitHub automatically creates a release for every tagged commit in the repository, creating a downloadable .zip and .tar.gz of the state of the repository at that commit. It is also possible to edit each of the releases, adding release notes or additional files like installers for a variety of platforms. In this case we are going to use the GitHub releases to store the compiled document for each tagged release providing a historical view of the document which is linked to the code generating it.

In writing of my PhD thesis it makes sense to tag releases using Semantic Versioning, or at least a version of it. Other documents are much less linear, it might make sense to tag a talk with the location it will be given, or the name of the conference. The requirements are basically use numbers, letters and any of ._-+/ --- see the git-check-ref-format documentation for more specific details).³

You can create a tag my_tag for a release by running the command

git tag -a my_tag

which will open an editor to write a message. A typical tag message is the repository name followed by the tag, although that isn't the only approach. Like commit messages, you can specify the message using the -m option.

By default, git doesn't push tags to a remote, requiring the --tags option

git push --tags

Before pushing your newly created tag, you are going to want to configure Travis to upload releases to GitHub. The best method for this is to use the Travis command line client, which can be installed by following these instructions. Once installed you can run the command

travis setup releases

which will prompt for your GitHub credentials and other information about the release. These are used to generate a personal access token for GitHub which Travis uses to authenticate when uploading the release. The travis client will encrypt the token, and update your .travis.yml with a deploy section which looks something like this;

deploy:
  provider: releases
  api_key:
    secure: # your encrypted token will be here
  file: thesis.pdf
  skip_cleanup: true

Since we only want to deploy on tagged commits, we can use the conditional deployment options to conditionally deploy. This gives the following deploy section.

deploy:
  provider: releases
  api_key:
    secure: # your encrypted token will be here
  file: thesis.pdf
  skip_cleanup: true
  on:
    tags: true

Conclusion

I have made the entire .travis.yml file available for download should you want to get started quickly. Or alternatively have a look at my repositories usyd-beamer-theme or phd-thesis which I have set up to use this workflow. This file along with the Makefile this should enable this process to work for the compilation of most documents.

While the process is complicated to set up, once completed it shouldn't require much effort to maintain. There is the rationale it might save you some time, however I think it is cool which is all justification I needed.

Update 2018-07-26: Since initially writing this post I have created a conda package to distribute the biber binary. I have updated the post and included files to use this package since it significantly simplifies the installation. The steps from the original travis.yml file to manually install biber are below.

  # Download and install biber installing executable as biber2.5
  - wget https://sourceforge.net/projects/biblatex-biber/files/biblatex-biber/2.5/binaries/Linux/biber-linux_x86_64.tar.gz -O $HOME/downloads/biber.tar.gz
  - tar xvzf $HOME/downloads/biber.tar.gz -C $HOME/bin
  - mv $HOME/bin/biber $HOME/bin/biber2.5
  - export PATH="$HOME/bin:$PATH"

I should note that MiKTeX will install biber on a Windows system. So if you wanted to set up a Windows CI config I guess MiKTeX is a great approach.

There are more special characters supported, I just listed the most common ones. See the docs for more specific information.

You may notice that the minimal is not listed in the documentation. There is a GitHub issues to rectify this in which there is documentation.

Experi: A tool for computational experiments

2018-06-25T00:00:00+00:00

Abstract

One of the key features of computational experiments is being able to run the experiment over a large variable space. However, in my experience there aren't tools available to assist with this, particularly in the realm of High Performance Computing (HPC), where bash arrays and loops are commonplace. Using the current toolset, I made lots of errors in the specification of files, turning a 'quick edit' into a tedious process of find the bug. To make complicated experiment variable expression a simple and intuitive task I have developed Experi, which takes a list of commands and variables from a YAML input file and either runs them for testing or creates and submits files to an HPC scheduler. I have designed Experi to be a simple replacement for current workflows, using shell commands into which you add variables, and variables defined along with how they are combined. I have been using Experi as I have been developing it and though it hasn't prevented me from running software with bugs, I can specify the variables I want to use for a simulation without having to worry about the execution. As an added benefit, I also have a complete record of every simulation that I run which is version controllable.

Experi is installable using both pip and conda,

$ pip install experi

Currently, Experi only runs on python 3.6, because that is what I use and "premature optimisation is the root of all evil." If you would like to use Experi and can't use python 3.6, please let me know along with which python version you do use and I can look into supporting it.

I work in science where most compute intensive problems are handled by High Performance Computing (HPC) clusters. From the user perspective the entire cluster appears as a single host, with resources accessible through a job scheduler. You submit work as a script to the scheduler along with the resources required, where it will wait in a queue for those resources to be available. Once your script is running, it can access the resources allocated. When using a parallelism technology like MPI, the MPI software will detect all the resources and connecting the appropriate pieces together. This makes for a reasonably simple method of accessing the vast compute capabilities of HPCs.

Bash scripts are normally used to specify the work to take place; with additional scheduler directives denoting the resources required. One of scheduler directives you can specify is to make the job as an array job, where multiple instances of your file are submitted at once with an array index. In some ways this is similar to a numpy array operation rather than a for loop. The array job is able to make best use of the scheduler, splitting a large task into many smaller tasks which are easier to fit onto the cluster. This kind of workflow is ideal for running an experiment at different conditions, or with many replications.

With the scheduler I use, PBSPro an array job is specified using the -J flag and takes an argument instructing how to create the values. The argument has the form <start_index>-<end_index>:<increment>, e.g. 1-10:2 will generate five jobs for the scheduler which will each have the bash variable PBS_ARRAY_INDEX containing one of 1,3,5,7,9. While this can be useful where you need to vary a parameter over a sequence of integers, the more likely use case is creating a list of values which you are then indexing using PBS_ARRAY_INDEX like the example below

ARRAY = (1.0 1.5 2.0 2.5 3.0 4.0)
value = "${ARRAY[$PBS_ARRAY_INDEX]}"

When specifying my jobs for the scheduler I was making lots of mistakes writing a job script that was valid and reflected my intentions. This was difficult in that each job involved changing multiple variables and I only had a single index. While Bash is excellent for specifying complex commands for the computer to interpret, when dealing with long commands and lots of variables it quickly becomes too complicated to easily read. When I am designing a computational experiment I think about the commands I want to run separately from the variables I am going to use in those commands.

My experiments have a recipe that look something like;

Create a state with the properties I desire. This could be a liquid, a crystal, or a combination of the two.
Bring the initial state to the conditions I want to collect data, ensuring the new state is representative of those conditions.
Collect lots of data on how my experiment behaves at this specific condition.

I have presented each of these steps abstractly as they are applicable to many different experiments. It is also the level of abstraction for the command line interface, sdrun, I have written to help run my experiments. A command I would run for the create step would be

$ sdrun --temperature 0.1 --num-steps 100 --space-group p2 create configuration.out

I specify the variables for the simulation, the type of simulation being create, and then the output file.

For each of the three steps above I now want to define the specific variables I am using;

Crystal Structure:
- p2
- p2gg
- pg
Temperature:
- 0.4
- 0.45
- 0.50
- 0.6
Steps:
- 1_000_000
- 1_000_000
- 500_000
- 100_000

I want to run the same simulation for each of three different crystal structures, p2, p2gg, and pg. This simulation will be over a range of temperatures, with smaller spacing at lower temperatures where there is interesting behaviour. Also the effects I am looking for happen faster at higher temperatures so I am going to run shorter simulations for the higher temperatures.

In presenting the values for the variables in a concise way, a table is possibly clearer though much less concise, I am nearly using YAML syntax. YAML is a method of encoding both structure and data in a human readable format. You can think about it as a way of defining python lists and dicts in a text file. Lists are a series of bullet points,

- 0
- 1
- 2
- 3
- 4

with the snippet above being the list [0, 1, 2, 3, 4] in python. Dicts are mappings of values using the colon : as a separator

hello: world
foo: bar

being the dict {"hello": "world", "foo": "bar"} in python.

With the goal of being able to easily specify commands required for an experiment in addition to the variables to fill into those commands, I have developed Experi. It takes a YAML file as input, declaring the experiment that will take place, and either creates and submits files to the scheduler or runs the experiment in the current shell for testing. Taking the example I constructed above, to completely specify the experiment for Experi;

# experiment.yml
commands:
  - sdrun --space-group {crystal} -t 0.3 -s 1000 create configuration-{crystal}-0.3.out
  - >
    sdrun
    --space-group {crystal}
    --init-temp 0.3
    --temperature {temperature}
    --num-steps {steps}
    equil
    config-{crystal}-0.3.out
    config-{crystal}-{temperature:.2f}.out
  - sdrun --space-group {crystal} -t {temperature} -s {steps} prod config-{crystal}-{temperature:.2f}.out

variables:
  crystal:
    - p2
    - pg
    - p2gg

  zip:
    temperature:
      - 0.4
      - 0.45
      - 0.50
      - 0.6
    steps:
      - 1_000_000
      - 1_000_000
      - 500_000
      - 100_000

pbs:
  ncpus: 12
  walltime: 100:00:00
  j: oe

There are three commands which will be run; firstly a create step, secondly an equil step, finally a prod step. The longer command of the equil step is broken over multiple lines, and since I am using the YAML syntax for a multiline string >, it is interpreted as a single long string hence no need for line continuation characters. Where I want to replace the variables I have enclosed the variable name in braces ({}), which will be automatically replaced using python string formatting. Using this format allows for modifiers like {temperature:.2f} which is replaced with the temperature as a float having two decimal places, i.e. 0.4 -> '0.40'.

Commands in Experi are guaranteed to be run in a linear fashion, i.e. none of the equil simulations will run until the create simulations are finished, with the additional constraint that all previous steps are successful. This holds for both running in the local terminal, or using the job scheduler.

Variables for the experiment are specified in their own section, with the variable name as the dictionary key, followed by a list of values---a single value is also supported. Variables can take on any valid string with the exception of zip or product which are used define how the variables are combined. zip acts on the variables like a zipper, taking variables with the same number of values and matching the values up 1:1. In this case we get

temperature: 0.4 steps: 1_000_000
temperature: 0.45 steps: 1_000_000
temperature: 0.50 steps: 500_000
temperature: 0.6 steps: 100_000

product is also somewhat self-descriptive, taking the 'product' of all the values to give all possible combinations. product is the default operation and is not explicitly required so each of the three crystals will have simulations with the temperature and steps listed above.

Where multiple zip operations are required, a list of dictionaries can be supplied to the zip key with each being zipped separately before taking the product of each item in the list.

The final part of the YAML file is the specification of the options for the PBS scheduler. There are some values which can be passed as keys;

select/nodes
ncpus/cpus
ngpus/gpus
memory/mem
walltime
cputime
project
setup

While any options not specifically encoded can be passed by using the flag as the key. In the example above I have used j: oe which will become the scheduler option -j oe which is specifying that the stdout (o) and stderr (e) will be joined in the stdout stream.

The setup key allows for commands to load the required variables, modules, or anything else that is required before each command is run.

It should be noted that YAML is not without idiosyncrasies. However, it is powerful and widespread and I like and understand the format. The structure of Experi is not specific to YAML and could easily work with JSON or TOML input files if someone wants to put effort into the implementation and documentation.

With the input file specified, assuming it is named experiment.yml, the simulation can be submitted to the scheduler using the command

$ experi

which will submit the job to the scheduler if it is available. Regardless of the presence of the scheduler, a file is created for each command which includes all the combinations of variables specified in a bash array which is suitable for manual submission.

Using Experi for the design and execution of computational experiments has significantly benefited my science workflow. This is most noticeable with the replication and modification of experiments. I have a complete record of all the commands and options required for the experiment to run and I can focus on modifying values without having to worry about how to loop over the range of variables I require.

Distributing a Hoomd Plugin

2018-06-03T00:00:00+00:00

A piece of software I have been using in my research is Hoomd, a 'relatively' new package for running Molecular Dynamics (MD) simulations. These MD simulations have the basic premise of throwing hundreds of balls into a box and shaking it to find out what happens. The relative newness of Hoomd is in comparison to other software packages like LAMMPS and GROMACS which have been around for decades, while the initial release of Hoomd was in 2012. There are some major benefits of a newer approach to MD simulations and Hoomd is most notably designed to leverage the computational power of GPUs. Despite the benefits of a modern approach, Hoomd doesn't have the range of built-in simulation types of the more mature software packages. To allow researchers with specific problems to still use Hoomd, it has a plugin architecture to simplify the implementation of custom functionality.

This article documents how I took a plugin I wrote for my research and made it installable in a conda environment alongside Hoomd.

The Plugin

The example plugin is one which I have written for my own research. It implements a harmonic pinning potential on the positions and rotations of rigid bodies in a MD simulation. The complete plugin is available for reference on GitHub.

To get started writing your own plugin there is a short guide in the documentation which directs you to the example_plugin in the Hoomd source code. This code is located on bitbucket and there is also a copy on GitHub. I also found that finding a class that shared the same base class was useful for writing the implementation.

The Makefile

The example code for the example_plugin doesn't include a Makefile, requiring multiple steps to compile the plugin. A Makefile is useful in this case for both being able to install with a single command and ensuring the install options are consistent with the conda installed Hoomd.

While my plugin doesn't use CUDA for calculations, for compatibility with the conda installed Hoomd enabling CUDA on Linux is required. When I was compiling with CUDA disabled I came across unusual behaviour, for example numbers being far too large, which appeared to be a buffer overflow. I believe this is because of conditional definitions in the header files resulting in a slightly different memory layout for a class defined with or without CUDA.

If you are able to build your plugin using cmake like the example plugin, the following Makefile will make compilation simpler and set the appropriate flags assuming Hoomd was installed using conda.

build_dir = build

# Check OS to determine if CUDA is enabled, defaults to True
CUDA_ENABLED=True
UNAME_S := $(shell uname -s)
ifeq ($(UNAME_S),Linux)
	CUDA_ENABLED=True
endif
ifeq ($(UNAME_S),Darwin)
	CUDA_ENABLED=False
endif

all: $(build_dir)
	cd $(build_dir); cmake .. -DENABLE_CUDA=$(CUDA_ENABLED)
	$(MAKE) -C $(build_dir)

install: all
	$(MAKE) -C $(build_dir) install

clean:
	rm -rf $(build_dir)

test:
	pytest

$(build_dir):
	mkdir -p $@

.PHONY: test clean

Apart from making it simpler to install the package manually, it also makes it simplifies the build using conda.

Packaging

The conda package manager is the recommended method of installing Hoomd, primarily because of the simplicity of installation. Conda allows you to upload your own packages for anyone to download, which is how we are going to make this plugin simple to install.

To define the package for upload to the Anaconda Cloud repository we have to write a meta.yaml file. This defines everything required to build the package including; the package details, where to find the source code, dependencies required to build the package, dependencies to install the package, and how to build the package. There is [extensive documentation][meta.yaml documentation] for the meta.yaml file which covers a variety of use cases. The file uses the yaml syntax, which is a method of encoding python data structures in a more human readable format.

The meta.yaml file contains a series of sections each containing different information about the package. Some of these keys are;

package,
about,
source,
dependencies, and
test.

The documentation link above contains more extensive information on the options available for each of these sections. I am going to explain how I have used all the different sections to create this plugin.

The package key contains the name and version of the package. This section is compulsory, while all others are optional, although that doesn't mean they aren't required.

package:
    name: hoomd-harmonic-force
    version: 0.1.7

The about key is where you can find more information on the package, including the homepage and the license the package uses. In this case the homepage is a link to the Github repository.

about:
  home: https://github.com/malramsay64/hoomd-harmonic-force
  license: MIT
  license_file: LICENSE

The source section defines where to find the source code for the build phase. I have specified a specific tag from the git repository, enabling me to checkout an old commit to build an older version.

source:
  git_url: https://github.com/malramsay64/hoomd-harmonic-force.git
  git_rev: v0.1.7

Another useful option for the source section is use the path key. This key allows for the specification of the relative (or absolute) path to the source code. This is most useful in the development process, allowing for quickly testing whether a change has worked.

source:
    path: ./

The next section is the requirements, defining both the packages required for the build phase, and those required when the package is installed. These package definitions include the specification of compatible versions of the dependencies. The version specification is flexible enough for complex build processes, while also having the option for simplicity. In this guide I describe the simplest method of version specification. When you need something more complex, the documentation on build variants covers a wide range of use cases.

The build requirements are all the programs required for the build; in this case compilation with cmake. Since we are building a python module we require both python and setuptools. I am using python 3.6, so have set the python version to 3.6.* which means any point release of python 3.6, at the time of writing being 3.6.5. The numpy and Hoomd dependencies are also requirements of the build process, for which I have specified the latest versions. For python, numpy and Hoomd you can specify the versions you use for your work. Where you use multiple versions, like python 2.7, 3.5, and 3.6, the start of the build variants documentation has examples to adapt. The next dependency, cudatoolkit, is not strictly necessary in this case, however before Hoomd v2.2.5 there was a typo in the Hoomd version specification allowing a newer incompatible version to be installed. The final requirement, cmake, builds the package and while you might have it installed on your system already, someone else may not. It should be noted that the nvcc binary is also required on linux which I can't find a conda package for.

requirements:
  build:
    - python 3.6.*
    - setuptools
    - numpy 1.14.*
    - hoomd 2.3.*
    - cudatoolkit 8.*
    - cmake >=2.8.0

With the build dependencies specified, we need to specify the run dependencies. Version numbers under the [semantic versioning][] scheme have a format of <major>.<minor>.<patch>. Version numbers with the same major and minor numbers are typically compatible which is why I have specified the run requirements with the same minor version as the build requirements. I have come across previous versions of Hoomd which are only compatible with a singe patch version of python. Unfortunately, you can't pin the python version to a single patch as conda build overrides the pinning.

requirements:
  run:
    - python 3.6.*
    - numpy 1.14.*
    - hoomd 2.3.*
    - cudatoolkit 8.*

Having waded through the complications of dependency specification we now need to tell conda how to build our plugin. This is where creating the Makefile is useful, we can now use the same commands for build process as for a manual installation; make clean && make install. The make clean command removes the build files when using the path option for the source. The other element of the build section is the number, incremented when uploading another package with the same version and reset to 0 on a new version. The number acts as a sub-version number allowing for small fixes, like correcting the version pinning, without incrementing the package version number.

build:
  script: make clean && make install
  number: 0

The final section I will discuss is the test section. At its simplest this can import your newly created package, or it can run a complex unit and integration test suite. The test section failing will be a fail the build, providing a final check for bugs before release. In the example below I am checking the package will import as a first simple test, before running more extensive testing with pytest. To ensure pytest is installed, there is a requirements key in the test section allowing for the specification of test specific requirements. I have also specified the files for running the tests using the source_files key, which is the entire directory of test files. The final key is the commands to run your test suite. Since I have the test rule configured in my Makefile I make use of it here.

test:
  imports:
    - hoomd.harmonic_force
  requires:
    - pytest
  source_files:
    - test/*
  commands:
    - make test

I have put all the snippets into a single block of code at the end of this article or downloadable here. For more complex examples, you can have a look at the Hoomd repository or the example in my repository.

Building and Distribution

With all the metadata defined, conda makes it straightforward to create the package for upload. There are two conda packages required for building and uploading to Anaconda Cloud, which are both required in the root environment. You can install both packages with the command below, where the -n flag specifies installing to the root environment.

$ conda install -n root conda-build anaconda-client

With the conda-build package installed, you can run conda build . where the . is the path to the directory containing meta.yaml file. As part of build process conda will create new environments for both the build and test phases, preventing the packages in your current environment from interfering with the build process and ensuring you have specified all requirements. The build phase prepares the package for upload to Anaconda Cloud, which is handled by anaconda client. If you have already provided credentials the upload should be automatic, though on failure the error messages include steps to complete the upload.

With your package uploaded to the Anaconda Cloud, anybody can install it by specifying your repository. You can install my plugin with the command

$ conda install -c malramsay hoomd-harmonic-force

although I wouldn't recommend using it at this stage. That said, contributions are most welcome.

While the process of packaging is difficult, I have hopefully made it somewhat more approachable. The good thing is that once you have a meta.yaml file for a project, maintenance will mostly be updates of version numbers. As an added benefit, installing or updating your software is much simpler for you, and importantly, anyone else that wants to try it out. For why open source software if you don't intend for someone else to actually use it.

Entire `meta.yaml`

I have included the entire meta.yaml file below for ease of copying.

# meta.yaml

package:
  name: hoomd-harmonic-force
  version: 0.1.7

about:
  home: https://github.com/malramsay64/hoomd-harmonic-force
  license: MIT
  license_file: LICENSE

source:
  git_url: https://github.com/malramsay64/hoomd-harmonic-force.git
  git_rev: v0.1.7

requirements:
  build:
    - python 3.6.*
    - setuptools
    - numpy 1.14.*
    - hoomd 2.3.*
    - cudatoolkit 8.*
    - cmake >=2.8.0

  run:
    - python 3.6.*
    - numpy 1.14.*
    - hoomd 2.3.*
    - cudatoolkit 8.*

build:
  script: make clean && make install
  number: 0

test:
  imports:
    - hoomd.harmonic_force
  requires:
    - pytest
  source_files:
    - test/*
  commands:
    - make test

Scraping Data from Facebook

2018-05-09T00:00:00+00:00

Competition is a strange thing making you suddenly interested in the most unusual of problems It has become a tradition of The Lancer Band, of which I am a member, to produce a video as part of our ANZAC Day commemorations. These videos have been highly successful garnering millions of views on Facebook and with some choice communications throughout the rest of the year have resulted in a commendable social media following.

After a successful drive to raise the number of page likes this year, we became more interested in how we compared to other similar pages, from other service bands in Australia, to other units of the Australian Army. Rather than manually collating all this data manually, I looked for an automated solution for this process, both to have up to date data, and to store historical data allowing for comparisons with these pages over time.

The method the internet recommends for getting data from Facebook is through their GraphAPI, a method of using http requests query specific data. The first step in using the GraphAPI is getting an access token, the simplest method though logging into the GraphAPI Explorer, a web based tool for testing GraphAPI. In the process of getting an access token from the GraphAPI Explorer, a prompt will ask for the permissions granted to the access token. You can uncheck all the permissions since they are not required for this example. The more permissions granted to an access token, the more data from the GraphAPI is accessible, here interested in the fan_count, which is the number of users who like the page. Documentation for this value, and all the other values can are available in the page documentation.

With the page and data field to query and the access token, it is possible to query the GraphAPI in python using the requests package as in the code example below.

import requests
page = 'thelancerband'
token = '<access_token>'
ret = requests.get(f'https://graph.facebook.com/{page}', params={'fields': 'fan_count', 'access_token': token})
ret.json()['fan_count']

Note that you will need to replace <access_token> with the token generated from the GraphAPI Explorer. At the time of writing The Lancer Band has 6546 likes, which is the value output when I ran the code snippet above. Hopefully when you are try this the result is somewhat larger.

While the GraphAPI has a relatively simple interface for querying many different types of data, it has a fairly significant drawback for accessing data over periods of time. The access tokens for the GraphAPI Explorer have an expiry of 1 hour, meaning nearly every time you want to perform data collection you need a new token. Programmatically generating access tokens requires the creation of a Facebook application. While this sounds simple enough, it requires getting the app reviewed and accepted by Facebook, far more work than I was willing to put into this. While these requirements are fair enough for all the personal information having access to a Facebook profile provides, I only want to access the publicly available likes data of a page.

With all the effort required to go through the 'proper' methods to obtain this data, I looked for alternate approaches. As far as I can discern coming from the scientific realm, two of the key packages for collecting and processing data from websites

requests for an interface with http requests and responses
beautifulsoup4 which provides an interface for extracting information from a returned html/xml document.

We are going to use requests to get the page which has the number of likes contained on it, then use beautifulsoup4 to extract the number required.

Finding the web address to use in the request is a case of navigating Facebook manually, with the number of likes appearing on the community page having the url [https://facebook.com/pg/thelancerband/community][]. The contents of this page can be downloaded using the code snippet below;

import requests
page_name = "thelancerband"
community_page = requests.get(f"https://facebook.com/pg/{page_name}/community")

This gets the html, from which we need to extract the number of likes. Printing the entire returned page to output

community_page.text

gives a huge wall of text. Using the search functionality to look for instances of 'Total Likes' finds the following code snippet

...<div class="_3xom">6,546</div><div class="_3xok">Total Likes</div>...

The number of likes is in the div element preceding the div containing the text Total Likes. Beautifulsoup4 allows us to programmatically traverse the document in a similar logic

from bs4 import BeautifulSoup
document = BeautifulSoup(community_page, 'html.parser')
document.find(text='Total Likes').parent.previous

Here we search for text matching 'Total Likes', which returns a reference to that location in the document. The number of likes is the content of the previous div element, by taking the parent element to select the surrounding div, then the previous element we extract the value of the previous div element.

With the total number of likes as string, we need to reformat it to convert to an integer. This involves removing the comma as the thousands separator and the SI prefixes k, M, etc. when the likes are in the hundreds of thousands or more. The quick and dirty method of removing the comma is using str.replace(',', ''), though a real project should use the locale module as described on stackoverflow. A simple fix for the prefixes is installing the humanfriendly package, and using the humanfriendly.parse_size() function which will handle the prefixes. This results in the code for getting likes from a list of pages looking like that below.

from bs4 import BeautifulSoup
import humanfriendly
import requests

page_list = [
    "thelancerband",
]

def get_num_likes(page_id):
    community_page = requests.get(f"https://facebook.com/pg/{page_id}/community")
    document = BeautifulSoup(community_page, 'html.parser')
    page_likes = humanfriendly.parse_size(
        document.find(text='Total Likes').parent.previous.replace(',', '')
    )
    return page_likes

for page in page_list:
    print(f'The Facebook page for {page} has {get_num_likes(page)} likes')

This prints the results to the screen, which is not useful for longer term analysis although it is easy to see that everything is working as expected.

There are many different methods for storing this data for later analysis, from a simple csv file to a sqlite database. I am most familiar with pandas and HDF5 files, hence these were my tools of choice resulting in the data collection loop below.

import pandas
data = []
for page in page_list:
    data.append({
        'page_id': page,
        'likes': get_num_likes(page),
        'time': pandas.Timestamp.now(),
    })
df = pandas.DataFraame.from_records(data)
df.set_index('time').to_hdf('likes_data.h5', 'data', format='table', append=True)

Note that the tables package is also a requirement for this code snippet to work.

This is my first foray into scraping websites for data analysis which is surprisingly simple with the appropriate tools. The biggest issue I encountered in this project was trying to understand the Facebook authentication tokens. While this is a simple example, it can easily be extended for different use cases.

A guide to setting up remote SSH

2018-04-09T00:00:00+00:00

In the right (or wrong) hands ssh is a powerful tool for the remote management of a Unix system. Most desktop, or workstation distributions of Linux disable remote access over ssh by default. The simplest method to check if you have ssh server running on your machine is to run

$ ssh localhost

If ssh is not installed or running this will print out a message

ssh: connect to host localhost port 22: Connection refused

most likely indicating that the ssh server is not running.

Before heading any further in enabling remote access over ssh, it is a good idea to ensure that you have a strong password for all of the accounts on the device, in particular the root account (even better is disabling it). By enabling remote access for yourself you are also enabling remote access for anyone else that might want to try and log in.

Installation

The package required for installation is in most distributions named openssh-server. So for Ubuntu running the command;

$ sudo apt install openssh-server

will install the package as appropriate. Other Linux distributions have different package managers so use the one for your distribution. So for fedora, replace apt with dnf and for CentOS replace apt with yum. On macOS everything is already installed and just needs to be enabled in System Preferences > Sharing then ensure Remote Login is checked.

Once the package is installed, it needs to be both added to the list of packages to run at boot, and started now to test.

To start the openssh server, run the below command

$ sudo systemctl start ssh

which is using the systemd init system to start the openssh server instance. For fedora and CentOS, the ssh service instead has the name sshd so run

$ sudo systemctl start sshd

to start the server instance. Since this is probably a service you want running automatically on boot, running the command

$ sudo systemctl enable ssh

will put files in the appropriate places to enable the service on boot. Other useful commands for systemctl are stop, restart, and disable

Once the ssh server is running our command to connect at localhost

$ ssh localhost

should present a new message like the one below.

The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is ba:ba:b1:4c:5a:e2:aa:40:aa:7f:aa:01:aa:df:aa:47.
Are you sure you want to continue connecting (yes/no)?

In some ways this is like the terms and conditions when you sign up for a website in that you should care about and understand it, however, typically you click through (or type yes) without reading. Since we are connecting to localhost, which has an IPV6 address of ::1 or IPV4 address of 127.0.0.1 we can trust the key fingerprint. This is a method of checking that whenever we log into a host over ssh, the host hasn't changed which if unexpected could be indicative of a man-in-the-middle attack or other nefarious actions.

Now we are able to connect to the machine locally, we need to work out how to connect from a remote machine.

How do I Find My Device?

With the previous commands we have been passing localhost to the ssh command, in the same way we might type google.com into a web browser. These names are a human readable format for addressing devices, which map to IPV6 or IPV4 addresses computers read to initiate a connection.

To find the name of the computer we are working on, also known as the hostname, we can use the hostname command.

$ hostname -f
lovelace.staff.sydney.edu.au

Here the -f gives the full hostname of the device you are using, with a -s flag also supported, however we need to full hostname.

With the full hostname, we know the name of our device, and we know that our device is aware of it's hostname. This doesn't necessarily mean that any other devices know what ours is called. The process of taking a hostname and turning it into an address the computer can use is known as DNS, which is provided by name servers. Typically in organisations there are multiple levels of name servers, which will handle both internal names, like the hostname of our device, and external names like github.com or wikipedia.org. Where a device is querying an external name server, it will not be able to resolve local names. To test this we can use the name server lookup nslookup command.

$ nslookup lovelace.staff.sydney.edu.au
Server:         172.18.240.210
Address:        172.18.240.210#53

Name:   lovelace.staff.sydney.edu.au
Address: 10.65.205.200

This tells us that the server at the ip address 172.18.240.210 which my laptop is using as the default name server, knows that lovelace.staff.sydney.edu.au exists at the address 10.65.205.200. If we instead used one of the large public DNS servers provided by Google (8.8.8.8) or CloudFlare (1.1.1.1) we will instead get an error message.

$ nslookup lovelace.staff.sydney.edu.au 1.1.1.1
Server:         1.1.1.1
Address:        1.1.1.1#53

** server can't find lovelace.staff.sydney.edu.au: SERVFAIL

Setting the hostname

When going through the initial setup of macOS or linux machines, there is a step where you give the machine the hostname. To later change the hostname requires the editing a couple of files. The file /etc/hostname stores the hostname of the machine, which will typically be the Fully Qualified Domain Name (FQDN) of the host. To actually change the hostname a reboot is required, as this file is read during the boot sequence. Before you do reboot, there is another file /etc/hosts that requires editing.

The /etc/hosts file is the fist place that is checked when looking to resolve a domain name and is sometimes used as a way to block websites or just to resolve hostname. We want to our domain name to resolve to 127.0.0.1 and ::1 so that services like web servers don't get confused. Ensure the following lines (with the appropriate hostname) are in the /etc/hosts file

127.0.0.1   lovelace lovelace.malramsay.com
::1         lovelace lovelace.malramsay.com

This sets the ip address we want the hostname to resolve to, which on the first line is 127.0.0.1, followed by a list of hostnames, being the short version lovelace and the FQDN lovelace.malramsay.com.

What do I do if I can't lookup my hostname?

The reason we use domain names and hostnames is that they are far easier to remember than a string of numbers. It is possible to navigate to Google by navigating to the ip address 216.58.200.110 which is the address returned for google.com.

$ nslookup google.com
Server:         172.18.240.210
Address:        172.18.240.210#53

Non-authoritative answer:
Name:   google.com
Address: 216.58.200.110

While using the IP address is possible, it is far more cumbersome to remember and to type. For a single IP address which we can put in a configuration file, this isn't too problematic.

To find the ip address of the current host run the command ip addr, which will give output like that below.

$ ip addr
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet 127.0.0.1/8 lo0
        inet6 ::1/128
        inet6 fe80::1/64 scopeid 0x1
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 78:4f:43:4d:cc:53
        inet6 fe80::18c5:93d4:e972:b798/64 secured scopeid 0x7
 >      inet 10.16.249.156/21 brd 10.16.255.255 en0
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484
        ether b6:54:4e:bd:a7:d7
        inet6 fe80::b454:4eff:febd:a7d7/64 scopeid 0xa
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
        inet6 fe80::1c95:35f4:acea:1034/64 scopeid 0x10
en5: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether ac:df:48:00:11:22
        inet6 fe80::aede:48ff:fe00:1122/64 scopeid 0x9
en7: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 00:0e:c6:e2:bb:69
        inet6 fe80::1871:5dc7:1f58:2127/64 secured scopeid 0x12
 >      inet 10.65.205.200/24 brd 10.65.205.255 en7

The lines we are looking for are those with inet at the start which I have denoted with a >.

This gives me two IP addresses, one for the wifi and another for the wired ethernet connection. You may have noticed that the 10.65.205.200 address matched that obtained from the nslookup of my hostname above. This tells me that the name server is directing queries of my hostname back to my device.

Now we have an ip address, how do we make the computer remember it when the name server doesn't.

Remembering IP addresses

The DNS lookup of IP addresses from hostnames works on the institutional network I have access to at the University of Sydney. You will likely not have this same configuration, so there are some alternate methods for getting your computer to remember ip addresses.

SSH configuration file

The simplest method of getting the computer to remember an ip address for you is the ssh config file ~/.ssh/config. This approach only works when connecting using ssh, however, it is useful for setting all the parameters for an ssh connection. By putting the code snippet below in ~/.ssh/config

Host lovelace
    Hostname 10.65.205.200
    User malcolm

running the command ssh lovelace, is equivalent to running ssh malcolm@10.65.205.200 and is much easier to both type and remember (and tab-complete). Additional useful parameters for the configuration are

LocalForward/RemoteForward for port forwarding,
Port when using a non-standard port, and
IdentityFile for using specific private keys.

The ssh_config man pages (man ssh_config) also contain all the parameters available and information about each of them.

Hosts file

Earlier we modified the /etc/hosts file to change the hostname of our system. We can also use it to remember the ip address of other systems. This is not recommended for managing many devices, use DNS for that, for a small number of systems we can use use this file for the translation of domain names to ip addresses. Instead of using the ip address of localhost, we use the ip address we want the hostname to resolve to as shown below.

10.65.205.200   lovelace.malramsay.com

The advantage this approach has over the ssh configuration is that it will also resolve the hostname in the web browser. This also makes it much easier to access web services like jupyter notebooks on the remote host.

Conclusion

This should be sufficient to get started using SSH within the same network. To access the service over the internet, where possible use the institutional VPN, or set up your own.

Adventures in the Python Visualisation Landscape

2018-03-17T00:00:00+00:00

Calling the current python visualisation landscape fragmented would probably be an understatement, since itself requires a visualisation to even begin to comprehend. For various reasons I have been unhappy with the tool I was using for visualisation at that time and I have been searching for the one visualisation package to rule them all (spoiler: it doesn't exist)

About a year ago I was using Matplotlib for all my figures, which enabled me to create anything I wanted, usually with a stackoverflow answer giving me a working example to adapt. Problem was, nearly everything required a stackoverflow post or the documentation, making the process of building these figures slow and tedious. Around the same time I was getting fed up with Matplotlib, I found Jake VanderPlas' PyCon talk on The Python Visualisation Landscape which has since acted as both a map and a list of achievements to unlock.

Altair 1.2

Somewhat naturally the first package I looked at for improving my visualisation workflow was Altair which was at the time a 1.2 release. At the time I thought it was fine for simple figures, however my data was not formatted to make the most of what Altair could offer, which as a relative newbie to pandas was frustrating. Another sticking point with Altair was working out how to customise figures, After Matplotlib, I was expecting to search what I wanted to in Google and have appropriate stack overflow answer on the first page of results. With Altair being a new package, this how-to style documentation didn't exist, which meant I never worked out how to customise the figure. The most frustrating of the default settings was the axis labels which use the SI Prefix for large numbers i.e. 1Mm for $1 \times 10^6$ and 1no for $1 \times 10^{-9}$, something which I never worked out how to change at the time. Another problem I had with Altair (and Matplotlib) was the lack of interactivity. I was in the early stages of data investigation which required both a high level overview of the trends, while also being able to look at smaller regions in more detail. Having to constantly change axis ranges was making it slow and frustrating to create a figure with the appropriate view.

Bokeh

With an emphasis on interactivity, Bokeh was the tool I was looking for to investigate data on different scales. With the ability to generate figures in a notebook, for quick visualisations of datasets to understand the data, and compare it with previous studies, and also as a Bokeh server application, to perform standard analyses on large volumes of data, like ensuring a simulation is running properly. Despite having these interactive visualisations, bokeh is lacking in the same way as Matplotlib, it takes a long time to specify everything required to create the figures. While there was a start on a high level plotting interface in the form of bkcharts, the code is now unmaintained and directs users to Holoviews.

Holoviews

Like Altair, Holoviews is a declarative plotting interface, providing a quick and simple interface for constructing figures. Holoviews is structures around the idea of describing data on creation of the dataset, rather than the construction of the figure, allowing for simple figure definitions. Rather than being a library that actually creates a figure, Holoviews performs the reasoning about the dataset and passing the rendering off to Bokeh, Matplotlib, or Plotly. I initially saw this as a big strength of Holoviews, that I could generate interactive visualisations in bokeh, change the output to Matplotlib and have a configurable figure for publication. In practice this is not so simple, with some modifications to plot style parameters changing between the different output formats and having a limited range of customisations.

The drawback of having this special annotated data object is that you lose all the flexibility of having a pandas DataFrame and the vast array of operations that allows. This is important to me because my field of science has a long history of researchers taking some quantities and combining them to give a describable temperature dependence. This lack of flexibility led me back to Altair, this time version 2.0.

Altair 2.0

Why Altair again? When I first tried Altair I was approaching it from Matplotlib, with its extensive documentation in the form of the technical reference and the numerous how to guides. This time I was approaching it from Bokeh and Holoviews, both of which are relatively new libraries with some teething problems. As a consequence I had become much better at problem solving issues and navigating technical reference materials. Another major moment of discovery was making the connection that since Altair implements the Vega-Lite specification, I should have a look at the Vega-Lite documentation. This turned out to be particularly helpful because the Vega-Lite documentation is currently more extensive than for Altair.

It was in reading the Vega-Lite documentation that I finally understood how to use transform functions in Altair. These are a set of functions that perform computations on the input dataset to generate the resulting figure. This allows me to have a single canonical dataset, with data transformations like ratios of two quantities tied to the figure, rather than following awkwardly named variables around. One example of these functions is to make the data in the cars dataset useful for the majority of the world's population by converting the units as part of the figure definition.

import altair as alt
from vega_datasets import data

cars = data.cars()

chart = alt.Chart(cars).mark_circle().transform_filter(
    alt.expr.datum.Miles_per_Gallon > 0
).transform_calculate(
    'Fuel Economy (L/100 km)', '235.2 / datum.Miles_per_Gallon'
).transform_calculate(
    'Weight (kg)', 'datum.Weight_in_lbs * 0.45'
)
chart.encode(
    x='Fuel Economy (L/100 km):Q',
    y='Weight (kg):Q',
    size='Acceleration:Q'
)

This allows for only storing the fundamental values in the dataset and being able to compute derived values as part of the figure, something that is particularly useful in my workflow.

This computing of values, also extends to the computation of histograms, complete with shortened notation. Using the chart object from above it is possible to easily create a histogram

chart.mark_bar().encode(
    x=alt.X('Fuel Economy (L/100 km):Q', bin=True),
    y='count():Q',
)

Where setting bin=True will create bins with the default parameters, and the count():Q on the y axis counts the elements in each bin. Instead of count() it is also possible to perform other aggregations, like computing the mean of a column.

chart.mark_bar().encode(
    x=alt.X('Fuel Economy (L/100 km):Q', bin=True),
    y='mean(Weight (kg)):Q',
)

For a more comprehensive view of using Altiar, have a look at either the Example Gallery, or a case study.

Each of the visualisation libraries in python have their own strengths and weaknesses, types of visualisations they excel at, and others you wouldn't want to try. For me, while Altair does still have a some quirks, most notably in the handling of large datasets, and a somewhat complicated method of setting titles, it provides a simple and intuitive interface to data which at the time of writing makes it the first tool I will reach for to understand a dataset.

Creating an HDF5 Bomb

2018-03-15T00:00:00+00:00

You may have heard of a zip bomb or other decompression 'bombs', which have the basic premise of containing a large volume of highly redundant data that when decompressed takes up more resources than the system can handle. Within the HDF5 file format there is support for compression, an excellent tool for reducing file sizes, however also ripe for exploitation. This 'issue'¹ of a file containing far more data expected, whether accidental or malicious, is not limited to HDF5 files, any filetype supporting compression is susceptible.

How do you actually create one a decompression bomb. The simplest method in python is to create a pandas DataFrame comprising lots of strings which are all the same.

import pandas as pd
large_number = 1_000_000
df = pd.DataFrame({'evil': ['😈']*large_number})

Using the sys.getsizeof function we can find the size in memory of this DataFrame

import sys.getsizeof
sys.getsizeof(df) / 1024 / 1024  # bytes / kilobytes / megabytes

which turns out to be 840 MB, 10 times larger than a DataFrame with the same number of integers. This significant overhead of using strings is because each element in the DataFrame contains all the storage overhead of a python object,² rather than each column for numerical types, and is the main reason I chose strings for this diabolical construct.

Now we have a large, low entropy dataset we need to save it to disk.

with pd.HDFStore('dataset.h5', mode='w', complevel=9, complib='bzip2') as dst:
    dst.append('data', df)

A number of keyword arguments are set in opening the HDFStore file handle,

mode='w' ensures that a file with the same name is overwritten if it exists, making testing this out much simpler.
complevel=9 sets the compression to the maximum possible not caring about the processing requirements of doing so.
complib='bzip2' sets the compression library to bzip2 which had significantly better compression ratios than the default zlib in my testing of this.

Additionally the append function has been used to write the DataFrame to disk since we want to create a dataframe that doesn't fit in memory, which will require appending to the file numerous times. With these options, the 840 MB DataFrame is a 2.2 MB HDF5 file on disk, a compression ratio greater than 200.

With all the pieces in place we can now construct our file.

num_iters = 20
with pd.HDFStore('dataset.h5', mode='w', complevel=9, complib='bzip2') as dst:
    for _ in range(num_iters):
        dst.append('data', df)

This generates a small unsuspecting 42 MB file³ which when loaded as a pandas DataFrame becomes an 18 GB object in memory, a compression ratio of 440.

While this is a little fun and devious, it does highlight the importance of thinking about how we represent the data we are processing, in particular text data. I originally came across this HDF5 bomb by accident, leaving a field as text when it should have been a category. In this devious case presented, using the type 'category' in the DataFrame

df['evil'] = df['evil'].astype('category')

our 840 MB DataFrame becomes 10 MB, and there are no issues storing 20 of them in memory. While it may be cool to use big data tools like Spark, Dask, or Hadoop sometimes the simplest approach is to make the big data small.

The compression is working exactly as intended, it just hides the true size of the underlying data.

From interrogating the size of the DataFrame using either sys.getsizeof(df) or df.memeory_useage(deep=True) it appears that the memory is allocated for each object. When querying the individual objects using id, they all return the same value, which is the same as just the string. I don't know what is going on and would be happy for someone to point me to a good resource.

Of course you are going to download a file some random stranger on the internet tells you is going to crash your python interpreter.

The Perils of Packaging in Python

2018-01-07T00:00:00+00:00

There are many guides to packaging a python application, including the official Python Packaging User Guide. While these guides offer step by step instructions for deploying a simple application, when deviating from a guide there can be unexpected problems.

The application I have been trying to deploy is this one, a collection of tools to assist in the analysis of molecular dynamics trajectories for my PhD. Most of the code I have written is python, with small sections of Cython code for performance. I have been running automated testing on Travis-CI and have made many different attempts at deploying the package to both Anaconda Cloud and PyPI. I have been trying to have a deployment that when a git tag is pushed to GitHub, Travis will test, build and upload both the Anaconda and PyPI packages for Linux and macOS.

Rather than document what I have found to be the 'perfect' method, something which I still don't have, this is a collection of my failures that will hopefully prevent someone else making the same mistakes.

Packaging and Deploying

Versioning

In deploying an application a version scheme provides an indication of the changes made, especially where a semantic versioning scheme is used. To make the versioning process simpler, I have been using git tags as the 'true' version number. To get the version number in the setup I have been using the setuptools_scm package. The versioning scheme that best fit how I have been thinking about release versions is the post-release scheme, using

use_scm_version={'version_scheme': 'post-release'}

The default state of setuptools_scm is to only include the dirty flag when files committed in the project have changed, rather than any untracked file in the directory. This change to a post release scheme also changed this default, meaning that when compiling on Travis every commit was dirty. This was fixed by also specifying the local version scheme

use_scm_version={'version_scheme': 'post-release',
                 'local_scheme': 'dirty-tag'},

While having a hash after the version number is not a problem for me, it is a large problem when trying to upload packages to PyPI, which rejects it with the error

HTTPError: 400 Client Error: version: Cannot use PEP 440 local versions. for url: https://upload.pypi.org/legacy/

Travis

While Travis appears to have good support for uploading to PyPI, having a deployment target, as far as I can tell it doesn't support non alphanumeric passwords. Unfortunately I was unable to get some alternative methods of including a password to work. Authentication information for uploading to PyPI can be in the ~/.pypirc file, however when trying to write this file using variable expansion

echo -e "[pypi]\nusername=$PYPI_USERNAME\npassword=$PYPI_PASSWORD" > ~/.pypirc

resulted in the file

[pypi]
username=$PYPI_USERNAME
password=$PYPI_PASSWORD

with neither of the variables being expanded. I quickly gave up on that approach since it worked in every shell I tested it on locally.

A solution that works is to use [twine][twine] which along with generally being more secure is able to read the environment variables TWINE_USERNAME and TWINE_PASSWORD to authenticate.

Installing from Pip

Once a package is uploaded to PyPI it needs to be installable, i.e. running

$ pip install <package>

actually works, even in a clean environment. A sample setup.py file for a python package typically looks something like the one below (1, 2, 3).

# setup.py
from setuptools import setup, Extension
from Cython.Build import cythonize
import numpy

my_extensions = [Extension(..., include_dirs=numpy.get_include())]

setup(
    name=...,
    ...,
    setup_requires=['Cython', 'numpy'],
    install_requires=['numpy']
    ...,
    ext_modules=cythonize(my_extensions),
)

Problem is, when installing into a new environment via pip the install fails as reading the setup.py file requires both Cython and numpy to be installed as they are imported at the top of the file. This is a significant problem with python packaging for which a solution has been accepted PEP 518 and work is in [progress][pep 518 progress]. In meantime, distributing a wheel is a solution to this problem.

Update 2018-08-12

The release of pip 10.0 introduced support for a pyproject.toml file in which installation dependencies can be specified. A pyproject.toml for the above project would look like:

# pyproject.toml
[build-system]
requires= ['setuptools', 'wheel', 'numpy', 'Cython']

Wheels

I have included wheels as their own section since I feel they are especially egregious. The PyPI documentation tells you that "You should also create a wheel for your project." Well sure, I will just follow the instructions in the docs

$ python setup.py bdist_wheel
$ twine upload dist/sdanalysis-0.4.4.post5-cp36-cp36m-linux_x86_64.whl
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: malramsay64
Enter your password:
Uploading sdanalysis-0.4.4.post5-cp36-cp36m-linux_x86_64.whl
HTTPError: 400 Client Error: Binary wheel 'sdanalysis-0.4.4.post5-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://upload.pypi.org/legacy/

Huh??? I also think it is great that I have to wait for the package to upload to get the error message.

Some googling later...

I need to use auditwheel to create a manylinux1 wheel. After installing via pip (conda-forge only has auditwheel versions for up to python 3.5)

$ auditwheel repair dist/sdanalysis-0.4.4.post5-cp36-cp36m-linux_x86_64.whl
Repairing sdanalysis-0.4.4.post5-cp36-cp36m-linux_x86_64.whl
usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: cannot repair "dist/sdanalysis-0.4.4.post5-cp36-cp36m-linux_x86_64.whl" to
"manylinux1_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to
compile the wheel on an older toolchain.

Some more googling and reading documentation ...

From the auditwheel README:

But in general, building manylinux1 wheels requires running on a CentOS5 machine, so we recommend using the pre-built manylinux Docker image.

$ docker run -i -t -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /bin/bash

Oh easy, now I have to set this up to build in a docker container...

For a great example of building a manylinux1 wheel, I can suggest checking out the pypa/python-manylinux-demo repository. Unfortunately it is not linked from the python packaging documentation and so is not that simple to find.

Packaging in python is hard. I have over 70 builds this week on travis just trying to get everything to actually work. At the end of all this I still don't really know what I am doing other than basically everything I try doesn't work and that I shouldn't try any of the above, since I already know it won't work.

Building and Testing

A crucial aspect of deploying a piece of software is having some confidence that it actually works, even if that just means it doesn't break on import. For my workflow this is an automated process, with builds running on Travis-CI when a new commit is pushed to Github. Even using a popular service live Travis didn't preclude me from having plenty of failures.

Travis, Python and macOS

I had automated testing set up on Travis-CI for my python module on linux, however trying to extent this to also using macOS I found some unusual behaviour. At first I thought that testing on travis was going to be as simple as adding it to the list of os's.

# .travis.yml
language: python

os:
  - linux
  - osx

However it really wasn't because the build raised an error before my code even started running.

$ sudo tar xjf python-3.6.tar.bz2 --directory /
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
The command "sudo tar xjf python-3.6.tar.bz2 --directory /" failed and exited with 1 during .

It turns out that on Travis-CI, the python language is unsupported on macOS. So I had to bodge some kind of workaround, especially since the travis docs on the issue assume you are using tox. I found were that when using conda to install python, the simplest workaround is to use language: generic and install python through conda for each OS.

# .travis.yml
language: generic

os:
  - linux
  - osx

before_install:
  - if [[ $(uname -s) == "Darwin" ]]; then
      wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh;
    else
      wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
    fi

Alternatively, when just using the latest version of python it can be installed using homebrew.

# .travis.yml
language: python

matrix:
  include:
    - os: linux
      python: 3.6

    - os: osx
      language: generic
      before_install: brew install python3

Frozen versions melt

When deploying with [Conda][conda] the recipe is created in a [meta.yaml][meta.yaml] file, containing the requirements for each step, in addition to instructions for building and testing of a package. A very simple version is shown below specifying a patch version of python (3.6.1) to use for both the build and the run phase.

# file: meta.yaml
package:
  name: pyver_test
  version: 1.0.0

requirements:
  build:
    - python 3.6.1
  run:
    - python {{ pin_compatible('python', max_pin='x.x.x') }}

build: number: 0
test:
  commands: python --version

While the version number is adhered to in the build phase, during the run phase the patch version is ignored, instead installing the latest patch version of python, which at the time of writing is 3.6.4. This is the intended behaviour, albeit somewhat unexpected.

The reason that this was a problem for me is that I came across an unusual bug in a conda package I was trying to use. When installed on macOS

$  conda create -n freud -c glotzer freud

importing the module would crash python

$ python -c "import freud"
Fatal Python error: PyThreadState_Get: no current thread

zsh: abort           python -c "import freud"

This environment would use the latest patch version of python (currently 3.6.4) and a simple workaround was to install python==3.6.1, although it did make installing the package more difficult than I would have liked. I don't know why this worked, it just did, although I suspect it has something to do with linking to the python C libraries.

When the porcelain stains

Pipenv is a packaging manager for python that automates the creation of virtual environments for python projects in addition to the packages installed within them. As Jannis Leidel is quoted in the testimonials;

Pipenv is the porcelain I always wanted to build for pip. However there are always unusual cases when abstracting that can cause issues.

The first of these issues I found was in the installation, I primarily use Conda for my python and virtual environment configuration,

$ pip install pipenv
$ pipenv install numpy
ERROR: virtualenv is not compatible with this system or executable

However, it is required to install pipenv outside of a conda distribution (issue#699, issue#288). So installation can be performed by running

$ /usr/local/bin/pip install pipenv

or with conda 4.4

$ conda deactivate
$ pip install pipenv

One of the major features of Pipenv over other package managers for python is that it includes hashes of the binaries as part of the version pinning. This both ensures you are installing exactly the same code on different systems, and provides security in preventing unknown changes from being installed. These hashes are collected for all the package versions available on PyPI, allowing for cross platform and python version use.

The problem that I came across is that a certain package numpy-quaternion has some nearly identical version numbers,

numpy-quaternion 2017.11.26.19.0.13
numpy-quaternion 2017.11.26.19.00.13

the second just having an additional zero. The first of these version numbers contains all the pre-built wheels, while the second contains the source. When downloading the hashes, pipenv downloads the hash for the source distribution rather than for the wheels. This means that when trying to install in a new virtualenv the downloaded package fails the hash comparison

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!

and aborts the install, incredibly frustrating when the initial install of the package works fine.

A workaround is to manually update the Pipfile.lock file with the correct hashes, which is fine while dependencies are not updated and the Pipfile.lock file remains unchanged.

Update 2018-01-09

In a previous revision of the Installing from Pip section I stated that importing the dependencies within a function call was a solution to the import problems. Thanks to /u/vorpalsmith for bringing it to my attention that this isn't a solution, and that the success that I observed was a result of installing a wheel rather than any changes to the code that I made.

Quaternions for Orientation in MD

2017-09-13T00:00:00+00:00

In the understanding of the dynamics of a molecule in a Molecular Dynamics (MD) Simulation the two main properties we use to understand liquid behaviour are the translational motion and the rotational motion of the atoms or molecules. These motions reflect changes in the position and the orientation of the molecules. When calculating these motions in an MD simulation it makes sense to represent each molecule as;

the position of the Center of Mass (COM), and
the orientation within the lab reference frame.

This representation allows for quick and simple calculation of the motion between two separate times. In the simplest case, conceptually this is just computing

\[ \begin{aligned} \text{displacement} &= \text{norm}(\text{position}_2 - \text{position}_1) \\ \text{rotation} &= \text{norm}(\text{orientation}_2 - \text{orientation}_1). \end{aligned} \]

For the displacement, we can just use a standard Euclidean distance norm. Which is simple enough to compute. The rotations are somewhat more difficult. In a two dimensional system it would be possible to use just the angle, where the norm is a function keeping the rotation within a sensible bounds, like $(-\pi, \pi]$. However in three dimensions using angles gets more complicated and computationally expensive. The approach that is used in hoomd and likely many other molecular dynamics programs is to use quaternions for the representation of orientation.

This post aims to give an understanding of quaternions and how to use them to compute a rotation from one orientation to another.

Orientation in 2D

We will start with a simple case, orientation in two dimensions. In two dimensions it is possible to describe the orientation of a molecules using a single angle, $\theta$. This is a perfectly reasonable representation. We can find the rotational distance $\Delta\theta$ between two orientations as

\[ \Delta\theta = \theta_2 - \theta_1 \]

This can become a bit of an issue if we want to keep our angle bounded. A reasonable range for an angle of rotation is $(-\pi,pi]$, as we can assume a rotation larger occurred in the opposite direction. The code to implement this in python would look something like the function below;

import math

def rotationalDistance(theta1, theta2):
    delta_theta = theta2 - theta1
    if delta_theta > math.pi:
        delta_theta -= 2*math.pi
    elif delta_theta <= -math.pi:
        delta_theta += 2*math.pi
    return delta_theta

Complex Numbers

An equally valid method for the representation of angle in 2D is to use complex numbers. Any complex number $z = a+ib$ can be represented in exponential form

\[ z = r\text{e}^{i\theta} \]

where $r$ is the length, or modulo, of $z$

\[ r = \sqrt{a^2 + b^2} \]

and $\theta$ is the argument of $z$, which for the first quadrant is

\[ \theta = \tan^{-1}\frac{b}{a} \]

Due to the range of the $\tan^{-1}$ function only being $[-\pi/2, \pi/2]$, the quadrant of the complex number is important to having values in the range $(-\pi,pi]$. There is a nice computational solution to this problem implemented in almost all languages, the atan2 function which works out the quadrant for us, giving an angle in our desired range.

\[ \theta = \text{atan2}(b, a) \]

Make special note of the order of arguments to the atan2 function.

Now that we understand how to convert our final complex number to a rotation, we need to construct it from the distance between two orientations. When we multiply two complex numbers together it can be thought of as rotating by an angle. This can be demonstrated by performing a multiplication with the polar form of a complex number.

\[ \begin{aligned} z &= r_1e^{i\theta_1} \times r_2e^{i\theta_2}\ & = r_1r_2e^{i\theta_1 + i\theta_2}\ & = r_1r_2e^{i(\theta_1 + \theta_2)} \end{aligned} \]

Conversely division of a complex number by another is the distance between the two angles

\[ \begin{aligned} z &= r_1e^{i\theta_1} / r_2e^{i\theta_2} \ &= \frac{r_1}{r_2}e^{i\theta_1 - i\theta_2} \ &= \frac{r_1}{r_2}e^{i(\theta_1 - \theta_2)} \end{aligned} \]

It is possible to do all our intermediate calculations using the mathematics of complex numbers. Then when we need an rotational distance the complex number representing that can be converted to a angle.

Implementing this is produces a function that looks like the one below,

import math

def complexRotation(initial, final):
    delta = final/initial
    return math.atan2(delta.imag, delta.real)

While the code is written in python, most programming languages will handle complex numbers in a similar fashion.

Orientation in 3D

Dealing with the 2D case was relatively simple. There was a single orientation to deal with, you are probably somewhat familiar with complex numbers (or at least were at some point), and it is simple to draw and visualise. In 3D things are a little more difficult. When expressing orientation using rotations in 3D, known as Euler angles, there is a phenomenon known as gimbal lock. This is where you lose a degree of freedom of expressing orientation, similar to in spherical coordinates where if you have $\theta = 0$ then all values of $\phi$ are equivalent i.e. that degree of freedom is lost. This means it is typically a bad idea to represent 3D orientation using angles. Instead we use the equivalent of complex numbers, quaternions.

Quaternions

Just like complex numbers, quaternions are a mathematical construct that allow us to represent rotations, only in three dimensions instead of two. As the name suggests quaternions consist of four values of the form

\[ a + b\mathbf{i} + c\mathbf{j} + d\mathbf{k} \]

where the relationships

\[ \mathbf{i}^2 = \mathbf{j}^2 = \mathbf{k}^2 = \mathbf{ijk} = -1 \]

hold true. These relationships are essentially giving us the rules for the operations on the quaternions.

These operations on quaternions are very similar to those on complex numbers. The conjugate of a quaternion $q$ is;

\[ q^* = a - b\mathbf{i} - c\mathbf{j} - d\mathbf{k} \]

The norm of $q$ is

\[ ||q|| = \sqrt{q^*q} = \sqrt{qq^*} = \sqrt{a^2 + b^2 + c^2 + d^2} \]

The reciprocal of $q$ is given as

\[ q^{-1} = \frac{q^*}{||a||^2} \]

Quaternions have the same properties of multiplication and division as complex numbers. Multiplying a quaternion by another is applying a rotation, and the division is finding the rotation from one quaternion to the other.

Rotational Magnitude

There are a number of methods for computing rotational magnitude using quaternions, some of which trade of speed of calculation for accuracy. For an overview of the different methods of computing rotational distance Du Q. Huynh¹ does an excellent comparison of six different methods.

Of the methods used to compute rotational magnitude $\Phi$, the one I think is most suitable for use in Molecular Dynamics is

[ \Phi = 2\ \text{arccos}(|\mathbf{q}_1 · \mathbf{q}_2|) ]

This gives a value on the range $[0, 2\pi]$ in units of radians. I think this is most suitable as it gives results in units of radians, rather than some approximate distance, while remaining simple and fast to compute. Implemented in python this function is;

import numpy

def quaternion_distance(initial, final):
    return 2*numpy.arccos(numpy.abs(numpy.dot(initial, final)))

For an optimised computation in python over an array of values, the below is the fastest implementation I could find.

import numpy
def quaternion_distance_array(initial, final):
    return 2*numpy.arccos(numpy.abs(numpy.einsum('ij,ij->i', initial, final)))

For more complicated operations using quaternions I would suggest having a look at quaternion, an open source python module by Mike Boyle which adds support for quaternions to numpy. Mike also has a C++ version of the library available.

Huynh, D. Q. (2009). Metrics for 3D rotations: Comparison and analysis. Journal of Mathematical Imaging and Vision, 35(2), 155–164. doi: 0.1007/s10851-009-0161-2 (#icanhazpdf)

Building interfaces with click

2017-08-03T00:00:00+00:00

Running computer simulations is great. You tell the computer what to do, it sits there crunching numbers for a while. Then you can come back and look at the results. The problem with this is that most simulation packages take the simulation parameters in an input file, so running a simulation becomes;

Write an input file
Run simulation
Edit input file
Run simulation
Forget what the old file looked like

Click can help us break this pattern, providing us with a simple way to pass values to scripts from the command line. Allowing us to build command line applications that take care of editing these input files for us.

What is Click?

Click is a python library that provides function decorators for passing arguments to function calls through arguments and options on the command line. Click handles all the parsing for us, including the parsing of types, and also allows for some simple validation of values.

Understanding click is probably best done through an example. Let's say we have a script to run a simulation that looks like below

#!/usr/bin/env python
# run_simulation.py

import time

def main():
    time.sleep(30)
    print('Simulation Finished!')

if __name__ == '__main__':
    main()

To change the length of time we seep for we could edit the script manually, but we want to sleep for a whole range of different times. To pass an sleep value from the command line we can modify the script as follows;

#!/usr/bin/env python
# run_simulation.py

import time
import click

@click.command()
@click.argument('sleep_time', type=int)
def main(sleep_time):
    time.sleep(sleep_time)
    print('Simulation Finished!')

if __name__ == '__main__':
    main()

The script can then be run as follows

$ python run_simulation.py 1
Simulation Finished!

which will sleep for 1 second then print the output.

Taking it up a level

Changing the time we sleep for is nice, but we are lazy and don't want to have to specify this every time. We also want to be able to specify the output string and the file the output is written to. Easy!!!

To make the sleep_time optional we can use the following like of code

@click.option('-t', '--sleep-time', type=int, default=5)

now rather than passing the time as an ordered argument, we pass it as an option as either the short -t <time> or long --sleep-time <time> way. Note that the variable name is extracted from the long form with the hyphen being converted to an underscore.

There is another improvement we can make to this sleep_time option. It is impossible to sleep for a negative time, and we don't want to wait for longer than a minute for this to run. Click allows us to set a range of suitable values

@click.option('-t', '--sleep-time', type=click.IntRange(min=0, max=60), default=5)

Now when we run the simulation

$ python run_simulation.py
Simulation Finished!
$ python run_simulation.py -t 5
Simulation Finished!
$ python run_simulation.py -t -1
Usage: run_simulation.py [OPTIONS]

Error: Invalid value for "-t" / "--sleep-time": -1 is not in the valid range of 0 to 60.
$ python run_simulation.py -t 61
Usage: run_simulation.py [OPTIONS]

Error: Invalid value for "-t" / "--sleep-time": 61 is not in the valid range of 0 to 60.

Awesome, we can stop ourselves accidentally running a simulation that takes forever and we get a helpful error message when we do. In fact click will generate help documentation for us

$ python run_simulation.py --help
Usage: run_simulation.py [OPTIONS]

Options:
  -t, --sleep-time INTEGER RANGE
  --help                          Show this message and exit.

We can help it generate this help by providing a help string to our arguments and options

@click.option('-t', '--sleep-time', type=click.IntRange(min=0, max=60), default=5,
              help='Specify the time for which the simulation will run.')

So we have the sleep time sorted, now for the output string and file. The output string is just like the sleep time, we can add another option;

@click.option('--outstring', type=str, default='Simulation Finished!',
              help='String to print to output file at the end of the simulation.')

Just make sure that when you pass a value to this option that spaces are either escaped or quoted, otherwise they will be parsed as separate arguments.

Click also has excellent support for file input and output, including reading from stdin and writing to stdout. We get this functionality by using click.File as the type like below

@click.argument('output', default='-', type=click.File('w'))

What does this all mean. There is an argument, output which has the type of a writable file. It will return a file object that we can write to directly, click handles the open/close for us. The default value '-' is stdout, like in other command line tools where you can use - to indicate read from or write to stdin/stdout, click has the same support.

Bringing it all together

With all these modifications we end up with a script that looks like this;

#!/usr/bin/env python
# run_simulation.py

import time
import click

@click.command()
@click.argument('output', default='-', type=click.File('w'))
@click.option('-t', '--sleep-time', type=click.IntRange(min=0, max=60), default=5,
              help='Specify the time for which the simulation will run.')
@click.option('--outstring', type=str, default='Simulation Finished!',
              help='String to print to OUTPUT at the end of the simulation.')
def main(output, sleep_time, outstring):
    time.sleep(sleep_time)
    output.write(outstring + '\n')

if __name__ == '__main__':
    main()

To work out how to run the script we can use the help

$ python run_simulation.py --help
Usage: run_simulation.py [OPTIONS] [OUTPUT]

Options:
 -t, --sleep-time INTEGER RANGE  Specify the time for which the simulation
                                 will run.
 --outstring TEXT                String to print to output file at the end of
                                 the simulation.
 --help                          Show this message and exit.

or just run the command

$ python run_simulation.py -t 1 --outstring "Hello World." hello.txt

We have taken a simple script that can perform a single task and with an extra 7 lines of code have turned it into a versatile tool. There is plenty more functionality available with Click, and you should check out the documentation.

Blog Posts

What's the time Azure Synapse

Installing Fedora 35 on Raspberry Pi 4

Installation Process

Standard install

Installing UEFI Firmware

Installing the DVD ISO

Why all this Effort?

A Workflow for Data Science

Idea Development

Organisation of Thoughts

Solidification of Idea

Does this code actually work

Package Creation

Conclusion

Updating Query with Soft Delete for SQLAlchemy 1.4

Why does this work?

Handling Exceptions in Numerical Python

Exploring the Solution Space

Catch-all

Hypothesis

Documentation

Rust

Conclusion

Failing Hard: How I 'lost' two years of data

The Problem

Finding the Bug

Lessons Learnt

Identifying Bugs

Rebuilding the Dataset

Conclusion

A Complete Guide to Docker on Fedora

Installation

Setting Permissions

Mounting Local Directories

Speed Through Specificity

Compiling LaTeX on Travis-CI

Choosing a LaTeX Distribution

Compiling Documents with Tectonic

Configuring Travis CI

Creating a .travis.ml file

Deploying to GitHub Releases

Conclusion

Experi: A tool for computational experiments

Abstract

Distributing a Hoomd Plugin

The Plugin

The Makefile

Packaging

Building and Distribution

Entire meta.yaml

Scraping Data from Facebook

A guide to setting up remote SSH

Installation

How do I Find My Device?

Setting the hostname

What do I do if I can't lookup my hostname?

Remembering IP addresses

SSH configuration file

Hosts file

Conclusion

Adventures in the Python Visualisation Landscape

Altair 1.2

Bokeh

Holoviews

Altair 2.0

Creating an HDF5 Bomb

The Perils of Packaging in Python

Packaging and Deploying

Versioning

Travis

Installing from Pip

Wheels

Building and Testing

Travis, Python and macOS

Frozen versions melt

When the porcelain stains

Quaternions for Orientation in MD

Orientation in 2D

Complex Numbers

Entire `meta.yaml`