Flexible conda package dependencies on Cloud Foundry

The official Python Cloud Foundry buildpack has support for conda environments using the environment.yml file. This provides a lot of flexibility for Python (and other) dependencies and helps you to use packages from other public and private sources including your own locally built ones.

Using other conda channels

One thing this allows you to do is reference other channels such as conda-forge, and get packages from there instead of the standard Anaconda provided default channel.

We can specify conda-forge as a channel in our environment.yml file and then install a package like prettytable. This enables a lot of flexibility to access the latest and greatest conda packages from anywhere.

name: myenv
channels:
  - conda-forge
dependencies:
  - python=3.6
  - flask
  - gunicorn
  - prettytable

Vendoring conda packages with your app

However, what happens if you need a particular package but you can’t access the public internet from your CF installation? In the Python buildpack, the pip-based approach supports the concept of “vendoring”, providing dependency packages alongside your application code when uploading to CF.

You can do this with conda packages as well, by pointing to a local conda channel created in your app directory.

If you are packaging your own library, there are lots of tutorials on how to build a conda package for it, depending on the complexity. If you need an existing third-party library you can use any of the existing conda packages from anaconda.org or conda-forge.

The next step is to create a local conda channel in your app directory. This is essentially a series of directories for each architecture you need (osx_64, linux, win, and noarch for pure Python packages) which you tell conda about by creating a channel index for each architecture directory:

conda index ./vendor/noarch

Once you have this channel in place in your app, you need to tell the CF buildpack about it by providing a local file path to the channel.

During the staging phase of CF app deployment, the buildpack installs your app and all its dependencies in a temporary container and the absolute path is /tmp/app. So we can provide conda with a reference to our local channel, knowing it will be located at /tmp/app/vendor.

name: myenv
channels:
  - /tmp/app/vendor
dependencies:
  - python=3.6
  - flask
  - gunicorn
  - mypkg

When we cf push our local conda package file will be uploaded along with our app, and the Python buildpack will install it along with any other dependencies required. In this way you can mix local and public packages easily.

If you want to see this in action, I’ve created a very simple example package, and a corresponding CF app that installs this package from a local channel, as well as getting prettytables from conda-forge. For more details on using Python on Cloud Foundry, take a look at my tutorial.