Building conda packages from scratch

Overview

In this tutorial we will build a conda package for Pyinstrument, but write the required files in the conda build recipe from scratch. We also explore the elements of a conda package in more detail.

Who is this for?

This tutorial is for Windows, Mac, and Linux users who wish to generate a conda package from scratch. Prior knowledge of conda-build and conda recipes is helpful.

You should have already completed Building conda packages with conda skeleton.

Before you start

You should already have installed Miniconda or Anaconda.

Install conda-build:

conda install conda-build

It is recommended that you use the latest versions of conda and conda-build. To upgrade both packages run:

conda upgrade conda
conda upgrade conda-build

Now you are ready to start building your own conda packages.

What is a conda package?

A conda package is a package that can be installed using the conda install [packagename] command.

It includes a link to a tarball or bzipped tar archive (.tar.bz2) which contains metadata under the info/ directory, and a collection of files which are installed directly into an install prefix.

The format is identical across platforms and operating systems. During the install process, files are extracted into the install prefix, except for files in the info/ directory. Installing the files of a conda package into an environment can be thought of as changing the directory to an environment, then downloading and extracting the zip file and its dependencies – all with the single conda install [packagename] command.

About conda build recipe

Building a package requires a recipe. A conda build recipe is a flat directory which contains the following files:

  • meta.yaml contains all the metadata in the recipe. Only package/name and package/version are required fields.
  • build.sh The Unix script that installs the files for the package on OS X and Linux, and is executed using the bash command.
  • bld.bat The build script that installs the files for the package on Windows, and is executed using cmd.
  • run_test.py Optional Python test file, a test script that will run automatically if it is part of the recipe.
  • Optional: patches that will be applied to the source.
  • Other resources, which are not included in the source and cannot be generated by the build scripts.

NOTE: When you use the conda skeleton command, the first three files (meta.yaml, build.sh and bld.bat) are automatically generated for you. See the previous tutorial Building conda packages with conda skeleton for more information.

Manually edit the file meta.yaml

Let’s start by making a new directory for this tutorial named pyinstrument, and change to the new directory.

All users:

mkdir pyinstrument
cd pyinstrument

To create a new meta.yaml file, open your favorite editor. Create a new text file and insert the following information. A blank sample meta.yaml follows to make it easier to match up the information.

name pyinstrument
version “0.13.1” (or latest from https://github.com/joerick/pyinstrument/releases)
git_rev v0.13.1 (or latest from https://github.com/joerick/pyinstrument/releases)
git_url https://github.com/joerick/pyinstrument.git
imports pyinstrument
home https://github.com/joerick/pyinstrument
license BSD
license_file LICENSE
package:
  name:
  version:

source:
  git_rev:
  git_url:

requirements:
  build:
    - python
    - setuptools

  run:
    - python

test:
  imports:
    -

about:
  home:
  license:
  license_file:

When you are finished, save the file in the same pyinstrument directory as meta.yaml. It should match this meta.yaml file.

NOTE: The version number must be specified as a string to allow correct sorting and comparison.

Write the build script files build.sh and bld.bat

The other two files you need for a build are

  • build.sh shell script for Linux and OS X, and
  • bld.bat batch file for Windows.

These two build files contain all the variables such as for 32-bit or 64-bit architecture (the ARCH variable) and the build environment prefix (PREFIX). The two files build.sh and bld.bat files must be in the same directory as your meta.yaml file. In this tutorial you’re going to make both build.sh and bld.bat so that other users can build the appropriate package for their architecture.

First we’ll write the build file for Windows and then the build file for Linux and OS X.

All users, in your favorite text editor, create a new file named bld.bat and enter the text exactly as shown:

"%PYTHON%" setup.py install
if errorlevel 1 exit 1

NOTE: In bld.bat, the best practice is to to add the if errorlevel 1 exit 1 after every command so if the command fails, the build fails.

Save this new file bld.bat to the same directory where you put your meta.yaml.

All users, in your favorite text editor, create a new file named build.sh and enter the text exactly as shown:

$PYTHON setup.py install     # Python command to install the script.

Save your new build.sh file to the same directory where you put the meta.yaml file.

build.sh is run with bash -x -e (the -x makes it echo each command that is run, and the -e makes it exit whenever a command in the script returns nonzero exit status). You can revert this in the script if you need to by using the set command in build.sh.

Build and install

Now that you have your three new build files ready, you are ready to create your new package with conda build, and install the package on your local computer.

Run conda build:

conda build pyinstrument

When conda-build is finished, it displays the filename and the file’s location at the end of the build. In our case the file was saved to:

~/anaconda/conda-bld/linux-64/pyinstrument-0.13.1-py27_0.tar.bz2

NOTE: Save this path and file information for the next step. The exact path and filename will vary depending on your operating system, and whether you are using Anaconda or Miniconda. Conda-build tells you the exact location and filename.

Now install your newly built program on your local computer by using the use-local flag:

conda install --use-local pyinstrument

We know that Pyinstrument installed successfully if there are no error messages.

Convert package for use on all platforms

Now that you have built a package for your current platform with conda-build, you can convert it for use on other platforms. This is why you made the two build files, build.sh and bld.bat.

Use the conda convert command with a platform specifier from the list {osx-64,linux-32,linux-64,win-32,win-64,all}. We will use the platform specifier all, as shown:

conda convert --platform all ~/anaconda/conda-bld/linux-64/pyinstrument-0.13.1-py27_0.tar.bz2 -o outputdir/

NOTE: change your path and filename to the exact path and filename you saved in Step 6.

Optional: How to use PyPI as the source instead of GitHub

What if you wanted to use PyPI or another repository instead of GitHub? There is little difference to conda-build between building from Git versus building from a tarball on a repository like PyPI. Because the same source is hosted on PyPI and GitHub, you can easily find a script on PyPI instead of GitHub. Simply replace this source section:

git_rev: v0.13.1
git_url: https://github.com/joerick/pyinstrument.git

With the following:

fn: pyinstrument-0.13.1.tar.gz
md5: e347036acc50720c0903dc2221b2605d
url: https://pypi.python.org/packages/source/p/pyinstrument/pyinstrument-0.13.1.tar.gz

NOTE: The md5 is found on the PyPI Pyinstrument page.

Optional: Upload new packages to Anaconda.org

After converting your files for use on other platforms, you may choose to upload your files to Anaconda.org, formerly known as binstar.org. It only takes a minute to do if you have a free Anaconda.org account.

If you haven’t already, open a free Anaconda.org account and record your new username and password.

Next, in your terminal window, run conda install anaconda-client and enter your new Anaconda.org username and password.

Again in your terminal window, log into your Anaconda.org account with the command:

anaconda login

And upload your package to Anaconda.org:

anaconda upload ~/miniconda/conda-bld/linux-64/pyinstrument-0.12-py27_0.tar.bz

NOTE: Change your path and filename to the exact path and filename you saved in Step 6.

TIP: To save time, you can set conda to always automatically upload a successful build to Anaconda.org with the command: conda config --set anaconda_upload yes

More resources

See more information about all the possible values that can go in the meta.yaml file on the Conda build recipe reference page.

Command reference