Software development and beyond

Starting new modern Python projects

Python ecosystem is thriving and there are many useful libraries that we can use in our projects for managing virtual environments, dependencies, formatting source code, debugging, testing, static code analysis and so on. So how would a modern Python project look like? And how to setup such project with one simple command?

In this article, I will present a modern Python project template that we can use for starting new projects and the reasoning behind it.

Let’s first go through the basic project dependencies and configuration files that we might want to have in every new project:

Dependency management & virtualenv

I used to use Pipenv before, but Poetry is now my default choice for managing Python virtual environments and dependencies. That’s because it it can be used both for applications as well as for creating Python library packages while Pipenv can be used for applications only. Poetry is also much more actively developed.

To use Poetry, we will need a pyproject.toml file where all our dependencies will be defined, together with basic project metadata such as a project name or description. It will look something like this:

[tool.poetry]
name = "project"
version = "0.1.0"
description = "Project description"
authors = ["John Snow <a href="mailto:jsnow@example.com">jsnow@example.com</a>"]

[tool.poetry.dependencies]
python = "^3.7"
dependency = "version"
...

[tool.poetry.dev-dependencies]
devdependency = "version"
...

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Once our project is properly described using this file, Poetry will be able to create a virtual environment for the project and install all of the specified dependencies in it with a simple command poetry install.

(Poetry can also create this file on its own and can add or remove dependencies interactively, but we want to define the file ourselves so that we can use it as a template.)

Debugging

There are many ways to debug Python applications, however there are two packages that can be useful to install in any project: PySnooper for automatic print()-ing to the console and stackprinter for nicer exception messages. I already described how to use them in the article Debugging Python programs.

Testing

Pytest has been my go-to testing library since I started with Python. It is basically the Python standard for automated testing. It is easy to use and has useful plugins available. One plugin that I like to always install is pytest-sugar for nicer console output. It is also very useful to leverage Pytest configuration file pytest.ini and specify the test folder:

[pytest]
testpaths = tests

This will make running tests faster in larger projects since Pytest doesn’t have to scan all folders in the project.

Code formatting

I now use Black basically everywhere I can. It is a no-configuration code formatting that will ensure that the code looks neat without the need for programmers to pay any attention to how they format their code. Being opinionated, is is also a bit controversial. In the end, a readable code is important and an automatic formatter is the best way to ensure the code style is consistent across the entire codebase. You can check Black Playground to try it out before adopting it.

Static code analysis

Flake8 is a popular checker to enforce various code rules in a Python codebase. Its plugin architecture allows us to pick and choose what rules should be enforced. I use this particular configuration to avoid conflicts with Black:

[flake8]
ignore = E203, E266, E501, W503, F403, F401
max-line-length = 79
max-complexity = 18
select = B,C,E,F,W,T4,B9

Other code analysis tools that we will include in our template are MyPy for checking Python types and Bandit for finding security issues.

Git

To help with source version control, it is useful to have a basic .gitignore file in place so as not to commit cache directories or .env files:

__pycache__
.mypy_cache/
.pytest_cache/
.env

pre-commit

To automatically check and format code before every commit, pre-commit is a perfect tool. To define what to run we will need to create .pre-commit-config.yaml file. In our project template, we will run Black and Flake8 before every commit:

repos:
- repo: https://github.com/psf/black
  rev: 20.8b1
  hooks:
  - id: black
- repo: https://gitlab.com/pycqa/flake8
  rev: 3.8.4
  hooks:
  - id: flake8

Command-line interfaces

While it may seem like something project-specific, I reckon that having a simple command line interface to various tasks is useful in almost every project. That’s why I recommend installing one of the Python command line libraries right away.

For this template, I picked Typer for declaring CLIs with the help of static types and rich, a library for rich console outputs.

README

Since every project should have a README file, it makes sense to include a blank README.md file in the project template.

Creating a cookiecutter template

So far we have selected all the tools and configuration that we want to have prepared in our new projects and now we need to put it all together in a project template.

Cookiecutter is a great tool for creating such project templates. We just need to:

Once the template is created, we can create new projects based on it with one simple command. Cookiecutter will go through all defined variables and interactively ask us to fill in the values for our new project.

Based on the requirements discussed, I have created a python-new-project cookiecutter template. Besides specifying all the variables and configuration files, a post-creation hook is necessary to prepare the project for us in hooks/post_gen_project.sh:

#!/bin/bash
poetry install && git init && poetry run pre-commit install

This will:

Using the template

Follow the installation instructions in the python-new-project repo to install Python, Git, cookiecutter and poetry. These are necessary requirements.

Then to create a new project based on this template, we can just run:

cookiecutter https://github.com/stribny/python-new-project

Last updated on 16.10.2020.

python