Starting new modern Python projects
Python ecosystem is thriving and there are many useful libraries that we can use in our projects for managing virtual environments, dependencies, formatting source code, debugging, testing, static code analysis and so on. So how would a modern Python project look like? And how to setup such project with one simple command?
In this article, I will present a modern Python project template that we can use for starting new projects and the reasoning behind it.
Let’s first go through the basic project dependencies and configuration files that we might want to have in every new project:
Dependency management & virtualenv
I used to use Pipenv before, but Poetry is now my default choice for managing Python virtual environments and dependencies. That’s because it it can be used both for applications as well as for creating Python library packages while Pipenv can be used for applications only. Poetry is also much more actively developed.
To use Poetry, we will need a
pyproject.toml file where all our dependencies will be defined, together with basic project metadata such as a project name or description. It will look something like this:
[tool.poetry] name = "project" version = "0.1.0" description = "Project description" authors = ["John Snow <a href="mailto:email@example.com">firstname.lastname@example.org</a>"] [tool.poetry.dependencies] python = "^3.7" dependency = "version" ... [tool.poetry.dev-dependencies] devdependency = "version" ... [build-system] requires = ["poetry>=0.12"] build-backend = "poetry.masonry.api"
Once our project is properly described using this file, Poetry will be able to create a virtual environment for the project and install all of the specified dependencies in it with a simple command
(Poetry can also create this file on its own and can add or remove dependencies interactively, but we want to define the file ourselves so that we can use it as a template.)
There are many ways to debug Python applications, however there are two packages that can be useful to install in any project: PySnooper for automatic print()-ing to the console and stackprinter for nicer exception messages. I already described how to use them in the article Debugging Python programs.
Pytest has been my go-to testing library since I started with Python. It is basically the Python standard for automated testing. It is easy to use and has useful plugins available. One plugin that I like to always install is pytest-sugar for nicer console output. It is also very useful to leverage Pytest configuration file
pytest.ini and specify the test folder:
[pytest] testpaths = tests
This will make running tests faster in larger projects since Pytest doesn’t have to scan all folders in the project.
I now use Black basically everywhere I can. It is a no-configuration code formatting that will ensure that the code looks neat without the need for programmers to pay any attention to how they format their code. Being opinionated, is is also a bit controversial. In the end, a readable code is important and an automatic formatter is the best way to ensure the code style is consistent across the entire codebase. You can check Black Playground to try it out before adopting it.
Static code analysis
Flake8 is a popular checker to enforce various code rules in a Python codebase. Its plugin architecture allows us to pick and choose what rules should be enforced. I use this particular configuration to avoid conflicts with Black:
[flake8] ignore = E203, E266, E501, W503, F403, F401 max-line-length = 79 max-complexity = 18 select = B,C,E,F,W,T4,B9
Other code analysis tools that we will include in our template are MyPy for checking Python types and Bandit for finding security issues.
To help with source version control, it is useful to have a basic .gitignore file in place so as not to commit cache directories or .env files:
__pycache__ .mypy_cache/ .pytest_cache/ .env
To automatically check and format code before every commit, pre-commit is a perfect tool. To define what to run we will need to create
.pre-commit-config.yaml file. In our project template, we will run Black and Flake8 before every commit:
- repo: https://github.com/psf/black
- id: black
- repo: https://gitlab.com/pycqa/flake8
- id: flake8
While it may seem like something project-specific, I reckon that having a simple command line interface to various tasks is useful in almost every project. That’s why I recommend installing one of the Python command line libraries right away.
For this template, I picked Typer for declaring CLIs with the help of static types and rich, a library for rich console outputs.
Since every project should have a README file, it makes sense to include a blank
README.md file in the project template.
Creating a cookiecutter template
So far we have selected all the tools and configuration that we want to have prepared in our new projects and now we need to put it all together in a project template.
Cookiecutter is a great tool for creating such project templates. We just need to:
- define variables that are needed for the template in cookiecutter.json
- create all files and folders we want to have in the project with the help of Jinja2 template tags (we can use in file and folder names or anywhere inside the files)
- create pre- and post- creation hooks in case we need to run some code before or after our folder structure is created
Once the template is created, we can create new projects based on it with one simple command. Cookiecutter will go through all defined variables and interactively ask us to fill in the values for our new project.
Based on the requirements discussed, I have created a python-new-project cookiecutter template. Besides specifying all the variables and configuration files, a post-creation hook is necessary to prepare the project for us in
poetry install && git init && poetry run pre-commit install
- tell Poetry to create a virtual environment for the project and install all dependencies
- initialize a new git repository
- install pre-commit git hooks
Using the template
Follow the installation instructions in the python-new-project repo to install Python, Git, cookiecutter and poetry. These are necessary requirements.
Then to create a new project based on this template, we can just run:
Last updated on 16.10.2020.