Keep Python & Packages up to Date

Last updated on 2024-03-28 | Edit this page

Estimated time: 10 minutes

Overview

Questions

  • Why would a newer version of Python or a package be faster?
  • Are there any risks to updating Python and packages?
  • How can reproducibility be ensured through package upgrades?

Objectives

  • Able to explain why using the latest versions of Python and packages is beneficial.
  • Able to identify when updating is not possible due to incompatibilities.
  • Able to ensure code remains reproducible through package changes.

Introduction

It’s important to use the latest Python wherever feasible. In addition to new features and fixes, much work has been completed over the lifetime of Python 3 to improve the performance of the language.

Python 3.11 is between 10-60% faster than Python 3.10. On average, we measured a 1.25x speedup on the standard benchmark suite.

Future proposals, such as changes to the JIT and GIL will provide further improvements to performance.

Similarly, major packages with a performance focus such as NumPy and Pandas, should be kept up to date for the same reasons.

These improvements are often free, requiring minimal changes to any code (unlike the jump from Python 2 to Python 3).

Performance regressions within major packages should be considered rare, they often track performance alongside their test suites.

However, the more packages and language features your code touches, and the older the Python it currently uses, the greater chance of incompatibilities making it difficult to upgrade.

Similar to optimising, when updating it’s important to have tests in place to validate the correctness of your code before and after changes. An update to a single small dependent package could introduce a breaking change. This could cause your code to crash, or worse subtly change your results.

Updating Python & Packages

This isn’t as relevant if you’re starting from scratch. Simply make sure you’ve installed the latest Python before you start.

Key Points

  • Where feasible, the latest version of Python and packages should be used as they can include significant free improvements to the performance of your code.
  • There is a risk that updating Python or packages will not be possible to due to version incompatibilities or will require breaking changes to your code.
  • Changes to packages may impact results output by your code, ensure you have a method of validation ready prior to attempting upgrades.