Library

pyflyby: Utilities and Extensions for Improving Productivity in Python

Pyflyby, developed by the D. E. Shaw group, offers developers a number of innovative utilities to mitigate the risk of interruption and improve efficiency when working in IPython and Jupyter. In this piece, originally published as a Quansight Labs blog post, Quansight introduces pyflyby and provides examples of its functions and features.

—

Few things hinder productivity more than interruption. A notification, random realization, or unrelated error can derail one's train of thought when deep in a complex analysis – a frustrating experience.

In the software development context, forgetting to import a statement in an interactive Jupyter session is such an experience. This can be especially frustrating when using typical abbreviations, like np, pd, plt, where the meaning is obvious to the human reader, but not to the computer. The time-to-first-plot, and ability to quickly cleanup one's notebook afterward are critical to an enjoyable and efficient workflow.

In this blogpost we present pyflyby, a project and an extension to IPython and JupyterLab, that, among many things, automatically inserts imports and tidies Python files and notebooks.

What is pyflyby?

Pyflyby is a set of tools designed to improve interactive and non-interactive workflows in Python. Pyflyby provides a number of utilities and extensions aimed at making day-to-day work in Python faster and simpler.

gif of pyflyby in action, IPython is opened, typing cos(arange(10)) automatically import arange and cos from numpy before executing the code.

Autoimport

One of pyflyby's key capabilities is automatic importing of commonly used modules and objects, leading to simpler, faster, and less disruptive coding. In a new session, for example, one can type: 

sin(arange(10))

Pyflyby will then hook into the execution mechanism of Python, execute the correct import, and confirm with a clear message: 

[PYFLYBY] from numpy import arange
[PYFLYBY] from numpy import sin

Pyflyby will also do so when running a command line file via the py executable replacing python.

With the jupyterlab-pyflyby extension, imports will be executed and inserted in the first cell of one's notebook:

pyflyby-jupyterlab animation, executing the second cell with missing imports still import those and insert them in the first cell

tidy-import

In addition to the extension that can seamlessly import the right libraries while exploring, another pyflyby feature helps to maintain scripts and notebooks with explicit and correct imports.

Pyflyby displays the tidy-import command line tool to gather, insert, and format imports in Python files. This is similar to tools such as black and isort, but with different styling options and with the ability to infer missing imports.

tidy-import includes the imports to pandas and matplotlib, as in the example below, and queries whether to update the file: 

$ tidy-imports example.py
[PYFLYBY] example.py: added 'import pandas as pd'
[PYFLYBY] example.py: added 'from matplotlib import pyplot'
--- example.py  2021-03-08 10:33:04.000000000 -0800
+++ example.py  2021-03-08 10:33:18.000000000 -0800
@@ -1,2 +1,7 @@
+from   matplotlib               import pyplot
+import pandas as pd
+
 data = pd.read_csv("./data/base-pop-2015.csv")
 pyplot.plot(data.population)

Replace example.py? [y/N]

Other utilities

Pyflyby contains a number of other utilities to improve the efficiency of manipulating and executing Python code. (Please refer to pyflyby's README for additional information.)

py is one such example. It is a flexible tool that can be used either to start IPython or to execute commands quickly from the developer's shell without the need for imports. It supports a range of syntax options, allowing for quick calculation and graph plotting.

  • Without any parameters, py will start IPython with the pyflyby extension activated.
  • With space-separated arguments, py will attempt to interpret each argument as a Python function call with the right imports:
$ py np.random.normal 0 1
[PYFLYBY] import numpy as np
[PYFLYBY np.random.normal(0, 1)
-0.027577422117386
  • When more control over values is necessary, py will run a Python expression:
$ py 'plot(scipy.stats.norm.pdf(linspace(-5, 5), 0, 1))'
[PYFLYBY] from numpy import linspace
[PYFLYBY] from matplotlib.pyplot import plot
[PYFLYBY] import scipy.stats
[PYFLYBY] plot(scipy.stats.norm.pdf(linspace(-5, 5), 0, 1))
[<matplotlib.lines.Line2D object at 0x132981940>]

using pyflyby from bash to plot with matplotlib with above snippet, at matplotlib widow is open and show the plot.

find-import , another utility available in pyflyby, can be deployed to find a particular function across many libraries by returning the relevant import. For example: 

$ find-import norm
from scipy.stats.distributions import norm

Notes on pyflyby codebase

The Pyflyby codebase provides a window into advanced data structures and programming paradigms. Its use of modules and programming concepts is unusual relative to those found in more classical data science-focused libraries. For example:

  • Pyflyby will conduct non-trivial manipulation of the Python Abstract Syntax Tree (AST), which represents code written in tree form. Pyflyby uses AST to find and insert missing imports, and it does so even as AST's exact representation changes with almost every minor release.

  • Pyflyby demonstrates the use of Aspect-Oriented programming, which highlights the flexibility of the Python programming model.

Conclusion

Pyflyby's utilities are designed to improve developer efficiency and materially reduce the impact of interruption on productivity. Its value, however, is broader than that. Pyflyby expands what one is capable of doing within the Python ecosystem and has helped identify a number of limitations and bugs in Python and IPython over time.

How to get pyflyby

Pyflyby is available on GitHub or, for terminal IPython users: 

$ pip install pyflyby 
$ py pyflyby.install_in_ipython_config_file

JupyterLab users can also install the JupyterLab Extension, which is notebook-aware and enables even more features.

Acknowledgements

Pyflyby was created by Karl Chen and is supported by the D. E. Shaw group in collaboration with Quansight Labs.

Disclosures

The contents of this post and any third-party content that is linked to or included in this post (“Linked Content”) are collectively referred to as the “Content.”

The Content is provided for informational purposes only. It does not convey an offer of any type and is not intended to be, and should not be construed as, an offer to sell, or the solicitation of an offer to buy, any interest in any entity, investment, or investment vehicle. The Content does not constitute investment advice. No assurances can be given that any aims, assumptions, expectations, and/or goals expressed or implied in the Content were or will be realized, or that the activities described in the Content have continued or will continue at all or in the same manner as is described.

The D. E. Shaw group, its affiliates and members, and any party or person associated with or acting on behalf of the foregoing shall have no liability for any errors (as a result of negligence or otherwise, to the fullest extent permitted by law in the absence of fraud) in the Content or its production or for the consequences of any party’s reliance on the Content. The D. E. Shaw group does not endorse any information or beliefs discussed in any Linked Content and makes no representation as to the accuracy or adequacy of any Linked Content.

All trademarks, logos, information, and photos are owned by D. E. Shaw & Co., L.P. or its affiliates or used with permission of the applicable third party. The reproduction or retransmission of the Content (other than Linked Content) is prohibited without the prior written consent of D. E. Shaw & Co., L.P.