Python, virtual environments and Robot Framework - Introduction to virtual environments

02 Feb 2021

Why an introduction?

I consider it good practice to reflect on a new technique, pattern or (in this case) tool, before actually using it. There are several benefits in doing so.

For one, we will be better equipped to understand, interpret, analyze and remove errors or other problems that we might encounter, when we understand the inner machinations of what we’re working with. We will also utilize our tool (or pattern, etc.) both more efficiently and effectively when we know which problem(s) it has been designed to solve and know how it solves this (these) problem(s). We will then also utilize its potential to a greater extent, because we know what it has to offer.

Apart from such practical benefits, I believe that as testers we should have a natural curiosity. A curiosity that drives us into investigating how stuff works and why it works that way.

Let’s therefore first have a closer look at virtual environments, before putting them to use:

1. What is a virtual environment?
2. What does a virtual environment look like?
3. Why virtual environments?
4. Next steps.

Note that this is just a first, quick peek under the hood: we will keep on peeking even in the remaining parts of this series of articles, where we will be working with virtual environments.

What is a virtual environment?

A virtual environment is a self-contained, isolated Python installation that (as such) is independent from any global/system Python installations (and their configurations) as well as from any other virtual Python environments (and their configurations). Within such an environment, we can create an ecosystem of third party libraries/packages that will be specific for (and dedicated to) that environment.^[1]

Installed libraries/packages will only be accessible to the Python interpreter of that specific environment. Vice versa, no packages from outside the environment will be accessible to that interpreter.^[2]

Once you have created an environment that fulfills the requirements of your specific development project, you can then proceed and bind the environment to that project. Or to multiple projects that have identical requirements.

What does a virtual environment look like?

Creating a virtual environment generates a (relatively) small directory structure. Depending on the tool (and depending on the parameters you apply when creating the environment) there are some variations within this structure. For instance, this is the folder structure for one of my virtual environments, created with the Python module virtualenv:

But regardless the tool, all virtual environments basically consist of three components:

1. The Python interpreter (usually with a version that you specified).
   
   This might be a copy of the system interpreter that you based your environment on or a so-called 'symlink' to that system interpreter. Again, this will depend on the tool used and/or on the parameters you select upon creating the env.


2. The Python \Scripts folder (\bin on Unix/Linux).
   
   This folder is part of every regular Python installation and also of every virtual environment. It is used by third-party modules/packages (which themselves get installed into the \Lib\site-packages folder; see below) to store scripts and/or executables that are associated with them.
   
   As this folder (together with the installation's root folder) gets dynamically added to the operating system's PATH variable (more on this in the third part), you can run these scripts/executables from the command line.
   
   For instance, if you were to invoke:
   
   pip install robotframework
   
   from the Windows command line within an active virtual environment, then the excutable pip.exe in the \Scripts folder would be called, which acts as a wrapper for the pip module in \Lib\site-packages. The same goes for the script ride.py, which is called when you start RIDE (Robot Framework Integrated Development Environment) from the command line and which, in turn, acts as an entry point to the RIDE package in the \Lib\site-packages folder.

Basically, this constellation is there to make sure we can access Python programs through the standard Windows shells, and not having to use the interactive Python interpreter. See this example of the \Scripts folder of my virtual environment:


You can see that the Python executable is also located here, which, in my case, is a copy of the system Python interpreter and not a symlink.


3. The Python \Lib\site-packages folder.
   
   This folder is used for third-party libraries/packages that are installed. For instance, Robot Framework, RIDE and all installed RF test libraries will reside in this folder.

See again my own virtual environment as an example:

So the three basic types of artifacts, that every virtual Python environment is composed of, are:

The scripts/executables and the packages are closely related, as was just explained.

These three basic components are, as an independent whole, isolated from any other (virtual or system) Python installation.

Why virtual environments?

Creating a virtual Python environment for your test automation project (and for any kind of development project, for that matter) can be beneficial for many reasons. Among those reasons are:

• Avoiding dependency hell.
  
  A project may contain dependencies towards specific versions of Python and/or third party libraries. This may conflict with dependencies within one or more of your other projects. For instance, it may conflict with a project that might require the same Python version, but at the same time requires divergent versions of the same third-party libraries.


• Keeping your system/global/base Python installation clean.
  
  You won't have to clutter your global \Lib\site-packages and \Scripts folders when installing third-party libraries and tools for your various projects. Usually an average project will require quite a few packages to be installed. Consequently, the mentioned folders will fill up fast and become rather hulking and unwieldy.


• Being able to easily share a specific Python environment.
  
  For instance, in order to facilitate that every contributing team member uses the same Python version and tool stack/ecosystem. As we will see in the remainder of this post, utilizing virtual environments makes sharing these environments very easy.


• Being able to restore your environment, in case of problems related to that environment.
  
  Let's say, for instance, that you updated multiple libraries in one, single (and bold) step and now find yourself overwhelmed by an avalanche of (mostly obscure) exceptions. Wouldn't it be nice to be able to quickly revert to the original (or last known functioning) environment and try your updates again, but now in a less rash fashion? Later on we will see just how effortless this can be done with a virtual environment!

Next steps.

Now that we have some idea of what a virtual environment is, let’s create one in the next part.

[1] Although you can configure your environments to share certain artifacts. (back)

[2] Again, depending on how you rig your environment(s), you could share artifacts between environments. (back)