Python relative import and dynamic file paths

Python code showing import of module from relative directory

How to import from other folders in Python, and how to read and write to parent directories using dynamic and relative file paths

Links to topics on this page

There are different methods to use in Python depending on whether you are importing modules from other folders, or reading and writing to parent directories.

How to organise folders to import modules in Python

The simplest way to enable importing of modules from files in other directories in Python is to organise your folders so that you are only accessing files and modules from child directories.

There are ways to enable importing modules from parent directories, but these aren’t necessarily easy or even advisable. A simple effective solution is to ensure that any files / functions that you want to import from are in child directories.

So in the example below if I place all my functions in the functions folder, and my output scripts in a parent directory, then I will be able to import the functions – see the example below.

├── my_app
     ├── data ── raw_data_file.csv
     └── outputs ── chart_outputs.qmd
	 └── functions ── get_file_path.py, interactive_chart.py
#=== CHART_OUTPUTS.QMD script ===#
# import from the get_file_path.py script which is in the functions folder (child) 
from functions.get_file_path import get_data_file_path

# use the function to read the csv file from the PARENT directory
def shipping_chart():
    file_path = f"{get_data_file_path()}raw_data_file.csv"

    shipping = pd.read_csv(file_path)

To import from a child folder in Python

from functions.get_file_path import get_data_file_path

In the example above:

  • I can refer to just functions because it is in a child directory of outputs;
  • the ' . ' denotes that the file get_file_path.py is within the functions folder.

How to read and write to parent directories in Python using a dynamic file path

In contrast to using the import function, when it comes to reading and writing files it is possible to access parent directories.

In the scenario above I could simply have added the data folder into the outputs folder. However, my preference is to keep the data folder as a sub-folder of the root my_app folder. It’s good practice to keep data files in a separate folder. The location can be added to the .gitignore file to avoid data being inadvertently published to the remote repository. It also makes it easier to locate all the data files.

Below is an example of a function which creates a dynamic file path depending on where in the app folder tree the function is run from. For example, you’ll find that jupyter notebook files .ipynb will run from a different working directory to python .py scripts even though they appear to be located in the same directory. The function will be transferable to other users because it extracts the folder names within the app tree without having to refer to the path higher up the user’s folder structure.

The function does involve hard-coding in the directory names, and makes use of the os package.

The code below refers to our folder structure from above. The function can be called for Python relative import to read and write data files in the data folder.

#=== GET_FILE_PATH.py script ===#
import os


def get_data_file_path():
    """Gets the file path of the data directory
    Args:
    cwd - gets the file path of the current working directory
    parent - the folder name of the parent directory
    """

    cwd = os.getcwd()
    parent = os.path.basename(os.path.dirname(cwd))
    if os.path.basename(cwd) == "my_app":
        return f"{cwd}/data/"
    elif parent == "outputs":
        return f"{os.path.dirname(os.path.dirname(cwd))}/data/"
    elif parent == "my_app":
        return f"{os.path.dirname(cwd)}/data/"
    return "Error - check file path"

Lightbulb icon demonstrating good idea, used to highlight solution in excelquick.com websiteExplanation of the code

  1. cwd = get the current working directory
  2. parent = the folder name of the parent directory
  3. If the name of the current working directory (fetched using basename()) is the root folder my_app then append /data/ to the root file path.
  4. If parent is the outputs folder, then move up one step in the directory tree by adding an additional nested dirname() call.
  5. If parent is the root folder my_app, then return the full path to my_app appended with /data/.

os.getcwd()

Gets the full path of the current working directory.

os.path.dirname()

Gets the full path of the parent directory. dirname() can be nested so that each call moves up one level in the folder tree. os.path.dirname(os.path.dirname(os.getcwd())

os.path.basename()

Gets just the folder name of the target path. For example, if combined with os.getcwd() will return just the name of the current working directory. os.path.basename(os.getcwd())

Call the function to read a csv from the data folder

The code snippet below from the interactive_chart.py file calls the get_data_file_path() function to get the relative path to the data folder, to open the raw_data_file.csv.

The variable file_path is created from a Python f string combining the output from the get_data_file_path() function, plus the name of the target csv file.

#=== INTERACTIVE_CHART.py script ===#
from functions.get_file_path import get_data_file_path


def shipping_chart():
    file_path = f"{get_data_file_path()}raw_data_file.csv"
    print(file_path)
    shipping = pd.read_csv(file_path)

Python relative import – read and write to files using a static file path

The examples below show how to use static relative file references to access files in child and parent folders.

To read from a child directory using a static file path:

# opens the file in the sub-folder /files/
# the '.' denotes the current directory
open("./files/my_text_file", "r")

To read from a file in a parent directory

# reads the file from the parent directory
import pandas as pd
df = pd.read_csv("../raw_data_file.csv")