Working with Python Configuration Files: Tutorial & Best Practices

What Are Configuration Files in Python? 

Configuration files in Python are components that store settings and preferences for applications, enabling developers to manage how software operates under different environments without changing the code. They typically contain key-value pairs which represent settings that can be read and modified.

By keeping configuration settings in separate files, developers can enhance Python code maintainability, avoid hard-coding values, and provide a flexible way to manage different application settings for development, testing, and production environments.
This is part of a series of articles about environment variables

Why Are Configuration Files in Python Needed? 

Configuration files are crucial in Python for several reasons: 

• Abstracting configuration details away from the code, which makes the software more adaptable to different environments. For example, database credentials, API keys, and other environment-specific settings can be stored in configuration files to manage the differences between development, staging, and production setups easily.
• Easier updates and changes to configuration settings without risking the integrity of the core application code.
• Improving security, because passwords and API tokens can be segregated from the source code, reducing the risk of accidental exposure. 

Important note: Storing credentials, and other sensitive data, in a configuration file is safer than saving them in your source code, but is also not secure. Anyone with access to your environment will be able to view and compromise the information. This is why it’s important to store sensitive data using a secrets management or configuration management system (like our very own Configu), which can encrypt and control access to sensitive information.

Related content: Read our guide to Python environment variables

Types of Configuration Files in Python 

INI Files

INI files are among the oldest formats used for configuration in Python applications. They are text files with a structure composed of sections, each containing key-value pairs. Libraries like ConfigParser in Python make it straightforward to read, write, and modify INI files, making them an accessible option for basic configurations.

However, INI files have an important limitation: They lack support for data structures like lists and dictionaries. This can be cumbersome when the application requires a more complex configuration setup.

YAML

YAML (Yet Another Markup Language) is another widely-used configuration file format due to its readability and flexibility. YAML supports nested structures such as lists and dictionaries, making it a choice for more complex configurations. The PyYAML library in Python makes it convenient to work with YAML files.

YAML’s primary advantage is its human-readable format, which is easier to edit directly. However, its flexibility can also be a drawback, as it allows for multiple ways to represent the same data, which can lead to inconsistencies.

JSON

JSON (JavaScript Object Notation) is a ubiquitous format for configuration files, favored for its lightweight and easy-to-parse structure. JSON supports nested structures similar to YAML, making it suitable for intricate configuration setups. The built-in json module in Python simplifies reading and writing JSON files.

One downside of JSON is that it is less readable compared to YAML, especially for complex configurations. Despite this, its widespread use and compatibility with other data interchange formats make it a popular choice.

Tips From the Expert

In my experience, here are tips that can help you better manage and optimize Python configuration files:

1. Implement  configuration schema validation: Use libraries like Pydantic or jsonschema to define and enforce a schema for your configuration files. This ensures that your configuration files adhere to the expected structure and data types, preventing runtime errors.
2. Automate configuration file management: Use tools like Ansible, Puppet, or Chef to automate the deployment and management of configuration files across different environments. This helps maintain consistency and reduces the risk of manual errors.
3. Leverage templating for configuration files: Use templating engines like Jinja2 to create dynamic configuration files. This allows you to generate configuration files based on different parameters or environment variables, enhancing flexibility and reusability.
4. Encrypt configuration files: Use configuration management tools like Configu to encrypt and secure your configuration files. This adds a layer of security, especially when dealing with sensitive information.

Use a layered configuration approach: Combine multiple configuration sources, such as default settings, environment-specific overrides, and local development configurations. Libraries like Dynaconf or ConfigObj support layered configurations, allowing for more flexible and manageable settings.

Ran Cohen

CTO & Co-Founder, Configu

Before co-founding Configu, Ran was a full stack developer at Testim.io and previously served in an elite cybersecurity unit in the Israeli Defense Forces.

What Is ConfigParser?

ConfigParser is a module in Python’s standard library that provides a way to handle configuration files with a structure composed of sections, each containing key-value pairs. It can read, write, and modify INI files, making it a practical choice for managing simple configurations in Python applications. 

ConfigParser enables developers to easily separate configuration settings from the main codebase, improving maintainability and flexibility. It supports interpolation, where values in the configuration file can reference other values. This can simplify managing related settings. Additionally, ConfigParser allows for the use of default values that can be overridden in specific sections, providing a mechanism for layered configurations. 

Tutorial: Managing Python Configuration Files

Creating Config File in Python

To create a configuration file in Python, you can use the configparser module. This module allows you to create and manipulate .ini files, which are text files with a simple structure composed of sections and key-value pairs.

Here’s how to create a config file with two sections: USERINFO and DBCONFIG.

from configparser import ConfigParser

# Create a ConfigParser object
config_object = ConfigParser()

# Add user information to the config object
config_object["USERINFO"] = {
    "admin": "Mark Taylor",
    "loginid": "marktaylor",
    "password": "abcd1234"
}

# Add server configuration to the config object
config_object["DBCONFIG"] = {
"db-url": "mydb.com/connection-string",
"port": "3306",
"ipaddr": "100.10.10.1"
}

# Write the configuration to a file named 'config.ini' with open('config.ini', 'w') as conf: config_object.write(conf)

After running this code, a file named config.ini will be created in your working directory with the following content:

[USERINFO]
admin = Mark Taylor
password = abcd1234
loginid = marktaylor

[DBCONFIG]
db-url = mydb.com/connection-string
port = 3306
ipaddr = 100.10.10.1

Reading a Key from Config File

To read values from the configuration file, you can use the read method of the ConfigParser object. This allows you to access specific settings by their section and key names.

Here’s an example of reading the password from the USERINFO section:

from configparser import ConfigParser

# Create a ConfigParser object
config_object = ConfigParser()

# Read the configuration from the 'config.ini' file
config_object.read("config.ini")

# Access the USERINFO section
userinfo = config_object["USERINFO"]

# Print the password from the USERINFO section
print("Password is {}".format(userinfo["password"]))

Output:

Password is abcd1234

Updating a Key in Config File

If you need to update a value in the configuration file, you can modify the desired key and write the changes back to the file.

Here’s how to update the password for the marktaylor user:

from configparser import ConfigParser

# Create a ConfigParser object
config_object = ConfigParser()

# Read the configuration from the 'config.ini' file
config_object.read("config.ini")

# Access the USERINFO section
userinfo = config_object["USERINFO"]

# Update the password in the USERINFO section
userinfo["password"] = "abrakadabra"

# Write the updated configuration back to the 'config.ini' file
with open('config.ini', 'w') as conf:
    config_object.write(conf)

After running this code, the config.ini file will be updated with the new password:

[USERINFO]
admin = Mark Taylor
password = newpassword
loginid = marktaylor

[DBCONFIG]
db-url = mydb.com/connection-string
port = 3306
ipaddr = 100.10.10.1

By using these methods, you can easily create, read, and update configuration files in Python, enabling you to manage application settings efficiently.

Best Practices for Managing Configuration Files in Python 

Keep Configurations Maintainable and Readable

Organize settings logically into sections and use descriptive key names to ensure clarity. Avoid inline comments in formats that do not support them, like JSON. Instead, use documentation to explain complex configuration settings.

Additionally, keeping configuration files small and modular helps. Split large configuration files into smaller, purpose-specific ones to ease management. For instance, have separate files for database settings, API keys, and general application settings. This approach enhances readability and maintainability.

Version and Document Configuration Changes

Proper versioning and documentation of configuration files are essential for tracking changes and ensuring consistency. Use version control systems like Git to manage configuration files. Commit changes with detailed messages explaining why the changes were made. This practice helps in understanding the history and reasoning behind configuration modifications.

Documenting configuration settings and their purpose within the codebase or using a dedicated documentation file enhances transparency. Include details about each key, its acceptable values, and the impact of changing it. This ensures that new team members can quickly understand and manage configurations effectively.

Test and Validate Configuration Files

Testing and validating configuration files are crucial steps to ensure application stability. Implement automated tests to check the presence of required keys and validate their values. Use testing frameworks like pytest to create unit tests for configuration files, verifying that they meet defined requirements.

Additionally, use schema validation libraries like voluptuous or cerberus to enforce structure and data type constraints on configuration files. This validation catches errors early and ensures that configuration files are correct, reducing runtime errors caused by misconfigurations.

Learn more in our detailed guide to testing environment 

Use Environment-Specific Configuration Files

Environment-specific configuration files help manage settings across different stages of development, such as development, testing, staging, and production. Use naming conventions or directories to separate configuration files for each environment. For example, have config.dev.ini, config.test.ini, and config.prod.ini for respective environments.

Implement logic in your application to load the appropriate configuration file based on the environment it is running in. This approach ensures that environment-specific settings are correctly applied, reducing the risk of configuration errors when deploying across different environments.

Managing and Python Configuration Files with Configu

In light of the limitations of Dotenv libraries, you should consider a more robust alternative to managing environment variables. Configu is a configuration management platform comprised of two main components, the stand-alone Orchestrator, which is open source, and the Cloud, which is a SaaS solution:

Configu Orchestrator

As applications become more dynamic and distributed in microservices architectures, configurations are getting more fragmented. They are saved as raw text that is spread across multiple stores, databases, files, git repositories, and third-party tools (a typical company will have five to ten different stores).

The Configu Orchestrator, which is open-source software, is a powerful standalone tool designed to address this challenge by providing configuration orchestration along with Configuration-as-Code (CaC) approach.

Configu Cloud

Configu Cloud is the most innovative store purpose-built for configurations, including environment variables, secrets, and feature flags. It is built based on the Configu configuration-as-code (CaC) approach and can model configurations and wrap them with unique layers, providing collaboration capabilities, visibility into configuration workflows, and security and compliance standardization.

Unlike legacy tools, which treat configurations as unstructured data or key-value pairs, Configu is leading the way with a Configuration-as-Code approach. By modeling configurations, they are treated as first-class citizens in the developers’ code. This makes our solution more robust and reliable and also enables Configu to provide more capabilities, such as visualization, a testing framework, and security abilities.

Learn more about Configu