Application Configuration Management

Aaron M Aaron   M.
June 24, 2020

Introduction

At JBS, High Velocity Software Development means hitting the ground running, utilizing proven architectural practices to find reusable solutions. The study presented below demonstrates how we identify redundant work and decompose it into reusable components, accelerating all of our development efforts.

With many of our clients, JBS augments multiple teams that each have different business initiatives and objectives. By maintaining lines of conversation between these teams, we are able to identify common tools and APIs used by our client. Once common APIs are identified, the next step is building shareable interface libraries. These shared libraries provide all teams with a common, clean interface to each API. Furthermore, all teams can benefit from the improvements made by each group. This shared body of work enables JBS and client employees to move between teams as needed while maintaining a high level of productivity.

Shared Library and Decomposition

To keep the interface libraries consistent, we continue to decompose common tasks or utilities into more reusable components. Chief among these is application configuration and secrets management.

Application Configuration Management

The ability to have feature flags or change configuration in real time without deployments is something most of our applications require. Since we use Python and Django to build our service side components, Constance was an obvious solution for real time configuration. Not all of our apps use it or require it, however, so we decided to make constance an optional requirement.

setup(
    name=‘config-loader',
    version=‘0.0.1',
    packages=[‘config_loader'],
    include_package_data=True,
    description='Support optional loading of constance configs over django settings for use in libraries.',
    install_requires=install_requires,
    setup_requires=['pytest-runner'],
    tests_require=['redis', 'pytest', 'pytest-coverage', 'pytest-django', 'pytest-mock'],
    long_description=README,
    author='',
    author_email='',
    classifiers=[
        'Programming Language :: Python :: 3.7',
    ],
    extras_require={
        'constance':  ['django-constance>=2.0.0,<3'],
        'credstash':  ['credstash>=1.0.0,<2'],
        'django': ['Django>=1.11'],
    }
)

By using extras_require in the setup.py for your library, you can specify optional packages that your library supports. To have them installed, install your package like `pip install config-loader[constance]`. Find more details in the [https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies|setuptools docs].

Most of our applications are deployed with Zappa to AWS Lambda. This makes it convenient to do per environment configuration in a zappa_settings.yml file. To support applications that go this route, our config library also supports reading from environment variables.

In our final solution libraries can declare their settings, along with a namespace in a `Settings` object.

from config_loader import Settings

loader_settings = Settings({'A_SETTING': 'DEFAULT'}, ’NAMESPACE')

And load values from a hierarchy of places:

  1. constance CONSTANCE_CONFIG = {’NAMESPACE_A_SETTING': (True, 'A fine setting'),}
  2. settings.py NAMESPACE = {'A_SETTING': 'NEW_VALUE’}
  3. os.environ export NAMESPACE_A_SETTING=env_Value
  4. The configured default

Secrets Management

Another optional requirement, used by applications that need credentials to other systems, is credstash. Credstash leverages AWS KMS and AWS Dynamo to maintain secure storage of secret data. With a little bit of code glue, we are able to specify the name of a secure value in our clear text config, protecting the actual secrets.

Any of the values above can be surrounded by %% eg %%app.API_PASSWORD.dev%% to fetch those values from credstash.

Results

With many developers from different teams contributing to and reviewing this library, we’ve accumulated a collection of robust features in a single reusable component. The final solution includes caching, logging, documentation, and testing - all of a higher quality than one person or team would likely be able to deliver. The end result is higher reliability, more team bandwidth, and "free" updates.

We have found that once a particular integration or feature has been moved to a shared library, it greatly reduces the overhead for a new team to develop that integration in the future. Rather than re-implementing the same (or similar) code in multiple applications, our teams stay efficient and deliver more high-quality software by working together.

Introducing the JBS Quick Launch Lab!

FREE 1/2 Day Assessment

Quantify what it will take to implement your next big idea! Our intensive 1/2 day session will deliver tangible timelines, costs, high-level requirements, and recommend architectures that will work best, and all for FREE. Let JBS show you why over 20 years of experience matters.
Yes, I'd Like A FREE Assessment