We're big fans of GCP at Qatalog, so of course we store all our production secrets in GCP Secret Manager. And as our backend runs on Node, we were looking for a good config loader that integrates nicely with Secret Manager. But we couldn't find one that worked how we wanted, so we built our own instead and have released it under an MIT license. Behold, @qatalog/gcp-config!

Headline features

What distinguishes it from the alternatives? A few things:

• The config schema is defined as a JSON structure, each item specifying options that control how values are loaded, validated and coerced to produce the resulting config object.
• Config values can be loaded from a number of sources; Secret Manager, environment variables, files and default values in the schema itself.
• Secrets are not leaked to the environment, so people with access to your production containers do not also gain access to your production secrets.
• Validation uses Joi, which we think is the best Node validation library bar none. If you're already using Joi for API validation, it follows naturally to also use it for config.
• The test suite runs against a real GCP project in CI, so you can have confidence that the integration with Secret Manager is robust.

How does it work?

First add @qatalog/gcp-config and joi to your dependencies in package.json (joi is marked as a peer dependency so you need to include it too).

Then you can use it like so:

const gcpConfig = require('@qatalog/gcp-config');
const joi = require('joi');

const CONFIG_SCHEMA = {
  foo: {
    env: 'NAME_OF_ENV_VAR',
    secret: 'name_of_secret_in_gcp',
    default: 'default value',
    schema: joi.string(),
  },

  bar: {
    baz: {
      env: 'BAR_BAZ',
      secret: 'bar.baz',
      schema: joi.number().positive(),
      required: true,
    },

    qux: {
      secret: 'bar.qux',
      schema: joi.string(),
    },
  },

  // ...
};

const { GCP_PROJECT, NODE_ENV } = process.env;

main();

async function main() {
  const config = await gcpConfig.load({
    file: `config/${NODE_ENV}.json`,
    project: GCP_PROJECT,
    schema: CONFIG_SCHEMA,
  });

  assert(typeof config.foo === 'string');
  assert(typeof config.bar === 'object');
  assert(typeof config.bar.baz === 'number');
  assert(typeof config.bar.qux === 'string');

  // ...
}

All properties in the config schema are optional, and the JSON can be nested as deeply as you like. The tree structure you specify will be mirrored in the returned config object.

Config values are loaded in the following order of precedence:

1. Environment variables. If an environment variable set, it takes precedence over everything else. This can especially useful in local dev if you want to override secrets without touching them in GCP.
2. GCP Secret Manager. Secrets are loaded optimistically, so it's not fatal when they don't exist in GCP unless they've been marked as required in the config schema.
3. File system. If you want to, you can specify the path to a JSON file from which to load values that aren't set via environment variable or secret. This can be useful if there are any default values that you're happy to commit to source control.
4. Config schema. Finally, you can also set default values directly in the config schema itself, if you find that more convenient.

There are lots more details in the readme, but in the remainder of this post I want to highlight three features in particular that have saved us from harm.

Error: Quota exceeded 😞

A problem we used to face repeatedly with Secret Manager is the dreaded quota exceeded error. We have a lot of integration tests and when multiple engineers run jobs in CI, we'd often see these errors cause tests to fail.

So we added an ignoreSecrets option to the load method that lets you skip reading from Secret Manager in those cases. For example, if you have a test.json that contains all the config settings you want to use in CI, you might do this:

const config = await gcpConfig.load({
  file: `${NODE_ENV}.json`,
  ignoreSecrets: NODE_ENV === 'test',
  project: process.env.GCP_PROJECT,
  schema: CONFIG_SCHEMA,
});

Then you just need to make sure that your CI jobs set NODE_ENV to test, and you know they'll never fail with quota exceeded.

Network errors 😞

On rare occasions, we'd see some kind of network error occur when attempting to read secrets. Either a connection reset or, on one occasion, a GCP outage that happened to coincide with a production deployment. Because load falls back to alternative options by default if reading a secret fails, it's important to have a way to make those errors fatal when you want them to be. For that, we added the required option to the config schema.

We add it to all of the nodes in our schema that absolutely must come from Secret Manager. Things like encryption keys, API tokens and database passwords. If the connection fails when reading any of those secrets, the promise returned from load will be rejected and we can either retry or abort the process:

try {
  const config = await gcpConfig.load({
    project: process.env.GCP_PROJECT,
    schema: {
      postgres: {
        host: {
          env: 'PGHOST',
          required: true,
          secret: 'postgres_host',
        },
        password: {
          env: 'PGPASSWORD',
          required: true,
          secret: 'postgres_password',
        },
        port: {
          default: 5432,
          env: 'PGPORT',
          schema: joi.number().port(),
        },
        user: {
          env: 'PGUSER',
          required: true,
          secret: 'postgres_user',
        },
      },
    },
  });
} catch (error) {
  log.error('config.error', { error });
  process.exit(1);
}

Working with durations 😞

In the JavaScript world, it's typical to work with durations as milliseconds, because those are the units used by standard libary functions such as setTimeout and Date.now. But in the operations space it's often more common to use seconds instead. To make mixups between those two conventions less likely, we added explicit coercion options to the config schema.

For any node that has schema: joi.string().isoDuration(), you can specify the value as an ISO-8601 duration string and add a coerce option indicating whether you want the resulting config value to be converted to integer seconds or milliseconds:

const config = await gcpConfig.load({
  project: process.env.GCP_PROJECT,
  schema: {
    cacheExpiry: {
      env: 'CACHE_EXPIRY',
      schema: joi.string().isoDuration(),
      coerce: { from: 'duration', to: 'milliseconds' },
      default: 'PT12H',
    },
 },
});

assert(typeof config.cacheExpiry === 'number');

And that's it!

If you're using GCP Secret Manager in your stack, we'd love you to try it out. Please open issues (or pull requests!) on GitHub if there are any changes you'd like to see.