Get started for free
Sign in
More posts
Understand the essential steps for achieving AI excellence in your organization, from establishing a clear vision and managing data effectively to ensuring seamless integration.
Explore how AI-powered tools like Qatalog can revolutionize knowledge management by breaking down information silos, enhancing search capabilities, and improving collaboration within organizations.
Qatalog announces partnership with Google Cloud.
Share this article:

A GCP-secrets integrated config loader for Node.js

We're big fans of GCP at Qatalog, so of course we store all our production secrets in GCP Secret Manager. And as our backend runs on Node, we were looking for a good config loader that integrates nicely with Secret Manager. But we couldn't find one that worked how we wanted, so we built our own instead and have released it under an MIT license. Behold, @qatalog/gcp-config!

Headline features

What distinguishes it from the alternatives? A few things:

  • The config schema is defined as a JSON structure, each item specifying options that control how values are loaded, validated and coerced to produce the resulting config object.
  • Config values can be loaded from a number of sources; Secret Manager, environment variables, files and default values in the schema itself.
  • Secrets are not leaked to the environment, so people with access to your production containers do not also gain access to your production secrets.
  • Validation uses Joi, which we think is the best Node validation library bar none. If you're already using Joi for API validation, it follows naturally to also use it for config.
  • The test suite runs against a real GCP project in CI, so you can have confidence that the integration with Secret Manager is robust.

How does it work?

First add @qatalog/gcp-config and joi to your dependencies in package.json (joi is marked as a peer dependency so you need to include it too).

Then you can use it like so:

const gcpConfig = require('@qatalog/gcp-config');
const joi = require('joi');

  foo: {
    env: 'NAME_OF_ENV_VAR',
    secret: 'name_of_secret_in_gcp',
    default: 'default value',
    schema: joi.string(),

  bar: {
    baz: {
      env: 'BAR_BAZ',
      secret: 'bar.baz',
      schema: joi.number().positive(),
      required: true,

    qux: {
      secret: 'bar.qux',
      schema: joi.string(),

  // ...

const { GCP_PROJECT, NODE_ENV } = process.env;


async function main() {
  const config = await gcpConfig.load({
    file: `config/${NODE_ENV}.json`,
    project: GCP_PROJECT,
    schema: CONFIG_SCHEMA,

  assert(typeof === 'string');
  assert(typeof === 'object');
  assert(typeof === 'number');
  assert(typeof === 'string');

  // ...

All properties in the config schema are optional, and the JSON can be nested as deeply as you like. The tree structure you specify will be mirrored in the returned config object.

Config values are loaded in the following order of precedence:

  1. Environment variables. If an environment variable set, it takes precedence over everything else. This can especially useful in local dev if you want to override secrets without touching them in GCP.
  2. GCP Secret Manager. Secrets are loaded optimistically, so it's not fatal when they don't exist in GCP unless they've been marked as required in the config schema.
  3. File system. If you want to, you can specify the path to a JSON file from which to load values that aren't set via environment variable or secret. This can be useful if there are any default values that you're happy to commit to source control.
  4. Config schema. Finally, you can also set default values directly in the config schema itself, if you find that more convenient.

There are lots more details in the readme, but in the remainder of this post I want to highlight three features in particular that have saved us from harm.

Error: Quota exceeded 😞

A problem we used to face repeatedly with Secret Manager is the dreaded quota exceeded error. We have a lot of integration tests and when multiple engineers run jobs in CI, we'd often see these errors cause tests to fail.

So we added an ignoreSecrets option to the load method that lets you skip reading from Secret Manager in those cases. For example, if you have a test.json that contains all the config settings you want to use in CI, you might do this:

const config = await gcpConfig.load({
  file: `${NODE_ENV}.json`,
  ignoreSecrets: NODE_ENV === 'test',
  project: process.env.GCP_PROJECT,
  schema: CONFIG_SCHEMA,

Then you just need to make sure that your CI jobs set NODE_ENV to test, and you know they'll never fail with quota exceeded.

Network errors 😞

On rare occasions, we'd see some kind of network error occur when attempting to read secrets. Either a connection reset or, on one occasion, a GCP outage that happened to coincide with a production deployment. Because load falls back to alternative options by default if reading a secret fails, it's important to have a way to make those errors fatal when you want them to be. For that, we added the required option to the config schema.

We add it to all of the nodes in our schema that absolutely must come from Secret Manager. Things like encryption keys, API tokens and database passwords. If the connection fails when reading any of those secrets, the promise returned from load will be rejected and we can either retry or abort the process:

try {
  const config = await gcpConfig.load({
    project: process.env.GCP_PROJECT,
    schema: {
      postgres: {
        host: {
          env: 'PGHOST',
          required: true,
          secret: 'postgres_host',
        password: {
          env: 'PGPASSWORD',
          required: true,
          secret: 'postgres_password',
        port: {
          default: 5432,
          env: 'PGPORT',
          schema: joi.number().port(),
        user: {
          env: 'PGUSER',
          required: true,
          secret: 'postgres_user',
} catch (error) {
  log.error('config.error', { error });

Working with durations 😞

In the JavaScript world, it's typical to work with durations as milliseconds, because those are the units used by standard libary functions such as setTimeout and But in the operations space it's often more common to use seconds instead. To make mixups between those two conventions less likely, we added explicit coercion options to the config schema.

For any node that has schema: joi.string().isoDuration(), you can specify the value as an ISO-8601 duration string and add a coerce option indicating whether you want the resulting config value to be converted to integer seconds or milliseconds:

const config = await gcpConfig.load({
  project: process.env.GCP_PROJECT,
  schema: {
    cacheExpiry: {
      env: 'CACHE_EXPIRY',
      schema: joi.string().isoDuration(),
      coerce: { from: 'duration', to: 'milliseconds' },
      default: 'PT12H',

assert(typeof config.cacheExpiry === 'number');

And that's it!

If you're using GCP Secret Manager in your stack, we'd love you to try it out. Please open issues (or pull requests!) on GitHub if there are any changes you'd like to see.

A London based VP of Engineering, Leo obsesses about productivity and efficiency
See their articles
Latest articles
Rapid setup,
easy deployment
Seamless onboarding β€’ Enterprise grade security β€’ Concierge support
Get started for free