A GCP-secrets integrated config loader for Node.js
We're big fans of GCP at Qatalog, so of course we store all our production secrets in GCP Secret Manager. And as our backend runs on Node, we were looking for a good config loader that integrates nicely with Secret Manager. But we couldn't find one that worked how we wanted, so we built our own instead and have released it under an MIT license. Behold, @qatalog/gcp-config
!
Headline features
What distinguishes it from the alternatives? A few things:
- The config schema is defined as a JSON structure, each item specifying options that control how values are loaded, validated and coerced to produce the resulting config object.
- Config values can be loaded from a number of sources; Secret Manager, environment variables, files and default values in the schema itself.
- Secrets are not leaked to the environment, so people with access to your production containers do not also gain access to your production secrets.
- Validation uses Joi, which we think is the best Node validation library bar none. If you're already using Joi for API validation, it follows naturally to also use it for config.
- The test suite runs against a real GCP project in CI, so you can have confidence that the integration with Secret Manager is robust.
How does it work?
First add @qatalog/gcp-config
and joi
to your dependencies in package.json
(joi
is marked as a peer dependency so you need to include it too).
Then you can use it like so:
const gcpConfig = require('@qatalog/gcp-config');
const joi = require('joi');
const CONFIG_SCHEMA = {
foo: {
env: 'NAME_OF_ENV_VAR',
secret: 'name_of_secret_in_gcp',
default: 'default value',
schema: joi.string(),
},
bar: {
baz: {
env: 'BAR_BAZ',
secret: 'bar.baz',
schema: joi.number().positive(),
required: true,
},
qux: {
secret: 'bar.qux',
schema: joi.string(),
},
},
// ...
};
const { GCP_PROJECT, NODE_ENV } = process.env;
main();
async function main() {
const config = await gcpConfig.load({
file: `config/${NODE_ENV}.json`,
project: GCP_PROJECT,
schema: CONFIG_SCHEMA,
});
assert(typeof config.foo === 'string');
assert(typeof config.bar === 'object');
assert(typeof config.bar.baz === 'number');
assert(typeof config.bar.qux === 'string');
// ...
}
All properties in the config schema are optional, and the JSON can be nested as deeply as you like. The tree structure you specify will be mirrored in the returned config object.
Config values are loaded in the following order of precedence:
- Environment variables. If an environment variable set, it takes precedence over everything else. This can especially useful in local dev if you want to override secrets without touching them in GCP.
- GCP Secret Manager. Secrets are loaded optimistically, so it's not fatal when they don't exist in GCP unless they've been marked as
required
in the config schema. - File system. If you want to, you can specify the path to a JSON file from which to load values that aren't set via environment variable or secret. This can be useful if there are any default values that you're happy to commit to source control.
- Config schema. Finally, you can also set default values directly in the config schema itself, if you find that more convenient.
There are lots more details in the readme, but in the remainder of this post I want to highlight three features in particular that have saved us from harm.
Error: Quota exceeded 😞
A problem we used to face repeatedly with Secret Manager is the dreaded quota exceeded
error. We have a lot of integration tests and when multiple engineers run jobs in CI, we'd often see these errors cause tests to fail.
So we added an ignoreSecrets
option to the load
method that lets you skip reading from Secret Manager in those cases. For example, if you have a test.json
that contains all the config settings you want to use in CI, you might do this:
const config = await gcpConfig.load({
file: `${NODE_ENV}.json`,
ignoreSecrets: NODE_ENV === 'test',
project: process.env.GCP_PROJECT,
schema: CONFIG_SCHEMA,
});
Then you just need to make sure that your CI jobs set NODE_ENV
to test
, and you know they'll never fail with quota exceeded
.
Network errors 😞
On rare occasions, we'd see some kind of network error occur when attempting to read secrets. Either a connection reset or, on one occasion, a GCP outage that happened to coincide with a production deployment. Because load
falls back to alternative options by default if reading a secret fails, it's important to have a way to make those errors fatal when you want them to be. For that, we added the required
option to the config schema.
We add it to all of the nodes in our schema that absolutely must come from Secret Manager. Things like encryption keys, API tokens and database passwords. If the connection fails when reading any of those secrets, the promise returned from load
will be rejected and we can either retry or abort the process:
try {
const config = await gcpConfig.load({
project: process.env.GCP_PROJECT,
schema: {
postgres: {
host: {
env: 'PGHOST',
required: true,
secret: 'postgres_host',
},
password: {
env: 'PGPASSWORD',
required: true,
secret: 'postgres_password',
},
port: {
default: 5432,
env: 'PGPORT',
schema: joi.number().port(),
},
user: {
env: 'PGUSER',
required: true,
secret: 'postgres_user',
},
},
},
});
} catch (error) {
log.error('config.error', { error });
process.exit(1);
}
Working with durations 😞
In the JavaScript world, it's typical to work with durations as milliseconds, because those are the units used by standard libary functions such as setTimeout
and Date.now
. But in the operations space it's often more common to use seconds instead. To make mixups between those two conventions less likely, we added explicit coercion options to the config schema.
For any node that has schema: joi.string().isoDuration()
, you can specify the value as an ISO-8601 duration string and add a coerce
option indicating whether you want the resulting config value to be converted to integer seconds or milliseconds:
const config = await gcpConfig.load({
project: process.env.GCP_PROJECT,
schema: {
cacheExpiry: {
env: 'CACHE_EXPIRY',
schema: joi.string().isoDuration(),
coerce: { from: 'duration', to: 'milliseconds' },
default: 'PT12H',
},
},
});
assert(typeof config.cacheExpiry === 'number');
And that's it!
If you're using GCP Secret Manager in your stack, we'd love you to try it out. Please open issues (or pull requests!) on GitHub if there are any changes you'd like to see.