Syncing Configs

Let’s say we want to configure a daemon called demonio (which is spanish for daemon).

About daemonio:

  1. It stores it’s configs in the directory /etc/daemonio.d
  2. It can detect configuration changes itself (See Reloading Configs)
  3. This config of daemon is named my_daemonio [1]
[1]If we ever run multiple instances of the daemon on the same machine or accross the same cluster we need to differentiate different config folders

Server Setup

  1. Configure the service to read configs from /etc/daemonio-configs/current instead of /etc/daemonio.d

    Note

    The point here is: ciruela can sync whole directory of configs and replace it atomically. But to do that we need to grant access for user to the parent directory of the actual config folder. We don’t want to grant access to the whole /etc.

  2. Drop the following file into /etc/ciruela/configs/my-daemonio.yaml:

    directory: /etc/daemonio-configs
num-levels: 1
append-only: false
upload-keys: [my_daemonio]

  3. Drop the public key allowed to upload new config into /etc/ciruela/keys/my_daemonio.key. (See Client Setup)

    Note

    Multiple keys can be put into the file as well as multiple files can be specified in upload-keys in the configuration of the directory. It’s useful to create a file per actual user (potentially having multiple keys) or organize keys in any way you want.

  4. Set ciruela as the owner of all the directories it should sync:

    chown -R ciruela /etc/daemonio-configs

Client Setup

Human Operator Setup

This is mostly useful for testing, because work well mostly for a single user.

  1. Generate a key, it’s same as ssh but in ed25519 format:

    ssh-keygen -t ed25519 -f ~/.ssh/id_ciruela -P ""

    Note

    Ciruela doesn’t support password-protected keys yet.

    This also means you can use your normal ~/.ssh/id_ed25519 key, if it isn’t password-protected, but leaving ssh key plain-text isn’t good idea. Ciruela keys are much more lower risk because they allow only uploading a limited set of directories rather than executing any script.

    Also you may wish to delete the key and use CI system when you finish testing the setup.

  2. Upload the key into /etc/ciruela/keys/my_daemonio.key.

  3. Run the following every time you need to upload new configs:

    ciruela sync server.name --replace local/config/path:/my-daemonio/current

CI Setup

Usually CI systems allow to put secret variables into environment, so you can use CIRUELA_KEY environment variable for storing keys.

  1. Generate a key, it’s same as ssh but in id_ed25519 format:

    ssh-keygen -t ed25519 -f tmp-key -P ""

  2. Upload the private key into CI for CIRUELA_KEY, for example for travis you may use travis encrypt:

    travis encrypt "CIRUELA_KEY=$(cat tmp-key)" --add

  3. Add upload command to the task:

    ciruela sync server.name --replace ./cfg:/my-daemonio/current

Reloading Configs

All of this works if your service can pick up configuration on the fly without any kind of signals.

Making signals for configuration reload is out of scope of this article but here are some ideas:

  1. You can set a script that compares directory timestamp and signals service if that changes. Ciruela replaces directory atomically so reloading is safe at any time (or as quick as the directory is new)
  2. You can use some special programs for that (but I’m not sure they are suited for production):
  3. Some supervisors like supervisord (API) and systemd (API) have RPC for the task
  4. Maybe you have a UI for the service?

Just to show you that (1) is not as scareful as it sounds, here is an example script for nginx:

#!/bin/sh

DIR=/etc/nginx/conf
CMD="nginx -s reload"

last_stat="$(stat --format="%Z/%Y/%d:%i" "$DIR" || "<absent>")"
while sleep 1; do
  new_stat="$(stat --format="%Z/%Y/%d:%i" "$DIR" || "<absent>")"
  if [ "$last_stat" != "$new_stat" ]; then
    $CMD
  endif
done

Note

The script doesn’t detect most changes done on individual config files, but ciruela always replaces the directory with the new one. And we detect it by checking inode number "%i". Other stat parameters here are just for being more cautious.

Additional Options

  • If your service has only one configuration file, you should put it into a directory anyway, as ciruela syncs directories. But it’s a good idea since you can add another include file later or just put a README into the dir.

  • You may want to check configs before uploading. For example run daemonio --config=./local_config_dir --check-config on the CI server before upload.

  • You can override keys via -i, -e (see ciruela sync --help)

  • You can upload multiple dirs simultaneously via:

    ciruela sync s1.example.org --replace ./dir1:/dest1 --replace ./dir2:/dest

  • If server name resolves to multiple IP addresses, ciruela will try to upload to at most three of them (random ones if there are more) and will return non-zero exit status if none of them accepts the upload.

  • Multiple names on command-line treated as a separate clusters. So ciruela will upload on three servers on each of them:

    ciruela sync s1.example.org s1.example.org --replace ./dir1:/dest1

    This will report upload progress for every cluster on it’s own.

    If these are individual servers use -m:

    ciruela sync -m s1.example.org s1.example.org --replace ./dir1:/dest1

    With > 4 servers this makes ciruela upload to at least 75% of them and tolerate few failures. Just like it does for a single cluster name and multiple servers behind.

  • Mutliple instances of daemonio can be configured with a single upload key you may put multiple configurations into the single directory:

    • /etc/daemonio-configs/my_daemonio
    • /etc/daemonio-configs/other_daemonio

  • Or you can group all configured services under single folder (if you don’t need to differentiate permissions for them):

    • /etc/syncing-configs/daemonio
    • /etc/syncing-configs/nginx
    • /etc/syncing-configs/my-other-service