OP5 Monitor - Using git repositories to backup, track and restore local configuration files

You may already know "git": released in 2005, it's today the most popular Version Control System (VCS) available, used by about 70% of code repositories online in 2019.

While git is normally used for collaborative development of code, it also works locally, and can be used for any use case where you want to track changes to text files over time. For this reason, it's a good fit for directories containing configuration files.

The first thing you want to do is to actually install git, if you don't already have it:

# yum install git -y

At this point, we probably want to switch to the "monitor" user, because using git as root doesn't make a lot of sense. To be able to do this stuff as "monitor", we may need to change permissions for the monitor home directory. This is unsupported until changed in the product, so you need to be aware of what you're doing and how to revert it. This permissions issue is tracked in MON-11796.

Check if /opt/monitor is writable by the "monitor" user:

# ls -alh /opt/monitor/ | head -n2
total 8,0K
drwxr-xr-x.  8 root    root     77  6 mar 19.08 .

It's not, so without modifying the base permissions for user/group/other, let's add an Access Control List (ACL) entry saying that the monitor user gets "rwx" on this directory.

# setfacl --modify u:monitor:rwx /opt/monitor/

Another ls indicates with a plus sign at the end that ACL is in effect:

drwxrwxr-x+  8 root    root     77  6
mar 19.08 .

View it with:

# getfacl /opt/monitor/
(...)
user:monitor:rwx

To revert this change, run "setfacl --remove-all", targeting the directory.

We should now be able to use git as "monitor".

Using git as monitor

First, let's switch to the "monitor" user:

# su - monitor

In order to use git, we must provide a name and an e-mail that will be included in all of our commits. 

$ git config --global user.email "monitor@localhost" && git config --global user.name "monitor"

You now need to go to the directory you want to create your repository in, and initialize it:

$ cd /opt/monitor/etc && git init
Initialized empty Git repository in /opt/monitor/etc/.git/

If you run "git status", you will note that all files in this folder are "untracked". By default, git will only track the files you tell it to, but since we assume that all files in this folder should be tracked, we can simply add all of them:

$ git add .

The files should now be marked as "new file" in git status.

Let's make our first commit manually, to create our first "snapshot" of the current directory state:

$ git commit -m "Initial commit"

This commit will now show up in "git log" with its own unique hash.

We'll go through how to revert local changes later under "failure scenarios". Obviously, making these commits manually whenever you feel like it is not a great idea, so there's a couple of things we can do to ensure commits happen automatically.

Automatic commits

Commit on every config save

By creating a Nacoma "save hook", we can make certain things happen on every config save. For more information, read /opt/monitor/op5/nacoma/hooks/save/README

You need to do this as root:

# vim /opt/monitor/op5/nacoma/hooks/save/git_hook.py

git_hook.py

#!/usr/bin/env python2

import sys, subprocess
from nacoma.hooks import Change

if sys.argv[1]:
    username = sys.argv[1]
else:
    username = "UNKNOWN"

multiline_commit = '''
'''

for line in sys.stdin:
    change = Change(line)
    if change.is_deleted():
        multiline_commit += "\n{} deleted: {}".format(change.username, change.oldname)
    elif change.is_new():
        multiline_commit += "\n{} created: {}".format(change.username, change.newname)
    elif change.is_renamed():
        multiline_commit += "\n{} renamed: {} -> {}".format(change.username, change.oldname, change.newname)
    else:
        multiline_commit += "\n{} made a change to an existing object".format(change.username)

subprocess.Popen(['/usr/bin/git', 'commit', '--author', '{} <{}@localhost>'.format(username, username), '-a', '-m', '{}'.format(multiline_commit).strip()], cwd='/opt/monitor/etc')

Note: For minor changes, the log will not be very useful as the exact change is not exposed very well by the library used. You can, however, see the actual change by using diff or "git show".

Mirror the existing permissions:

# chmod a+x /opt/monitor/op5/nacoma/hooks/save/git_hook.py

In some cases, a restart may be required at this point:

mon restart && systemctl restart httpd

Given two changes by users foo and bar, committed by foo, the git log will now look like this after the GUI save was executed:

Author: foo <foo@localhost>
Date:   Mon Mar 8 13:44:07 2021 +0100
    bar deleted: baz
    bar deleted: baz;PING
    foo created: foozz
    foo created: foozz;PING

Pretty neat, we can tell who staged the changes, and who pushed them through. But what if this change was catastrophically bad and we want to undo it? See "failure scenarios" below.

Commit daily via cron

If you already "commit on every config save" you may not want or need this. To run a daily commit as "monitor" via cron, run "cron -e" to edit your crontab and add:

0 2 * * * cd /opt/monitor/etc && /usr/bin/git commit -a -m "Nightly cron commit"

This will run the command daily at 2am.

Failure scenarios

Note that some of this usage is considered bad practice in shared repositories, but we are not sharing this local repo with anyone else.

Remember that Naemon does not detect configuration changes done during runtime, so you should "mon stop" first before doing any of this.

All of these commands assume you're already in the repository directory (/opt/monitor/etc). Since "mon stop" requires root, we'll do the "reset" as root as well, but it's not required.

1: One or more commits are bad, I want to revert the changes

If we want to get rid of one or multiple commits forever and restore the local files in the process, we should "reset". We first need to figure out which commit we want to reset to though.

Use "git log" to find one that you want to go back to. Copy the hash of it, and use it in the below command:

git reset --hard 5ff88721c5ae64701efad9107b5962b0d1393b60

The output will tell you where you moved to:

HEAD is now at 5ff8872 (...)

In this case, an object had been deleted, but was restored by the above command. If we do "mon start", we can refresh a list view in the GUI and see the object is back.

Note that the configuration database used by the GUI is not updated automatically, you need to run the following to make that happen:

# asmonitor /usr/bin/naemon --verify-config /opt/monitor/etc/naemon.cfg && echo "Now running undo_config..." && php /opt/monitor/op5/nacoma/api/monitor.php -u monitor -a undo_config

This will validate your on-disk configuration files, and if they pass, import them to the database.

You also need to run "mon restart" to apply the changed configuration files.

2: I want to see what changed between two commits

"git diff" takes commit hashes like described above, you can use these to show the difference between any two arbitrary commits. Find them with "git log" and then do:

git diff 5ff88721c5ae64701efad9107b5962b0d1393b60 0355442688d99596504f38789be8bd196d342dac | less

You can now page through the output and see what files changed, and what changed in them.

3: I reset like in #1, but I still have untracked (new) files

If, in addition to a reset, you also want to delete non-tracked files, you can run:

git clean -f -d

Where "-f" means force, and "-d" means that untracked directories, in addition to files, should also be deleted.

If you wish to ignore the .gitignore rules, you can also add "-x"