Confetti

From Outrun Wiki
Jump to: navigation, search

Overview

Confetti is a tool for Enterprise Linux 6 that manages system configuration files and settings. The name Confetti relates to the small Snippets of text being glued together into complete configuration files.

Explanation

Confetti was invented as a way to dynamically manage configuration files for services that use a single file and don’t have an 'includedir' option.

For example, logrotate(8) has an /etc/logrotate.d/ directory where sub-configuration files are placed. That way, if an administrator wants to add a logfile to the logrotate facility, he does not have to edit the main /etc/logrotate.conf file (which easily could result in errors) but instead he would place a small snippet in /etc/logrotate.d/ just to add the entries for rotating one configuration file.

Other familiar examples are:

Service Default includedir
Sudo /etc/sudoers.d/
xinetd /etc/xinetd.d/
shell profiles /etc/profile.d

To explain in more detail, let's take a look at xinetd. There is a global configuration file /etc/xinetd.conf (man xinetd.conf) that specifies the global parameters for xinetd such as where xinetd sends logging. The last line of xinetd.conf shows

includedir /etc/xinetd.d

If you want to add a service to xinetd then you would only place a service definition file in the includedir instead of editing the main configuration file. The end result is that xinetd behaves as if it would have one configuration file constructed from /etc/xinetd.conf and all the snippets in /etc/xinetd.d/

See The /etc/xinetd.d/ Directory for more info.

Not all Linux facilities have such a flexible includedir option. Examples of facilities NOT offering an includedir (.d) directory:

Service Configurarion file
NFS /etc/exports
Services /etc/services
Sysctl /etc/sysctl.conf

Confetti allows administrators to add extra configuration options to such facilities without having to carefully modify the primary configuration file.

Problem description

Let's say there is a service 'foo' that uses a configuration file /etc/foo.conf. The foo service does not offer an "includedir ..." option so creating a /etc/foo.d/ directory with config snippets would not help us much. If we need to add items to the "foo" service, we need to edit /etc/foo.conf. If we want to automate such additions (i.e. in install scripts), we quickly end up with having to write SED and AWK programs every time. Such programs have to take care of:

  • Existing entries to avoid double adding the same item
  • Existing entries with different values
  • Different spacings between assignments:
VAR=VALUE
VAR= VALUE
VAR = VALUE
VAR="VALUE"
  • Comments:
# comment
#comment
VAR=VALUE # comment
; comment
// comment
 # comment not starting on left column
  • Multiple packages/services attempting to set the same configuration items

Solution

Confetti solves this problem by allowing us to create a directory named /etc/foo.conf.d/ and a config file /etc/confetti/foo. We create the following files:

File Purpose
/etc/foo.conf.d/ Top level directory for our service
/etc/confetti/foo Global (master) configuration file
/etc/foo.conf.d/item1.txt Foo service addition for "item1"

Let's say /etc/foo.conf.d/foo.conf contains the following contents:

# /etc/foo.conf.d/foo.conf
#
# Some comment describing this parameter
global setting 1 = some param

# Another comment
global setting 2=another param

We add the file /etc/confetti/foo.conf.d/item1.txt with contents:

# /etc/foo.conf.d/item1.txt
#
# An extra service item for the FOO service
foo-service item1 {
  name = item1
  desc = Some example item
}

Now we run confetti: /usr/sbin/confetti and take a look at the resulting file:

# /usr/sbin/confetti 
Updating /etc/foo.conf
# cat /etc/foo.conf

# /etc/foo.conf - generated via confetti from /etc/foo.conf.d
# Do not directly edit this file! It will be overwritten during boot
# or configuration changes (see man 8 confetti)
# To add entries or change the configuration, edit the files in
# /etc/foo.conf.d
# ------------------------------------------------------------------
global setting 1 = some param
global setting 2=another param
foo-service item1 {
  name = item1
  desc = Some example item
}

Notes:

  • As you can see the comments from our own foo.conf file are replaced by a generic header. It's possible to get the original comments (see below)
  • The resulting file is by default placed in /etc/. It's also possible to change that location.
  • Confetti is configured to run each hour from cron.hourly as well as during each reboot. If you temporarily edit the /etc/foo.conf file, expect it to be replaced by the original one within one hour.
  • The original /etc/foo.conf (if one existed) is moved to /tmp/confetti-backup/ in case you accidentally overwrite settings.

Example

It's easy to see how Confetti can be used to add all kinds of stuff to existing configurations. For example, take the case of /etc/services. Say we run VNC services on the default VNC port 5900. /etc/services has an entry for that:

# cat /etc/services|grep ^vnc
vnc-server      5900/tcp                # VNC Server
vnc-server      5900/udp                # VNC Server

Now we want to add an entry for an extra VNC port on 5902:

vnc-2           5902/tcp                # VNC Server Persistent
vnc-2           5902/udp                # VNC Server Persistent

Classic option: add something like "vnc-2 5902/tcp # vnc-server-1" to /etc/services. How do you do that using a script - without accidentally creating double entries etc?

Classic method

Create a script that does something like:

# Add an entry if it does not exist yet
grep -qs "^vnc-2" /etc/services || echo "vnc-2           5902/tcp                # VNC Server Persistent" >> /etc/services
# Modify the entry regardless of the parameters
sed -i -e "s/^vnc-2.*/vnc-2           5902/tcp                # VNC Server Persistent/" /etc/services

You can see where this becomes problematic:

  • We need to make things much more complicated because we need to add two entries (one for tcp and one for udp)
  • Need to write a set of commands for each and every line entry you want to add
  • We are overwriting /etc/services every time we run our configuration script
  • Bugs are subtle: What if there is another entry named vnc-22? (matches the ^vnc-2.* pattern - recipe for trouble)
  • What if an administrator made a typo and entered "Vnc-2" or " vnc-2"?
  • How do you delete/modify entries?
Confetti method
  • Create a directory /etc/confetti/services.d (Outrun already did this)
  • Copy /etc/services to /etc/confetti/services.d/services (if you run Outrun then this is already part of the system. This file is the exact same services file that CentOS installs by default)
  • add a file with the extra entries: /etc/confetti/services.d/lightvnc with contents:
# ports for another VNC service (lightvnc)
vnc-2           5902/tcp                # VNC Server Persistent
vnc-2           5902/udp                # VNC Server Persistent
  • Run confetti (or reboot or wait for cron.hourly to kick in)

Check the contents of /etc/services:

# tail -20 /etc/services
... removed ...
iqobject        48619/tcp               # iqobject
iqobject        48619/udp               # iqobject
... removed ...
vnc-2           5902/tcp                # VNC Server Persistent
vnc-2           5902/udp                # VNC Server Persistent

Later you want to add another entry for another VNC console on port 5903. Instead of editing /etc/services, add a new file /etc/confetti/services.d/lightvnc-3 with contents:

vnc-3           5903/tcp                # VNC Server 3
vnc-3           5903/udp                # VNC Server 3

and re-run confetti.

Removing vnc-3 from /etc/services is as simple as removing /etc/confetti/services.d/lightvnc-3 and re-running confetti again.

Running Confetti

Confetti runs by default:

  • At each reboot
  • Every hour (if Anacron is installed)

You may manually run confetti any time. If you only want to re-configure one section, you may enter the section name as parameter:

confetti services
confetti sysctl.conf

Advanced features

Hourly cron job

The confetti RPM comes with an hourly cron job (requires Anacron) so that Confetti frequently enforces correct contents of all config files. If you want to modify generated files directly you probably want to temporarily disable this job.

This can be done as follows:

echo confetti >> /etc/cron.hourly/jobs.deny

Re-enable by removing 'confetti' from jobs.deny.

Confetti during boot

The confetti RPM installs a service "confetti" that runs during boots. Disable/re-enable using the standard "chkconfig confetti <on|off>" method.

Config file

In /etc/confetti/ we need to store a file with configuration details for foo.conf.

Using
confetti template
we can generate a template, which looks like:
# /etc/confetti/demo
#
# Define file to be updated
target    = /etc/demo.conf

# Define primary source file (default: same as target filename)
# source  = /etc/demo.d/demo.conf

# Single file (don't process a .d directory
# single  = yes

# Define .d directory (default: /etc/<target name without extension>.d/
# dir     = /etc/demo.d

# Define postprocess command
# post    = service demo restart

# Skip header
# header  = no

# Keep blank lines
# blank   = yes

# Remove comments starting with character
# comment = #

# Set owner, group, mode (octal)
# owner   = root
# group   = nobody
# mode    = 644

Skipping files

Sometimes you want to temporarily disable updates to one config file. To achieve that, place a file named .ignore in the config subdir. Example: /etc/foo.conf.d/.ignore

Master file required

If there is no master file (i.e. ../foo.conf.d/ contains item1.txt but not foo.conf) then generation of the config file is skipped (we cowardly refuse to build config files without a global section). If you don't want this, create a master file of zero bytes.

Override

If a confetti subdir is provided by an RPM package but you want to change the master config file, put that in a <name>.override file. Confetti will use the .override file instead

Distribution-specific files

Obsolete. You may get this behaviour with custom scripts.

Dynamic file content

Sometimes you want to configure settings dynamically rather than using static content. Example: You need the current IP address inside a config file. Or set memory tuning parameters based on the available RAM (i.e. /etc/sysctl.conf).

If a (static) config file exists named 'foo' combined with a (dynamic) foo.sh component, AND the .sh file is executable, it will be executed and the stdout output will be appended. Example: We want to set 'kernel.shmmax' in /etc/sysctl.conf to 50% of available RAM. Assume we already have a confetti configuration for sysctl.conf i.e. /etc/confetti/sysctl.conf.d with a 'meta' file and a 'sysctl.conf' master file (which provides the static sysconf settings)

we add an empty file 'shmmax' to trigger shell script execution. We also need a 'shmmax.sh' script that spits out the kernel.shmmax=... entries when executed.

Script shmmax.sh:

#!/bin/bash
echo "# Calculated value for shmmax - 50% of ram"
echo "kernel.shmmax = $(awk '/MemTotal/ {print $2"*1024/2"}' /proc/meminfo|bc)"

Note that this requires the "bc" calculator tool to be installed.

Now every time confetti runs, the shmmax value will be calculated. If we run on VMware and we change the virtual amount of RAM, after reboot the system will reset shmmax to 50% of the new RAM.

Note that the config file should contain:

post = sysctl -q -p

So it reloads /etc/sysctl.conf every time it is updated.

This shmmax config method is available from the 'outrun-oracle' package if you run the Outrun distro.

Ignored files

Some tools (RPM, vi, sed etc) sometimes create backup files and we don't want them to be included. Files with these extensions will be skipped:

~ .bak .orig .rpmnew .rpmorig .rpmsave .org .tmp

Parsing order of configuration snippets

The order of parsing all snippet files is how the "find" command reports them and then sorted in alphabetic order.

Testing

You can test confetti behaviour using the "-n" option - this will only show which files would be updated and not really overwrite them. Be aware that the hourly cron job does not run with this option so during tests, disable the cron job.

Backup files

Before overwriting config files, confetti will copy the original file to /tmp/confetti-backup/<file>.<timestamp> in case you destroyed precious manually edited configurations.

Network restarts

The confetti package also includes a method for dynamically updating files after network restarts rather than reboots or hourly cron jobs.

The way this works is by using a feature of Enterprise Linux that is part of the network service:

  • If a script /sbin/ifup-local exists, it will be executed directly after the network restart of each network interface.
  • If a script /sbin/ifup-pre-local exists, it will be executed directly before the network restart of each network interface.

Confetti provides both scripts:

  • ifup-pre-local will run all executable scripts /etc/ifup-pre-local.d/*.sh before network restart.
  • ifup-local will run all executable scripts /etc/ifup-pre-local.d/*.sh after network restart.

Initially I used "eth0" as trigger to start dynamic configuration but eth0 is not always the primary interface. So now the more independent method works as follows:

  • ifup-pre-local: will be triggered during configuration of "lo" (localhost)
  • ifup-local: will be triggered after configuration of the interface that is linked to the default gateway

This way even if we have multiple NICs we will only run the .d/ scripts once, and only after configuring the primary interface (assuming it has a default gateway).

Using these scripts help when IP addresses or even networks change dynamically (i.e. during movement of virtual machines across networks).