How to apply and maintain integrity on your Linux box in 3 steps

Introduction to Integrity management

The CIA triad is the security standard every CISO (Chief Information Security Officer) should know and adopt: Confidentiality, Integrity, availability.

In this article we will discuss about the Integrity concept and how to apply it to a Linux server; we will refer to CentOS 7 being a server platform similar to Red Hat 7.

The information an organization hold should be reliable. It should not change and if a change has happened we should be able to detect it. Integrity management means taking care of detecting and managing changes.

We need to prevent unauthorized changes such as: adding new users, deleting or changing logs (to erase your footprint).

To protect files from being manipulated, deleted or modified, there are several security layers you need to consider:

• File permission
• Network controls (Intrusion detection and prevention such as firewalls)
• Authentication

File permissions is handle mainly by ACL: Access Control List (wrxwrxwrx) while the best way to protect integrity of a Linux OS is to use MAC (Mandatory Access Control) which is obtained through SELinux. Even in this scenario, some remote and several local attacks can be done!

You can also manage special file attribute such as the “immutable” feature which make a file impossible to be manipulated (even by root); you can also set an attribute to a file so that you can only add information to it but you cannot modify the existing data (Append only). This is particularly good when it comes to system, audit or application logs.

To check the current attributes of a file:

# lsattr name-of-the-file

To make a file immutable

# chattr +i name-of-the-file

To remove the immutable attribute:

# chattr –i  name-of-the-file

To add an extra layer of security you can mount your filesystem “read-only”. If you do things right you can make life difficult to an attacker who has not gain root access yet.

Android has the OS binaries stored in a partition which is mounted read-only and can only be modified while in recovery mode; this an extreme solution as it does not allow you to do system updated without rebooting the device.

As things will happen (they will … I mean it) and you cannot prevent all the possible scenario within a limited budget, you need to be able to detect changes: Host-based Intrusion Detection System (H-IDS).

The simplest way (but slowest…) is to make a backup of your baseline system and periodically compare your current system/files with the backup (using diff and rsync). While doing a backup is always to be considered part of good security practice, using hash functions allow you to make a baseline of the hash of the files and then compare the current hash with the baseline.

It is almost impossible for an attacker to break it. Currently we use SHA-2 or SHA-3 hashing families. Comparing 2 hashes is a lot quicker than comparing 2 files. Anyhow there are several negative aspects that you need to take into account:

• If something has changed within a file you won’t know what has changed (Lack of Change Detail): perhaps an attacker can modify a configuration file, compromise a file which is not monitored by the HIDS and then restore the original file before a check of the integrity will happen so nobody will ever notice the attack (Static Anomaly Detection)
• If you change a file attribute the hash will not change (to be verified)
• Signing your hashes can be compromised (stripwire is an evasion tool built to specifically create files which can evade Tripwire detection)
• Restricted Operating Environment: “Checking the integrity of a program binary on disk … does not ensure that the corresponding in memory image of the program has not been modified (Garfinkel & Rosenblum, 2003)

Three applications are mainly used by system administrators to look after integrity and they basically use the concept of having a good baseline stored in a database:

• Another File Integrity Checker (AFICK): (Author: Eric Gerbier – Current version 3.5.2 – 2016-08-05 - http://afick.sourceforge.net/ ) Does not require any mail server so you don’t have to twick the installation; but it may use available mail client to send emails in case of alerts. It is a Perl based application hence slowest compared to the other rivals (which are binaries).
  It is the best in terms of CPU resources usage during the system scan and re-indexing.
• Tripwire: it is possibly the oldest solution (the idea date back in 1994 by Gene Kim and Eugene Spafford). It is the most aggressive in terms of CPU usage.
• AIDE (Advanced Intrusion Detection Environment)

We are going to provide you wit a step by step procedure to install, configure and use AFICK (reviewed version 3.5.2).

Step 1: Installing AFICK 3.5.2

We searched for the package in several repository to be able to install on CentOS 7 box using yum but it seems like it is not available, so the installation may be a little tricky. 

First you need to check if your system already have all the required dependencies (using yum command):

• perl
• perl-Digest-MD5
• perl-Tk
• perl-Digest-SHA1

Then you need to download the rpm package from the sourceforge website:

# wget https://sourceforge.net/projects/afick/files/afick/3.5.2/afick-3.5.2-1.noarch.rpm/download

Then you can proceed with the installation … you will get several errors (the installer is really not so user friendly).

# rpm -Uvh package-name.rpm

I would suggest you to forward the output to a file and then review it to eventually fix any problem.

Step 2: Configuration

In your /etc folder you should now have a file “afick.conf”. Below you can see my file (which is the default one for the current version). As you can see the file has a lot of comments which explain the meaning of each parameter. For further help you can consult the documentation

[start the conf file]

# afick config sample file

# $Id$

# see afick.conf documentation for more informations

####################

# directives section

####################

# binary values can be : yes/1/true or no/0/false

# database : name with full path to database file

database:=/var/lib/afick/afick

# history : full path to history file

history := /var/lib/afick/history

# archive : full path to directory for archived results

archive := /var/lib/afick/archive

# report_url : where to send the result : stdout/stderr/null

report_url := stdout

# report_syslog : send output to syslog ?

report_syslog := no

# verbose : (obsolete) boolean value for debugging messages

# use debug parameter below

verbose := no

# debug : set a level of debugging messages, from 0 (none) to 4 (full)

debug := 0

# warn_dead_symlinks : boolean : if set, warn about dead symlinks

warn_dead_symlinks := no

# follow_symlinks : boolean : if set, do checksum on target file (else on target file name)

follow_symlinks := no

# allow_overload : boolean : if set, allow to overload rules (the last rule wins), else put a warning

allow_overload := yes

# report_context : boolean : if set, display all changed attributes, not just those selected by rules

report_context := no

# report_full_newdel : boolean : if set, report all changes, if not set, report only a summary on top directories

report_full_newdel := no

# report_summary : boolean ; if set, report the summary section

report_summary := yes

# warn_missing_file  : boolean : is set, warn about selected files (in this config), which does not exist

warn_missing_file := no

# running_files : boolean : if set, warn about files changed during a program run

running_files := yes

# timing : boolean : if set, print timing statistics about the job

timing := yes

# ignore_case : boolean : if set, ignore case on file name

ignore_case := no

# max_checksum_size : numeric : only compute checksum on first max_checksum_size bytes ( 0 means unlimited)

max_checksum_size := 10000000

# allow_relativepath : boolean : if set, afick files, config and databases are stored as relative path

allow_relativepath := 0

# only_suffix : list of suffix to scan (and just this ones) : is empty (disabled) by default

# not very usefull on unix, but is ok on windows

# this will speed up the scan, but with a lesser security

# only_suffix :=

# the 3 next directives : exclude_suffix exclude_prefix exclude_re

# can be written on several lines

# exclude_suffix : list of suffixes to ignore

# text files

exclude_suffix := log LOG html htm HTM txt TXT xml

# help files

exclude_suffix := hlp pod chm

# old files

exclude_suffix := tmp old bak

# fonts

exclude_suffix := fon ttf TTF

# images

exclude_suffix := bmp BMP jpg JPG gif png ico

# audio

exclude_suffix := wav WAV mp3 avi

# exclude_prefix : list of prefixes to ignore

#exclude_prefix :=

# exclude_re : a file pattern (using regex syntax) to ignore (apply on full path)

# one pattern by line

#exclude_re :=

#################

# macros section

#################

# used by cron job (afick_cron)

# define the mail adress to send cron job result

@@define MAILTO root@localhost

# truncate the result sended by mail to the number of lines (avoid too long mails)

@@define LINES 1000

# REPORT = 1 to enable mail reports, =0 to disable report

@@define REPORT 1

# VERBOSE = 1 to have one mail by run, =0 to have a mail only if changes are detected

@@define VERBOSE 0

# define the nice value : from 0 to 19 (priority of the job)

@@define NICE 18

# = 1 to allow cron job, = 0 to suppress cron job

@@define BATCH 1

# (optionnal, for unix) specify a file system to mount before the scan

# it must be defined in /etc/fstab

#@@define MOUNT /mnt/dist

# if set to 0, keep all archives, else define the number of days to keep

# with the syntaxe nS , n for a number, S for the scale

# (d for day, w for week, m for month, y for year)

# ex : for 5 months : 5m

@@define ARCHIVE_RETENTION 0

# send nagios messages by NSCA (= 1 to allow, = 0 to block)

@@define NAGIOS 0

# address of the nagios server to send messages to

@@define NAGIOS_SERVER my.nagios.server.org

# NSCA configuration file

# @@define NAGIOS_CONFIG /etc/send_nsca.cfg

# name used for nagios passive check on the nagios server side

@@define NAGIOS_CHECK_NAME Another File Integrity Checker

# number c of the changes that are considered critical => nagios state CRITICAL

# (0 changes => nagios state OK; 0> and <c changes => nagios state WARNING)

@@define NAGIOS_CRITICAL_CHANGES 2

# path to nsca binary

# @@define NAGIOS_NSCA /usr/sbin/send_nsca

###############

# alias section

###############

# action : a list of item to check :

# md5 : md5 checksum

# sha1 : sha-1 checksum

# sha256 : sha-256 checksum

# sha512 : sha-512 checksum

# d : device

# i : inode

# p : permissions

# n : number of links

# u : user

# g : group

# s : size

# b : number of blocks

# m : mtime

# c : ctime

# a : atime

#all:        p+d+i+n+u+g+s+b+m+c+md5

#R:      p+d+i+n+u+g+s+m+c+md5

#L:      p+d+i+n+u+g

#P:         p+n+u+g+s+md5

#E:      ''

# action alias may be configured with

# your_alias = another_alias|item[+item][-item]

# all is a pre-defined alias for all items except "a"

DIR = p+i+n+u+g

ETC = p+d+u+g+s+md5

Logs = p+n+u+g

MyRule = p+d+n+u+g+s+b+md5

##############

# file section

##############

# 3 syntaxe are available :

# file action

#     to scan a file/directory with "action" parameters

# ! file

#     to remove file from scan

# = directory action

#             to scan the directory but not sub-directories

# file with blank character have to be quoted

#

# action is the list of attribute used to detect a change

= /  DIR

/bin        MyRule

/boot    MyRule

# ! /boot/map

# ! /boot/System.map

/dev p+n

# ! /dev/.udev/db

# ! /dev/.udev/failed

# ! /dev/.udev/names

# ! /dev/.udev/watch

! /dev/bsg

! /dev/bus

! /dev/pts

! /dev/shm

# to avoid problems with pending usb

# = /dev/scsi p+n

/etc        ETC

/etc/mtab ETC - md5 - s

/etc/adjtime ETC - md5

/etc/aliases.db ETC - md5

# /etc/mail/statistics ETC - md5

/etc/motd ETC

# /etc/ntp/drift ETC - md5

# /etc/urpmi/urpmi.cfg Logs

# /etc/urpmi/proxy.cfg Logs

# /etc/prelink.cache ETC - md5 - s

! /etc/cups

# ! /etc/map

# ! /etc/postfix/prng_exch

# ! /etc/samba/secrets.tdb

# ! /etc/webmin/sysstats/modules/

# ! /etc/webmin/package-updates/

# ! /etc/webmin/system-status/

/lib         MyRule

/lib64     MyRule

/lib/modules MyRule

# /lib/dev-state MyRule -u

/root MyRule

# ! /root/.viminfo

! /root/.bash_history

# ! /root/.mc

# ! /root/tmp

/sbin      MyRule

/usr/bin               MyRule

/usr/sbin             MyRule

/usr/lib MyRule

/usr/lib64            MyRule

/usr/local/bin    MyRule

/usr/local/sbin  MyRule

/usr/local/lib      MyRule

# /var/ftp MyRule

/var/log Logs

# ! /var/log/journal

= /var/log/afick Logs

# ! /var/log/ksymoops

# /var/www MyRule

# ! /var/www/html/snortsnarf

############################################

# to allow easier upgrade, my advice is too separate

# the default configuration file (above) from your

# local configuration (below).

# default configuration will be upgraded

# local configuration will be kept

########## put your local config below ####################

[end of conf file]

 Step 3: Usage

After the installation and configuration has completed you need to create the baseline (the database of hashes which will be used as “good” when comparing with the running system):

# afick –i

The database itself has a hash (MD5) so you can check manually if somebody hacked into the database by comparing the new hash with the one which I “painted in blue”.

So now that we have our baseline database of hashes there is only one operation required to keep your system under control: run the afick.pl command after any system update or any time you change a configuration file; you should run a compare before applying any changes and then an update after the change has been done.

Update command:

# afick.pl –u

Compare command:

# afick.pl –k

Now you need to take a look to the /etc/cron.daily/afick_cron file (a Bash script). In the distro I am using currently (CentOS 7) you need to check the /etc/anacrontab to see if you have an entry such as the following:

This show that all the scripts within the /etc/cron.daily folder will be executed daily starting at 3am with a delay of 5 minutes between a job and the following. If for some reasons the job (script) is not executed then anacron will try again! To know if anacron is working check the file /var/spool/anacron/cron.daily this should contain the current date … if not verify the logs and troubleshoot cron (that is not the topic of this article !). On CentOS 7 (and Red Hat 7) anacron is executed by crond through a script within cron.hourly. For more information check the related documentation.

Conclusion

For an absolute secure usage of afick:

• The database should be stored on a read-only media
• The afick command should be run from another “clean operating system” (in example from a bootable media): this is not an option in production environment but it is suggested in the afick.sourcforge.net website and we are reporting here for completeness.

We hope you found this article useful and that it will help in your daily effort to keep servers secure. Feel free to comment and share.

Author:

Francesco Campanini

Reference

A full comparison of the 3 HIDS application can be found in the following link:

https://www.sans.org/reading-room/whitepapers/detection/ids-file-integrity-checking-35327