Howto deploy VM using KVM & kickstart on CentOS/Red Hat - Part 2

Introduction

So far we have been able to deploy a VM using KVM but we need to reply to all the questions the installer will ask us.
As our final objective is to automate the VM deployment, a possible solution is to proceed with a manual installation, selecting the option we would like to replicate in our cluster deployment, and copy the automatically generated kickstart file.

In this post we will show you the steps to automate the tasks and in particular how to manipulate the kickstart file to allow the customization of the partitions.
Let's get started !

Kickstart file

We need to create a kickstart configuration file. The Red Hat documentation provide a good description about kickstart: link.

First of all you should remember the followings:
1) There are 4 important sections:

• command sections: this section must be the first and it contains something like the "expected question/answer" to all the installation questions.
• %packages sections: this section should be the second and it includes the list of packages to be installed.
• %pre %post script sections: these sections contain script to be executed before and after the installation. These sections can be omitted (not required).

2) If you omit some item, the installation may interrupt in a "wait for customer input" mode ... it may be tricky initially to find the right setup but it will be rewarding at the end!
3) Like for Bash scripts, lines which start with a # (pound sign) are treated like comments.

We can start by manipulating the kickstart file that anaconda created when we installed the host OS (se Part1). The file should be located under the root home directory: /root/anaconda-ks.cfg

This is my anaconda-ks.cfg file (depurated of the password hashes !!):

# cat /root/anaconda-ks.cfg

#version=DEVEL

# System authorization information

auth --enableshadow --passalgo=sha512

# Use CDROM installation media

cdrom

# Use text mode install

text

# Run the Setup Agent on first boot

firstboot --enable

ignoredisk --only-use=vda

# Keyboard layouts

keyboard --vckeymap=us --xlayouts=''

# System language

lang en_US.UTF-8

# Network information

network  --bootproto=dhcp --device=eth0 --onboot=off --ipv6=auto --no-activate

network  --hostname=localhost.localdomain

# Root password

rootpw --iscrypted HASHED-PASSWORD

# System services

services --enabled="chronyd"

# Do not configure the X Window System

skipx

# System timezone

timezone Europe/London --isUtc

# System bootloader configuration

bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda

autopart --type=lvm

# Partition clearing information

clearpart --all --initlabel --drives=vda

%packages

@core

chrony

kexec-tools

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%anaconda

pwpolicy root --minlen=6 --minquality=50 --notstrict --nochanges --notempty

pwpolicy user --minlen=6 --minquality=50 --notstrict --nochanges --notempty

pwpolicy luks --minlen=6 --minquality=50 --notstrict --nochanges --notempty

%end

As you can see the file is pretty easy and self explanatory. When it comes to using the kickstart file to create/install a VM guest, you need to have particular care for the network and the disks/partitions configuration. Then you have the package selection: the difficult part is to satisfy the dependencies.

Let's modify the partition entries ... we want to create our "standard" set of partitions. The boot partition cannot be created on an LVM partition. We will create a swap and a root partition. I like to suggest to create other partitions as well, such as the home and the var partitions as they may grow significantly and you may eventually need to increase the size or move out to a network file system, but this is not the scope of this article.

So this is my partition command list:

# Partition clearing information
clearpart --all --initlabel --drives=vda

####################################
part /boot --fstype ext4 --size=500
part swap --size=1024
part pv.01      --size=1000     --grow  --ondisk=sda
volgroup vg00 pv.01
logvol / --vgname=vg00  --fstype=xfs  --size=10240 --name=lv_root

We start by clearing any existing partitions (clearpart --all --initlabel --drives=vda) on the virtual drive we will use (vda) which we have created with this command (see Part 1 for details):

# qemu-img create -f qcow2 /opt/VM/CentOS-server1/guest.qcow2 32768M

Formatting '/opt/VM/CentOS-server1/guest.qcow2', fmt=qcow2 size=34359738368 encryption=off cluster_size=65536 lazy_refcounts=off

Then we provide a list of the partition, indicating the size and type of the filesystem. I didn't use all the space available on the virtual drive (32GB) as I would like to be able to dinamically increase the space of the existing partitions later or create new partitions.

About the %post section, the following is an example of a command you may find useful to modify an entry in a script file.

%post
sed -i 's/ONBOOT=no/ONBOOT=yes/g' /etc/sysconfig/network-scripts/ifcfg-eth0
%end

At the end of the installation you should reboot the guest VM.
This is the complete kickstart file I have been using:

#version=DEVEL
# System authorization information
auth --enableshadow --passalgo=sha512
# Use CDROM installation media
cdrom
# Use text mode install
text
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=vda
# Keyboard layouts
keyboard --vckeymap=us --xlayouts=''
# System language
lang en_US.UTF-8

# Network information
network  --bootproto=dhcp --device=eth0 --onboot=off --ipv6=auto --no-activate
network  --hostname=localhost.localdomain

# Root password
rootpw --iscrypted HASHED-PASSWORD
# System services
services --enabled="chronyd"
# Do not configure the X Window System
skipx
# System timezone
timezone Europe/London --isUtc
# System bootloader configuration
bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda
# Partition clearing information
clearpart --all --initlabel --drives=vda

####################################
part /boot --fstype ext4 --size=500
part swap --size=1024
part pv.01      --size=1000     --grow  --ondisk=sda
volgroup vg00 pv.01
logvol / --vgname=vg00  --fstype=xfs  --size=10240 --name=lv_root

%post
sed -i 's/ONBOOT=no/ONBOOT=yes/g' /etc/sysconfig/network-scripts/ifcfg-eth0
%end

%packages
@core
chrony
kexec-tools

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%anaconda
pwpolicy root --minlen=6 --minquality=50 --notstrict --nochanges --notempty
pwpolicy user --minlen=6 --minquality=50 --notstrict --nochanges --notempty
pwpolicy luks --minlen=6 --minquality=50 --notstrict --nochanges --notempty
%end

Verify the Kickstart file

You can make sure your Kickstart file is valid by using “ksvalidator”.

Install ksvalidator:
# yum install pykickstart

Run ksvalidator on your Kickstart file:
# ksvalidator /path/to/anaconda-ks.cfg

Deploy the VM

As explained along part one, to create and deploy a VM we will use the command virt-install. The additional option for the automation with kickstart file are:
--initrd-inject=/opt/VM/ks.cfg
--extra-args "ks=file:/ks.cfg"
We use the initrd-inject, to tell anaconda to use the kickstart file in the ramdisk. The extra args help anaconda later to "reply" the questions automatically.

This is the command I have tested so far:
# virt-install --name CentOS-LDAP-server --ram 1024 --disk path=./ldap-guest.qcow2,size=32 --vcpus 1 --os-type linux --os-variant centos7.0 --network bridge=virbr0 --nographics --location /opt/VM/CentOS-7-x86_64-DVD-1611.iso --extra-args "console=ttyS0" --initrd-inject=/opt/VM/ks.cfg --extra-args "ks=file:/ks.cfg"

We got our VM created and deployed automatically without any user interation. In the next posts I will explore a basic setup of Ansible in a docker and following, how to deploy VM using Ansible.

RHCSA Howto ... Tips and Tricks - Episode 2

Introduction

In this episode of my journey to the desired certification (RHCSA), we are going to provide some information about one of the topic of the chapter "Understand and use essential tools":

• access a shell prompt and issue commands with correct syntax

There is no "official" list of commands which Red Hat suggest you to know to pass the certification as it does not really matter how you solve the question; so every command or option you use, which is available in the distribution ISO and you are familiar with, will be good.

Feel free to add your comments and recommendation; we will be happy to reply and or update the post!

Let's get started !

access a shell prompt and issue commands with correct syntax

There are 3 type of commands in Linux:

• internal
• external
• aliases

An internal command is part of the shell itself. An example of internal command is echo which print to the standard output whatever is the argument.

An external command is an executable file located somewhere in the fylesystem. An example of external command is /bin/hostname which print the name of your system.
To know if you are calling an internal or external command you can use the command type, while with which you will know the exact command the shell will use:

$ type ls
ls is aliased to `ls --color=auto'

$ which ls
alias ls='ls --color=auto'
        /usr/bin/ls

The output shows that there is an alias called ls and an executable file called /usr/bin/ls. Aliases have precedence on both internal and external commands so Bash will use the alias when we will use the ls command (see below for more info on the aliases).

If you want to create a custom command, you can use an alias. If for example you frequently use a specific command with some option (like ls -al or df -h) you can use the Bash command alias to create a "custom command" (alias). There are some useful aliases available by default in most distribution. To check which are available for your user just type alias at the Bash prompt:

# alias

alias cp='cp -i'

alias egrep='egrep --color=auto'

alias fgrep='fgrep --color=auto'

alias grep='grep --color=auto'

alias l.='ls -d .*'

alias ll='ls -l'

alias mv='mv -i'

alias rm='rm -i'

alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'

To create a new alias you just have to type "alias" followed by the command name you want to use, equal the command that it will execute (as you can see in the output of the raw alias command). Let's look at an alias example:

# df

Filesystem              1K-blocks     Used Available Use% Mounted on

/dev/mapper/centos-root 385867688 58806536 327061152  16% /

devtmpfs                  3885068        0   3885068   0% /dev

tmpfs                     3901000      320   3900680   1% /dev/shm

tmpfs                     3901000    42304   3858696   2% /run

tmpfs                     3901000        0   3901000   0% /sys/fs/cgroup

/dev/sda2                84907480 30177444  50393860  38% /home

/dev/sda1                  508588   363028    145560  72% /boot

tmpfs                      780204       16    780188   1% /run/user/1001

tmpfs                      780204       24    780180   1% /run/user/1000

# alias df='df -h'

# df

Filesystem               Size  Used Avail Use% Mounted on

/dev/mapper/centos-root  368G   57G  312G  16% /

devtmpfs                 3.8G     0  3.8G   0% /dev

tmpfs                    3.8G  320K  3.8G   1% /dev/shm

tmpfs                    3.8G   42M  3.7G   2% /run

tmpfs                    3.8G     0  3.8G   0% /sys/fs/cgroup

/dev/sda2                 81G   29G   49G  38% /home

/dev/sda1                497M  355M  143M  72% /boot

tmpfs                    762M   16K  762M   1% /run/user/1001

tmpfs                    762M   24K  762M   1% /run/user/1000

As you can see from now on, typing df I will get the output of "df -h" which is much more easy to read (human readable format) than the raw command.

The issue here is that whenever you close the shell or restart your server, the alias you manually enforced will disappear. To make the change persistend and customize many other shell behaviour there are 4 files to consider:

• ~/.bashrc    -   user specific config, read by a non-login shell
• ~/.bash_profile   -   user specific config, read by a login shell
• /etc/bashrc   -   system config, read by a non-login shell
• /etc/profile   -   system config, read by a login shell

I suggest you to have a look at the content of each of the mentioned files to discover which configuration are applied in each of the situation. Be aware that: when you open a login shell, Bash read the /etc/profile file first and then the ~/.bash_profile. The user specific settings have priority and override the system settings. The same for the "non login" shell.

Some more alias-stuff ! If you just created an alias for the current session, you can remove it by typing unalias followed by the command name (unalias df). You can also force Bash to use the internal or external command instead of the alias by unticipating the command with a backslash (\df).

Other interensting topic about the shell is the environment variables. A variable is a way to assign a value (which can be a string or a number) to a label (i.e. a name) which can only start with a letter. As a style rule it is better to use capital letters for variable names. Variables that are "available" within a working shell are called environmental variable. You can get a list executing the command env.
When you create a Bash script and you need to use multiple times the same value, it is a good idea to create a variable and call it within the script. As the variable has been created in the script, it will be available just within it (you need to export it to make it available within the running shell, outside the scope of the script).
You can permanently define environment variable by adding or modifying them within the Bash script we have been reviewing above. You can create or assign a value to a variable for the current session in the following way:

$ TEST="This is a test"
$ echo $TEST
This is a test

As you can see, you can use echo to print to the value of a variable to the standard output (just precede the variable name with a $).

To make your life easier you can recall commands that you have executed before by using the Bash internal command history. The command history is stored in RAM until the session is closed; at that point the list is appended to the ~/.bash_history text file.
So typing history you can get the list printed in your shell. Each entry start with a number to make it easy for you to execute the same command again:

$ history
   .........
   88  su
   89  df
   90  su
   91  alias
   92  su
   93  env
   94  TEST="This is a test"
   95  echo $TEST
   96  which ls
   97  type ls
   98  type /usr/bin/ls
   99  type ls
  100  history

$ !99

type ls

ls is aliased to `ls --color=auto'

You can scroll backward the list using CTRL-R. With CTRL-R you enter into an interactive session... type part of a command you already entered and see what happen!!

You can also call a command by typing ! followed by part of the command.

A good security measure is to clear the root history when closing the session. In this way, if an cracker succeed to compromise your system, there will be no traces of any possible "clear" password you typed within your commands (in example when you login to a database by providing the password while starting the CLI ... dangerous!!). It is also something crackers do to remove their footprints. Clearing the history is a 2 steps process: you need to clear the commands in memory (history -c) and you need to clear the content of the file ~/.bash_history (history -w).

RHCSA Howto ... Tips and Tricks - Episode 1

Introduction

As I recently started to study to improve my knowledge in Red Hat Enterprise Linux (RHEL), I thought it was a good idea to use the Blog like a notepad where I will write information on the topics of the certification, links to external websites which provide tutorials and whatever matters with the topic!

The RHCSA certification is the introductory Red Hat System Administrator certification and a mandatory step for the Red Hat Certified Engineer (RHCE) certification.

As the topic are subject to change by Red Hat, if you want to take my same journey, I suggest you to check the following link and print-out the "Study Point".

At the moment of writing this post (July 2017), the topics are divided into the following chapters:

• Understand and use essential tools
• Operate running systems
• Configure local storage
• Create and configure file systems
• Deploy, configure, and maintain systems
• Manage users and groups
• Manage security

As written at the end of the webpage of the mentioned Red Hat link, quote "Red Hat reserves the right to add, modify, and remove objectives. Such changes will be made public in advance through revisions to this document" ... unfortunately there is no "revision number" in the document itself so ... check the document multiple times during the preparation.

I would not suggest to take this certification journey if you don't already have a couple of year of experience with Linux! You will possibly waste time and money and this certification is not cheap... currently the exam cost around 470£ + VAT ... quite expensive!

Training Material

After downloading the topic list you need to figure out where to get information, which training material to study, which books, videos ...
I love Linux for many reasons, one is that you can find so many information for free. Perhaps when it comes to a certification, in my opinion you need to have an official certification guide: this will help you to divide and conquer each topic, have a set of exercises and labs and have one or more exam simulation to verify if what you learnt is enough to pass the exam or if you need to review any topic.

I have been able to review several books (thanks to some friends) and the one I found perfect for my study-style is:
Cert Guide - Red Hat RHCSA/RHCE 7 - by Sander Van Vugt

As I like to read the same topic from different authors I am using these other books as well:

The Linux Bible - Ninth Edition - by Christopher Negus
which is a good "comprehensive" book full of tutorials and resources.

How Linux works, what every superuser should know - by Brian Ward
more about troubleshooting but with interesting information even for a System Admin

Note: I don't have any interest/profit by mentioning the above books but I found good to suggest them and promote the authors work.

As you will read on all the books, for any command you will come across during your journey, you should have a proper look to the man page and to the info page (if any): these resources will be the only one available to you during the exam ... so better to know how to use it to your advantage!!

Lab and first experiences

That's the first step! What Sander suggest is to create 3 virtual environments:

• server with CLI
• server with GUI
• server for LDAP authentication and other shared services

You can create an account and download the official Red Hat 7 image and then create the 3 VMs or, more simply, download CentOS 7. At the moment of writing (July 2017) it seems like the exam will focus on versions 7.1 and 7.2 of Red Hat Enterprise Linux. You can use CentOS with the same version number as basically CentOS is a Red Hat Enterprise "stripped of the Red Hat" logo.
So what is the difference between Red Hat and CentOS and why companies are paying for Red Hat? When it comes to the ISO, the same versions of the 2 distribution has the same packages version and pretty similar configurations; they are basically interchangeable.
When it comes to apply patches, CentOS provides free access to its repositories while to get access to Red Hat you need to pay an annual fee. The fee does not entitle you only to get access to the repository but also to get free high quality, 24H 7/7 365 support! That's the main difference ... an enterprise level support which will take care of providing you quick and expert answer to your queries.
So, setup your VMs and start to play around with the basics! If you already have a Linux machine running CentOS you may find useful to read my post about deployment of VM using KVM.

In my next post I will focus on the objective of the first chapter "Understand and use essential tools":

• access a shell prompt and issue commands with correct syntax
• use input-output redirection
• use grep and regular expression to analyze text
• access remote system using ssh
• log in and switch users in multiuser targets
• archive, compress, unpack, and uncompress files using tar, star, gzip and bzip2
• create and edit text files
• create, delete, copy, and move files and directories
• create hard and soft links
• list, set, and change standard ugo/rwx permissions
• locate, read and use system documentation including man, info and files in /usr/share/doc

In the post I will introduce the commands that you need to know focusing on the tasks which maybe required in the exam.

Recommendations

If you decide to take a journey into a certification the most common recommendation is to schedule your study activities and have a target day for the exam! There is no better recommendation. I know that sometimes life and work does not help to keep you on track but without any target milestone you will never meet the desired result!
Please feel free to add comments; they may be helpful for other geek who would like to learn the topic! I hope you enjoyed reading this post.
Stay tuned!

5 Linux command line tips & tricks

In this post we will show you some useful commands which you can use within your BASH shell.

1) repeat the last command argument: lets say that you need to check the content of a file without editing it and you use the command:
       # cat /etc/ssh/sshd_config
then you find out that you need to edit the same file. Using the following bash shortcut you can repeat the argument of the last command and use it as argument for the new command:
       # vi !$

2) you need to check the content of a specific line or range of lines of a script or configuration file. There are several ways to achieve this:
- use less and jump to the line you need to review ( if you want to go to line 27, just open the file with less and type 27g ). The same option is valid using vi
- print only the content of the specified lines using sed (my favorite way)
       # sed -n -e 25,30p name_of_the_file
the above command will print the line between 25 and 30

3) you need to determine which were the latest 10 users logged to your server:
       # last -a | head

4) You want to disable your printer service (as this is a server and does not have any printers connected ...):
a) find if "cups" service is running:
       #lsof -i | grep -i cups
   alternatives:
       #service cups status
       #systemctl status cups
b) if you find the service started and enabled you need to stop it first:
       #systemctl stop cups
c) now you need to disable so it will not start at next system reboot:
       #systemctl disable cups
That's it. The same procedure can be applied to any other services. I have been using commands tested on CentOS7. These commands will run on Red Hat 7 and on most of the other up to date Linux distributions.

5) You need to convert a unix text file into a windows (replace end of the line character). While using unix2dos is the fastest way, the utility is not available as default in all the distributions (perhaps it is not included in the minimal CentOS 7 installation). More than this, why you have to install a package to do a specific task when you can adopt the creative "unix-way"... there are several unix-way, I am going to provide you a few, but with your fantasy and experience you may create new one (and choose which one you prefer)
       #awk 'sub("$","\r")' unix.txt >  win.txt
awk is really powerful ... in this example it is gonna "sub"-stitute the character at the end of each line ($) with the windows end of line (\r) from the file unix.txt. The output is forwarded line per line to the file win.txt
       #sed 's/$/\r/' < input.txt > output.txt
sed is possibly more difficult compare to awk but the above command shows a really similar structure. Perhaps both the commands (awk and sed) are manipulating regular expressions and that's waht we need. Again, s stand for substitute ... what ? the end of the line character ($) with \r ...

Notes:
Other useful commands are published in the following posts of Cyber Sec Linux:
How to hack your Linux Wireless interface in 5 steps
8 Daily Tasks every System Administrator need to know

Please, feel free to share and comments !

8 Daily Tasks every System Administrator need to know

In the daily task of a Linux system administrator the most common duties are related to maintain performance, reliability and security to the systems. In this post I will describe the main daily tasks.

Let's get started !

Checks for software updates

A priority is to keep your system updated and patched (especially with those related to security). It's a good attitude to spend some time every day reading forum about systems security! It could be necessary to modify system configuration scripts and to apply these new changes. To keep your system up-to-date there are different package system such as RedHat RPM (used by RedHat, Fedora, CentOS...) and Debian dpkg (used by Debian, Ubuntu, Linux Mint...). Both systems keep a database of all the software installed in your system; you can use their specific tools to keep your system update. Checking for available updates is good ... but for the installation you need to pay some more attention as the behavior of your system can be unexpected (cause of incompatibility or bugs in the updates !!). Better to test the updates within a sandbox before applying to a production environment (VM are perfects for such type of tests).

Perform Disk Management

Check for any disks and RAID issues (it's better to have a monitoring system that inform you whenever there is a new issue instead of discovering it too later!!). There are some basic commands you need to know and use daily, such as du ("disk usage": is a standard program used to estimate file space used under a particular file system), df (display the amount of available disk space for file-systems), fdisk (to manage disk partitions),...

• several application use the /tmp folder as a cache. Sometimes (often) the "cache" is not cleaned properly and the system become unresponsive!! /tmp folder can have several folders in it ... to check the folder size better to use the du command with the "d" option (after d I placed a 1 "the number", not the letter l "the letter" ;) ) which allows you to select how much to dig within the tree: du -h -d 1 /tmp
• monitor the system using nagios or other system monitoring software to be alerted when the free space is running low. You can also install SNMP and send traps to an SNMP manager.

Perform Data Backup

It can be scheduled daily or weekly using many different tools); you need to have a good Disaster Recovery Procedure (DRP) in case you have to restore data or a totally damaged systems!!! In these days of Ransomware: for more information read my post about Ransomware.
The most easy tools are TAR and CPIO but they need to be tuned and can be tricky. You can implement several backup procedure such as: Full, Incremental, Differential.

Perform Data-Base check

If you have to manage a data-base server (MySQL for example) you can use the command line shell or many other different tools such as: MySQL WorkBench, MySQL WebMin module (less powerful than WorkBench in my opinion but still a complete tool), phpMyAdmin (my favorite web tool, really easy to use and it handles every aspect of managing a MySQL database server).

Perform Server Load control

You need to check CPU, memory, I/O, network devices. To perform such control you can use a monitor software or command line tools such as top (that display system resources and information) ps (that display all the systems process...you can use a combination of ps and grep to filter what you are searching for)...

Check your log files

you need to periodically check log files (/var/log) and system messages. When you really know your system, it will be quite easy to find an issue checking how big are getting your log files or how often they get "rotated" !!

User and group management

It could be necessary to add or remove user, to change their group or the user disk quota. Other than command line there are many different tools helping you in this task.

Verify network performance

It often happens that this task is handled by monitoring software or external devices (manageable switches that send log information to an SNMP server)