Notes for the CompTIA Linux+ certification.

CompTIA Linux+ XK0-004 Certification Exam Notes

Recommended Pre-Req's:

CompTIA A+ Certification
CompTIA Network+ Certification
LPI linux Essentials Course

Hardware and System Configuration

Boot Process

Kernel Ring Buffer hold volatile logs in RAM
dmesg
journalctl -k -k for kernel messages

GRUB - Grand Unified Boot Loader
Stage 1 - MBR in first 512 bytes of boot disk boot.img Stage 1.5 - core.img Stage 2 - /boot/grub files

Legacy Grub

grub-install [device]

use TARGET SOURCE as [ device ]

> findmnt /boot
    TARGET SOURCE
    /boot    /dev/vda1

> grub-install /dev/vda1

Also

grub-install '(hd0)' # install on first drive in system

Or grub then find
/boot is considered / in grub. (relative!)

This is normally done from a live cd/usb to install GRUB on a new disk.

Installing GRUB on running linux session is potentially dangerous

as root:
grub
help
find grub/stage1
quit

GRUB2

MBR (Master Boot Record)

• Only supported 26 total partitions
• Actually 4 partitions with one being extended to 23 logical partitions
• Each partition limited to 2TB

GPT (GUID Partition Table)

• Supports 128 partitions
• Partitions in the ZB (petabyte) range
• Needs UEFI (Unified Extensible Firmware Interface)
  • replacement for Traditional BIOS
  • requires 64bit OS
  • prevents unauthorized OS from booting on system

on Red Hat distros = grub2-<command> on Debian distros = grub-<command>

grub2-editenv list
cat /etc/default/grub
code /etc/default/grub

Change SUBMENU to false
grub2-mkconfig to read updated grub file and create new GRUB2 config file
less /boot/grub2/grub.cfg
ls /etc/grub.d

Interacting with GRUB

from GRUB menu
[A] append options to kernel boot line
[C] key will bring up GRUB Command Line
help
install - advanced and detailed
setup - automates much of the process
in GRUB setup (hd0) - reinstalled GRUB
reboot to check

GRUB2
[E] to edit GRUB menu item
use arrows to get to end of arrow line
need to use systemd - systemd.unit=rescue.target
f10 or ctrl+x
reboot
[C] to get to command line
ls
ls (hd0,1)/ # looks at entries under that drive partition

boot one step at a time
set root=(hd0,1)
linux /boot/vmlinuz- (tab complete)
select newest (or other)
linux /boot/vmlinuz-4.13.0-43-generic root=/dev/vda1
initrd /boot/initrd.img-4.13.0-43-generic
boot

initramfs

lsinitrd - view contents of initramfs file
dracut - create new initramfs for kernels on system. Add or remove modules and drivers from initramfs builds
/etc/dracut.conf - primary config file for dracut - typically points to /etc/dracut.conf.d
View some info
lsinitrd | less
lsinitrd
cp /boot/initrd<xxx> /boot/initrd<xxx>.bak
ls /boot
dracut --list-modules

remove FCOE modules (Fiber Channel Over Ethernet)
dracut -o "fcoe fcoe-uefi" -M -f
reboot
lsinitrd
lsinitrd | less # verify the fcoe modules are removed

This is not persistent across kernel updates... so need to make a config file
cd /etc/dracut.conf.d/
ls
code no-fcoe.conf

omit_dracutmodules+="fcoe fcoe-uefi"

sudo dracut -f

PXE Boot
_ Preboot Execution Environment - method to boot system through a network interface card _ TFTP - Trivial File Transfer Protocol

Install, Configure, and Monitor Kernel Modules

Working with Kernel Modules

uname displays info about the current kernel

uname -rm
uname -a

lsmod displays listing of all currently loaded kernel modules modinfo displays info about a specific module modprobe dynamically loads and unloads kernel modules at runtime (CAREFUL)

lsmod
modinfo floppy
modprobe -r floppy
lsmod
modprobe floppy # loads dependents if they exist=
lsmod

Third party modules...

More Kernel Module Commands

ls /lib/modules/$(uname -r)/ List kernel modules directory for the currently running kernel
rmmod removes module from currently running kernel
insmod inserts module to currently running kernel. Does NOT take into account dependencies.
depmod generates a list of dependencies and map files
/etc/modprobe.d/ directory location where kernel modules can be listed in blacklist files

(centOS)

lsmod
rmmod floppy
lsmod

insmod floppy
ls /lib/modules/$(uname -r)/kernel/drivers/
ls /lib/modules/$(uname -r)/kernel/drivers/block
insmod /lib/modules/$(uname -r)/kernel/drivers/block/floppy.ko.xz
lsmod

rmmod virtio_blk # fails due to dependencies

ls /etc/modprobe.d/
cat /etc/modprobe.d/dccp-blacklist.conf

vim modname-blacklist.conf

blacklist <modulename>
blacklist <dependency modulename>
blacklist <dependency modulename>

nano /etc/modprobe.d/floppy-blacklist.conf
blacklist floppy
/etc/modprobe.d/

If using modules from outside the distro, these may be useful from time to time when adding a new module to the system

ls /lib/modules/$(uname -r)
depmod

Dealing with Kernel Panic

Kernel Panic is a safety measure to prevent further damage to the system _ Hardware failure _ Software bug, device driver bug, OS bug

Debugging

kdump if it is installed will write ram logs to disk
can be analyzed with gdb or crash by Red Hat
proc/sys/kernel/panic file contains a number of seconds that a system will wait to reboot. Default is 0 and will not reboot.
/etc/sysctl.conf config with various kernel params that can be altered

cat proc/sys/kernel/panic
echo 5 > proc/sys/kernel/panic
cat proc/sys/kernel/panic

proc file system only exists in mem while the system is running
to make the kernel reboot log permanent - need to edit the /etc/sysctl.conf

code /etc/sysctl.conf
kernel.panic=15
cat proc/sys/kernel/panic
systctl -p # reparse the sysctl file
cat proc/sys/kernel/panic

Networking

Configure and Verify Connection Parameters

Recommended to review the Basic Networking video from LPI Linux Essentials Course

ip addr show
ip a s
ping -c 1 example.com
dig example.com
cat /etc/resolv.conf
host example.com
dig @8.8.8.8 example.com
cat /etc/hosts #can add ip and name then whatever here

ip route show
ip r s
ip a s

man route
man nmcli
man netstat
man ss
netstat -tlnp # tcp listen numeric showPIDs

Networking Fundamentals

TCP : Transmission Control: Protocol (Relies on a send and acknowledge system)
UDP: User Datagram Protocol: Stateless - no acknowledgement or resend. "fire and forget system"
ICMP: Internet Control Message Protocol (used fro sending error messages)

• often used to query a network device to check availability
• examples that use ICMP are ping and traceroute

IP Addresses

class ranges
RFC 1918 memorize table

Network Mask: defines a logical network called a subnet that indicates the start and end range of IP addresses
Classless inter domain registration (CIDR)
Subnet Masks

• Class A 255.0.0.0/8
• Class B 255.255.0.0/16
• Class C 255.255.255.0/24

Private IP Address Ranges: used for internal networks.

• Prevents need for every host to have an IP address from a central authority
• Not publicly routable - not on the internet
• internet just sees the public IP on the outside of the router

commit to memory IP Address range table with # of hosts, CIDR notation, class description

Network Gateway

• destination for traffic with no matching route or not intended for local network

Broadcast Address

• used to broadcast messages to all hosts on a particular network
• Ends with the last octet containing 255
• ex. 192.168.0.255 is the broadcast address for 192.168.0.0/24 (only hosts on 198.168.0.0 network will see them)

NetworkManager

https://www.linux-kvm.org/page/Main_Page

nmcli dev show

Device Naming

en ethernet wl wireless

eno1 (o)board ens1 pcie hotplug (s)lots, indexed by BIOS enp2s0 devices in specific (p)hysical locations eth0 older traditional naming

nmcli dev show

GENERAL.HWADDR is the MAC Address associated with the NIC

Deactivate the connection

nmcli con show
nmcli con down "Wired connection 2"
nmcli con show
nmcli device status

Reactivate

nmcli con show
nmcli con up "Wired connection 2"
nmcli con show
nmcli device status

nmcli con delete "Wired connection 2"
nmcli dev show enp6s0
nmcli con add con-name "backup" type ethernet \
    ip4 <interface static ip address 192.168.122.75/24> \
    gw4 <gateway address 192.168.122.1> \
    ifname enp6s0
    autoconnect

nmcli dev show enp6s0

    ax@n:~$ nmcli dev show enp6s0
    GENERAL.DEVICE:                         enp6s0
    GENERAL.TYPE:                           ethernet
    GENERAL.HWADDR:                         38:D5:47:C8:A2:BC
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          100 (connected)
    GENERAL.CONNECTION:                     TBSC Ethernet
    GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/4
    WIRED-PROPERTIES.CARRIER:               on
    IP4.ADDRESS[1]:                         192.168.167.24/24
    IP4.GATEWAY:                            192.168.167.254
    IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.167.254, mt = 100
    IP4.ROUTE[2]:                           dst = 192.168.167.0/24, nh = 0.0.0.0, mt = 100
    IP4.ROUTE[3]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000
    IP4.DNS[1]:                             192.168.167.2
    IP4.DOMAIN[1]:                          thebiosciencecenter.local
    IP6.ADDRESS[1]:                         fe80::5588:c2c3:42c9:ec73/64
    IP6.GATEWAY:                            --
    IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 100
    IP6.ROUTE[2]:                           dst = ff00::/8, nh = ::, mt = 256, table=255

nmcli con delete "TBSC Ethernet"
nmcli con add con-name "TBSC Ethernet" type ethernet ifname enp6s0
nmcli dev show enp6s0

    GENERAL.DEVICE:                         enp6s0
    GENERAL.TYPE:                           ethernet
    GENERAL.HWADDR:                         38:D5:47:C8:A2:BC
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          100 (connected)
    GENERAL.CONNECTION:                     TBSC Eth
    GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/5
    WIRED-PROPERTIES.CARRIER:               on
    IP4.ADDRESS[1]:                         192.168.167.24/24
    IP4.GATEWAY:                            192.168.167.254
    IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.167.254, mt = 100
    IP4.ROUTE[2]:                           dst = 192.168.167.0/24, nh = 0.0.0.0, mt = 100
    IP4.ROUTE[3]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000
    IP4.DNS[1]:                             192.168.167.2
    IP4.DOMAIN[1]:                          thebiosciencecenter.local
    IP6.ADDRESS[1]:                         fe80::a853:f22a:16fa:8f92/64
    IP6.GATEWAY:                            --
    IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 100
    IP6.ROUTE[2]:                           dst = ff00::/8, nh = ::, mt = 256, table=255

nmcli edit
nmcli con mod <conName> ipv4.dns "192.168.122.1"
nmcli -f ipv4.dns con show <conName>

ip a s

#set temp IP
ip add add 192.168.122.76/24 dev enp6s0
ip addr show enp6s0

ip addr del 192.168.122.75/24 dev enp6s0
ip addr show enp6s0

ip link set enp6s0 down
ip link set enp6s0 up
ip addr sho enp6s0
# new IP is gone

# set a default route
ip route add default via 192.168.122.2 dev eth0
ip route del default via 192.168.122.1 dev eth0
ip route show

hostnamectl set-hostname "centos7"
hostname

Legacy Tools

apt instal net-tools

Change the IP address of enp6s0 (breaks the internet) ifconfig enp6s0 192.168.167.21

ifdown enp6s0
ifup enp6s0

configs have been reset and enp6s0 has active internet connection once again.
i.e., stateless ip assignement connect to another network in the organization but not thorugh the default gateway

route add -net 192.168.10.0 netmask 255.255.2555.0 gw 192.168.122.25

Testiing Connectivity

ping
ping6
traceroute
traceroute6

ping loopback

nmcli dev show lo
ping 127.0.0.1

# just send 5 (count = 5)
ping -c 5 127.0.0.1

# implied ipv4 address - if explicit do:
ping -4 -c 2 127.0.0.1

# for ipv6
nmcli dev show lo
ping -6 -c 2 ::1

Hops and traceroute

sudo apt install inetutils-traceroute
traceroute 127.0.0.1
# default gateway test broken but should be 1 also

nmcli dev show enp6s0
    GENERAL.DEVICE:                         enp6s0
    GENERAL.TYPE:                           ethernet
    GENERAL.HWADDR:                         38:D5:47:C8:A2:BC
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          100 (connected)
    GENERAL.CONNECTION:                     TBSC Ethernet
    GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/11
    WIRED-PROPERTIES.CARRIER:               on
    IP4.ADDRESS[1]:                         192.168.167.24/24
    IP4.GATEWAY:                            192.168.167.254
    IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.167.254, mt = 100
    IP4.ROUTE[2]:                           dst = 192.168.167.0/24, nh = 0.0.0.0, mt = 100
    IP4.ROUTE[3]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000
    IP4.DNS[1]:                             192.168.167.2
    IP4.DOMAIN[1]:                          thebiosciencecenter.local
    IP6.ADDRESS[1]:                         fe80::8dda:5ece:f92a:5d4e/64
    IP6.GATEWAY:                            --
    IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 100
    IP6.ROUTE[2]:                           dst = ff00::/8, nh = ::, mt = 256, table=255

traceroute 192.168.167.254
#broken - probably from previous tinkering

route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    0.0.0.0         192.168.167.254 0.0.0.0         UG    100    0        0 enp6s0
    169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 enp6s0
    172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
    192.168.167.0   0.0.0.0         255.255.255.0   U     100    0        0 enp6s0

Test for hops to something out in the wild.

# one of Googles DNS servers
traceroute 8.8.8.8
    traceroute to 8.8.8.8 (8.8.8.8), 64 hops max
      1   192.168.167.254  0.659ms  0.158ms  0.170ms
      2   23.31.113.30  1.216ms  0.661ms  0.945ms
      3   96.120.0.145  12.203ms  12.099ms  9.423ms
      4   69.139.207.101  13.660ms  10.364ms  8.588ms
      5   162.151.130.117  10.636ms  8.998ms  13.947ms
      6   68.86.95.237  19.992ms  17.795ms  20.452ms
      7   68.86.83.94  17.979ms  18.793ms  18.799ms
      8   96.87.8.210  21.856ms  20.564ms  17.581ms
      9   108.170.254.65  18.298ms  22.926ms  18.512ms
     10   8.8.8.8  18.534ms  18.076ms  23.627ms

# tracepath is like traceroute but does not always need elevated priv
# uses UDP packets

tracepath google.com
# UDP scannign not allowed at some point and trace fails.

netstat
netstat -tl # where are we listenting to TCP packets?

sudo netstat -utpl #udp tcp port listen

netstat -r
netstat -rn

#on modern linux use ss (socket statistics)
ss -tl
ss -ul

# no route table option though

Basics of DNS Resolution from Linux

cat /etc/hosts
cat /etc/hostname
ping -c 1

host-utils

host localhost
# really querrying A reccords (address)

# will also get mx mail records
host google.com

dig google.com
dig @8.8.8.8 google.com

dig -t MX google.com
dig -t A google.com

getent hosts

Bonding and Link Aggregation

Configuration that treats 2 or more network interfaces as a single interface aka teaming

Bonding Modes
Mode=1 active backup policy

• all nics to backup while one remains active

Mode=2 XOR policy

• selects an interface based on result of an xor operation

Mode=4 IEEE 802.3ad policy

• creates aggregation groups for which included interfaces share the speed and duplex settings
• ex. Bond 2 1GB NICs in Mode4 to have a 2GB NIC
• requires switch that can support at other end also

Mode=5 adaptive transmit load balancing policy

• ensures outgoing traffic distribution is according to the load on each NIC, and that current NIC receaves all incoming traffic

Bridging

Combines 2 or more networks into a new logical network
Often used in virtualization settings wher virt guest network is configed to communicate on the same network as the host system
Needs the bridge-utils library and tools

sudo apt install bridge-utils

# Create bridge interface
brctl addbr br0

# Add ni(c) to bridge
brctl addif br0 [veth0]

# check status
brctl show

Manage Storage in a linux Environment

# Psuedo File Systems
/proc
/sys

cat /proc/cpuinfo
man proc

# Main File System
mount # list all mount points
lsblk
fdisk -l /dev/diskname

# Legacy style MBR
lsblk
fdisk /dev/sda # work on sda in fdisk interactive
parted /dev/sdb # work on drive in parted

gdisk #for GPT

#swap space
# swap file or swap partition -> partition more performant

fdisk
gdisk
parted
mkswap
swapon
swapoff

fdisk sda
    p
    n
    8200 # linux swap partition

lsblk

mkswap -L SWAP /dev/sda2
free -m
swapon -a
swapon -U <UUID>
swapon -L SWAP # label

#add SWAP to /etc/fstab
## UUID= or LABEL=SWAP swap swap defaults 0 0 #[defaults, dump(0), fs checking (0)]

swapoff

# File Systems
#   non-journaling (ext2)
#   journaling (ext3, ext4m XFS)

# Btrfs - uses CoW / copy on write
#   uses subvolumes - stores changes in copy of file - can be accessed like a directory
# FAT File Allocation Table
#   linux uses vfat for longer file names
# exFAT - larger than 2GB files supported

mkfs -t [fs-type]
mkfs.ext4 -L <label> <targetdevice>

lsblk -f
blkid /dev/sda1

#/etc/fstab
LABEL=<label> <mntPoint> <fsType> defaults 1 2

# mount

mount -t ext4
cat /etc/mtab # smlknk
cat /proc/mounts

man mount
    esc 501 # fs-independent munt options

mount -L <label> -t <fs> -o rw,noexec <mntPnt>
# e.g. mount -L OPT -t xfs -o rw,noexec /opt

nano /etc/fstab
LABEL=OPT /opt  xfs  rw,exec,suid,
auto 1 2

mount -a
mount -t xfs
ls /media


## Multipath

DM-Multipath - kernel module that  akes routing decisions
multipath # command for listing/viewing devices
Multipathd # daemon that monitors paths, marks fails, and reactivates paths
kpartx # command used to create mapper entries

## File system Maintenance

lsblk
fsck
mke2fs # creates new fs on partition
less /etc/mke2fs.conf
mke2fs -t ext4 -L EXTRA /dev/sdc1

tune2fs # utility to adjust params on ext(2,3,4) filesystems


xfs_repair
xfs_fsr # reorganize data stored in blocks on xfs, similar to defrag on MS Windows fs
xfs_db # used to debug xfs fs


# Disk Usage

ls -i (inode number)
df -h
df --total -h
free -mh
du -sh --max-depth=2

# inode stores info about files and folders such as permissions and ownership, filetype
# dont run out of inodes!

# personal note
mkfs.ext4 /dev/...
mkfs.ext4 -L "label" /dev/...
    # or
e4label <blk-device> <new-label>

sudo wipefs /dev/sda # list all partitions
sudo wipefs -a -t dos -f /dev/sdx # wipe specific partition
mkfs.ext4 -L "label" /dev/...
sudo tune2fs -l /dev/sda1

Localization

locale
localectl
UTF-8
iso-8859
iconv # convert between char encodings

date
date -u
date +%F
date +%a
date +%A

timedatectl
timedatectl set-time "2018-12-1 01:00:00"
timedatectl set-timezone "Antartica/Davis"
timedatectl set-timezone "America/New_York"
tzselect

System Operations and Maintenance

APT

# debian / ubuntu / mint / etc.
apt # advanced -- includes dependencies

apt remove
apt
autoremove
apt purge
apt dist-upgrade
apt download # doesnt need sudo - downloads but doesnt install
apt-cache search
apt-cache show
apt-cache showpkg

Reads /etc/apt/source.list and handles dpkg from there

DPKG

dpkg # no dependnecies unless specified
dpkg --info <package-name>
dpkg --status <package-name>
dpkg -l #list
dpkg -i <package-name> # installs package
dpkg -L <package-name> # list all files that were installed with the package
dpkg -r <package-name> # remove package but not config files
dpkg -P <package-name> # remove everything (Purge)
dpkg -S <search string>
dpkg-reconfigure # re-run applications configuration tool

Shared Libraries

/lib and /usr/lib/ contains .so files for "shared object" - Dynamic file Also .a statically linked file.

ldd <program-name> # see shared object dependencies
ldconfig #configures dynamic linker run-time bindings, creates cache, can show current cache
cat /etc/ld.so.conf

Remote Desktop

Old method... but worth noting

xhost + <ip addr> export DISPLAY="<ip.10.10> ssh -y <ip>

Newer Methods

VNC Insecure by default SPICE TLS encrypted protocol

Investigating Hardware

udev handles hardware devices and /dev psuedo-file system for devices

ls /dev/cpu

lspci
lsusb
lscpu
lsblk

CUPS Common Unix Printing System

apt install cups printer-driver-cups-pdf
localhost:631

Line Print Deamon // LPD
lpstat -s

lpadmin  #add, modify, delete from cmdline
lpinfo   #displays devices and drivers
lpc      # older command to show info. can pass `status`

# e.g.
lpadmin -p ENVY-4512 -L "downstairs printer" -v socket://<printer-ip>:<port> -m everywhere # -m for driver search
lpinfo --make-and-model "HP ENvy 4510" -m
lpadmin -p ENVY-4512 -m "drv:///hpcups.drv/hp-envy_4510_series.ppd" -E
lpc status

# print from cmdline
lpr /etc/passwd
lpq -a # see the queue
lpstat -l
lprm <job number>
lpq -a
lpadmin -x <brokenPrinter> # remove printer

cups reject prevent jobs being sent to specific printer

Manage Users and Groups

Users

useradd, passwd, and usrdel commands for user management

Add a user

useradd -m <usrName>
ls /home
passwd <usrName>

useradd -m -c "Geroge Freeman" -s /bin/tcsh gfreeman
passwd -e gfreeman # require password change at first login

userdel gfreeman # preserves home dir
userdel -r gfreeman # removes home directory also

Groups

Collections of user accounts Used to set permissions by group groups, groupadd, and groupdel

# create a group for curators
groupadd curators
useradd -G curators -m -c "Barney Calhoun" bcalhoun # capital G for supplementary group lowercase g for primary
groups bcalhoun
groupdel curators
groups bcalhoun

User and Group Config Files

ls -l /etc/passwd #root permission level
less /etc/passwd
# name:passwd:userNumber:PrimaryGroupID:path-to-home:defaultShell

less /etc/shadow
name:encryptionAlg-salt-hashedValue:age:minDaysPWChange:maxDaysPWChange:PW-expiry-warning-days:pw_status:inacctiveDays:expiry:

Encryption Algorithm Designations $1$ = MD5
$2a$, $2y$ = Blowfish
$5$ = SHA256
$6$ = SHA512

less /etc/group
GroupName:PW:GroupIDNum

less etc/skel what is added to new user's home directory

less etc/default/useradd User add defaults

getent to querry a db for userid or name

getent group 100

/etc/login.defs takes precidence over etc/default/useradd

User and Group Modifications

getent, groupmod, and system accounts

usermod -s /bin/bash bcalhoun
getent passwd bcalhoun

groupadd engineering
usermod -a -G engineering bcalhoun
groups bcalhoun
getent group engineering

usermod -L bcalhoun # lock a user account
getent shadow bcalhoun # exclamation in PW field shows lock
usermod -U bcalhoun # unlock a user account

useradd -r projectx #add system account with -r
getent passwd projectx
usermod -s /sbin/nologin projectx
getent passwd projectx
usermod -d /opt/projectx projectx #change location of home dir
mkdir /opt/projectx
cd /opt
chown projectx:projectx
getent passwd projectx

chage -E 2020-06-01 #change aging params of pw
chage -l bcalhoun
chage -E -1 bcalhoun #remove expiry
chage -W 14 bcalhoun #two week warning period

getent group engineering
groupmod -g 1100 engineering #change groupid number
getent group engineering

groupmod -n Engineering engineering
getent group engineering

Securing Local Logins

etc/nologin, etc/passwd, and /etc/shadow

usermod -L avance #lock account
usermod -e 1 avance
getent shadow avance
getent passwd avance

usermod -s /sbin/nologin avance
getent passwd avance

usermod -U -e "" avance

vi /etc/nologin
<Message to locked user logins>

usermod -s /bin/bash avance
ls -l /etc/passwd #anyone can read - only root can mod
ls -l /etc/shadow #may appear odd

Bash Shell Environment

env, echo, set, and unset

Example bash function

function yo()
{
    echo "sup g"
}

unset -f yo

Environment Variables

env #to view all environment variables
echo $PWD #print single env
echo $LOGNAME

set
set | less
set -x #turns on debugging

shopt, export, pwd, which, and type

shopt displays shell options and the current settings

shopt -s <optionName> # enable option
export YO=yo
echo $YO

'Strong' vs "Weak" Quotes

echo '$PATH'
echo "$PATH"
echo $PATH

-bash vs bash after echo $0 differentiates between login and non-login shells. -bash is login shell.
/etc/skel contains template files

Automate and Schedule Jobs

Cron Table

crontab

cat /etc/crontab

crontab -e

# backup documents every saturday at 5am
# minutes hours date month day usrname command
0 5 * * sat ax0n /usr/bin/tar -cfz documents-$(/bin/date +%F).tar.gz /home/ax0n/Documents

crontab -l sudo cat /var/spool/cron/ax0n crontab -r # deletes crontable

# every three minutes - do something useless
*/3 * * * * ax0n /bin/echo "Hi There!" >> /home/kenny/hi_there.txt

crontab -u <username> crontab -r -u <username> deletes entire cron table cat /etc/cron.deny users listed here will not be able to create cron jobs

At

at runs a task at a later time atq views the at job queue atrm remove a scheduled job by job number

Example 1
at now + 5 minutes

echo "notes for later:" > /root/notes.txt

[ctrl+d]

Example 2
at 4:00 AM tomorrow

rm /root/notes.txt
atq
atrm
atq

control useage with at.allow, at.deny files.

Systemd Timer Unit Files

monotonic deleted on reset or shutdown OnBootSec= and OnUnitActiveSec= Realtime more like cron or at, uses somethign like -OnCalendar=

Simpler syntax then cron or at

man 5 systemd.timer
man 7 systemd.time

systemctl list-timers --all
systemctl cat systemd-tmpfile-clean.timer systemct cat systemd-tmpfiles-clean.service
vi /etc/systemd/system/<service.timer>

[Unit]
Description=Fire off the backup

[Timer]
OnCalendar=*-*-* 21:06:00
Persistent=true
unit=<servicename.service>

[Install]
WantedBy=multi-user.target

systemctl enable <.timer> systemctl start <.timer> systemctl list-timers --all

Create, Modify, and Redirect Files

Basic File Operations

# Basics
cd
mkdir
$PATH
ls -la
touch
cp
rm
mv
file

File Globbing - Global Command - GLOB
* wildcard match zero or more
? wildcard for any single character
[abc] matches any one of the charicters in the list, case sensitive
[^abc] matches any one character except those in the list, case sensitive
[0-9] matches a range of numbers

Regular Expressions

man 7 regex
. represents a single character grep g.m passwd or grep s..m passwd
^ search the beginnings of lines grep ^rpc passwd
$ search end of line grep bash$# passwd
[abc] - specific characters grep [v] passwd grep -i [v] passwd
[^abc] - exclude these characters from a search
* match zero or more of the preceding characters or expressions

Regular Expression Tools
sed - Stream Editor

cat passwd | sed -n '/nologin$/p'
cat passwd | sed -n '/nologin$/d' >> filter.txt #filters (d for delete)

egrep - extended grep - equivilent to grep -E
fgrep - searches strings not patterns equivilent to grep -F

egrep  'bash$' passwd
egrep -c 'bash$' passwd

Symbolic Links

ln -s creates a soft link
ln creates a hard link
unlink remmoves a link from a file or folder

ln -s <target-of-link> <link-name> # name.lnk is a good convention
unlink <lnik-name>

Finding Files

find

find -name
find -ctime #based on last change, works on 24hr period i.e., 1 for past 24 hours
find -atime #based on access time
find -empty #find files or folders that are emtpy
find -exec [command]{}\; #acts on results of find command

find . -name exmaple.txt #recursive search through . dir
find /  -name passwd

find . -ctime 1 #all files in . that have been changed in last 24 hours
find . -atime 2 #all files in . accessed in the last 48 hours

find . -newer passwd #files newer than passwd

find  . -empty -type f -exec rm -f {} \;

Standard INput, Output, and Error

stdout, stdin, and stderr

script.sh                #stderr printed in console
script.sh 2> error.log   #send stderr to error.log file
script.sh 2>&1 | less    #send stderr and stdout to less

Finding Commands

locate searches a local db of files and folders
updatedb updates the db for locate
whereis locates binary source or man pages fro a command

Redirecting

tee and xargs

echo "yep" > myfile.txt
echo "yep some more" >> myfile.txt
cat < myfile.txt

ls -d  /usr/share/doc/lib[Xx]* | tee > lib-docs.txt

grep -l "junk example text search" test/file_* | xargs -I {} mv {} test/bak/
find ~ -name "*.sh" | xargs ls -lah {} > scripts-info.txt

File and Folder Compression

dd copies and converts often used to create files of arbitrary size, and back up to disc drives
tar wraps up files and folders into an archive filedoes not compress on it's own
gzip creates .gz compressed files
gunzip extracts .gz files bzip2
bunzip2
xz
unxz

dd if=<input.file> of=<output.file>
dd if=<bootdisk> of=<file.img> bs=512 count=1 #backup MBR

dd if=/dev/urandom of=file bs=1024 count=10
ls -h file

tar -cf <tarname.tar> <content-directory> #create tarball
tar -tf <tarname.tar> # read files in tarball
tar -x <tarnam.tar> #unpacks tarball

tar -czf tarName.tgz <content-directory> # -czf order is important - .tgz or .tar.gz are common naming conventions
tar -xvzf ... # uncompress

tar -cjf tarname.bz2 ... # bzip2 compression method
tar -xvjf ... #unzipfor bz2

Text Editors

nano and vim : basic useage notes.

Manage Services

Init

init - startup daemon that was on most Linux distributions. It is still in use on some distributions to this day, but has mostly been supplanted by the modern systemd.

• Runlevel levles 0-6, determines system-wide runlevel.
• /etc/inittab - each line specifies a runlevel

Red Hat Service Tools

Classic init daemons were managed on Red Hat based distributions using the tools service, chkconfig, and ntsysv. These tools are legacy utilities, but you may still encounter them when administering older systems. Also, the service command currently can be mapped to equivalent systemd commands for managing daemons at runtime.

Upstart

Upstart helped to bridge the gap between the classic init system and the modern sytemd initialization system. Ubuntu was the first distribution to fully tackle parallel daemon initializations, and was used for a time on other Linux distributions as well. We will discuss the architecture of Upstart, and how it differed from the classic init daemon

man upstart

Systemd

systemd removed the need to have shell scripts.

Systemctl

systemd deals with everything as a 'unit'
i.e. something.unit

• session-3.scope
• user.slice
• etc

status, enable, and disable

enable creates a symlnk and sets the service to start on boot.
disable prevents a service from starting, removes symlink.

systemctl defaults to '.service' - best to be explicit ... i.e., systemctl enable httpd.service

Unit Files

.target, .service, and .timer

Delineate how a systemd unit will operate.
systemd replaces bash scripts with compiled C code.

Drop-in files: e.g. Modifying the HTTP Service:

1. Create a directory named in the format <iunit.d> e.g. /etc/systemd/system/httpd.service.d/
2. Create a conf file that contains the change e.g., /etc/systemd/system/httpd.service.d/my-httpd.conf

Safest method is to systemctl edit --full <unit> where --full completely overrides the existing unit files.
systemd-delta command (systemctl-delta?) to see changes
systemctl daemon-reload after any changes - reruns all dependency generators without having to restart computer.

Files are located at:
/usr/lib64/systemd/system/
/etc/systemd/system/ - only place one should make edits
/run/systemd/system/

systemctl list-unit-files to view all unit files
systemctl cat to view contents of unit file

Target Unit Files

This type of unit describes how our computer's operating environment will be set up. We can quickly and easily change between different types of targets with the systemctl isolate command.

multi-user.target multi user system
graphical.target multi-user system with Desktop Environment
rescue.target pulls basic system and file mounts and provides a rescue shell
basic.target
sysint.target
man 5 systemd.target - defines target unit configs man 7 systemd.special - lists and defines all targets

ex. systemctl cat graphical.target

Service Unit Files

Stuff that gets things done on the systems.

[Service]
Type= # 'simple' is default, 'forking' creates children then exits - use with PIDFile - dbus BusName= - 'notify'
ExecStart=

[Install]
WantedBy=

Types:
'simple' is default
'forking' creates children then exits - use with PIDFile
'dbus' BusName=
'notify'

systemctl mask <daemon.service> will prevent accidental service start by symlinking to /dev/null
systemctl unmask <daemon.service> ...

Timer Unit Files

This unit type is meant as a replacement for the atd and cron systems for scheduling jobs or system events to take place at specific times.

foo.timer file must have foo.service monotonic 'OnBootSec=', 'OnActiveSec='
realtime 'OnCalendar='

[Timer]
OnBootSec=
OnActiveSec=

# or
OnCalendar=

Unit= # not always used

systemctl list-timers --all
systemctl cat <foo.timer>

systemd-run --on-active= used to craete a transient timer without a service file.

Reviewing the State of the System

ps
proc
man proc
man signal

Monitoring Process

uptime shows how long the system has been up, and a small bit of other usefull info.
free view used and available memory and swap space
pgrep find process info based on processor name
kill send a signal (usally SIGTERM) to a process based on PID
pkill send a signal (usally SIGTERM) to a process based on process name
killall kills all processes based on a name (sudo)

Keep a Process Running

watch runs a command at specified intervals
screen terminal window manager tmux if you dont know... know you know.

Summarize and Explain Server Roles

Core Network Servers

DNS Domain Name Server
Purpose: Translate Domain Names in IP Addresses
Ports: 53 UDP/TCP
Service Provided by: BIND, dnsmasq, Unbound

NTP
Network Time Protocol
Purpose: Synchronizes the date and time on local system with an upstream, network-connected time provider
Ports: 123 UDP
Service Provided By: ntpd, chronyd, systemd-timesyncd (client only)

DHCP
Dynamic Host Configuration Protocol
Purpose: Provides an automatically assigned IP address to a client on a network Ports: 67 UDP, 68 UDP Service Provided By: dhcpd, dnsmasq

Infrastructure Servers

Authentication Server
Purpose: centralized server system on a network where multiple users and/or services can receive authentication credentials to network systems Ports: 389 TCP for LDAP, 636 TCP for encrypted LDAP (LDAPS), 88 TCP for Kerberos Service Provided By: openLDAP, Active Directory, Kerberos, Red Hat Identity Management

Load Balancing Servers Purpose: routes network traffic to various servers based on network load Ports: depends on service Service Provided By: haproxy, Apache, Nginx, BIND, dsnmasq, Keepalived

Server Clustering Purpose: Failover protection using redundant servers

Monitoring and Security Servers

Monitoring, Logging, and Proxy servers.

Logging Services
Purpose: logging services are crucial sources of info that contain details about the systems state and operations.
Ports: 54 UDP (syslog - unsecure), 6514 TCP (secure sysslog communications) ,19531 TCP (systemd-journal-gatewayd serves jounral events over network)

M0nitoring Services Purpose: RAM and CPU Metrics, useage, uptime, logged-in users, etc.usually gathered by SNMP traps Ports: 161, 162 UDP (SNMP); 10161, 10162 SNMP over TLS; typically HTTP ports for web applications. Service Provided By: SNMP, Nagios, Munin, Zabox, Monit

Proxy Server Purpose: Centralized server that computers on a network connect to prior to reaching the internet. cache frequently visited pages, block others. Ports: 3128 TCP Squid proxy Service Provided By: Squid, Varnish, Apache, Nginx

Common Server Systems for Client Use

Print Server, File Server, Database Server

File Server Purpose: Provide clients access to remote files and folders from centralized server
Ports: 137, 138, 139, 445 TCP (Samba); 111 TCP/UDP, 2049 TCP/UDP (NFS)
Service Provided By: Samba, NFS

Email Servers

• 25 TCP unecrptyed SMTP
• 465 TCP SMTPS, SMTP+SSL
• 587 TCP MSA
• 110 TCP POP3 unencrypted
• 995 TCP encrypted SSL/TLS POP3
• 143 TCP unencrypted IMAP
• 993 TCP encrypted SSL/TCP IMAP

Service Provided By: Sendmail (SMTP), Postfix (SMTP), Dovecot (IMAP, POP)

Web Server
Purpose: Content Delivery
Ports: 80TCP; 443TCP (encrypted)
Service Provided By: Apache, Nginx

Print Server
Purpose: Centralized Print Server
Ports: 631TCP default for CUPS
Service Provided By: CUPS

Database Servers
Purpose: Store data of various kinds
Ports: 3306 TCP (MySQL, MariaDB); 5432 PostgreSQL)
Service Provided By: MySQL, MariaDB, PostgreSQL

Container Basics

docker is a runtime (container tool)
kubernetes is a container orchestration tool

Security

File and Directory Permissions

Basic File Permissions

Symbolic Permissions: (broken down to "user group other/world")
r = read w = write
x = execute permission
- = no permission

Octal Permissions: sums apply to user/group/world e.g., 7 == rwx, and 0 == --- 4 = read
2 = write
1 = execute
0 = no permission

[d,b,c,-]uuugggooo e.g. drwxrwxr-x
file type, user permissions, group permissions, world permissions.

chown - change owner
chmod - change mode
chgrp - change group ownership

Advanced Permissions

SUID Set User ID
SGID Set Group ID
Sticky Bit puts 't' in place of 'x' in other's collumn allows only the creater to remove the file

Default Permissions

umask shows default maskign value Default permission for directories = 777. umask = 0002, directories default to 775

/etc/bashrc has umask for whole system.
/home/<user>/.bashrc has umask for individual user

File Access Control Lists

getfacl <filename>
setfacl -m u:<user>:r <filename> Modify acces for user to read permissions for file.

will see + at end of permission string when ls -l

Context-Based Permissions

SELinux

Mandatory Access Control
access denied events logged to /var/log/messages
Components:
Policy enforcement server
Access Vector Cache (AVC)
Access Matrix
Installed Pollicy

cat /etc/selinux/config to see SELINUX= variable - never set to disabled. use permissive and check logs.

Policies
/etc/selinux/ Targeted vs Strict

SELinux Tools

semanage login -l
semanage user -l
sestatus
getenforce : Permissive, Enforcing, or Disabled
sudo setenforce <value>
getsebool -a Get a list of SELinux booleans without descriptions
semanage boolean -l Get a long list of SELinux booleans
set sebool Change an SELinux boolean
ls -Z
ps -Z
chcon
restorecon

AppArmor

Debian and SUSE Linux security tool

apparmor_status
apt install apparmor-utils
aa-complain /sbin/dhclient
apparmor_status
systemctl reload apparmor.service
aa-unconfined
aa-genprof Xtightvnc
aa-complain /usr/bin/Xtightvnc
systemctl reload apparmor.service

Privilege Escalation

Wheel group is allowed to run sudo commands and elevate their priviledges

su
su -
sudo
sudoedit

Configure sudo ONLY WITH visudo

sudo visudo -c #check for errors
sudo visudo -f #file name
sudo visudo -s #strict  mode
sudo visudo -x #output `sudoers` file as json

sudo vim /etc/pam.d/password-auth-ac as non wheel group member will throw reported error

As admin or wheel member sudo visudo

# root, standard user, and service user accounts
ps -aux | grep -v root
grep ^dbus /etc/passwd
grep nologin /etc/passwd #see service accounts

Access and Authentication Configuration

PAM Basics
Pluggable Authenticaiton Modules
Designed to provide a single interface for processes and services to leverage built-in authentication and authorization methods within a Linux system
The /etc/pam.d/ directory contains the PAM configuration files for each PAM-aware application.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/managing_smart_cards/pam_configuration_files

Set lockout paramters on failed logins
set password policies

PAM and LDAP pam_ldap
Lightweight Directory Access Protocol

vim common-session
Put pam_ldap.so just above the pam_unux.so

The pam_ldap module can allow authentication against the Lightweight Directory Access Protocol (LDAP).
This is a centralized credential store that can be used for single sign on.

Secure Shell (SSH)

known_hosts

Creating keys and copy them over to a server

ssh_keygen
  #id_rsa
  #id_rsa.pub

ssh-copy-id <user>@<publicIP>
ssh-add
authorized_keys

ssh_config #global config for client
sshd_config #config for ssh daemon

grep Root ./ssd_config
vim sshd_config
# AllowUsers, AllowGroups
# default entry for ssh port
# PasswordAuthentication yes/no -- disable password for extra security!

SSH and TCP Wrappers
Secures incoming connections by using the hosts.allow and hosts.deny files located in the /etc directory
hosts.allow supersedes hosts.deny

vim hosts.allow

sshd : <publicIP>,<privateIP>

vim hosts.deny

sshd : ALL

now only that sshd.allow ip is allowed to ssh into the server with the allow file. everything else denied.

TCP wrappers can be used to lock down ssh for increased security

# Example
sudo nano /etc/ssh/sshd_config
sudo service sshd restart
sudo nano /etc/hosts.deny #sshd : ALL
sudo nano /etc/hosts.allow #sshd : 10.0.0.0/24

Terminal Types

TTYs and PTYs
Psuedo and Actual Terminals

/dev/tty
/dev/pts
/dev/securetty

sudo chvt 3

tty2 connected to psuedo terminal 0 so sudo chvt 2 will bring us back to the gnome terminal.

head /etc/securetty
cat /etc/securetty | grep pts - no result. dangerous for root to log ingto psuedo terminals.

Public Key Infrastructure (PKI)

Digital Signature
Encrypted hash of a message sent with the message.

Digital Certificate
Basic component of a PKI
Contains information about the user's identity

Certifiacte Authority (CA)
Server that issues certxificates
maintains key pairs

Certificate Signing Request (CSR)
Message sent to CA to apply for a digital certificate

Certificates, Session Keys,

openssl dgst -sha256 <file>
openssl dgst -sha256 -verify <pubkey.pem> -signature <file.sha1> <file>

Virtual Private Networking (VPN)

IPSec

Internet Protocol Security (IPSec) One implentation for Virtual Private Netowrks (VPNs)
IPSec VPN is implemented at OSI Layer 3, therefore it is application independant

Transport Mode Data encrypted, headers not
Remote Acces VPNs

Tunnel Mode
Everything encrypted.
Site to Site VPNs

Keeps people from eavesdropping or sniffing packets.

SSL / TLS

Application Dependent
Application layer implementation of Virtual Private Networks (VPNs)
Uses SSL/TLS to secure data in transit
The most popular software for this purpose is OpenVPN

OpenVPN
Certificate-based
Password-based
TCP

Open Connect
Datagram Transport Layer Security (DTLS)
Uses UDP
Less Latency but no redundancy (as TCP packet verification)

Security Best Practices and System Hardening

Protecting the boot sequence

BIOS / UEFI Password
Boot Loader Password (Usually in GRUB / GRUB2)

Additional Authentication Methods

default login method - least secure
OTP and Tokens
Biometrics

Combination - Multi-Factor Authentication

RADIUS
Remote Access Dial-In User Service is an itnernet protocol
provides Authentication, Athorization, and Accounting (AAA) services

TACACS+
Terminal Access Controller Access-Control System
Also provides AAA but more scaleable and more secure than RADIUS

LDAP
klist -v

Kerberos Implementation of LDAP
Ticket graninting system for single-sign-on
kinit, kpassword, klist, and kdestroy

Secure System Design

Isolate Users and Services
Dont share IDs
Use a chrooted jail
Separate application data from OS data using disk partitions
Discourage USB devices
use disk encryption (LUKS) cryptsetup -y -v luksformat /dev/<disk>
disable ctrl+alt+del

restrict cron access
ls /etc | grep cron
probably missing a cron.deny file
echo root > /etc/cron.allow is a good idea - lets only root use cron

Add banners and MOTD cat /etc/motd

Securing Network Services

Change default ports
Uninstall or disable unused insecure services

• FTP
• Telnet
• Sendmail

Enable SSL/TLS
Setup Passwordless Login (PKI)
netstat -plnt to check ports
Enable auditd
Monitor CVE have a patch policy

Isolating Services

Partition service data
Use chroot jail
Enforce context permissions
Ensure logging

Restricing Remote and Local Access

Disable root logins (require all users to sudo up)
vim etc/passwd to add nologin to root line (probably top line)
cat /etc/hosts.deny Explicit SSH permissions vim /etc/ssh/sshd_config AllowUsers line to whitelist
AppArmor/SELinux running, auditd running

Logging Services

Log Locations

ls -l /var/log/

syslog
messages
auth.log
secure
# <application name> i.e., nginx
nginx

head -n 20 <file>
tail -n 20 <file>
tial -f <file> to watch the tail of a log file

Log Management

Log rotation and remote logging

cd /etc/logrotate.d
ls -la
vim nginx

Commands

last
lastb
lastlog

Can use third party agent to get syslog compliant logging from other machines and services (i.e., Windows)

cd /etc/rsyslog.d
ll
cat <>

journalctl

journalcrtl is used to view logs created by the journald service
/etc/systemd/journald.conf
/var/log/journal
journalctl [opt] [match]

File lives in memory, can preven tby creating the journal directiy in /var/log/

journalctl
jounralctl -f
jounralctl -p  err
jounralctl -p  err -u ssh
jounralctl -p  err -u ssh -o
jounralctl -p  err -u ssh -v

Firewalls in Linux

iptables -nL

OSI Layer 3 - Network Layer Firewalls

Stateless firewalls (packet filters)
Single packet filtering - no introspection

Stateful firewalls can recognize traffic
e.g., in iptables if we see ctstate RELATED, ESTABLISHED - it is a statful rule

Application Layer Firewall

Not a default in linux
protocol introspection

Access Control Lists

NACL
Packet Filtering

#ubuntu
iptables -I INPUT -p tcp -s <allowedIP> --dport 80 -j ACCEPT
iptables -nL | grep <allowedIP>

ufw allow 80/tcp

#fedora
firewall-cmd --permanent --zone=public --add-rich-rule='
  rule family="ipv4"
  source address="<ip/CIDR>"
  port protocol="tcp" port="22" accept'

firewall-cmd --list-all --zone=public

IP Forwarding

ip addr
curl 127.0.0.1 #curl nginx on fedora

switch servers to ubuntu

ip addr
curl 127.0.0.1 #gets nothing

cd /proc/sys/net/ipv4
ls
cat ip_forward #1 means IPTables can forward

cat /etc/sys/ctlconfig #uncomment ip-forward line
iptables-save

now, traffic on port 80 in ubuntu forwarded to fedora machine

Dynamic Rule Sets

ipset create 80_allow hash:ip
ipset add 80_allow <ip>
ipset list 80_allow

iptables -I INPUT -m set --match-set 80_allow src -p tcp --dport 80 -j ACCEPT

iptables add 80_allow <newIP> # automaically added to iptables rule above
iptables list

Dynamic Toolkits
denyhosts track failed ssh logins
fail2ban updates firewall rules to reject ip addresses for a certain amoutn of time

Common Firewall Configs

cat /etc/services
Trusted or Privelidged ports 0-1023

Backup, Restore, and Compress Files

Archives

tar -cvf backup.tar /some/source/ #compress
tar -xvf backup.tar

ls | cpio -ov > archive.cpio
cpio -idv < archive.cpio

dd if=/dev/sda2 of=backup.img
dd if=backup.img of=/dev/sda2

Compression

gzip
xz
bzip2
zip

# -c <input files>
# -d <archive.gzip>

tar -czvf <output> #gzip
tar -cvjf <output> #bzip2

Backups

Storing Redundant Data - growing disk size unnecessarily

full backups - including data that hasnt changed - high redundancy

differential backup - only stores changes since the last full backup
(diff#2 contains diff#1) diff#3 etc...

Incremental Backups incremental#1 same as differential backup #1
incremental#2 just changes since inc#1
restore each one all the way to most current increment
then make a new complete and process starts over

Remote System Storage

scp #ssh
sftp #ssh+ftp
rsync #can be run over ssh

File Integrity Checks

md5sum <file1 file2> > manifest.txt #128 hash - not super strong

sha256sum <file1 file2> > manifest.txt
sha512sum <file1 file2> > manifest.txt

Linux Troubleshooting and Diagnostics

Analyze System Properties

Network Monitoring and Configuration

nmcli
ip
nmap
tshark #wireshark
tcpdump
netcat

lspci | grep -i ethernet

RDMI is a high speed network interface card

ip a s #[ip addr show]
nmcli device status
nmap -p 1-1023 -sV -sS -T4 192.168.1.1 #ip is gateway ip from output of `ip`

nc -zv 192.168.1.1 20:80

tshark > packet.txt
tcpdum > dump.txt

Latency, Bandwidth

ping
traceroute
tracepath
netstat #depricated for ss
ss

Routing

route
arp

Saturation, Throughput

iftop
iperf

Packet drop, Timeout

mtr

Name Resolution in DNS

dig
nslookup
host
whois

locahost vs Unix socket

cat /etc/resolve.conf
cat /etc/hosts

Storage Configuration and Management

LVMtools

pvscan
pvcreate /dev/sdb /dev/sdc
pvscan
vgcreate vg1 /dev/sdb /dev/sdc
vgscan

lvscan
lvcreate -n new_vol -L 10G vg1
lvscan
lvdisplay vg1/new_vol

mkfs.ext4 /dev/vg1/new_vol
mkdir /mnt/lvm1
mount /dev/lg1/new_vol /mnt/lvm1
cd /mnt/lvm1
ls

lsblk
partprobe -s
partprobe #update kernel

lsblk
partprobe -s
du
df #disk free
iostat
ioping -c 12 .

ulimit -a

IO Scheduling

cat /sy/sblock/sda/queue/scheduler
cfq #completely fair queue - round robin
noop #no sorting - just pass to device
Deadline #FIFO

fsck /dev/sdb
fsck /dev/sdb1

CPU Monitoring and Configuration

cat /proc/cpuinfo
uptime
loadaverage
sar
sysctl

cat /etc/default/sysstat
# change to true

systemctl restart sysstat
cat /etc/cron.d/sysstat #10 minute interval
#option to run manually

sysstat
sar

Memory Monitoring and Configuration

swapon
swapoff
mkswap
vmstat

Out of memory killer

free
cat /proc/meminfo | grep -i swap

Buffer cache output

Recovering a lost root password

from GRUB(2)
Single User Mode / Rescue Mode to Reset PW

e
linux ... `rw inti=/bin/bash`
ctrl+x

cat /etc/passwd | grep <user>
passwd <usrname>
exec /sbin/init

mount -o remount,rw / #if mount is read only

Analyze System Processes

Process Management

man ps
ps ax # == ps -e

ps -e | grep <process>
ps -el | grep <process>

• running R
• interuptible sleep
• uninteruptible sleep
• zombie
• stopped/paused T

fg

ping google.com >> ping.txt
ctrl+z
ps -l
fg
ctrl+c
cat ping.txt

nice value is priority: -20 to 19. -20 is highest priority
nice -n 10 ping google.com >> ping.txt
renice -n 0 <pid>

top

pgrep -a ssh
pgrep -l ssh

time
lsof
pkill

kill #(default graceful interupt 2)
kill -9 #(force kill right meow)

time ping -c 4 google.com

Troubleshoot User Issues

Methodology

1. Verify Complaint
2. Check inheritance with lsattr ../
3. Remove immutable flag with chattr -i <dir>
4. test permission vs ownership

Access

Access issues vs Authentication issues

1. ping to test one-way access
2. check firewalls
3. reverse ping wiht traceroute
4. check table with iptables -nL
5. check route table

Local Access
Remote Access

Authentication

Local Auth
Remote Auth
Policy Violations

1. is the issue remote or local
2. is there a policy violation PAM i.e. authentication failures - user exists but password is wrong
3. what is the mode of the failure cat /var/log/secure | grep -e <username>

File Creation

Quotas
Storage
Inode exhaustion
Immutable files

df -i to check inode useage

Insufficient Priveledges for Authorization

SELinux Violations
sealert -a /var/log/audit/audit.log > auditresults
cat auditresults
look at Raw Audit Messages

Environment and Shell Issues

Search Paths
Aliases
~/.bash_profile
echo $PATH
export
alias ll='ls -la'

Troubleshoot Application and Hardware Issues

Storage Issues

Small Computer Systems Interface (SCSI)
Serial Advanced Technology Attachment (SCADA)

(IDE adn PATA also exist - replaced by SCADA)

Host Bus Adapter (HBA)

Redundant Array of Independent (inexpensive) Disks (RAID) Card
Software / Hardware RAID (Kernel/Card)

Firewall Issues

Restrictive ACLs
Blocked ports
Blocked protocols

cat deny.hosts to look for errors
cat /etc/hosts to look for errors
vim iptables (?) firewalls rules

Audit the Firewall!
iptables rules should be in an iptables save file
Firewall should be logging

Hardware Issues

lsusb
lspci
lshw
localectl
dmidecode

Automation and Scripting

Deploy and Execute Basic Bash Scripts

Shell and Environment Variables

Shell variable looks like example='other data'
Envrionment Variable looks like UPPER_CASE

Bash Scripting

Shell exapnsion possible with 'weak' quotes "
example.sh

#!/bin/bash
#this is a regular comment
source otherscript.sh


echo  $sourced_var
#this is the variable that we will use for later
data='somedata'

#echo 'this is the variable without shell expansion $data'

#echo "this is the variable with shell expansion $data"

#echo "this is the variable with shell expansion $data" > test.txt


#find ./ -type f -name "file?"
#find ./ -type f -name "file*"


#echo "This is the first parameter that we passed in $1"
#echo "This is the second parameter that we passed in $2"

#echo -e "This is an example of\n escaping characters \t\t !"

otherscript.sh

#this script will get sourced from the other script

sourced_var='This is imported from our other script'

Exit Codes

$? gets last exit code
0 = no issues
&2 is stderr
exit keyword will set exit in bash script

Redirection and Piping

> #overwrites file
>> #appends to end
| #pipe output on left to input on right

Conditional Statements

conditional.sh

#!/bin/bash

var=$1
if [ $var == 1 ]
then
    echo  "$var eq one"
fi

if [ $var == 2 ]
then
    echo "$var eq two"
else
    echo "$var does not equal 2"
fi

if [ $var -ne 2 ] && [ $var -ne 1 ]
then
    echo "$var equals 3"
fi

case $var in
    1)
        echo "in this case the positional parameter eq 1"
        ;;
    2)
        echo "in this case the positional parameter eq 2"
        ;;
    3)
        echo "in this case the positional parameter eq 3"
        ;;
esac

&& is both true
|| is OR
-eq equal to
-ne not equal to

looping.sh

#!/bin/bash

for i in {1..5}
do
    echo $i
done

echo 'for loop is complete'

counter=6
while [ $counter -le 9 ]
do
    echo $counter
    counter=$[ $counter+1 ]
done

echo 'while loop is complete'


number=10
until [ $number -eq 15 ]
do
    echo $number
    ((number++))
done

echo 'until has completed'

Positional Paramters

position.sh

#!/bin/bash

set -- first1 second1 third1

echo "the first arg is $1"
echo "the second arg is $2"
echo "the third arg is $3"

echo "the name of the script is $0"

scope.sh

#!/bin/bash


variable='this is the global setting'
echo -e "before $variable\n"

set_var (){

local variable='this is local'
echo -e  "inside $variable\n"

}
set_var
echo -e  "this is after $variable\n"

Version Control Using Git

See Git Blog

will link when it's posted...

Orchestration Processes and Concepts

central Configuration Management

dif a file to check for differences
Configuration management
Patch management
Inventory management
Deployment Orchestration Lifecycle management Monitoring

Agent vs Agentless

Agentless

• Does not require specific software on target machine
• typically from an account with elevated privelidges
• Ansible is an example of and agentless solution - uses Playbooks written in YAML to execute commands over SSH
• typically sudo no passwrd configuration

Agent

• requiresa software agent to be installed on the target machine
• usually exchange keys for auth
• agent is given elevated priveledges on the remote machine and no external elevated priviledges are required
• Puppet is an example of a configuration system that uses an agent to execute config manifests written in Ruby to enforce desired state

Inventory

Managing inventory is important for configuration management
inventory is a vital part of configuration management
It changes the agents you need to install as well as monitoring settings and patch management

Infrastructure as Code

Automatically instantiate resources by runninga script

• deploying new assets
• configuring assets
• deploying/manageing applications/databases
• deploying managing monitoring and patching

Resiliency and Scaling

Infrastructure Automation

Procedures - steps taken to complete a task
Automation - logical ordering of procedures to achieve end result
Orchestration - process of managing automation to achieve deployed infrastructure

Automated Configuration Management

Automated process for testing target server's current state against master server's desired state md5 hashes to tack file status

The Exam

90%