Use when: onboarding a new account that needs password-less login

useradd -m -s /bin/bash USERNAME
chage -d 0 USERNAME                    # force password reset on first login

# on the server, for the user's public key:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Gotcha: wrong permissions on .ssh or authorized_keys is the single most common reason key auth silently fails — always check permissions before assuming the key itself is wrong.

Use when: verifying what a user can do, or shutting off access without deleting the account

sudo -l
passwd -S USERNAME
usermod -L USERNAME
visudo -f /etc/sudoers.d/USERNAME

Gotcha: always edit sudo rules through visudo, never a normal editor — it validates syntax before saving, so you can't lock yourself out with a typo.

Use when: generating a new SSH key pair

ssh-keygen -t ed25519

# RHEL system running in FIPS mode:
ssh-keygen -t rsa -b 4096

Gotcha: ed25519 keys are rejected outright on RHEL boxes running in FIPS mode. Check FIPS status before generating the key, not after a confusing auth failure.

Use when: you manage several servers and don't want to type full ssh -i ... commands each time

# ~/.ssh/config
Host server01
    HostName server01.example.com
    User USERNAME
    IdentityFile ~/.ssh/server01_ed25519

ssh server01

ssh-copy-id -i ~/.ssh/server01_ed25519.pub USERNAME@SERVER_NAME

Note: ssh-copy-id is the fast path, but only works while password auth is still enabled — once it's locked down you're back to manually placing the key in authorized_keys.

Use when: auditing or troubleshooting what a server actually allows for SSH login

grep -Ei "PasswordAuthentication|PubkeyAuthentication|PermitRootLogin|AllowUsers|AllowGroups" \
  /etc/ssh/sshd_config /etc/ssh/sshd_config.d/* 2>/dev/null

sshd -t

Gotcha: always run sshd -t before restarting sshd on a remote box, and keep a second session open — a bad config on restart can lock you out entirely.

Use when: setting up a new network printer queue

lpadmin -p OfficePrinter -E -v ipp://192.168.1.50/ipp/print -m everywhere
lpstat -p
lpadmin -d OfficePrinter

Note: driverless IPP Everywhere is the preferred modern method — no vendor driver needed. Fall back to a PPD-based AppSocket queue (socket://IP:9100) only when the printer specifically requires it.

Use when: jobs show as processing but nothing prints

lpstat -t
tail -f /var/log/cups/error_log

Gotcha: most driverless failures trace back to the wrong PPD being assigned, or the printer dropping off mDNS — restarting cups-browsed often clears the second case.

Use when: a queue needs pausing, or a job is stuck

cupsdisable PRINTER
cupsreject PRINTER
cancel JOB_ID
cancel -a PRINTER

Gotcha: cancel -a clears everything queued — confirm nobody else is waiting on a pending job first.

Use when: the default tray, paper size, or duplex mode needs changing for a queue

lpoptions -p PRINTER -l
lpadmin -p PRINTER -o InputSlot=Tray1
lpadmin -p PRINTER -o PageSize=A4
lpadmin -p PRINTER -o Duplex=DuplexNoTumble

Note: to change settings for a single print job without touching the default, use lp -d PRINTER -o PageSize=A4 file.pdf instead of lpadmin.

Use when: a network printer is unreachable or won't accept a connection

ping -c 4 PRINTER_IP
nc -vz PRINTER_IP 631
nc -vz PRINTER_IP 9100

Note: 631 is IPP, 9100 is raw AppSocket/JetDirect, 515 is old-style LPD — knowing which port responds narrows down whether it's a network issue or a queue configuration issue.

Use when: a filesystem is full and the volume group has free space

vgs
lvextend -r -L +10G /dev/VG_NAME/LV_NAME

Note: the -r flag grows the filesystem automatically in the same step (XFS via xfs_growfs, ext4 via resize2fs). XFS can be grown online but never shrunk — size volumes with that in mind.

Use when: a new disk was attached and needs to join existing storage

lsblk
pvcreate /dev/sdb1
vgextend VG_NAME /dev/sdb1
lvextend -l +100%FREE /dev/VG_NAME/LV_NAME

Gotcha: misidentifying the disk in step one is the riskiest part of this entire workflow — double-check before running pvcreate on anything.

Use when: a system needs more swap without touching LVM

fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
# then add to /etc/fstab:
/swapfile none swap sw 0 0

Note: swap masks memory pressure, it doesn't fix a leak. If usage keeps climbing, check journalctl -k | grep -i "out of memory" for the real cause.

Use when: df -h shows free space but writes still fail

df -i
find /path -xdev -type f | sed 's#/[^/]*$##' | sort | uniq -c | sort -n | tail

Gotcha: a filesystem with millions of tiny files can exhaust its inode table long before running out of actual space. Always list candidates with find ... -mtime +N -ls before adding -delete.

Use when: testing fstab changes, or a mount won't release

mount -a
fuser -vm /mount/point

Note: a bad fstab entry found via mount -a today is far better than the same entry found during a 3am reboot.

Use when: a virtual disk or SAN LUN was expanded, but Linux still sees the old size

echo 1 | tee /sys/class/block/sdb/device/rescan
lsblk
growpart /dev/sdb 1
pvresize /dev/sdb1

Note: this is different from adding a brand-new disk — here the same disk got bigger, so the OS needs a rescan before pvresize will see the extra space at all.

Use when: assigning a fixed IP to a server

nmcli con mod "Wired connection 1" \
  ipv4.addresses 192.168.1.20/24 \
  ipv4.gateway 192.168.1.1 \
  ipv4.dns 1.1.1.1 \
  ipv4.method manual

nmcli con up "Wired connection 1"

Gotcha: confirm with nmcli con show before bringing a connection down on a box you're remoted into — it's an easy way to lock yourself out.

Use when: a service needs a port opened, permanently

firewall-cmd --add-port=8080/tcp --permanent
firewall-cmd --reload

Gotcha: forgetting --reload is the most common reason a "fixed" firewall rule doesn't actually take effect. Without --permanent, the rule vanishes on the next reload/reboot too.

Use when: a hostname won't resolve but you're not sure if it's DNS, hosts file, or network

ping -c 4 8.8.8.8
ping -c 4 example.com
cat /etc/resolv.conf
dig example.com
dig @8.8.8.8 example.com
cat /etc/hosts

Note: if ping-by-IP works but ping-by-name fails, the problem is DNS, not the network — that one comparison saves a lot of wrong-direction troubleshooting.

Use when: a service should be listening but a client can't connect

ss -tulpn | grep :PORT
lsof -i :PORT

Note: confirms the service is actually bound to the port before spending time on firewall rules or network paths that were never the problem.

Use when: a connection is failing and you need to see the actual packets

tcpdump -nn -i INTERFACE host HOST_IP and port PORT

tcpdump -nn -s 0 -i INTERFACE host HOST_IP and port PORT -w /tmp/capture.pcap

tcpdump -nn -s 0 -i INTERFACE -G 60 -W 5 -w /tmp/capture-%Y%m%d-%H%M%S.pcap

Reading it: repeated SYN with no SYN-ACK back means nothing's answering. A returned RST means something's actively rejecting the connection — two different problems that look the same from the client side.

Use when: a service needs checking, restarting, or its logs reviewed

systemctl status SERVICE
journalctl -u SERVICE -n 100
systemctl restart SERVICE
systemctl status SERVICE

Note: prefer reload over restart when a service supports it — it reapplies config without a hard stop/start, which matters more than it sounds on anything user-facing.

Use when: something broke and you don't yet know what

journalctl -xe
journalctl -p err -b
journalctl -u SERVICE --since "1 hour ago"

Note: -xe is almost always the right first move — it adds explanations to cryptic entries, not just raw log lines.

Use when: ps shows processes marked Z / defunct

ps -eo stat,pid,ppid,cmd | grep -w Z
ps -fp PARENT_PID

Gotcha: a zombie is already dead — you fix the parent, not the zombie. One or two is usually nothing; a steadily growing count means the parent isn't reaping its children, which is an application bug, not something memory tuning fixes.

Use when: you don't know what's broken yet

hostnamectl
uptime
df -h
free -h
systemctl --failed
ip a
journalctl -xe

Note: covers identity, load, disk, memory, failed units, and network in one pass — narrows down which of the "big five" is actually the problem before going deeper into any one of them.

Use when: checking what's installed, or which package owns a file

dnf info PACKAGE_NAME
rpm -qa | grep PACKAGE_NAME
rpm -qf /path/to/file

Note: rpm -qf is underused — genuinely handy when you find an unfamiliar binary or config file and need to know what put it there.

Use when: starting any new script, before adding real logic

#!/bin/bash
set -euo pipefail

echo "Starting script..."
# commands here
echo "Done."

Note: set -e exits on any failed command, -u errors on undefined variables, -o pipefail catches failures inside a pipe — without these three, a script can silently continue past a failure and do real damage downstream.

Use when: scripting a simple tar-based backup with automatic cleanup

#!/bin/bash
set -euo pipefail

DATE=$(date +"%Y-%m-%d_%H-%M-%S")
BACKUP_DIR="/path/to/backups"
SOURCE_DIR="/path/to/source"
RETENTION_DAYS=14

mkdir -p "$BACKUP_DIR"
tar -czf "$BACKUP_DIR/backup-$DATE.tar.gz" -C "$SOURCE_DIR" .
find "$BACKUP_DIR" -type f -name "backup-*.tar.gz" -mtime +"$RETENTION_DAYS" -delete

Note: the retention cleanup line is the part people forget to add until disk usage becomes a problem — build it in from the start, not after the fact.

Use when: scheduling a recurring script

crontab -e
crontab -l

0 2 * * * /path/to/script.sh >> /path/to/script.log 2>&1

# 0 2 * * *    = every day at 02:00
# */5 * * * *  = every 5 minutes
# 0 8 * * 1    = every Monday at 08:00

Note: always redirect output (>> log 2>&1) — a cron job that fails silently with no log is one of the most annoying things to debug later.

Use when: managing containerized services defined in a compose file

docker compose up -d
docker compose down
docker compose logs -f
docker compose pull && docker compose up -d
docker compose restart SERVICE_NAME

Note: pull then up -d is the safe update pattern — it fetches new images first and only recreates containers once they're actually available.

Use when: copying files locally or to a remote host, especially repeatedly

rsync -avh --progress SOURCE/ DESTINATION/
rsync -avh -e ssh SOURCE/ USERNAME@SERVER:/DESTINATION/
rsync -avh --dry-run SOURCE/ DESTINATION/

Gotcha: a trailing slash on the source means "copy the contents," no trailing slash means "copy the folder itself" — easy to get backwards and end up with an extra nested directory. Always --dry-run first on anything you didn't write yourself.

Use when: something is denied for no obvious permission reason

getenforce
grep denied /var/log/audit/audit.log | tail
restorecon -Rv /path/to/fix

Gotcha: never leave a system in setenforce 0 (permissive) to "fix" a problem — that disables the protection instead of fixing the actual context. Find the denial, then restorecon or semanage fcontext the correct label in.

Use when: a service's logs are growing unbounded and there's no built-in rotation

cat /etc/logrotate.d/SERVICE_NAME
logrotate -d /etc/logrotate.d/SERVICE_NAME
logrotate -f /etc/logrotate.conf

Note: most packaged services already ship a logrotate config — check before writing one from scratch. This is usually the actual fix for "disk filling up with logs," not manual deletion.

Use when: clock drift is suspected — auth failures, cluster weirdness, cert errors

chronyc tracking
chronyc sources -v
timedatectl

Note: clock drift is an underrated cause of weird failures — Kerberos/SSO auth and cluster heartbeats are especially sensitive to it, and the symptoms rarely look time-related at first glance.

Use when: applying updates, especially on anything production

dnf check-update
dnf update --setopt=install_weak_deps=False PACKAGE_NAME
needs-restarting -r

Gotcha: a blanket dnf update on production without checking what's included first is how unrelated things break. Patch narrowly when possible, and always check needs-restarting afterward — a patched-but-not-rebooted kernel is still running the old, vulnerable one.

Use when: proactively checking a physical disk, or investigating unexplained I/O errors

smartctl -a /dev/sda
smartctl -H /dev/sda
dmesg -T | grep -Ei "ata|scsi|i/o error"

Note: SMART isn't infallible, but a failing "reallocated sector count" trending upward is one of the most reliable early-warning signs of a dying disk — worth checking before it becomes an outage.

Use when: running something long, or working over an unreliable connection

tmux new -s work
# Ctrl+b, then d
tmux attach -t work
tmux ls

Note: the habit worth building is starting anything long-running (a migration, a big rsync, a patch job) inside tmux by default — a dropped SSH connection no longer means starting over.

Use when: editing a config file on a server with no other editor available

i
Esc
:wq
:q!
/searchterm

Gotcha: :q! exists for a reason — if you're not sure what state the file is in, that's the safe exit, not closing the terminal window and hoping.

Use when: checking overall cluster and service group status

hastatus -sum
hagrp -display SERVICE_GROUP

Note: check whether a group is frozen (automatic actions blocked) before assuming a fault needs manual intervention.

Use when: growing storage that's under Veritas, not plain LVM

vxdg list
vxprint -g DISK_GROUP -ht
vxassist -g DISK_GROUP growby VOLUME_NAME SIZE
# then grow the filesystem: xfs_growfs or resize2fs, matching the fs type

Gotcha: this is deliberately not a copy-paste recipe. Running plain OS LVM commands against Veritas-managed disks by mistake is one of the most common ways this goes wrong.

Use when: checking replication health, or investigating a DCM mode alert

vradmin -g DISK_GROUP repstatus RVG_NAME
# healthy output shows:
#   Data status: consistent, up-to-date
#   Replication status: replicating (connected)

# if vradmin can't connect:
service vras-vradmind status

Note: a failed VRAS daemon alongside a healthy vvr_stats is a known, restartable state — not a full outage.