A hand-held build guide: from an empty vSphere VM to a working Red Hat Satellite 6.19 server with no internet, FIPS mode, and the DISA STIG profile. Every command, every flag, and every symbol is explained in plain English.
Start Here: What You're Building and How to Read This Guide
Your organization runs many Red Hat Enterprise Linux (RHEL) servers on a network with no internet connection. That's called an air-gapped or disconnected network. Those servers still need software and updates. Red Hat Satellite is the answer: it's a private "app store and update center" that lives inside your network. You load software into Satellite by hand (from DVDs and approved transfer media), and every other server gets its software from Satellite instead of the internet.
In this guide you build that Satellite server from nothing: an empty virtual machine, the operating system, the network, the security hardening, and finally Satellite itself.
💡 Why is this so careful?
This server is built to DISA STIG rules. STIGs are the U.S. Department of Defense's security checklists, and they decide things like how disks are split up, how passwords work, and which ports can be open. Many steps look "extra." They exist because an auditor will check them.
Words to know
You'll see these words on every page. Come back here whenever one feels fuzzy.
Word
What it means in plain English
Server / VM
A computer that provides a service to other computers. A VM (virtual machine) is a computer made of software, running on a bigger physical machine called an ESXi host. You manage VMs in vSphere.
ISO
A single file that is an exact copy of a DVD. You "insert" it into the VM's virtual DVD drive.
Terminal / shell
The text screen where you type commands. The program reading your commands is called the shell (on RHEL it's called bash).
Command
One instruction you type and run by pressing Enter, like ls ("list files").
Flag / option
Extra settings added to a command, usually starting with - or --. ls -l means "list files, long format."
root
The all-powerful administrator account on Linux. Like "Administrator" on Windows.
sudo
"Do this one thing as root." You type your own password, not root's.
File path
A file's address. /etc/fstab means "the file fstab inside the folder etc, which is inside the top folder /." Linux uses forward slashes. / by itself is the very top folder.
Directory
Linux's word for a folder.
Partition / filesystem
A disk split into separate sections. Each section has its own fixed size and can't overflow into another.
Mount
Connecting a disk (or ISO) to a folder so you can open it. Like plugging in a USB stick and having it show up as a drive letter on Windows, except on Linux it shows up as a folder.
Package / RPM
A piece of software packed into one installable file (.rpm). Like an .msi on Windows.
Repository (repo)
A collection of packages the system is allowed to install from. Like a shelf in a store.
dnf
RHEL's package installer. It finds packages in repos and installs them. (yum is its older name and still works.)
Service
A program that runs in the background all the time, like a web server.
systemd / systemctl
systemd starts and stops every service. systemctl is the command you use to talk to it.
IP address
A computer's number on the network, like a street address (for example 10.10.208.50).
Port
A numbered door on a server. One IP can run many services, each behind its own port. Web pages use 443, SSH uses 22, and Cockpit uses 9090 by default.
Firewall
The gatekeeper that decides which ports outside computers may knock on. On RHEL it's firewalld.
SELinux
A second, stricter guard inside the server. Even if the firewall lets traffic in, SELinux decides which programs may use which ports and files.
DNS
The network's phone book: it turns names into IP addresses and back.
FQDN
Fully Qualified Domain Name: a server's full name, like sat01.lab.example, not just sat01.
Certificate
A digital ID card that proves a server is who it says it is. Your browser checks it before showing a secure page.
CA
Certificate Authority: the trusted office that signs ID cards. If your computer trusts the CA, it trusts every certificate the CA signed.
PKI
Public Key Infrastructure: the whole system of CAs, rules, and people that issues certificates. The DoD runs its own, called the DoD PKI.
Private key
The secret half of a server's identity. Whoever has it can pretend to be the server, so it never leaves the server.
Cockpit
A web page for managing a Linux server from your browser (logs, terminal, files).
How to read a command (the symbols)
Commands are built from small pieces. Here are the symbols this guide uses. You don't need to memorize them, since each command block explains its own pieces again.
Symbol
What it does
Everyday comparison
>
Send the command's output into a file, erasing whatever was in the file before.
Pouring water into a glass after dumping out what was in it.
>>
Send the output into a file, adding to the end and keeping what's there.
Topping off the glass.
|
"Pipe": hand one command's output to the next command.
An assembly line: station 1 passes its work to station 2.
&&
Run the next command only if the first one worked.
"Unlock the door, and if that worked, walk in."
;
Run the next command no matter what happened.
"Do this, then do that."
\ at line end
"This command continues on the next line." It lets one long command be written on several lines.
A hyphen at the end of a line in a book.
# at line start
A comment. The computer ignores the line. It's a note for humans.
A sticky note.
*
Wildcard: "anything." *.rpm means "every file ending in .rpm."
A blank in a crossword that any letter fits.
' ' single quotes
"Take this text exactly as written. Don't interpret any symbols in it."
Putting something in a sealed envelope.
Flags: when to use one dash (-) or two (--)
Flags are the extra settings after a command. You'll see two styles, and they usually do the same job. One is short and quick to type; the other is a full word that's easier to read.
🔒 The fixed rules
One dash + one letter = a short flag.ls -a means "list all files."
Two dashes + a word = a long flag.ls --all is the exact same thing, spelled out.
Short flags can be stacked behind one dash.ss -tlnp is the same as ss -t -l -n -p. Long flags can never be stacked; each one gets its own --: ss --tcp --listening --numeric --processes.
Some flags need a value after them. Short: a space, then the value: findmnt -o TARGET. Long: an = or a space: findmnt --output=TARGET.
Two dashes all alone (--) means "no more flags after this." Everything after it is treated as a name, even if it starts with a dash.
A letter only means something inside its own command.-p is a good example. It means something different in almost every command (see the table below). Never assume. Check the command's help.
🔀 Which one should you use?
Typing at the terminal: short flags. They're faster, and it's what you'll see in most examples.
Writing scripts, Ansible playbooks, or documentation: long flags. Someone reading --human-readable six months from now knows what it does; -h makes them look it up.
Both work the same. It's a readability choice, not a right-or-wrong choice.
The same letter, different meanings: -p
Command
What -p means there
Long form
Example
mkdir
Make parent folders too, and don't complain if it already exists
--parents
mkdir -p /media/rhel mkdir --parents /media/rhel
ss
Show the process (program) behind each port
--processes
ss -tlnp ss --tcp --listening --numeric --processes
semanage port
The network protocol (tcp or udp)
--proto
semanage port -a -t websm_port_t -p tcp 9091 semanage port --add --type websm_port_t --proto tcp 9091
systemctl show
Which property to show
--property
systemctl show cockpit.socket -p Listen systemctl show cockpit.socket --property=Listen
ssh
Which port to connect to
(none)
ssh -p 22 @
Short and long forms of the flags this guide uses
Command
Short
Long
What it does
Example (short → long)
ls
-a
--all
Include hidden files
ls -a /media/rhel ls --all /media/rhel
ls
-d
--directory
Show the folder itself, not what's inside
ls -ld /var/local/rpm-staging ls -l --directory /var/local/rpm-staging
Create an empty file literally named -test.txt. Without --, touch would complain that -t is a flag it wasn't expecting like that.
ls -l -- -test.txt
-l is a real flag (before the stop sign). -test.txt comes after it, so it's a file name.
rm -- -test.txt
Delete it, same idea.
📣 Commands that break the rule
One dash + a whole word: some older tools do this. find /var -type f -name '*.rpm' (not --type), and openssl req -new -key cockpit.key -out cockpit.csr. For these, -name is one flag, not -n -a -m -e.
No dashes at all: subcommands are plain words. systemctl restart, nmcli connection modify, hammer repository list. The words come first; flags come after.
PowerShell (Windows): always one dash + a word, and capital letters don't matter: Test-NetConnection -ComputerName sat01 -Port 9091.
How to look up any flag yourself
bashss --help
man ss
--help
Almost every command prints a short list of its flags, both short and long forms side by side, like -p, --processes.
man ss
Opens the full manual. Inside it, type /-p and press Enter to jump to -p, press n for the next match, and press q to quit.
💡 What does cat > file <<'EOF' really mean? (You'll use it a lot.)
Many steps need you to create a small settings file. The usual way is to open a text editor, but editors are clumsy inside the vSphere console. This trick lets you type the whole file right in the terminal instead:
examplecat > /etc/example.conf <<'EOF'
line one of the file
line two of the file
EOF
Read it left to right like a sentence:
cat means "take what I give you and hand it back out." By itself it just repeats text.
> /etc/example.conf means "don't show that text on screen. Put it into this file instead." If the file already exists, it is replaced.
<< means "the text I'm giving you is coming right here, on the next lines, not from another file." This is called a here-document ("the input is here").
EOF is the stop word. It stands for "End Of File." It tells the shell: "keep reading lines until you see a line that says only EOF." The stop word isn't saved into the file. You could use any word, like DONE. EOF is just the tradition, so everyone recognizes it.
The quotes around 'EOF' mean "copy every line exactly." Without them, the shell would try to "fill in" anything starting with $ (it would think $HOME is a variable and swap in your home folder). With the quotes, what you type is what goes in the file.
In one sentence: "Create this file, and here is everything that goes in it, up to the word EOF."
📣 The final EOF line must be alone on its line, with no spaces before or after it. If the terminal shows a > prompt and seems stuck, it's still waiting for that line. Type EOF and press Enter.
Your values
Every environment has its own names and addresses. Type yours below and every command in this guide updates to use them. Text highlighted like is one of these values. They are saved only in this browser.
📣 Keep real values out of GitHub
The defaults are made-up lab values. Type real hostnames and IPs only here in your browser. Never commit them to the repo or show them in portfolio screenshots.
Rules this guide always follows
🔒 Always true in this guide
You work as root. Log in as your normal user, then type sudo -i once. sudo means "as the administrator," and -i means "give me a full administrator session." After that, the prompt ends in # instead of $, and you never need to type sudo again.
Keep the vSphere console open. It's like plugging a keyboard and monitor straight into the server. It still works when the network, SSH, or Cockpit are broken.
Check before moving on. Every step ends with a ✔ Checkpoint. If a checkpoint fails, stop and fix it first. Problems pile up otherwise.
Order matters. For example, Cockpit must move off port 9090 before Satellite is installed, because Satellite needs 9090.
What you need before you start
Permission in vSphere to create a VM on the approved host, folder, and datastore (the datastore is the storage where VM disks and ISOs live).
rhel-9.8-x86_64-dvd.iso, the full RHEL DVD (not the small "boot" ISO), uploaded to the datastore.
Satellite-6.19.1-rhel-9-x86_64.dvd.iso uploaded to the datastore.
Your approved server name, IP addresses, gateway, DNS servers, and time server, typed into the form above.
Someone who can add records in Windows DNS.
A PKI contact (the people who run your certificate authority). Only needed for the trusted certificate in Step 6.
The approved process for moving files onto the air-gapped network.
📣 Three wrong passwords locks you out
STIG locks an account after 3 failed logins, and it usually stays locked until an admin unlocks it. That can include root. Know your root password and keep vSphere console access before you start. The unlock steps are on the Troubleshooting tab.
Step 1
Create the Empty VM in vSphere
First you build the "computer": an empty VM with the right amount of processing power, memory, and disks. Nothing is installed yet. Think of it as buying the hardware.
💡 Why two separate disks?
The small disk (200 GB) holds only the operating system. The big disk (2 TB) holds Satellite's software library, which keeps growing. If the library ever fills its disk, the operating system is on a different disk, so the server can still boot and you can still log in to fix it. It's like keeping your kitchen and your warehouse in separate buildings.
1a. Name, location, and operating system type
In vSphere, right-click the approved ESXi host → New Virtual Machine → Create a new virtual machine → Next.
Name it and pick the approved folder → Next.
Confirm the host (the "compute resource") → Next.
Pick the approved datastore → Next.
Compatibility: use your environment's approved level → Next.
Guest OS family Linux, version Red Hat Enterprise Linux 9 (64-bit) → Next. This tells vSphere which drivers to expect.
1b. The virtual hardware
Setting
Value
Plain-English reason
CPU
8 vCPU
Processing power. Satellite needs at least 4; 8 makes installs and syncs faster.
Memory
32 GiB
Working memory. 32 GiB is Satellite's documented minimum.
Hard disk 1
200 GB
The operating system's disk.
Hard disk 2 (Add New Device)
2 TB
Satellite's library, database, and logs.
Network adapter 1
208 Network, Connected unchecked
The network "cable" is unplugged on purpose, so nothing reaches the network during install. You plug it in during Step 4.
USB controller (Add New Device)
xHCI
USB 3 support for approved transfer drives. Don't swap it for the older EHCI+UHCI type.
CD/DVD drive 1
Datastore ISO File → rhel-9.8-x86_64-dvd.iso, Connect At Power On checked
"Inserts" the RHEL DVD so the VM boots from it.
VM Options → Boot Options → Firmware
EFI
The modern startup system (replaces old "BIOS"). Set it now. Changing it after install breaks booting.
Set every value in the table.
Compare the summary page against the table, then click Finish.
✔ Checkpoint 1
The VM has 8 vCPU, 32 GiB memory, EFI, a 200 GB disk, a 2 TB disk, and xHCI USB.
Network adapter 1 is on the 208 Network and not connected. The DVD drive holds the RHEL ISO.
🔧 If this happens
xHCI isn't in the list → the VM's compatibility level is too old. Ask your vSphere admin. Don't downgrade the USB type.
You can't find the ISO → you're in the wrong datastore, or your account can't browse it.
Step 2
Test That the VM Boots From the RHEL DVD
Before installing anything, make sure the right DVD is in the drive and it works. This takes two minutes and catches the most common setup mistake: the wrong ISO.
Select the VM → Launch Remote Console (or Launch Web Console). This is your screen and keyboard for the VM.
Power on the VM.
You should see a menu with Install Red Hat Enterprise Linux 9.8.
Choose it and wait for the installer's Installation Summary screen. (The installer is called Anaconda.) Don't start installing yet.
Look at Installation Source. It should say Local media with no warning. That means the installer found all the software on the DVD.
Power off the VM from vSphere.
🔧 If this happens
You see a Satellite screen or a "rescue" menu → the Satellite ISO is in the drive. It can't boot a computer. Swap it for rhel-9.8-x86_64-dvd.iso.
Installation Source shows an error → you used the small "boot" ISO, which has no software on it. Use the full DVD ISO.
✔ Checkpoint 2
The installer reached Installation Summary, the source said "Local media," and the network was still unplugged.
Screenshot saved for your portfolio.
Step 3
Install RHEL 9.8: FIPS, STIG, and Separate Partitions
Now you install the operating system. Three security choices happen here, and they're hard to add later. FIPS means only government-approved encryption. STIG means the DoD security checklist is applied automatically. Separate partitions means each important folder gets its own fenced-off space on the disk.
💡 Why does STIG want separate partitions?
Picture one big room where everything is stored. If one pile grows out of control, like logs from a noisy program, it fills the whole room and the system crashes. Separate partitions are walls: each folder gets its own room with a fixed size. If the log room fills up, the operating system's room is untouched.
Walls also let you put rules on each room, called mount options:
noexec: "nothing in this room may run as a program." It stops an attacker from dropping a program into a temp folder and running it.
nosuid: "ignore special 'run as admin' markers on files here." It blocks a common privilege trick.
nodev: "no device files here." Device files are special files that talk directly to hardware.
Those rules can only be set on a whole partition, never on a single folder. That's why STIG requires the walls.
3a. Start the installer in FIPS mode
Power on the VM. At the menu, highlight Install Red Hat Enterprise Linux 9.8 and press e ("edit this boot option").
Find the line that starts with linuxefi or linux. Move to the end of that line and type a space, then fips=1.
Press Ctrl+X to start with that change.
💡 What is FIPS, and why turn it on this early?
FIPS 140 is a U.S. government standard that lists which encryption methods are approved. fips=1 tells the installer "use only approved encryption from the first second." During install, the system creates its own secret keys (for example, SSH keys that prove the server's identity). If FIPS is already on, those keys are made the approved way. Turning FIPS on later would leave keys that were made the unapproved way. Also, since RHEL 9.6, picking the STIG profile no longer turns FIPS on for you, so you must do this yourself.
3b. Software and security profile
Software Selection → Server (no desktop) or Minimal Install. Add nothing extra. Less software means fewer things to patch and secure.
Security Profile → select DISA STIG for Red Hat Enterprise Linux 9 → Select profile → Done. The installer will now apply hundreds of security settings for you.
Leave Network & Host Name alone. The network is unplugged on purpose.
Time & Date: pick your time zone. The time server comes in Step 4.
3c. Split up the disks
This is the biggest part of the install. You'll use LVM (Logical Volume Manager). LVM pools a disk's space into a volume group, then carves that pool into logical volumes (the "rooms"). The big advantage: you can make a room bigger later without reinstalling.
🔒 Always true for the disk layout
The 200 GB disk gets its own pool named rhel (operating system). The 2 TB disk gets its own pool named data (Satellite). The two never mix.
These folders must each get their own room (STIG): /boot, /boot/efi, /home, /tmp, /var, /var/log, /var/log/audit, /var/tmp.
Satellite's two heaviest folders also get their own rooms: /var/lib/pulp (the software library) and /var/lib/pgsql (the database).
Leave some of the data pool unused, so you have room to grow later.
🔀 Your choice
The sizes below are a sensible start, not a rule. You can grow any room later. What must not change is which folders get their own room.
Folder (mount point)
Size
Type
Disk / pool
What lives there
/boot/efi
600 MiB
EFI System Partition
200 GB disk
Startup files the EFI firmware reads first
/boot
2 GiB
XFS, standard partition
200 GB disk
The Linux kernel (core of the OS)
swap
4 GiB
LVM
rhel
Overflow space when memory is full
/
150 GiB
XFS, LVM
rhel
The operating system itself ("root" folder)
/home
10 GiB
XFS, LVM
rhel
Users' personal folders
/var
250 GiB
XFS, LVM
data
Data that changes: caches, queues, service data
/var/lib/pulp
1200 GiB
XFS, LVM
data
Satellite's software library (the biggest)
/var/lib/pgsql
50 GiB
XFS, LVM
data
Satellite's database
/var/log
20 GiB
XFS, LVM
data
Log files
/var/log/audit
20 GiB
XFS, LVM
data
Security audit records (who did what)
/var/tmp
20 GiB
XFS, LVM
data
Temporary files that survive a reboot
/tmp
20 GiB
XFS, LVM
data
Temporary files
XFS is the type of filesystem ("the way files are organized inside a room"). It's RHEL's default. The data pool uses about 1,600 GiB of roughly 1,860 GiB, leaving about 260 GiB to grow into.
First, the operating system disk
Installation Destination → check only the 200 GB disk. Leave the 2 TB disk unchecked for now.
Under Storage Configuration choose Custom → Done.
In the partitioning scheme dropdown pick LVM.
Click + and add /boot/efi 600 MiB, then /boot 2 GiB. Click each one and set Device Type to Standard Partition. (The computer must read these before LVM starts, so they can't be inside LVM.)
Add swap 4 GiB, / 150 GiB, and /home 10 GiB. Check that each shows Volume Group rhel.
Done → Accept Changes.
Then, the data disk
Open Installation Destination again. Check both disks → Custom → Done.
Add /var 250 GiB. In its Volume Group dropdown choose Create a new volume group.
Name it data, select only the 2 TB disk, leave RAID as is, set Size policy to As large as possible → Save.
Check that /var now says Volume Group data.
Add the rest, each in pool data: /var/lib/pulp 1200 GiB, /var/lib/pgsql 50 GiB, /var/log 20 GiB, /var/log/audit 20 GiB, /var/tmp 20 GiB, /tmp 20 GiB.
Compare every row with the table → Done → Accept Changes.
🔧 If this happens
/ or the rhel pool is on both disks → click Reset All and redo "First, the operating system disk." Nothing is written to disk until you click Begin Installation, so this is safe.
The dropdown only shows rhel → you haven't created data yet. Pick Create a new volume group.
The STIG profile shows a partition warning → one of the required folders is missing. Compare with the table.
3d. Accounts
Root Password: set the approved root password.
User Creation: create your admin account and check Make this user administrator (this lets it use sudo).
📣 The console can mistype symbols
The vSphere console sends which key you pressed, not which letter you meant. If your computer's keyboard layout doesn't match what the VM expects, symbols like @ # $ \ | can come out different. Password typing is hidden, so you'd never see it, and 3 misses locks you out.
For your first login, pick a password (STIG still needs 15+ characters with upper, lower, a number, and a symbol) whose symbol you've tested in the console first. Safe bets are - and .. Change it later from SSH or Cockpit if you like.
3e. Install and first boot
Check that every item on the summary is ready → Begin Installation. This is when the disks are actually written.
When it finishes, don't click Reboot yet. In vSphere: Edit Settings → CD/DVD drive 1 → uncheck Connect At Power On → OK. Otherwise the VM boots from the DVD again and restarts the installer.
Now click Reboot System.
Log in on the console as , then type sudo -i and your password. The prompt should end in #, which means you're root.
A one-line file that says which RHEL version is installed.
/proc/sys/crypto/fips_enabled
A live "switch" the kernel exposes as a file. 1 = FIPS on, 0 = off.
fips-mode-setup --check
A FIPS tool. --check means "just tell me the status, don't change anything."
getenforce
"Is SELinux enforcing its rules right now?"
You should see:Red Hat Enterprise Linux release 9.8 (Plow), then 1, then FIPS mode is enabled., then Enforcing.
bash — check that every room existsfindmnt -t xfs,vfat -o TARGET,SOURCE,SIZE,OPTIONS
lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS
vgs
findmnt
"Find mounts": lists every connected filesystem (every room).
-t xfs,vfat
type filter: only show XFS rooms and the vfat (EFI) room. Hides dozens of internal system entries.
-o TARGET,SOURCE,SIZE,OPTIONS
output only these columns: the folder, the disk piece behind it, its size, and its rules (mount options).
lsblk
"List block devices": draws a tree of disks → pieces → LVM rooms, so you can see which disk each room sits on.
-o NAME,SIZE,TYPE,MOUNTPOINTS
Only these columns.
vgs
"Volume groups summary": shows each pool and how much is still unused (VFree).
You should see: one line per folder from the table. In lsblk, rhel-… rooms appear only under the 200 GB disk and data-… rooms only under the 2 TB disk. vgs shows free space left in data.
3g. Check the STIG room rules
The STIG profile already wrote these rules into /etc/fstab. fstab ("filesystem table") is the list the server reads at every boot to know which room connects to which folder, and with what rules. Confirm the rules took effect:
Folder
STIG expects
/boot
nodev, nosuid
/boot/efi
nosuid
/home
nodev, nosuid, noexec
/tmp
nodev, nosuid, noexec (see the Satellite exception below)
STIG says /tmp should be noexec (no programs run there). Satellite's installer needs to run small helper programs from /tmp and fails without that. The accepted fix: remove noexec from /tmponly, keep nodev and nosuid, and write it down as an approved exception (a POA&M or risk acceptance) with your ISSO. That way an auditor sees a documented decision, not a mistake.
bash — only if /tmp shows noexecgrep -n ' /tmp ' /etc/fstab
sed -i '/[[:space:]]\/tmp[[:space:]]/ s/noexec,\?//; /[[:space:]]\/tmp[[:space:]]/ s/,\([[:space:]]\)/\1/' /etc/fstab
grep -n ' /tmp ' /etc/fstab
systemctl daemon-reload
mount -o remount,exec /tmp
findmnt -o TARGET,OPTIONS /tmp
grep
"Search for text." Finds lines in a file that contain what you typed.
-n
Show the line number, so you can see exactly which line you're about to change.
' /tmp '
The text to find. The spaces stop it from matching /var/tmp.
sed
"Stream editor": finds and replaces text automatically.
-i
Edit the file in place (save the change), instead of just printing the result.
s/noexec,\?//
s/old/new/ means "substitute." Here it replaces noexec (and a comma after it, if there is one) with nothing, which deletes it. The part before s limits this to the /tmp line only.
second s/…/
Tidies up a leftover comma if noexec was last in the list.
systemctl daemon-reload
Tells systemd "re-read your settings." systemd builds its mount list from fstab, so it has to re-read after any edit.
mount -o remount,exec /tmp
-o passes options. remount means "change the rules now without disconnecting." exec means "allow programs again."
You should see: the /tmp line in fstab and the findmnt output both show nodev,nosuid and no noexec. If the sed result looks wrong, fix the line by hand with vi /etc/fstab. A broken fstab line can stop the next boot.
✔ Checkpoint 3
RHEL 9.8, FIPS = 1, SELinux Enforcing.
Every folder from the table is its own room, in the right pool.
STIG rules are in place, and /tmp has no noexec (exception written down).
Step 4
Give the Server Its Network, Name, and Clock
A server on a network needs three things to be trusted: a fixed address (IP), a name everyone agrees on (DNS), and the correct time. Satellite is picky about all three.
💡 Why does Satellite care so much?
When Satellite installs, it writes its own name into its ID cards (certificates), its database, and dozens of settings. If the name, the DNS "phone book" entry, and the reverse lookup (IP back to name) don't all agree, the install fails or builds a broken server. Time matters because ID cards have start and end dates. If the server's clock is wrong, it thinks valid cards are expired or not valid yet.
4a. Plug in both network "cables"
📣 Don't forget adapter 1
Adapter 1 has been unplugged since Step 1. If you only add adapter 2, the main address, gateway, and DNS never work, and it looks like everything else is broken.
vSphere → Edit Settings → Network adapter 1 (208 Network): check Connected and Connect At Power On.
Add New Device → Network Adapter → pick the .210 repository network → check Connected and Connect At Power On.
Click OK.
4b. Set the server's name
bashhostnamectl set-hostname
hostnamectl status --static
hostnamectl
The tool that controls the server's name.
set-hostname
"Set the permanent name to this." It's saved in /etc/hostname.
status --static
"Show me only the permanent (static) name."
You should see: . Always the full name, never the short one.
4c. Find what the network cards are called
bashnmcli device status
nmcli connection show
nmcli
The command-line tool for NetworkManager, the program that manages network settings on RHEL.
device status
Lists the network cards (hardware) and whether each is connected.
connection show
Lists saved connections (settings profiles). A card is the hardware; a connection is the settings that card uses.
You should see: two ethernet cards, often named ens192 and ens224. Type the .208 one in the NIC1 field at the top of the page and the .210 one in NIC2.
4d. Main network card (.208): fixed address, gateway, DNS
The name you're giving the profile. Any name works; this one describes its job.
ifname NIC2
Which network card (interface) it belongs to.
type ethernet
The kind of connection. Wired cards are always ethernet.
ipv4.method manual
Fixed address, no DHCP.
ipv4.addresses
The address with its /24. There's no gateway line on purpose.
ipv4.never-default yes
"Never use this card as the main exit door." All normal traffic keeps using .208, which avoids confusing routing.
connection.autoconnect yes
Turn on at every boot.
connection up
Turn it on now.
🔧 If this happens
nmcli connection show already lists something like "Wired connection 1" on NIC2 → RHEL made an automatic profile. Delete it first with nmcli connection delete "Wired connection 1" (the quotes keep the spaces together as one name). Otherwise two profiles fight over the same card.
4f. Check the network
baship -4 address show
ip -4 route show
cat /etc/resolv.conf
ip
The low-level network tool.
-4
Only show IPv4 (the familiar 4-number addresses). Hides IPv6 noise.
address show
Which address each card has.
route show
The "map" of where traffic goes. The default line is the main exit door.
cat /etc/resolv.conf
Shows the DNS servers the system is actually using.
You should see: on NIC1 and the .210 address on NIC2. Exactly one line starting default via . Both DNS servers in resolv.conf.
4g. Add the server to the DNS phone book (Windows DNS)
Two entries are needed. An A record is name → IP ("look up sat01, get 10.10.208.50"). A PTR record is IP → name, the reverse lookup. Satellite checks both.
On an approved DNS server, open DNS Manager → Forward Lookup Zones → right-click your zone → New Host (A or AAAA).
Name: the first part of your FQDN. Check that the full name shown reads .
Open Reverse Lookup Zones and confirm the PTR record is there. If not, the reverse zone doesn't exist yet. Ask the DNS admin.
bash — back on the servergetent hosts
getent hosts
hostname -f
getent hosts NAME
"Look up this name the same way the system would." This is a forward lookup, name → IP.
getent hosts IP
The reverse lookup, IP → name.
hostname -f
"What's my full name?"
You should see: all three agree on ↔ . If they don't, stop here. Don't go on to Satellite until they match.
Optional: let this lab look up names in a production DNS zone
Only if the lab must find names in production. It needs a network path and port 53 (DNS) open from the lab's DNS servers to production's.
On the lab DNS server: DNS Manager → right-click Conditional Forwarders → New Conditional Forwarder. (This says "for names ending in X, ask those servers.")
Enter the production zone name and both production DNS server IPs. Only choose "store in Active Directory" if the DNS team approves.
Test from the server: getent hosts <a-production-name>.
4h. Set the clock source (chrony)
💡 Why the clock needs a source
Computer clocks drift a few seconds a day. Normally servers correct themselves from internet time servers, but this network has no internet. So you point the server at an internal time server your team runs. The time program on RHEL is chrony, and its settings file is /etc/chrony.conf.
Show the lines that name time sources. -n adds line numbers. -E turns on "either/or" patterns. ^ means "at the start of the line," and (pool|server) means "pool OR server."
sed -i 's/^pool /#pool /'
Adds a # in front of every pool line, turning them into comments. Those are internet time servers this network can't reach.
grep -q … ||
-q means "quiet: just tell me yes or no." || means "if the answer was no, run the next part." So the server line is added only if it isn't already there.
echo '…' >> file
echo prints the text, and >>adds it to the end of the file.
iburst
"Send a quick burst of requests at startup," so the clock syncs in seconds instead of minutes.
systemctl restart chronyd
Restart the time service so it reads the new setting. (chronyd is the service; the d means "daemon," a background program.)
chronyc sources -v
List the time sources. -v (verbose) adds a legend explaining the symbols.
chronyc tracking
Shows how far off the clock currently is.
You should see: your time server marked ^* (the * means "currently using this one") and a tiny "System time" offset in tracking. If STIG already added approved server lines, keep them.
✔ Checkpoint 4
Both network cards connected. One default route, via .208.
Name, forward lookup, and reverse lookup all match.
chrony is synced to the internal time server.
Step 5
Build an Offline Software Shelf and Install Cockpit
To install anything, dnf needs a repository, a shelf of packages. Normally that shelf is on Red Hat's website. Here you turn the RHEL DVD itself into the shelf, and make sure it's still there after every reboot.
5a. Put the RHEL DVD back in the drive
vSphere → Edit Settings → CD/DVD drive 1 → Datastore ISO File = rhel-9.8-x86_64-dvd.iso. CheckConnected and Connect At Power On again. You turned Connect At Power On off in Step 3.
5b. Connect the DVD to a folder, permanently
💡 Why use the DVD's label?
Linux names DVD drives /dev/sr0, /dev/sr1, and so on, in the order it finds them. When you add a second DVD in Step 7, those names can swap after a reboot, and you'd connect the wrong disc. Every ISO also has a label (like a name written on the disc) that never changes. Connecting by label always gets the right one.
bashblkid /dev/sr0
blkid
"Block ID": shows a disk's label, its unique ID, and its filesystem type.
/dev/sr0
The first DVD drive. /dev is where Linux lists all hardware.
You should see: something like LABEL="RHEL-9-8-0-BaseOS-x86_64". Copy the exact label into the RHEL ISO label field at the top of the page.
bashmkdir -p /media/rhel
echo 'LABEL= /media/rhel iso9660 ro,nofail 0 0' >> /etc/fstab
systemctl daemon-reload
mount /media/rhel
ls -a /media/rhel
mkdir -p /media/rhel
"Make directory": creates the folder where the DVD will appear. -p means "don't complain if it already exists."
echo '…' >> /etc/fstab
Adds one line to the boot-time mount list. It must be >> (add). A single > would erase every other line, and the server wouldn't boot.
LABEL=…
"Find the disc by this label," not by its drive name.
/media/rhel
The folder where the DVD's contents will show up.
iso9660
The type of filesystem DVDs use.
ro
Read-only. You can't write to a DVD anyway.
nofail
"If the DVD isn't there, keep booting." Without it, a missing DVD would freeze the boot in emergency mode.
0 0
Two old settings: "don't back this up" and "don't disk-check it at boot."
systemctl daemon-reload
Makes systemd re-read fstab.
mount /media/rhel
Connect it now. Giving only the folder makes mount look up the rest in fstab, which also proves your new line works.
ls -a
"List" the folder. -a shows all files, including hidden ones that start with a dot.
You should see:.treeinfo, media.repo, BaseOS, and AppStream. If BaseOS or AppStream is missing, it's the wrong ISO.
5c. Tell dnf about the shelf (one file, used for the whole build)
RHEL splits its software into two shelves: BaseOS (the core operating system) and AppStream (applications and tools). You create one settings file that points dnf at both. This uses the <<'EOF' trick from Step 0.
"Create this file with everything I type until the word EOF." See Step 0.
/etc/yum.repos.d/
The folder where dnf looks for shelf definitions. Any file ending in .repo here is read.
[rhel-local-baseos]
The shelf's ID, its short unique name. The square brackets start a new shelf.
name=
A friendly description for humans.
baseurl=file:///…
Where the shelf is. file:// means "a folder on this machine" instead of a website. (The third / is the start of the path.)
enabled=1
Turn this shelf on. 1 = on, 0 = off.
gpgcheck=1
Check Red Hat's digital signature on every package before installing, like checking the tamper seal on a medicine bottle. Keep it on. STIG requires it.
gpgkey=
Where Red Hat's public "seal checker" key is stored. It's already on the system from the install.
rpm --import
Tell the package system "trust packages sealed with this key."
dnf clean all
Throw away any old saved shelf lists so dnf reads the new file fresh.
dnf repolist
List the shelves dnf can use right now.
You should see:rhel-local-baseos and rhel-local-appstream in the list. A message about "subscription management" is harmless, because this server isn't registered with Red Hat online.
The DVD connects from fstab by label, so it survives a reboot.
One repo file, both shelves listed, signature checking on.
Cockpit and semanage installed.
Step 6
Move Cockpit to Port 9091 and Give It a Trusted ID
Cockpit normally answers on port 9090. Satellite also needs 9090 for one of its own services (the Capsule/foreman-proxy). Two programs can't share one door, so Cockpit moves next door to 9091 now, before Satellite arrives.
🔒 The three-layer rule for ANY port change
On a hardened RHEL server, a port change has to pass three separate checkpoints. Miss one and it fails, usually with no error message.
The internal guard (SELinux): is the program allowed to use this door?
The outer fence (firewall): may outside traffic reach this door?
The service itself (systemd): is the program actually standing behind this door?
This is always true, for every service. Moving SSH to 2222 or Apache to 8080 works the same way; only the names change.
"Save this change so it survives a reboot." Permanent changes don't take effect until --reload.
--add-service=cockpit
"Let Cockpit traffic through the fence."
--service=cockpit --add-port=9091/tcp
Changes what "Cockpit" means to the firewall: add port 9091 to its definition.
--remove-port=9090/tcp
Take 9090 out of Cockpit's definition, freeing it for Satellite.
--reload
"Apply the saved changes now."
--info-service=cockpit
Show which ports the "cockpit" definition now includes.
--list-services
Show every service allowed through the fence.
You should see:ports: 9091/tcp for cockpit, and cockpit in the services list.
6d. Layer 3: the service itself (systemd)
Cockpit uses socket activation: systemd stands at the door on Cockpit's behalf and wakes Cockpit up only when someone knocks. So you tell systemd which door to stand at. You do it with a small override file called a drop-in, instead of editing Cockpit's original file, so software updates won't undo your change.
Creates the drop-in folder. Any folder named <unit>.d holds overrides for that unit.
cat > … <<'EOF'
Create the override file with the next three lines (see Step 0).
[Socket]
"These settings are about the door."
ListenStream= (blank)
"Forget the doors you were told about before." This erases the built-in 9090. Without this line, Cockpit would answer on both 9090 and 9091.
ListenStream=9091
"Now stand at door 9091."
systemctl daemon-reload
Make systemd re-read its settings files.
enable --now
enable means "start at every boot." --now means "and start it right now too."
restart
Stop and start again so it moves to the new door.
show -p Listen
Show one property, the door it's listening on.
ss -tlnp | grep …
Double-check from the outside, same as 6a.
You should see:Listen=[::]:9091 (Stream), and ss shows 9091 owned by systemd, with nothing on 9090. Seeing systemd instead of cockpit is correct, since systemd is holding the door.
🔀 Your choice: systemctl edit vs. this method
systemctl edit cockpit.socket does the same job through a text editor, but saves to a file named override.conf. Pick one method. If both files exist, both apply, which gets confusing.
6e. Test it, from the inside out
Always test the closest thing first (the server itself), then work outward to your browser. The first test that fails tells you which layer is broken. The full checklist is in Troubleshooting → port won't load.
bash — on the servercurl -skI https://localhost:9091 | head -1
curl
A command-line web browser.
-s
Silent: no progress bar.
-k
Skip the certificate (ID card) check. This test only asks "is Cockpit answering at all?"
-I
Only fetch the page's header, not the whole page.
https://localhost:9091
localhost means "this same machine."
| head -1
Show only the first line.
You should see: a line starting with HTTP/1.1 200. 200 means "OK."
PowerShell — on the Windows jump serverClear-DnsClientCache
Test-NetConnection -ComputerName -Port 9091
Clear-DnsClientCache
Makes Windows forget names it looked up earlier, so it asks DNS fresh.
Test-NetConnection
Windows' "can I reach it?" tester.
-ComputerName
Which server. Using the full name tests DNS at the same time.
-Port 9091
Try to knock on this door.
You should see:TcpTestSucceeded : True. Then open Edge to https://:9091.
6f. Install the trusted certificate chain (recommended)
Right now Cockpit shows your browser a self-signed certificate, an ID card it made for itself. Edge doesn't trust that and shows a warning. The fix is to get a card signed by your organization's certificate authority.
Your Windows DNS servers also run Active Directory Certificate Services (AD CS), so the organization already has an internal certificate authority. DNS publishes the VM's name, while the CA signs proof that the name belongs to this server.
📣 Keep private keys separate
A private key proves the server's identity. Keep it in its service-specific, root-only location on the server.
Never transfer /root/cockpit_cert/cockpit.key to AD CS.
Never place it in /etc/pki/ca-trust/source/anchors/. That folder is for public CA certificates only.
Rule: public certificates travel; private keys never leave the server.
Show the certificate steps
Step A: Get the CA's public certificate
Ask the PKI owner whether the AD CS instance is the root CA (the top of the chain of trust) or an issuing CA (a middle CA that the root trusts).
If it's an issuing CA, use the approved root CA server for the next steps.
On the root CA server, open Windows PowerShell as Administrator (right-click → Run as administrator).
Export the root CA's public certificate to your Desktop with the commands below.
PowerShell — on the root CA serverSet-Location "$HOME\Desktop"
certutil -ca.cert internal-root-ca.cer
Set-Location
PowerShell's "go to this folder" (like cd).
"$HOME\Desktop"
$HOME is your user folder, so this is your Desktop. The quotes keep the path together.
certutil
Windows' built-in certificate tool.
-ca.cert
"Export this CA's own public certificate."
internal-root-ca.cer
The file name to save it as.
💡 Where do the CA files come from?
The exported internal-root-ca.cer file contains the root CA's public certificate. It does not contain the CA's private key, so it's safe to copy around.
If AD CS uses an issuing CA, get that public intermediate certificate from the PKI owner as well. The issuing certificate connects the Cockpit certificate to the trusted root, like a middle link in a chain: root → intermediate → Cockpit.
Step B: Teach the RHEL server to trust your CA
Move the .cer file(s) through the approved transfer process into /etc/pki/ca-trust/source/anchors/ on the server. They must be in PEM (Base-64) format. If head -1 on the file shows -----BEGIN CERTIFICATE-----, it is.
bashupdate-ca-trust extract
update-ca-trust extract
"Rebuild the server's list of trusted CAs, including anything new in the anchors folder."
Step C: Make the private key and the certificate request on the server
You create two things. The private key is the secret, and it stays here. The CSR (Certificate Signing Request) is a form that says "please sign an ID card for this name." Only the CSR goes to the CA.
Make a folder, and if that worked (&&) lock it so only root can open it. 700 = owner can do everything, nobody else can do anything.
openssl
The Swiss-army knife for keys and certificates.
genrsa
"Generate an RSA private key." RSA is a FIPS-approved key type.
-out FILE
Where to save the result.
4096
Key size in bits. Bigger means harder to break.
chmod 600
Lock the key file so only root can read or change it. 600 = owner read and write, nobody else anything.
prompt = no
"Don't ask me questions; take the answers from this file."
CN =
"Common Name": the server's name on the card.
basicConstraints CA:FALSE
"This card can't sign other cards." It's a server ID, not a CA.
keyUsage / extendedKeyUsage serverAuth
"This card may only be used to prove a server's identity."
subjectAltName = DNS:…
The names the card is valid for. Browsers check this field, so it must match the URL you type exactly.
req -new
"Make a new certificate request."
-key
Which private key to build the request from.
-config
Which settings file to use (the one you just wrote).
-out
Where to save the CSR.
Transfer onlycockpit.csr to AD CS. Have it signed with the approved web server template, in Base-64 (PEM) format.
Bring the signed card back and save it as /root/cockpit_cert/cockpit.crt.
Step D: Give the new ID card to Cockpit
bashcat /root/cockpit_cert/cockpit.crt > /etc/cockpit/ws-certs.d/50-internal.cert
# only if you have an intermediate CA:
# cat /etc/pki/ca-trust/source/anchors/intermediate.pem >> /etc/cockpit/ws-certs.d/50-internal.cert
install -m 600 /root/cockpit_cert/cockpit.key /etc/cockpit/ws-certs.d/50-internal.key
/usr/libexec/cockpit-certificate-ensure --check
systemctl restart cockpit.socket
cat … > …50-internal.cert
Copy the signed card into Cockpit's certificate folder. > creates or replaces the file.
# lines
Comments, which do nothing. If you have an intermediate, remove the # from the second one. Its >>adds the intermediate after your card, forming the chain.
install -m 600
Copy the key and lock it to root-only in one step (-m = set mode, meaning permissions).
50-internal.cert / .key
Cockpit picks the card and key that share the same name and come last alphabetically. The 50- makes sure yours wins.
cockpit-certificate-ensure --check
"Which card will you use, and does the key match?" It only checks and changes nothing.
systemctl restart cockpit.socket
Restart so Cockpit picks up the new card.
Step E: Teach the Windows workstation to trust your CA
Press Win+R, type mmc → File → Add/Remove Snap-in → Certificates → Computer account → Local computer.
Look in Trusted Root Certification Authorities for your internal root CA. If it's missing, right-click → Import internal-root-ca.cer there.
Import any intermediate into Intermediate Certification Authorities.
Open Edge to https://:9091. There should be no warning.
6g. Lab vs. production: what changes when DoD issues the certificate
In this lab, your own internal CA (AD CS) signs the certificate. In production on a DoD network, you don't sign it yourself. You send the request to the DoD PKI (Public Key Infrastructure, the DoD's official certificate system), and it sends back an official DoD certificate. The steps on the server look almost the same. What changes is who signs, who is trusted, and how much paperwork is involved.
💡 Three levels of ID card
Self-signed (Cockpit's default): the server made its own ID card. Like writing your own name on a sticky note. Nobody has a reason to trust it, so browsers warn.
Internal CA (this lab): your organization's own office signs it. Trusted only by computers you told to trust that office. Fine for a lab or a small enclave.
DoD PKI (production): the Department of Defense's official office signs it. DoD computers already trust the DoD root CAs (they're installed by the DoD InstallRoot tool or by group policy), so every DoD workstation trusts your server with no extra setup. Auditors expect this on DoD systems.
What
Lab (internal CA)
Production (DoD PKI)
Who signs
Your AD CS server.
A DoD issuing CA (for servers, usually one of the DOD SW CA series).
Who you send the CSR to
Whoever runs your AD CS.
Your organization's PKI sponsor or Registration Authority (RA) / Trusted Agent, through the enrollment process your command uses. You can't just sign it yourself.
Approval
Usually immediate.
A formal request that checks you own the server and the name. Expect days, not minutes. Plan it into the schedule.
Root of trust
Your internal root CA (you import it everywhere).
The DoD Root CAs. Already trusted by DoD Windows machines. Not trusted by RHEL out of the box, so you add them (below).
Chain
Root → (maybe) your issuing CA → server.
DoD Root CA → DOD SW CA (intermediate) → your server. You must send the intermediate with your certificate.
Name on the card
Whatever you put in the CSR.
The DoD CA may rewrite the subject into DoD format (e.g. CN=…, OU=PKI, OU=DoD, O=U.S. Government, C=US). What browsers check is the SAN (the DNS name), so that must still match your FQDN exactly.
Key rules
Your choice.
Must follow DoD policy (RSA 2048 is the common minimum). Ask your RA which key sizes they accept before generating the key, because some enrollment systems reject sizes they don't expect.
Revocation checking
Usually skipped.
DoD requires checking whether certificates were cancelled (CRLs, the "cancelled cards" lists). On an air-gapped network you have to bring updated CRLs in regularly, or checks start failing when the old list expires.
Expiration
Whatever you set.
Fixed by DoD policy. Put the expiry date in your calendar and start the renewal request weeks early, since approval takes time.
Which network
Lab.
On NIPRNet (unclassified) it's the DoD PKI. On SIPRNet (classified) it's a separate system, the NSS PKI, with its own roots and its own RA. The steps are the same shape, but the CAs and contacts are different.
What gets a cert
Cockpit only (optional).
Cockpit and Satellite's own web page (port 443). Satellite part below.
🔒 Always true in lab and production
The private key is made on the server and never leaves it. Only the CSR travels. That's exactly the same with DoD.
The SAN must match the name people type in the browser.
You install the certificate plus its intermediate chain, and the key stays chmod 600.
Production step 1: Teach RHEL to trust the DoD roots
DISA publishes the DoD root and intermediate CA certificates as a downloadable bundle on the public DoD Cyber Exchange (public.cyber.mil, PKI/PKE section). Download it on an approved connected system, bring it across with the approved transfer process, then:
Find the bundle file. .p7b (PKCS#7) is a "bag" that holds many certificates in one file.
openssl pkcs7
Open a PKCS#7 bag.
-in FILE
Which bag. Use the file whose name ends in .pem.p7b. If you only have a plain .p7b, add -inform DER ("the file is in binary format").
-print_certs
"Take every certificate out of the bag and write them as normal text certificates."
-out FILE
Save them straight into the trust anchors folder.
grep -c 'BEGIN CERTIFICATE'
-c = count matching lines. Each certificate starts with that line, so this counts how many you got.
update-ca-trust extract
Rebuild the server's trusted list with the new CAs.
trust list | grep -i 'DoD Root'
trust list shows every trusted CA. grep -i finds DoD roots, ignoring upper/lower case.
You should see: a count of several dozen certificates, and DoD Root CA entries in the trust list.
📣 Only trust what came from DISA
Take the bundle only from the official DISA source through your approved transfer process. Adding a CA to the anchors folder tells the server "trust anything this CA signs," so a fake CA there would be a serious hole.
Production step 2: Make the key and CSR
Same as Step C above. Only change the key size if your RA requires it (for example 2048 instead of 4096), and use the exact name format they ask for. Then send onlycockpit.csr to your PKI sponsor or RA using your command's process.
Production step 3: Check what DoD sent back
You'll get your server certificate, and often the DOD SW CA intermediate. Before installing, check the card is right:
Show the expiration date. Put it in your calendar.
-ext subjectAltName
Show the DNS names it's valid for. It must include your FQDN.
openssl verify
"Does this card chain up to a CA I trust?"
-CAfile …ca-bundle.crt
Use the server's full trusted list, which now includes the DoD roots and intermediates from step 1.
diff <(…) <(…)
diff compares two things and prints only differences. <( ) means "treat this command's output like a file." It compares the public key inside the certificate with the public key made from your private key.
openssl pkey -in KEY -pubout
Read the private key and print only its public half.
You should see: the issuer is a DoD CA, the SAN has your FQDN, verify says OK, and diff prints nothing (nothing printed = the key matches the card). If diff prints lines, this certificate was made from a different key. Don't install it.
Production step 4: Install it for Cockpit
Same as Step D. Put your certificate first, then the DOD SW CA intermediate after it with >> (remove the # on that line). The DoD intermediate is not optional.
Production step 5: Give Satellite's own web page the DoD certificate
In production, Satellite's main page (port 443) needs an official certificate too, not the one it made for itself during install. Satellite's certificate must include the Satellite FQDN in its SAN, so you can reuse the same request process with its own key and CSR (for example in /root/satellite_cert/).
Glue the intermediate and the root into one "chain" file. > creates it.
katello-certs-check
Satellite's certificate checker. It only checks, changing nothing, and at the end prints the exact installer command to use.
-c
Your server certificate.
-k
Its private key.
-b
The CA bundle (the chain file).
You should see: "Validation succeeded," followed by a ready-made satellite-installer command. Run the command it prints. It will look like this:
bash — example; use the one katello-certs-check printssatellite-installer --scenario satellite \
--certs-server-cert /root/satellite_cert/satellite.crt \
--certs-server-key /root/satellite_cert/satellite.key \
--certs-server-ca-cert /root/satellite_cert/dod-chain.pem \
--certs-update-server --certs-update-server-ca
--scenario satellite
Same setup as before, just updating it.
--certs-server-cert
The DoD certificate for the web page.
--certs-server-key
Its private key.
--certs-server-ca-cert
The DoD chain that signed it.
--certs-update-server
"Replace the current web certificate with this one."
--certs-update-server-ca
"Also update the CA that clients are told to trust," so registered clients trust the new certificate.
📣 Clients need the new CA too
Servers already registered to Satellite trusted the old self-made certificate. After this change, update them. The simplest way is to re-run the registration command from Satellite (Hosts → Register Host) or reinstall the katello-ca-consumer package from https://FQDN/pub/. Do this in a planned maintenance window, because clients can't check in until they trust the new chain.
✔ Checkpoint 6
ss shows 9091 listening and nothing on 9090.
Cockpit loads at https://FQDN:9091, you can log in, and Cockpit's Terminal shows the right name for hostname -f.
Step 7
Install Satellite's Software From Its DVD
Satellite comes on its own DVD. You "insert" it as a second DVD drive, connect it, and run Red Hat's script, which installs Satellite and pulls any extra pieces it needs from the RHEL shelf you built in Step 5.
vSphere → Edit Settings → Add New Device → CD/DVD Drive → Datastore ISO File → Satellite-6.19.1-rhel-9-x86_64.dvd.iso. Check Connected and Connect At Power On → OK.
bashlsblk -o NAME,SIZE,TYPE,LABEL,MOUNTPOINTS
lsblk
List disks and drives.
-o NAME,SIZE,TYPE,LABEL,MOUNTPOINTS
Only these columns. TYPE rom means a DVD drive. LABEL tells the two discs apart.
You should see: two rom drives. The one not connected to /media/rhel, with a Satellite label, is the new one, usually sr1.
bashmkdir -p /media/sat6
mount -o ro /dev/sr1 /media/sat6
findmnt -t iso9660
ls /media/sat6
mkdir -p /media/sat6
Make a folder for the Satellite DVD.
mount
Connect a disk to a folder.
-o ro
Options: read-only. No fstab line this time, because you only need this DVD during install.
/dev/sr1 /media/sat6
"Connect drive sr1 to this folder." Swap in sr0 if lsblk showed that.
findmnt -t iso9660
List every connected DVD (-t filters by type).
ls
List what's on the DVD.
You should see: both /media/rhel and /media/sat6, and an install_packages file on the Satellite DVD.
bashdnf upgrade -y
cd /media/sat6
./install_packages
dnf upgrade -y
Bring every installed package up to the newest version on your shelves. -y means auto-yes. It usually finds nothing, since you installed from the same DVD.
cd /media/sat6
"Change directory": go into the Satellite DVD folder. The script expects to run from there.
./install_packages
./ means "the file right here in this folder." Linux doesn't run programs from the current folder unless you say so, for safety. The script creates a temporary shelf from the DVD and installs Satellite.
You should see: it ends with Install is complete. Please run satellite-installer --scenario satellite.
🔧 If this happens
"No match" or a missing dependency → check both DVDs are connected (findmnt -t iso9660) and both RHEL shelves are listed (dnf repolist).
One specific package is truly missing → bring that exact RPM over through the approved transfer process, then run the script again.
✔ Checkpoint 7
The script finished with "Install is complete."
Step 8
Set Up Satellite and Open It to Client Servers
The software is installed but not set up yet. Now you run satellite-installer, which creates the database, the web interface, the ID cards, and all the services. Before that, do one last check that everything it depends on is right.
"Build a full Satellite server" (with its own local database), not a smaller Capsule.
--foreman-initial-organization
The first organization, the top-level container for everything. Foreman is the open-source project Satellite's web interface is built on, which is why the options start with "foreman."
--foreman-initial-location
The first location (usually a site or building).
--foreman-initial-admin-username
The first admin login name.
\
The command continues on the next line.
💡 Why isn't there a password option?
Anything you type in a command is saved in your command history (~/.bash_history), where anyone with root can read it later, and STIG auditors check that file. If you leave the password out, the installer makes up a strong random one and shows it at the end. Save it in your approved password vault, then change it after your first login.
You should see: after 20 to 40 minutes, a success message with the Satellite web address, the admin name, and the generated password.
🔧 If it fails
Read the log. The last error lines name the part that broke: grep -n ERROR /var/log/foreman-installer/satellite.log | tail -20. Here grep -n finds lines with ERROR and shows line numbers, and tail -20 shows only the last 20 of them.
After fixing the cause → run the exact same command again. The installer is safe to re-run.
bash — check every Satellite service is runningsatellite-maintain service status --brief
satellite-maintain
Satellite's maintenance tool.
service status
Check every Satellite service (web server, database, content service, and others).
--brief
One line per service instead of pages of detail.
You should see: every line ending in [OK].
8c. Put away the install DVD
bashcd /
umount /media/sat6
findmnt -t iso9660
cd /
Leave the DVD folder first. You can't disconnect a disk while you're "standing in" it; you'd get "target is busy."
umount
"Un-mount": disconnect it. Note the spelling: umount, not "unmount."
findmnt -t iso9660
Confirm only /media/rhel is left.
Then remove the Satellite DVD drive in vSphere.
🔀 Your choice: keep or remove the RHEL DVD
Keep it (default): future dnf install commands keep working from the RHEL shelf.
Remove it: then switch the shelves off, or every dnf command errors: sed -i 's/^enabled=1/enabled=0/' /etc/yum.repos.d/rhel-local.repo. Here sed -i edits the file in place, s/old/new/ swaps text, and ^ means "at line start." Also delete or comment out (#) the /media/rhel line in fstab.
8d. Log in to Satellite
In Edge go to https://. Always use the full name, never the IP.
A certificate warning is normal here. Satellite made its own CA during install. Continue only if the certificate shows your server's name.
Log in, check the organization and location, then change the admin password.
8e. Let approved client servers reach Satellite
Right now the firewall only lets in SSH and Cockpit. Your client servers need to reach Satellite's web ports. You'll open them only for the approved client network, not for everyone.
bashfirewall-cmd --get-active-zones
--get-active-zones
Firewall rules are grouped into zones. This shows which zone each network card uses.
You should see: a zone name with NIC1 listed under it. Type that zone in the firewalld zone field.
A "rich rule" is a rule with conditions. The single quotes keep the whole rule together as one piece.
rule family="ipv4"
This rule is about IPv4 traffic.
source address="CIDR"
"Only from this network." A CIDR like 10.10.220.0/24 means a whole range of addresses.
service name="https"
Port 443, the secure web port clients use to register and download.
service name="http"
Port 80, also used for downloads and one registration step.
accept
Let matching traffic in.
--reload
Apply the saved rules now.
--list-all
Show everything configured in that zone.
You should see: two rich rules for your client network. Also make sure the output does not show target: ACCEPT or plain http/https under services. Either of those would let everyone in and undo your restriction.
🔒 Always true for ports on this server
Client servers → Satellite: 443 and 80 only, and only from approved networks.
Admins only: 22 (SSH) and 9091 (Cockpit).
Every other port stays closed until the matching feature is approved and turned on.
bash — from a client server inside the approved networkgetent hosts
curl -I --connect-timeout 10 http:///
curl -I --connect-timeout 10 https:///
getent hosts
Can the client find Satellite's address?
curl -I
Fetch just the page header.
--connect-timeout 10
Give up after 10 seconds instead of waiting forever.
You should see: the right IP, then a header for both. A timeout means a firewall or network path problem. A certificate error means the client doesn't trust Satellite's CA yet (normal until the client is registered).
✔ Checkpoint 8
All services [OK], and the dashboard loads by full name.
A client in the approved network gets a response on 80 and 443.
Step 9
Lock Satellite Offline and Load Your Own Packages
Two last jobs. First, tell Satellite it will never talk to Red Hat online, so it stops trying. Second, load a set of approved packages (RPMs) into Satellite so client servers can install them.
9a. Switch Satellite to offline mode
Content → Subscriptions → Manage Manifest → CDN Configuration tab → choose Export Sync → Update. (This means "content arrives by exported files, not by download.")
Administer → Settings → Content tab → Subscription Connection Enabled = No. Reload the page and confirm it saved.
9b. Make a drop-off folder on the big disk
💡 Why not your home folder?
/home is small on purpose (10 GiB) and has noexec. A real batch of packages can fill it quickly. /var lives on the 2 TB data disk, so the drop-off folder goes there.
"Change owner": make your admin user the owner, so it can upload files there.
chmod 750
"Change mode" (permissions). Each digit is one audience: 7 = owner can read, write, and open; 5 = the group can read and open; 0 = everyone else gets nothing.
ls -ld
List with details (-l), showing the folder itself (-d) rather than what's inside it.
You should see:drwxr-x--- … … /var/local/rpm-staging. The letters are the 750 permissions written out: d = directory, rwx = owner, r-x = group, --- = everyone else.
9c. Copy the packages over with WinSCP
WinSCP is a Windows program for copying files to Linux over SSH (the encrypted remote-login protocol).
WinSCP → New Site → protocol SFTP, host , port 22, user → Login.
WinSCP shows the server's fingerprint, a short code unique to this server. Accept it only if it matches the one from the server itself: run ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub. Here -l means "show the fingerprint" and -f means "of this file." This protects you from connecting to an impostor.
Right panel: go to /var/local/rpm-staging. Left panel: the approved package share. Upload, and wait until the queue is empty.
Only names ending in .rpm. The quotes stop the shell from expanding * too early.
| sort
Put the list in alphabetical order, so it's easy to compare with the source.
rpm -K
Check each package's tamper seal (digest and signature). OK is good. NOT OK means damaged in transit or unsigned. Unsigned may be fine for in-house packages if approved.
9d. Create a product and repository in Satellite
In Satellite, a product is a labeled box, and a repository is a shelf inside it that holds packages.
Repositories tab → New Repository → Name Approved RPMs, Type yum, leave Upstream URL empty (there's no internet source), HTTP Proxy Policy No HTTP Proxy → Save.
bashhammer repository list --organization "" --product "Offline Custom Packages"
hammer
Satellite's command-line tool, which does what the web page does, from the terminal. The installer already saved root's login for it.
repository list
"List repositories."
--organization "…"
Only in this organization. Quotes keep names with spaces together.
--product "…"
Only in this product.
You should see: a table row for Approved RPMs. Type its ID number into the Custom repo ID field.
"Put packages onto this shelf." Satellite builds the shelf's index for you.
--id
Which repository (the number from the last step).
--path
Where the packages are. A folder uploads everything in it.
repository info
Show the repository's details.
grep -A3 -i 'content counts'
Find the "Content Counts" section. -i ignores upper/lower case; -A3 also shows the 3 lines After it.
You should see: a Packages: number equal to how many RPMs you uploaded. In the web page, Products → Approved RPMs → Packages lists them by name.
✔ Checkpoint 9
Export Sync selected and Subscription Connection = No.
Package count matches what you uploaded, and there's no upstream URL.
Bonus
Secret Mission and Cleanup
Secret mission: move a file using only your browser
In Cockpit (:9091) open File browser → go to /var/tmp → upload an approved, non-sensitive test file.
In Cockpit's Terminal run ls -l /var/tmp/<file> (list with details) and stat -c '%n %s bytes' /var/tmp/<file>. stat shows file facts; -c sets a custom format, where %n is the name and %s the size.
Take a screenshot, then delete the file.
Cleanup
If you want to…
Do this
Keep it running
Keep Export Sync on. Empty /var/local/rpm-staging per your retention policy. Check df -h /var/lib/pulp now and then, because the library grows.
Pause it
On the console run satellite-maintain service stop (stops every Satellite service cleanly), then shut the VM down from vSphere.
Delete it
Delete the VM from disk in vSphere, remove its A and PTR records from DNS, and remove any lab-only CA from Windows certificate stores.
Automate the boring, repeatable work
Ansible 101 & YAML Basics
Write your first playbooks from scratch: patch servers, install an agent, and handle everyday production jobs. Every line, indent and flag is explained, and every rule is labeled ALWAYS, BEST ORDER or CHOICE.
Lesson A0
What Ansible Is, in Plain English
Imagine you have 50 servers and they all need the same update. Logging in to each one and typing the same commands 50 times is slow, and you'll make a mistake on number 37. Ansible lets you write the instructions once, in a file, and it logs in to every server and follows those instructions for you, the same way, every time.
💡 The kitchen picture
Think of a head chef running a big kitchen:
The control node is the head chef: the one computer where Ansible is installed and where you type commands.
The managed nodes are the cooks: the servers that receive instructions. They don't need Ansible installed. They only need SSH (remote login) and Python, which RHEL already has.
The inventory is the staff list: every cook's name, grouped by station.
A playbook is the recipe book: one file of instructions.
A play is one recipe in the book: "these cooks do these steps."
A task is one step in a recipe: "chop the onions."
A module is the tool used for that step: the knife. Ansible ships with hundreds of tools, one for each kind of job (install packages, copy files, restart services).
Words to know
Word
What it means
Control node
The computer where Ansible is installed and where you run it. Only one needed.
Managed node / host
A server Ansible works on. "Host" and "server" mean the same thing here.
Inventory
The file listing your servers' names, sorted into groups.
Group
A named bunch of servers, like webservers. Lets you say "do this to all web servers" in one word.
Playbook
A YAML file (ends in .yml) holding one or more plays.
Play
A section of a playbook: which servers, and what to do to them.
Task
One single action, like "install this package."
Module
The built-in tool a task uses, like ansible.builtin.dnf for installing packages.
YAML
The simple text format playbooks are written in. It uses indentation (spaces) to show what belongs to what.
Become
Ansible's word for "use sudo" (act as root).
Variable
A named value you can reuse, like agent_package: acme-agent. Change it in one place and it changes everywhere.
Handler
A task that only runs when another task "calls" it, like "restart the service, but only if the config file changed."
Idempotent
A big word for a simple idea: running it twice gives the same result as running it once. See below.
💡 Idempotent: "make sure" instead of "do it again"
A light switch that says "flip" is not idempotent: press it twice and the light goes off again. A switch that says "make sure the light is ON" is idempotent: press it 10 times and the light is still just on.
Ansible modules work like the second switch. state: present means "make sure it's installed." If it's already installed, Ansible does nothing and says ok. If it wasn't, Ansible installs it and says changed.
Does this happen every time? Yes, for Ansible's built-in modules that describe a state (dnf, copy, user, systemd_service, lineinfile). No, for command and shell. Those run their command every single time, because Ansible can't know what your command does. That's why this guide uses real modules whenever one exists.
How to read the labels in this tab
You asked for definite answers, so every rule here carries one of these labels:
Label
Meaning
ALWAYS
Required. If you break it, Ansible stops with an error or does the wrong thing. No exceptions.
BEST ORDER
Ansible would still work another way, but professionals always do it this one way, so every playbook reads the same. Do it this way every time.
CHOICE
More than one right answer. The one this guide uses is marked. Pick it unless your team has its own standard.
Lesson A1
Set Up the Control Node
The control node is where you'll write and run playbooks. It can be any RHEL 9 server you log in to, often an admin or "jump" server. You install Ansible here only. The servers you manage get nothing new.
A1a. Install Ansible and tmux (offline)
Both packages are on the RHEL 9 DVD shelves you set up in Step 5 of the Build Guide (ansible-core is in AppStream; tmux is in BaseOS). On a Satellite-registered server, they come from Satellite instead.
bash — on the control node, as rootdnf install -y ansible-core tmux
ansible --version
dnf install -y
Install the packages. -y answers "yes" for you.
ansible-core
Ansible itself, plus the ansible.builtin tools (modules).
tmux
A "session keeper" (explained in A1d). The RHEL 9 STIG already requires it, so it may be installed.
ansible --version
Prints the version and which config file Ansible is using (the config file = line). You'll use that line in A1c.
🔒 Air-gap rule ALWAYS
On a disconnected network, only use modules whose names start with ansible.builtin.. They come inside ansible-core. Anything else (like ansible.posix.firewalld or community.general.*) is a separate download called a collection, and it won't exist here unless someone brings it in through the approved transfer process. Every example in this tab uses ansible.builtin only.
A1b. Give Ansible a key to log in with (SSH keys)
Ansible logs in to each server over SSH. Instead of typing a password 50 times, you create a key pair: a private key (stays on the control node, like your house key) and a public key (copied to each server, like the lock that key fits).
bash — as your normal user (not root)ssh-keygen -t ecdsa -b 521 -C "ansible control node"
ssh-copy-id -i ~/.ssh/id_ecdsa.pub @web01.lab.example
ssh-keygen
"SSH key generate": makes a new key pair.
-t ecdsa
The key type. ALWAYS use ecdsa or rsa on FIPS servers. The popular ed25519 type is not FIPS-approved on RHEL 9 and will be refused.
-b 521
Key size in bits. 521 is the strongest ECDSA size. (For RSA you'd use -t rsa -b 4096.)
-C "…"
A comment stored with the key, so you can tell later where it came from.
passphrase prompt
It asks for a passphrase. Set one. STIG requires private keys to be protected. See the ssh-agent box below so you only type it once.
ssh-copy-id
Copies your public key onto a server so it will accept your key. Run it once per server.
-i FILE
Which public key (identity) to copy. It always ends in .pub. Never copy the file without .pub; that's your private key.
user@server
Who to log in as, @, which server.
🔀 Type the passphrase only once per session CHOICE (use this)
basheval "$(ssh-agent)"
ssh-add ~/.ssh/id_ecdsa
ssh-agent
A small helper that holds your unlocked key in memory until you log out.
eval "$( … )"
$( ) runs the command inside and captures what it prints. ssh-agent prints two settings lines; eval runs them so your shell knows where the helper is. The quotes keep the output together.
ssh-add FILE
Unlock the key (you type the passphrase here, once) and hand it to the helper.
A1c. Make a project folder (the layout is fixed)
Every Ansible project gets its own folder. Inside it, the layout below is the standard. BEST ORDER
folder layout~/ansible/
├── ansible.cfg ← settings for this project (read FIRST)
├── inventory.ini ← your server names
└── playbooks/
├── ping.yml
├── patch.yml
└── agent.yml
bashmkdir -p ~/ansible/playbooks
cd ~/ansible
chmod 755 ~/ansible
mkdir -p
Make the folder and the playbooks folder inside it in one go (-p = parents).
~
Shortcut for your home folder.
cd ~/ansible
Move into the project. ALWAYS run Ansible from inside this folder, or it won't find your ansible.cfg.
chmod 755
Owner can do everything; others can only look. This matters: Ansible ignores an ansible.cfg sitting in a folder that anyone can write to, as a safety rule.
The main settings section. Square brackets start a section.
inventory = ./inventory.ini
"My server list is this file." Now you never have to type -i inventory.ini.
remote_user
Which account Ansible logs in as on every server.
host_key_checking = True
Check each server's identity before logging in. Keep it True. Many internet tutorials turn this off; that removes protection against impostor servers and is a STIG finding.
retry_files_enabled = False
Don't leave .retry clutter files behind after a failure.
interpreter_python = auto_silent
Find Python on each server automatically, without printing a warning every time.
[privilege_escalation]
Settings for becoming root.
become_method = sudo
Use sudo to become root.
🔒 Which ansible.cfg wins? ALWAYS this order
Ansible looks in these places, in this order, and uses the first one it finds. It never mixes them.
A file named in the ANSIBLE_CONFIG environment variable (if set)
ansible.cfg in the folder you're standing in
~/.ansible.cfg in your home folder
/etc/ansible/ansible.cfg
That's why you always cd into the project first. Check which one is in use with ansible --version and read the config file = line.
A1d. tmux: keep your work alive if the connection drops
Patching 50 servers can take an hour. If your SSH session to the control node drops halfway (VPN hiccup, jump box reboot, laptop sleeps), the playbook dies with it, and some servers are left half-updated. tmux fixes that. It runs your work inside a session that lives on the server, not in your window. If you get disconnected, the work keeps going, and you reconnect and pick up where you were.
🔒 The tmux rules
ALWAYSStart tmux BEFORE you start the playbook. tmux can only protect things started inside it. You can't move an already-running playbook into tmux afterward.
ALWAYS Start it on the control node (where Ansible runs), not on your own PC.
BEST ORDER Use tmux for every run that changes servers (patching, installs). A quick ping test doesn't need it.
Does it need to be "enabled" like a service? No. tmux isn't a service. You start a session when you need one, and it ends when you type exit inside it.
bash — the full tmux routinetmux new -s patch
# ...now inside tmux: cd ~/ansible and run your playbook...
# to step away without stopping it: press Ctrl+b, let go, then press d
tmux ls
tmux attach -t patch
exit
tmux new -s patch
Start a new session and name it patch (-s = session name). A green bar appears at the bottom; that means you're inside.
Ctrl+b, then d
"Detach": step out and leave it running. Ctrl+b is tmux's "listen to me" key; press it, let go, then press d. Every tmux shortcut starts with Ctrl+b.
tmux ls
List running sessions.
tmux attach -t patch
Reconnect to the session named patch (-t = target). Use this after a disconnect too.
exit
Typed inside tmux, this ends the session for good. Only do it once the playbook has finished.
📣 STIG lock screen inside tmux
The STIG sets tmux to lock after 15 minutes of no typing. The screen goes blank and asks for your password. Your playbook keeps running underneath. Type your password to see it again.
✔ Checkpoint A1
ansible --version works and shows config file = /home/…/ansible/ansible.cfg.
Your public key is copied to each server, and ssh web01.lab.example logs in without asking for the account password.
You can start, detach from, and re-attach to a tmux session.
Lesson A2
The Inventory: Where Server Names Go
The inventory is your class roster. It lists every server by name and sorts them into groups. Playbooks never contain server names directly; they name a group, and the inventory says who's in it.
bash — create inventory.inicat > ~/ansible/inventory.ini <<'EOF'
# Satellite is in its own group so patch playbooks never touch it
[satellite]
[rhel_servers]
web01.lab.example
web02.lab.example
db01.lab.example
[webservers]
web01.lab.example
web02.lab.example
EOF
# …
A comment. Ansible ignores it. Use it to explain why.
[satellite]
Square brackets = a group name. Every server listed under it (until the next bracket) belongs to that group.
one name per line
Each line is one server. Use the full name (FQDN) that DNS knows.
same server, two groups
web01 is in both rhel_servers and webservers. That's allowed and normal.
🔒 Inventory rules
ALWAYS Group names use only letters, numbers, and underscores. No dashes, no spaces (rhel_servers, not rhel-servers).
ALWAYS Two groups exist without you writing them: all (every server in the file) and ungrouped (servers not under any bracket).
BEST ORDER Name groups by what the servers do (webservers, dbservers), not by where they are.
BEST ORDER Keep the Satellite server in its own group, and never include it in a general patch group. Satellite must be updated with satellite-maintain, not a plain dnf update.
CHOICE Inventories can also be written in YAML format. This guide uses INI (the bracket style) because it's easier to read. Both work the same.
A2a. Check the inventory, then say hello to every server
bash — from ~/ansibleansible-inventory --graph
ansible all -m ansible.builtin.ping
ansible-inventory --graph
Draws your groups as a tree. It's a quick way to catch typos before anything runs.
ansible
Runs one module right now, without a playbook. Good for quick checks.
all
Which servers: the built-in group of everyone. You could put a group name here instead, like webservers.
-m
Which module to run (long form: --module-name).
ansible.builtin.ping
Not a network ping. It logs in over SSH, checks Python works, and replies "pong." If this works, Ansible can manage that server.
You should see: a green block per server ending in "ping": "pong". Red UNREACHABLE means SSH couldn't log in. See A9's error table.
Lesson A3
YAML: The Grammar Rules
Playbooks are written in YAML. YAML has very few rules, but the ones it has are strict. Almost every first-time Ansible error is a YAML spacing mistake, not an Ansible mistake.
💡 Indentation is the whole language
In a school outline, you indent sub-points under a main point to show they belong to it. YAML works exactly the same way: how far a line is pushed in decides what it belongs to. There are no brackets or braces to help. The spaces are the structure.
🔒 The YAML rules
ALWAYSSpaces only. Never the Tab key. YAML refuses tabs, and they're invisible, which makes the error confusing.
ALWAYSThings at the same level line up exactly, to the same column. One space off and it means something different, or breaks.
ALWAYSkey: value has a space after the colon.name:web is wrong. name: web is right.
ALWAYS- (dash, space) starts an item in a list. Plays are a list, and tasks are a list, so each one starts with a dash.
ALWAYSPut quotes around a value that starts with a special character:*{[@%&!. Example: name: "*". Without quotes, YAML thinks * is one of its own symbols and errors.
ALWAYSQuote numbers that must keep a leading zero, like file permissions: mode: "0644". Without quotes YAML treats it as a number and the leading zero is lost or misread.
BEST ORDERIndent 2 spaces per level. YAML allows any amount, as long as you're consistent. The Ansible world uses 2. Always use 2.
BEST ORDERStart every playbook with --- on the first line. It means "a YAML document starts here." Ansible works without it, but it's the standard.
BEST ORDER For yes/no settings, write true and false (lowercase), never yes/no. All four work today; true/false is the modern standard.
# starts a comment. Everything after it on that line is ignored.
A3a. See the spaces
Below is a tiny playbook with every space shown as a dot (·). Each level is pushed in exactly 2 more dots than the level above it.
yaml — spaces shown as dots---
-·name:·Say·hello ← level 0: the play starts with "- "
··hosts:·all ← level 1 (2 spaces): lines up with "name"
··tasks: ← level 1: tasks belong to the play
····-·name:·Print·a·message ← level 2 (4 spaces): a task, starts with "- "
······ansible.builtin.debug: ← level 3 (6 spaces): the module
········msg:·Hello·from·Ansible ← level 4 (8 spaces): the module's setting
Here's the same playbook as a clean file you can copy. Save it as ~/ansible/playbooks/ping.yml:
yaml — ~/ansible/playbooks/ping.yml---
- name: Say hello
hosts: all
tasks:
- name: Print a message
ansible.builtin.debug:
msg: Hello from Ansible
bash — from ~/ansibleansible-playbook playbooks/ping.yml
ansible-playbook
Runs a playbook file (where ansible runs a single module).
playbooks/ping.yml
The path to the playbook, starting from the folder you're in. No -K is needed here, because this play has no become.
You should see:"msg": "Hello from Ansible" for every server, then a PLAY RECAP (explained in A9).
💡 Why does "hosts" line up with "name" and not with the dash?
The dash is a bullet point, not part of the text. name, hosts, and tasks are all facts about the same play, so they line up with each other, 2 spaces in, right where the text after - begins. This is the rule that trips up most beginners. ALWAYS true.
A3b. What a spacing mistake looks like
yaml — WRONG (the module is pushed in 1 extra space) - name: Print a message
ansible.builtin.debug:
msg: Hello
What you'd see:ERROR! We were unable to read either as JSON nor YAML or mapping values are not allowed in this context, with a line and column number. Always go to that line and count the spaces on it and the line above. The fix is almost always "line it up."
bash — check any playbook's grammar without running itansible-playbook playbooks/ping.yml --syntax-check
--syntax-check
Reads the file and reports grammar mistakes, but changes nothing and contacts no servers. Run it every time you edit a playbook.
Lesson A4
Playbook Anatomy: What Goes Where, and in What Order
Every playbook has the same skeleton. Once you know the skeleton, you can read any playbook, and write your own by filling in the blanks.
A4a. The hierarchy (what's inside what)
Playbookthe file, e.g. patch.yml. Starts with ---
Playstarts with - name:. Says who (hosts) and how (become)
tasks:the list of steps, in order
Taskstarts with - name:. One step
Modulee.g. ansible.builtin.dnf:. The tool
Module settingse.g. name:, state:. How to use the tool
Task keywordswhen:register:notify:. Line up with the module, not under it
handlers:tasks that only run when "notified"
A4b. The play: which lines, in which order
Here's the honest, definite answer on order. Ansible reads the whole play before doing anything, so it doesn't care what order these lines are in. People do, though. A fixed order means you always know where to look. This is the standard order. Use it every time. BEST ORDER
#
Line
Required?
What it answers
1
- name:
Optional, but always write it
"What is this play for?" It's printed on screen when the play runs.
2
hosts:
ALWAYS required
"Who?" A group name from the inventory (or all).
3
become:
Needed for anything that changes the system
"Do I need root?" true = use sudo.
4
serial:
Optional
"How many at a time?" For rolling changes.
5
vars:
Optional
"What values will I reuse?"
6
tasks:
ALWAYS required (or roles:)
"What do I do?" The steps.
7
handlers:
Optional
"What runs only if something changed?" Always last.
💡 Why this order makes sense
It reads like a sentence: "This play (name) is for these servers (hosts), as root (become), one at a time (serial), using these values (vars), do these steps (tasks), and if needed, clean up with these (handlers)." Who and how come before what.
A4c. The task: which lines, in which order
🔒 Task rules
ALWAYS Each task starts with - (it's an item in the tasks list).
ALWAYSOne module per task. Never two. If you need two actions, write two tasks.
ALWAYS The module's own settings go under the module, pushed in 2 more spaces.
ALWAYS Task keywords (when, register, notify, changed_when, failed_when) line up with the module name. They're about the task, not settings for the tool. Push them under the module and Ansible thinks they're module settings, and errors.
BEST ORDER- name: first → module and its settings → then task keywords.
BEST ORDER Always use the full module name (ansible.builtin.dnf), never the short one (dnf). The short one usually works, but the full one can never be confused with a same-named module from another collection. Like writing a full street address instead of "the blue house."
BEST ORDER Start every task name with a capital letter and a verb: "Install…", "Copy…", "Restart…". It's printed on screen and becomes your run log.
yaml — one task, fully labeled - name: Install the web server ← 1. name (what this step does)
ansible.builtin.dnf: ← 2. module (the tool)
name: httpd ← 2a. module setting: which package
state: present ← 2b. module setting: make sure it's installed
when: ansible_facts['os_family'] == "RedHat" ← 3. task keyword: lines up with the module
A4d. Does it run top to bottom every time?
🔒 How a playbook runs ALWAYS
Plays run in order, top to bottom.
Tasks run in order, top to bottom. Task 1 finishes on every server before task 2 starts on any of them. (That's the default; serial changes how many servers are in each batch.)
If a task fails on a server, that server stops. It's dropped from the rest of the play. The other servers keep going.
Handlers run at the end of the play, once, and only if a task notified them and that task reported changed. If nothing changed, handlers do not run.
Before task 1, Ansible runs a hidden step called Gathering Facts. It collects information about each server (OS version, IPs, memory) into ansible_facts so tasks can use it.
Lesson A5
If This, Then That: Decisions, Fallbacks, and Messages
Real jobs need decisions: "if a reboot is needed, reboot," "if you can't find the file, do this instead," "tell me what you found." This lesson shows how to write those in Ansible, and in plain shell, since you'll see both.
A5a. Remember an answer: register
register saves a task's result under a name, like writing the answer on a sticky note so a later task can read it.
yaml - name: Look for a server line in the chrony config
ansible.builtin.command:
cmd: grep -c '^server' /etc/chrony.conf
register: ntp_check
changed_when: false
failed_when: false
ansible.builtin.command
Runs a command on the server. Use it only when no real module does the job.
cmd:
The command to run. grep -c counts matching lines (-c = count).
register: ntp_check
Save the result as ntp_check. Inside it: ntp_check.rc (the return code: 0 = success/found, anything else = not), ntp_check.stdout (what it printed), and ntp_check.stdout_lines (the same, split into lines).
changed_when: false
"This task only looks. Never report it as a change." Without it, commandalways says changed, which is misleading.
failed_when: false
"Never treat this as a failure." grep returns 1 when it finds nothing, and Ansible would normally call that a failure. Here "not found" is a valid answer, not an error.
A5b. Only do it if: when
yaml - name: Add the internal time server if none is configured
ansible.builtin.lineinfile:
path: /etc/chrony.conf
line: "server iburst"
when: ntp_check.stdout == "0"
when:
"Only run this task if the test is true." If false, the task is skipped (shown in cyan).
ntp_check.stdout == "0"
"If grep counted zero server lines." == means "is equal to." (One = is not a test.)
🔒 when rules ALWAYS
Never put {{ }} inside when. Write when: ntp_check.rc == 0, not when: "{{ ntp_check.rc }} == 0". when already understands variable names.
when is a task keyword, so it lines up with the module name (A4c).
Useful tests: == equals, != not equal, > greater than, and, or, not, is defined (the variable exists).
Several lines under when: as a list means all must be true (AND).
A5c. Print a message (Ansible's "echo"): debug
yaml - name: Report what was found
ansible.builtin.debug:
msg: "{{ inventory_hostname }} has {{ ntp_check.stdout }} time server line(s)"
ansible.builtin.debug
Prints a message on your screen during the run. It changes nothing on the server. It's Ansible's version of echo.
msg:
The message. Use var: ntp_check instead of msg: to dump a whole variable, which is great for seeing what a register captured.
{{ … }}
"Fill in this variable's value here." Called Jinja templating.
inventory_hostname
A built-in variable: the server's name as written in the inventory.
🔒 Curly-brace quoting rule ALWAYS
If a value starts with {{, the whole value must be in quotes: name: "{{ agent_package }}". YAML sees { as its own symbol otherwise, and errors. Wrapping every value that contains {{ }} in quotes is the safe habit. BEST ORDER
A5d. "If this fails, do that instead": block / rescue / always
Like a try / plan B / clean up checklist. Ansible tries the block. If any task in it fails, it jumps to rescue (plan B). always runs no matter what.
yaml - name: Install the agent, with a fallback
block:
- name: Try the agent from Satellite
ansible.builtin.dnf:
name: acme-agent
state: present
rescue:
- name: Tell me it failed on this server
ansible.builtin.debug:
msg: "Agent install FAILED on {{ inventory_hostname }}. Check the Satellite repo."
always:
- name: Record that this server was attempted
ansible.builtin.lineinfile:
path: /var/log/agent-rollout.log
line: "attempted by Ansible"
create: true
mode: "0600"
block:
The group of tasks to try. They're indented under block: as a list.
rescue:
Runs only if something in block failed. When rescue succeeds, the server is not marked failed (it shows as rescued) and keeps going.
always:
Runs every time, whether block succeeded or failed.
create: true
(lineinfile) Make the file if it doesn't exist.
🔒 Order ALWAYS block → rescue → always
block, rescue, and always line up with each other. rescue and always are both optional, but if present, they come after block in that order.
A5e. The same ideas in plain shell (bash)
You'll see these in scripts and in the Build Guide. Same logic, different words.
bash — "if not found, say so"grep -q '^server' /etc/chrony.conf || echo "No time server configured"
grep -q '^server' /etc/chrony.conf && echo "Time server is configured"
grep -q
Search quietly (-q): print nothing, just succeed (found) or fail (not found).
||
"OR ELSE": run the right side only if the left side failed. So: "look for it, or else say it's missing."
&&
"AND THEN": run the right side only if the left side succeeded.
echo "…"
Print the text in quotes on screen.
bash — full if / then / elseif grep -q '^server' /etc/chrony.conf; then
echo "Found a time server"
else
echo "Not found - adding one"
echo "server iburst" >> /etc/chrony.conf
fi
if COMMAND; then
"If this command succeeds, do the lines after then." The ; ends the test so then can share the line.
else
"Otherwise, do these lines instead."
fi
"End of the if." It's if spelled backwards. ALWAYS required; forget it and bash says syntax error: unexpected end of file.
>>
Add to the end of the file (never >, which would erase it).
bash — test if a file existsif [ -f /etc/chrony.conf ]; then echo "File exists"; else echo "File missing"; fi
[ -f FILE ]
A test: "is this a regular file that exists?" (-d = is it a directory.) ALWAYS put a space after [ and before ]. [-f without the space fails.
; between parts
Lets you write the whole if on one line.
🔀 Shell or Ansible for this? CHOICE
For one server, right now: shell is fine. For more than one server, or anything you'll repeat: use Ansible, with real modules like lineinfile that are idempotent. The shell >> example above would add a second copy of the line if you ran it twice; Ansible's lineinfile never would.
Lesson A6
Playbook 1: Patch (Update) Servers Safely
This playbook updates every package, checks whether the update needs a reboot, and reboots only if it does, one server at a time, so you never take everything down at once. On a disconnected network, the updates come from Satellite (for registered servers) or your local repos.
yaml — ~/ansible/playbooks/patch.yml---
- name: Patch RHEL servers one at a time
hosts: rhel_servers
become: true
serial: 1
tasks:
- name: Update every installed package
ansible.builtin.dnf:
name: "*"
state: latest
- name: Check whether a reboot is needed
ansible.builtin.command:
cmd: dnf needs-restarting -r
register: reboot_check
changed_when: false
failed_when: reboot_check.rc > 1
check_mode: false
- name: Tell me the result
ansible.builtin.debug:
msg: "{{ inventory_hostname }}: {{ 'REBOOT needed' if reboot_check.rc == 1 else 'no reboot needed' }}"
- name: Reboot if the update needs it
ansible.builtin.reboot:
reboot_timeout: 900
when: reboot_check.rc == 1
Line by line
---
Start of the YAML document. BEST ORDER
- name: Patch…
The play begins (dash = first item in the list of plays).
hosts: rhel_servers
Who: the group from inventory.ini. The Satellite server is not in this group, on purpose.
become: true
How: use sudo. Installing updates needs root.
serial: 1
One server at a time, fully (update → check → reboot) before the next starts. If one breaks, the rest are untouched. Use serial: "25%" to go a quarter at a time.
(blank line)
Blank lines are ignored. Use one before tasks: to make the file easier to read.
tasks:
What: the steps, in order.
ansible.builtin.dnf:
The package tool.
name: "*"
Every package. The quotes are required; a bare * is a YAML symbol (A3 rule 5).
state: latest
"Make sure it's the newest version available." (present would mean "just make sure it's installed, any version.")
cmd: dnf needs-restarting -r
Asks RHEL "do the updates need a reboot?" It answers with its exit code: 0 = no reboot needed, 1 = reboot needed. (-r = only report on reboot need.)
register: reboot_check
Save the answer.
changed_when: false
It only asks a question, so never report "changed."
failed_when: reboot_check.rc > 1
Exit code 1 is an answer here, not an error. Only 2 or higher is a real failure.
check_mode: false
In a dry run (--check), Ansible normally skipscommand tasks. This line says "always run this one, even in a dry run," which is safe because it only asks. Without it, the next tasks would have no answer to read and would error.
'A' if … else 'B'
Inside {{ }}: "say A if the test is true, otherwise say B."
ansible.builtin.reboot:
Reboots the server, waits for it to come back, and confirms it's reachable before moving on.
reboot_timeout: 900
Wait up to 900 seconds (15 minutes) for it to come back. STIG servers can boot slowly.
when: reboot_check.rc == 1
Only reboot if the check said "reboot needed."
📣 Never point this at the Satellite server
Satellite has its own update tool (satellite-maintain) that stops and starts its services in the right order. A plain dnf update on Satellite can break it. That's why Satellite lives in its own inventory group.
A6a. Run it, the safe way, every time
BEST ORDER Follow these steps in this order. Each one catches a different kind of mistake before it can hurt anything.
bash — on the control nodetmux new -s patch
cd ~/ansible
ansible-playbook playbooks/patch.yml --syntax-check
ansible-playbook playbooks/patch.yml --list-hosts
ansible-playbook playbooks/patch.yml --check --diff -K
ansible-playbook playbooks/patch.yml --limit web01.lab.example -K
ansible-playbook playbooks/patch.yml -K
tmux new -s patch
Step 0: always first. Protects the run if you get disconnected (A1d).
--syntax-check
Step 1: grammar check. Contacts no servers.
--list-hosts
Step 2: prints exactly which servers it would touch. Make sure Satellite isn't there.
--check
Step 3: dry run (short form -C). Shows what would change, and changes nothing.
--diff
With --check: also shows the before/after of any file changes (short form -D).
-K
Ask for your sudo password once, at the start (long form --ask-become-pass). Needed because become: true uses sudo and STIG requires a password for it. Capital K. Lowercase -k asks for the SSH password instead.
--limit NAME
Step 4: test on ONE server first (short form -l). Only that server is touched. You can also give a group name.
(no limit)
Step 5: the real run on everyone in hosts.
🔒 "Will it do the same thing every time?"
Yes. Run it again right after, and every server reports ok (nothing new to update) and no reboot, because dnf and the reboot check are idempotent. That's the proof it worked.
Each new month's run will update whatever is new in Satellite at that time. The behavior is identical; the packages depend on what's been published.
Lesson A7
Playbook 2: Install and Configure an Agent
Security and monitoring teams often need a small program, an agent, on every server. It usually comes as an RPM package, needs a config file saying which server to report to, and needs its service running. This playbook does all three. acme-agent is a made-up name; swap in your real agent's package name, service name, and config.
It assumes the agent's RPM is already in Satellite (like the "Approved RPMs" repo from Build Guide Step 9) and the servers are registered to it.
yaml — ~/ansible/playbooks/agent.yml---
- name: Install and start the monitoring agent
hosts: rhel_servers
become: true
vars:
agent_package: acme-agent
agent_service: acme-agent
agent_config: /etc/acme-agent/agent.conf
agent_server:
agent_port: 8443
tasks:
- name: Install the agent package
ansible.builtin.dnf:
name: "{{ agent_package }}"
state: present
- name: Write the agent config file
ansible.builtin.copy:
dest: "{{ agent_config }}"
content: |
server={{ agent_server }}
port={{ agent_port }}
owner: root
group: root
mode: "0600"
notify: Restart the agent
- name: Start the agent now and at every boot
ansible.builtin.systemd_service:
name: "{{ agent_service }}"
state: started
enabled: true
handlers:
- name: Restart the agent
ansible.builtin.systemd_service:
name: "{{ agent_service }}"
state: restarted
Line by line
vars:
Values reused below. BEST ORDER Anything you might change later (names, paths, ports) goes here, so you edit one line, not five.
agent_package: …
A variable: a name, a colon, a value. Names use lowercase and underscores only. ALWAYS no dashes or spaces in variable names.
name: "{{ agent_package }}"
Fill in the variable. Quotes required because the value starts with {{.
state: present
"Make sure it's installed." Unlike latest, it won't upgrade an agent that's already there.
ansible.builtin.copy:
Puts a file on the server.
dest:
Where the file goes (destination).
content: |
The file's text, written right here. The | means "keep the following lines exactly as written, line breaks included." The lines under it must be indented further than content.
owner / group
Who owns the file.
mode: "0600"
Permissions: only root can read or write it. Quotes required to keep the leading zero (A3 rule 6).
notify: Restart the agent
"If this task changed the file, call the handler with this exact name." ALWAYS must match the handler's name:exactly, including capital letters.
ansible.builtin.systemd_service:
Controls services. (On older Ansible versions this module is called ansible.builtin.systemd. If you get "couldn't resolve module," use that name instead.)
state: started
"Make sure it's running." Does nothing if it already is.
enabled: true
"Make sure it starts at every boot."
handlers:
Only-when-called tasks. BEST ORDER always last in the play.
state: restarted
Stop and start the service so it reads the new config.
🔒 Does the handler run every time? No.
First run: the config file is new → the copy task reports changed → the handler runs once, at the end of the play.
Second run, nothing edited: the file already matches → ok → the handler does not run. The service is not restarted for no reason.
You change agent_port and run again: the file content changes → changed → the handler runs.
Handlers run at the end of the play, not right after the task that notified them. ALWAYS
If several tasks notify the same handler, it still runs only once. ALWAYS
💡 Why is the order install → config → start?
It's the only order that works. ALWAYS You can't write a config file into a folder the package hasn't created yet, and you shouldn't start a service before it knows where to report. Tasks run top to bottom (A4d), so the order in the file is the order of work: install, then configure, then start. Use that order for any piece of software.
Lesson A8
More Useful Production Tasks
Short, ready-to-use recipes. Each one is a task (or two) you can paste into the tasks: list of any play that has become: true.
A8a. Set the login banner (a STIG requirement)
yaml - name: Set the approved login banner
ansible.builtin.copy:
dest: /etc/issue
content: |
Paste your organization's approved banner text here.
owner: root
group: root
mode: "0644"
/etc/issue
The text shown before the console login prompt.
content: |
Paste the exact approved banner. Keep every line indented under content.
mode: "0644"
Everyone can read it, only root can change it.
A8b. Make sure every server uses the internal time server
yaml - name: Point chrony at the internal time server
ansible.builtin.lineinfile:
path: /etc/chrony.conf
regexp: '^server '
line: "server iburst"
notify: Restart chronyd
handlers:
- name: Restart chronyd
ansible.builtin.systemd_service:
name: chronyd
state: restarted
ansible.builtin.lineinfile
Makes sure one specific line is in a file.
regexp: '^server '
"Find the line that starts with server ." (^ = start of line.) If found, it's replaced. If not found, the new line is added at the end.
line:
What the line should say.
handlers:
This goes at the play level (lined up with tasks:), not inside the task list.
A8c. Create an admin account
yaml - name: Make sure the admin account exists
ansible.builtin.user:
name:
groups: wheel
append: true
shell: /bin/bash
state: present
ansible.builtin.user
Creates or changes a user account.
groups: wheel
Add to the wheel group, which is RHEL's "allowed to use sudo" group.
append: true
ALWAYS include this with groups. It means "add to these groups." Without it, Ansible removes the user from every group not listed.
state: present
Make sure the account exists. (absent would delete it.)
A8d. Disk space report from every server
yaml - name: Check disk space
ansible.builtin.command:
cmd: df -h /var /var/log /tmp
register: disk
changed_when: false
- name: Show disk space
ansible.builtin.debug:
var: disk.stdout_lines
df -h
Disk free, human-readable (G/T).
var: disk.stdout_lines
Print what df printed, one line per row, so it's readable.
🔀 Reading-only jobs don't need becomeCHOICE
A report like this one only reads, so it could run without root. If you make it its own playbook, you can leave out become: true and -K. Rule of thumb: only ask for root when the task changes something.
Lesson A9
Run It, Read the Results, Fix the Errors
A9a. The ansible-playbook flags you'll use
Short
Long
What it does
When to use it
(none)
--syntax-check
Grammar check only
Every time you edit
(none)
--list-hosts
Show which servers it would touch
Before every real run
(none)
--list-tasks
Show the task names in order
To review a playbook you didn't write
-C
--check
Dry run: show changes, make none
Before every real run
-D
--diff
Show file before/after
With --check, and on real runs that edit files
-l
--limit
Only these servers or groups
Test on one server first
-K
--ask-become-pass
Ask for the sudo password
Any play with become: true on STIG servers
-i
--inventory
Use a different inventory file
Only if not set in ansible.cfg
-v … -vvvv
--verbose
More detail. Each extra v = more
-vvv when debugging SSH or errors
(none)
--start-at-task "Name"
Skip ahead to a task by its name
Resuming after fixing a failure
(none)
--step
Ask yes/no before every task
Walking through a new playbook slowly
🔒 Flag order CHOICE, so use this one
Flags can go in any order after the playbook name. Ansible doesn't care. Use this one every time so your commands look the same: ansible-playbook PLAYBOOK --check --diff --limit HOST -K (what to run → how to run it → where → password last).
Tasks that finished fine (this count includes changed ones).
changed
Tasks that made a change.
unreachable
Couldn't connect. Must be 0.
failed
Tasks that failed. Must be 0 for a clean run. Here, db01 failed and stopped early.
skipped
Tasks skipped by a when.
rescued
Failures caught and handled by a rescue block.
ignored
Failures ignored on purpose (ignore_errors: true). Avoid that keyword; it hides real problems.
A clean run:unreachable=0 and failed=0 on every line. Run it a second time:changed=0 everywhere proves the playbook is idempotent.
A9d. Common errors and exactly what to do
You see
It means
Do this
mapping values are not allowed / could not find expected ':'
YAML spacing or a missing space after a colon
Go to the line number it gives. Count spaces against the line above (A3).
found character that cannot start any token
A Tab character, or an unquoted *, {, or @
Replace tabs with spaces; quote the value.
UNREACHABLE! … Permission denied (publickey)
Your SSH key isn't on that server, or it's the wrong user
Rerun ssh-copy-id; check remote_user; test with plain ssh.
UNREACHABLE! … Name or service not known
DNS can't find the name in your inventory
Check spelling; getent hosts NAME.
Host key verification failed
First connection to that server, or its identity changed
SSH to it once by hand and verify the fingerprint. Never turn off host_key_checking.
Missing sudo password
become: true with no password given
Add -K.
couldn't resolve module/action
Typo in the module name, or it's not in ansible-core
Check spelling; use only ansible.builtin.* in the air gap.
No package … available
The package isn't in any repo that server can reach
Check it's in Satellite and the server is registered; dnf repolist on that server.
The task includes an option with an undefined variable
A {{ name }} that was never set, or a typo
Check spelling against vars:. Names are case-sensitive.
Unsupported parameters … when
when was indented under the module instead of lining up with it
Move it left to line up with the module name (A4c).
Warning: could not match supplied host pattern
The hosts: group name isn't in the inventory
Check spelling; ansible-inventory --graph.
✔ Checkpoint A9
You ran a playbook with --syntax-check, --check --diff, then --limit on one server, then for real.
The PLAY RECAP showed failed=0 and unreachable=0.
A second run showed changed=0.
When something doesn't go to plan
Troubleshooting & Fixes
Problems met during real builds, with what you see, why it happens, the fix, and how to confirm it worked. Search by error text or command.
🔒 The golden rule: ask before you change
Never guess. Let the server tell you what's wrong first, using commands that only look and never change anything. Always check from the inside out: the service → SELinux → firewall → network → your computer. Stop at the first thing that fails. That's your problem.
journalctl -u <service> -b --no-pager | tail -50: a service's log. -u = which service ("unit"), -b = only since the last boot, --no-pager = print it all at once instead of one screen at a time, tail -50 = only the last 50 lines.
tail -f /var/log/secure: watch login attempts live. -f = "follow," keep showing new lines as they happen. Press Ctrl+C to stop.
ausearch -m AVC -ts recent: what SELinux blocked. -m AVC = the "access denied" message type, -ts recent = from the last 10 minutes.
df -h and vgs: how full each room is (-h = readable sizes) vs. how much unused space is left in each disk pool.
klist -kte /etc/krb5.keytab: (only if joined to Active Directory) the exact Kerberos identities this server holds. -k = read the keytab file, -t = show dates, -e = show encryption types.
The password is right, but login keeps failing
Step 3 onward · logins · faillock · STIG
What you see
"Authentication failed" even with the right password. Resetting the password as root doesn't help.
Why
STIG turns on pam_faillock, a lockout counter. After 3 bad tries it locks the account, and it usually never unlocks by itself. The counter is stored separately from the password, so changing the password doesn't reset it. The login prompt still appears on purpose, so an attacker can't tell the account is locked.
The tool that reads and clears the lockout counter.
--user NAME
Show that user's failed attempts.
--reset
Clear the counter, which unlocks the account.
grep -E '^(…)'
Show the lockout settings: how many tries (deny), the auto-unlock time (unlock_time, where 0 = never), where the counter is stored (dir), and whether root can be locked too (even_deny_root).
Check it
faillock --user <user> shows no entries, and login works.
touch /.autorelabel: touch creates an empty file. This one tells SELinux to re-check every file's label on the next boot, because you changed things while SELinux wasn't watching.
exit, then reboot. The relabel takes a few minutes.
Check it
Root can log in on the console after the reboot.
The vSphere console types the wrong characters
Step 3 · console · keyboard
What you see
A password with symbols fails in the console but works over SSH or Cockpit. Symbols come out wrong when you type visible commands.
Why
The console sends which key you pressed (a "scan code"), not which character you meant. If your computer's keyboard layout doesn't match what the VM expects, symbols like @ # $ | \ turn into different characters. This happens from Windows and Mac alike. Passwords are hidden, so you never see it.
Fix
Test the symbols (not your password) at a normal prompt:
bashecho '@#$\|-.'
echo
"Print back exactly what I typed."
' '
Single quotes stop the shell from treating the symbols as special, so you see exactly what the console sent.
Use only symbols that came out right. STIG still requires 15+ characters with upper, lower, number, and symbol, so "lowercase and numbers only" won't be accepted. A long passphrase with - or . is a safe choice. If the account is already locked, reset the password (passwd <user>) and clear faillock (above).
Don't
Don't type your real password on a visible command line to "see" it. It stays on the screen and in ~/.bash_history, the saved command history that auditors check.
"Permission denied" when writing a file, even with sudo
shell · sudo · the redirection trap
What you see
sudo echo "x" > /etc/file or sudo cat > /etc/file says "Permission denied."
Why
Think of it as two workers. sudo echo runs as the admin, but the > (put it into a file) is done by your normal shell, before sudo even starts. Your normal user isn't allowed to write in /etc, so the file part fails.
Fix
Option 1 (this guide's rule): become root first with sudo -i, then there's only one worker and it's the admin.
Option 2: hand the text to a command that runs as admin and writes the file itself:
bashprintf '[Socket]\nListenStream=\nListenStream=9091\n' | sudo tee /etc/systemd/system/cockpit.socket.d/listen.conf
printf '…'
Print text. \n means "new line," so this makes three lines.
|
Hand that text to the next command.
sudo tee FILE
tee writes what it receives into a file (and also shows it on screen). Because tee itself runs under sudo, it has permission. Add -a to add to the file instead of replacing it.
The port moved and the service is "listening," but the browser won't load
Step 6 · ports · SELinux · firewall · the ladder
What you see
systemctl show cockpit.socket -p Listen says 9091, but Edge shows a blank page or "can't reach this page."
Fix
Climb this ladder in order and stop at the first rung that fails:
#
Check
Where
If it fails
1
ss -tlnp | grep 9091 (TCP, listening, numbers, program)
Server
The service isn't behind the door. Check the drop-in file (is the blank ListenStream= line there?), then systemctl daemon-reload and restart the socket.
2
semanage port -l | grep -w 9091
Server
SELinux hasn't allowed it. Add websm_port_t (Step 6b). See what was blocked with ausearch -m AVC -ts recent.
3
firewall-cmd --info-service=cockpit and firewall-cmd --list-all
Server
The fence is closed. Add 9091, then --reload. The usual slip: --permanent without a reload, or a reload without --permanent.
4
curl -skI https://localhost:9091
Server
Cockpit itself is failing. Read journalctl -u cockpit -b and check the certificate files.
5
Test-NetConnection <FQDN> -Port 9091
Jump server
If the name doesn't resolve, it's DNS: check the records and run Clear-DnsClientCache. If the name resolves but TCP fails, a network firewall or route between them is blocking.
6
Edge at https://FQDN:9091
Jump server
Hard refresh (Ctrl+F5) or open an InPrivate window. Use the exact name on the certificate. If the IP works but the name doesn't, it's DNS or the certificate name.
Also check
That you typed https://. Cockpit won't answer plain http://.
Choice
Your original notes opened the port with firewall-cmd --permanent --add-port=9091/tcp. That works too: --add-port opens a raw numbered door. This guide instead changes the cockpit service definition, so the firewall list still says "cockpit" and a reviewer can see why the door is open. Use one method, not both. With both, removing one still leaves the door open.
Everything broke after the jump server rebooted or DNS changed
jump server · DNS · sessions
What you see
Sessions freeze, pages hang, or a name goes to an old address.
Why
Your open sessions died with the reboot. Also, your computer remembers DNS answers for a while (a cache), so it may still use an old address. It's the DNS answers that are cached, not the jump box's network routes.
Fix
Close every old tab and console window. Then clear the memory and look the name up fresh.
Windows — PowerShell as AdministratorClear-DnsClientCache
Resolve-DnsName
Clear-DnsClientCache
Forget every remembered DNS answer.
Resolve-DnsName
Ask the DNS server directly, right now, and show the answer (record type and IP).
Mac — Terminalsudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder
dscacheutil -flushcache
Clears the Mac's lookup memory, including DNS.
;
Then run the next command no matter what.
killall -HUP mDNSResponder
killall sends a signal to every program with that name. -HUP ("hang up") tells the Mac's DNS program to reload and forget its cache. Despite the name, it doesn't kill it.
Then open fresh sessions.
Note
A dropped connection doesn't scramble your password. If logins start failing after this, it's the lockout counter from retries. See account locked.
No network: no gateway, no DNS, or nothing on .208
Step 4 · vSphere · NetworkManager
What you see
nmcli connection up fails, there's no default route, or getent can't find anything.
Why
Usually one of three things:
Network adapter 1 is still unplugged in vSphere from Step 1.
The address is missing its /24.
Two settings profiles are fighting over one card.
Fix
vSphere → Edit Settings → Network adapter 1 → check Connected. Then run nmcli device status: the STATE column should say connected. Run nmcli connection show to spot duplicate profiles, and delete the extra one with nmcli connection delete "<name>".
Check it
ip -4 route show shows exactly one default via line.
dnf fails after a reboot: "Cannot find a valid baseurl"
Step 5 · software shelves · DVD
What you see
dnf worked before the reboot. Now every dnf command fails.
Why
The shelf is the DVD, and the DVD isn't connected anymore. Either it was connected by hand with no fstab line, "Connect At Power On" is unchecked in vSphere, or it was connected by /dev/sr0 and the drive names swapped.
Fix
In vSphere, check Connected and Connect At Power On. Make sure fstab has the label line from Step 5. Then:
bashmount /media/rhel
dnf clean all && dnf repolist
mount /media/rhel
Connect using the fstab line.
dnf clean all
Forget old saved shelf information.
&&
Only if that worked…
dnf repolist
…show the shelves.
dnf hangs, trying to reach repositories that don't exist here
Step 5 · dnf · air gap
What you see
dnf install sits on "Updating Subscription Management repositories" or waits on a web address it can't reach, then fails.
Why
An old shelf file (like redhat.repo) still points at Red Hat's website or a company mirror. dnf checks every enabled shelf before it installs anything, so one unreachable shelf blocks everything.
"Turn off every shelf, just for this command." The quotes stop the shell from turning * into a list of file names.
--enablerepo='rhel-local-*'
"Then turn back on only the shelves whose names start with rhel-local-." Order matters: turn all off first, then the ones you want back on.
Your original notes used /usr/bin/yum and also listed cockpit-ws. On RHEL 9, yum is just another name for dnf, and cockpit pulls in cockpit-ws by itself, so neither is needed.
Show only the names of files that contain enabled=1, not the lines themselves.
config-manager --set-disabled ID
Sets enabled=0 for that shelf, whichever file it's in.
To silence the "subscription management" message on a server that will never register, set enabled=0 in /etc/dnf/plugins/subscription-manager.conf.
Check it
dnf repolist shows only rhel-local-baseos and rhel-local-appstream.
dnf refuses a package: "GPG check FAILED" or "public key is not installed"
Step 5 · signatures
What you see
Installs from the local shelves fail right after "downloading."
Why
Signature checking (gpgcheck=1) is on, which is correct and required by STIG, but the server doesn't have Red Hat's "seal checker" key loaded, or the gpgkey= path in the repo file is wrong.
Confirm the key file is where the repo file says it is.
rpm --import
Load the key, so packages sealed with it are trusted.
rpm -q gpg-pubkey
-q = query. Loaded keys show up as pretend "packages" named gpg-pubkey.
--qf '…'
"Query format": print just these details. \t = tab, \n = new line.
Don't
Your original notes used gpgcheck=0. That hides the error by skipping the tamper-seal check entirely, which is a STIG finding. Load the key instead.
The VM boots the wrong thing, or the installer says the source is missing
Step 2 · DVDs
Why
The Satellite DVD (which can't boot a computer) or the small RHEL "boot" ISO is in the drive instead of the full RHEL DVD.
Fix
Power off. Set CD/DVD 1 to rhel-9.8-x86_64-dvd.iso with Connect At Power On checked, then power on.
The RHEL installer starts over after the reboot
Step 3
Why
The DVD is still set to connect at power-on, so the VM boots from it again.
Fix
Uncheck Connect At Power On in vSphere and reboot. Your installed system is fine.
Satellite's installer fails with permission errors in /tmp
Step 3 / 8 · STIG · room rules
Why
STIG set /tmp to noexec (no programs may run there), but Satellite needs to run helper programs from it.
Fix
Remove noexec from the /tmp line only (Step 3g), run systemctl daemon-reload, then mount -o remount,exec /tmp (change the rules now, allowing programs). Write it down as an approved exception with your ISSO.
Check it
findmnt -o OPTIONS /tmp shows no noexec.
satellite-installer fails: something already has port 9090
Step 8 · ports
Why
Cockpit is still at door 9090. Usually the blank ListenStream= line is missing, so Cockpit took 9091 and kept 9090.
Fix
Run ss -tlnp | grep 9090 to see who has it (-t TCP, -l listening, -n numbers, -p program). Fix the file in /etc/systemd/system/cockpit.socket.d/, then systemctl daemon-reload, systemctl restart cockpit.socket, and run the installer again.
satellite-installer stopped partway
Step 8 · logs
Fix
Find the first real error:
bashgrep -n -E 'ERROR|Error:' /var/log/foreman-installer/satellite.log | head -20
grep -n -E
Find matching lines, with line numbers (-n), allowing "either/or" patterns (-E).
'ERROR|Error:'
Lines containing ERROR or Error:.
/var/log/foreman-installer/satellite.log
The installer's log file.
| head -20
Only the first 20 matches. The first error is usually the real cause, and later ones are knock-on effects.
Common causes: name or DNS mismatch, clock not synced, 9090 in use, /tmp noexec, or a full disk. Fix the cause and run the same installer command again.
Edge shows a certificate warning
Step 6 / 8 · certificates
Why
One of three things:
Your internal root CA isn't in the Local Computer trusted list.
The intermediate is missing.
The address you typed doesn't match the name on the card (IP instead of name, or short name instead of full).
Fix
Use the exact full name. Check mmc → Certificates (Local Computer). On the server, check which names the card covers:
Don't print the scrambled certificate text itself.
-ext subjectAltName
Just show the names the card is valid for.
Note
A warning on Satellite's own page (port 443) right after install is normal until a custom certificate is added to Satellite.
Uploaded packages don't show up (count is 0)
Step 9 · hammer
Why
Wrong repository ID, the repository type isn't yum, the files aren't .rpm, or the upload job failed.
Fix
bashhammer repository list --organization ""
hammer task list --search 'state=stopped and result=error'
repository list
Confirm the right ID number.
task list
Satellite does big jobs in the background as "tasks." This lists them.
--search '…'
Filter the list: only finished tasks that ended in an error.
A room (filesystem) is full
storage · LVM
Fix
df -h shows which room is full. vgs data shows how much unused space the data pool has (VFree). If there's space, make the room bigger while it's running:
bashlvs
lvextend -r -L +100G /dev/data/<lv_name>
lvs
"Logical volumes summary": lists the rooms by name.
lvextend
Make a room bigger.
-L +100G
Add 100 GiB. The + means "add this much," not "make it this total."
-r
Also resize the filesystem inside the room in the same step, so the extra space is usable right away with no restart.
/dev/data/<lv_name>
Which room: pool data, room name from lvs.
Note
XFS rooms can grow but can never shrink. Grow in modest steps.
The DoD certificate is installed, but the browser or Satellite still complains
Step 6g · production · DoD PKI
What you see
Browser says the certificate isn't trusted, katello-certs-check fails, or Cockpit keeps using its old certificate.
Why
Usually one of four things:
Missing intermediate. The DOD SW CA wasn't added after your certificate.
RHEL doesn't trust the DoD roots yet (production step 1 skipped).
Key mismatch. The CSR was made from a different key than the one installed.
Stale revocation list. The CRL copy on the air-gapped network expired.
Fix
Run the three checks from production step 3 (x509 … -issuer -ext subjectAltName, openssl verify, and the diff key match). For Cockpit also run /usr/libexec/cockpit-certificate-ensure --check to see which file it's actually using. If revocation is the issue, bring in fresh CRLs through your approved process.
Check it
verify says OK, diff prints nothing, and Edge shows the lock icon with a DoD issuer.
"Kerberos authentication did not pass" or "Cannot find KDC"
Only if the server is joined to Active Directory (not part of the base build) · SSSD · Kerberos
Words first
Kerberos is the ticket system Active Directory uses for logins: you prove who you are once and get a ticket (a TGT) that works for hours. The KDC is the ticket office (your domain controllers). A keytab is a file holding the server's own password, so it can get tickets without a person typing anything. SSSD is the RHEL program that talks to AD for logins.
What you see
AD logins or single sign-on worked, then fail, often after a long break or a network or DNS change.
Why
Check these in order:
Clock off by more than 5 minutes. Kerberos refuses tickets from a clock that's too far off. This is the most common cause on an air-gapped server.
Expired tickets. AD tickets last 10 hours by default.
Can't find the ticket office. DNS isn't returning the domain controllers, or /etc/krb5.conf has the wrong realm.
Ask first
bash — on the serverchronyc tracking | grep 'System time'
klist -kte /etc/krb5.keytab
dig +short -t SRV _kerberos._tcp.
chronyc tracking | grep
How far off is the clock? It must be far under 300 seconds.
klist -kte
List what's in the keytab: -k = keytab, -t = dates, -e = encryption types. The names it shows are the exact names to use below.
dig
Ask DNS a question directly (it's in the bind-utils package).
-t SRV
Ask for "service location" records, which say where a service lives.
_kerberos._tcp.REALM
"Where is the Kerberos ticket office for this realm?"
+short
Only print the answers. Expect one line per domain controller.
Fix
bash — on the serverkinit -k '$@'
klist
sss_cache -E
systemctl restart sssd
kinit -k
"Get a fresh ticket using the keytab" instead of a typed password. Add -t FILE to use a keytab other than the default /etc/krb5.keytab.
'SHORTHOST$@REALM'
An AD-joined server logs in as its computer account: short name in CAPITALS, then $, then @ and the realm in CAPITALS. The single quotes stop the shell from treating $ as a variable. Copy it exactly as klist -kte showed it.
klist
With no options: show the tickets you now hold and when they expire.
sss_cache -E
Mark everything SSSD remembers (users, groups, and more) as out of date, so it asks AD again.
systemctl restart sssd
Restart SSSD so it reconnects fresh.
Restart httpd (the web server) only if the web page itself uses Kerberos logins (Satellite "external authentication"). Otherwise it's unrelated.
Windows workstation — PowerShell or CMDklist purge
klist purge
Delete your saved tickets on this PC. Windows gets new ones automatically next time (or lock and unlock the screen).
Check it
klist shows a fresh ticket (krbtgt/REALM@REALM), and id <aduser> lists that user's groups.
Corrected
Your original notes used a lowercase realm and a host/ name that didn't match the server. Realm names are written in CAPITALS by convention, and the match is case-sensitive. Always copy names from klist -kte, not memory. The original real-looking internal hostnames were replaced with lab placeholders.
🔀 What changed from the original session notes
"Mac keyboard mangling": broadened to any mismatched keyboard layout, including Windows.
"Type your password where you can see it": replaced with a symbols-only test.