Failed to mount /sysroot on CentOS/RHEL

Power outage, VM tanked, whatever the reason you may need to run xfs_repair from the recovery console.
Mounting /sysroot... [ ***] A start job is running for /sysroot (3min 59s / 4min 31s)[240.527013] INFO: task mount:406 blocked for more than 120 seconds. [ 240.527056] "echo 0 > /proc/sys/kernel/hung_task_timeout+secs" disables this message." [FAILED] Failed to mount /sysroot. See 'systemctl status sysroot.mount' for more details. [DEPEND] Dependency failed for Initrd Root File System. [DEPEND] Dependency failed for Reload Configration from the Real Root. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Reached target Initrd File System. [ OK ] Stopped dracut mount hook. [ OK ] Stopped target Basic System. [ OK ] Stopped System Initialization. Starting Emergency Shell... Genrating "/run/initramfs/rdsosreport.txt" Entering emergancy mode. Exit the shell to continue. Type "journalctl" to view system logs. You might want to save "/run/initramfs/rdsosreport.txt" to usb stick or /boot after mounting them and attach it to a bug report. :/#

xfs_repair -v /dev/dm-0
Be sure to include any other volumes before rebooting, i.e.: /dev/dm-1.

Credit to https://unix.stackexchange.com/questions/337289/how-to-repair-centos-failed-to-mount-sysroot for full error message copypasta. Mine was in a GUI in an RDP in a bump in a hole in the log down by the river~

The BIOS has Corrupted hw-PMU Resources

Modern HP servers, among others, may display the following warning when booting RHEL7+ and associated flavours:

The BIOS has corrupted hw-PMU resources (MSR 30d is 330)

This can be safely ignored. However, if it bothers you, you can disable it thus:

  1. Boot machine to RBSU (F9).
  2. Press CTRL+A.
  3. Select "Service Options."
  4. Select "Processor Power and Utilization Monitoring."
  5. Select "Disable."
  6. Press F10 to save and exit.
  7. Reboot.

Per HPE advisory https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c03265132

Discover All Tenda Devices in ARP Database

Tenda makes embedded devices that are frequently compromised and/or used in DoS attacks. You can actively scan or sniff for the following list of vendor IDs live but I wrote an SQL backed multi-router ARP table reporting system for an ISP years back. This query will locate all clients operating such a device.

select distinct `ip` from `arp` where ( `mac` like 'E8:65:D4%' or `mac` like 'D8:32:14%' or `mac` like 'CC:2D:21%' or `mac` like 'C8:3A:35%' or `mac` like 'B8:3A:08%' or `mac` like 'B4:0F:3B%' or `mac` like 'B0:DF:C1%' or `mac` like '58:D9:D5%' or `mac` like '50:2B:73%' or `mac` like '50:0F:F5%' or `mac` like '08:40:F3%' or `mac` like '04:95:E6%' )

I have had enough problems with these devices that I suggest preemptively locating them, blocking typical remote management/access ports upstream and have all affected users return or upgrade their router. Follow up and remove corresponding upstream rules once the devices have been removed (use arping to verify) to recover netfilter resources.

Search All Files Recursively for a String

find / -xdev -type f -print0 | xargs -0 grep -H "string"
The -xdev argument ignores other filesystems mounted under the given path, avoiding /proc and /dev for example. If, for example, your /home folder is on a different partition, it won't be searched. Run the command again on that specific path.