=^.^=

Searching for Strings and Regular Expressions in Multiple Files on Prehistoric Windows

karma

Before Windows got its WSL swagger installing grep was cumbersome (but at least it was possible, thanks for the memories, Cygwin!)

That being said, you could count on Rando McClient's winb0x to ship with with its own take, the command line tools find:

c:\>find /? Searches for a text string in a file or files. FIND [/V] [/C] [/N] [/I] [/OFF[LINE]] "string" drive:][path]filename[ ... /V Displays all lines NOT containing the specified string. /C Displays only the count of lines containing the string. /N Displays line numbers with the displayed lines. /I Ignores the case of characters when searching for the string. /OFF[LINE] Do not skip files with offline attribute set. "string" Specifies the text string to find. [drive:][path]filename Specifies a file or files to search. If a path is not specified, FIND searches the text typed at the prompt or piped from another command.

... and findstr:

c:\>findstr /? Searches for strings in files. FINDSTR [/B] [/E] [/L] [/R] [/S] [/I] [/X] [/V] [/N] [/M] [/O] [/P] [/F:file] [/C:string] [/G:file] [/D:dir list] [/A:color attributes] [/OFF[LINE]] strings drive:][path]filename[ ... /B Matches pattern if at the beginning of a line. /E Matches pattern if at the end of a line. /L Uses search strings literally. /R Uses search strings as regular expressions. /S Searches for matching files in the current directory and all subdirectories. /I Specifies that the search is not to be case-sensitive. /X Prints lines that match exactly. /V Prints only lines that do not contain a match. /N Prints the line number before each line that matches. /M Prints only the filename if a file contains a match. /O Prints character offset before each matching line. /P Skip files with non-printable characters. /OFF[LINE] Do not skip files with offline attribute set. /A:attr Specifies color attribute with two hex digits. See "color /?" /F:file Reads file list from the specified file(/ stands for console). /C:string Uses specified string as a literal search string. /G:file Gets search strings from the specified file(/ stands for console). /D:dir Search a semicolon delimited list of directories strings Text to be searched for. [drive:][path]filename Specifies a file or files to search. Use spaces to separate multiple search strings unless the argument is prefixed with /C. For example, 'FINDSTR "hello there" x.y' searches for "hello" or "there" in file x.y. 'FINDSTR /C:"hello there" x.y' searches for "hello there" in file x.y. Regular expression quick reference: . Wildcard: any character * Repeat: zero or more occurrences of previous character or class ^ Line position: beginning of line $ Line position: end of line [class] Character class: any one character in set [^class] Inverse class: any one character not in set [x-y] Range: any characters within the specified range \x Escape: literal use of metacharacter x \ Word position: end of word For full information on FINDSTR regular expressions refer to the online Command Reference.

It is also possible to leverage the Windows Explorer shell by navigating to a folder containing the file or files you wish to search and entering your query into the search box integrated into the top-right corner of the window. You can improve the results and reduce query time/resource load by filtering the file type or file name with the file: modifier and conventional Windows filename wildcards like ?, _ and *. For example, to limit the query to plaintext files one would append file: *.txt after the search term(s), separated by a space.

[attachment-0rsB3s]
Explorer's Search functionality is virtually identical from v.7 through 11.

I compiled a small testing sample of various formats and some ZIP archives out of the first two acts of Shakespeare's Antony and Cleopatra, with gratitude to MIT.

[attachment-1vq4rm]
Explorer didn't adopt Ribbon graphical widgets until v. 2008 but Folder Options has managed searching inside ZIP files since XP.

The ability to search inside of ZIP files appeared with the introduction of Windows XP and its broad, built-in support abstract "Compressed Folders" based on the format.

[attachment-0zj7yn]
Cabinet (.CAB) Files support multiple compression algorithms but are usually encoded similarly to .ZIPs

...Searching inside archive files can also be enabled through Folder Options which can be accessed via Explorer's View Tab > View Ribbon > Options Button or...

[attachment-6MCJKC]
A side-by-side comparison of Windows 10's apparently stand-alone File Folder Options and File Explorer's Folder Options

...you can modify the exact same settings in what appears to be the exact same interface, other than the title: File Folder Options which is easily dredged up by way of the the taskbar's integrated Search Bar or the reduced-width Search Button or, if you have wisely deleted it altogether: Meta (Windows) Key & Start Typing...

[attachment-9Rdjvo]
I personally tend to disable Indexing, especially on SSD/Flash backed storage.

For improved performance going forward you may wish to take this opportunity to (same as above) Search for Indexing Options then click the Advanced Button > File Types Tab and find the 'zip' Entry and change the How should this file be indexed? radio button to Index Properties and File Contents; rinse and repeat for any of the other file formats Windows thinks it can handle...

[attachment-AE0kv4]
WIndows has an impressive ability to search the contents of many formats./figcaption>

Unfortunately Windows does not support Search and Replace natively. Therefore if you need that ability in conjunction with multi-file search you must adopt a third party solution.

If one is to use a GUI-first operating system I think it's only appropriate to exercise its strengths; running operations against groups of files should be any GUI's cakewalk, especially with some arbitrary file and recursive directory selection/exclusion. I can't tell you how much I wish I could slap a mouse and control/shift keys on the Linux find command some days (fortunately KATE has me covered where interfacing through NFS/FUSE sshfs is feasible). If you often find yourself having to search multiple files at once, as one certainly does in analytical, systems/network administration and software development roles, I have recently had cause to evaluate these two free, GPLv3 solutions for Windows environments:

[attachment-3lhTxo]
Side-by-Side Comparison of grepWin and AstroGrep

grepWin (Downloads | Source @ GitHub) will only run on Windows 7 SP1 and above. You have the option of 64 and 32 bit conventional installers or a 1.1MB (uncompressed) portable executable which makes it perfectly suited to bootable USB and DVD environments, PXE or staging on a remote server or site (like this one! :o) for rapid access in the field. Supports PERL-inspired (PCRE) Regular Expressions (RegEx) and recommends a third party a tutorial here, while the software has a powerful on the CLI.

[attachment-DF47eD]
grepWin substantially outperforms AstroGrep

The first thing I noticed running grepWin against my sample set was an at-least 2x possibly three-fold reduction in processing time compared to AstroGrep, having run the same search pattern with it a dozen times prior. An identically matching set across text and RTF files was acquired, only among the plain text (*.TXT) and rich text (*.RTF) files. String replace functionality was fast and I was admonished both to enable backups and disable UTF-8 mode (which I had changed from the default off value; while that shouldn't have caused any issues it is nice to know that an appropriate level of hand-holding is implemented before executing a potentially destructive and ultimately embarrassing act.

[attachment-q4HVob]
AstroGrep: Does what it claims to. And not a penny more.

AstroGrep (Downloads | Source @ SourceForge) has a long history and an established userbase. AstroGrep adds a context menu to Explorer when you right-click on directories while grepWin adds a context menu for individual files. Even if your main use-case substantially differentiates between their featuresets or if the lack of replace functionality is a deal-killer (as it sadly is for me) that could be enough utility to warrant installing both.

Out of the box AstroGrep was able to parse the plain text (*.TXT) and rich text format (*.RTF) files from my sample set. Enabling the "File Handler" plugin added the two OpenDocument Format (*.ODF) files but got nothing out of the HTML, PDFs or the files in archives (as expected). Unfortunately, while it could see the matches I could not: despite the plugin apparently parsing the ODF format enough to find text strings, the reproduction inside of AstroGrep's interface was binary and largely unintelligible.

According to this thread from 2017 that appears to document a discussion between to developers a Replace functionality may have been planned but in the intervening five years apparently that has gone nowhere. A test version of the app (in "portable" form) is provided in the thread; after completing your search the greyed out "fast-forward" button at the bottom of the search form in the left panel of the UI becomes slightly less grey; clicking on it reveals the Replace field. While the replace procedure did work, it only worked for my plain text (*.TXT) files; the rich texts and ODF remained unchanged.

I was hoping I would be able to recommend both applications but the lack of a replace function in AstroGrep - at least for now - cements my personal endorsement with winGrep.

Tools for Boot Images, LiveCDs and Windows PEs

karma

The following are some bootable Windows (WinPE) environments & builders/utilities, Linux-based recovery/utility LiveCD/LiveUSB images and some general purpose multiple image boot stick tools I have come across and want to come back to when I have the time but (for the most part) have not tested yet - so before using any of them be sure to perform due diligence. I have not and have no idea how trustworthy or effective any of the following projects, sites and executables are; do not consider this list to be an endorsement. As such this list doesn't belong on my Favourite Windows Software page so I'm putting it here for our (my) benefit. I will come back to update this page once I have something educated to say. Feel free to let us know your experience - once I finish programming the comments thing. Some day. Never. Maybe. Gesundheit.

  • According to the Wikipedia article on WinBuilder this package used to be available at https://winbuilder.net but at the time of this writing the domain redirects to https://reboot.pro probably the result of expired domain sniping. Which may be for the best: it appears to be a relatively active mutual support community that hosts downloads for numerous delicious looking utils and some member-rolled bootable images. No idea as to the safety of this site or its downloads, you may prefer to find the official home pages for those utilities before downloading - but as far as WinBuilder is concerned this seems (at present) to be the best place to get it at a glance. The wiki handily links to a bunch of bootable images that were created with WinBuilder.
  • At present writing if you navigate to http://wimbuilder.world you are redirected to http://hello.wimbuilder.world/hello/ which is... uh... hella SEO. >.> anyway in turn it links to the ostensibly official github source repo at https://github.com/slorelee/wimbuilder2, the Issues section of which indicates a healthy userbase. That's right, WIM Builder:

    WimBuilder2 is an open source lightweight editor for wim file. but be powerful, fully customizable and easy to use.

    That's a new one for me. https://fileinfo.com/extension/wim:

    What is a WIM file?

    A WIM file is saved in a file-based imaging format that was introduced with Windows Vista. It allows a single disk image to be deployed to multiple computer platforms. WIM files are used to manage files such as drivers, updates, and components without booting the operating system image.

    Ahhh. So that's how the wind0ze buggers do "slipstreaming". https://theoven.org/ seems strongly correlated with support for this utility.

    NOTE: funnily enough, about a week after writing this I ended up doing much of the above by hand when faced with a USB 3.0-only machine that needed drivers slipstreamed to install Windows 7. I documented the effort in detail at Add Missing Hardware Support (Drivers) to Windows Install Media (DVDs, USB Sticks).

  • RMPrepUSB at https://rmprepusb.com/:

    RMPrepUSB is a Windows 32-bit utility to partition and format a drive (especially USB drives).

    • Install/repair boot code
    • Clear write-protect\read-only drive flags
    • Sector editor
    • SD cards can be up to 10% faster (when formatted as FAT32)
    • Test for ‘Fake’ size drives
    • Fix various USB boot issues
    • Run a 32-bit MBR Emulator to boot from a USB drive (QEMU) with full write access
    • Includes the command line utility RMPartUSB.exe

  • Easy2Boot at https://easy2boot.xyz/:

    E2B – the free USB multiboot solution for professionals

    • Boot to (almost) anything (Legacy/UEFI64/UEFI32/Secure Boot) from just one USB drive for free!
    • E2B includes three menu systems – E2B grub4dos, agFM/grubfm and Ventoy for the best chance of booting any payload!
    • E2B and agFM use a generic method of booting Linux ISOs. This means it is often more successful than Ventoy.
    • Over 1 million downloads – E2B is used by techs, repair shops, schools and Universities all over the world.

  • Ventoy at https://www.ventoy.net/ is included as part of E2B above.

    Ventoy is an open source tool to create bootable USB drive for ISO/WIM/IMG/VHD(x)/EFI files.
    With ventoy, you don't need to format the disk over and over, you just need to copy the ISO/WIM/IMG/VHD(x)/EFI files to the USB drive and boot them directly.
    You can copy many files at a time and ventoy will give you a boot menu to select them (screenshot).
    x86 Legacy BIOS, IA32 UEFI, x86_64 UEFI, ARM64 UEFI and MIPS64EL UEFI are supported in the same way.
    Most type of OS supported (Windows/WinPE/Linux/ChromeOS/Unix/VMware/Xen...)

    [attachment-xpK9rA]
    Dang. I need the hell out of this...

    I would make the observation that you will want to include version numbers for your isos in the boot menu.

  • Gandalf’s Windows 10PE x64 seems active, well-liked and comes with a ridonkulous list of installed utilities.
  • MediCat USB - A Multiboot Linux USB for PC Repair A glorious ~25 gig multiboot usb stick formulation with frequent, rolling updates.

While I haven't tried some of these options personally and/or very seldomly use them, I think you can reasonably take their safety for granted:

  • SystemRescue is a conventional system recovery linux LiveCD distro that has been around forever and remains up-to-date.

    ...a Linux system rescue toolkit available as a bootable medium for administrating or repairing your system and data after a crash. It aims to provide an easy way to carry out admin tasks on your computer, such as creating and editing the hard disk partitions. It comes with a lot of Linux system utilities such as GParted, fsarchiver, filesystem tools and basic tools...

    What I think they should mention up front is that it includes PhotoRec which makes this the ISO I'll be reaching for next time a grampa drive dies.

  • UNetbootin runs on Windows, Linux and MacOS. The official project page at https://unetbootin.github.io/ describes:

    UNetbootin allows you to create bootable Live USB drives for Ubuntu and other Linux distributions without burning a CD.

    You can either let UNetbootin download one of the many distributions supported out-of-the-box for you, or supply your own Linux .iso file.

  • Clonezilla is a modern, open-source and more fully featured disk imaging/cloning suite than the long-defunct Norton Ghost which I used to see running in the shop at all hours in my youth. It is included as part of Diskless Remote Boot in Linux (DRBL) https://drbl.org/ for turn-key network booting
  • ophcrack has a livecd!!! Get it from https://ophcrack.sourceforge.io/download.php and be sure to select the correct image for your target version of Windows to ensure it comes with the correct rainbow tables or download the table-less image and choose a specific table from https://ophcrack.sourceforge.io/tables.php for better coverage (many will not fit on smaller media/drives).
    NOTE If you already have NT or LM hashes extracted and ready to crack you can quickly just plug them into the rainbow tables live via their site at https://www.objectif-securite.ch/en/ophcrack!
  • If you don't care to crack Windows account passwords you can simply disable them; this feature is widely available on distributions like Hiren's but since that hasn't been officially maintained in years and it doesn't look like a supported feature of SystemRescue (above) I feel I should include https://www.supergrubdisk.org/rescatux/ and https://pogostick.net/~pnh/ntpasswd/.
  • The Microsoft Desktop Optimization Pack (MDOP) "is available as an additional subscription for Software Assurance customers" which means you have to pay for it. Apparently. Anyway it comes with the Diagnostic and Recovery Toolkit (DaRT) which is a bootable image full of recovery tools. However, I am way more interested in the virtualization features...
  • Another official Microsoft offering was the Windows to Go Live USB functionality available from Windows 8 Enterprise through Windows 10 Enterprise until the May 2020 update

This PC CAN run Windows 11: How to Disable Windows 11 Installer's Minimum Hardware Requirements Checks

karma

As we have known for some time Windows 11 will install from bootable DVD or USB stick (but not Windows Update) on virtually any x86-64 (aka amd64 - I have noticed in some documents Microsoft is calling this architecture "IA 64-bit" which is trop droll considering it was Microsoft's pressure on AMD to innovate the 64 bit 80x86 platform at Intel's expense at a time when IA64 clearly referred to the ill-fated Itanium architecture) CPU, including for a short and embarrassing moment where it would "officially" be happy to install on Pentium 4 661s. The catch is that new Windows Updates will not be delivered to so-called unsupported systems which sounds rather pointless to me; as I recall Windows Updates are made available to the public freely as individual executable installers from their Knowledge Base and if a dozen people haven't already written an automatic scraper for Windows 11 updates I would be utterly shocked and disappointed.

Unfortunately there are a few more minimum requirements that the installer checks for that may get in our way, particularly if we are trying to simply preview what is to come in this new version from the safe confines of a Virtual Machine instead of risking an otherwise perfectly good and still supported Windows 10 installation at this early stage in its release:

  • System must present a Unified Extensible Firmware Interface (UEFI) instead of a Basic Input/Output System (BIOS) and further:
    • The firmware must be Secure Boot capable. You might be able to enable Secure Boot on a physical machine.
    • A Trusted Platform Module meeting the version 2.0 specification must be provided by a discreet chip/module, as a component of the integrated chipset or as a firmware implementation. Newer machines should be able to enable their TPM if it appears to be missing.
  • At least 4GB RAM must be available to the system. This could be a problem on bare-metal hosts with 4GB of RAM but a portion allocated to shared graphics memory.

I think it's interesting that Microsoft would impose such a strict requirement on RAM while in many areas it has embraced virtualization technology and employed it in impressive, creative ways to make the Windows operating system vastly more secure, relevant and capable than it ever has been before. It seems in these early days at least Microsoft is aiming for a very smooth, very optimal first impression among early adopters at the expense of bringing more budget-constrained users and virtualization-backed service providers along to this new paradigm of sophisticated security measures, Android application compatibility and more.

That said, they couldn't stop us if they put their backs into it, and we're going to leverage some handy functionality they left in the installer for us that will shimmie right past the most egregious of its hardware requirement checks. Upon running the installer you have likely encountered the error:

This PC Can't Run Windows 11

This PC doesn't meet the minimum system requirements to install this version of Windows. For more information visit https://aka.ms/WindowsSysReq

[attachment-4r091J]
Anything you can do I can do better.

Use the Back button embedded (practically camouflaged) in the top left corner of the window's title bar to return to the edition selection menu.

[attachment-yvpUd4]
The one that doesn't get in my way, please.

From here you can press Control + F10 on your keyboard to launch a command prompt.

[attachment-l7IxKZ]
We're taking command of this operation!

Type regedit and hit Enter to launch the registry editor.

[attachment-tYpuGu]
The registry is the key.

Navigate to HKEY_LOCAL_MACHINE\SYSTEM\Setup. Right-click on the Setup key and select New > Key. Name the new key LabConfig.

[attachment-t1Qk35]
Bypass is DE-WORD!

Right click on the LabConfig key or anywhere in the right-hand pane while it is selected. Under New click on DWORD (32-bit) value and create the following values:

  • BypassRAMCheck
  • BypassSecureBootCheck
  • BypassTPMCheck
[attachment-VfIObu]
I value control over my own system.

Now double click on each value or Right Click/press the Context Menu key and Modify each item to contain the value 1 (integer one).

[attachment-9MuE7D]
I put a hex on you! Now you're mine~

Exit the registry editor and carry on with your installation as usual.

[attachment-hIKvkx]
How do you like me now? :/

Hopefully at this point your installation continues without a hitch. It's not certain whether Microsoft will stick to these draconian requirements in the future or ease them up to make Windows 11 a more realistic virtualization candidate (among many other great reasons) but one thing is certain: there will always be a way around them - whether provided as a somewhat convenient override by Microsoft themselves or through unofficial channels as part of yet another arms race.

The Superb Prebuilt Virtual Machine Image Repository Short-List

karma

For your pleasure and my rapid deployment I present my personally endorsed and expertly curated short-list of sources for professionally appointed virtual appliances and environments:

  • OSBoxes maintains an impressively varied repertoire of meat-and-potatoes server and desktop/end user installations of the most relevant flavours, lightweight performance-oriented distributions particularly suited to use in VM-optimized projects and embedded essentials like Android-x86, FreeDOS, Raspbian and more.
  • Kali Linux stays on top of their VM images - get it from the source.
  • Microsoft provides free images of Internet Explorer on Windows 7 through Edge on Windows 10. Web developers don't need to worry about appeasing Internet Exploder anymore but I don't need to tell you how valuable these images are to neglected and abused admins forced to wrangle legacy systems.
  • Univention provides mostly free, mostly open source dockerized turn-key IT infrastructure and economically relevant virtual appliances based on its custom distribution Univention Corporate Server through an in-house app store model. You can choose to deploy a full-scale, centrally managed cloud solution composed of up to thousands of virtual machines or download a single virtual machine image directly from the web front-end to their marketplace, the Univention App Center and it will happily run, receive updates and benefit from the same standardized management capabilities as a massive cloud would all by itself. You can of course scale up later if you choose. It's.... actually pretty cool... (you guys hiring?)
  • The best place to get Oracle Solaris images (among other Oracle offerings) is Oracle. Go figure.

Honourable mentions:

  • Linux VM Images - we've already ticked almost all of these boxes, I'm including it since it's such a large collection but have never personally used it. Maybe today's the day you'll find something exotic in there.
  • VirtualBoxes provides a few interesting non-linux images like Plan9 and MINIX.
  • There is no better place to snag a weird Windows beta or nostalgia trip than Virtual Disk Images
  • Microsoft provides evaluation versions of some editions of Windows and Windows Server; the CLI-only distribution of Hypervisor 2012 is provided here free of charge.
  • Server 2008 R2 is available as a 180 day evaluation but Server 2003 R2's product page does not indicate a time limit (you tell me - I have better things to do! :D)
  • Well, this doesn't look like a sanctioned release of Windows 98 SE but I won't tell if you won't.
  • The Naval Postgraduate School's Center for Cybersecurity and Cyber Operations provides "Labtainers"- Fully packaged Linux-based computer science lab exercises with an initial emphasis on cybersecurity. God damn youth is wasted on the young. I wish i had time to play with this.
  • Oh hey! TurnKey is still a thing. I'm afraid its integration/management features are about as compelling as its selection of software in comparison to UCS; particularly as the TurnKey Hub requires AWS... and I think at least half of the half dozen "fine publications" it touts having received coverage in went out of business while I was yet nae but knee-hight... >.>
  • This disposable browser appliance had a lot more appeal 10 years ago when it was maintained around 300MB, the Firefox installation kept only a few versions behind and both virtualization technology and free, pre-configured virtual machine images were much harder to come by. I suppose I'm including it for nostalgia's sake, you're better off downloading the CentOS desktop version from OSBoxes.

Replicate Installed Packages on New RHEL/Fedora/CentOS/Debian/Ubuntu Depoloyment

karma

You've spend a lot of time getting a particular installation just right, whether it's a bare metal server, virtual machine, desktop workstation or laptop: the role is clearly defined and you'd like to replicate it as quickly as possible either with a fresh base installation or on a totally separate new host. This is particularly salient when upgrading major versions of Qubes Fedora TemplateVMs: generally speaking not a lot of customization goes into these base layers on which AppVMs and DisposableVMs etc. are meant to be built - except for all of the package management that goes into fleshing out a comfortable and usable default environment.

One option is to follow the Qubes documentation for upgrading a Fedora template in place: https://www.qubes-os.org/doc/template/fedora/upgrade/ however I am inclined to take advantage of the template packages as outlined at https://www.qubes-os.org/doc/templates/fedora/ both for the additional management capabilities (e.g. one-line reinstall, version management) and the clean start and distinction between images.

Although some articles recommend obtaining your package list from:
# dnf repoquery --userinstalled acl-0:2.2.53-3.fc30.x86_64 adobe-release-x86_64-0:1.0-1.noarch alternatives-0:1.11-4.fc30.x86_64 attr-0:2.4.48-5.fc30.x86_64 audit-libs-0:3.0-0.15.20191104git1c2f876.fc30.x86_64 basesystem-0:11-7.fc30.noarch ...
There are four issues with this approach:

  • Specific architectures are specified. It's rare that you will be switching architectures but one of the most amazing aspects of Linux is its platform versatility and since things can slip into and out of noarch all the more reason to let the package manager's default settings handle the unforseeable. Agnosticism is next to godliness - but I might be biased :)
  • Base packages, the packages that came preinstalled with the Minimal Server role or the TemplateVM etc. are included. I need a list of only those packages I have intentionally, specifically installed myself or I run the risk of trying to install deprecated, merged, removed, abandoned, unnecessary, etc. packages. This is of particular concern if I am upgrading to a new major release version and/or switching my base installed package set/"server role".
  • Specific versions are specified which is begging for trouble even outside of the context of a global update (ask any Gentoo admin!)
  • This is not a list of the packages that I have chosen to manually install; it is a list of every package installed after the base installation. In other words, it is every package I have chosen to install AND each one of its dependencies. Ask a Gentoo admin how they feel about explicitly installing dependencies!

The thing about dependencies is they like to change and when a dependency has been abandoned by an intentionally installed package yet is itself explicitly installed you are open to the liabilities (dependency hell (eek!), wasted space and update time, tool for intruders...) of keeping that package around and it can be quite unclear months or years after the fact if an abandoned dependency is safe to eliminate or if it provides the crucial library or shim or goo or magic smoke that makes some special, foreign or from-source software go~.

dnf history gets us a lot closer:
# dnf history ID | Command line | Date and time | Action(s) | Altered ------------------------------------------------------------------------------- 24 | install tigervnc | 2020-07-29 21:15 | Install | 4 23 | install k3b | 2020-07-29 13:47 | Install | 28 22 | install deluge | 2020-07-28 23:06 | Install | 47 21 | install gnome-tweak-tool | 2020-07-28 10:14 | Install | 16 20 | install mlocate youtube- | 2020-07-25 03:57 | Install | 2 19 | install elinks links lyn | 2020-07-21 23:05 | Install | 1 18 | install nano psmisc nmap | 2020-07-21 22:58 | Install | 1 17 | install libreoffice | 2020-07-21 22:52 | Install | 99 16 | install vlc | 2020-07-20 23:31 | Install | 32 15 | install ffmpeg | 2020-07-20 23:10 | Install | 31 14 | upgrade --refresh | 2020-07-20 23:05 | Upgrade | 2 EE 13 | install kate gimp | 2020-07-20 01:07 | Install | 93 12 | install chromium | 2020-07-19 06:09 | Install | 13 11 | install screen sshfs nma | 2020-07-19 06:05 | Install | 19 10 | update | 2020-07-19 05:52 | I, O, U | 304 EE 9 | install -y --cacheonly - | 2019-12-25 18:19 | Install | 1 8 | install -y --cacheonly - | 2019-12-25 18:18 | Install | 4 7 | install -y --cacheonly - | 2019-12-25 18:17 | Install | 16 EE 6 | install -y --cacheonly - | 2019-12-25 18:16 | Install | 4 EE 5 | install -y --cacheonly - | 2019-12-25 18:12 | Install | 1 4 | install -y --cacheonly - | 2019-12-25 18:12 | Install | 1 3 | install -y --cacheonly - | 2019-12-25 18:11 | Install | 1 2 | install -y --cacheonly - | 2019-12-25 18:10 | Install | 125 EE 1 | install -y --cacheonly - | 2019-12-25 18:02 | Install | 785 EE
Unfortunately, even if we expand the terminal really far the Command line column is prone to truncating on long package lists. As an aside, I think it's really neat that we can use dnf history info number to zero in and look at, for example, the entries from 1 to 9 in this TemplateVM's history. They show us first the complete base installation at entry 1, then the standard constellation of packages Qubes adds to implement its unparalleled integration and enhancements at slot 2, then every package added and updated before this particular version of the official fedora-30 TemplateVM image was itself rolled into an RPM and deployed. It's always worth taking the time to get to know what you're made of - time permitting!

I must regretfully report that at the time of this writing, having spent hours digging through the dnf sqlite DBs, JSON files, API documentation etc it seems that while it was possible with yumdb to pull a simple list of explicitly user-installed packages free of their dependencies there is simply no facility in current dnf implementations to demarcate the difference between a user-installed package and a package that was installed as a dependency of one. We can at least address two issues: it is easy to get rid of the architecture and version information from our package list but it may be necessary to manually edit the list to remove dead packages, particularly if upgrading by major version number revisions. This is accomplished by using the --queryformat/--qf filter:
# dnf repoquery --userinstalled --qf "%{name}"

The same effect can be achieved through the dnf history userinstalled route via the application of sed:
# dnf history userinstalled | sed 's/-[0-9].*//' | sed '1d' | sed '/.kernel./d'

Direct an itemised list like the preceeding to a text file from stdout using the > operator, copy it to the receiving host and it can be easily edited and batch processed through xargs:

# < package_list.txt xargs dnf -y install

There is one more imperfect option which I have incidentally been using for years, it relies on:

  • Your having used bash to perform most/all of your dnf install operations
  • Gracefully loging out of/closing your shell session(s) afterwards (as opposed to exiting via SIGTERM or segfault or loss of power etc.)
  • Not having exceeded the default .bash_history buffer length (very hard to do in a TemplateVM)

You guessed it...
# cat ~/.bash_history | grep "dnf install" dnf install screen sshfs nmap links lynx nano whois bind-utils dnf install chromium dnf install kate gimp dnf install vlc dnf install ffmpeg dnf install epel-release dnf install nano net-misc psmisc nmap screen dnf install nano psmisc nmap screen dnf install elinks links lynx w3c dnf install elinks links lynx dnf install bind-utils whois sshfs

This method's saving grace is the ease with which it is copied and pasted between remote SSH sessions. I'm the kind of person who uses sudo bash and su so my installations will be found under the root user's .bash_history; if you are more of a sudoer type grep your regular user's log accordingly. Note that a little search-and-replace in a text-editor to add the -y flag to dnf would allow one to copy and paste the entire block into a remote session and let it run non-interactively.

Finally, as the Qubes documentation suggests, you can simply record the changes you make to your TemplateVMs. For instance, I have been compiling a shortlist of so-called favourite programs for RHEL/CentOS so I can quickly assemble an environment I'm used to on the numerous virtual machines I end up configuring month to month regardless of where I am and without having to hunt down an already configured image to crib off of. It's much less frustrating to get the utilities I reflexively expect to be available installed up front instead of as I notice their absence.

As for Debian/Ubuntu and derivatives that ship with dpkg the situation is not perfect - in that we are still going to end up with a list of base packages and dependencies that may need to be edited - but there are facilities built-in to address this simple type of migration.

On the egress system run:
# dpkg --get-selections > /tmp/package_list.txt

Then on the ingress system, after copying over the package_list.txt file, run:
# dpkg --set-selections < /tmp/package_list.txt # apt-get -y update # apt-get dselect-upgrade

Of course it is also possible to grep our .bash_history as with redhat but depending on your system and habits it might be necessary to check for more than one command:
# cat ~/.bash_history | grep "apt install" # cat ~/.bash_history | grep "apt-get install" # cat ~/.bash_history | grep "dpkg install"