=^.^=

Recover a Solaris Service Stuck in Maintenance Mode

karma

Something updated, a configuration file changed, there was a power outage, aliens descended from the heavens... and now a service that was running yesterday is nowhere to be found. Even the last bastion of an unraveling mind - turning it off and on again - has failed you. Welcome to SMF's maintenance mode.
# svcs | grep gdm maintenance 9:33:00 svc:/application/graphical-login/gdm:default

According to the official documentation (Managing Services (Overview) Introduction to SMF > SMF Concepts > Service States) a service enters the maintenance state when:

The service instance has encountered an error that must be resolved by the administrator.

This is done to prevent a critical error from running out of control or unnecessarily suspending system or service startup procedures when a service fails to start multiple times. The documentation (How to Restore a Service That Is in the Maintenance State) further recommends:

Determine if any process that are dependent to the service have not stopped.

Normally, when a service instance is in a maintenance state, all processes associated with that instance have stopped. However, you should make sure before you proceed. The following command lists all of the processes that are associated with a service instance as well as the PIDs for those processes.
# svcs -p service-name

(Optional) Kill any remaining processes.

Repeat this step for all processes that are displayed by the svcs command:

# pkill -9 process-name

Before we can clear our problem service we need to figure out what went wrong. We can take a detailed look with svcs -xv service-name:
# svcs -xv gdm svc:/application/graphical-login/gdm:default (GNOME Display Manager) State: maintenance since Sat Nov 20 09:33:00 2021 Reason: Method failed repeatedly. See: http://support.oracle.com/msg/SMF-8000-8Q See: man -M /usr/share/man -s 8 gdm See: /var/svc/log/application-graphical-login-gdm:default.log Impact: This service is not running.

It is so delightful to have a pertinent log file suggested in the middle of a potential crisis instead of having to sort through reams of journalctl vomit.

Once the issue has been resolved bring the service out of maintenance mode:
# svcadm clear service-name

Start it from the top:
# svcadm start service-name

And verify all is well:
# svcs -x gdm svc:/application/graphical-login/gdm:default (GNOME Display Manager) State: online since Sat Nov 20 10:06:16 2021 See: gdm(8) See: /var/svc/log/application-graphical-login-gdm:default.log Impact: None.

Update the Default Global Time Zone on Solaris 11

karma

There are two ideologically correct methods to change the default system time zone on Solaris 11:

Using nsladm:

Install the nls-administration package if it is not already available:
# pkg install nls-administration

List the available time zones:
# nlsadm list-timezone

Finaly, set the correct time zone:
# nlsadm set-timezone timezone # nsladm set-timezone Canada/Eastern

Updating the timezone:default SMF property timezone/localtime:

# svccfg -s timezone:default setprop timezone/localtime = astring: Canada/Eastern # svcadm refresh timezone:default

Make Remote Firefox Run Faster over SSH with X11 Forwarding and X2Go

karma

Running X clients (graphical applications) over SSH with X11 Forwarding is notoriously slow, even over high speed remote links because the nature of the X protocol demands a lot of pinging-and-ponging back and forward which stacks latency up regardless of your connection speed. For performance reasons alone you are better off using VNC, NoMachine NX/FreeNX or X2Go if you find yourself doing a lot of work with remote graphical applications.

However performance isn't the only reason you should stop using X11 Forwarding over SSH if you can; for a very long time this functionality has been considered harmful and the reasons to abandon it have only kept growing.

As always, there are practical reasons to bend or break the rules and no shortage of situations where a thoughtfully configured, trusted environment can exist and X11 Forwarding is appropriate. Firefox is particularly slow over remote X and there are a couple of things you can do to improve the experience:

  • Use the -C option when you connect to the remote SSH daemon. This enables compression, which will only help when your problem is low connection speed. Enabling compression on a high speed link will only add to your latency which in 2021 is probably your real problem. Some articles recommend specifying the gzip CompressionLevel (from 0 to 9, default 4), a ssh_config option with the -o flag on the command line (-o CompressionLevel 9) but this directive is only applicable to SSH protocol version 1 and will likely be ignored.
  • Specify a cypher that is less computationally expensive for streaming data, i.e. RC4 or BlowFish-CBC; RC4 is deprecated and may not be available so we can specify a list of supported cyphers in decreasing order of preference with -c arcfour,blowfish-cbc. Be advised that these algorithms are considered less secure than the default AES, with RC4 being considered substantially less secure though much faster than BlowFish - apply appropriate judgement when considering the trade-offs of choosing alternate cyphers.
  • Use -Y instead of -X to enable "trusted" X11 Forwarding and disable the X11 SECURITY extension controls. This ability relies on the ForwardX11Trusted setting being enabled in your local ssh_config and is generally discouraged however it will reduce the overhead of remote X forwarding at the expense of security considerations like inter-client snooping, key grabs and injection, among many more. That being said, it is debatable how much of a difference this really makes to overall security and running certain apps without trusted X11 Forwarding will cause them to crash. Some excellent reading fodder on the shotcomings of the X11 SECURITY extension can be found in this reddit discussion: What is up with the X11 SECURITY extension? The bottom line here is regardless of whether you use -X or -Y you need to have a lot of faith in the remote system when using X11 Forwarding.
  • If it is still available in your version of Firefox, the most impactful thing you can do is enable XRender support in Firefox's about:config by toggling gfx.xrender.enabled from its default false to true. The X Rendering Extension was developed to provide a facility for rendering transparency - either with hardware acceleration (on the GPU) where supported or falling back to software rendering (on the CPU) where it isn't.

    XRender was disabled in Firefox by default when Firefox first moved to a built-in off-screen surface rendering implementation that is, in a conventional setting where the X server and clients are all located on the same machine, more stable and effective at taking advantage of hardware acceleration. Then a whole new compositing engine coalesced under the WebRender moniker. WebRender is a great advancement that takes advantage of OpenGL to vastly speed up virtually all aspects of rendering a page. By design it is tied to the GPU of the machine where firefox is running; this is a desirable trait because it won't suffer from the unnecessary overhead of the legacy, network-oriented X protocol calls XRender uses.

    Unfortunately, disabling XRender has the effect of forcing the entire Firefox window to be composed and redrawn with every change and that has subsequently devastated firefox performance over X11 Forwarding and X2Go. This is covered in numerous Bugzilla tickets where the firefox developers have spoken disparagingly about its use-cases. Worse, firefox is poised to drop support for XRender entirely; although it is obviously no match for WebRender when run locally, those costly X protocol calls and ability to render on whatever machine is actually displaying the application (including any hardware acceleration it makes available) are what makes it ideal for remote use.

    If your browser lacks the gfx.xrender.enabled option you may be able to enable the same functionality by setting gfx.webrender.force-disable to true, however:

    ...with the launch of Firefox 93, the support for the options to disable WebRender will be discontinued and this engine will be mandatory.

    It seems our only option going forward to use Firefox remotely with acceptable responsiveness is to either adopt a bit-scraping remote desktop protocol like VNC or run an insecure and outdated version to continue support for legacy use-cases like multi-seat configurations.

Fix Remote Firefox over SSH with X11 Fowarding Runs Local Instance

karma

You have logged into a remote host with X11 Forwarding enabled and at the command line you run firefox or thunderbird. To your surprise, a local version launches and loads your locally configured home page, extensions, etc. - or an already running instance opens a new tab. You close it, return to the remote host's command line, try again - the same thing happens! What in god's name is going on?

It turns out that the firefox command uses X11 inter-client communication and, assuming it is being used on a single machine, notices the already-running session you have open on your local machine and tries to work with that instead of launching a new session on its own host. You will notice if you terminate the locally running session (make sure firefox and any of its related processes have actually exited) and try again that the expected behaviour occurs: a new firefox session is started on the remote machine and forwarded to your local X server.

The remedy to this situation, if you intend to run a local and remote session at the same time, is to use the --no-remote flag on the command line.
{local} $ ssh user@remote-host -Y {remote} $ firefox --no-remote

Why no-remote when running firefox remotely is exactly what we're trying to do? It turns out the nomenclature was established in the Netscape days to refer to the CLI command's ability to remotely issue commands to an already-running, separate, remote, if you will, instance:

For more details on this functionality and its history I direct you to the excellently written and well-researched Dissecting Firefox's -no-remote option article by Bryce Van Dyk.

ps Output is Truncated when Piped to grep on Oracle Solaris

karma

You may have noticed running ps from the command line produces output that you later can not filter for using grep on Oracle Solaris. You can get a visual understanding of what grep is seeing by piping the output into less:
$ ps aux USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND user 5403 5.7 9.463597966305772 ? S 16:42:27 48:30 /opt/VirtualBox/amd64/VBoxHeadless --comment mail --startvm user 1313 2.8 3.322088802159016 ? S Nov 15 696:52 /opt/VirtualBox/amd64/VBoxHeadless --comment nextcloud --startvm user 3973 1.3 1.611256801074444 ? S 23:01:16 59:10 /opt/VirtualBox/amd64/VBoxHeadless --comment piranha1 --startvm user 3974 0.6 1.711521601106504 ? S 23:01:19 60:23 /opt/VirtualBox/amd64/VBoxHeadless --comment piranha2 --startvm user 997 0.5 0.110100053588 ? S Nov 15 31:50 /opt/VirtualBox/amd64/VBoxSVC --auto-shutdown user 5404 0.4 0.4257364227628 ? S 16:42:54 1:38 /opt/VirtualBox/amd64/VirtualBoxVM --comment mail --startvm user 987 0.2 0.13010412384 ? S Nov 15 8:32 /opt/VirtualBox/amd64/VBoxXPCOMIPCD root 3 0.2 0.0 0 0 ? S Nov 15 43:57 fsflush root 5 0.2 0.0 0 0 ? S Nov 15 31:34 zpool-rpool ~ vs ~ $ ps aux | less USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND user 5403 7.8 9.463597966305772 ? O 16:42:27 48:03 /opt/VirtualBox/am user 1313 3.1 3.322088802159016 ? S Nov 15 696:41 /opt/VirtualBox/am user 3973 0.6 1.611236321072396 ? S 23:01:16 59:06 /opt/VirtualBox/am user 3974 0.5 1.711521601106504 ? S 23:01:19 60:21 /opt/VirtualBox/am user 997 0.5 0.110100053588 ? S Nov 15 31:48 /opt/VirtualBox/am user 5404 0.4 0.4257364227628 ? S 16:42:54 1:36 /opt/VirtualBox/am user 987 0.2 0.13010412384 ? S Nov 15 8:31 /opt/VirtualBox/am root 3 0.2 0.0 0 0 ? S Nov 15 43:56 fsflush root 5 0.2 0.0 0 0 ? S Nov 15 31:33 zpool-rpool ...

Typical Linux versions of ps will detect whether they are outputting to a terminal and truncate accordingly or if they are being piped and will automatically output in "arbitrarily wide" format. We can get the same functionality by adding the ww flag, as in:
ps auxww

From the man page for ps:

       w

           Uses a wide output format, that is, 132 columns rather than 80.  If
           the option letter is repeated, that is, -ww, this option uses arbi-
           trarily wide output. This information is used to decide how much of
           long commands to print.

           Note -

             The  wide  output option can be viewed only by a superuser or the
             user who owns the process.