qemu disk format autodetection

Lets say you use qemu like this:

sudo qemu-system-x86_64 \
    -hda system.raw	\
    -hda data.raw

In recent versions of qemu, you might see this error message:

WARNING: Image format was not specified for 'system.raw' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

To avoid the warning and remove the restriction on block 0 write operations, do what the error message suggests and explicitly set the disk file format to raw, ie:

sudo qemu-system-x86_64 \
    -drive file=system.raw,format=raw,index=0,media=disk \
    -drive file=data.raw,format=raw,index=1,media=disk

Happy qemuing!

qemu disk format autodetection

Choosing a domain registrar

For a long time I’ve registered my domains with godaddy. They were recommended by a friend back in the early 00’s, and they worked well enough, but there have always been a few gotchas that made using the service unpleasant:

  • Heavy UI, required more work than needed to get things done. To be fair, they have improved this over the time.
  • Constant up selling of things I don’t need.
  • Bait tactic of charging for a cheap one year domain, only to be hit with a high renewal fee the year after.

Still, for a relatively light domain registrar user like myself, those kinks weren’t really enough to justify the trouble of moving.

Until recently a couple of my domains came up for renewal, and I finally decided it was time to try something different. My first port of call was reddit.com, and as usual, they didn’t disappoint. The site is packed full of threads discussing domain registrars. Out of these, a gem came out.

https://domcomp.com/ is a service that allows you to compare different domain registrars. It’s pretty cool, if you are shopping for a domain registrar, you should definitely check it out.

I like the ability to compare prices for different .TLDs and purchase options (New, Renew, Transfer), while the discussions on reddit gave me a general idea of what kinds of features/service levels to expect.

After some research, I’ve narrowed it down to either namecheap or namesilo and will transfer a domain or two to try them out.

Choosing a domain registrar

Redirect STDOUT/STDERR

Sometimes you want to redirect your shell script’s STDOUT/STDERR to a log file. This is pretty simple to do, ie:

./myscript.sh &> output.log

But sometimes you want to do the redirect inside the script itself ( ie, maybe redirecting is conditional, or maybe the file name is conditional on something only the script knows about)


#!/bin/bash

exec >> output.log
exec 2>&1

>&2 echo "This was printed to stderr"
echo "This was printed to stdout"

Or if you want to write to both a file and the terminal:

#!/bin/bash

exec 1> >(tee -a output.log)
exec 2>&1

>&2 echo "This was printed to stderr"
echo "This was printed to stdout"

Stackoverflow has other ways to do it:

Redirect STDOUT/STDERR

How to rotate pdf pages permanently

You scan a document, only to find out it’s upside down. You could head to the scanner again and get some movement into your body, but you’re not sure which way to orient the page. Here’s how to do it from your computer instead.

  1. Open the file in Adobe Reader
  2. Choose the menu option View->Rotate View->Clockwise
  3. Choose the menu option File->Print
  4. Choose to print to a PDF driver.  This is native in Windows 10.
  5. If you don’t have a PDF printer driver, try this one: http://www.cutepdf.com/products/cutepdf/writer.asp
How to rotate pdf pages permanently

Wait for a file in bash

Sometimes your bash script needs to wait for a file to exist before proceeding. Here is one way to do it.

function wait_till_exists {
    file=$1
    timeout=${2:-60}
    counter=1
    until ls -l $file &> /dev/null
    do
        if [ $counter -gt $timeout ]; then
            return 1
        fi
        sleep 1
        ((counter++))
    done
    return 0
}

wait_for_file="/tmp/myfile"
if ! wait_till_exists "$wait_for_file" 5; then
    echo Timed out waiting for $wait_for_file to exist
exit 1
fi

echo "$wait_for_file found"
Wait for a file in bash

Capture stdout/stderr and exit code in perl

The modern recommended way:

use strict;
use warnings;

use Capture::Tiny qw/capture/;

my $cmd = "ls -l /tmp /non_existant_path";
my ($stdout, $stderr, $exitcode) = capture {
    system( $cmd );
};
$exitcode >>= 8;

print "STDOUT:\n", $stdout, "\n";
print "STDERR:\n", $stderr, "\n";
print "Exit code: $exitcode\n";

If Capture::Tiny is not available, IPC::Open3 might be ( but this has some caveats ).

use strict;
use warnings;

use IPC::Open3;

local(*MY_STDIN, *MY_STDOUT, *MY_STDERR);

my $cmd = "ls -l /tmp /nonexistant_path";

my $childpid = open3(*MY_STDIN, *MY_STDOUT, *MY_STDERR, $cmd);
my @outlines = <MY_STDOUT>;
my @errlines = <MY_STDERR>;  # XXX: block potential if massive
close MY_STDOUT;
close MY_STDERR;

print "STDOUT:\n", @outlines, "\n";
print "STDERR:\n", @errlines, "\n";
waitpid($childpid, 0);
my $exitcode = $? >> 8;
print "Exit code $exitcode\n";

Using open3 may fail if the output being thrown in stdout/stderr is too large and cannot be buffered. This can be made to work with IO::Select. See also http://docstore.mik.ua/orelly/perl2/prog/ch32_31.htm .

Capture stdout/stderr and exit code in perl

Speed up ssh logins

In this article we are going to explore two ways to speed up ssh logins. Reusing existing connections and disabling GSSAPI authentication.

The changes can be made on the ssh client side by adding the relevant config to the ssh config file at $HOME/.ssh/config .

Reusing existing connections

If you login to the same host using the same credentials multiple times (ie: different windows), one way to speed up connections is to share the same ssh connection. This means only the first connection actually needs to login. This also speeds things up if you open an ssh session, and later use scp/rsync to copy files across. Instead of opening a new connection, scp/rsync will reuse the existing ssh connection.

Host *
    ControlMaster auto
    ControlPath /tmp/ssh_mux_%h_%p_%r

Disabling GSSAPI Authentication

The GSSAPI is a IETF standard that provides an alternative to public/private keys as a means of authentication, suitable for doing strong encrypted authentication1. Sometimes ssh takes a long time figuring out if it needs to use GSSAPIAuthentication or not. If you don’t use it, disabling it can speed things up.

Host *
    GSSAPIAuthentication no

Footnotes

1. Using GSSAPI authentication at SLAC

Speed up ssh logins

Force cfengine client to download/execute new promises

CFEngine is a configuration management system, once upon a time famous for being used by facebook (who since migrated to chef).

CFEngine has a server component where configuration changes are written to, and a client side component that downloads and applies them. When testing around, it’s useful to be able to tell the client to download and execute these “promises” (as they are called in cfengine speak) immediately rather than waiting for the various daemons to kick in. This is how it can be done:

On the hub:

cf-agent -K -f failsafe.cf
cf-promises

On the client

rm -f /var/cfengine/inputs/cf_promises_validated
cf-agent -K -f failsafe.cf   # Download new promises
cf-agent -vK                     #execute new promises in verbose mode

Happy configuration managing !

Force cfengine client to download/execute new promises