Windows 7 Update troubles

While working on someone else's computer I noticed that svchost.exe -k netsvcs was completely hogging one CPU. Another symptom was that Windows Update was stuck at Checking for Updates... The first thing I tried was installing a new update client that allegedly fixes the problem, KB3050265. However the situation remained unchanged. I then learned about %WINDOR%\WindowsUpdate.log which contained the lines
2015-12-11 10:58:15:789 4440 3bc Handler   : Updates to install = 1
2015-12-11 10:58:46:178  964 944 Handler FATAL: UH: 0x80070490: EvaluateApplicability failed in CCbs::EvaluateApplicability
2015-12-11 10:58:46:880  964 944 Agent WARNING: Failed to evaluate Installed rule, updateId = {F9164872-3E27-4410-B4FC-038814549BB1}.105, hr = 80070490
2015-12-11 10:59:09:157  964 944 Handler FATAL: UH: 0x80070490: EvaluateApplicability failed in CCbs::EvaluateApplicability
Unfortunately it did not mention which update was corrupted. The error message took me to the System Updatze Readiness Tool via KB947821. Unfortunately, its error report was clean.
=================================
Checking System Update Readiness.
Binary Version 6.1.7601.22471
Package Version 26.0
2015-12-11 11:00

Checking Windows Servicing Packages

Checking Package Manifests and Catalogs

Checking Package Watchlist

Checking Component Watchlist

Checking Packages

Checking Component Store

Summary:
Seconds executed: 522
 No errors detected
Customer Experience report successfully uploaded.  Thank you for participating.  For more information, see the Microsoft Customer Experience Improvement Program on the Microsoft web site.

Restoring previous Perl library in El Capitan

Because I need an historic Perl 5.10 library for a program I use (GPSPhotoLinker), after upgrading to El Capitan I wanted to recover the old libraries in /System/Library/Perl/5.10 from Time Machine. Unlike the previous times I did this, Time Machine refused. So I restored the folder elsewhere and was in for a surprise.

$ sudo mv 5.10 /System/Library/Perl
mv: rename 5.10 to /System/Library/Perl/5.10: Operation not permitted

Hmmm, that's not supposed to happen. A bit of googling revealed the cause.

$ ls -lO /System/Library/Perl/
total 0
drwxr-xr-x  130 root  wheel  restricted 4420 23 Aug 02:09 5.16/
drwxr-xr-x  130 root  wheel  restricted 4420 23 Aug 02:08 5.18/
drwxr-xr-x    4 root  wheel  restricted  136 23 Aug 05:49 Extras/

There's a new restricted flag at work that's not documented in chflags(1).

Some more googling produced the video that explains it all, including how to disable the system integrity protection from Recovery as a developer.

Rather than jumping through all the hoops of installing perl in the expected location only to lose it again after the next OS upgrade, I decided to copy perl into the application bundle instead.

Playing with otool(1) and install_name_tool(1) let me change the binary's search path, but as soon as the perl code attempted to load a library it failed. In the end my solution was to create a small wrapper script to set PERL5LIB and DYLD_LIBRARY_PATH to the perl embedded into the application bundle.

$ cd /Applications/GPSPhotoLinker.app/Contents/MacOS
$ mv GPSPhotoLinker GPSPhotoLinker.bin
$ cat >GPSPhotoLinker <<EOF
#!/bin/sh

PERL5LIB=${PERL5LIB}:${0%/*}/../Perl/5.10 \
        DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:${0%/*}/../Perl/5.10/darwin-thread-multi-2level/CORE \
        "${0}.bin"
EOF
$ chmod a+x GPSPhotoLinker

Installing CM13 on Moto G from OS X

Motorola having officially abandoned my Moto G XT1032 (Falcon), it was time to install CyanogenMod again…

First, I downloaded the Android SDK. After unzipping the folder, I ran tools/android sdk and selected just the Platform Tools.

Stupidly, I then decided to install Motorola Device Manager thinking I needed its USB driver when in reality just turning on USB debugging on the phone would have sufficed. I spent the next 15 minutes poring over BOMs in /var/db/receipts, unloading LaunchDaemons and removing kexts to get rid of everything that had installed…

This allowed me to run platform-tools/adb devices and see my phone show up in the list. Now it was time to actually start trying to install CM13.

$ ./platform-tools/adb reboot bootloader
$ ./platform-tools/fastboot flash recovery twrp-2.8.6.0-xt1032.img
target reported max download size of 536870912 bytes
sending 'recovery' (8480 KB)...
OKAY [  0.287s]
writing 'recovery'...
(bootloader) Preflash validation failed
FAILED (remote failure)
finished. total time: 0.311s

The flashing failed as indicated by the »Preflash validation failed« message because my boot loader wasn't unlocked yet. Another indication is a line saying »Device is LOCKED« while in Fastboot. BE AWARE THAT THE FOLLOWING STEP WIPES THE ENTIRE DEVICE.

$ ./platform-tools/fastboot oem get_unlock_data
...
(bootloader) 0000000000000000#00000000000000
(bootloader) 00000000000000000000000000#0000
(bootloader) 0000000000000000000000000000000
(bootloader) 00000#0000000000000000000000000
(bootloader) 0000000
OKAY [  0.147s]
finished. total time: 0.147s
$ ./platform-tools/fastboot oem unlock 000000000000000000
...
(bootloader) Unlock code = 000000000000000000

(bootloader) Unlock completed! Wait to reboot

ERROR: usb_read failed with status e00002ed
FAILED (status read failed (No such file or directory))
finished. total time: 27.398s

Despite all the error messages, this seems to have worked.

Then, we can finally install TWRP recovery (which supports encryption in contrast to ClockWorkMod).

$ ./platform-tools/fastboot flash recovery twrp-2.8.6.0-xt1032.img
target reported max download size of 536870912 bytes
sending 'recovery' (8480 KB)...
OKAY [  0.287s]
writing 'recovery'...
OKAY [  0.300s]
finished. total time: 0.587s

It is now imperative to choose the »Recovery« option from the Fastboot menu using Volume Down and selecting it using Volume Up. This is what prevents the recovery partition we have just flashed from being overwritten during the next boot.

The first thing I did was to backup the PDS partition because when it is lost or damaged the phone's IMEI is lost. We don't want that. In order for the following commands to work, root access via ADB must be enabled in the developer options.

$ ./platform-tools/adb root
restarting adbd as root
$ ./platform-tools/adb shell
# dd if=/dev/block/platform/msm_sdcc.1/by-name/pds of=/sdcard/pds.img
# ^D
$ ./platform-tools/adb pull -p /sdcard/pds.img
Transferring: 3145728/3145728 (100%)
5569 KB/s (3145728 bytes in 0.551s)

Afterwards, it's necessary to remove the warning message from the boot logo. In order to do that, you need to patch the partition containing the boot logos.

$ ./platform-tools/adb root
restarting adbd as root
$ ./platform-tools/adb shell
# dd if=/dev/block/platform/msm_sdcc.1/by-name/logo of=/sdcard/logo.img

If you look at the header of logo.img, you will notice there are four images named logo_boot, logo_battery, logo_unlocked and logo_charge. The names are followed by the offset and length of the image file. In my case, to replace the logo_unlocked image by logo_boot, I used the following command.

$ ./platform-tools/adb root
restarting adbd as root
$ ./platform-tools/adb shell
# dd bs=1 count=8 of=/dev/block/platform/msm_sdcc.1/by-name/logo seek=101 if=/sdcard/logo.img skip=37

Unfortunately, device encryption doesn't work yet with TWRP v2.8.6.0 and I'm not the only one having the problem. An unofficial version of TWRP v2.6.7.0 at least asks for the password, but fails to decrypt the partition.

There is also currently a settings bug if you try to set a PIN for the screen lock. The PIN doesn't actually get set, but when you try to go back into settings to change the screen lock, the Settings app asks for the PIN and then crashes. I didn't find the password.key file in /system/data or anywhere else, so I wiped the data partition to get rid of the PIN.

However, Facebook Messenger keeps crashing and GPS isn't working for me yet, so maybe 20151108 is a little too early for Marshmallow on Falcon. Here are the commands to get back to stock. The system partition can only be written using the moto-fastboot tool from Motorola. When flashed with Android's fastboot, the boot never gets to the animation.

$ fastboot oem lock begin
...
(bootloader) Ready to flash signed images

OKAY [  2.839s]
finished. total time: 2.839s
$ fastboot flash partition gpt.bin
$ fastboot flash motoboot motoboot.img
$ fastboot flash logo logo.bin
$ fastboot flash boot boot.img
$ fastboot flash recovery recovery.img
$ moto-fastboot-osx64 flash system system.img_sparsechunk.0
$ moto-fastboot-osx64 flash system system.img_sparsechunk.1
$ moto-fastboot-osx64 flash system system.img_sparsechunk.2
$ moto-fastboot-osx64 flash system system.img_sparsechunk.3
$ fastboot flash modem NON-HLOS.bin
$ fastboot erase modemst1
$ fastboot erase modemst2
$ fastboot flash fsg fsg.mbn
$ fastboot erase cache
$ fastboot erase userdata
$ fastboot oem lock

The last command might prompt an error to be shown in Fastboot unless all partitions are signed. When it succeeds, it reboots the device.

I only had to do logo, boot, recovery and system which was enough to make the oem lock command succeed.

SocketTimeoutException when sending with K-9 Mail via smtp.gmail.com

Today I noticed an odd problem: K-9 Mail reported SocketTimeoutException when sending mails from an account configured to use smtp.gmail.com. When I set the server to smtp.googlemail.com without changing any of the credentials, it started working again. I didn't dig deeper, but I assume it might have something to do with certificates. It's still odd though because smtp.gmail.com and smtp.googlemail.com resolve to different addresses.

TimeMachine and ReadyNAS OS

Every once in a while, I get a message from Time Machine telling me that it cannot back up because »the sparse bundle is in use«.

Restarting the NAS fixes the problem, but that's a bit hard-core. I've found that it starts working again if I disable and re-enable the AFP service through the web interface. An even easier way to do this is to ssh into the NAS and say

# systemctl restart netatalk

Day Numbers

For reference, this is relatively canonical code for calculating day numbers.

# turn a March 1600 based date into a day number
def daynum(y, m, d):
	# years consist of repeating 5-month periods of 153 days each
	m = divmod(m, 5)
	return 365 * y + y / 4 - y / 100 + y / 400 + \
		153 * m[0] + \
		31 * m[1] - m[1] / 2 + \
		d

# turn a number into a March 1600 based date
def numdate(dnum):
	# over-estimate year
	year = dnum / 365
	day = dnum - daynum(year, 0, 0)
	while day < 0:
		year = year - 1
		day = dnum - daynum(year, 0, 0)

	# over-estimate month
	month = day / 30
	day = dnum - daynum(year, month, 0)
	while day < 0:
		month = month - 1
		day = dnum - daynum(year, month, 0)

	return year, month, day

# turn a date into a March 1600 based date
def to1600(year, month, day):
	month = divmod(month - 3, 12)
	return year + month[0] - 1600, month[1], day - 1

# turn a March 1600 based date into a real date
def from1600(y, m, d):
	m = divmod(2 + m, 12)
	return 1600 + y + m[0], 1 + m[1], 1 + d

def from_date_str(date):
	return daynum(*to1600(*map(int, date.split("-"))))

def to_date_str(d):
	return "%04d-%02d-%02d" % from1600(*numdate(d))

How PHP is broken

Some objective reasons.

  • PHP's handling of variables violates a fundamental programming paradigm. I call it the »substitution principle«. It stipulates for an operation op and an expression a such that op(a) is a valid expression, then op(x) should also be a valid expression for x = a.

    There are very many cases where this is not the case in PHP. For example, a property of an object may be accessed indirectly using the notation $this->$prop when the variable $prop is a string specifying a property's name. However, the substitution principle is violated because $this->'propertyname' is not a valid expression. Instead, a bare word must be used to access the property directly.

    This also happens when instantiating a class: an unknown class can be instantiated using the expression new $class when the variable $class contains the class name. But substituting its value yields the invalid expression new 'ClassName'.

    Another violation of the substitution principle is that the following won't work:

    $func = 'array';
    $func();

    Even more bizarrely, sometimes it's impossible to use a value in any other way than a local variable. For example, if $this->someClass contains a class name, its value has to be copied to a local variable in order to access static properties or methods of the referenced class.

    $className = $this->className;
    $className::s_property;
    $className::makeInstance();
    The syntax provides no way to use $this->className directly in these contexts, violating the substitution principle.

  • Another violation of the substitution principle arises with function literals: while it's possible to pass a function literal as a callback, for example in
    array_map(function ($a) { return 1 + $a; }, $ary)
    the same is not true in other places where expressions are allowed:
    class C {
      static private $l = function () { return 'hello'; };
    }
    is a parse error.
  • One of PHP's important use-cases is presenting data retrieved from a database. That considered, I was dismayed how difficult it is to use prepared statements or database transactions with this language. I know, both MySQLi and PDO now provide them, but what in the world has taken them so bloody long?!

    I mean, if it's taken the developers of a database connection toolkit decades of coming up with quoting functions after quoting function ( addslashes, addcslashes, quotemeta, magic_quotes_runtime, mysql_escape_string, mysql_real_escape_string, mysqli_real_escape_string, PDO::quote) without realising that in the presence of NO_BACKSLASH_ESCAPES and ANSI_QUOTES the client cannot know how to correctly quote arbitrary data, how can I trust their software not to randomly delete my hard drive on a wrong keypress? Or indeed, why doesn't this happen all the time!?

  • Given that PHP is used for presenting database contents on the web, you'd think it's a given that various character encodings and their conversion are well supported. I was disappointed when I found out that the function mb_internal_encoding() returns the string 'UTF-8' if that's selected as the internal encoding, but the function tidy_parse_string() requires the string 'utf8' in order to correctly parse a UTF-8 string! This virtually requires hard-coding a specific character encoding, making it nearly impossible to write code that deals correctly with various character encodings in different situations. In comparison, Java and Perl provide full conversion toolsets that seamlessly support any encoding situations imaginable.
  • There's such a thing as »catchable fatal errors« in PHP. When I first heard this I didn't think much about it, assuming that if the language or interpreter encountered some error, an exception would be thrown and my code could deal with it appropriately by displaying some message in a suitable place (log file or user interface).

    But that's not what a catchable fatal error is. What actually happens is that PHP prints a message into your web page (thereby usually causing Quirks Mode) and the program continues.

    In order to make these errors actually »catchable«, you must first install a custom error handler:

    set_error_handler(function ($severity, $message, $file, $line) {
        if (!(error_reporting() & $severity)) {
            return;
        }
        throw new \ErrorException($message, 0, $severity, $file, $line);
    });
    PHP even helpfully provides an exception subclass and sample code in the manual — then why not report all errors like this in the first place?

    On the upside, this has also finally allowed me to get rid of that annoying message from file_get_contents() if a file could not be opened. I had given up on being able to prevent this from breaking my user interface until I installed the above error handler, which now lets me catch and deal with the exception as expected.

  • PHP's object-oriented support has been bolted on in a big rush. This becomes apparent, for example, when using static methods. Incredibly, PHP treats such methods, which are also known as »class methods«, as being inherited by subclasses rather than merely visible to them. This is an important distinction: since every call of a static method has to be qualified with a class name, it can be statically determined at compile time which method will be called. Hence the name, d'uh. There is no run-time polymorphism involved due to inheritance.

    And yet, when compiling code like the following,

    class A {
        static public function f() {}
    }
    
    class B extends A {
        static public function f($a) {}
    }

    PHP's compiler generates this error:

    Declaration of B::f() should be compatible with A::f()
  • It's really difficult to find good advice on how to use PHP well. Even a published book (Peter MacIntyre's »PHP — The Good Parts« at O'Reilly) advocates code like the following without any warning:

    $sql = "INSERT INTO guests (fname, lname, comments)
    	VALUES ('$_POST[fname]', '$_POST[lname]', '$_POST[comments]')";
    $query = mysql_query($sql);
    At the end of this quite horrifying chapter is a suggestion to also read the chapter on security, which makes a passing mention of filtering user data for semicolons and encrypting passwords using SHA0 or SHA1, without giving any reasons or code examples… That something like this could be published in 2010 makes me speechless.

  • PHP's ternary operator is very limited. While every other language that implements this operator (C, C++, Java, …) evaluates it right-to-left, PHP evaluates it left-to-right. Why is this wrong? Let's assume we have a statement
    a ? b : c ? d : e
    When this is evaluated right-to-left, there are three possible outcomes b, d or e, as becomes obvious when adding implied parentheses:
    a ? b : (c ? d : e)
    The way PHP evaluates from left to right, the only outcomes are d or e:
    (a ? b : c) ? d : e
    This ordering only makes it possible to provide elaborate conditions, but breaks multiple outcomes. This is relevant when doing range tests which cannot otherwise be written as an expression but require resorting to if-statements:
      high < value    ? something
    : middle < value  ? other
    : low < value     ? allgood
                      : dflt
  • Functions and even classes are no objects in their own right. For example, in order to call a function with a list of arguments from an array, its name has to be known and passed to the function call_user_func_array:

    call_user_func_array('array_merge', $arrays)

    This can be done much more elegantly in JavaScript using Function.prototype.apply:

    Array.prototype.concat.apply([], arrays)

  • While on the subject of array_merge(), it doesn't return an empty array when called without arguments like it obviously should do. So you end up with this idiom:

    $arrays ? call_user_func_array('array_merge', $arrays) : []
  • It's seriously scary how many situations call for the name of a function or class in PHP. For example, in order to access static methods or properties of an unknown class:

    $model = '\\Model\\Employee';
    $model::load(123);

    It really feels a lot like programming Tcl.

  • Getting the last element of an array is something that many languages have special tools for, be they vector.back() or array[-1]. PHP lacks such tools and the idiomatic solution is to use the array's internal iterator <shudder>. There's no way to avoid the side-effect of moving that iterator, since PHP provides no way to save and restore the iterator's position.

    $last = end($array);
    reset($array);
    This situation is described very succinctly by Keeth's comment on stackoverflow: »Only in PHP would the idiomatic inspection of an array element cause a side effect.«

    One frequently proposed way to avoid the side-effect is to say

    $last = end(array_values($array))
    which conceptually creates a copy of the array. Even if that doesn't actually happen because it's implemented lazily, that's simply too much of an intellectual overload to justify calling »idiomatic«.

  • The sign of PHP's »modulus« operator % is that of its first operand (the dividend in the integral division) rather than the second operand (the divisor). The former choice of sign corresponds with rounding towards zero in the associated integral division (commonly known as »truncation«), while the latter corresponds to rounding towards negative infinity (commonly known as »floor«).

    To see this, recall that the relation between an integral division div and its modulus mod is given by the equation

    p = (p div q) * q + p mod q
    for two integers p and q.

    Sign of »p mod q« when rounding towards negative infinity, i.e. »p div q = floor(p / q)«. The sign of the result of »mod« corresponds to that of the divisor q.

    0 < q floor(p / q) ≤ p / q floor(p / q) * qp 0 ≤ p mod q
    q < 0 p ≤ floor(p / q) * q p mod q ≤ 0

    In comparison, rounding towards zero (or »truncation«) produces a slightly more complex case table.

    Sign of »p mod q« when truncating, i.e. »p div q = (int)(p / q)«. The sign of the result of »mod« corresponds to that of the dividend p.

    0 ≤ p 0 < q (int)(p / q) ≤ p / q (int)(p / q) * qp 0 ≤ p mod q
    q < 0 p / q ≤ (int)(p / q)
    p < 0 0 < q p / q ≤ (int)(p / q) p ≤ (int)(p / q) * q p mod q ≤ 0
    q < 0 (int)(p / q) ≤ p / q

    In addition to simpler formulae, floor division has the attractive property that for a given q, each number n is the result of dividing numbers i ∈ [qn, qn + q) from an interval of length q. This is important when working with time periods such as hours, days and weeks, for example. In contrast, division and truncation means that 0 is the result of dividing numbers i ∈ (−q, q) from a much larger interval of length 2q − 1.

    PHP's preference of »truncation« makes writing algorithms that work correctly with negative integers more difficult than it needs to be.

    /*
     * Polyfill for Python's divmod: return two numbers [ x, y ] where
     * x = floor(p / q) and p = x * q + y. As a result, abs(y) < abs(q).
     */
    function divmod($p, $q) {
    	$x = floor($p / $q);
    	return [ $x, $p - $x * $q ];
    }
  • The standard library function to create a timestamp expects its arguments in the order $month, $day, $year.

    int mktime(int $hour, int $minute, int $second,
    	   int $month, int $day, int $year, int $is_dst)

    The standard library function to calculate the Julian day number for a Gregorian date has the signature

    int gregoriantojd(int $month, int $day, int $year)

    Wow, this is messed up!

The cause for the low quality of this language and its implementation is its community: it quickly becomes obvious that the developers are not professionals, as illustrated by the hilarious tale that started with

int size;
…
if (size > INT_MAX) …

As I come to know PHP more, my feelings are summed up precisely by Eevee in this beautiful analogy:

I can’t even say what’s wrong with PHP, because— okay. Imagine you have uh, a toolbox. A set of tools. Looks okay, standard stuff in there.

You pull out a screwdriver, and you see it’s one of those weird tri-headed things. Okay, well, that’s not very useful to you, but you guess it comes in handy sometimes.

You pull out the hammer, but to your dismay, it has the claw part on both sides. Still serviceable though, I mean, you can hit nails with the middle of the head holding it sideways.

You pull out the pliers, but they don’t have those serrated surfaces; it’s flat and smooth. That’s less useful, but it still turns bolts well enough, so whatever.

And on you go. Everything in the box is kind of weird and quirky, but maybe not enough to make it completely worthless. And there’s no clear problem with the set as a whole; it still has all the tools.

Now imagine you meet millions of carpenters using this toolbox who tell you “well hey what’s the problem with these tools? They’re all I’ve ever used and they work fine!” And the carpenters show you the houses they’ve built, where every room is a pentagon and the roof is upside-down. And you knock on the front door and it just collapses inwards and they all yell at you for breaking their door.

That’s what’s wrong with PHP.

Recommended reading: I got much more useful advice from this post than from the above-mentioned book »The Good Parts«.

My analogy for knocking at the door is trying to find a way to use prepared statements or database transactions with this language.

Recovering backups from differently formatted CD-ROMs

Over the past 20 years, I have built up a collection of backup CD-ROMs. There is quite large variation in their formats due to the different platforms I was using at various times and the development of the standards. I have plain ISO-9660 media, some using Unix Rock Ridge extensions, and others using Windows Joliet extensions. Most of my media are multi-session because I thought it might be useful to be able to add some data later on.

Getting data from these media isn't actually very difficult in itself. As long as there are no read errors, the file contents can be retrieved using any system.

However, the problem starts when one would like to restore correct file names, directory hierarchies and properties such as ownership, read/write/execute permissions or hard links.

Note for the future: even if it's slightly less convenient, it's actually a good idea to put archives on CDs (or backups in general)... much easier to read later on than some silly file system that you have to find the right mount options for.

Image creation

After a day of fiddling, here's a workflow that I settled on:

I use Harald Bögeholz's H2cdimage to read the contents of the CD into an .ISO file. This tool can be run on multiple machines using different drives which increases the chances of recovering data from defective media.

Unfortunately, H2cdimage doesn't create a .CUE file with the track layout information which appears to be needed to correctly mount multi-session CDs. I use IsoBuster which creates beautifully annotated .CUE files that also show the linear block address (LBA) of sessions which will be needed for mounting.

FILE "CD.iso" BINARY

REM ORIGINAL MEDIA-TYPE: CD

  REM SESSION 01        ; Not supported by other applications (*)
    TRACK 01 MODE2/2352
      INDEX 01 00:00:00
      REM MSF: 00:00:00 = LBA: 0

  REM RUN-OUT  18:12:70 ; Not supported by other applications (*)
  REM LEAD-OUT 18:12:72 ; Not supported by other applications (*)
  REM SESSION 02        ; Not supported by other applications (*)
    TRACK 02 MODE2/2352
      INDEX 01 20:44:72
      REM MSF: 20:44:72 = LBA: 93372

REM (*) SESSION directives are unfortunately not properly supported
REM     'out there'.  IsoBuster however supports them !

IsoBuster also provides »managed image files« which are based on the same idea of using different hardware to read as much data as possible from damaged discs, but I have not tried that.

File extraction

In order to extract the files, here are the things I've tried:

7zip

The simplest and most pleasant way is to just use

$ 7z x CD.iso
This works great for plain ISO9660 files and even deals very well with Joliet extensions, but when I tried it on a multi-session Rock Ridge disc it only saw the first session, so I'm not sure whether it would have dealt correctly with the RR_MOVED folder, for example.

IsoBuster

IsoBuster can extract data from any session and even allows specifying the filesystem character set which can be a problem for incorrectly burned media. For example, one of my ISO9660 discs uses the ancient DOS codepage 437, and one of my Rock Ridge discs uses ISO8859-1 filenames. I have no idea whether this is correct or not (since I burned it myself), but the names definitely have to be converted before being stored on a modern filesystem.

However, if there are any hard linked files on the medium, IsoBuster will extract them multiple times and the information which files are identical is lost in the process. Additionally, the resulting folder might be much bigger than the original CD image.

Also, IsoBuster is unable to place files that were moved to the RR_MOVED folder back into the correct place in the hierarchy. As a reminder, here's the deal with RR_MOVED:

If mkisofs is creating a filesystem image with Rock Ridge attributes and the directory nesting level of the source directory tree is too much for ISO-9660, mkisofs will do deep directory relocation. This results in a directory called RR_MOVED in the root directory of the CD. You cannot avoid this directory in the directory tree that is visible with ISO-9660 but it it automatically hidden in the Rock Ridge tree.

Lastly, since it runs on Windows IsoBuster cannot extract some files from Rock Ridge discs. For example, I have one file that contains a ':' colon (created by CVSup back in the day) which Windows does not allow in file names.

mount

The only way I found to correctly restore file hierarchies from Rock Ridge discs is to use Unix. You could even say that's fair enough since those discs were created on Unix in the first place, and contain Unix files.

But there's also an advantage over the 7z or IsoBuster for pure ISO9660 filesystems: filenames are presented in lower-case.

If you need to read later sessions of a multi-session disc it's not sufficient to just mount -o loop CD.iso /mnt/iso it. Apparently, there is information on the disc that allows mount to pick the latest session as advertised, but it's missing from the image file and mount itself does not understand .CUE files and needs a little help. This is where the LBA from IsoBuster comes into play:

# mount -o loop,sbsector=93372 CD.iso /mnt/iso

Now you can use

$ cd /mnt/iso
$ pax -r -w -p p . /.../dest
to copy the files to the new backup medium.

Unfortunately, Linux's mount only supports character set conversions via the iocharset option for Joliet extensions. If you're stuck with a Rock Ridge or ISO9660 disc in a different encoding, you're going to need a script to rename those files manually after copying

#!/bin/sh

# uncomment for debugging
#ECHO=echo

if [ $# -lt 2 ]
then
	cat >&2 <<EOF
usage: $0 from to <file> ...

	Convert file name encodings. Use "iconv -l" for supported encodings.
EOF
	exit 1
fi

from="$1"
shift
to="$1"
shift

for p in "$@"
do
	f="${p##*/}"
	d="${p%"$f"}"
	t=$(echo "$f" | iconv -f "$from" -t "$to")
	if [ "$f" != "$t" ]
	then
		echo "$d$f -> $t"
		$ECHO mv -i "$d$f" "$d$t"
	fi
done

This can be run on the entire tree using, for example

$ find . -depth -exec /.../fconv cp437 utf8 {} \+
The -depth primitive is important in case directories are renamed, after which subsequent commands won't find the remaining files to rename.

As a final step, don't forget to mark the extracted images and files read-only just to prevent stupid mistakes.

$ chmod -R a-w *

Removing duplicates

When working with backups it's common to get some duplication of files. To deal with this, I looked at the fslint suite (which includes a GUI), rmlint which is extremely complex, fdupes which didn't seem very encouraging, duff which seemed promising but was a bit slower than rdfind that I eventually settled on.

My requirement was to find duplicate files in two (or more) given directories and delete duplicates in all but one of them, which was supposed to remain untouched. That is, even if the first directory contained duplicates, I didn't want to remove them. While rmlint supports this out of the box using »tagging,« it wasn't available on my platform and so I wrote a small script to process the rdfind output.

#!/bin/sh

# uncomment for debugging
#ECHO=echo

if [ $# -lt 2 ]
then
	cat >&2 <<EOF
usage: $0 <dir> ...

	Find duplicate files in the named directories. Files from the first
	named directory are never deleted even if they are duplicates.
EOF
	exit 1
fi

rdfind -checksum sha1 -outputname dedup.$$ "$@"
cat dedup.$$ | \
	grep -v '^#' | \
	grep -v '^DUPTYPE_FIRST_OCCURRENCE' | \
	cut -d' ' -f8- | \
	grep -v "^$1/" | \
while read f
do
	$ECHO rm "$f"
done
rm dedup.$$

C++14 best practices

I just watched Herb Sutter's CppCon 14 address and felt a need to make a note to myself of a couple of things:

  • Call functions with raw references or pointers to entities that live at least as long as your scope.
  • Don't overuse pass-by-value even for parameters you're going to copy unconditionally — a combination of copy and move assignment perform better in important cases because an unconditionally copied parameter cannot take advantage of existing capacity at the final destination. Prefer
        void set_name(std::string const& name);
        void set_name(std::string&& name);
    
    to this
        void set_name(std::string name);
    
  • Assuming you have a class hierarchy C0 <| C1 <| … and corresponding factory template functions. If you'd like to keep an arbitrary instance around for the duration of a scope while erasing its type, you might think this requires heap allocation. But it's not true: a reference to the base type will do just fine!
    {
        C0 const& c0 = cFactory(…);
        /* … */
    }
    The result of the factory is stored in a temporary and its lifetime is determined by the lifetime of the reference. The idea came from Andrei Alexandrescu's ScopeGuard.

Mercurial EolExtension is broken beyond repair...

I have an admittedly complex setup with EOL conversion because development is on a Windows host where some programs require CRLFs.

But even so, I've never had any trouble with git in similar circumstances, and this happens multiple times per week: at some point during MQ patch refreshment, EolExtension gets confused and reports »abort: inconsistent newline style in ...«. Subsequently, hg stat takes forever (once) and hg diff shows all files as added with Unix line endings, even though the actual physical files have Windows line endings just like before.

While trying to get all the files to show up as unmodified again, the following happens.

And the real fun part: because the files are actually still using the correct line endings on disk, hg revert is completely ineffective! How is that even possible?!

Because it's a really large repo, I don't really fancy going via hg up -C null since that takes forever. But in the end it was the only way to remedy the »situation«.

And don't even get me started on the observed fact that hg stat does caching – why else would it take so long only the first time after EolExtension screws up? I'll never use Mercurial on any project if I get a choice, and that's especially true on Windows where according to some it works so much better than git.

Doing some more investigating, it turns out that the problematic files show up with missing data in hg debugstate:

n   0         -1 unset               browser/components/places/tests/unit/test_txnGUIDs.js
n   0         -1 unset               browser/components/search/test/browser_415700.js
n   0         -1 unset               browser/components/sessionstore/test/browser/browser_346337_sample.html

This suggests that running hg debugrebuildstate -r tip might help to fix the problem. After this, hg status takes a long time again, but without setting the correct file size.

n 644         -1 1970-01-01 01:00:00 browser/components/places/tests/unit/test_txnGUIDs.js
n 644         -1 1970-01-01 01:00:00 browser/components/search/test/browser_415700.js
n 644         -1 1970-01-01 01:00:00 browser/components/sessionstore/test/browser/browser_346337_sample.html

After going through the lengthy hg co -C null and hg update -C tip exercise, the state of the above files looks like this:

n 644       4030 2016-07-16 02:48:59 browser/components/places/tests/unit/test_txnGUIDs.js
n 644       3729 2016-07-16 02:49:00 browser/components/search/test/browser_415700.js
n 644       1049 2016-07-16 02:49:00 browser/components/sessionstore/test/browser/browser_346337_sample.html

Reloading services and kexts using AppleScript

This weekend, I had to try to convince a slightly reluctant (no-name) bluetooth mouse to work on an Apple laptop. Especially after a sleep-wakeup cycle it tends to stop reacting and my theory is that it confuses OS X's bluetooth driver to the point where it stops functioning.

As a small test, I decided to try to unload both the bluetooth daemon blued and the IOBluetoothHIDDriver when the problem occurs and see if that fixes it. For this purpose, I came up with the following one-click experiment, my very first AppleScript.

set bluedController to serviceController for "com.apple.blued" given defaults:"/Library/Preferences/com.apple.Bluetooth", preference:"ControllerPowerState"

set driverController to kextController for "com.apple.driver.IOBluetoothHIDDriver"


restart({bluedController, driverController})


-- Shutdown a list of services, waiting for each to stop, and restart them in reverse order

on restart(controllers)

repeat with controller in controllers

repeat

tell controller

shutdown()

if not isRunning() then exit repeat

end tell

display dialog "Waiting for " & controller's name & " to stop" giving up after 1

end repeat

end repeat

repeat with controller in reverse of controllers

repeat

tell controller

start()

if isRunning() then exit repeat

end tell

display dialog "Waiting for " & controller's name & " to start" giving up after 1

end repeat

end repeat

end restart


-- Return a script object that controls a launch service using its associated preference.

on serviceController for serviceLabel given defaults:prefsFile, preference:prefName

script controller

property name : "service " & last item of (wordList of serviceLabel at ".")

on launchCtl(command)

-- launchctl only knows about blued if run as admin

do shell script "launchctl " & ¬

quoted form of command & " " & ¬

quoted form of serviceLabel ¬

with administrator privileges

end launchCtl

on setPref(value)

-- turn off the service's preference

do shell script "defaults write " & ¬

quoted form of prefsFile & " " & ¬

quoted form of prefName & " " & ¬

quoted form of (value as text) ¬

with administrator privileges

end setPref

on isRunning()

return launchCtl("list") contains "\"PID\" = "

end isRunning

on shutdown()

setPref(0)

launchCtl("stop")

end shutdown

on start()

setPref(1)

launchCtl("start")

end start

end script

return controller

end serviceController


-- Return a script object that loads and unloads a kernel extension.

on kextController for bundleIdentifier

script controller

property name : "kernel extension " & last item of (wordList of bundleIdentifier at ".")

on isRunning()

do shell script "kextstat -l -b " & ¬

quoted form of bundleIdentifier

return the result contains bundleIdentifier

end isRunning

on shutdown()

try

do shell script "kextunload -b " & ¬

quoted form of bundleIdentifier ¬

with administrator privileges

end try

end shutdown

on start()

do shell script "kextload -b " & ¬

quoted form of bundleIdentifier ¬

with administrator privileges

end start

end script

return controller

end kextController


on wordList of theWords at delimiters

set oldDelimiters to text item delimiters

set text item delimiters to delimiters

try

set res to text items of theWords

set text item delimiters to oldDelimiters

return res

on error m number n

set text item delimiters to oldDelimiters

error m number n

end try

end wordList

Normalise line endings in Git repository without external tools

As it happens, I recently »inherited« a Git repository. It turned out that all files had been submitted with Windows line endings, also known as CRLF. Since the core.autocrlf wasn't set, they were stored as such in the repository.

Since it was a private repository that wasn't published anywhere (that I knew of or cared about), I decided to rewrite its history as a first step. I wanted to use Git's own conversion rather than relying on external tools such as fromdos, so I activated it using

$ git config core.autocrlf input
Then I wrote a small shell script called readd.sh that re-adds the current commit and triggers Git's line-ending conversion:
#!/bin/sh

git rm --cached -r .
git add -f .
The -f option to add ensures that even files that are in .gitignore are re-added. Yes, they even managed to submit ignored files into their repository...

As a last step, I used filter-branch to process every commit and apply my script.

$ git filter-branch --tree-filter "$HOME/readd.sh" -- --all

Since it was a very small repository, the entire process took only a matter of seconds, and I was left with a clean state with normalised line endings.

If I ruled the world...

To whom it may concern, particularly any political party that would like my vote:

  • Religious taxation and religious education at public schools is abolished to be replaced by philosophy. The subject currently called »maths« is renamed »arithmetic«.
  • Recreational drugs are legalised. The gained enforcement capacity is used to police speeding and parking offences and any other traffic violation that compromises the safety of others. Every incident is fined at a flat rate of €200.
  • The proceeds of traffic fines are used to actually make roads cycling friendly.
  • Tax relief and subsidies for commuting is scrapped. The funds are used to make public transport free for everyone to use.
  • Privately-owned compulsory health insurance is abolished. Doctors and hospitals are made free for everyone to use (administrative fees of €5-€10 per use apply), the health system is financed through income tax. Whether the income was gained through employment, capital gains or otherwise makes no difference and there are no upper or lower limits to each individual's contributions. Tax relief and subsidies for private health insurance is scrapped.
  • Tax relief and subsidies for private pension plans are scrapped. The public pension system based on annual turnover is adjusted such that 40 years' worth of contributions yield an equivalent pension income. Pension contributions are subsumed by income tax irrespective of whether the income was gained through employment, capital gains or otherwise.
  • Unemployment insurance and incapacity benefits are subsumed by income tax irrespective of whether the income was gained through employment, capital gains or otherwise.
  • Tax relief and subsidies for real estate are abolished, personal use excepted. Income from rent is treated like any other income.

As you have probably guessed I'm not an economist. But I figure that just the simplifications from these changes would make for a much more pleasant life...

Yosemite with audio on DH87RL with Realtek ALC892

The following is now obsolete thanks to AppleALC which only requires the kernel parameter :Boot:Arguments alcid=1.


In order to get audio functionality in Yosemite on my DH87RL motherboard, I follow toleda's method using Clover. The exact method is encoded as a shell script with the required files in a couple of repositories.

The first task is to find out the current layout-id using the command

$ ioreg -nHDEF -r -w0
There should be a section called "HDEF@1B" with a key "layout-id" showing a value.

In my case, the value was 0 which is not a good sign. Thankfully, Clover can patch it by adding the string value :Devices:Audio:Inject with the desired value "1".

After rebooting, the command

$ ioreg -rxnIOHDACodecDevice
identifies my audio chip as vendor and device 0x10ec0892 (Realtek ALC892) and revision 0x100302 (current).

Now I downloaded and unpacked config-audio_cloverALC.plist.zip and copied the two relevant entries from :KernelAndKextPatches:KextsToPatch to my config.plist: Item 0 with comment »10.9-10.10-AppleHDA/Resources/xml>zml« (the purpose of which will become clear in a little while) and Item 4, »10.9-10.10-AppleHDA/Realtek ALC892«.

Then I downloaded and unpacked realtekALC.kext and installed it into my EFI/CLOVER/OEM/DH87RL/kexts/10.10 folder. Finally, I renamed the files Platforms.xml.zlib and layout[123].xml.zlib from 892.zip to *.zml.zlib, installed them in my /System/Library/Extensions/AppleHDA.kext/Contents/Resources folder and made them owned by root:wheel.

Next, I ensured that the :SystemParameters:InjectKexts was enabled in my config.plist ("Detect") and that :Boot:Arguments contained "kext-dev-mode=1" and rebooted with rebuilding kext cache.

An active remote client has recently sent requests to this machine

Trying to get Windows 7 to sleep is notoriously difficult. Like many, I was in the situation that I got the following message:

C:\Windows\system32>powercfg /requests
DISPLAY:
None.

SYSTEM:
[DRIVER] \FileSystem\srvnet
An active remote client has recently sent requests to this machine.

AWAYMODE:
None.

Needless to say, the Management Console's fsmgmt.msc was silent about any active sessions, so the message seems to be a lie.

In order to make the message disappear, I ended up creating a homegroup, activating media streaming and waiting for the whole process to finish (there is a message in the control panel while it is still busy »sharing«). Afterwards, I simlpy »left« the homegroup again, and voilá, the srvnet request disappeared.

There is also a setting in the advanced power options about letting the computer sleep while sharing media, but that seems to have no effect either way.

Running Yosemite on UEFI hardware

I use the Clover boot loader in a UEFI-only installation on the ESP with the »BootCamp« theme. The only UEFI driver I need is OsxAptioFix2Drv-64 -- without it I get ???.

The best source for up-to-date kexts I've found is to download the current MultiBeast installer. The »Contents/Resources« folder of »MultiBeast.app« contains packages which can be expanded using

$ pkgutil --expand FakeSMC-v6.14.1364.pkg p
$ pax -rzf p/Payload

For my Intel DH87RL, I drop FakeSMC.kext and AppleIntelE1000e.kext into the corresponding folder OEM/DH87RL/kexts/10.10 on the ESP.

After installing new kexts, a reboot »without caches« must be performed by selecting the boot entry with space bar in Clover and selecting the corresponding entry.

My configuration file OEM/DH87RL/config.plist contains

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ACPI</key>
 <dict>
  <key>SSDT</key>
  <dict>
   <key>EnableC6</key>
   <true/>
   <key>Generate</key>
   <dict>
    <key>CStates</key>
    <true/>
    <key>PStates</key>
    <true/>
   </dict>
  </dict>
 </dict>
 <key>Boot</key>
 <dict>
  <key>DefaultVolume</key>
  <string>LastBootedVolume</string>
  <key>Timeout</key>
  <integer>5</integer>
  <key>Log</key>
  <false/>
 </dict>
 <key>GUI</key>
 <dict>
  <key>Scan</key>
  <true/>
  <key>Custom</key>
  <dict>
   <key>Legacy</key>
   <array>
    <dict>
     <key>Type</key>
     <string>Windows</string>
     <key>Hidden</key>
     <true/>
    </dict>
   </array>
  </dict>
 </dict>
 <key>KernelAndKextPatches</key>
 <dict>
  <key>Debug</key>
  <false/>
  <key>KernelPm</key>
  <true/>
 </dict>
 <key>RtVariables</key>
 <dict>
  <key>MountEFI</key>
  <false/>
 </dict>
 <key>SystemParameters</key>
 <dict>
  <key>InjectKexts</key>
  <string>Detect</string>
 </dict>
</dict>
</plist>

Deploy Windows RE on system partition

I have been fighting with REAgentC to allow me to install the WinRE image to the system partition (8).

I have gone through numerous errors like »the product has been uninstalled«, »there is insufficient space on the device«, »the operation cannot be completed due to a file system limitation« and the pair of »cannot find the file/path specified« or »file cannot be created because it exists«.

I think the »cannot be created« error was the result of copying an existing boot.sdi ramdisk along with the winre.wim. This is wrong, REAgentC creates the ramdisk.

One thing I've noticed is that at least on an MBR disk, an NTFS partition is required in order to successfully enable WinRE on the system partition. If it's FAT32, I got misleading errors about »insufficient space« until I enlarged it, which then told me it could not complete the operation due to »file system limitations.«

If REAgentC is confused by the current contents of %systemroot%\System32\Recovery\ReAgent.xml, replace by a blank file:

<?xml version='1.0' encoding='utf-8'?>
<WindowsRE version="1.0">
    <WinreBCD id=""/>
    <WinreLocation path="" id="0" offset="0"/>
    <ImageLocation path="" id="0" offset="0"/>
    <OsInstallLocation path="" id="0" offset="0"/>
    <InstallState state="0"/>
    <OsInstallAvailable state="0"/>
    <WinREStaged state="0"/>
    <OperationParam path=""/>
    <OsBuildVersion path=""/>
    <OemTool state="0"/>
    <BootKey state="0"/>
    <IsServer state="0"/>
    <ScheduledOperation state="4"/>
</WindowsRE>

With the recovery partition on R:, the commands

md r:\Recovery\WindowsRE
xcopy /h %systemroot%\System32\Recovery\winre.wim r:\Recovery\WindowsRE
reagentc /setreimage /path R:\Recovery\WindowsRE /target %systemroot%
reagentc /enable
succeed and eventually produced this file:

<?xml version='1.0' encoding='utf-8'?>
<WindowsRE version="1.0">
    <WinreBCD id="{30f4463f-add1-11e4-97b2-e8db73286102}"/>
    <WinreLocation path="\Recovery\WindowsRE" id="196858822" offset="1048576" guid="{00000000-0000-0000-0000-000000000000}"/>
    <ImageLocation path="\Recovery\WindowsRE" id="196858822" offset="1048576" guid="{00000000-0000-0000-0000-000000000000}"/>
    <OsInstallLocation path="" id="0" offset="0" guid="{00000000-0000-0000-0000-000000000000}"/>
    <InstallState state="1"/>
    <OsInstallAvailable state="0"/>
    <WinREStaged state="0"/>
    <OperationParam path=""/>
    <OsBuildVersion path="7601.18700.amd64fre.win7sp1_gdr.141211-1742"/>
    <OemTool state="0"/>
    <BootKey state="0"/>
    <IsServer state="0"/>
    <ScheduledOperation state="4"/>
</WindowsRE>

Note that it's apparently impossible to issue the »/enable« command from Recovery Environment because it doesn't support the »/target« switch.

And here are the relevant boot loader entries created by REAgentC for this case:

Windows-Start-Manager
---------------------
Bezeichner              {bootmgr}
device                  partition=\Device\HarddiskVolume1
description             Windows Boot Manager
locale                  de-de
inherit                 {globalsettings}
default                 {current}
resumeobject            {30f44639-add1-11e4-97b2-e8db73286102}
displayorder            {current}
toolsdisplayorder       {memdiag}
timeout                 30

Windows-Startladeprogramm
-------------------------
Bezeichner              {current}
device                  partition=C:
path                    \Windows\system32\winload.exe
description             Windows 7
locale                  de-de
inherit                 {bootloadersettings}
recoverysequence        {30f4463f-add1-11e4-97b2-e8db73286102}
recoveryenabled         Yes
osdevice                partition=C:
systemroot              \Windows
resumeobject            {30f44639-add1-11e4-97b2-e8db73286102}
nx                      OptIn
detecthal               Yes

Windows-Startladeprogramm
-------------------------
Bezeichner              {30f4463f-add1-11e4-97b2-e8db73286102}
device                  ramdisk=[\Device\HarddiskVolume1]\Recovery\WindowsRE\Winre.wim,{30f44640-add1-11e4-97b2-e8db73286102}
path                    \windows\system32\winload.exe
description             Windows Recovery Environment
inherit                 {bootloadersettings}
osdevice                ramdisk=[\Device\HarddiskVolume1]\Recovery\WindowsRE\Winre.wim,{30f44640-add1-11e4-97b2-e8db73286102}
systemroot              \windows
nx                      OptIn
winpe                   Yes

Geräteoptionen
--------------
Bezeichner              {30f44640-add1-11e4-97b2-e8db73286102}
description             Ramdisk Options
ramdisksdidevice        partition=\Device\HarddiskVolume1
ramdisksdipath          \Recovery\WindowsRE\boot.sdi

For the tools to work, the system and recovery partition must be type id 0x27 on an MBR disk. The MBR and GPT partition types are:

Recovery300MBNTFS0x27de94bba4-06d1-4d40-a16a-bfd50179d6ac
ESP100MBFAT320x27c12a7328-f81f-11d2-ba4b-00a0c93ec93b
MSR128MBNTFS0x7e3c9e316-0b5c-4db8-817d-f92df00215ae
OSNTFS0x7ebd0a0a2-b9e5-4433-87c0-68b6b72699c7

Other tools

Helpful tools I've used (apart from REAgentC and bcdedit, obviously) include bcdboot %systemroot% /l en-us /s s: to copy the required files to the boot partition and set the desired locale and the Recovery Environment commands bootsect /nt60 %systemroot%, bootrec /fixboot and bootrec /fixmbr (the latter apparently Windows XP relics) to restore a partition's boot loader.

Did you know

…you can start WordPad using "write" from the cmd line? Very useful when editing LF-only files as administrator…