Denon RC-1146 remote with LibreELEC 9

In LibreELEC 9, the IR driver was changed from lircd to in-kernel decoding with ir-keytable.

I'm still using the GPIO IR receiver from this tutorial on my Raspberry Pi.

To activate it, I need to add a dtoverlay.

# mount -o remount,rw /flash
# echo dtoverlay=gpio-ir >> /flash/config.txt
# mount -o remount,ro /flash

I then proceeded along this tutorial.

# systemctl stop kodi
# systemctl stop eventlircd

I get from ir-keytable:

# ir-keytable
Found /sys/class/rc/rc0/ (/dev/input/event2) with:
 Name: gpio_ir_recv
 Driver: gpio_ir_recv, table: rc-rc6-mce
 lirc device: /dev/lirc0
 Supported protocols: other lirc rc-5 rc-5-sz jvc sony nec sanyo mce_kbd rc-6 sharp xmp
 Enabled protocols: lirc nec rc-6
 bus: 25, vendor/product: 0001:0001, version: 0x0100
 Repeat delay = 500 ms, repeat period = 125 ms

After some trial and error with different protocols, I finally found:

# ir-keytable --protocol sharp --test
Protocols changed to sharp
Testing events. Please, press CTRL-C to abort.
1009.423304: lirc protocol(sharp): scancode = 0x8ac
1009.423341: event type EV_MSC(0x04): scancode = 0x8ac
1009.423341: event type EV_SYN(0x00).

As with all things Linux, documentation is notoriously difficult to come by. According to ir-keytable --version, LibreELEC-9.0.1 uses v4l-utils-1.14.2. This is an important detail since v4l-utils have since changed their keytable format to toml!

I could not find any documentation for the old keytable format, so I had a look at the source code of keytable.c.

Apparently, a # in the first line can be followed by table and type keywords specifying the name of the table and what is otherwise known as the protocol, separated by various combinations of whitespace, equal signs, colons and commas. Otherwise, lines beginning with # are ignored. The remaining lines may start with an optional keyword "scancode", followed by whitespace, equals sign or colon. This is followed by the scancode (in a format understood by strtoul(3)) and terminated by whitespace, equals sign or colon. Finally, there should be a keycode terminated by whitespace, equals sign, colon or an open parenthesis(!). The keycodes can be ones listed by irrecord --list or, better yet, ones from the devinput section of /usr/share/kodi/system/Lircmap.xml to make sure they are assigned the desired function in Kodi, or in a format suitable for strtol(3). The rest of the line is ignored and can thus be used for comments.

Since Kodi is connected to the DVD input of my amp, I used the remote's default DVD profile. Eventually, I ended up with the following /storage/.config/rc_keymaps/denon1146dvd:

# table denon1146dvd, type: sharp
0x6c0 KEY_POWER
0x881 KEY_0
0x882 KEY_1
0x883 KEY_2
0x884 KEY_3
0x885 KEY_4
0x886 KEY_5
0x887 KEY_6
0x888 KEY_7
0x889 KEY_8
0x88a KEY_9
0x88c KEY_SUBTITLE 10+
0x891 KEY_INFO menu
0x892 KEY_BACK
0x898 KEY_NEXT
0x899 KEY_PREVIOUS
0x89a KEY_FASTFORWARD
0x89b KEY_REWIND
0x89d KEY_PAUSE
0x8a0 KEY_PLAY
0x8a1 KEY_STOP
0x8ac KEY_UP
0x8ad KEY_DOWN
0x8ae KEY_RIGHT
0x8af KEY_LEFT
0x8bb KEY_ENTER
0x8bc KEY_EPG search
0x8bd KEY_MENU top menu

The non-obvious assigments are: the key labelled »menu« on the remote sends a KEY_INFO code since that seems to be the most useful code in Kodi (file info or playing info, respectively). The »top menu« key sends KEY_MENU which takes you to Kodi's main menu and toggles the play menu. I assigned KEY_EPG to the »search« key since KEY_EPG invokes the context menu in the menu and the decoder info while playing. Finally, I cheekily remapped the »10+« key to KEY_SUBTITLE since the former seems to be unused and I use the latter a lot.

Last, I activated it like this:

# echo '* * denon1146dvd' >> /storage/.config/rc_maps.cfg

Logging https traffic using Raspbian

In order to intercept https traffic, it is necessary to run software on a router or access point that provides its own certificates to clients. Using Mirko Dölle's beautiful instructions, today I set up mitmproxy on a Raspberry Pi 2 running Raspbian based on Debian Jessie. I have an RT5572 based dual band wifi dongle that works very well with Raspbian which I am going to use for clients to connect while the Raspberry Pi 2 itself is connected to the internet via ethernet.

First, I set up Raspbian in the usual way. Then I modified the dhcpcd configuration to assign a static IP address to the wifi adaptor which is going to be my access point. I added the following lines to the end of /etc/dhcpcd.conf.

interface wlan0
static ip_address=192.168.6.1/24

Next, I added the hostapd package and created a configuration file for it:

# apt-get install hostapd
# cat >/etc/hostapd/hostapd.conf <<EOF
interface=wlan0
driver=nl80211
country_code=GB
ctrl_interface=/var/run/hostapd
ctrl_interface_group=0

ssid=rpi-mitm

# 802.11g on #8
hw_mode=g
channel=8

# encryption
wpa=2
wpa_passphrase=pleasehackme
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

# gobbledigook
beacon_int=100
auth_algs=1
wmm_enabled=1
EOF

Before activating, one line in /etc/default/hostapd has to be adjusted:

DAEMON_CONF="/etc/hostapd/hostapd.conf"

In order to assign IP addresses to clients and handle DNS forwarding, I then installed dnsmasq, again creating a configuration for it.

# apt-get install dnsmasq
# cat >>/etc/dnsmasq.conf <<EOF
interface=wlan0
dhcp-range=192.168.6.50,192.168.6.100,12h
EOF

Next, I activated package forwarding by uncommenting the following lines in /etc/sysctl.conf:

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

I then set up network address translation by adding a POSTROUTING rule for my WAN interface to the NAT iptables. This setting needs to be persisted by installing the iptables-persistent package.

# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# apt-get install iptables-persistent

Then I restart all the services involved. I disable legacy Debian networking because it takes the wlan0 interface up which prevents hostapd from starting. This completes the setup of the access point.

# systemctl disable networking
# systemctl start hostapd
# systemctl start dnsmasq
# systemctl restart procps

In order to inspect client traffic through the access point, I installed mitmproxy. The version 0.10.1-2 installed by apt-get (on Jessie) is too old, so I install the current version 0.18.2 manually. Since Jessie only provides Python 3.4, I went with Python 2.7.

# apt-get install python-pip python-dev libffi-dev libssl-dev libxml2-dev libxslt1-dev libjpeg8-dev zlib1g-dev g++
# pip install mitmproxy

To run it, I add a couple of redirections to the inbound interface.

# iptables -t nat -A PREROUTING -i wlan0 -p tcp --dport 80 -j REDIRECT --to-port 8080
# iptables -t nat -A PREROUTING -i wlan0 -p tcp --dport 443 -j REDIRECT --to-port 8080
# mitmproxy -T --host

Install mitmproxy's certificate on the device by browsing to http://mitm.it/.

To deactivate the capture, it suffices to remove the iptables rules.

# iptables -t nat -F PREROUTING

Correct local domain and host-name DD-WRT setup

For a long time, I've struggled with inconsistent results for name queries on my local network. For example, sometimes OS X (El Capitan) couldn't ping a host by name while dig (host, nslookup) worked fine.

Also, I had never really tried to understand several options in the DD-WRT setup for domain and host names and the way they affected dnsmasq's configuration, the default DHCP and DNS server on DD-WRT.

My situation is that of a typical home user with a DHCP-assigned IPv4 address and no public host name or top-level domain.

As it turned out, I had got the setup wrong: the »hostname« and »domain name« on the main setup page refer to DHCP options sent to the ISP by the router's DHCP client, udhcpc. My ISP doesn't require any of those, so the fields should be left blank on my setup.

Instead, the right place to set the router's host name is the aptly-named »Router name« field.

The correct place to define the top-level domain of the local network is the »domain name« field in the »DHCP Server« section on the »Services« tab. In addition, »Used domain« must be set to »LAN and WLAN« instead of »WAN« because we don't actually have a »WAN« domain.

By the way, I chose »lan« as the name of my local domain because »local« has been specified for use by mDNS.

As a result, the router's host name is correctly entered into /etc/hosts (both with and without my local domain) and dnsmasq resolves names of local DHCP clients both in their short and long forms.

Use all USB3 ports of Intel DH87RL on El Capitan

Today I succeeded in enabling all of my Intel DH87RL motherboard's USB3 ports.

As it turns out, El Capitan's XHCI driver has a limit of 15 ports which was taken up by the board's USB2 ports, leaving just one USB3 port usable.

Since only 8 USB2 ports are actually physically accessible from the outside, disabling the unused ports makes more USB3 ports available.

In theory, this should happen automatically because OS X evaluates the board's DSDT which should flag the user-accessible ports. Needless to say, the DH87RL's DSDT is buggy, so this method fails. ioreg is a helpful tool to find out the current configuration.

Helpfully, Apple has implemented a port disabling feature to handle the case of buggy DSDTs. Examples can be seen by examining the existing plug-ins of the XHCI drivers, to be found at /System/Library/Extensions/IOUSBHostFamily.kext/Contents/PlugIns/AppleUSBXHCIPCI.kext/Contents/Info.plist. This file contains port specifications for each different computer model.

In order to create my own port specification for my DH87RL, I used the fact that the XHCI controller uses a label of 'XHC1' on Apple's DSDT's, while the DH87RL's XHCI is labelled 'XHC' by the DSDT. This allowed me to create my own kext by matching on the emulated model name 'iMac14,2' in combination with 'XHC'.

By trial and error I found out that the USB2 ports provided by the DH87RL are numbered 1, 2, 3, 4, 6, 8, 9 and A. I created a file /EFI/CLOVER/OEM/DH87RL/kexts/10.11/InjectUSB.kext/Contents/Info.plist with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>CFBundleDevelopmentRegion</key>
 <string>English</string>
 <key>CFBundleGetInfoString</key>
 <string>1.0.1, Copyright © 2000-2014 Joki Inc. All Rights Reserved.</string>
 <key>CFBundleIdentifier</key>
 <string>org.joki.injector.usb.USBXHCIDH87RL</string>
 <key>CFBundleInfoDictionaryVersion</key>
 <string>6.0</string>
 <key>CFBundleName</key>
 <string>XHC1 injector for DH87RL</string>
 <key>CFBundlePackageType</key>
 <string>KEXT</string>
 <key>CFBundleShortVersionString</key>
 <string>1.0.1</string>
 <key>CFBundleSignature</key>
 <string>????</string>
 <key>CFBundleVersion</key>
 <string>1.0.1</string>
 <key>IOKitPersonalities</key>
 <dict>
  <key>iMac14,2-XHC1</key>
  <dict>
   <key>CFBundleIdentifier</key>
   <string>com.apple.driver.AppleUSBMergeNub</string>
   <key>IOClass</key>
   <string>AppleUSBMergeNub</string>
   <key>IONameMatch</key>
   <string>XHC</string>
   <key>IOProviderClass</key>
   <string>AppleUSBXHCIPCI</string>
   <key>IOProviderMergeProperties</key>
   <dict>
    <key>port-count</key>
    <data>FQAAAA==</data>
    <key>ports</key>
    <dict>
     <key>HS01</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>AQAAAA==</data>
     </dict>
     <key>HS02</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>AgAAAA==</data>
     </dict>
     <key>HS03</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>AwAAAA==</data>
     </dict>
     <key>HS04</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>BAAAAA==</data>
     </dict>
     <key>HS06</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>BgAAAA==</data>
     </dict>
     <key>HS08</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>CAAAAA==</data>
     </dict>
     <key>HS09</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>CQAAAA==</data>
     </dict>
     <key>HS0A</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>CgAAAA==</data>
     </dict>
     <key>SSP1</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>EAAAAA==</data>
     </dict>
     <key>SSP2</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>EQAAAA==</data>
     </dict>
     <key>SSP3</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>EgAAAA==</data>
     </dict>
     <key>SSP4</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>EwAAAA==</data>
     </dict>
     <key>SSP5</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>FAAAAA==</data>
     </dict>
     <key>SSP6</key>
     <dict>
      <key>UsbConnector</key>
      <integer>3</integer>
      <key>port</key>
      <data>FQAAAA==</data>
     </dict>
    </dict>
   </dict>
   <key>model</key>
   <string>iMac14,2</string>
  </dict>
 </dict>
 <key>OSBundleRequired</key>
 <string>Root</string>
</dict>
</plist>

After reboot, I could use all of my USB3 ports (for reference, numbered 9, A, B, C, D and E).

A handy command to see the kernel boot messages is

$ log show --debug --last boot --style compact

Setting WiFi regulatory domain on OpenELEC

When I connected my new 802.11 dongle to a Raspberry Pi running OpenELEC, I got the following messages in the system log:

cfg80211: World regulatory domain updated:
cfg80211:  DFS Master region: unset
cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
cfg80211:   (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
cfg80211:   (2457000 KHz - 2482000 KHz @ 20000 KHz, 92000 KHz AUTO), (N/A, 2000 mBm), (N/A)
cfg80211:   (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A)
cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A)
cfg80211:   (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s)
cfg80211:   (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s)
cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A)
cfg80211:   (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A)

A quick check with cat /sys/module/cfg80211/parameters/ieee80211_regdom revealed that the regulatory domain was set to 00.

In order to fix this, I executed

# echo 'options cfg80211 ieee80211_regdom="DE"' > ~/.config/modprobe.d/cfg80211.conf
and rebooted.

cfg80211: Regulatory domain changed to country: DE
cfg80211:  DFS Master region: ETSI
cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
cfg80211:   (2400000 KHz - 2483000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
cfg80211:   (5150000 KHz - 5250000 KHz @ 80000 KHz, 200000 KHz AUTO), (N/A, 2000 mBm), (N/A)
cfg80211:   (5250000 KHz - 5350000 KHz @ 80000 KHz, 200000 KHz AUTO), (N/A, 2000 mBm), (0 s)
cfg80211:   (5470000 KHz - 5725000 KHz @ 160000 KHz), (N/A, 2700 mBm), (0 s)
cfg80211:   (57000000 KHz - 66000000 KHz @ 2160000 KHz), (N/A, 4000 mBm), (N/A)

Denon RC-1146 remote with Kodi

Today I created a configuration file for my Denon RC-1146 remote. Since Kodi is connected to the DVD input of my amp, I used the remote's default DVD profile.

First, I wired up the IR detector to the Raspberry Pi's GPIO pins as shown in the tutorial.

Because I use OpenELEC rather than RaspBMC, I then had to edit the configuration file to load the device tree overlay.

# mount -o remount,rw /flash
# echo dtoverlay=lirc-rpi >> /flash/config.txt
# mount -o remount,ro /flash

From LibreELEC 9, rc-core is used instead of lircd, so the instructions below will need to change.

For the /storage/.config/lircd.conf file, I tweaked some of the codes for Kodi: the key labelled »menu« on the remote sends a KEY_INFO code since that seems to be the most useful code in Kodi (file info or playing info, respectively). The »top level« key sends KEY_MENU which takes you to Kodi's main menu and toggles the play menu. I assigned KEY_EPG to the »setup« key since KEY_SETUP doesn't seem to do anything while KEY_EPG invokes the context menu in the menu and the decoder info while playing. Finally, I cheekily remapped the »10+« key to KEY_SUBTITLE since the former seems to be unused and I use the latter a lot.

The numeric keys are assigned KEY_NUMERIC prefixes because Kodi requires it.

Also, the key labelled »enter« is assigned to KEY_OK and »return« is assigned to KEY_EXIT since that seems to be more appropriate in the context of lircd.

# Please make this file available to others
# by sending it to <lirc@bartelmus.de>
#
# this config file was automatically generated
# using lirc-0.9.1-git(default) on Tue Apr  5 23:36:06 2016
#
# contributed by <jocki84@googlemail.com>
#
# brand: Denon
# model no. of remote control: RC-1146
# devices being controlled by this remote: Denon DVD Players
# This multi-purpose remote has many emulations, these codes
# were generated using the default setting for DVD, 32134.
#

begin remote

  name  DENON_RC1146_DVD
  bits           15
  flags SPACE_ENC|CONST_LENGTH
  eps            30
  aeps          100

  one           290  1815
  zero          290   760
  ptrail        290
  gap          66911
  toggle_bit_mask 0x0

      begin codes
          KEY_POWER                0x300C
          KEY_UP                   0x08D4
          KEY_DOWN                 0x092B
          KEY_LEFT                 0x082B
          KEY_RIGHT                0x09D4
          KEY_OK                   0x088B # KEY_ENTER
          KEY_INFO                 0x09DB # KEY_MENU
          KEY_MENU                 0x090B # KEY_DVD
          KEY_EPG                  0x08F4 # KEY_SETUP
          KEY_EXIT                 0x0924
          KEY_REWIND               0x089B
          KEY_PLAY                 0x0814
          KEY_FASTFORWARD          0x0964
          KEY_PREVIOUS             0x099B
          KEY_PAUSE                0x091B
          KEY_STOP                 0x09EB
          KEY_NEXT                 0x0864
          KEY_NUMERIC_1            0x0904
          KEY_NUMERIC_2            0x08FB
          KEY_NUMERIC_3            0x0884
          KEY_NUMERIC_4            0x097B
          KEY_NUMERIC_5            0x0984
          KEY_NUMERIC_6            0x087B
          KEY_NUMERIC_7            0x0844
          KEY_NUMERIC_8            0x09BB
          KEY_NUMERIC_9            0x0944
          KEY_NUMERIC_0            0x09FB
          KEY_SUBTITLE             0x08C4 # KEY_102ND
      end codes

end remote

Note for LibreELEC 8.2 In order for lircd to start, it's necessary to

$ touch /storage/.cache/services/lircd.conf

Note for LibreELEC 7.90.008: lircd should be started when the /dev/lirc0 device is created by the lirc_rpi kernel module. This is achieved using udev which starts an appropriate instance of /usr/lib/systemd/system/lircd.service (such as lircd@lirc0:default:lircd.conf.rpi) from /usr/lib/udev/rules.d/98-lircd.rules when the device appears. Occasionally, there might be a link in .config/system.d/multi-user.target.wants which leads to lircd being started before the device node has been created. Such a link should not exist.

Restoring previous Perl library in El Capitan

Because I need an historic Perl 5.10 library for a program I use (GPSPhotoLinker), after upgrading to El Capitan I wanted to recover the old libraries in /System/Library/Perl/5.10 from Time Machine. Unlike the previous times I did this, Time Machine refused. So I restored the folder elsewhere and was in for a surprise.

$ sudo mv 5.10 /System/Library/Perl
mv: rename 5.10 to /System/Library/Perl/5.10: Operation not permitted

Hmmm, that's not supposed to happen. A bit of googling revealed the cause.

$ ls -lO /System/Library/Perl/
total 0
drwxr-xr-x  130 root  wheel  restricted 4420 23 Aug 02:09 5.16/
drwxr-xr-x  130 root  wheel  restricted 4420 23 Aug 02:08 5.18/
drwxr-xr-x    4 root  wheel  restricted  136 23 Aug 05:49 Extras/

There's a new restricted flag at work that's not documented in chflags(1).

Some more googling produced the video that explains it all, including how to disable the system integrity protection from Recovery as a developer.

Rather than jumping through all the hoops of installing perl in the expected location only to lose it again after the next OS upgrade, I decided to copy perl into the application bundle instead.

Playing with otool(1) and install_name_tool(1) let me change the binary's search path, but as soon as the perl code attempted to load a library it failed. In the end my solution was to create a small wrapper script to set PERL5LIB and DYLD_LIBRARY_PATH to the perl embedded into the application bundle.

$ cd /Applications/GPSPhotoLinker.app/Contents/MacOS
$ mv GPSPhotoLinker GPSPhotoLinker.bin
$ cat >GPSPhotoLinker <<EOF
#!/bin/sh

PERL5LIB=${PERL5LIB}:${0%/*}/../Perl/5.10 \
        DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:${0%/*}/../Perl/5.10/darwin-thread-multi-2level/CORE \
        "${0}.bin"
EOF
$ chmod a+x GPSPhotoLinker

TimeMachine and ReadyNAS OS

Every once in a while, I get a message from Time Machine telling me that it cannot back up because »the sparse bundle is in use«.

Restarting the NAS fixes the problem, but that's a bit hard-core. I've found that it starts working again if I disable and re-enable the AFP service through the web interface. An even easier way to do this is to ssh into the NAS and say

# systemctl restart netatalk

Recovering backups from differently formatted CD-ROMs

Over the past 20 years, I have built up a collection of backup CD-ROMs. There is quite large variation in their formats due to the different platforms I was using at various times and the development of the standards. I have plain ISO-9660 media, some using Unix Rock Ridge extensions, and others using Windows Joliet extensions. Most of my media are multi-session because I thought it might be useful to be able to add some data later on.

Getting data from these media isn't actually very difficult in itself. As long as there are no read errors, the file contents can be retrieved using any system.

However, the problem starts when one would like to restore correct file names, directory hierarchies and properties such as ownership, read/write/execute permissions or hard links.

Note for the future: even if it's slightly less convenient, it's actually a good idea to put archives on CDs (or backups in general)... much easier to read later on than some silly file system that you have to find the right mount options for.

Image creation

After a day of fiddling, here's a workflow that I settled on:

I use Harald Bögeholz's H2cdimage to read the contents of the CD into an .ISO file. This tool can be run on multiple machines using different drives which increases the chances of recovering data from defective media.

Unfortunately, H2cdimage doesn't create a .CUE file with the track layout information which appears to be needed to correctly mount multi-session CDs. I use IsoBuster which creates beautifully annotated .CUE files that also show the linear block address (LBA) of sessions which will be needed for mounting.

FILE "CD.iso" BINARY

REM ORIGINAL MEDIA-TYPE: CD

  REM SESSION 01        ; Not supported by other applications (*)
    TRACK 01 MODE2/2352
      INDEX 01 00:00:00
      REM MSF: 00:00:00 = LBA: 0

  REM RUN-OUT  18:12:70 ; Not supported by other applications (*)
  REM LEAD-OUT 18:12:72 ; Not supported by other applications (*)
  REM SESSION 02        ; Not supported by other applications (*)
    TRACK 02 MODE2/2352
      INDEX 01 20:44:72
      REM MSF: 20:44:72 = LBA: 93372

REM (*) SESSION directives are unfortunately not properly supported
REM     'out there'.  IsoBuster however supports them !

IsoBuster also provides »managed image files« which are based on the same idea of using different hardware to read as much data as possible from damaged discs, but I have not tried that.

File extraction

In order to extract the files, here are the things I've tried:

7zip

The simplest and most pleasant way is to just use

$ 7z x CD.iso
This works great for plain ISO9660 files and even deals very well with Joliet extensions, but when I tried it on a multi-session Rock Ridge disc it only saw the first session, so I'm not sure whether it would have dealt correctly with the RR_MOVED folder, for example.

IsoBuster

IsoBuster can extract data from any session and even allows specifying the filesystem character set which can be a problem for incorrectly burned media. For example, one of my ISO9660 discs uses the ancient DOS codepage 437, and one of my Rock Ridge discs uses ISO8859-1 filenames. I have no idea whether this is correct or not (since I burned it myself), but the names definitely have to be converted before being stored on a modern filesystem.

However, if there are any hard linked files on the medium, IsoBuster will extract them multiple times and the information which files are identical is lost in the process. Additionally, the resulting folder might be much bigger than the original CD image.

Also, IsoBuster is unable to place files that were moved to the RR_MOVED folder back into the correct place in the hierarchy. As a reminder, here's the deal with RR_MOVED:

If mkisofs is creating a filesystem image with Rock Ridge attributes and the directory nesting level of the source directory tree is too much for ISO-9660, mkisofs will do deep directory relocation. This results in a directory called RR_MOVED in the root directory of the CD. You cannot avoid this directory in the directory tree that is visible with ISO-9660 but it it automatically hidden in the Rock Ridge tree.

Lastly, since it runs on Windows IsoBuster cannot extract some files from Rock Ridge discs. For example, I have one file that contains a ':' colon (created by CVSup back in the day) which Windows does not allow in file names.

mount

The only way I found to correctly restore file hierarchies from Rock Ridge discs is to use Unix. You could even say that's fair enough since those discs were created on Unix in the first place, and contain Unix files.

But there's also an advantage over the 7z or IsoBuster for pure ISO9660 filesystems: filenames are presented in lower-case.

If you need to read later sessions of a multi-session disc it's not sufficient to just mount -o loop CD.iso /mnt/iso it. Apparently, there is information on the disc that allows mount to pick the latest session as advertised, but it's missing from the image file and mount itself does not understand .CUE files and needs a little help. This is where the LBA from IsoBuster comes into play:

# mount -o loop,sbsector=93372 CD.iso /mnt/iso

Now you can use

$ cd /mnt/iso
$ pax -r -w -p p . /.../dest
to copy the files to the new backup medium.

Unfortunately, Linux's mount only supports character set conversions via the iocharset option for Joliet extensions. If you're stuck with a Rock Ridge or ISO9660 disc in a different encoding, you're going to need a script to rename those files manually after copying

#!/bin/sh

# uncomment for debugging
#ECHO=echo

if [ $# -lt 2 ]
then
	cat >&2 <<EOF
usage: $0 from to <file> ...

	Convert file name encodings. Use "iconv -l" for supported encodings.
EOF
	exit 1
fi

from="$1"
shift
to="$1"
shift

for p in "$@"
do
	f="${p##*/}"
	d="${p%"$f"}"
	t=$(echo "$f" | iconv -f "$from" -t "$to")
	if [ "$f" != "$t" ]
	then
		echo "$d$f -> $t"
		$ECHO mv -i "$d$f" "$d$t"
	fi
done

This can be run on the entire tree using, for example

$ find . -depth -exec /.../fconv cp437 utf8 {} \+
The -depth primitive is important in case directories are renamed, after which subsequent commands won't find the remaining files to rename.

As a final step, don't forget to mark the extracted images and files read-only just to prevent stupid mistakes.

$ chmod -R a-w *

Removing duplicates

When working with backups it's common to get some duplication of files. To deal with this, I looked at the fslint suite (which includes a GUI), rmlint which is extremely complex, fdupes which didn't seem very encouraging, duff which seemed promising but was a bit slower than rdfind that I eventually settled on.

My requirement was to find duplicate files in two (or more) given directories and delete duplicates in all but one of them, which was supposed to remain untouched. That is, even if the first directory contained duplicates, I didn't want to remove them. While rmlint supports this out of the box using »tagging,« it wasn't available on my platform and so I wrote a small script to process the rdfind output.

#!/bin/sh

# uncomment for debugging
#ECHO=echo

if [ $# -lt 2 ]
then
	cat >&2 <<EOF
usage: $0 <dir> ...

	Find duplicate files in the named directories. Files from the first
	named directory are never deleted even if they are duplicates.
EOF
	exit 1
fi

rdfind -checksum sha1 -outputname dedup.$$ "$@"
cat dedup.$$ | \
	grep -v '^#' | \
	grep -v '^DUPTYPE_FIRST_OCCURRENCE' | \
	cut -d' ' -f8- | \
	grep -v "^$1/" | \
while read f
do
	$ECHO rm "$f"
done
rm dedup.$$

Reloading services and kexts using AppleScript

This weekend, I had to try to convince a slightly reluctant (no-name) bluetooth mouse to work on an Apple laptop. Especially after a sleep-wakeup cycle it tends to stop reacting and my theory is that it confuses OS X's bluetooth driver to the point where it stops functioning.

As a small test, I decided to try to unload both the bluetooth daemon blued and the IOBluetoothHIDDriver when the problem occurs and see if that fixes it. For this purpose, I came up with the following one-click experiment, my very first AppleScript.

set bluedController to serviceController for "com.apple.blued" given defaults:"/Library/Preferences/com.apple.Bluetooth", preference:"ControllerPowerState"

set driverController to kextController for "com.apple.driver.IOBluetoothHIDDriver"


restart({bluedController, driverController})


-- Shutdown a list of services, waiting for each to stop, and restart them in reverse order

on restart(controllers)

repeat with controller in controllers

repeat

tell controller

shutdown()

if not isRunning() then exit repeat

end tell

display dialog "Waiting for " & controller's name & " to stop" giving up after 1

end repeat

end repeat

repeat with controller in reverse of controllers

repeat

tell controller

start()

if isRunning() then exit repeat

end tell

display dialog "Waiting for " & controller's name & " to start" giving up after 1

end repeat

end repeat

end restart


-- Return a script object that controls a launch service using its associated preference.

on serviceController for serviceLabel given defaults:prefsFile, preference:prefName

script controller

property name : "service " & last item of (wordList of serviceLabel at ".")

on launchCtl(command)

-- launchctl only knows about blued if run as admin

do shell script "launchctl " & ¬

quoted form of command & " " & ¬

quoted form of serviceLabel ¬

with administrator privileges

end launchCtl

on setPref(value)

-- turn off the service's preference

do shell script "defaults write " & ¬

quoted form of prefsFile & " " & ¬

quoted form of prefName & " " & ¬

quoted form of (value as text) ¬

with administrator privileges

end setPref

on isRunning()

return launchCtl("list") contains "\"PID\" = "

end isRunning

on shutdown()

setPref(0)

launchCtl("stop")

end shutdown

on start()

setPref(1)

launchCtl("start")

end start

end script

return controller

end serviceController


-- Return a script object that loads and unloads a kernel extension.

on kextController for bundleIdentifier

script controller

property name : "kernel extension " & last item of (wordList of bundleIdentifier at ".")

on isRunning()

do shell script "kextstat -l -b " & ¬

quoted form of bundleIdentifier

return the result contains bundleIdentifier

end isRunning

on shutdown()

try

do shell script "kextunload -b " & ¬

quoted form of bundleIdentifier ¬

with administrator privileges

end try

end shutdown

on start()

do shell script "kextload -b " & ¬

quoted form of bundleIdentifier ¬

with administrator privileges

end start

end script

return controller

end kextController


on wordList of theWords at delimiters

set oldDelimiters to text item delimiters

set text item delimiters to delimiters

try

set res to text items of theWords

set text item delimiters to oldDelimiters

return res

on error m number n

set text item delimiters to oldDelimiters

error m number n

end try

end wordList

Yosemite with audio on DH87RL with Realtek ALC892

The following is now obsolete thanks to AppleALC which only requires the kernel parameter :Boot:Arguments alcid=1.


In order to get audio functionality in Yosemite on my DH87RL motherboard, I follow toleda's method using Clover. The exact method is encoded as a shell script with the required files in a couple of repositories.

The first task is to find out the current layout-id using the command

$ ioreg -nHDEF -r -w0
There should be a section called "HDEF@1B" with a key "layout-id" showing a value.

In my case, the value was 0 which is not a good sign. Thankfully, Clover can patch it by adding the string value :Devices:Audio:Inject with the desired value "1".

After rebooting, the command

$ ioreg -rxnIOHDACodecDevice
identifies my audio chip as vendor and device 0x10ec0892 (Realtek ALC892) and revision 0x100302 (current).

Now I downloaded and unpacked config-audio_cloverALC.plist.zip and copied the two relevant entries from :KernelAndKextPatches:KextsToPatch to my config.plist: Item 0 with comment »10.9-10.10-AppleHDA/Resources/xml>zml« (the purpose of which will become clear in a little while) and Item 4, »10.9-10.10-AppleHDA/Realtek ALC892«.

Then I downloaded and unpacked realtekALC.kext and installed it into my EFI/CLOVER/OEM/DH87RL/kexts/10.10 folder. Finally, I renamed the files Platforms.xml.zlib and layout[123].xml.zlib from 892.zip to *.zml.zlib, installed them in my /System/Library/Extensions/AppleHDA.kext/Contents/Resources folder and made them owned by root:wheel.

Next, I ensured that the :SystemParameters:InjectKexts was enabled in my config.plist ("Detect") and that :Boot:Arguments contained "kext-dev-mode=1" and rebooted with rebuilding kext cache.

Running Yosemite on UEFI hardware

I use the Clover boot loader in a UEFI-only installation on the ESP with the »BootCamp« theme. The only UEFI driver I need is OsxAptioFix2Drv-64 -- without it I get ???.

The best source for up-to-date kexts I've found is to download the current MultiBeast installer. The »Contents/Resources« folder of »MultiBeast.app« contains packages which can be expanded using

$ pkgutil --expand FakeSMC-v6.14.1364.pkg p
$ pax -rzf p/Payload

For my Intel DH87RL, I drop FakeSMC.kext and AppleIntelE1000e.kext into the corresponding folder OEM/DH87RL/kexts/10.10 on the ESP.

After installing new kexts, a reboot »without caches« must be performed by selecting the boot entry with space bar in Clover and selecting the corresponding entry.

My configuration file OEM/DH87RL/config.plist contains

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ACPI</key>
 <dict>
  <key>SSDT</key>
  <dict>
   <key>EnableC6</key>
   <true/>
   <key>Generate</key>
   <dict>
    <key>CStates</key>
    <true/>
    <key>PStates</key>
    <true/>
   </dict>
  </dict>
 </dict>
 <key>Boot</key>
 <dict>
  <key>DefaultVolume</key>
  <string>LastBootedVolume</string>
  <key>Timeout</key>
  <integer>5</integer>
  <key>Log</key>
  <false/>
 </dict>
 <key>GUI</key>
 <dict>
  <key>Scan</key>
  <true/>
  <key>Custom</key>
  <dict>
   <key>Legacy</key>
   <array>
    <dict>
     <key>Type</key>
     <string>Windows</string>
     <key>Hidden</key>
     <true/>
    </dict>
   </array>
  </dict>
 </dict>
 <key>KernelAndKextPatches</key>
 <dict>
  <key>Debug</key>
  <false/>
  <key>KernelPm</key>
  <true/>
 </dict>
 <key>RtVariables</key>
 <dict>
  <key>MountEFI</key>
  <false/>
 </dict>
 <key>SystemParameters</key>
 <dict>
  <key>InjectKexts</key>
  <string>Detect</string>
 </dict>
</dict>
</plist>

Installing Mavericks on UEFI hardware

I decided to try and build a Mavericks installer for my new Intel DH87RL based computer. My aim was to use as few extra files as possible from a plain Mavericks installation medium, and especially no tools to perform unknown tasks for me.

I create a Mavericks install medium using the suggestively named script createinstallmedia:

$ cd <Path to Install OS X Mavericks.app>
$ sudo ./Contents/Resources/createinstallmedia --volume /Volumes/<scratch media> --applicationpath "$PWD"

Try to boot...

Next, there is an initially confusing number of boot loader options: Chameleon/Chimera and Clover. According to my research, the former are legacy boot loaders for pre-UEFI boards and will utilise the compatibility module when booted on UEFI hardware -- yuck!

So Clover seems to be the way to go for modern hardware. It even offers a friendly manual installation using only terminal commands without resorting to magic tools.

I am following this handy guide.

Attempts on Asus P6T SE LGA 1366 board with Radeon HD4790

A plain Mavericks installer with a Clover boot loader and an SMBIOS for a MacPro5,1 (LGA1366) boots to a grey screen with an apple logo and a SBBOD (Spinning Beach Ball Of Death).

Adding FaceSMC.kext to /EFI/CLOVER/OEM/P6T SE/kexts/10.9 and setting "InjectKexts" to "Detect" in the "SystemParameters" section of my config.plist successfully brings up the installer. Adding "RealtekRTL81xx.kext" additionally enables ethernet.

The system boots both with my Sapphire HD4350 and an Asus HD7790, but the former chooses the wrong video mode and seems very slow while the second one seems to be better supported by Mavericks. In the installer, Safari flickers a lot with both video cards.

Next, I formatted the EFI partition that the OS X installer had created using FAT32 to enable Clover to write log files or DSDTs (FAT32 is the only writable file system for Clover).

$ sudo newfs_msdos -F 32 -v EFI /dev/disk0s1

Note: FAT32 (-F 32) is important because newfs_msdos picks FAT16 by default which isn't supported by Clover's 'boot1f' code, resulting in the messages:

boot1f: init
boot1f:error

Then I installed Clover to this EFI partition on the target disk with the following settings:

  • Install Clover in the ESP (because I don't want it on my HFS hard drive)
  • Bootloader: Install boot0af in MBR (because I want the 'active' partition flag to work)
  • CloverEFI: CloverEFI 64-bits SATA (I think it does not make a difference on my BIOS-based computer)
  • Install RC scripts on target volume (this was needed to make the NVRAM work, which again is required to set the startup volume and use automatic boot)

This is the config.plist file I use (yes, that's all):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ACPI</key>
 <dict>
  <key>SSDT</key>
  <dict>
   <key>EnableC6</key>
   <true/>
   <key>Generate</key>
   <dict>
    <key>CStates</key>
    <true/>
    <key>PStates</key>
    <true/>
   </dict>
  </dict>
 </dict>
 <key>Boot</key>
 <dict>
  <key>Timeout</key>
  <integer>0</integer>
 </dict>
 <key>GUI</key>
 <dict>
  <key>Scan</key>
  <true/>
 </dict>
 <key>SystemParameters</key>
 <dict>
  <key>InjectKexts</key>
  <string>Detect</string>
 </dict>
</dict>
</plist>

The resulting system is really cute, but not really usable because the graphics support is too bad. For example, the adjustments in the Colour Calibration preference pane have no effect. iMovie doesn't even start because the requirements aren't satisfied, and iPhoto doesn't show photos in single-photo or edit mode.

Intel DH87RL with i5-4670

On this board, using Intel's »Visual BIOS«, the VT for Directed I/O (VT-d) switch must be turned off in the Security settings for Yosemite to boot successfully.

I copied my existing configuration into the /EFI/CLOVER/OEM/DH87RL/ folder and removed the RealtekRTL81xx.kext from kexts/10.9.

The drive booted fine in BIOS mode, but in UEFI mode I got an "Error exiting boot services" from Clover. From a successful BIOS boot, I added OsxAptioFixDrv and that fixed the problem.

For the network card, this board requires the AppleIntelE1000e driver in the kexts/10.9 folder.

Results with this board and the i5-4670's integrated Intel HD4600 graphics are looking much better than with the HD7790 above: colour calibration and iPhoto just work, for example.

There is still one mysterious problem: "Shut Down" in the system menu has the same effect as "Restart"...

In order to dual-boot Windows with OS X, I used Clover's "Add Clover as EFI boot option", which created a "Firmware Application" entry for Clover visible in bcdedit /enum all. I made this the default boot entry using the command

bcdedit /set {fwbootmgr} displayorder {...} /addfirst
where the last item is the identifier value of Clover's entry. This motherboard doesn't actually show a boot menu with different options as suggested by the name displayorder, but simply boots the first entry in the list. The timeout value in the {fwbootmgr} section refers to the time that the Intel logo is shown.

Abandoned initial attempts for reference

Using a guide, I extracted BaseSystem.dmg onto a USB key and inserted the Packeges folder. I then used pkgutil to extract mach_kernel and dropped it in the root folder.

$ cd
$ xar -xf "/Volumes/OS X Install ESD/Packages/BaseSystemBinaries.pkg" Payload
$ pax -rjf Payload ./mach_kernel
$ mv mach_kernel "/Volumes/OS X Base System"
$ rm Payload

At this point, the new install medium should boot fine.

In order to enable booting on generic UEFI hardware, I downloaded FaceSMC.kext from tonymacx86 and copied it to /System/Library/Extensions on the installation medium.

Using RaspBMC 1.0 (Frodo) with bluetooth keyboard

Without having checked whether each step was actually necessary, this is how I got a Bluetooth keyboard working with RaspBMC 1.0 (Frodo). Mostly based on this installation and this setup.

SSH into your RaspBMC with username pi and password raspberry.

$ sudo su
# update-rc.d dbus defaults # prevents bluez installation failure
# apt-get install bluez python-gobject # gobject required by bluez-simple-agent

If the installation succeeded, the bluetooth service should be up and running (version bluez-4.99_2 in my case):

# service bluetooth status
[ ok ] bluetooth is running.

Otherwise it can be started with

# service bluetooth start

Put the keyboard into pairing mode and find out its address using

# hcitool scan
Scanning ...
        00:18:00:0A:F7:E3       BTKB-F7E3

With the keyboard still in pairing mode, I then attempted the pairing process with

# bluez-simple-agent hci0 00:18:00:0A:F7:E3
Creating device failed: org.bluez.Error.AlreadyExists: Already Exists

Apparently, the error indicates that the device is already paired (because I had previously used it on my PC), and the remedy is adding an arbitrary 3rd argument to the invocation.

# bluez-simple-agent hci0 00:18:00:0A:F7:E3 a
RequestPinCode (/org/bluez/1981/hci0/dev_00_18_00_0A_F7_E3)
Enter PIN Code: 1234
Release
New device (/org/bluez/1981/hci0/dev_00_18_00_0A_F7_E3)

I first entered the PIN code when prompted and had typed it on the keyboard followed by Enter before the line that says "Release".

I then created the entry to automatically pair the device after it disconnects:

# bluez-test-device trusted 00:18:00:0A:F7:E3 yes

This adds a line containing the device address to the file /var/lib/bluetooth/<adaptor address>/trusts.

Now connect the keyboard and load the necessary uinput module.

# bluez-test-input connect 00:18:00:0A:F7:E3
# modprobe -i uinput
# echo uinput >> /etc/modules

After restarting XBMC, the keyboard worked.

# restart xbmc

Booting GParted on MacBook Air 11" (late 2010)

I wanted to boot GParted-0.13.0-1 (kernel 3.2.0-3-486) from a USB stick on my MacBook Air 11" (late 2010) using the EFI bootloader which appears when holding down Option while turning on the laptop. However, only HFS+ partitions appear to be attached in a way that GRUB can see, but GParted's initrd cannot mount HFS+ (or even exFAT) partitions, saying hfsplus not found in modules.dep or exfat not found in modules.dep, respectively.

So I used Disk Utility to create a HFS+ as well as a FAT partition on the USB key and copied the contents of the GParted image into both partitions using

$ pax -rw

A possible optimisation would be to try and work out which files exactly are really required on each partition, but since GParted unly uses about 120MB space shouldn't be an issue even for small USB sticks.

Interestingly, whether the USB stick is partitioned using MBR or GPT doesn't make a difference.

After that, the MBA's EFI bootloader (hold down Option while turning on) shows two items called "EFI boot", the first of which should refer to the HFS+ partition. After booting, GParted's initrd then finds the required boot image on the FAT partition. GParted loads and runs successfully using default settings.

There was still another small hurdle before I could reset the SSD:

# hdparm -I /dev/sda
showed the SSD's Security state as frozen which disallows issuing the SECURITY_ERASE command. Googling revealed a trick to change the state to not frozen by putting the laptop to sleep and waking it up. Amazingly, simply issuing
# echo mem > /sys/power/state
and closing and pressing the shift key to wake it up worked perfectly. If you're working on a detachable disk, it also helps to only attach the disk once GParted has booted.

Note: I tried again using GParted 0.19.0-1 (kernel 3.14). I had to select "VGA" mode in the boot menu in order to make X start and waking up from sleep didn't work. So I used 0.13.0-1 instead.

Lucid Lynx on Sony Vaio P VGN-P19WN

After successfully running Xubuntu Karmic Koala 9.08 on the Sony Vaio P VGN-P19WN, the pleasant surprise after upgrading to Lucid Lynx was that the GMA500 »poulsbo« drivers were easy to install from the PPA repository. However, I was quite disappointed to discover that many things were broken after upgrading to Lucid Lynx (10.04):

  1. The brightness keys trigger the on-screen brightness display, but the display brightness doesn't change.
  2. After selecting »suspend« from the menu or closing the lid, the laptop suspends (to RAM) but cannot wake up.
Both features were working flawlessly before in Karmic.

The backlight starts working as soon as the kernel option acpi_backlight=vendor is removed from the variable GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub. I don't remember whether that option was there in Lucid Lynx by default or whether it was carried over from the previous Karmic Koala installation which needed that option.

After installing the uswsusp package and uninstalling the vbetool package (not needed, see below!), suspend works if calling

$ sudo s2ram -f
manually. The -f switch is needed because otherwise s2ram just displays the message
Machine is unknown.
This machine can be identified by:
    sys_vendor   = "Sony Corporation"
    sys_product  = "VGN-P19WN_Q"
    sys_version  = "J003KYE9"
    bios_version = "R2073U3"
However, setting SLEEP_MODULE="uswsusp" in /etc/pm/config.d/00sleep_module isn't enough to make the suspend work from the menu or by running pm-suspend from the command line. The suspend process looks the same as when calling s2ram manually, but the system does not wake up. So something that pm-suspend does before or after suspending using s2ram is breaking the wake-up.

There is also some advice on the web to deactivate the ath9k module, but this does not affect the Vaio P.

Deactivating all the scripts in /usr/lib/pm-utils/sleep.d by appending a ~ to their names also doesn't solve the issues. Turning them back on again one by one isolated the issue to 98video-quirk-db-handler (and occasionally 49bluetooth). So by adding these names to HOOK_BLACKLIST in a file in /etc/pm/config.d calling pm-suspend from the command line now works.

However, when pm-suspend is called using the Xfce interface or by closing the lid, an additional parameter --quirk-s3-bios is passed which again prevents resume from working. Also, lshal reports power_management.quirk.s3_bios = true. The reason turns out to be a double semicolon in a match clause in the file /usr/share/hal/fdi/information/10freedesktop/20video-quirk-pm-sony.fdi which adds the s3_bios to every product. The upstream bug report and fix.

While trying to manually restart hald I notice that Lucid Lynx no longer has a file /etc/init.d/hald, nor does the service command list or know about hal, hald or haldaemon. How to manually restart hald now escapes me. Shoddy user interface change!

Ironically, that same error was copy-and-pasted into the file /usr/lib/pm-utils/video-quirks/20video-quirk-pm-sony.quirkdb (turning into a double pipe symbol along the way), which was the reason I had to disable the module 98video-quirk-db-handler above. Upstream bug report. After deleting the spurious pipe symbol, I can enable the module again.

Also, it is not necessary to install the uswsusp package, the default kernel suspend method works fine. In that case, it might be necessary to unload and reload the ath9k module using SUSPEND_MODULES in a file in /etc/pm/config.d in case wireless networking frequently doesn't work after resume.

GNOME username login order

Current versions of gnome-simple-greeter which is part of the gdm package that manages the GNOME login screen display the users ordered by how frequently they log in to the system. This can be quite confusing as it is generally different from the obvious sorting criteria like user name or numeric user id.

It took a little while of digging around to find out how the login frequency is calculated. As it turns out it is a feature of ConsoleKit which can be accessed from the command line as

$ ck-history --frequent
In order to reset the count, it suffices to delete the file /var/log/ConsoleKit/history. By editing that file the login count can also be modified at will.

Debugging shared libraries

If an application crashes inside a shared library, you need source level debugging in that shared library to find the cause.

Let's assume the shared library is part of the base system. First of all, you need to build the library with debugging information. Unfortunately, MKDEBUG and MKDEBUGLIB don't help because they only affect applications and static libraries, respectively. Assuming you still have the obj tree of the last build around, the easiest way to build is by using the make wrapper it created:

$ MAKE="`echo /usr/obj/usr/src/tooldir*/bin/nbmake-amd64`"

As an alternative, you could also use normal make. This will use its own obj directory and might trip over missing dependencies as a result:

$ MAKE="make USETOOLS=no"

Now build the debugging library:

$ cd /usr/src/lib/libedit
$ $MAKE obj
$ $MAKE CFLAGS=-g distclean dependall

When loading the application into the debugger, you can use LD_LIBRARY_PATH to make sure that your debug version of the library is loaded. To run sh with our debugging version of libedit:

$ LD_LIBRARY_PATH="`$MAKE print-objdir`" gdb /bin/sh
(gdb) b el_gets
Function "el_gets" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
(gdb) r
Breakpoint 2, el_gets (el=0x7f7ffdb11800, nread=0x621b1c)
    at /usr/src/lib/libedit/eln.c:75
75  el->el_flags |= IGNORE_EXTCHARS;

Avoid timecounter "TSC" on modern PCs and laptops!

Most times when I boot NetBSD, the system time runs at only about half the real speed, meaning the time displayed by date is way behind. This would only be a nuisance if it didn't also mean that my e-mail client puts the wrong timestamps on every e-mail I send, which can be confusing or even embarrassing. Sometimes, very rarely, all is fine and the clock runs quite accurately.

I managed to capture the dmesg output for both cases, and this is the relevant difference:

--- ./dmesg.accurateclock       2009-09-08 16:20:59.000000000 +0100
+++ ./dmesg.clockslow   2009-09-07 09:49:39.000000000 +0100
@@ -12,14 +12,14 @@
 timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
 System manufacturer System Product Name (System Version)
 mainbus0 (root)
-cpu0 at mainbus0 apid 0: Intel 686-class, 2672MHz, id 0x106a5
-cpu1 at mainbus0 apid 2: Intel 686-class, 2672MHz, id 0x106a5
-cpu2 at mainbus0 apid 4: Intel 686-class, 2672MHz, id 0x106a5
-cpu3 at mainbus0 apid 6: Intel 686-class, 2672MHz, id 0x106a5
-cpu4 at mainbus0 apid 1: Intel 686-class, 2672MHz, id 0x106a5
-cpu5 at mainbus0 apid 3: Intel 686-class, 2672MHz, id 0x106a5
-cpu6 at mainbus0 apid 5: Intel 686-class, 2672MHz, id 0x106a5
-cpu7 at mainbus0 apid 7: Intel 686-class, 2672MHz, id 0x106a5
+cpu0 at mainbus0 apid 0: Intel 686-class, 3741MHz, id 0x106a5
+cpu1 at mainbus0 apid 2: Intel 686-class, 3741MHz, id 0x106a5
+cpu2 at mainbus0 apid 4: Intel 686-class, 3741MHz, id 0x106a5
+cpu3 at mainbus0 apid 6: Intel 686-class, 3741MHz, id 0x106a5
+cpu4 at mainbus0 apid 1: Intel 686-class, 3741MHz, id 0x106a5
+cpu5 at mainbus0 apid 3: Intel 686-class, 3741MHz, id 0x106a5
+cpu6 at mainbus0 apid 5: Intel 686-class, 3741MHz, id 0x106a5
+cpu7 at mainbus0 apid 7: Intel 686-class, 3741MHz, id 0x106a5
 ioapic0 at mainbus0 apid 8: pa 0xfec00000, version 20, 24 pins
 ioapic1 at mainbus0 apid 9: pa 0xfec8a000, version 20, 24 pins
 acpi0 at mainbus0: Intel ACPICA 20090730
@@ -170,7 +170,7 @@
 ieee1394if0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
 ieee1394if0: bus manager 0 (me)
 timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
-timecounter: Timecounter "TSC" frequency 2672861800 Hz quality 3000
+timecounter: Timecounter "TSC" frequency 3741967920 Hz quality 3000
 azalia0: codec[0]: ATI R600 HDMI (rev. 1.0), HDA rev. 1.0
 audio0 at azalia0: full duplex, independent
 azalia1: codec[0]: Realtek ALC888 (rev. 1.1), HDA rev. 1.0
This shows that the processor clock is detected differently in the two cases: in the cases when my clock runs slow, the processor is detected to be running at 3.7GHz, otherwise it's 2.6GHz. As a consequence of misdetecting the processor clock, the kernel also assumes the incorrect frequency for the "TSC" timecounter.

The reason this happens is that my CPU is an Intel i7 which uses the a technology called Turbo Boost. Basically, when there isn't much work to do the processor gets clocked down to save energy. Processors designed for mobile devices have been doing this for quite a while.

The manual page timecounter(9) indicates that it originated on FreeBSD, and in fact I found the solution to my problem in the FreeBSD FAQ.

Luckily, the "TSC" timecounter isn't my only timing source on this machine. As a matter of fact, the kernel variable kern.timecounter.choice offers plenty of such:

kern.timecounter.choice = TSC(q=3000, f=3741919640 Hz)
        clockinterrupt(q=0, f=100 Hz) hpet0(q=2000, f=14318179 Hz)
        ACPI-Fast(q=1000, f=3579545 Hz) lapic(q=-100, f=133687675 Hz)
        i8254(q=100, f=1193182 Hz) dummy(q=-1000000, f=1000000 Hz)
NetBSD has even already gone to the trouble of rating the quality of each of these, so all I need to do is to pick the one with the next lower value:
# sysctl -w kern.timecounter.hardware=hpet0
kern.timecounter.hardware: TSC -> hpet0
This seems to have solved the problem.

I think the code for the "TSC" timecounter should either be adapted to modern hardware with changing processor speeds. At the very least, the quality rating should be lowered significantly if such hardware is detected so it isn't chosen as the default time source.

Building new packages from scratch

Today I tried yet another way to upgrade my packages. Since it was recently fixed and after thinking more about Hubert's upgrade procedure I decided to give mk/bulk/mksandbox another chance. For more discussion about upgrading packages, also see the NetBSD wiki. I have also described a method to upgrade only packages that changed.

This time, I wanted to skip the step of building packages and installing them later. Instead, I decided to build the packages in a sandbox on my /usr partition and simply move the resulting /usr/pkg, /var/db/pkg and /var/db/pkg.refcount directories into the right place when finished. This way, I get the benefit of having a working system while building the new one and being able to switch over to the new packages in one swell foop when finished. This requires the build environment in the sandbox to be fairly similar to the real system: in particular, the user and group ids in the sandbox must match the ones in the real system. Luckily, this is exactly the way that mk/bulk/mksandbox sets up the sandbox, so nothing special needs to be done.

Because all packages are rebuilt during the following upgrade procedure, there are no problems when dynamic libraries in the base system are upgraded. This happens fairly frequently in my case because the base system is NetBSD-current.

To update, I first get a new pkgsrc tree:

$ cd /usr/pkgsrc
$ mv cvs.up cvs.up.0
$ cvs up -A 2>&1 | tee cvs.up
$ grep ^[^cUP] cvs.up
Then I generate a pkg_chk configuration file of my required packages:
$ pkg_leaves -a | \
>       while read pkg; do
>               pkg_info -Q PKGPATH "$pkg";
>       done > /usr/pkgsrc/pkgchk.conf
or without pkg_leaves:
$ pkg_info -e "*" | \
>       while read pkg; do
>               if [ -z "`pkg_info -q -R "$pkg"`" ]; then
>                       pkg_info -Q PKGPATH "$pkg";
>               fi;
>       done > /usr/pkgsrc/pkgchk.conf
Don't fall into the trap of trying to use pkg_info -u for this purpose: it uses the package variable "automatic" which gets set to a completely random value in my experience. Instead, the above script prints all packages that are not depended on like in pkg/32827. This speeds up the following build procedure by reducing the number of lines in pkgchk.conf since each line of pkgchk.conf gives rise to an execution of make clean CLEANDEPENDS=yes.

I then save all my binary packages and clean any obsolete distfiles using lintpkgsrc.

$ rm -r packages-old
$ mv packages packages-old
$ lintpkgsrc -mopr

I also clean any stale working directories that might still be around:

$ rm -r /usr/pkgsrc/*/*/work

The only package that needs to be installed in the »host« environment is pkgtools/mksandbox. Note that it has to be called with an absolute path or bad things will happen! This is how I build a new sandbox in my home directory and jump inside:

$ sudo mksandbox $HOME/sb
$ cd $HOME/sb
$ sudo sh sandbox chroot
#
I then set up the /etc/mk.conf file with my desired settings for the build:
.ifdef BSD_PKG_MK
# Setting WRKOBJDIR is inconvenient for development:
#WRKOBJDIR=     /usr/obj/usr/pkgsrc
PKG_DEVELOPER=  yes

UPDATE_TARGET=  bin-install
BINPKG_SITES=   # prevent bin-install from accessing ftp
USE_DESTDIR=    yes
MAKE_JOBS=      8
SKIP_LICENSE_CHECK=yes

# Make Firefox look like Firefox
PKG_OPTIONS.firefox+= official-mozilla-branding
PKG_OPTIONS.gecko+=   official-mozilla-branding
PKG_OPTIONS.thunderbird+=official-mozilla-branding

PKG_DEFAULT_OPTIONS+= gnome

ALLOW_VULNERABLE_PACKAGES=yes

.if exists(/usr/pkg/bin/sudo)
SU_CMD=        /usr/pkg/bin/sudo /bin/sh -c
.endif
.endif

The first package that needs to be installed is pkg_chk because it is in charge of the following procedure.

# cd /usr/pkgsrc/pkgtools/pkg_chk
# make update
Don't do this, it breaks at least textproc/icu: I then tend to install pkgtools/autoswc which provides a cache for configure. This is useful because these tests are typically performed by every package just to produce the same answers to identical questions.

Now, all that is left to do is to start the build and get some coffee while it's working away:

# pkg_chk -akr 2>&1 | tee /usr/obj/pkgchk-akr.0.log

After the build is finished, the sandbox can be unmounted:

# ^D
$ sudo sh sandbox umount

It's probably worth checking for any special configuration in /usr/pkg/etc and merge differences you care to keep into the new tree:

$ diff -ur /usr/pkg/etc $HOME/sb/usr/pkg/etc

Also, you should check for any changes to files in /etc and apply them to your system's configuration. In particular, many packages set up users and groups that you have to copy with identical id's as in the chroot in order to match the owners of the installed files.

$ diff -ur /etc $HOME/sb/etc

Then I switch into single-user mode and exchange my /usr/pkg, /var/db/pkg and /var/db/pkg.refcount directories with the newly built ones:

$ sudo shutdown now
# mv /usr/pkg /usr/pkg-old
# mv /var/db/pkg /var/db/pkg-old
# mv /var/db/pkg.refcount /var/db/pkg.refcount-old
# mv $HOME/sb/usr/pkg /usr/pkg
# mv $HOME/sb/var/db/pkg /var/db/pkg
# mv $HOME/sb/var/db/pkg.refcount /var/db/pkg.refcount
# ^D
At this point, the system should boot back into multi-user mode using the new packages. If anything goes wrong, it's fairly easy to revert back to the old packages using the backups. For this to work properly it's essential that the sandbox directories are on the same partition as their final locations: if they're not, moving the created files while preserving hard links and the like is best done using pax -rw.

It can't hurt to read the install messages for the installed packages using

$ pkg_info -Da
and do as they suggest (create users and groups, copy files from /usr/pkg/share/examples/rc.d to /etc/rc.d etc.)

After everything works to satisfaction,

# rm -fr $HOME/sb
# rm -fr /usr/pkg-old
# rm -fr /var/db/pkg-old
# rm -fr /var/db/pkg.refcount-old