Tuesday, 10 April 2018

Understanding and debugging fontconfig font fallback on macOS

Recently, I used an app on macOS that uses fontconfig. While this filled me with an appropriate sense of dread, I encountered an issue which forced me into debugging its font selection process.

Unfortunately, the app could not display certain symbols I was using (⚭, ⚯) even though my system had fonts providing those glyphs. Using FontBook.app, I found out that the two glyphs are contained in Apple Symbols.

In order to debug the issue, I used fc-match as follows.

$ fc-match -s :charset=26ad # ⚭
LastResort.otf: ".LastResort" "Regular"
$ fc-match -s :charset=26af # ⚯
LastResort.otf: ".LastResort" "Regular"

This result shows us that fontconfig doesn't find a font for the requested glyphs and returns the LastResort font which instead provides replacement glyphs with the Unicode character code inside them.

Knowing that a font was actually available for those glyphs, I started debugging the issue:

$ FC_DEBUG=4 fc-match -s :charset=26ad
FcConfigSubstitute donePattern has 5 elts (size 16)
        family: "Arial"(s) […]
        hintstyle: 1(i)(w)
        0026: 00000000 00000000 00000000 00000000 00000000 00002000 00000000 00000000(s)
        lang: "en"(w)
        prgname: "fc-match"(s)


The FC_DEBUG variable selects »match/test/edit execution« debug output.

This shows the process by which fontconfig builds up the pattern that it uses to search for a font. The string donePattern indicates the final step which is used to find a font. The output shows the list of font families that are considered and the remaining criteria, specifically the hintstyle, the required charset and the language.

Digging deeper, I wanted to find out what information fontconfig has about the Apple Symbols font.

$ FC_DEBUG=2 fc-match -s :charset=26ad
Font 7 Pattern has 22 elts (size 22)
        family: "Apple Symbols"(w)
        familylang: "en"(w)
        0026: ffffffff ffffffff ffffffff ffffffff 1fffffff 0007ffff 00000000 00000000
        lang: el|fj|ho|ia|ie|io|nr|om|sm|so|ss|st|sw|to|ts|uz|xh|zu|kj|kwm|ms|ng|rn|rw|sn|za(w)

Here, the FC_DEBUG setting requests »extensive font matching information«.

This snippet shows that the font does indeed contain the requested glyph. However, for some reason fontconfig has not assigned English (en) to the list of languages with which Apple Symbols can be used. I assume that this is the result of some algorithm based on the characters contained in the font. According to FontBook.app, the font itself definitely claims to support English, among many other languages:

Asu, Bemba(bem), Bena(bez), Chiga, Cornish(kw), English(en), Greek(el), Gusii(guz), Indonesian(id), Kalenjin(kln), Kinyarwanda(rw), Luo(luo), Luyia(luy), Machame, Makhuwa-Meetto(mgh), Makonde(kde), Malay(ms), Morisyen, North Ndebele(nd), Nyankole(nyn), Oromo(om), Rombo, Rundi(rn), Rwa, Samburu(saq), Sangu, Shambala(ksb), Shona(sn), Soga(xog), Somali(so), Swahili(sw), Taita(dav), Teso(teo), Uzbek(uz), Vunjo(vun), Zulu(zu)

In order to fix this problem, I added a section to my ~/.config/fontconfig/fonts.conf which assigns the correct language codes to the fontconfig cache for the Apple Symbols font.

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
    <!-- Apple Symbols loses some language tags and just ends up with
         el, fj, ho, ia, ie, io, nr, om, sm, so, ss, st, sw, to, ts, uz,
         xh, zu, kj, kwm, ms, ng, rn, rw, sn, za.

         Full list taken from FontBook.app:

         Asu, Bemba(bem), Bena(bez), Chiga, Cornish(kw), English(en),
         Greek(el), Gusii(guz), Indonesian(id), Kalenjin(kln),
         Kinyarwanda(rw), Luo(luo), Luyia(luy), Machame,
         Makhuwa-Meetto(mgh), Makonde(kde), Malay(ms), Morisyen, North
         Ndebele(nd), Nyankole(nyn), Oromo(om), Rombo, Rundi(rn), Rwa,
         Samburu(saq), Sangu, Shambala(ksb), Shona(sn), Soga(xog),
         Somali(so), Swahili(sw), Taita(dav), Teso(teo), Uzbek(uz),
         Vunjo(vun), Zulu(zu)
    <match target="scan">
        <test compare="eq" name="family">
            <string>Apple Symbols</string>
        <edit binding="same" mode="assign_replace" name="lang">

Note that properties of type langset do not support appending, so the full resulting set needs to be specified. Also, the replacement needs to take place during »scanning« when the cache is built in order to affect the information stored for the font. As a consequence, the fontconfig cache must be force-rebuilt because fc-cache doesn't consider changes to the configuration when it decides whether to rebuild a cache.

$ fc-cache -f

Now, fc-match should give the following output:

$ fc-match -s :charset=26ad
Apple Symbols.ttf: "Apple Symbols" "Regular"
LastResort.otf: ".LastResort" "Regular"

Additionally, I also added the following section to my fonts.conf which adds Apple Symbols as a fallback font for the sans-serif family. This slightly increases the likelihood of it being picked. The fact that it was already picked up beforehand shows that when it is really desparate to find a glyph, fontconfig considers all the fonts it knows about and not just ones with a matching family.

    <!-- Consider glyphs from Apple Symbols, e.g. U+26ad, U+26af. -->
            <family>Apple Symbols</family>

In the end, I discovered that my app still did not work because it turned out that it wasn't actually using Pango Cairo's fontconfig backend but the CoreText backend instead. After changing this using PANGOCAIRO_BACKEND=fontconfig I was now able to display the glyphs I needed.

Tuesday, 6 March 2018

Finding a build tool

As someone who actually thinks that make(1) is a wonderful tool if used correctly, that's what I've normally been using. I'm talking about BSD make, mind you, not the abomination that is GNU make.

For those who are unaware, a typical Makefile looks somewhat like this:

PROG_CXX=       hello

.include <bsd.prog.mk>

Since I also take note of my environment, it hasn't escaped me that that's not what most people seem to be doing nowadays. While I don't have much time for people whining about make(1) given its clear, straightforward and problem-oriented syntax, I do believe there is room for improvement and I'm curious what people have been coming up with in the decades since make's inception.

One of the most widely used tools nowadays appears to be CMake. However, looking at the syntax of CMake's language reminds me mostly of Tcl, and that's not a good thing. Passing sequences of strings as arguments where certain strings modify the interpretation of the strings that come after them is not just a glaring lack of design but also neither fault tolerant nor easy to read. It is also based on the idea of (badly) generating code which I find revolting. As if that wasn't bad enough, it also clutters the source tree with cache files and other clutter which is unacceptable for someone refusing to go down the .*ignore route. Next.

The next big candidate that comes up is Boost.Build. Interestingly, the Jam language suffers almost identical shortcomings as CMake's with lists of strings and certain keywords modifying their semantic, making me almost believe that this might be endemic to people designing build systems. Fail.

Another popular option that comes up is Gradle. It requires a JDK in order to run the build process. What is more, it has a feature matrix showing which platforms and compilers it supports – how is that even relevant!? No thanks.

The search continues…

To be evaluated (in no particular order apart from alphabetical):

Saturday, 6 January 2018

IPsec VPN for macOS High Sierra and Android Nougat

After switching to OpenWRT from DD-WRT which doesn't support IPsec, this is post describes the settings I use.

Since Android (at least up to Nougat) does not support IKEv2, we will use a certificate-based IKEv1 setup (IPsec Xauth RSA in Android speak).

strongSwan configuration

The configuration is taken from strongSwan's usable examples.

# /etc/ipsec.conf - strongSwan IPsec configuration file

config setup
        #charondebug="asn 2, cfg 2, chd 2, dmn 2, enc 1, esp 2, ike 2, imc 2, imv 2, job 2, knl 2, lib 2, mgr 2, net 2, pts 2, tls 2, tnc 2"

conn %default




        # Android Nougat native client.
        # IKE:AES_CBC_256/HMAC_SHA2_384_192/PRF_HMAC_SHA2_384/MODP_1024
        # macOS High Sierra Cisco IPsec.
        # IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048

        # For some reason, data cannot be transferred using Android's second suggestion.
        # ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ
        # ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ

conn pubkey

conn xauth-pubkey
        # Android's native client requires XAuth in addition to certificates.

conn xauth-hybrid

conn ipsecuritas
        # IPSecuritas apparently needs a specific IP or the CHILD_SA cannot be found.

In order to prevent routing headaches, it's advisable to assign addresses to IPsec clients (rightsourceip) from a different subnet than the LAN ( in this example).

I did not set up IKEv2 connections (for use on macOS, for example) because that would require installing additional modules for EAP, such as eap-md5 or eap-tls.

# /etc/ipsec.secrets - strongSwan IPsec secrets file

: RSA moonKey.pem
client : XAUTH "password"

I have chosen the legacy ipsec configuration in favour of swanctl because the latter is a much bigger packet to install.

Firewall configuration

Confusingly, OpenWRT /etc/config/firewall already has a couple of rules for IPsec (Allow-IPSec-ESP and Allow-ISAKMP), but another important one is missing. Moreover, as of this writing these rules handle IPsec forwarding to the LAN. In order for the router to be able to receive IPsec traffic, the option dest lan must be dropped. Here's the result:

config rule
        option name             Allow-IPSec-ESP
        option src              wan
        #option dest             lan
        option proto            esp
        option target           ACCEPT

config rule
        option name             Allow-ISAKMP
        option src              wan
        #option dest             lan
        option proto            udp
        option dest_port        500
        option target           ACCEPT

config rule
        option name             Allow-NATT
        option src              wan
        option proto            udp
        option dest_port        4500
        option target           ACCEPT

The WAN interface needs to accept packets unwrapped from IPsec tunnels which can be recognised by their »ipsec« policy using the »policy module«. In order for clients connected via IPsec to be reachable from the LAN, packets bound for an IPsec tunnel must not be subjected to NAT. This can be accomplished using the following rule in /etc/firewall.user, for example:

# Accept tunnelled traffic.
iptables \
        -t filter \
        -A input_wan_rule \
        -m policy \
        --dir in \
        --pol ipsec \
        --mode tunnel \
        -j ACCEPT \

# Prevent NATting traffic bound for IPsec tunnel.
iptables \
        -t nat \
        -A postrouting_wan_rule \
        -m policy \
        --dir out \
        --pol ipsec \
        --mode tunnel \
        -j ACCEPT \


Activating the line charondebug in /etc/ipsec.conf (above) or adding the following section to /etc/strongswan.d/charon-logging.conf is helpful when debugging issues:

charon {
    syslog {
        daemon {
            default = 1
# logread -f

The following commands can help with routing and firewall debugging:

# ip -f inet route list table all
# iptables -t [filter|nat] -L -v
# iptables -t filter -I reject --log-prefix 'reject ' -j LOG
# /etc/init.d/firewall stop

Other settings

In order to provide DNS to VPN clients, the DNS server must not be configured as »local only« (localservice=0).

config dnsmasq
        option domainneeded      1
        option rebind_protection 1
        option rebind_localhost  1
        option local             /lan/
        option domain            lan
        option authoritative     1
        option readethers        1
        option leasefile         /tmp/dhcp.leases
        option resolvfile        /tmp/resolv.conf.auto
        option nonwildcard       1
        option localservice      0
        option add_local_fqdn    3

Key and certificate generation

Generating certificates requires strongswan-pki which is a separate install. First, generate the key for your certificate authority.

# ipsec pki --gen --outform pem >caKey.pem

The CA key is particularly sensitive and must never leak. Next, create a certificate for your certificate authority.

# cacert=/etc/ipsec.d/cacerts/caCert.pem
# ipsec pki --self \
        --in caKey.pem \
        --dn 'C=CH, O=rienajouter, CN=rienajouter CA' \
        --ca \
        --outform pem \

The »distinguished name« for every certificate you create must be unique – otherwise the certificates will be considered interchangeable and authentication will fail. Next, create a key and certificate for the VPN server.

# key=/etc/ipsec.d/private/serverKey.pem
# cert=/etc/ipsec.d/certs/serverCert.pem
# ipsec pki --gen --outform pem >"$key"
# ipsec pki --pub --in "$key" \
| ipsec pki --issue \
        --cacert "$cacert" \
        --cakey caKey.pem \
        --dn 'C=CH, O=rienajouter, CN=vpn.example.com' \
        --san 'vpn.example.com' \
        --flag serverAuth \
        --flag ikeIntermediate \
        --outform pem \

For the VPN server certificate to work reliably with macOS, the CN of the distinguished name as well as the subject alt name (--san) must match the DNS name of the VPN gateway and the ikeIntermediate flag must be present. Windows on the other hand requires the serverAuth flag.

Last, for each client create a certificate to be shared with the client.

# key=/etc/ipsec.d/private/clientKey.pem
# cert=/etc/ipsec.d/certs/clientCert.pem
# ipsec pki --gen --outform pem >"$key"
# ipsec pki --pub --in "$key" \
| ipsec pki --issue \
        --cacert "$cacert" \
        --cakey caKey.pem \
        --dn 'C=CH, O=rienajouter, CN=client' \
        --outform pem \

Generating keys and certificates in PEM format helps when exporting them to the clients.

Doing the same with openssl is a nightmare because openssl was created in order to prevent the widespread use of cryptography, and quite successfully so far.

$ openssl req \
        -new \
        -newkey rsa:2048 \
        -subj '/C=CH/O=rienajouter/CN=client' \
        -keyout "$key" \
        -nodes \
| openssl x509 \
        -req \
        -CA "$cacert"  \
        -CAkey caKey.pem \
        -sha256 \
        -days $((3 * 365)) \

In case you forget, here's how to find out what's inside one of the key or certificate files:

$ ipsec pki --print --in "$cert"
$ openssl rsa -in "$key" -text -noout
$ openssl x509 -in "$cert" -text -noout

Certificate distribution

In order to distribute the certificates generated in the previous step, they must be packaged together into PKCS#12 files. The command to do so is:

$ openssl pkcs12 \
        -certfile "$cacert" \
        -inkey "$key" \
        -in "$cert" \
        -export \
        -out client.p12 \

When exporting for macOS, make sure to use a non-empty password when prompted by the above command. After importing the CA and client certificates into the system keychain on macOS, set the CA certificate's »trust« for »IPsec« to »always trust«. You also need to grant access to the private key inside the client certificate to /usr/sbin/racoon. The »access control« tab is only shown for keys, which in turn only appear when selecting »keys« or »certificates« in the left-hand pane of »Keychain Access«.