flowblok’s blog

Advent of Code 2023: Day 6

2024-04-13T00:00:00+10:00

Hurl is a dynamically-typed programming language that uses exception handling as its only form of control flow. Much of the syntax is unsurprising, but the forms of exception raising and handling deserve close examination.

Here is the canonical way to do conditional branches in Hurl:

try {
    hurl 1 < 2;
} catch (true) {
    println("hooray");
} catch (false) {
    println("what weird machine are you running on this on?");
};

In this example, the hurl keyword raises an exception with the value true, which then is matched by the catch (true) block. There are three forms of the catch block:

catch (<expr>) { ... } will match if the exception value evaluates equal to <expr>¹
catch as <identifier> { ... } will match any value and make it available as an <identifier> variable in the catch body
catch into <identifier> will match any value and assign it to an already-declared variable <identifier>

Hurl has functions, but since we have exceptions, there is no need for that pesky return keyword: we can just hurl the value we want to return:

let add = func(x, y) {
    hurl x + y;
};

try {
    add(3, 4);
} catch as result {
    println(result);
};

There is a second way to throw exceptions: if you toss a value, it will be match against the parent catch blocks, but when that block calls return² execution will resume from after the toss. Crucially, toss can be called multiple times to invoke the parent catch block as many times as desired, which gives us a tool to build continuations.

Making Hurl even more exceptional

Hurl is an exceptional language, but some parts seem unexceptional to me. For example, we only really need the let <ident> part of let <ident> = <value>, since we can write:

# Today's Hurl:
let add = 0;
# Tomorrow's Hurl?
#let add;

try {
    hurl func(x, y) {
        hurl x + y;
    };
} catch into add;

Since the value part is unnecessary and the language is dynamically-typed, I will restrict myself to only ever assigning the value 0 with let, with the hope that this is removed from the language in the future.

Another unexceptional part of the language is the collection of built-in functions for I/O and manipulating numbers, strings and lists. Having such a collection is necessary to make the language practical to use, but they currently evaluate as expressions rather than hurl'ing their results—something that user-defined functions cannot do.

Fortunately, it is not too difficult to build a compatibility layer to remove their special status. Wherever you see FOO_proper, this is a version of the built-in function FOO that hurl's the result rather than evaluating to it. For example:

let read_file_proper = 0;

try {
    hurl func(filename) { hurl read_file(filename); };
} catch into read_file_proper;

Finally, there is no need for builtin comparison operators in the language. We can make these truly exceptional with a three-way comparison operator for numbers:

let lt = 0;
let eq = 0;
let gt = 0;
let cmp = 0;

# Some enums or list unpacking would be neat...
try { hurl 0; } catch into lt;
try { hurl 1; } catch into eq;
try { hurl 2; } catch into gt;

# cmp(a, b) hurls lt (if a < b), eq (if a == b) or gt (if a > b).
try {
    hurl func(a, b) {
        try {
            hurl a < b;
        } catch (true) {
            hurl lt;
        } catch (false) {
            try {
                hurl a > b;
            } catch (true) {
                hurl gt;
            } catch (false) {
                hurl eq;
            };
        };
    };
} catch into cmp;

And for strings (which do not support lexicographic comparison in Hurl) we need an explicit equality function:

let equals = 0;

try {
    hurl func(a, b) {
        hurl a == b;
    };
} catch into equals;

Wait, what was I doing again?

Oh, that’s right, we’ve got an Advent of Code problem to solve. Today we have data about some boat races. Each race has a duration: we can use some portion of this duration to charge our boat, and then we spend the remainder of the duration moving at a speed determined by how long we charged our boat for. If we hold the button for hold of the duration milliseconds, then we will travel hold * (duration - hold) millimeters.

If we wanted to maximise that distance, we'd want the two parts of that multiplication to be as close to each other as possible, so hold would be either floor(duration / 2) or ceil(duration / 2). But that's not our problem: we want to know how many different ways we could beat a competitor who achieved a certain distance (target).

We could do maths to find an analytical solution, but that sounds like work, so let's not. We could also just evaluate every possible hold value, but it seems likely that Part 2 of the problem will make the problem bigger or more complicated, and maybe that would be too slow (but maybe not?). Or we could implement binary search. That sounds more fun.

In order to binary search, we make use of the half-way point: we can split our problem into the intervals [0, floor(duration / 2)] and [ceil(duration / 2), duration]: hold * (duration - hold) will be strictly increasing for the first interval, and strictly decreasing for the second. So we can binary search the first interval for the smallest value of hold where hold * (duration - hold) > target (we start beating the competitor), and the second interval for the smallest value where hold * (duration - hold) <= target (we stop beating the competitor). The difference between these hold values is the number of ways we can beat them.

So anyhow, we need an implementation of binary search. Here’s one using the loop function from Hurl’s standard library:

let binary_search = 0;

try {
    hurl func(lower, upper, fn) {
        try {
            loop(func(locals) {
                try { hurl locals.1; } catch as lower {
                try { hurl locals.2; } catch as length {
                try { hurl floor(length / 2); } catch as half_length {
                try { hurl lower + half_length; } catch as mid {
                try {
                    fn(mid);
                } catch (true) {
                    toss [lower, half_length];
                    try {
                        cmp(half_length, 0);
                    } catch (eq) {
                        hurl false;
                    } catch as ne {
                        hurl true;
                    };
                } catch (false) {
                    length = length - half_length + 1;
                    toss [mid + 1, length];
                    try {
                        cmp(length, 0);
                    } catch (eq) {
                        hurl false;
                    } catch as ne {
                        hurl true;
                    };
                }; }; }; }; };
            }, [lower, upper - lower]);
        } catch as result {
            hurl result.1;
        };
    };
} catch into binary_search;

On each iteration, loop invokes the function in its first argument with the parameter locals, which starts as the second argument ([lower, upper - lower]). Each time the function is invoked, it should toss a new value which will replace locals for the next invocation, and hurl either true or false to indicate if the loop should continue. Finally, loop will hurl locals so the caller can produce a result.

This example also shows off our three-way comparison operator³: we catch eq explicitly, but then aggregate lt and gt into a variable (the name doesn’t matter, but ne aids readability). You will also notice a bunch of faces that look like they want to Hurl (}; }; }; }; };). These are from using catch as blocks in sequence, to avoid the use of let.

Anyhow, we define a distance function and implement lower and upper as functions which do the comparisons used by the binary search, find the two values of hold and subtract them.

let distance = 0;
let ways_to_beat = 0;

try {
    hurl func(hold, duration) {
        hurl hold * (duration - hold);
    };
} catch into distance;

try {
    hurl func(duration, target) {
        try {
            hurl func(hold) {
                try {
                    distance(hold, duration);
                } catch as d {
                    try {
                        cmp(target, d);
                    } catch (lt) {
                        hurl true;
                    } catch as ge {
                        hurl false;
                    };
                };
            };
        } catch as lower {
        try {
            hurl func(hold) {
                try {
                    distance(hold, duration);
                } catch as d {
                    try {
                        cmp(target, d);
                    } catch (lt) {
                        hurl false;
                    } catch as ge {
                        hurl true;
                    };
                };
            };
        } catch as upper {
        try {
            binary_search(0, floor(duration / 2), lower);
        } catch as l {
            try {
                binary_search(ceil(duration / 2), duration + 1, upper);
            } catch as u {
                hurl u - l;
            };
        }; }; };
    };
} catch into ways_to_beat;

That’s the core of the problem, but we also need to parse the input and multiply together the results of multiple races. The second part is fairly straightforward, but parsing the input is a little more tricky, since we have numbers with any number of spaces between them:

Time:      7  15   30
Distance:  9  40  200

There is a split function in the standard library, but it will turn "7 15" into ["7", "", "15"], so we need to be able to remove empty elements. All that’s needed for this is filter(xs, pred) (returns a new list of the items in xs for which pred(item) returned true), but I thought I might need map later⁵, so implemented map_filter_nil(xs, fn) (returns a new list of fn(item) for each item in xs, skipping items when fn returns nil). Unfortunately, while nil is a valid value at runtime, it’s not part of the language syntax, so we need to jump through a hoop to make it accessible:

let nil = 0;
try {
    try {
        hurl func() {};
    } catch as nil_producer {
        hurl nil_producer();
    };
} catch into nil;

And then we can implement map_filter_nil. It might be possible to write this better using one of the other loop functions in the standard library, but I didn’t immediately see anything which would work.

let map_filter_nil = 0;

try {
    hurl func(xs, fn) {
        try {
            loop(func(locals) {
                try {
                    at_proper(xs, locals.1);
                } catch as item {
                    try {
                        try {
                            fn(item);
                        } catch (nil) {
                            hurl locals.2;
                        } catch as value {
                            hurl locals.2 + [value];
                        };
                    } catch as new_list {
                        toss [locals.1 + 1, new_list];
                        try {
                            cmp(locals.1, len(xs));
                        } catch (lt) {
                            hurl true;
                        } catch as ge {
                            hurl false;
                        };
                    };
                };
            }, [1, []]);
        } catch as rv {
            hurl rv.2;
        };
    };
} catch into map_filter_nil;

And then we can construct a split function that drops empty values:

let split_ignoring_whitespace = 0;

try {
    hurl func(string, sep) {
        try {
            split(string, sep);
        } catch as rv {
            map_filter_nil(rv, func(s) {
                try {
                    equals(s, "");
                } catch (false) {
                    hurl s;
                } catch as ne {
                    hurl nil;
                };
            });
        };
    };
} catch into split_ignoring_whitespace;

Part 2

My guess from earlier turned out correct: Part 2 makes the numbers bigger by combining the races into one (in the example above, "7 15 30" is read as 71530 and "9 40 200" as 940200). These numbers don’t seem that big though, so I suspect trying every hold value is perfectly viable.

So there were some minor changes to input parsing, but nothing particularly interesting.

Anyhow, I recommend having a read of Hurl’s documentation, there are a bunch of interesting examples that demonstrate the capabilities of exceptional control flow. And if you’re the kind of person who thinks this is all very sensible, I can also recommend SIGBOVIK, whose 2024 proceedings are how I discovered this language.

As usual, a complete code listing is available in the repository.

The documentation says that it merely matches literals, but the implementation handles expressions, which seems intentional. ↩
There are some bugs (features?) in the implementation, so I can get this working without doing this. Also, hurl and toss behave slightly differently when invoked from a function within a try block than if inlined in that block. None of this really matters, if you hold the language as intended, it works fine. ↩
This example could also be achieved with equals, but you would still need two catch blocks to invert the boolean⁴. ↩
Or I guess you could use the decidedly unexceptional unary not operator. ↩
I didn’t. Oh well. ↩

Advent of Code 2023: Day 5

2024-03-16T00:00:00+11:00

My language choice for Day 5’s puzzle is brought to you by the wonderful combination of a what if spark of inspiration and a concerning amount of stubbornness. The inspiration was this: what if we could make a text adventure game where the puzzle input resulted in the game computing the correct output?

Inform 7 is a domain specific language for creating interactive fiction, where the source code is largely written in natural language, though punctuation and indentation is structurally significant, and (unsurprisingly) sentences often need to be written in specific ways to avoid ambiguity.

Part 1

Our input is structured as a list of seeds, followed by a number of maps that transform the seed numbers to soil numbers, to fertilizer numbers, and so on. The types of things don’t really matter, so all we need to do is read the seed list, then (repeated seven times) read a map and transform the numbers, and finally output the smallest value.

Numbers

There is an unfortunate complication though. The actual input has unsigned 32-bit integers, but the best native integer type Inform offers is signed 32-bit (targeting Glulx) or 16-bit (targeting Z-machine) integers. There’s a few options for working around this:

Restrict to Glulx, use signed 32-bit integers, representing 0 as -2³¹, up to 2³²-1 as 2³¹-1. This lets us use signed comparison, addition and subtraction, but would require custom parsing and display routines. The representation is also somewhat annoying to understand when debugging.
Restrict to Glulx, combine the unsigned parts of two 32-bit integers to implement up to 62-bit unsigned integers. This would require custom operators for comparison, addition, subtraction, parsing and display. The representation is also somewhat annoying to understand when debugging.
Combine the unsigned parts of at least three 16-bit integers to implement up to 45-bit unsigned integers. Again, requires implementing everything, and the representation is also not suited to debugging.
Use a string of the decimal representation. This does not require parsing and display routines¹, but would require custom comparison, addition and subtraction routines. This is likely the least performant option, but does have the advantage of being arbitrary precision and having an easily understood representation when debugging.

Since performance is not a concern (and because, as far as I can tell, bitwise operators are only available by embedding Inform 6 code), I went with the last option.

Reading the seeds list

Let’s get onto some code:

"Advent of Code 2023 Day 5.1" by flowblok

Chapter 0 - The Setting

The Garden is a room.

This is just a preamble, specifying the title and author of the story. Inform 7 source code can be organised by a hierarchy of headings (volumes, books, parts, chapters and sections), primarily to improve navigation of the source. Inform 7 organises the world into rooms, so we need at least one room (for the player to start in), so we make one called "The Garden".

Chapter 1 - On the Identification of Seeds

Defining the seed list is an action applying to one topic.
Understand "seeds [text]" as defining the seed list.
The seed list is a list of text which varies.

Carry out defining the seed list:
    now the seed list is the words of the topic understood.

There’s a couple of things going on here: firstly, we define an action called "defining the seed list", which applies to (has a parameter of) one topic (some text). We also need to specify how the player can trigger this action, so we configure Inform 7’s player command parsing to understand that seeds [text] should invoke that action (with everything after the word seeds as the topic). We then define a global variable called "the seed list" which is a list of text (since we’re using the decimal representation for our numbers). And finally, we specify what should happen when the "defining the seed list" action is carried out: now the seed list is the words of the topic understood would be written in traditional programming languages something like seed_list = input_topic.split(" ").

There’s no input validation here, we’re just going to assume that the player actually entered numbers. Keen readers may also note that the puzzle input looks like "seeds: 79 14 55 13", but our parsing only works with "seeds 79 14 55 13". Unfortunately, Inform assumes that the colon means we’re asking something called "seeds" to perform the "79" action on "14 55 13"², which is not very easy to make work for our processing. To work around this, we can hook into the command parsing and remove the colons before the parser tries to understand what action is being invoked on what:

Section - Input Sanitization

After reading a command:
    let T be "[the player's command]";
    replace the regular expression ":" in T with "";
    change the text of the player's command to T.

Reading maps

Each map is a list of (destination start, source start, range length) entries that specifies the interval [source start, source start + range length) should be remapped to the interval [destination start, destination start + range length). We only need to store one map at a time, so we do so in a table.

Chapter 2 - On the Reading of Various Kinds of Maps

Table of Translation
Destination start (text)    Source start (text) Source end (text)
with 100 blank rows

Defining a map is an action applying to one topic.
Understand "[text] map" as defining a map.

Carry out defining a map:
    blank out the whole of the Table of Translation.

Entering a map row is an action applying to one topic.
Understand "[text]" as entering a map row.

Carry out entering a map row:
    let the number list be the words of the topic understood;
    choose a blank row in the Table of Translation;
    now destination start entry is entry 1 of the number list;
    now source start entry is entry 2 of the number list;
    now source end entry is (source start entry + entry 3 of the number list) - "1".

Our Table of Translation has three columns, and many blank rows. Rather than storing the length of the range, we store source start + range length - 1 (the inclusive end of the source range), which will aid in searching through the map.

When we see "seed-to-soil map" or "soil-to-fertilizer map" (or similar), we understand that as the "defining a map" action, which clears the table of its previous values.

Each map row will be three numbers, but there’s no word at the start we can use to help with parsing. Fortunately though, our overall format is not overly complex, so we can understand anything (which doesn’t match another action) as the "entering a map row" action. The arithmetic in this action is using [text] + [text] and [text] - [text] operators (which we haven’t implemented yet, but will define later).

Transforming the seeds list

Once the map has been entered, the player will enter a blank line, and we should apply the mapping to the seeds list (which then should be called the soil list, but we won’t bother tracking the name changes).

Chapter 3 - On the Application of Maps to Seeds

To decide which text is (input - text) after being mapped (this is mapping):
    repeat through the Table of Translation:
        if input is numerically at least source start entry and input is numerically at most source end entry:
            let offset be input - source start entry;
            decide on the destination start entry + offset;
    decide on input.

Finishing a map is an action applying to nothing.
[Understand "" as finishing a map.]

Carry out finishing a map:
    now the seed list is mapping applied to the seed list.

We define a phrase (a first-class function) that takes a string (decimal number) and remaps it according to the current entries in the Table of Translation. The remapping is pretty simple: we loop through all the entries in our map, and if our input is in the source interval, we figure out its offset to determine its place in the destination interval. If it doesn’t match anything, it stays as-is.

Then when the player enters a blank line, we carry out the "finishing a map" action, which applies the "mapping" phrase to every entry in the seed list. There is (of course) a complication: you’d think that Understand "" as finishing a map. would configure Inform to understand the empty line, but it’s actually a parser error, so we need to hook into the parser in a different way to get it to actually work:

Rule for printing a parser error when the latest parser error is the I beg your pardon error:
    try finishing a map instead.

Computing the answer

The actions above don’t keep track of which maps is being entered, so we need to figure out which one is the last and print out the answer after its mapping has been applied. This turns out to be beautifully simple:

Chapter 4 - On the Production of the Answer

After finishing a map for the eighth time:
    say "[the minimization reduction of the seed list]".

Inform keeps track of how many times the action is invoked for us, so we just need to find the smallest entry in the seed list. To do this, we rely on having a phrase for finding the smaller of two decimal numbers, and then reduce over the list (this is more or less the maximization example in the docs).

Interlude: decimal numbers

That application logic wasn’t too hard to implement, but that’s mostly because I hand-waved over all those numerical operations. We can skip equality (assuming we never have leading zeros in numbers), so let’s start by writing a phrase that can decide if one number is smaller, equal, or larger than another:

Comparisons

[This is an enum. Maybe there's an existing/better way of doing this, but I couldn't find it.]
Comparison is a kind of value. The comparisons are smaller, equal and larger.

To decide what comparison is (N - text) compared to (M - text):
    if the number of characters in N < the number of characters in M, decide on smaller;
    if the number of characters in N > the number of characters in M, decide on larger;
    repeat with i running from 1 to the number of characters in N:
        if character number i in N < character number i in M, decide on smaller;
        if character number i in N > character number i in M, decide on larger;
    decide on equal.

If one number has more digits than the other, it’s the larger one. If we have the same number of digits, then moving from the highest-value digit to lowest, whichever number has a higher value digit is larger. There is an assumption in this code: it compares the digits as strings, so it relies on the lexicographic comparison being the same, which appears to be true for the Z-machine and Glulx targets. We could do the string to number conversion, but uh, you’ll see why I wanted to avoid that soon enough.

We can then build on top of this phrase to provide the <, <=, >= and > relations:

[<]
Numerically less than relates a text (called N) to a text (called M) when N compared to M is smaller.
The verb to be numerically less than means the numerically less than relation.
[<=]
Numerically at most relates a text (called N) to a text (called M) when N compared to M is not larger.
The verb to be numerically at most means the numerically at most relation.
[>=]
Numerically at least relates a text (called N) to a text (called M) when N compared to M is not smaller.
The verb to be numerically at least means the numerically at least relation.
[>]
Numerically greater than relates a text (called N) to a text (called M) when N compared to M is larger.
The verb to be numerically greater than means the numerically greater than relation.

There is also one last thing to define: in order to use the minimization reduction when computing the answer, we need:

To decide what text is the smaller of (N - text) and (M - text) (this is minimization):
    if N compared to M is smaller, decide on N;
    decide on M.

Addition

Consider how you would perform 972 + 35 by hand:

add the last digit of each: 2 + 5 = 7, so write 7 (→ 7)
add the second-last digit of each: 7 + 3 = 10, so write 0 and carry the 1 (→ 07)
add the carry to the first digit of the left number: 9 + 1 = 10, so write 0 and carry the 1 (→ 007)
write the carry (→ 1007)

We’re going to do something very similar:

convert "972" + "35" to lists: [9, 7, 2] + [3, 5]
reverse each list: [2, 7, 9] + [5, 3]
pad with zeros to make them the same length: [2, 7, 9] + [5, 3, 0]
produce a new list adding the digits together, keeping track of the carry:
- [2, 7, 9] + [5, 3, 0] = []
- [7, 9] + [3, 0] = [7]
- [9] + [0] = [7, 0] (+carry)
- [] + [] = [7, 0, 0] (+carry)
add the final carry (if there is one):
- = [7, 0, 0, 1]
reverse the list: [1, 0, 0, 7]
convert back into a string: "1007"

Here’s the Inform implementation:

To decide what text is the (N - text) + (M - text):
    let N digits be the digits of N;
    let M digits be the digits of M;
    reverse N digits;
    reverse M digits;
    extend N digits to the number of entries in M digits entries;
    extend M digits to the number of entries in N digits entries;
    let result digits be a list of numbers;
    let carry be 0;
    repeat with i running from 1 to the number of entries in N digits:
        let digit sum be entry i of N digits + entry i of M digits + carry;
        add the remainder after dividing digit sum by 10 to result digits;
        now carry is digit sum / 10;
    if carry > 0, add carry to result digits;
    reverse result digits;
    decide on the concatenation reduction of str applied to result digits.

This code has a suprising numbers of dependencies. The line let N digits be the digits of N uses this phrase:

To decide what list of numbers is the digits of (N - text):
    let digits be a list of numbers;
    repeat with i running from 1 to the number of characters in N:
        add the digit value of character number i in N to digits;
    decide on digits.

Which itself relies on this phrase:

To decide what number is the digit value of (N - text):
    if N is "1", decide on 1;
    if N is "2", decide on 2;
    if N is "3", decide on 3;
    if N is "4", decide on 4;
    if N is "5", decide on 5;
    if N is "6", decide on 6;
    if N is "7", decide on 7;
    if N is "8", decide on 8;
    if N is "9", decide on 9;
    decide on 0.

Sigh.

Moving swiftly along, the line extend N digits to the number of entries in M digits entries is also notable: it fills any extra values needed with the default value of the list, which since we have a list of numbers will be zero. The final point of interest is turning the list back into a string: the concatenation reduction of str applied to result digits says to apply the str phrase to each entry of the list (converting each number into a string) and then reduce the list with the concatenation phrase, which joins the strings together. Of course, we need to define these phrases:

To decide what text is the concatenation of (X - text) and (Y - text) (this is concatenation):
    decide on "[X][Y]".

To decide what text is the str of (X - number) (this is str):
    decide on "[X]".

Subtraction

You know, let’s just skip over this: it’s very similar to addition. The only thing to call out is that I decided not to implement negative numbers, so my subtraction operator only implements N - M under the condition N >= M.

Part 2

It turns out we’ve been reading the input wrong! The line seeds: 79 14 55 13 actually specifies two intervals of seeds to plant: [79, 79 + 14) and [55, 55 + 13). This is a modest 27 seed numbers, but looking at the actual input suggests we’re going to have billions of seeds, so expanding these ranges would be rather ill-advised.

Reading the seed ranges

Instead, we’ll track each range as a list of [start, end], and adjust our algorithms to work on lists of ranges.

Chapter 1 - On the Identification of Seeds

Defining the seed ranges is an action applying to one topic.
Understand "seeds [text]" as defining the seed ranges.
The seed ranges is a list of lists of text which varies.

Carry out defining the seed ranges:
    change the seed ranges to have 0 entries;
    repeat with pair running through the pairs of words of the topic understood:
        let range be a list of text;
        add entry 1 of pair to range;
        add (entry 1 of pair + entry 2 of pair) - "1" to range;
        add range to the seed ranges.

When the seed list is entered, we convert each pair of start/length into start/end ranges. Of course, we also have to define what a pair of words is:

Section - Text Manipulation

[This phrase assumes there are an even number of words.]
To decide which list of lists of text is the pairs of words of (strings - text):
    let result be a list of lists of text;
    repeat with i running from 1 to the number of words in strings:
        let pair be a list of text;
        add word number i in strings to pair;
        add word number i + 1 in strings to pair;
        add pair to result;
        now i is i + 1;  [skip the second word of the pair]
    decide on result.

Reading maps

This section is slightly different to Part 1: for reasons that will shortly become apparent, we’re going to add a column for the number of digits in the source start.

Chapter 2 - On the Reading of Various Kinds of Maps

Table of Translation
Destination start (text)    Source start (text) Source end (text)   Source start length (number)
with 100 blank rows

Defining a map is an action applying to one topic.
Understand "[text] map" as defining a map.

Carry out defining a map:
    blank out the whole of the Table of Translation.

Entering a map row is an action applying to one topic.
Understand "[text]" as entering a map row.

Carry out entering a map row:
    let the number list be the words of the topic understood;
    choose a blank row in the Table of Translation;
    now destination start entry is entry 1 of the number list;
    now source start entry is entry 2 of the number list;
    now source start length entry is the number of characters in source start entry;
    now source end entry is (source start entry + entry 3 of the number list) - "1".

Transforming the seed ranges

In Part 1, we just needed to find the mapping for a single number, but now we need to find the mapping for intervals. Let’s work through each step of that process:

Chapter 3 - On the Application of Maps to Seeds

Carry out finishing a map:
    sort the Table of Translation in source start order;
    sort the Table of Translation in source start length order;

The first thing we do is to sort the Table of Translation by the source start order. Unfortunately, this is using lexicographical order, so "10" < "2". But fear not: sorting tables is defined to be stable, so if we next sort by the length of the source start column (helpfully in its own explicit column), the rows within each length will remain sorted lexicographically, which gives us a numerical sort.

    now the seed ranges is the sorted interval union of the seed ranges;
    let the destination ranges be a list of lists of text;

Next, we get the "sorted interval union" of the seed ranges. We’ll define this later, but this takes the seed ranges, sorts them by their interval start and then merges any adjacent/overlapping intervals together, so we have a canonical, minimal representation of the intervals.

We also define the destination ranges to store the mapped intervals as we go.

    let the starting row be 1;
    repeat with seed range running through the seed ranges:
        repeat with N running from the starting row to the number of filled rows in the Table of Translation:
            now the starting row is N;
            choose row N in the Table of Translation;

Nested loops, in my interactive fiction? But fear not: both the seed ranges and the Table of Translation are sorted, so for each seed range, we scan through the Table of Translation to find out how it should be mapped, but then we know that the rows before that point cannot possibly be useful for processing future seed ranges, so we keep starting row updated to note where we can start from (so this is linear time).

Now we need to match a few cases for how the seed range intervals correspond to the intervals in the Table of Translation:

            if entry 1 of seed range is numerically less than source start entry:
                [seed range start < source start, so we have an unmapped subrange]
                let unmapped subrange be a list of text;
                add entry 1 of seed range to unmapped subrange;
                let last possible unmapped number be the source start entry - "1";
                add the smaller of entry 2 of seed range and the last possible unmapped number to unmapped subrange;
                let new source end be entry 2 of unmapped subrange + "1";
                now entry 1 of seed range is new source end;
                add unmapped subrange to the destination ranges;
                [if seed range is now empty, there's no point looking at any more translations]
                if seed range is an empty interval, break;

If we have a seed range like [2, 20] and the first row of the Table of Translation is for the source range [10, 30], then we need to first split off the [2, 9] part, as it is unmapped and should just be added to the destination ranges as-is. We’d then continue processing with the interval [10, 20].

But we also could have had a seed range [0, 5]: in that case, we add it entirely to the destination ranges, and then update the seed range to be [6, 5]. The if seed range is an empty interval condition catches this case and breaks out of the loop: we entirely processed this seed range and can move onto the next one. Note that starting row remains at this row, as our next seed range could also hit this case (e.g. [7, 9]).

            [seed range start >= source start, but we also need seed range start <= source end for overlap]
            if entry 1 of seed range is numerically at most source end entry:
                let mapped subrange be a list of text;
                let destination start be the mapping of entry 1 of seed range on row N;
                let subrange source end be the smaller of entry 2 of the seed range and source end entry;
                add destination start to mapped subrange;
                add the mapping of subrange source end on row N to mapped subrange;
                add mapped subrange to destination ranges;
                now entry 1 of seed range is subrange source end + "1";
                [if seed range is now empty, there's no point looking at any more translations]
                if seed range is an empty interval, break;

The next case is for when the seed range interval overlaps the source range in the Table of Translation. The previous case ensures that the seed range start is now at least the source range start³, but our interval might be entirely after this source range, so we need to check our start is before the source range end.

If it is, then we need to find the subinterval which overlaps and apply the mapping to the destination range. The destination start is simple: we know the seed range start is in the source range, so we just need to map it, which we do with the mapping of entry 1 of seed range on row N. The arithmetic here is a little messy inlined, so I’ve split off the source → destination mapping for a single number to a separate phrase.

The end of the interval is a little trickier, since it could extend outside the source range of the current table row: so we find the subrange source end as the smaller of the seed range and and the source range end, and then map it (using the source → destination mapping phrase again) to find the destination end.

In case the seed range does extend past the source range, we adjust the seed range start to source range end + 1. If it didn’t, then this will make the interval empty and we’ll break out of the loop. If it did, then we’ll continue on to the next table row with the rest of the interval to be mapped.

        [we reached the end of the (possibly-empty) table]
        unless seed range is an empty interval:
            [with a non-empty range, so it must be unmapped]
            add seed range to destination ranges;

Dropping out of the Table loop, if we reached the end of the table and our seed range is still not empty, then we have another unmapped range.

    now the seed ranges is the destination ranges.

And finally, once we’ve mapped all the seed ranges into destination mappings, it replaces seed ranges. Phew.

And let’s tidy up a couple of loose threads:

To decide which text is the mapping of (input - text) on row (N - number):
    choose row N in the Table of Translation;
    decide on destination start entry + (input - source start entry);

To decide if (X - list of text) is an empty interval:
    if entry 1 of X is numerically greater than entry 2 of X, decide yes;
    decide no.

Finding the sorted interval union

We tidied up the loose threads, now let’s tidy up a loose rope: we need a phrase to find the "sorted interval union" of the seed ranges, which I defined as sorting by the interval start and merging any adjacent or overlapping intervals together.

To decide if (R1 - list of text) is before (R2 - list of text) (this is interval start order):
    if entry 1 of R1 is numerically less than entry 1 of R2, decide yes;
    decide no.

To decide which list of lists of text is the sorted interval union of (ranges - list of lists of text):
    let result be a list of lists of text;
    now ranges is ranges sorted by interval start order;
    repeat with range running through ranges:
        [The first entry cannot be merged with a previous interval.]
        if the number of entries in result is 0:
            add range to result;
            next;
        [We can merge if the start of this range <= the end of the previous range + 1]
        let this interval start be entry 1 of range;
        let the previous interval end be entry 2 of the last entry of result;
        if this interval start is numerically at most the previous interval end + "1":
            let this interval end be entry 2 of range;
            now entry 2 of the last entry of result is the larger of the previous interval end and this interval end;
        otherwise:
            add range to result;
    decide on result.

As you can see, we sort the ranges by interval start order, which is defined above as comparing the first entries (interval start) numerically. Then we go through each interval. The first gets added to result unconditionally. For every interval after that though, we look at the previous interval in result and see if we can merge it: if this interval’s start is at most the previous interval’s end + 1, it is either adjacent or overlapping, and instead of adding the interval to the list, we just update the previous interval’s end (if this interval’s end is larger). If they’re not adjacent/overlapping, we just add the range to the list.

This is all well and good, but there’s another slight hiccup. Inform’s built-in sorting will sort a list of values by their natural ordering, but does not let us specify a custom comparator (our lists of intervals are lists of lists of text, and so the natural ordering would sort by the interval starts lexicographically, not numerically).

That’s fine, let’s just bang out a quick and terrible implementation of quicksort:

To decide which list of lists of text is (ranges - list of lists of text) sorted by (cmp - phrase (list of text, list of text) -> truth state):
    if the number of entries in ranges <= 1, decide on ranges;
    let pivot be entry 1 of ranges;
    let left be a list of lists of text;
    let right be a list of lists of text;
    repeat with i running from 2 to the number of entries in ranges:
        let range be entry i of ranges;
        if cmp applied to range and pivot is true:
            add range to left;
        otherwise:
            add range to right;
    now left is left sorted by cmp;
    now right is right sorted by cmp;
    add pivot to left;
    add right to left;
    decide on left;

While implementing this, I discovered two bugs: firstly because I initially tried to remove the pivot from the list, which corrupts the list, and then when I tried to use kind variables (generic types, more or less), as To decide what list of K is the (input - list of values of kind K) sorted by (cmp - phrase (K, K) -> truth state) does not compile (this is apparently a regression).

Computing the answer

The answer will then be the smallest start of any interval, which we could find in linear time, but instead of writing more code, let’s just sort the intervals and look at the first one:

After finishing a map for the eighth time:
    now the seed ranges is the sorted interval union of the seed ranges;
    let the answer be entry 1 of entry 1 of the seed ranges;
    say the answer.

Transcript

The sections above focus on the programming elements of Inform, but what’s the point of using an Interactive Fiction tool if you’re not going to tell a story? So my actual code adds descriptions and some flavour text as the story goes through the process of computing the answer.

Here’s a transcript for the example for Part 2:

Advent of Code 2023 Day 5.2
An Interactive Fiction by flowblok
Release 1 / Serial number 240316 / Inform 7 v10.1.2

Garden
A giant garden that looks more to you like a farm.

The latest Island Island almanac, containing an overwhelmingly amount of information on seeds, soils, fertilizers, water, lighting, temperature and humidity.

You can also see the gardener here.

> seeds: 79 14 55 13
You open the almanac and read out the long list of seeds, as the gardener looks on in appreciation.

>
You flip to the next page of the almanac.

> seed-to-soil map:
> 50 98 2
> 52 50 48
>
One by one, you slowly find the right type of soil for each kind of seed. The gardener looks fascinated to understand how you're figuring everything out.

> soil-to-fertilizer map:
> 0 15 37
> 37 52 2
> 39 0 15
>
As you find the right type of fertilizer for each kind of soil, the gardener looks on with fading interest.

> fertilizer-to-water map:
> 49 53 8
> 0 11 42
> 42 0 7
> 57 7 4
>
The gardener is looking very bored now and wanders off to tend to other tasks.

> water-to-light map:
> 88 18 7
> 18 25 70
>
Without an audience, you're starting to tire of this onerous task. You wonder how there could possibly be this many types of water.

> light-to-temperature map:
> 45 77 23
> 81 45 19
> 68 64 13
>
Some of these temperature numbers look a bit concerning, but there's no time to stop and think, you have to keep translating these numbers.

> temperature-to-humidity map:
> 0 69 1
> 1 0 69
>
As you flip to the next page of the almanac, you realise you're almost at the end. A wave of determination washes over you, and you set yourself to the task anew.

> humidity-to-location map:
> 60 56 37
> 56 93 4
>
The gardener wanders over to you. "You found it already?" he exclaims. "It would have taken me ages to figure out it was 46!"

Source code is in my repository here, as always.

Unless you’re the kind of person who asserts the identity function requires implementation. ↩
Well, not necessarily, it depends on what understanding sentences have been written. But colons are pesky in general. ↩
This may not be obvious, but if we’re here, we have a non-empty interval which does not have any part before the source range start (as we removed it as unmapped). ↩

Advent of Code 2023: Day 4

2024-02-04T00:00:00+11:00

On Day 4, we journey up the gondola lift to Island Island and meet an elf with a pile of colourful square scratchcards. They look just like this one, which also happens to be a valid Piet program that solves Part 2 of today’s problem:

It’s only 48x48 pixels, so here it is with 16x zoom:

In 2023, my solution for Day 2 was in Piet (David Morgan-Mar’s website, esolang), a stack-based language where the source code is an image, and instructions are encoded by transitions in colours as execution moves around the image. At the time, I wrote the instructions as text, and used a Piet assembler to construct an image. I don’t think I would have finished within 24h without that, but it left me wanting to construct a Piet program by hand, and so I was delighted when I realised this day’s problem was suitable.

Part 1

Our input is a sequence of cards, where each card has a set of winning numbers (which I will call “card numbers”) and a list of numbers you have (which I will call “test numbers”). We need to find out how many of our test numbers are in the set of card numbers to determine how many points each card is worth (one point for the first match, and then doubling for each subsequent match) and output the sum of all cards’ points.

Piet is a stack-based language, and it provides operators for:

basic stack manipulation: push a positive number on top of the stack, pop off the top of the stack, duplicate the top of the stack
arithmetic: add, subtract, multiply, divide and mod for taking the remainder, which all pop the top two numbers on the stack and push their result
comparison: not (replaces the top of the stack with 1 if nonzero, else 0), and greater (replaces the top two values of the stack with 1 if the second is greater than the top, else 0)
data-driven movement: pointer pops the top value of the stack and rotates the direction pointer by that amount, and switch does the same for the codel chooser
data-driven stack manipulation: roll pops the top two values of the stack and uses them to rotate a section of the stack by a certain amount
I/O: in and out do what you expect, but there are variants for characters (specified as Unicode, but with no encoding) or numbers (but exactly what is consumed from STDIN is undefined).

My approach to constructing a Piet program is structurally similar to what I did for Brainfuck in Day 1: write the algorithm, decompose that into a lower-level representation and then implement in the low-level language. For Piet, I’ll write the algorithm, decompose it into basic blocks and then lay out those basic blocks as an image.

Here’s the high-level algorithm (in pseudo-Python):

total = 0
for each line in STDIN:
    STDIN.discard("Card {N}: ")

    # read the winning numbers
    card_numbers = []
    while STDIN.read(1) == " " (or not "|"):
        n = parse_integer(STDIN.read(2))
        card_numbers.append(n)

    # test how many numbers match
    matches = 0
    while STDIN.read(1) == " " (or not "\n"):
        test_number = parse_integer(STDIN.read(2))
        for each card_number in card_numbers:
            if card_number == test_numbers:
                matches += 1
                break

    # convert the number of matches to points
    points = 0
    if matches > 0:
        points = 1
        matches -= 1
        while matches > 0:
            points *= 2
            matches -= 1

    total += points

print(total)

The next step is to convert this into basic blocks, where each block contains a sequence of Piet instructions that is run in sequence and ends with an unconditional jump to another block, a conditional jump (zero or nonzero) to two¹ other blocks, or by terminating the program.

This representation is too verbose for this blog post, so I’ll leave the link for interested readers, and just call out a few interesting² aspects. I wrote this representation in Python, where each basic block is a function that calls functions implementing the Piet instructions and ends in an unconditional jump (calling another basic block function), a conditional jump (calling a jnz function with the functions to call if the top value is zero or nonzero) or doing nothing (which results in termination). Implementing the framework for this is less than 100 lines of code, but does require a sys.setrecursionlimit call for non-trivial inputs, so not the kind of code you want to put into production, but I think we’d already passed that point by the second paragraph of this blog post.

Anyhow, we start by setting total to zero:

def bb0_init_total():
    push(1)
    not_()
    bb1_check_eof()

We can’t just push(0) because the value pushed is the size (number of ~~pixels~~codels) of the previous colour block, so instead we push a positive number (any will do, but one takes the least space) onto the stack and then use not to turn it into a zero. We then jump to bb1_check_eof, which implements the looping behaviour of for each line in STDIN: by consuming a character from STDIN and testing if we reached the end. This also consumes the 'C' from "Card {N}: ", but that’s okay.

def bb1_check_eof():
    # inchr: If no input is waiting on STDIN, this is an error and the command
    # is ignored.
    # To detect EOF, we push 0 onto the stack, then call inchr.
    push(1)
    not_()
    inchr()
    # If we read nothing, we will have a zero on the stack.
    not_()
    jnz(bb23_exit, bb2_not_eof)

In Piet, any instructions that cannot be performed (including attempting to read from STDIN when no input is available) are ignored and do nothing. So if inchr reads any character (except NUL), we’ll have a non-zero value on top of the stack, which not will turn into a zero, and jnz will jump to bb2_not_eof. If it does nothing, not will turn the zero we put into the stack into a one, and jnz will jump to bb23_exit.

That’s all well and good, but let’s skip ahead to the parsing code. We need to implement STDIN.discard("Card {N}: "), where {N} can be any (space-prefixed) integer. To achieve this without excessive instructions, we’re going to assume our input is well-formed. This is not an assumption one should usually make, but in this situation, we only have two inputs our program is expected to handle (the example and the actual input), which are well-formed, so we can make this assumption.

Having assumed our input is well-formed, let’s move the goalposts a little closer: rather than discarding "Card {N}: " specifically, we can discard any character that appears in "Card 0123456789" and is not ":", and then discard two characters. That may not seem like a huge improvement, but it allows us to disregard any character not in those sets, which lets us ask: are there integers x and m such that the expression (ord(character) + x) % m (where ord is the Unicode value of a character) is zero for ':' and non-zero for each of "Card 0123456789"?

A tiny amount brute-force later, we find the answer is yes (x = 0, m = 29):

assert ord(':') % 29 == 0
assert all(ord(c) % 29 != 0 for c in 'Card 0123456789')

This means we can construct a basic block which discards characters until it reaches a colon (and then calls bb4_read_card_numbers which needs to consume the space):

def bb3_read_until_colon():
    inchr()
    push(29)
    mod()
    not_()
    jnz(bb4_read_card_numbers, bb3_read_until_colon)

Parsing numbers

Our next interesting challenge is to parse a one or two digit integer, where one digit integers are space-prefixed. It would be lovely to use the instruction that reads numbers, but as noted above, the specification doesn’t define what state STDIN should be left in, so I am reluctant to write any program that would need to call any input instruction after using it. But fear not, we have made a bold assumption that lets us use a dirty trick, so let’s just do that again. Can we find x and m such that each of the Unicode characters "0123456789" map to their integer values and ' ' maps to 0?

assert all(ord(c) % 16 == int(c) for c in '0123456789')
assert ord(' ') % 16 == 0

Huh. That’s a power of two, so we’re just bit-masking to the four least-significant bits. That feels awfully convenient, very similar to the case-conversion trick in ASCII ('a-z' can be converted to 'A-Z' and vice-versa by XORing with 0x20), and seems like the kind of thing my predecessors would have known about, but I’ve never seen written down (and nobody pointed me to references when I asked about it). If you’ve seen this before, please leave a comment, I’d love to know!

Unfortunately, if we read three characters at a time (space, first maybe-digit, second digit), the terminating '|' will be in the first maybe-digit space, which means we actually need to handle the character set " 0123456789|". Fortunately, ord('|') % 16 == 12, so we can use (ord(c) + 4) % 16 to get 0 when c == '|'.

In the case where the first maybe-digit isn’t '|', we subtract 4 to get its value, multiply by 10 (to move it to the tens place), read the next character, modulo by 16 and add to put the second digit in the ones place.

Lists

The next point of interest is how to store card_numbers in our stack. We will need the length of the list to do various manipulations, so that should be at the top of the stack, and then we can just put each element of the list in order: B| list[0] … list[n-1] list_length |T (where B| and |T indicate the bottom and top of the stack, respectively). An empty list would just have the list length of zero: B| 0 |T.

As we read each card number, we will need to append it to the card_numbers list. If we have parsed the card number to the top of the stack, then we merely need to do some stack manipulation to move the card number into the list representation:

# STACK: B| card_numbers[0] … card_numbers[n-1] n new_card_number |T
push(2)
push(1)
roll()
# STACK: B| card_numbers[0] … card_numbers[n-1] new_card_number n |T

In this case, we’re using the roll instruction to take the top two values of the stack and rotate them by one so they get swapped. Then we just need to increment n so we include new_card_number in the list:

# STACK: B| card_numbers[0] … card_numbers[n-1] new_card_number n |T
push(1)
add()
# STACK: B| card_numbers[0] … card_numbers[n-1] new_card_number n+1 |T
# which is equivalent to
# STACK: B| card_numbers[0] … card_numbers[n-2] card_numbers[n-1] n |T

We will also need to be able to iterate over each element of the list. If we wanted the first element of the list to be at the top of the stack, we can achieve it with the sequence:

# STACK: B| card_numbers[0] … card_numbers[n-1] n |T
dup()
push(1)
add()
dup()
push(1)
sub()
# STACK: B| card_numbers[0] … card_numbers[n-1] n n+1 n |T
roll()
# STACK: B| card_numbers[1] … card_numbers[n-1] n card_numbers[0] |T

Once we are done with this element, we can return the list representation to its canonical state:

# STACK: B| card_numbers[1] … card_numbers[n-1] n card_numbers[0] |T
push(2)
push(1)
roll()
# STACK: B| card_numbers[1] … card_numbers[n-1] card_numbers[0] n |T
dup()
# STACK: B| card_numbers[1] … card_numbers[n-1] card_numbers[0] n n |T
push(3)
push(1)
# STACK: B| card_numbers[1] … card_numbers[n-1] card_numbers[0] n n 3 1 |T
roll()
# STACK: B| card_numbers[1] … card_numbers[n-1] n card_numbers[0] n |T
push(1)
add()
push(1)
# STACK: B| card_numbers[1] … card_numbers[n-1] n card_numbers[0] n+1 1 |T
roll()
# STACK: B| card_numbers[0] … card_numbers[n-1] n |T

If you are still reading this, then either you skipped over that low-level nonsense (fair), read through it and learnt something new (good for you!), or have already dealt with this level of bullshit (my condolences) and thought this was rather ho-hum. If you were in the latter two groups, you’ll be delighted to know that it’s not this simple, as you may have other variables on top of your stack (the number of matches so far, and the current test number), and will need to keep a counter of how far through the list you are. But if you did follow that sequence, those are merely a matter of perseverance.

Structure

Many hours later, I had the high-level algorithm decomposed into basic blocks in my Python representation, and I produced this Graphviz representation:

There are no crossing edges in this layout, which means we could assemble this into an image fairly mechanically: draw the instructions for each basic block vertically, leaving enough space around it for the edges to route the control flow correctly, and then we just need a little extra logic for conditional edges.

One of the things I disliked about using the Piet assembler was the output was sparse and not very artistic, so we’re not going to do that: instead we’re going to try to keep the basic blocks next to each other and keep the image as dense as possible.

Assembly

I used a combination of programs for assembly and debugging:

GNU Image Manipulation Program for drawing pixels. I created a layer for each basic block: toggling visibility of layers made it easy to match sections of the image to the graph structure, and was useful for isolating sections for debugging.
Piet Studio, an unreleased Piet debugger I wrote. Like many Piet interpreters, it lets you step through the program showing the current cursor position, direction pointer, codel chooser and stack, but it also lets you manually edit the first three of those (stack editing is still on the backlog 😉).
npietedit just for figuring out what the colour transitions were for the instructions I needed. These two web-based editors also provide this functionality and may be useful for following along with this section.
npiet as an independent Piet implementation to verify my program doesn’t rely on bugs in my Piet implementation.

Let’s have a look at how I assembled out the four basic blocks at the top of my graph: bb0_init_total, bb1_check_eof, bb23_exit, and bb2_not_eof:

We start at the top-left, and the first three codels are bb0_init_total: we can choose the first arbitrarily (■), but then we need to push 1, which means our first colour block must only have one codel (which it does), and we need to shift the colour one darker (■ → ■). We next need not, which means we need to shift the colour two darker (i.e. one lighter) and two hue steps (■ → ■). This isn’t our only choice: we could interrupt the flow of colours with a white codel, which would not be treated as an instruction, but would let us start again at any colour. This does add two extra codels to the program, so I’ve only used white colour blocks when they’re needed for the control flow.

The control flow continues on into bb1_check_eof, which ends with not on a dark blue colour block (■). We then need to branch: to bb23_exit if the top of the stack is nonzero, or bb2_not_eof if it is zero. Except I have actually made this easier, and ensured that whenever we call jnz the top of the stack will always be either 0 or 1 (which is why so many blocks end with not). There are two instructions for data-driven movement: in this case, we’ll use pointer (which is one darker and three hue shifts, so ■ → ■): which will rotate the direction pointer by the top value on the stack. If that’s a zero, we’ll keep moving right (■ → ■ is the pop of bb2_not_eof), otherwise it will be a one, which will start moving us downwards.

I made a few mistakes in Part 1, and one of the big ones was forgetting to actually output the answer. The program in the graph structure above computes the answer and leaves it on top of the stack, but bb23_exit should have included an out instruction. Unfortunately I had assembled bb23_exit to just exit and put other blocks around it before I realised this, so I was left the unpleasant task of figuring out how to retrofit it in. So the instructions in bb23_exit (■ → ■ → ■) are out (number) and (uselessly) dup.

The other thing bb23_exit needs to do is terminate the program. From the Piet specification:

Black colour blocks and the edges of the program restrict program flow. If the Piet interpreter attempts to move into a black block or off an edge, it is stopped and the CC is toggled. The interpreter then attempts to move from its current block again. If it fails a second time, the DP is moved clockwise one step. These attempts are repeated, with the CC and DP being changed between alternate attempts. If after eight attempts the interpreter cannot leave its current colour block, there is no way out and the program terminates.

In this trace from npiet (edited to improve legibility), we can see how that works in our program: when we enter the final blue block, the direction pointer is down and the codel chooser is left, so we attempt to move from the codel marked with 9.0 to the codel below it. That’s black, so we toggle the codel chooser to right and try again from the codel marked with 9.1. It’s also black, so we rotate the direction pointer and try again. When the eighth attempt (9.7) also fails, we realise there’s no way out and the program terminates.

Having dealt with the end of file, let’s take a look at the assembly of bb3_read_until_colon (bb2_not_eof and bb4_read_card_numbers are also included to give context):

That’s a big block of dark red. If you recall, we found 29 was the magic number for distinguishing colon/non-colon characters, so we need to push 29 in this block, and you’ll be unsurprised that there are 29 dark red codels in that block. This is not really a mistake, but there are more efficient ways we could have achived the same thing. For example, I could have split this into a sequence of instructions using only 10 codels:

push 3 (stack: [3], +3 codels)
dup (stack: [3 3], +1 codel)
dup (stack: [3 3 3], +1 codel)
mul (stack: [3 9], +1 codel)
mul (stack: [27], +1 codel)
push 2 (stack: [27 2], +2 codels)
add (stack: [29], +1 codel)

Anyhow, let’s plow on with the straightforward approach and take a look at how we loop back to the start of the block. Here’s a trace:

We perform the sequence in (character), push 29, mod, not (■ → ■ → ■ → ■ → ■), and then we need to go back to the start of bb3_read_until_colon if the top of the stack is zero, or on to bb4_read_card_numbers if it’s one. But it makes more sense for the layout if we turn right to get back to the start of bb3_read_until_colon and do nothing for bb4_read_card_numbers, so we add another not before we use pointer to change our direction (■ → ■ → ■).

But there are two more instructions before we go back to the start: push 1 and switch (■ → ■ → ■), what’s with that? Well, when we were on the light red codel at the top right, we were moving right and then hit the black codel to the right, which rotated our direction pointer so we started moving down, but it also flipped our codel chooser from left to right (following the rules quoted above). But it also gets flipped in the white colour block (changing from moving left to up) and the dark yellow colour block (changing from moving up to moving right), which means on the second iteration our codel chooser would be right: then when we exit the big dark red block, we’d be choosing the rightmost codel on the right edge to exit from, which would put us on the light-blue codel, skipping the light red and green codels! So we just flip the codel chooser once more to get us back into the correct state.

This is probably a good time to bring up another mistake I made in constructing Part 1: I used a black background, which means it’s not clear which black codels are load-bearing for the program’s control flow and which are unused. This is good for speed, since I didn’t have to draw in the black codels whenever I need them, but bad for knowing whether I can use adjacent codels for other parts of the program flow, or use for decoration at the end.

There’s not too many things to talk about for Part 1, so here’s my solution:

To help with understanding the high-level control flow, here’s a version where I’ve coloured each basic block a distinct colour and overlaid the npiet trace from running with the example input:

You’ll notice that some parts of the image are coloured but unvisited: these are either the handling for empty lists (which don’t appear in the input) or just decoration.

Part 2

In Part 2, we discover there is no such thing as points. We’re actually winning copies of subsequent scratchcards, and need to count how many we end up with. Here’s the high-level algorithm:

# Number of instances of each scratchcard we've processed so far.
seen = []

# Number of copies for scratchcards yet to be processed.
# (i.e. copies[0] + 1 will be the number of instances for the next scratchcard)
copies = []

for each line in STDIN:
    STDIN.discard("Card {N}: ")

    # read the winning numbers
    card_numbers = []
    while STDIN.read(1) == " " (or not "|"):
        n = parse_integer(STDIN.read(2))
        card_numbers.append(n)

    # test how many numbers match
    matches = 0
    while STDIN.read(1) == " " (or not "\n"):
        test_number = parse_integer(STDIN.read(2))
        for each card_number in card_numbers:
            if card_number == test_numbers:
                matches += 1
                break

    # record the number of instances of this scratchcard in the seen list
    instances = 1
    if len(copies) > 0:
        instances += copies.pop(0)
    seen.append(instances)

    # make sure copies is long enough
    while len(copies) < matches:
        copies.append(0)

    # add `instances` copies of the next `matches` scratchcards
    for each i where 0 <= i < matches:
        copies[i] += instances

total = 0
for each instances in seen:
    total += instances
print(total)

The first half of the loop is the same as Part 1, so while I did go through everything again (to be certain of correctness, and add comments about the stack state to aid in debugging), I was able to copy many of the basic blocks. You may also notice that the seen list doesn’t need to be a list: we could just keep a total and add to it instead of appending. This choice was also made in aid of debugging: if something goes off the rails and I only notice later, I can look through the list to figure out at which point it happened.

Decomposing this algorithm into basic blocks was still many hours of effort, but with the aid of rigorous commenting, being able to copy bits from Part 1 and just having more experience, it was a much smoother process than Part 1. Here’s a link to the Python representation, and the Graphviz representation:

This time, I started with a grey background, and marked load-bearing black codels explicitly in their basic block layers. I also wanted to have more room for decoration, so I tried to lay things out so I would get a border around the image, and then put basic blocks inside that. I initially aimed to just leave the “scratch!” box free, but then it became clear I’d have some more space, and so I tried to line up some cells for numbers, which worked out pretty well.

If I turn off the decoration layers, I can get a bare version of the program at the top of this article and get a trace using the example input to show the control flow:

The last step is decoration, and anything can go in the grey areas, so long as it doesn’t change the existing colour blocks by having the same colour adjacent³. I drew in the text and numbers, and then just filled the rest with random greens and blues to be thematic.

As always, code and intermediate stages are in my repository.

There are eight possible combinations of the direction pointer and codel chooser, so more sophisticated control flow mechanisms are possible! But we’re going to keep things simple, at least for now… ↩
So long as you share my definition of interesting, which… well, I’m not sure how good an idea that is. ↩
I was too lazy to do this manually, so all the greens and blues I added are one step away from the correct Piet colours in RGB space. ↩

Advent of Code 2023: Day 3

2023-12-22T00:00:00+11:00

On Day 3 we reach the gondola lift station, but the engine is broken and for some inexplicable reason the only computer we have is a mainframe running COBOL (which appears to be the COmmon Bauble Ornamentation Language on this machine).

To repair the engine, we’re going to need a program:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. aoc.

And we’re going to need it to be able to read the engine schematic:

       ENVIRONMENT DIVISION.
          INPUT-OUTPUT SECTION.
             FILE-CONTROL.
             SELECT STDIN ASSIGN TO KEYBOARD
             ORGANIZATION IS LINE SEQUENTIAL.

       DATA DIVISION.
           FILE SECTION.
           FD STDIN.
           01 STDIN-LINE PIC X(200).

There’s a lot of verbosity here, but the key points are we declare a file called STDIN which comes from the keyboard input and is a sequence of lines, with each line containing 200 characters (this is what PIC X(200) indicates: X means any character). Our actual input is only 140 characters wide, I’m just rounding up.

We’re going to want to read the input into memory so we can manipulate it, so we’ll define a bunch of variables for doing this:

       WORKING-STORAGE SECTION.
           01 WIDTH PIC 9(3).
           01 HEIGHT PIC 9(3).
           01 X PIC 9(3).
           01 Y PIC 9(3).
           01 CHAR-GRID.
               02 CHAR-GRID-ROW OCCURS 200 TIMES.
                   03 CHAR-GRID-CELL PIC X OCCURS 200 TIMES.

The first four variables (WIDTH, HEIGHT, X and Y) are all defined with PIC 9(3), meaning they can store three-digit numbers (000–999), which is all we need. The definition of CHAR-GRID will be a little less obvious to COBOL newcomers. I’d say that it’s effectively char[200][200], but that’s not exactly true, since we give names to each level of the array, so it’s more like this C++ code:

struct CharGridRowRecord {
    char CHAR_GRID_CELL[200];
};

struct CharGridRecord {
    CharGridRowRecord CHAR_GRID_ROW[200];
};

CharGridRecord CHAR_GRID;

Anyhow, let’s read into this structure:

       PROCEDURE DIVISION.
      * Read the grid.
           MOVE 1 TO Y
           OPEN INPUT STDIN
               PERFORM UNTIL EXIT
                   READ STDIN INTO CHAR-GRID-ROW(Y)
                       AT END EXIT PERFORM
                       NOT AT END ADD 1 TO Y
                   END-READ
               END-PERFORM
           CLOSE STDIN
      * Calculate width and height.
           COMPUTE HEIGHT = Y - 1
           INSPECT CHAR-GRID-ROW(1) TALLYING WIDTH FOR ALL " "
           COMPUTE WIDTH = 200 - WIDTH

Reading the grid is pretty straightforward: PERFORM UNTIL EXIT is what most other languages call while (true), and EXIT PERFORM is break. Computing the width is messier than I’d like: I’m counting the number of spaces (which occur at the end of the line, after the read characters) in the first line, and then subtracting from 200 (the size I picked above). I’d love to know if someone has a better way to do this.

Part 1

Now we’ve got that boilerplate out of the way, let’s get onto the main part of the problem. We’re given a grid of characters, and are asked for the sum of numbers which are adjacent (including diagonals) to symbols (excluding periods). Here’s what an example input looks like:

467..114..
...*......
..35..633.
......#...
617*......
.....+.58.
..592.....
......755.
...$.*....
.664.598..

The way I’m going to approach this is to build a grid of booleans (i.e. a bitmap or 2D bitmask), each indicating whether the corresponding cell in the input grid can reach a symbol. The first step is to construct the bitmap with 1s where there are symbols in the input, and 0s everywhere else:

We’ll need some more data structures to represent this in our working storage section:

       WORKING-STORAGE SECTION.
      * … (as above)
           01 FILTER.
               02 FILTER-ROW OCCURS 200 TIMES.
                   03 FILTER-CELL PIC 1 USAGE BIT OCCURS 200 TIMES.

FILTER is a 2D array like CHAR-GRID, but this time of booleans (PIC 1 USAGE BIT¹). We can construct the bitmap for the first step with:

       PROCEDURE DIVISION.
      * … (as above)
      * Construct the filter grid.
           PERFORM VARYING Y FROM 1 BY 1 UNTIL Y > HEIGHT
               PERFORM VARYING X FROM 1 BY 1 UNTIL X > WIDTH
                   IF CHAR-GRID-CELL(Y,X) IS NOT NUMERIC
                       AND CHAR-GRID-CELL(Y,X) <> "."
                   THEN
                       MOVE 1 TO FILTER-CELL(Y,X)
                   ELSE
                       MOVE 0 TO FILTER-CELL(Y,X)
                   END-IF
               END-PERFORM
           END-PERFORM

We then expand this bitmap: for each cell, we can set it to a 1 if the corresponding character in the input is a digit and one of the eight neighbours in the bitmap is a 1. The first application of this rule produces:

Rinse and repeat: once we stop making changes, we end up with:

To implement this, we need a few more bits of storage: a boolean indicating whether applying the rule resulted in any changes (FILTER-CHANGED), some integers tracking the delta to each neighbour (DX, DY) and each neighbour’s position (X2, Y2):

       WORKING-STORAGE SECTION.
      * … (as above)
           01 FILTER-CHANGED PIC 1 USAGE BIT.
           01 DX PIC S9.
           01 DY PIC S9.
           01 X2 PIC 9(3).
           01 Y2 PIC 9(3).

The additions to the procedure division are a mess of nested loops:

       PROCEDURE DIVISION.
      * … (as above)
      * Expand the bitmap.
           MOVE 1 TO FILTER-CHANGED
           PERFORM UNTIL FILTER-CHANGED = 0
               MOVE 0 TO FILTER-CHANGED
               PERFORM VARYING Y FROM 1 BY 1 UNTIL Y > HEIGHT
               PERFORM VARYING X FROM 1 BY 1 UNTIL X > WIDTH
                   IF FILTER-CELL(Y,X) = 0 AND
                       CHAR-GRID-CELL(Y,X) IS NUMERIC
                   THEN
                       PERFORM VARYING DY FROM -1 BY 1 UNTIL DY = 2
                       PERFORM VARYING DX FROM -1 BY 1 UNTIL DX = 2
                           COMPUTE X2 = X + DX
                           COMPUTE Y2 = Y + DY
                           IF X2 > 0 AND X2 <= WIDTH AND
                               Y2 > 0 AND Y2 <= HEIGHT AND
                               FILTER-CELL(Y2,X2) <> 0
                           THEN
                               MOVE 1 TO FILTER-CELL(Y,X)
                               MOVE 1 TO FILTER-CHANGED
                           END-IF
                       END-PERFORM
                       END-PERFORM
                   END-IF
               END-PERFORM
               END-PERFORM
           END-PERFORM

The structure here is:

Outermost loop: keep running until we do an iteration without changing any cells (indicated by setting FILTER-CHANGED to 1)
Second/third loops: look at each cell (X,Y) in the bitmap, check if the corresponding input character is a digit (IS NUMERIC), and the bitmap cell is not already a 1.
Fourth/fifth loops: loop over each of the relative offsets to the eight neighbouring cells: (-1, -1), (0, -1), (1, -1), (-1, 0), ..., (1, 1). This does also include (0, 0), but in that case we already know that FILTER-CELL(Y2,X2) will be 0, so it doesn’t matter.
Innermost body: compute (X2, Y2) as the absolute position of the neighbour, check if it’s in bounds and the neighbour’s cell is a 1. If so, we can set this cell to a 1 and set FILTER-CHANGED to 1 to indicate we’ve made a change.

We can then apply this bitmask back to the original input to hide the numbers which aren’t touching a symbol, leaving us with the simpler problem of adding the remaining numbers:

           PERFORM VARYING Y FROM 1 BY 1 UNTIL Y > HEIGHT
               PERFORM VARYING X FROM 1 BY 1 UNTIL X > WIDTH
                   IF FILTER-CELL(Y,X) = 0 THEN
                       MOVE "." TO CHAR-GRID-CELL(Y,X)
                   END-IF
               END-PERFORM
           END-PERFORM

After this, our example input is now missing the numbers which weren’t touching a non-period symbol:

467.......
...*......
..35..633.
......#...
617*......
.....+....
..592.....
......755.
...$.*....
.664.598..

The only thing left is to add up the remaining numbers and print the total. We need two last variables in our working storage section:

       WORKING-STORAGE SECTION.
      * … (as above)
           01 PARSED-NUMBER PIC 9(3).
           01 TOTAL PIC 9(9).

To add up the numbers, we first need to assemble each number out of its digits. We loop over each cell in the grid: when we see a digit, we multiply the existing digits in PARSED-NUMBER by 10 to shift them up a place, then add on the digit. Unfortunately while we know that CHAR-GRID-CELL(Y,X) is a digit, it’s still a PIC X, so we need FUNCTION NUMVAL(…) to turn it into a PIC 9 so we can add it.

Whenever we reach a non-numeric character or the end of the row, we add PARSED-NUMBER to the total and reset it to zero. It doesn’t matter if we hadn’t actually seen a number: in that case, PARSED-NUMBER will be zero and adding it to TOTAL will have no effect.

      * Find numbers in the grid, add them up.
           MOVE 0 TO PARSED-NUMBER
           MOVE 0 TO TOTAL
           PERFORM VARYING Y
               FROM 1 BY 1 UNTIL Y > HEIGHT
               PERFORM VARYING X
                   FROM 1 BY 1 UNTIL X > WIDTH
                   IF CHAR-GRID-CELL(Y,X) IS NUMERIC THEN
                       COMPUTE NUMBER = NUMBER * 10 +
                           FUNCTION NUMVAL(CHAR-GRID-CELL(Y,X))
                   ELSE
                       ADD PARSED-NUMBER TO TOTAL
                       MOVE 0 TO PARSED-NUMBER
                   END-IF
               END-PERFORM
               ADD PARSED-NUMBER TO TOTAL
               MOVE 0 TO PARSED-NUMBER
           END-PERFORM
      * Print the total.
           DISPLAY TOTAL
           STOP RUN.

Part 2

Having found the missing part, we need to take things up a gear. We now only care about the numbers which are touching a * symbol: when there are two of these, we need to multiply them together and add them to the total.

Let’s give each gear (a * symbol) a number, starting from 1. Then we can turn our bitmap into something a bit more detailed: rather than just storing 0 or 1, it’ll store 0 or the gear number that digit is connected to.

       WORKING-STORAGE SECTION.
      * … (as above, but without FILTER / FILTER-CHANGED)
           01 GEAR-GRID.
               02 GEAR-GRID-ROW OCCURS 200 TIMES.
                   03 GEAR-GRID-CELL PIC 9(3) OCCURS 200 TIMES.
           01 GEARS-CHANGED PIC 1 USAGE BIT.
           01 GEARS-COUNT PIC 9(3).
           01 GEAR-INFO OCCURS 1000 TIMES.
               02 GEAR-RATIO PIC 9(6).
               02 GEAR-COUNT PIC 9.
           01 CURRENT-GEAR PIC 9(3).

We also have the number of gears (GEARS-COUNT) and for each gear, a record (GEAR-INFO) which will have the result of multiplying the touching numbers (GEAR-RATIO) and the count of how many touching numbers there are (GEAR-COUNT²). There’s also CURRENT-GEAR, which is an index into GEAR-INFO we’ll need later.

To construct the gear grid, we loop over each cell: whenever we see a * we increment the gear count to get the current gear number, put it in the gear grid, and initialize the corresponding entries in GEAR-INFO. We initialize GEAR-RATIO to 1, which will come in handy later, when we’re multiplying numbers into it.

      * (replacing the "Construct the filter grid" section)
      * Construct the gears grid.
           INITIALIZE GEAR-GRID
           MOVE 0 TO GEARS-COUNT.
           PERFORM VARYING Y
               FROM 1 BY 1 UNTIL Y > HEIGHT
               PERFORM VARYING X
                   FROM 1 BY 1 UNTIL X > WIDTH
                   IF CHAR-GRID-CELL(Y,X) = "*" THEN
                       ADD 1 TO GEARS-COUNT
                       MOVE GEARS-COUNT TO GEAR-GRID-CELL(Y,X)
                       MOVE 1 TO GEAR-RATIO(GEARS-COUNT)
                       MOVE 0 to GEAR-COUNT(GEARS-COUNT)
                   ELSE
                       MOVE 0 TO GEAR-GRID-CELL(Y,X)
                   END-IF
               END-PERFORM
           END-PERFORM

For our example input, this gives us (pretending that GEAR-GRID-CELL is a single-digit number):

Iteratively expanding the gears grid is very similar to the filter:

      * (replacing the "Expand the bitmap" section)
      * Expand the gears grid.
           MOVE 1 TO GEARS-CHANGED
           PERFORM UNTIL GEARS-CHANGED = 0
               MOVE 0 TO GEARS-CHANGED
               PERFORM VARYING Y FROM 1 BY 1 UNTIL Y > HEIGHT
               PERFORM VARYING X FROM 1 BY 1 UNTIL X > WIDTH
                   IF GEAR-GRID-CELL(Y,X) = 0 AND
                       CHAR-GRID-CELL(Y,X) IS NUMERIC THEN
                       PERFORM VARYING DY FROM -1 BY 1 UNTIL DY = 2
                       PERFORM VARYING DX FROM -1 BY 1 UNTIL DX = 2
                           COMPUTE X2 = X + DX
                           COMPUTE Y2 = Y + DY
                           IF X2 > 0 AND X2 <= WIDTH AND
                               Y2 > 0 AND Y2 <= HEIGHT AND
                               GEAR-GRID-CELL(Y2,X2) <> 0
                           THEN
                               MOVE GEAR-GRID-CELL(Y2,X2)
                                   TO GEAR-GRID-CELL(Y,X)
                               MOVE 1 TO GEARS-CHANGED
                           END-IF
                       END-PERFORM
                       END-PERFORM
                   END-IF
               END-PERFORM
               END-PERFORM
           END-PERFORM

Aside from a bunch of renaming, pretty much the only change is that we now need to copy the gear number from the neighbour (GEAR-GRID-CELL(Y2,X2)) when setting each updated cell’s gear number (GEAR-GRID-CELL(Y,X)).

Once expanded, we get (again pretending that GEAR-GRID-CELL is a single-digit number):

Like in Part 1, we filter the CHAR-GRID using GEAR-GRID to remove anything which isn’t touching a gear, but now our final step is a bit more complicated. When we find a number, we need to figure out which gear it corresponds to, multiply the number into GEAR-RATIO and increment GEAR-COUNT (so we’ll be able to tell if there’s exactly two touching numbers we multiplied):

      * (replacing the "Find numbers in the grid, add them up" section)
      * Find numbers in the grid, multiply them into their gears.
           MOVE 0 TO CURRENT-GEAR.
           MOVE 0 TO PARSED-NUMBER
           PERFORM VARYING Y FROM 1 BY 1 UNTIL Y > HEIGHT
               PERFORM VARYING X FROM 1 BY 1 UNTIL X > WIDTH
                   IF CHAR-GRID-CELL(Y,X) IS NUMERIC THEN
                       COMPUTE PARSED-NUMBER = PARSED-NUMBER * 10 +
                           FUNCTION NUMVAL(CHAR-GRID-CELL(Y,X))
                       MOVE GEAR-GRID-CELL(Y,X) TO CURRENT-GEAR
                   ELSE
                       IF CURRENT-GEAR <> 0 THEN
                           MULTIPLY PARSED-NUMBER
                               BY GEAR-RATIO(CURRENT-GEAR)
                           ADD 1 TO GEAR-COUNT(CURRENT-GEAR)
                           MOVE 0 TO CURRENT-GEAR
                           MOVE 0 TO PARSED-NUMBER
                       END-IF
                   END-IF
               END-PERFORM
               IF CURRENT-GEAR <> 0 THEN
                   MULTIPLY PARSED-NUMBER
                       BY GEAR-RATIO(CURRENT-GEAR)
                   ADD 1 TO GEAR-COUNT(CURRENT-GEAR)
                   MOVE 0 TO CURRENT-GEAR
                   MOVE 0 TO PARSED-NUMBER
               END-IF
           END-PERFORM

In this part, we need to track which gear the number belongs to, so when we find a digit, we store GEAR-GRID-CELL(Y,X) (which has the gear number) in CURRENT-GEAR. Unlike addition, multiplying by zero does have an effect, so we also need to check when we actually have a number to do something with, which is provided by CURRENT-GEAR being non-zero.

All that’s left is to add up the gear ratios for any gear where we saw exactly two touching numbers and print it out:

      * Add up the gear ratios.
           MOVE 0 TO TOTAL
           PERFORM VARYING CURRENT-GEAR
               FROM 1 BY 1 UNTIL CURRENT-GEAR > GEARS-COUNT
               IF GEAR-COUNT(CURRENT-GEAR) = 2 THEN
                   ADD GEAR-RATIO(CURRENT-GEAR) TO TOTAL
               END-IF
           END-PERFORM
           DISPLAY TOTAL
           STOP RUN.

As always, the code is in my repository.

Unfortunately, my COBOL implementation doesn’t implement this efficiently and currently my blog’s syntax highlighter treats this as an error. But it’s totally a thing! ↩
I regret this is very similar to GEARS-COUNT, but I couldn’t find an alternative I liked. ↩

Advent of Code 2023: Day 2

2023-12-12T00:00:00+11:00

I don’t know about you, but when I see a problem that needs some basic numeric operations like min, max, addition and multiplication, I think to myself: “this is a great opportunity to pull out sed”. Oh, just me then?

sed is a line-oriented text processor: you give it a sequence of commands, and it applies those commands to each line of the input. The substitution command is the most well-known, and many folks will only have ever used sed 's/regex/replacement/g' on the command line, opting for other tools for more complex problems. But sed has plenty more functionality to offer!

The problem input

We have a game where we’re pulling marbles from a bag, and we’re given a record of what happened for these games. The input format is:

Game 1: 3 blue, 4 red; 1 red, 2 green, 6 blue; 2 green
Game 2: 1 blue, 2 green; 3 green, 4 blue, 1 red; 1 green, 1 blue
Game 3: 8 green, 6 blue, 20 red; 5 blue, 4 red, 13 green; 5 green, 1 red
Game 4: 1 green, 3 red, 6 blue; 3 green, 6 red; 3 green, 15 blue, 14 red
Game 5: 6 red, 1 blue, 3 green; 2 blue, 1 red, 2 green

It turns out we don’t need to care about the commas and semicolons: neither of the parts have any kind of dependency on the sequence or conjunction of cubes.

Part 1

To work with numbers in sed, we’re going to convert our numbers between numeral systems. The three forms we will have are:

Decimal: 43 (the coefficients of 4×10¹ + 3×10⁰)
Tally: BBBBAAA (B = 10¹, A = 10⁰, add together the values)
Unary: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (count the symbols)

In our input, the largest number is three digits (Game 100), so we can convert from decimal to tally format like so:

# Convert all numbers to tally format.
s/9([0-9]{2})/C8\1/g
s/8([0-9]{2})/C7\1/g
s/7([0-9]{2})/C6\1/g
s/6([0-9]{2})/C5\1/g
s/5([0-9]{2})/C4\1/g
s/4([0-9]{2})/C3\1/g
s/3([0-9]{2})/C2\1/g
s/2([0-9]{2})/C1\1/g
s/1([0-9]{2})/C\1/g
s/9([0-9])/B8\1/g
s/8([0-9])/B7\1/g
s/7([0-9])/B6\1/g
s/6([0-9])/B5\1/g
s/5([0-9])/B4\1/g
s/4([0-9])/B3\1/g
s/3([0-9])/B2\1/g
s/2([0-9])/B1\1/g
s/1([0-9])/B\1/g
s/9/A8/g
s/8/A7/g
s/7/A6/g
s/6/A5/g
s/5/A4/g
s/4/A3/g
s/3/A2/g
s/2/A1/g
s/1/A/g
s/0//g

The first line turns any 9xx decimal number into C8xx, which then gets turned by the second line into CC7xx, and so on, until the coefficient of 10² is entirely in tally format. The next ten lines do the same for the coefficient of 10¹, and the last ten lines for 10⁰.

Our example now looks like:

Game A: AAA blue, AAAA red; A red, AA green, AAAAAA blue; AA green
Game AA: A blue, AA green; AAA green, AAAA blue, A red; A green, A blue
Game AAA: AAAAAAAA green, AAAAAA blue, BB red; AAAAA blue, AAAA red, BAAA green; AAAAA green, A red
Game AAAA: A green, AAA red, AAAAAA blue; AAA green, AAAAAA red; AAA green, BAAAAA blue, BAAAA red
Game AAAAA: AAAAAA red, A blue, AAA green; AA blue, A red, AA green

We can then turn this into unary format with:

# Convert tally format to unary.
s/C/BBBBBBBBBB/g
s/B/AAAAAAAAAA/g

Which turns our example into:

Game A: AAA blue, AAAA red; A red, AA green, AAAAAA blue; AA green
Game AA: A blue, AA green; AAA green, AAAA blue, A red; A green, A blue
Game AAA: AAAAAAAA green, AAAAAA blue, AAAAAAAAAAAAAAAAAAAA red; AAAAA blue, AAAA red, AAAAAAAAAAAAA green; AAAAA green, A red
Game AAAA: A green, AAA red, AAAAAA blue; AAA green, AAAAAA red; AAA green, AAAAAAAAAAAAAAA blue, AAAAAAAAAAAAAA red
Game AAAAA: AAAAAA red, A blue, AAA green; AA blue, A red, AA green

Now we need to discard any game which has more than 12 red cubes, 13 green cubes or 14 blue cubes. Consider the red cubes: if we see "AAAAAAAAAAAAA red" (that’s 13 As) somewhere in the line, then it means that one of the draws had at least 13 red cubes. So, we can discard the games with:

# 13+ red cubes -> impossible
s/^.*AAAAAAAAAAAAA red.*$//
# 14+ green cubes -> impossible
s/^.*AAAAAAAAAAAAAA green.*$//
# 15+ blue cubes -> impossible
s/^.*AAAAAAAAAAAAAAA blue.*$//

Typically, you’d discard lines using the d command (e.g. /AAAAAAAAAAAAA red/d), but I couldn’t figure out how to make that work with the next part, so we’re just emptying the line. Our example is now:

Game A: AAA blue, AAAA red; A red, AA green, AAAAAA blue; AA green
Game AA: A blue, AA green; AAA green, AAAA blue, A red; A green, A blue


Game AAAAA: AAAAAA red, A blue, AAA green; AA blue, A red, AA green

We next need only the game ids, so we filter out everything else:

# Get just the ids.
s/^Game (A+):.*$/\1/

Which gives us:

A
AA


AAAAA

And then we need to merge all the lines together to add our numbers, which obviously is just:

# Merge the lines together.
H
$!d
x
s/\n//g

Obviously? Not in the slightest, so let’s break this down. Remember how sed processes text line-by-line? In order to do multi-line operations, we need to make use of the hold buffer. The H command appends the current line to the hold buffer, which is great, we can now accumulate all our lines into one place.

But we only want to do anything with what we put into the hold buffer once we reach the ends, so what $!d says is “if it’s not (!) the last line ($), then delete the line (and go back to the start of the program to process the next line)”.

On the last line¹ however, we get to continue with x, which (to skip some detail) says that our hold buffer (of all the lines we accumulated) is now the line we’re processing. This “line” actually contains multiple lines with newline characters between, which the s/\n//g removes to give us a single line.

Anyhow, this mess does what we need, and we get:

AAAAAAAA

We then just need to turn this back into a decimal number, which we do by first turning it into the tally format:

# Convert the tally back to a number.
# We have 100 input lines, so at worst we can have 1 + 2 + ... + 100 = 5050
s/AAAAAAAAAA/B/g
s/BBBBBBBBBB/C/g
s/CCCCCCCCCC/D/g

And then decimal:

# Insert zeros.
s/D([^CD]|$)/D0\1/
s/C([^BC]|$)/C0\1/
s/B([^AB]|$)/B0\1/

# Convert back to digits.
s/([ABCD])(\1{8})/9/g
s/([ABCD])(\1{7})/8/g
s/([ABCD])(\1{6})/7/g
s/([ABCD])(\1{5})/6/g
s/([ABCD])(\1{4})/5/g
s/([ABCD])(\1{3})/4/g
s/([ABCD])(\1{2})/3/g
s/([ABCD])(\1{1})/2/g
s/[ABCD]/1/g

Part 2

For the second part, we don’t care about the game ids, and the commas and semicolons are going to get in the way, so we start by removing them:

# Remove unnecessary parts.
s/^.*://
s/[,;]//g

Which turns our example into:

 3 blue 4 red 1 red 2 green 6 blue 2 green
 1 blue 2 green 3 green 4 blue 1 red 1 green 1 blue
 8 green 6 blue 20 red 5 blue 4 red 13 green 5 green 1 red
 1 green 3 red 6 blue 3 green 6 red 3 green 15 blue 14 red
 6 red 1 blue 3 green 2 blue 1 red 2 green

We then convert our numbers to unary format as before (though this time we only need to do two digits), and then turn the colours into a single-letter suffix on our numbers:

# (decimal->tally->unary from above)

# Turn the colour into a suffix.
s/ red/r/g
s/ green/g/g
s/ blue/b/g

Our example is now:

 AAAb AAAAr Ar AAg AAAAAAb AAg
 Ab AAg AAAg AAAAb Ar Ag Ab
 AAAAAAAAg AAAAAAb AAAAAAAAAAAAAAAAAAAAr AAAAAb AAAAr AAAAAAAAAAAAAg AAAAAg Ar
 Ag AAAr AAAAAAb AAAg AAAAAAr AAAg AAAAAAAAAAAAAAAb AAAAAAAAAAAAAAr
 AAAAAAr Ab AAAg AAb Ar AAg

In order to find the minimum number of cubes for any game, we need to find the maximum of each colour. We can do this by repeatedly removing any number where there is a larger number in the same line (e.g. in the first line, we should remove AAAb because there is also AAAAAAb which is bigger). The implementation:

# Find the minimum number of cubes of each colour required.
:min1
    s/ (A+[rgb])(.*)\1/\2\1/g
    tmin1

:min2
    s/(A+[rgb])(.*) \1/\1\2/g
    tmin2

There are two new features shown here: :min1 defines a label named min1, and tmin1 says to jump to min1 if an s command has succeeded since the last input line was read or a conditional branch (which includes t) was taken. Combined, this just says “keep running the s command in between until it doesn’t do anything”.

So what does the s command do? It matches a space, then a number with a colour-suffix (which is the first capture group \1), then anything (the second capture group \2), then a repetition of the first capture group. Consider the first line of our example, where we can find this match: (AAAb)( AAAAr Ar AAg AAA)AAAb. This finds an example where the first capture group is a number which is less than or equal to a number (of the same colour) appearing later in the line. This gets substituted with \2\1, which would be ( AAAAr Ar AAg AAA)(AAAb) (without the parentheses), which removes the left number.

Unsurprisingly, min2 does the same thing, but for the cases where the larger number appears earlier in the line. Once these have both run to completion, we’ll be left with one number per colour, though in no particular order. Here’s what the result looks like:

 AAAAr AAAAAAb AAg
 AAAg AAAAb Ar
 AAAAAAb AAAAAAAAAAAAAAAAAAAAr AAAAAAAAAAAAAg
 AAAg AAAAAAAAAAAAAAAb AAAAAAAAAAAAAAr
 AAAAAAr AAAg AAb

We don’t need the order though, we just need to multiply them together, which we’ll do with:

# Discard colour suffixes.
s/[rgb]//g

# Multiply the first two numbers together.
s/^/=/
:mul1
    s/= (A+) AA/\1= \1 A/
    tmul1
s/(A*)= (A+) A / \1\2 /

# Multiply the third number in.
s/^/=/
:mul2
    s/= (A+) AA/\1= \1 A/
    tmul2
s/(A*)= (A+) A$/\1\2/

Consider the first line of our example, " AAAAr AAAAAAb AAg". We start by discarding the colour suffixes and adding a = at the start of the line, giving "= AAAA AAAAAA AA". The mul1 loop then performs naïve multiplication of AAAA and AAAAAA, by adding AAAA (4) to a total AAAAAA (6) times. Here’s each substitution (parentheses indicate capture groups, square brackets indicate the entire match):

[= (AAAA) AA]AAAA AA → [AAAA= AAAA A]AAAA AA
AAAA[= AAAA AA]AAA AA → AAAA[AAAA= AAAA A]AAA AA
AAAAAAAA[= AAAA AA]AA AA → AAAAAAAA[AAAA= AAAA A]AA AA
AAAAAAAAAAAA[= AAAA AA]A AA → AAAAAAAAAAAA[AAAA= AAAA A]A AA
AAAAAAAAAAAAAAAA[= AAAA AA] AA → AAAAAAAAAAAAAAAA[AAAA= AAAA A] AA

And then post-loop, we make the substitution:

[(AAAAAAAAAAAAAAAAAAAA)= AAAA A ]AA → [AAAAAAAAAAAAAAAAAAAAAAAA] AA

Which leaves us with 24×2 as the multiplication for mul2, which operates almost exactly the same.

Once we’ve done that, it’s just a matter of merging the lines together to add all the numbers, then converting back to the tally format and then back to decimal. Simple.

As always, the code is in my repository.

This is why I couldn’t use the delete command earlier: if the last line of the input was an impossible game, we’d delete it and the program would terminate without getting to the processing of the hold buffer. Perhaps there’s a nicer solution? I’d love to know. ↩

Advent of Code 2023: Day 1

2023-12-10T00:00:00+11:00

We’re kicking things off the same way as last year: taking the (hopefully) easiest problem and solving it in one of the more difficult languages to code in: Brainfuck.

A brief introduction to Brainfuck

As a brief reminder (or introduction if you’ve never had the pleasure), Brainfuck programs run using an array of memory cells, each storing an integer. There is a pointer into this array, initially pointing at the first cell.

There are only eight instructions for Brainfuck programs:

Command	Description
`>`	Move the pointer one cell to the right.
`<`	Move the pointer one cell to the left.
`+`	Increment the integer in the pointed-at cell.
`-`	Decrement the integer in the pointed-at cell.
`,`	Read from the input into the pointed-at cell.
`.`	Write the pointed-at cell to the output.
`[`	Jump past the matching `]` if the pointed-at cell is 0.
`]`	Jump back to the matching `[`.

With these instructions, we can start to build higher-level constructs. The simplest of these is [-]. This repeatedly decrements the pointed-at cell until it is equal to zero, or more simply, it sets the pointed-at cell to zero. Add some +s after that, and we now can set the pointed-at cell to any number, regardless of its prior state.

We also will want to be able to copy a value from one cell to another. Let’s first look at how we’d move a value from one cell to another: [->+<]. This decrements the first cell, moves to the next cell, increments it, and then moves back to the first cell. The loop will repeat until the first cell is zero, at which point we will have finished moving the value.

To copy a cell then, we need a temporary cell: we’ll start by moving from the first cell to the second and third cells with [->+>+<<], and then follow up by moving the third cell back to the first cell with >>[-<<+>>]. Note that in the constructs we’ve seen so far, the < and > within […] loops are all balanced: this is usually the case, until you start wanting to use the memory cells as an array.

Another useful high-level construct is to be able to read the input one character at a time until you encounter some specific value. Here’s how you’d read a single line of input: ,----------[++++++++++ $inner ,----------]. This is pretty straightforward, we read a character and subtract 10 so the newline character ('\n' = 10) will become a zero that terminates the loop. The ++++++++++ restores the original character value, though this is optional if you’re happy to process a shifted character value.

We’ll also want conditionals (if/else statements): we can use [[-] $ifbody] to run $ifbody once if the pointed-at cell is nonzero, though this also sets the pointed-at cell to zero, so we may wish to use a temporary and then move the value back with [[->+<] $ifbody]>[-<+>]<.

To implement an else statement, we need to use another memory cell as a flag:

# set the "else flag" to one
>[-]+<

# if the initial memory cell is nonzero (destructive)
[
    [-]
    $ifbody
    # clear the "else flag"
    >-<
]

# if the "else flag" is nonzero, we didn't run $ifbody
>[
    [-]
    $elsebody
]<

Some conventions I will continue to use: # indicates a comment (there is a way to do comments natively in Brainfuck, but this should be familiar to readers), and $ indicates that the supplied code is merely a template for a Brainfuck program: in the case above, $ifbody and $elsebody would need to be replaced with some actual Brainfuck instructions.

Part 1: the problem

Heavens, all that and we haven’t even looked at the problem yet?! As expected, the problem is pretty straightforward: we want the sum of calibration values for each line, and each calibration value is formed by concatenating the first and last digits found in the line. The last line of the example input helpfully points out the edge case where there is only one digit in the line, so the first and last digits will be the same.

We’ll start by designing an algorithm for this that works character-by-character (since that’s how we’ll need to implement it):

Until the end of the input:
1. Read characters until we find a digit
2. Add the digit 10 times to the total
3. Store the digit as the last-seen digit
4. Read characters until a newline, updating the last-seen digit as we go
5. Add the last-seen digit to the total
Print the total

Is it a digit?

We have most of the pieces we need already, but there is one tricky part: how do we tell if a character is a digit or not? The digits '0123456789' have ASCII/Unicode values 48, 49, …, 57, so if we knew we had a digit, we could get its value by subtracting 48. But if we have another character like 'a' (value 97), then subtracting 48 will give us 49. So we could subtract 48 and then test if the value is in the range 0-9… but we don’t have a construct for doing that in Brainfuck (we have "if nonzero", and that can be combined with subtraction to get "if equal to", but we don’t have an "if less than" or "if greater than").

Rather than being clever and coming up for a construct for that, let’s do something simpler and just test each of the 10 different digits. Here’s some pseudo-code to explain the approach:

function is_digit(in c, out is_digit, temp c_copy, temp else_flag):
    is_digit = 0
    unroll for $digit in [48...57]:
        c_copy = c - $digit (temporary=else_flag)
        else_flag = 1
        if c_copy != 0:
            else_flag = 0
            c_copy = 0
        else: (flag=else_flag)
            else_flag = 0
            is_digit = 1

We need four memory cells:

c: the character to test, which we pinky-promise will be unchanged at the end of the function
is_digit: a flag indicating if c is a digit (1) or not (0)
c_copy: a temporary variable which will hold c - '0', c - '1', etc. If this is ever zero, it means we’ve found a digit.
else_flag: a temporary variable used for copying c to c_copy and indicating if the else conditional needs to be run.

Let’s now turn that into Brainfuck:

# is_digit = 0
>[-]<

# $digit = '0'
    # c_copy = c
    # else_flag = c
    # c = 0
    [->>+>+<<<]

    # c = else_flag
    # else_flag = 0
    >>>[-<<<+>>>]<<<

    # c_copy -= '0'
    >>------------------------------------------------

    # else_flag = 1
    >[-]+
    # if c_copy != 0
    <[
        # c_copy = 0
        [-]
        # else_flag = 0
        >[-]<
    ]
    # else
    >[
        # else_flag = 0
        [-]
        # is_digit = 1
        <<[-]+>>
    ]<<<

# $digit = '1'
    [->>+>+<<<]
    >>>[-<<<+>>>]<<<

    # c_copy -= '1'
    >>-------------------------------------------------

    >[-]+
    <[
        [-]
        >[-]<
    ]
    >[
        [-]
        <<[-]+>>
    ]<<<

# $digit = '2'
    [->>+>+<<<]>>>[-<<<+>>>]<<<
    >>--------------------------------------------------
    >[-]+<[[-]>[-]<]>[[-]<<[-]+>>]<<<
# $digit = '3'
    [->>+>+<<<]>>>[-<<<+>>>]<<<
    >>---------------------------------------------------
    >[-]+<[[-]>[-]<]>[[-]<<[-]+>>]<<<
# $digit = '4'
    [->>+>+<<<]>>>[-<<<+>>>]<<<
    >>----------------------------------------------------
    >[-]+<[[-]>[-]<]>[[-]<<[-]+>>]<<<
# $digit = '5'
    [->>+>+<<<]>>>[-<<<+>>>]<<<
    >>-----------------------------------------------------
    >[-]+<[[-]>[-]<]>[[-]<<[-]+>>]<<<
# $digit = '6'
    [->>+>+<<<]>>>[-<<<+>>>]<<<
    >>------------------------------------------------------
    >[-]+<[[-]>[-]<]>[[-]<<[-]+>>]<<<
# $digit = '7'
    [->>+>+<<<]>>>[-<<<+>>>]<<<
    >>-------------------------------------------------------
    >[-]+<[[-]>[-]<]>[[-]<<[-]+>>]<<<
# $digit = '8'
    [->>+>+<<<]>>>[-<<<+>>>]<<<
    >>--------------------------------------------------------
    >[-]+<[[-]>[-]<]>[[-]<<[-]+>>]<<<
# $digit = '9'
    [->>+>+<<<]>>>[-<<<+>>>]<<<
    >>---------------------------------------------------------
    >[-]+<[[-]>[-]<]>[[-]<<[-]+>>]<<<

Gorgeous.

For the "read characters until we find a digit" step, we’re going to want to have a loop that terminates when we’ve found a digit. Since loops terminate on zero, it’s convenient to have a variant of this which returns 0 if the character is a digit, else 1, which can be constructed by swapping all is_digit = 0 with is_digit = 1 (and vice versa).

Putting it together

We start with our memory layout:

total: The total of all calibration values.
last_seen: The last-seen digit (as a value 0–9).
c: The current character.
is_digit: Whether the current character is a digit or not (though the assignment of 0/1 to true/false is contextual).
c_copy: Temporary variable, primarily for copying c.
else_flag: Temporary variable, primarily for else statements.

The overall program structure then looks like:

# Read a character until EOF.
>>,[
    # Check if that character was a digit.
    $is_digit(false→1, true→0)

    # Until we find a digit:
    >[
        # Read the next character.
        <,
        $is_digit(false→1, true→0)
    ]

    # Set last_seen to the first digit, and add it ten times to the total.
    <------------------------------------------------
    [-<+<++++++++++>>]

    # Read characters until a newline
    ,----------[
        ++++++++++
        # If this character is a digit, store it in last_seen.
        $is_digit(false→0, true→1)
        >[
            [-]
            <------------------------------------------------
            <[-]>[-<+>]>
        ]

        # Read the next character.
        ,----------
    ]

    # Add last_seen to total.
    <[-<+>]>

    # Read the next character (start of the next line).
    ,
]

<<
# Print total,
[>>>>++++++++++<<<<[->+>>+>-[<-]<[<<[->>>+<<<]>>>>+<<-<]<<]++++++++[->++++++<]>[-<+>]>>>>[-<<<<+>>>>]<[-]<<<]<[.<]
# then a newline.
++++++++++.

Running this is a bit of a challenge: the input is 1000 lines long, and each line can add at most 99 to the total, so our total cell needs to be able to hold the number 99000, which rules out 8-bit and 16-bit cell widths. Fortunately, there is an optimizing brainfuck compiler which can recognise some of the constructs in our program and produce equivalent code that runs much faster.

Onwards to Part 2

That wasn’t too tricky, was it? Unfortunately, Part 2 is here to ruin our fun. Now our digits can also be spelled out with letters, so we need to recognise one, two, …, nine. Uh oh, that’s going to be a lot of conditionals…

Here’s the new memory layout:

total: The total of all calibration values.
last_seen: The last-seen digit (as a value 0–9).
multiplier: How many times to add a digit to the total.
c4: The fifth-last character.
c3: The fourth-last character.
c2: The third-last character.
c1: The second-last character.
c0: The last (current) character.
match: does the current word match so far? / else_flag: temporary variable for else statements.
c_copy: Temporary variable, primarily for copying cN.

There’s really only two changes here: we’ve introduced multiplier and a sliding window of the last five input characters. One of the problems of the Part 1 solution is that $is_digit had to be repeated three times. With the addition of the words for digits, we’re going to have a lot of extra code for recognising digits, so we really want to restructure the code to avoid that repetition.

This is where the multiplier cell comes in. We’re going to simplify the algorithm so we just have one loop reading input characters:

Set multiplier to 10
Until the end of the input:
1. Shift the sliding window (c0 goes to c1, … c3 goes to c4)
2. Read a character into c0
3. If that character was a newline:
  1. Add the last-seen digit to the total
  2. Set multiplier to 10
4. For each digit word (including the actual digits), if the word appears at the end of the sliding window:
  1. Set the last-seen digit to the digit’s value
  2. Add the digit’s value multiplier times to total
  3. Set multiplier to 0
Print the total

The first time we see a digit, multiplier will be 10, so we’ll add it to the total ten times. But for every digit after that, multiplier will be 0, so although the same code path will be run, we’ll add it to the total zero times, having no effect. Then once we see the newline, we’ll add the last digit to the total, and reset multiplier for the first digit of the next line.

Finding words in the sliding window

The phrase "if the word appears at the end of the sliding window" in the algorithm above is suspiciously high-level for a Brainfuck program. Implementing it turns out to be not too difficult, albeit extremely verbose.

Here’s some pseudo-code to explain the approach:

function test_character($letter, in c, out match, temp c_copy):
    c -= $letter
    if c != 0:
        c_copy += c
        c = 0
        match = 0
    c += c_copy
    c += $letter

function test_word($word, in c4, in c3, in c2, in c1, in c0, out match, temp c_copy):
    match = 1
    $if length($word) >= 5:
        $test_character($word[-5], c4, match, c_copy)
    $if length($word) >= 4:
        $test_character($word[-4], c3, match, c_copy)
    $if length($word) >= 3:
        $test_character($word[-3], c2, match, c_copy)
    $if length($word) >= 2:
        $test_character($word[-2], c1, match, c_copy)
    $if length($word) >= 2:
        $test_character($word[-1], c0, match, c_copy)

For 'two', that turns into the following Brainfuck code (the last-seen digit and multiplier adjustments are included):

# match = 1
[-]+

<<<
# $index = 2, $letter = 't'
    # c2 -= 't'
    --------------------------------------------------------------------------------------------------------------------
    # if c2 != 0:
    [
        # c_copy = c2; c2 = 0
        [->>>>+<<<<]
        # match = 0
        >>>[-]<<<
    ]
    # c2 += c_copy; c_copy = 0
    >>>>[-<<<<+>>>>]<<<<
    # c2 += 't'
    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    >

# $index = 1, $letter = 'w'
    # c1 -= 'w'
    -----------------------------------------------------------------------------------------------------------------------
    # if c1 != 0:
    [
        # c_copy = c1; c1 = 0
        [->>>+<<<]
        # match = 0
        >>[-]<<
    ]
    # c1 += c_copy; c_copy = 0
    >>>[-<<<+>>>]<<<
    # c1 += 'w'
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    >
# $index = 0, $letter = 'o'
    # c0 -= 'o'
    ---------------------------------------------------------------------------------------------------------------
    # if c0 != 0:
    [
        # c_copy = c0; c0 = 0
        [->>+<<]
        # match = 0
        >[-]<
    ]
    # c0 += c_copy; c_copy = 0
    >>[-<<+>>]<<
    # c0 += 'o'
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    >

Putting it together (again)

Once we’ve tested a word, match will be 1 if we found the digit word at the end of the sliding window, so we’re nicely set up to make the adjustments to last_seen, multiplier and total from our algorithm.

function check_digit_word($word, $value, in c4, in c3, in c2, in c1, in c0, temp match, temp c_copy):
    $test_word($word, c4, c3, c2, c1, c0, match, c_copy)
    if match != 0:
        match = 0
        last_seen = $value
        total += $value * multiplier
        multiplier = 0

For 'two', $value = 2 and will turn into (omitting $test_word) this Brainfuck code:

# if match != 0
[
    # match = 0
    [-]
    # last_seen = 2
    <<<<<<<[-]++
    # total += 2 * multiplier; multiplier = 0
    >[-<<++>>]
    >>>>>>
]

We now have everything we need to implement the entire algorithm:

# set multiplier to 10
>>++++++++++
# c0 = read()
>>>>>,
# while c0 != eof:
[
    # else_flag = 1
    >+<
    # if c0 != '\n'
    ----------[
        # c_copy = c0; c0 = 0
        [->>+<<]
        # else_flag = 0
        >[-]<
    ]
    # c0 += c_copy
    >>[-<<+>>]<<
    # c0 += '\n'
    ++++++++++
    # else / if else_flag == 1 / if c0 == '\n'
    >[
        # else_flag = 0
        [-]
        # add last-seen digit to total
        <<<<<<<[-<+>]
        # multiplier = 10
        >[-]++++++++++
        >>>>>>
    ]

    $check_digit_word($word="one", $value=1)
    $check_digit_word($word="two", $value=2)
    $check_digit_word($word="three", $value=3)
    $check_digit_word($word="four", $value=4)
    $check_digit_word($word="five", $value=5)
    $check_digit_word($word="six", $value=6)
    $check_digit_word($word="seven", $value=7)
    $check_digit_word($word="eight", $value=8)
    $check_digit_word($word="nine", $value=9)
    $check_digit_word($word="0", $value=0)
    $check_digit_word($word="1", $value=1)
    $check_digit_word($word="2", $value=2)
    $check_digit_word($word="3", $value=3)
    $check_digit_word($word="4", $value=4)
    $check_digit_word($word="5", $value=5)
    $check_digit_word($word="6", $value=6)
    $check_digit_word($word="7", $value=7)
    $check_digit_word($word="8", $value=8)
    $check_digit_word($word="9", $value=9)

    # update the character window
        # c4 = 0
        <<<<<[-]
        # c4 += c3; c3 = 0
        >[-<+>]
        # c3 += c2; c2 = 0
        >[-<+>]
        # c2 += c1; c1 = 0
        >[-<+>]
        # c1 += c0; c0 = 0
        >[-<+>]
        # c0 = read()
        ,
]

<[-]<[-]<[-]<[-]<[-]<[-]<
# Print total
[>>>>++++++++++<<<<[->+>>+>-[<-]<[<<[->>>+<<<]>>>>+<<-<]<<]++++++++[->++++++<]>[-<+>]>>>>[-<<<<+>>>>]<[-]<<<]<[.<]
# Print '\n'
++++++++++.

A complete code listing, debugging interpreter and a tool to strip comments/whitespace are available in the repository.

Advent of Code 2023: Introduction

2023-12-10T00:00:00+11:00

Advent of Code is a set of Christmas-themed programming problems. The problems are released daily from December 1–25, and each problem consists of two parts, with the second part not visible until the first part has been solved (usually, this is an extension that makes the problem slightly harder).

In previous years (2021, 2022), I’ve had the self-imposed restriction of using a different language for each day’s puzzle. I’ve enjoyed this restriction: it has prompted me to use all kinds of different langauges that I wouldn’t normally use. I view this restriction as encouraging me to create art with my code: rather than engineering the best solution (however you want to define that) to the problem, I’m trying to create interesting and beautiful solutions.

Unfortunately, I was busy at the start of December, so was unable to do the problems as they were released. But I’ve chosen to view this as a blessing in disguise: rather than pushing myself to solve each day’s problem before the next one is released, I can take my time to produce refined solutions. Hopefully, this will allow me to use interesting languages for more days.

I’m also going to use the lack of time pressure to write up some more detailed commentary on my solutions, which I’ll post here.

Ringsteady Sorting in Linear Time

2023-08-23T00:00:00+10:00

Last year, I wrote an article for ACM Queue describing an algorithm I co-invented for backend subsetting (and recently Netflix published that they’d adopted parts of it). This post is really just a footnote to that article, something which was originally included but had to be dropped due to word limits.

As a quick recap, to build our algorithm we start with Consistent Hashing and extend it to return a subset of backend tasks rather than just a single task (coined Consistent Subsetting). Ringsteady Subsetting is then a specific case of Consistent Subsetting that improves connection balance, by replacing the randomly chosen positions with positions from a low-discrepancy sequence.

The particular sequence we chose was the binary van der Corput sequence, and the article notes that one of the reasons is the ease of computing any individual element, but another reason is that it enables a particular optimisation.

The most straightforward way to implement this is to generate pairs of each backend’s index and position, and sort them by position. Here’s a Rust implementation¹ (this uses backend scaling², so corresponds to Figure 4c in the article):

// Copyright 2023 Google LLC.
// SPDX-License-Identifier: Apache-2.0

struct Backend {
    /// The backend's index in the range [0, n)
    index: usize,
    /// Its position on the circle, as a fixed-point number
    /// (i.e. usize::MAX / 2 is halfway around the circle)
    position: usize,
}

impl Backend {
    pub fn new(index: usize, position: usize) -> Self {
        Backend { index, position }
    }
}

fn ringsteady_naive(num_backends: usize, frontend_index: usize, subset_size: usize) -> Vec<usize> {
    // Generate a vector of [index, position] for each backend.
    let mut backends = Vec::new();
    for backend_index in 0..num_backends {
        let backend_position = backend_index.reverse_bits();
        backends.push(Backend::new(backend_index, backend_position));
    }

    // Sort the vector by each backend's position.
    backends.sort_by(|a, b| a.position.cmp(&b.position));

    // Get position of this frontend on the circle as a float in the range [0, 1).
    let frontend_unit_position = (frontend_index.reverse_bits() as f64) / 2f64.powi(64);
    // Figure out how many backends this frontend needs to rotate by.
    let rotation = (frontend_unit_position * (num_backends as f64)).ceil() as usize;

    // Output the backend indices.
    let mut subset = Vec::new();
    for i in 0..subset_size {
        subset.push(backends[(i + rotation) % num_backends].index);
    }
    subset
}

fn main() {
    // OUTPUT:
    // Frontend #0: [0, 4]
    // Frontend #1: [1, 5]
    // Frontend #2: [2, 1]
    // Frontend #3: [3, 0]
    // Frontend #4: [4, 2]
    for m in 0..5 {
        let subset = ringsteady_naive(6, m, 2);
        println!("Frontend #{}: {:?}", m, subset);
    }
}

Sorting these pairs is the most expensive part of this algorithm. Wouldn’t it be neat if we could side-step that and just write out the backends in the correct order?

The key insight is that if we have 2^w backends (for some integer w), then the positions of our backends will be a bit-reversal permutation, which is an involution (i.e. reversing the bits of a number twice gives the same number). This means that not only does reversing the bits give us the position for a backend index, it also gives us the backend index for a position. For example, here’s w = 3:

Backend Index	Position	Position	Backend Index
0	0	0	0
1	4	1	4
2	2	2	2
3	6	3	6
4	1	4	1
5	5	5	5
6	3	6	3
7	7	7	7

To extend this to any number of backends N, we just need to find the smallest integer w such that N ≤ 2^w, iterate over the positions [0, 2^w), reversing each position (as a w bit number) to get the backend index at that position, skipping any backend index that isn’t in [0, N).

Here’s what that looks like in Rust:

// Copyright 2023 Google LLC.
// SPDX-License-Identifier: Apache-2.0

fn ringsteady_linear_time(num_backends: usize, frontend_index: usize, subset_size: usize) -> Vec<usize> {
    // Generate a vector of backend indices in the correct order.
    let mut backends = Vec::new();
    let num_positions = num_backends.next_power_of_two();
    let w = num_positions.trailing_zeros();
    for backend_position in 0..num_positions {
        let backend_index = backend_position.reverse_bits() >> (usize::BITS - w);
        if backend_index < num_backends {
            backends.push(backend_index);
        }
    }

    // Get position of this frontend on the circle as a float in the range [0, 1).
    let frontend_unit_position = (frontend_index.reverse_bits() as f64) / 2f64.powi(64);
    // Figure out how many backends this frontend needs to rotate by.
    let rotation = (frontend_unit_position * (num_backends as f64)).ceil() as usize;

    // Output the backend indices.
    let mut subset = Vec::new();
    for i in 0..subset_size {
        subset.push(backends[(i + rotation) % num_backends]);
    }
    subset
}

fn main() {
    // OUTPUT:
    // Frontend #0: [0, 4]
    // Frontend #1: [1, 5]
    // Frontend #2: [2, 1]
    // Frontend #3: [3, 0]
    // Frontend #4: [4, 2]
    for m in 0..5 {
        let subset = ringsteady_linear_time(6, m, 2);
        println!("Frontend #{}: {:?}", m, subset);
    }
}

It’s fairly straightforward to show that 2^w < 2N (if it wasn’t, there would be a smaller valid choice of w), so the time complexity is linear in N. It’s also much faster in practice (even for small N), since we avoid the memory read/writes from swapping items in the sort algorithm.

Disclaimer: Any opinions stated here are my own, and not necessarily shared by my employer (Google).

I have preferred readability over performance or idiomatic code in these snippets, in the hope that it will be understandable to readers unfamiliar with Rust. For instance, there is no need for floating-point when calculating the frontend rotation, but I couldn’t figure out how to do that without distracting amounts of boilerplate. ↩
It’s possible to use this optimization without backend scaling, but it made the sample code too verbose for this blog post. ↩

Galactic Puzzle Hunt 2019: Reflection

2019-04-08T00:00:00+10:00

I recently organized and participated in a team for the 2019 edition of the Galactic Puzzle Hunt. This post is my reflections on that process.

Organizing a team

Teams for the puzzle hunt can be up to 10 people [1], so the first step was to attempt to find other people who are at least as insane as I am. I put out a tweet linking to a Google Form (and also messaged a few friends), asking (amongst a few other things) for a time commitment, with the intent that I’d select team members most happy to spend time on it.

As it happened, I only got seven signups (including myself), so that was my team formed. I thought that was a reasonable number, but it did have a lot of people only able to make a small time commitment, and I should have kept searching for more people. My perception was we had ~2.5 daytime solvers on average.

There were a few people who gave us some help during the hunt, so I think that means I need to promote signups more next year.

Having brought together some people, we needed a team name. A bunch of ideas we had (some of which followed the Antarctica theme for the puzzle hunt):

Production Freeze
By accepting this team name you agree to our EULA
Fizz puzz
Ice climbers
TE-901
UNTITL~1.GPH

But “How about ☃ repeated as many times as the registration form will allow?” was most liked. Confession time: I didn’t actually do this. Instead, I went for three rows of snowmen (as displayed on my laptop). So while we didn’t have the longest team name, we did have the most vertical space in the team list.

Perhaps next year we’ll figure out how to consume even more vertical space with our team name?

Team coordination

We used Slack and Google Docs to coordinate. In the Slack workspace, there was a #coordination channel which had login details, a periodically updated “best puzzles to work on” post, celebrations of solved puzzles, and occasional calls for help. Puzzle solving didn’t happen in that channel: instead, we created a channel per puzzle.

It’s hard to predict what technology will be best to solve a puzzle, but generally we always want a document to describing what we’ve tried and with partial progress (which eventually morphs into a “how to solve the puzzle” doc). All of these documents got put into a shared Google Drive folder, and each puzzle channel in Slack had a topic linking to the relevant doc for easy access.

A few of the puzzles needed more than a doc. We also used:

Photos shared in Slack (for a physical jigsaw-like puzzle)
Images shared in Slack
Google My Maps in Drive (super-useful for visualising latitude/longitude pairs and looking for patterns)
Google Spreadsheets in Drive (although a few times I started to create one, and then replaced it with a table in a Google Doc)
Google Colaboratory in Drive (omg sharable Python code with notes!)

I was generally happy with the technology choices, although I need to figure out how to use Colaboratory in a way that involves less clicking and makes better use of my vim muscle memory.

Puzzle solving

Something I keep saying about puzzle hunts is that it’s important to have lots of people on the team, even if they aren’t spending gross amounts of time solving puzzles. What’s really important is that there’s a continuous stream of “have you tried <thing>?” ideas and “that looks kinda like <other thing>” observations: otherwise, you get stuck trying to solve a puzzle and then get frustrated at not having ideas, which isn’t a great mental state to be producing ideas with.

I’m very grateful to my team members for contributing their time: I think I spent less time in that stuck state than I have in previous years, but there’s definitely still room for improvement there.

There were also a few times where I think we had trouble because we didn’t quite know what “rules” the puzzle authors were playing by, so to speak. In one case, I wrote a bunch of Python code looking for patterns in some data because of a perceived hint in the puzzle title, which turned out to be a wild goose chase. There was also at least one hint spent on something where we should have just been able to get it out by being more rigorous.

Results

Our team solved the first 12 puzzles, which put us in 239th place.

Out of 717 registered teams: top 33%
Out of 505 teams who solved at least one puzzle: top 47%

I’m pretty happy with how we went: I only wish we’d managed to finish our half-solved metapuzzle in time!

For next year’s Galactic Puzzle Hunt (or any other ones for that matter), I’ve been thinking lately about doing some training sessions (where you solve a puzzle, but there’s zero-cost hints as soon as you get stuck). If this sounds interesting, hit me up?

[1]	Although the rules also say “You may use any external sources for help, including other people, as long as they aren’t helping other teams and aren’t actively participating in the hunt.”

Python C Extension Module Corruption

2018-06-23T00:00:00+10:00

If you follow me on Twitter, you might have seen this recent thread, in which I spent a few days metaphorically banging my head against a wall trying to figure out why my Python program would occasionally segfault (with no apparent trigger).

This post is an attempt to write up the cause (and my workaround) in a clearer form, and hopefully make it Google-able for the next poor soul to encounter this (if it's not fixed before then).

Dramatis Personæ

The program in question here is a Python web application, written in Flask, and run using gunicorn (using a few sync workers). It also uses SQLAlchemy (via Flask-SQLAlchemy) to access a sqlite3 database. It’s all running on a Google Compute Engine VM running Debian Linux.

Most of those details aren’t relevant, but I mention them just to give a sense of the scope: each of them was a suspect at some stage or another during my investigation.

The Evidence

Aside from seeing 5xx responses from my web server, the only information I had was in syslog:

kernel: [1234.567890] gunicorn[1234]: segfault at ad0 ip 0000000000000ad0 sp 00007fffc0401448
                                      error 14 in python3.5[55f0889ee000+3ef000]

The gunicorn logs noted that it had started a new worker to replace the segfaulted one, but nothing more. The syslog entry is really quite odd: the at ad0 and ip 0000000000000ad0 parts indicate that the program tried to jump to (and hence execute) code at the address 0xad0, which is quite suspicious: I don't know offhand what's at that address, but it’s not usually program code.

Clearly I needed more information to debug this, so I tried to get a core dump. For reasons I’m still not entirely sure of, getting a core dump didn’t just work out-of-the-box—I believe Linux is meant to write a file named core to the process’s working directory (provided ulimit is set correctly).

Some quick Googling revealed that systemd-coredump is a thing, so I installed that, and then my program didn’t segfault for an hour, so I tweeted thinking that this was a Heisenbug. It turned out not to be the case: the segfault only happened a few times an hour, and I was just unlucky.

Anyhow, I later came back and used coredumpctl list to see when the program had segfaulted, and coredumpctl gdb to launch gdb with the core dump loaded. For all the hate systemd seems to get for reinventing the wheel, I will say that systemd-coredump seems quite good at its job.

$ coredumpctl gdb
...
Core was generated by `/usr/bin/python3'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000000ad0 in ?? ()
(gdb)

Great: there's the 0xad0 address from my syslog entry, so the crash is the same as the one I’m seeing! This might seem like a minor point, but fervently verifying the world is as you expect it to be is critically important in debugging. Let’s look at a backtrace to see what the program was doing when it segfaulted:

(gdb) bt
#0  0x0000000000000ad0 in ?? ()
#1  0x0000556dce8116df in PyCFunction_Call () at ../Objects/methodobject.c:109
#2  0x0000556dce7d16e9 in call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e1e0) at ../Python/ceval.c:4720
#3  PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#4  0x0000556dce7d193f in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffd2fc7e310, func=<optimized out>) at ../Python/ceval.c:4818
#5  call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e310) at ../Python/ceval.c:4745
#6  PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#7  0x0000556dce7d193f in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffd2fc7e440, func=<optimized out>) at ../Python/ceval.c:4818
#8  call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e440) at ../Python/ceval.c:4745
#9  PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#10 0x0000556dce7d6286 in _PyEval_EvalCodeWithName () at ../Python/ceval.c:4033
#11 0x0000556dce7d24c9 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffd2fc7e650, func=<optimized out>) at ../Python/ceval.c:4828
#12 call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e650) at ../Python/ceval.c:4745
#13 PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#14 0x0000556dce7d193f in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffd2fc7e780, func=<optimized out>) at ../Python/ceval.c:4818
#15 call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e780) at ../Python/ceval.c:4745
#16 PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#17 0x0000556dce7d70df in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=<optimized out>, argcount=<optimized out>,
    args=<optimized out>, locals=<optimized out>, globals=<optimized out>, _co=<optimized out>) at ../Python/ceval.c:4033
#18 PyEval_EvalCodeEx () at ../Python/ceval.c:4054
#19 0x0000556dce8135d3 in function_call.lto_priv () at ../Objects/funcobject.c:627
#20 0x0000556dce85a647 in PyObject_Call () at ../Objects/abstract.c:2166
#21 0x0000556dce77a94e in method_call.lto_priv () at ../Objects/classobject.c:330
#22 0x0000556dce85a647 in PyObject_Call () at ../Objects/abstract.c:2166
#23 0x0000556dce81aebf in slot_tp_iter () at ../Objects/typeobject.c:6214
#24 0x0000556dce85a4a3 in PyObject_GetIter () at ../Objects/abstract.c:2755
#25 0x0000556dce84cd56 in listextend.lto_priv.683 (
    b=<BaseQuery(_group_by=False, _params={<_anonymous_label at remote 0x7f26c3adbd48>: 'flowblok.id.au'}, _entities=[<_MapperEntity(entity_zero=<Mapper(self_and_descendants=<WeakSequence(_storage=[<weakref at remote 0x7f26c3a9bf98>]) at remote 0x7f26c3aae898>, inherits=None, version_id_generator=<function at remote 0x7f26c3e837b8>, confirm_deleted_rows=True, _readonly_props=set(), local_table=<Table(name=<quoted_name at remote 0x7f26c3e8c1c8>, description=<...>, _cloned_set={<...>}, constraints={<ForeignKeyConstraint(name='domain_admin_group_constraint', initially=None, dialect_kwargs=<_DialectArgView(obj=<...>) at remote 0x7f26c82b5160>, parent=<...>, _referred_table=<Table(name=<quoted_name at remote 0x7f26c3e8cbc8>, description=<...>, _cloned_set={<...>}, constraints={<ForeignKeyConstraint(name=None, initially=None, parent=<...>, _referred_table=<...>, columns=<ColumnCollection at remote 0x7f26c3e22cc0>, ondelete=None, elements=[<ForeignKey(name=None, column=<Column(_cloned_set={<...>}, constraints=set(), forei...(truncated),
    self=0x7f26c3433608) at ../Objects/listobject.c:831
...
(truncated for your sanity)

This isn’t terribly useful: it’s full of Python interpreter stuff. Knowing the title of the blog, you might notice that the calling function is PyCFunction_Call, which you might start to get ideas from: but that could be from anything. Fortunately, gdb has amazing Python integration, so I can do this instead:

(gdb) py-bt
Traceback (most recent call first):
  <error reading variable: Cannot access memory at address 0xe90>
  File "/…/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1024, in _execute_clauseelement
    distilled_params = _distill_params(multiparams, params)
  File "/…/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/…/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2958, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2935, in __iter__
    return self._execute_and_instances(context)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2864, in one_or_none
    ret = list(self)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2894, in one
    ret = self.one_or_none()
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/loading.py", line 250, in load_on_pk_identity
    return q.one()
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 962, in _get_impl
    return db_load_fn(self, primary_key_identity)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 887, in get
    ident, loading.load_on_pk_identity)
  File "/…/app.py", line 47, in setup_context
...
(truncated for your sanity)

The last frame of that backtrace is my application code, trying to run a SQLAlchemy query. The rest of the backtrace is SQLAlchemy trying to execute it. But the first frame is most interesting: SQLAlchemy is calling a function _distill_params, but we don't have the source for that call.

If you look at the source for sqlalchemy/engine/base.py, you’ll see that _distill_params gets imported from sqlalchemy/engine/util.py, which has this:

try:
    from sqlalchemy.cutils import _distill_params
except ImportError:
    globals().update(py_fallback())

That is, _distill_params comes from a C extension module, unless it’s not importable, in which case there’s a pure Python version as a fallback.

C Extension Modules

At this point, I’ve figured out the first part of what’s gone wrong: something is causing Python to think that the address of the _distill_params function is 0xad0, instead of whatever it’s meant to actually be.

For the next part, we need to delve a little into how C extension modules work. Here’s part of the source for sqlalchemy/cextension/utils.c, which provides the sqlalchemy.cutils module imported above.

static PyMethodDef module_methods[] = {
    {"_distill_params", distill_params, METH_VARARGS,
     "Distill an execute() parameter structure."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

This C file gets compiled into a shared library: on my system it’s a file named cutils.cpython-35m-x86_64-linux-gnu.so. The CPython interpreter loads extension modules in Python/importdl.c, which uses the _PyImport_FindSharedFuncptr function in Python/dynload_shlib.c, which uses dlopen to load that .so file into memory and get a function pointer.

Most of those details weren’t important: the key part is that we have some C code, being loaded into the process’s memory at runtime. So, let’s attach GDB to a running process, and have at look at that module_methods array:

$ gdb -p 1234
...
(gdb) p 'sqlalchemy/cextension/utils.c'::module_methods
$1 = {{ml_name = 0x7f66abdfde90 "_distill_params", ml_meth = 0x7f66abdfdad0 <distill_params>,
       ml_flags = 1, ml_doc = 0x7f66abdfdef8 "Distill an execute() parameter structure."},
      {ml_name = 0x0, ml_meth = 0x0, ml_flags = 0, ml_doc = 0x0}}

That’s more or less what I’d expect to see: ml_name has the Python name of the function, and ml_meth is a pointer to the distill_params C function. What happens if we try the same thing in our core dump?

$ coredumpctl gdb
...
(gdb) p 'sqlalchemy/cextension/utils.c'::module_methods
$1 = {{ml_name = 0xe90 <error: Cannot access memory at address 0xe90>, ml_meth = 0xad0,
       ml_flags = 1, ml_doc = 0xef8 <error: Cannot access memory at address 0xef8>},
      {ml_name = 0x0, ml_meth = 0x0, ml_flags = 0, ml_doc = 0x0}}

Well. That’s not right.

But wait, those hex numbers look kinda similar to the correct numbers…

>>> print('{:48b}'.format(0x7f66abdfde90))
 11111110110011010101011110111111101111010010000
>>> print('{:48b}'.format(0xe90))
                                    111010010000

>>> print('{:48b}'.format(0x7f66abdfdad0))
 11111110110011010101011110111111101101011010000
>>> print('{:48b}'.format(0xad0))
                                    101011010000

Huh? Sometime between being a perfectly happy process chugging along, and it segfaulting, the module_methods struct in a C extension module gets all its numbers truncated to 12 [*] bits‽

Time and Space

Thinking that the problem was something writing to the memory to corrupt it, I spent quite some time looking through strace and ltrace outputs to try and find something touching the relevant addresses in memory, but to no avail.

My breakthrough came when I compared /proc/$PID/smaps from a running process and a almost-segfaulted process. To capture the latter, I attached GDB, and then typed in shell cat /proc/1234/smaps > 1234.smaps and hit enter, so it would get run when the program had segfaulted (but before Linux forgot the process state). I also attempted using a breakpoint, but in hindsight I don’t think that worked. Regardless, I found the section of memory in which the module_methods struct resided, and found this difference:

Before:

7f26c44e2000-7f26c44e3000 rw-p 00002000 08:10 666070 /…/cutils.cpython-35m-x86_64-linux-gnu.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac sd

After:

7f26c44e2000-7f26c44e3000 rw-p 00002000 08:10 666070 /…/cutils.cpython-35m-x86_64-linux-gnu.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB

The key difference here is that this section of memory went from having 4 kB of Private_Dirty data to having 4 kB of Private_Clean data.

Combined with the knowledge that the numbers inside that section of memory seemed to be being “truncated”, I realised what was happening: this section of memory was being reloaded from the shared library file. I’m glossing over some details here, but on disk the shared library has various pointers to code and data within that shared library, but when it gets loaded into a process’s memory, it’ll be loaded at some offset, and all the pointers within the shared library need to be adjusted in order to work.

So on disk, inside the module_methods struct, the ml_name pointer is actually 0xe90, but when it initially gets loaded into memory, it gets adjusted to 0x7f66abdfde90 so it’s pointing at the location where the string "_distill_params" got loaded into memory.

The Piping Gun

Knowing that I was looking for something which was causing the shared library file to be reloaded from disk, I remembered that I had a bug in my automation which was unnecessarily reinstalling all Python libraries using pip. I didn’t think it was a huge problem: the downloads are cached locally, and reinstalling a package with the same version should just be a no-op, right?

Turns out not so much. When pip installs a wheel, it replaces all the files using shutil.copyfile. This opens the file for writing, truncates it, and then copies the content in.

Because this is changing the same file which is also memory mapped into a running process, Linux notices that section of data has been re-written (even with exactly the same bytes), and invalidates all unwritten changes in active processes (this is what makes the memory mapping go from Private_Dirty to Private_Clean), thus undoing the offset adjustments the process made when loading the shared library.

Honestly, I’m not sure where the bug is here. Probably after loading a shared library, Python should unassociate the data section from the file it was loaded from: though I’m not sure if the C functions to do that even exist. I’d also say pip should do atomic file replacements to avoid the problem, but this is more of a bandaid than an actual fix.

My Workaround

After figuring out what was happening, I had a realisation that this problem could be mitigated in the application, with a pure Python solution.

Instead of trying to unassociate the memory mapping, I can atomically replace the file it loaded with a copy of itself, thus making the memory mapping in the process point at data which no longer exists in the filesystem.

So here is a Python module which you can import, which will periodically find C extension modules loaded into the Python process and perform shenanigans on them to make them resilient against this failure mode. This potentially costs memory, but I doubt it’ll ever be an amount you care about.

# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
'''
Module sopin implements "pinning" of shared libraries.

This is a workaround for the problem where rewriting a Python C extension module
on disk corrupts the state of any Python process which has previously imported
it: for more details, see
https://blog.flowblok.id.au/2018-06/python-c-extension-module-corruption.html

Usage:
    import sopin
    sopin.start()
'''
import os
import shutil
import sys
import tempfile
import threading
import warnings

SUFFIXES = ('.so',)

# Modules previously pinned.
_pinned = set()
# Modules which we couldn't pin (so shouldn't reattempt).
_blacklisted = set()

def pin(filename):
    if filename in _pinned or filename in _blacklisted:
        return
    try:
        # Atomically create a unique tempfile.
        # Don't delete, because we'll move it over the old file.
        with tempfile.NamedTemporaryFile(delete=False) as f:
            # Copy the content+permissions of the old file into our new file.
            shutil.copy2(filename, f.name)
            # Move the new file over the old file.
            os.rename(f.name, filename)
    except IOError:
        warnings.warn('could not pin %s' % filename)
        _blacklisted.add(filename)
    else:
        _pinned.add(filename)

def pin_all():
    for name, module in sys.modules.items():
        try:
            filename = module.__file__
        except AttributeError:
            continue
        if filename.endswith(SUFFIXES):
            pin(filename)

def start(period_seconds=10):
    threading.Timer(period_seconds, pin_all).start()

[*]	Twitter followers may note this is not the 13 or 14 that I tweeted: I was actually debugging `processors.c` at that point, so these details are slightly different.

Synchronized Music on micro:bits

2018-02-24T22:02:00+11:00

It’s been a while since I last posted. So, uh, sorry about that. I’m going to make it up to y’all with this post. It’s in two parts: the video demonstrates the project in action, and the rest of this blog post details all the technical details. I’d recommend starting with the video to see the cool demo, and if you make it to the end, read the rest of the blog post.

Introduction

For many years now, I’ve tutored at the National Computer Science School, which teaches computer science and programming to Australian high school kids. The school is divided into two parts: a web stream, which builds a website using Python, and an embedded stream, which (nowadays) uses MicroPython on the micro:bit.

Last year I tutored the web stream, and so it was only during the closing ceremony while watching videos demonstrating all the embedded students’ projects that I started wondering about the potential of the micro:bit, and whether I could build a mesh network so they could play music synchronized across a large area. So when work started getting quieter over the Christmas / New Year break, and this year’s NCSS started approaching, I decided to buy a couple of micro:bits and start hacking.

Breaking it down

I broke down the project into a number of distinct components:

network protocol: for sending structured data as micro:bit radio packets.
time sync: getting all micro:bits on the network to share a common time.
triggers: getting all micro:bits to do an action at a given time.
music: making the micro:bits play notes on a piezo buzzer.

Writing micro:bits all the time is too hard, so I’m going to refer to them from here on as nodes.

Time Sync

Time is hard. It’s a little easier in this application, because we don’t care what the actual wall-clock time is, just that all the nodes have the same time.

The first step in figuring out how to synchronize the nodes was to head off to Wikipedia and do some reading on the way I already knew computers synchronized time, Network Time Protocol (NTP). After very little reading, it seemed fairly obvious that NTP was far too complicated for this application: I didn’t need to deal with leap seconds, and polling multiple servers and doing filtering on the answers seemed like too much effort.

Fortunately it took little effort to find the Precision Time Protocol (PTP). PTP is a really nice simple protocol, though the Wikipedia page does have much more detail than is really needed. It’s designed to work on multiple network segments (hey, that sounds kinda like the network topology we have), and on each segment, there’s a master which all the other nodes synchronize to. Each node periodically pings (called Delay_Req and Delay_Resp, but I like ping better) the master to approximate their Round Trip Time (RTT), and the master periodically sends out a Sync message with its current time. Add in some maths, and hey presto, you have synchronized time.

In order to adapt the algorithm for a mesh network, there were a few changes I needed to make. PTP has a best master clock algorithm for selecting the master, but since my nodes might drop in and out of network connectivity all the time, my system needed to continuously adapt which nodes were sending timestamps to synchronize to, based on which nodes could talk to them. I decided to term those nodes thought leaders (partially because I really disliked the "master" terminology, partially because I like being stupid 😛).

Network Protocol

All nodes in my network are assigned an id and level. Given the scale of my network, both of these are unsigned bytes (i.e. 0–255). The id is chosen randomly at startup. Astute readers will note this falls victim to the Birthday Problem, so after 20 nodes there’s a 50% chance two nodes will share an id. This turns out not to actually be a problem in practice: the presence of duplicate ids on the network is only an issue if another node can see both of them (cough and the nodes are sufficiently unreliable that they restart occasionally anyway cough).

Levels: One node is designated as the "root" (by pressing buttons on the micro:bit), and is assigned level 0. All other nodes start with level 31 (chosen because it can be easily displayed in binary on the micro:bit LEDs). The level is an estimate of how close the node is to the root, and nodes will only synchronize with nodes closer to the root than it. As nodes do so, they decrease their own level (but never more than one more than the level of the node it’s synchronizing with). If the node hasn’t managed to synchronize for a few seconds, its level increases by one, ensuring that nodes always choose good nodes to synchronize with.

Here’s an example network:

Diagram showing several nodes with levels, with arrows indicating which nodes synchronize from each other.

The level 1 and 4 nodes can both synchronize from the level 0 node (since it’s less than 1 and 4). The level 4 doesn’t synchronize from the level 2 node in this example, just because it can’t see it over the radio network (but it would if it could). As the level 4 node becomes more in sync, it will reduce to a level 3 node, then a level 2 node, and finally a level 1 node—at which point it would stop synchronizing from the other level 1 node.

Pings: All nodes periodically send PING_REQUEST messages on the network, which contain:

req_node: the id of the node requesting the ping
req_level: the node’s level
ping_id: a unique (16 bit) id for this ping request
votes: a list of node ids this node would like to sync from.

The votes list is just the ids of nodes which the node has seen PING_REQUEST messages from, where req_level was less than its own: this may be out-of-date, but if so, the node can just ignore the SYNC message later (and it won't send that node id next time).

Any node which sees a PING_REQUEST message from another node responds back with a PING_RESPONSE message, which contains:

req_node: copied from the PING_REQUEST
resp_node: the id of the node responding to the ping
resp_level: the node’s level
ping_id: copied from the PING_REQUEST
req_end_timestamp: the time at which the PING_REQUEST was received, according to the responding node’s clock

Syncs: Periodically, nodes use the votes it has seen from other nodes to see if it’s the best node to send a SYNC message. If it is, it does, which contains:

node: the node sending the SYNC
level: the node’s level
timestamp: the node’s timestamp at time of sending
triggers: a list of trigger ids and timestamps (sent as deltas relative to timestamp)

When a node receives a PING from a node with a lower level, it uses the same formula from PTP to adjust its own clock. Assuming:

$T_1$ is the time the node sent a PING_REQUEST to the node sending a SYNC
$T'_1$ is the req_end_timestamp from the corresponding PING_RESPONSE
$T_2$ is the timestamp in the SYNC message
$T'_2$ is the time the SYNC message was received

Then the offset $õ$ between the two nodes’ clocks is:

\begin{equation*} õ = \frac{T'_1 - T_1 - T'_2 + T_2}{2} \end{equation*}

In practice, I use exponential smoothing on this process, which reduces the impact of outliers. Earlier I mentioned that nodes reduce their level as they become "more in sync": nodes actually reduce their level every time they do an offset adjustment where the adjustment is small enough ($|õ| < \varepsilon $, where $\varepsilon =$ 10ms).

Side note: Originally, I intended to use MicroPython on the micro:bit to implement this project. However, after just trying to implement something to decode the network protocol messages into in-memory data structures, I ran out of memory using MicroPython. I briefly toyed with the idea of having some code to generate the MicroPython code (so I could write sensible high-level code, and have it spit out stupid code), but that was far too much effort, so I switched to using the micro:bit C++ API.

Simulation

There’s a lot of detail in the previous section, without much explanation of why I designed it that way. This section isn’t going to help much, but it will explain the process a little better. When I started, I realised that attempting to develop the time synchronization protocol on the actual micro:bits would be very slow and tedious, so I started by writing a Python simulation.

The simulation let me quickly iterate on the design of the protocol, testing various number of nodes with different network connection topologies, and meant that when the network didn’t converge to a common time, I could debug the algorithm by adding things into my Python code, rather than messing around with micro:bit serial cables.

Unfortunately, I didn’t save all the diagrams along the way, but here’s the last one I generated, which shows 13 nodes all converging at different times to the same time (within an accuracy of 20ms).

A diagram showing latencies of various nodes, and lots of coloured circles. It’s not entirely clear what it represents.

The 0.1ms line in the diagram is a lie: it should be 0, but that doesn’t work so well with the logarithmic y-axis. The coloured dots on the lines indicate the level of the node at that point in time. Well. That’s what my notes claim, at least.

Triggers

So far, we’ve managed to get a mesh network of nodes synchronized to the same time, but we haven’t managed to get them to do anything useful yet. This is where triggers come in.

A trigger is just an event with an id (saying what you’d like to happen), and a timestamp in the future (when you’d like it to happen). Every SYNC message contains a list of triggers: the root node can schedule triggers, and every node in the network propagates on that list (while the triggers’ timestamps are still in the future) to all the other nodes.

In order to get triggers into the network, the root node listens on the serial port for a six-character hex number: the first two characters are a trigger id, and the remaining four are in how many milliseconds from now to schedule the trigger (for example, 2a0fa0 schedules trigger #42 to happen in 4 seconds).

Just before a trigger is scheduled to happen, the node checks if it’s reasonably in sync (the last adjustment $õ < \varepsilon $): if not, it skips the trigger (on the assumption that not playing some music is better than playing out of time).

Music

When a trigger fires, I want to play a segment (roughly five seconds long) of music. This means that if the node gets out of sync, it’ll only be silent for five seconds. Also, if the time synchronization is off by a noticable amount, it’ll only sound weird for five seconds.

Since I want the micro:bits to play music, I need some way to input music into the program. I started with Lilypond, an excellent music engraving program that I've used before (it’s kinda like LaTeX for music). As an example for this section, I’ve written Beethoven’s Ode to Joy in Lilypond:

\version "2.18.2"

\include "articulate.ly"

music = \new Staff \relative c'' {
  \tempo 4=160
  \time 4/4
  \key c \major
  e4 e f g |
  g f e d |
  c c d e |
  e4. d8 d2 |
  \bar "|."
}

\score {
  \articulate \music
  \midi {}
}

\score {
  \music
  \layout {}
}

This Lilypond file results in this PDF output:

It also generates a MIDI file (in case you’re wondering: the \articulate command just makes the MIDI output sound a lot nicer), which I can conveniently read using in Python using the Mido library, and thus generate some C++ arrays representing the data. These look like this:

typedef struct {
    uint16_t period_us;
    uint16_t duration_ms;
} music_event_t;

typedef struct {
    size_t start_event;
    size_t n_events;
} music_segment_t;

typedef struct {
    const music_event_t* events;
    size_t n_events;
    const music_segment_t* segments;
    size_t n_segments;
}

const music_event_t _song_events[] = {
    {.period_us = 1517, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1517, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1432, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1276, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1276, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1432, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1517, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1703, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1911, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1911, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1703, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1517, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1517, .duration_ms = 492},
    {.period_us = 0, .duration_ms = 70},
    {.period_us = 1703, .duration_ms = 164},
    {.period_us = 0, .duration_ms = 23},
    {.period_us = 1703, .duration_ms = 656},
};

const music_segment_t _song_segments[] = {
    {.start_event = 0, .n_events = 26},
    {.start_event = 26, .n_events = 3},
};

const music_data_t song_data = {
    .events = _song_events,
    .n_events = sizeof(_song_events) /
                sizeof(_song_events[0]),
    .segments = _song_segments,
    .n_segments = sizeof(_song_segments) /
                  sizeof(_song_segments[0]),
};

The resulting song_data has two segments, which index into the events array. The intention there was that if music has repeating sections, the segments could reference the same events, saving program space: as it turned out, I had plenty of program space and finding repeating sections is hard, so this was a bit of a premature optimization.

Then all that remained was to hook up trigger n to play segment n of the music, using PWM on the piezo buzzer I’d attached to pin 0 of my micro:bits.

Reliability

Having written all this code, I found that on the micro:bits, the program would occasionally get stuck: either the CPU would freeze up entirely, or the radio module would get into a bad state and not receive / transmit messages, or other weirdness would happen.

Instead of trying to fix these, I used the watchdog timer of the nRF51 (the CPU which runs the program): this lets me configure up to 8 "reload requests", and a period. If my application code doesn’t trigger all the enabled reload requests within the period, the CPU will reset itself.

I configured the period to two seconds, and set up two reload requests: one in the main loop that I use for display (as a "the program has just stopped" check), and one inside the time sync logic, which triggers the reload request iff the time has synced (again, $|õ| < \varepsilon $) within the last 30 seconds. This second check is really useful: if the network gets into a weird state where none of the nodes can sync, the nodes will all be obviously displaying this by resetting themselves every 30 seconds.

The other huge reliability concern was the root node in the network: if that dies, all the other nodes lose their sync. As a way to reduce this impact, I made it so any node could become the root node (by holding down some buttons): that way, if I noticed the root node die, I could promote a standby node, and since it already had almost the same time, the other nodes wouldn’t lose their synchronization.

Epilogue

So while this project was really cool and fun to build, I didn’t really have a use for it until NCSS started. We run a programming competition at the camp, which this year was Hitchhiker’s Guide to the Galaxy themed. I did the theming for the scoreboard, and had a little intro animation (the words “Don’t Panic” zooming in over a background of starts), and had the micro:bits play Journey of a Sorcerer (the theme music for HG2G) during that.

I think it went reasonably well, though the big unforeseen issue was that the piezo buzzers on most of the micro:bits weren’t as loud as the ones I had been testing with, so the effect was somewhat, uh, muted. 😛

If I was to do more on this project, there’s a few things I would improve:

The time sync algorithm doesn’t actually need the SYNC packets: it could be done as part of the PING_RESPONSE packets, which would simplify it somewhat.
Send the music data across the network: compiling and flashing all the nodes every time I want to play a different song is very tedious.
Relatedly, move the song data to flash storage: that way the music data from the network would be persisted across reboots.
Use something other than piezo buzzers. I was watching the Wintergatan marble machine (well, actually the videos for the construction of its replacement), and it’s made me really want to do something where the micro:bit triggers physical actions to make noise (maybe plucking a violin string)?
Support multiple channels of MIDI. There’s actually very little stopping the micro:bits from playing polyphonic sounds: if I had compiled the nodes with different song data (but matching segment lengths), they could already do it pretty easily.

Source code: is in this Mercurial repository. It’s open source, under the Apache 2.0 license.

Finding Good Answers Quickly

2015-10-08T00:05:00+11:00

I recently had a difficult problem to solve: assigning 94 students into teams of four. The first reason it’s difficult is that you end up with 23.5 teams, and since we can’t chop students in half, we have to go with 22 teams of four, and two teams of three.

But we’re not done yet: the competition should be fair, in the sense that teams should be roughly equally matched: we don’t want to have one really good team that easily wins. So we get some estimations of students’ abilities— let’s say they get a score of 1 (worst), 2 or 3 (best). Then we can define a team’s strength as the sum of the students’ ability scores, and we want teams strengths to be roughly equal.

But we might even still have further requirements: maybe we want students to not be in teams with students they’ve already worked with (so they meet more people), or not put students who have done the competition together (as experience isn’t quite the same thing as ability).

Now this problem isn’t difficult because it’s hard to solve, but because it’s difficult to figure out exactly what the requirements are. The way I’d like to solve this problem is by specifying a small part of the problem, having something give me a possible solution, and then iterating to further define the problem (to avoid the bad cases I see).

I’ve had to solve a few problems of this nature, and after writing almost the same Python script each time, I refactored out the common code into Hufflepuff, an open-source project I released today. Like any good UNIX tool, it’s not especially interesting or complex itself, but lets you glue together other simple tools to produce something useful.

Let’s see how it works with a cut-down version of the example above. First, create a working directory, install a Python virtualenv and install Hufflepuff from pypi, and test it works:

$ mkdir -p ~/tmp/teams; cd ~/tmp/teams
$ virtualenv .
$ . bin/activate
(teams)$ pip install hufflepuff
(teams)$ hufflepuff -h
usage: hufflepuff [-h] [-c CONFIG_FILE] [--initial-binary INITIAL_BINARY]
                  [--initial-file INITIAL_FILE]
                  [--mutate-binary MUTATE_BINARY]
                  [--score-binary SCORE_BINARY] [--beam-width BEAM_WIDTH]
                  [--expand-mantissa EXPAND_MANTISSA]
                  [--expand-multiplier EXPAND_MULTIPLIER] [--target TARGET]
                  [--num-iterations NUM_ITERATIONS]
                  [--parallelism PARALLELISM]
...

We’ll also need some test data to play with, and for simplicity, we’ll just have a few students:

(teams)$ python
>>> import json, random
>>> with open('students.json', 'w') as f:
...   json.dump([
...     {
...       'name': name,
...       'skill': random.randint(1, 3),
...       'experience': random.random() < 0.3,
...     }
...     for name in 'ABCDEFGHIJKLMN'
...   ], f)
...
>>> [Ctrl-D]

You should now have a students.json file with some students, each with a unique name, skill level, and experience indicating whether they’ve done the competition before or not.

Now we need to provide three things to Hufflepuff: an initial assignment of students to teams, a program to mutate those teams, and a program to score a team assignment.

For the initial assignment, we need to specify the team sizes, and provide a valid assignment: so let’s just put the students in order. Here’s initial.py, which goes into your working directory:

import json

# For 14 students, two teams of four, and two of three.
team_sizes = [4] * 2 + [3] * 2

with open('students.json') as f:
    students = json.load(f)
assert sum(team_sizes) == len(students)

print json.dumps([
    [students.pop(0)['name'] for _ in range(team_size)]
    for team_size in team_sizes
])
assert len(students) == 0

Now when you run it, you should get the following output. It should be fairly obvious: the first team has four people (A, B, C, and D).

(teams)$ python initial.py
[["A", "B", "C", "D"], ["E", "F", "G", "H"], ["I", "J", "K"], ["L", "M", "N"]]

The next thing to provide to Hufflepuff is a way of mutating the state. In this example, the easiest thing to do is to swap a pair of students in different teams. We need to be carefuly to only produce valid states (every student should be in exactly one team), and ideally we shouldn’t duplicate states— so since teams are a set of students, let’s keep them in sorted order within the team.

Here’s mutate.py, also to be placed in the working directory:

import json
import random
import sys

# Read lines from stdin (ending cleanly on EOF):
for line in iter(sys.stdin.readline, ''):
    teams = json.loads(line)

    # Swap between 1-3 pairs.
    for _ in range(random.randint(1, 3)):
        # Select two teams to swap between.
        team_a, team_b = random.sample(teams, 2)

        # Choose a person from each team.
        person_a = random.choice(team_a)
        person_b = random.choice(team_b)

        # Swap them.
        team_a[:] = sorted(
            person_b if name == person_a else name
            for name in team_a
        )
        team_b[:] = sorted(
            person_a if name == person_b else name
            for name in team_b
        )

    print json.dumps(teams)

You should be able to run it as below. The first JSON list is input, the second is output, and it should continue accepting input until you send EOF (by pressing Ctrl-D).

(teams)$ python mutate.py
[["A", "B", "C", "D"], ["E", "F", "G", "H"], ["I", "J", "K"], ["L", "M", "N"]]
[["B", "C", "J", "M"], ["E", "F", "G", "L"], ["D", "I", "K"], ["A", "H", "N"]]

You’ll have noticed that it swaps a few pairs of people around. The initial state will be pretty bad, so we want to make a lot of swaps, to quickly head towards the better state. Towards the end of the search, lots of swaps will usually be bad, and we want single swaps to settle on the local optimum. There are more sophisticated things you could do, but I’ve found this is easy and works well enough in practice.

The last thing we need to provide is some way of scoring the state— nicely congregating all your experimental decisions of what to optimise into a simple problem: giving a score for a team assignment.

In this case, it’s easiest to specify the score as a “cost”, and say how bad an assignment is. We’ll inflict penalties for teams with more than one experienced student, and for the difference between a team’s strength and the average team strength. Here’s score.py, also to be placed in the working directory:

import json
import sys

with open('students.json') as f:
    students_by_name = {
        student['name']: student
        for student in json.load(f)
    }

def experience_penalty(teams, penalty=100):
    score = 0
    for team in teams:
        n_experienced = sum(1 for student in team if student['experience'])
        if n_experienced > 1:
            score += penalty
    return score

def skill_penalty(teams):
    strengths = [
        sum(student['skill'] for student in team)
        for team in teams
    ]

    average_strength = sum(strengths) / float(len(strengths))

    return sum(
        abs(average_strength - strength)
        for strength in strengths
    )

for line in iter(sys.stdin.readline, ''):
    teams = json.loads(line)

    # Replace students' names with their full info.
    teams = [
        [students_by_name[name] for name in team]
        for team in teams
    ]

    score = experience_penalty(teams) + skill_penalty(teams)
    print score

If you run it, and provide it the initial team JSON, it should print out some score (which will vary based on your students.json file):

(teams)$ python score.py
[["A", "B", "C", "D"], ["E", "F", "G", "H"], ["I", "J", "K"], ["L", "M", "N"]]
104.0

A side note: it’s handy to put a “debug mode” into the scoring program, so you can give it a solution, and quickly see how well it does on each of the conditions. This can be useful for distinguishing between multiple equally-scoring solutions, and also for telling if the search found the optimum.

Finally, we’ll write a search.cfg file to specify what we want Hufflepuff to do (we could also just pass the parameters as command line args, but this is nicer):

initial-binary = python initial.py
mutate-binary = python mutate.py
score-binary = python score.py
target = min

Then, all we need to do is to run Hufflepuff, which will take all those things, and run a beam search. As it runs, it prints out the scores of the items in it’s current search beam to stderr, and finally prints out [score, state] JSON-encoded lists to stdout of the best states it ever saw.

(teams)$ hufflepuff -c search.cfg
...
[2.0, [["A", "C", "D", "K"], ["B", "G", "J", "L"], ["F", "H", "I"], ["E", "M", "N"]]]
[2.0, [["A", "C", "D", "K"], ["B", "G", "J", "L"], ["F", "H", "N"], ["E", "I", "M"]]]
[2.0, [["A", "C", "E", "G"], ["D", "H", "J", "M"], ["B", "I", "L"], ["F", "K", "N"]]]
[2.0, [["A", "C", "G", "L"], ["I", "J", "K", "M"], ["D", "F", "N"], ["B", "E", "H"]]]
[2.0, [["A", "C", "L", "N"], ["I", "J", "K", "M"], ["D", "F", "G"], ["B", "E", "H"]]]

As you can see, it found a bunch of possible solutions, all of which had score 2.0. I can then manually inspect those states, determine if there’s any major problems with them, and refine the scoring program if necessary.

A cute little trick is that you can take those states (without the score), put them into best.json, and then run hufflepuff -c search.cfg --initial-binary='cat $initial-file' --initial-file=best.json to start the search from those states (and you can make it less ugly by defining initial-binary in search.cfg since that’s the default).

Pull requests, bugs and feature requests are welcome, and I’m also happy to answer questions in the comments below. And if case you were wondering, here’s the inspiration for the name.

Solving Problems Badly

2015-10-03T14:02:00+10:00

Perfect is the enemy of good.

I have an interesting story to tell today, and I’m sure the title of this post probably surprised you. Well, it’s not satire or schadenfreude, but actually serious advice. This is the story of some code I wrote (back in 2013) to solve a problem, where I deliberately chose a “bad” solution.

One of the things I’ve done for many years now is tutor high school students to learn programming as part of the NCSS Challenge. Each week, for five weeks, students are given a set of problems to solve, accompanied by notes explaining all the Python features and concepts they need to solve them. Many high school computing teachers sign up their class (sometimes using it as an assessment task), but many students also sign up themselves. In addition to the notes, students can also chat and discuss the problems (but not share code) on forums, or seek help privately from our volunteer tutors.

The challenge uses Discourse for the forums, which works well, but also for the private messaging (with some custom modifications), which works less well. There are two main problems, which the following sequence illustrates:

Student 1 sends a message (sending a notification to all tutors).
Tutor A starts writing a response.
Tutor B starts writing a response
Tutor A posts their response.
Tutor B either doesn’t notice (meaning the student gets two responses, which might attempt to solve the problem in different ways), or does notice and is annoyed that their time has been wasted.
Tutor A goes back to the notifications, and starts clicking through the unread posts, but they’ve all been answered by other tutors.

To solve the problem of multiple tutors answering the same post, we set up an IRC channel that tutors joined, so they could announce which threads they were going to work on. Various tutors wrote IRC bots to automate bits of this, and then I combined all the functionality into a single bot for keeping track of what state (unreplied/replied) threads were in, and which tutor was currently assigned to answer it.

And now you have enough background to get onto what my bad solution was. One of the parts of this IRC bot was the “threads backend”, which stores thread metadata: author, title, unreplied/replied state, and the tutor assigned to it. Now the obvious answer to this is to use a database: SQLite if it’s a toy problem, and PostgreSQL if it’s big enough.

Instead of doing that, however, I went with something far more crude:

import os

class Thread(object):
    def __init__(self, title=None, ...):
        self.title = title
        ...

    def to_json(self):
        return {
            'title': self.title,
            ...
        }

def load_threads():
    with open('threads.json', 'r') as f:
        threads = json.load(f)
    return {
        key: Thread(**kwargs)
        for key, kwargs in threads.iteritems()
    }

def save_threads(threads):
    with open('threads.json.new', 'w') as f:
        # If this crashes, we end up with an empty threads.json.new file,
        # but threads.json is still intact.
        json.dump({
            key: thread.to_json()
            for key, thread in threads.iteritems()
        }, f)
    os.rename('threads.json.new', 'threads.json')

...

def request_handler(request):
    threads = load_threads()

    if request.action == 'get_post':
        return get_post(threads, request)

    elif request.action == 'new_post':
        response = new_post(threads, request)
        save_threads(threads)
        return response
    ...

It’s terrible! On every request, the entire database is read from disk, and whenever a thread gets modified, the entire database has to be written back to disk (not just the parts which changed). There’s some other problems too: it doesn’t let you handle requests concurrently, and it doesn’t let you create secondary indices to speed up queries. It’s also missing an os.fsync() call to ensure the data is actually written to disk, but let’s not talk about that one. 😉

The secondary indices isn’t just a throwaway comment either: one of the operations this backend supports is filtering threads by various attributes. Obviously, the only way this backend can support that is by doing a linear scan over all threads, and returning the ones which match.

But actually, despite the number of raised eyebrows, it’s not actually quite as terrible as it first seems. Firstly, it’s a backend to an IRC bot, which means that any response it gives is going to be seen on an IRC channel, which is (sort of) a serialised output: so not handling concurrent requests isn’t a big deal. It also doesn’t have to handle a lot of traffic (much less than one request per second), and even a 1s latency wouldn’t be obviously bad on IRC.

The other thing to consider is that the database isn’t actually that big! In the first year, there were ~500 threads: even with a generous 1KiB of metadata per thread, that’s still less than a mebibyte, which is easily written to disk in less than a second. It’s also small enough to fit in the OS buffer cache, so reading the database from “disk” isn’t expensive either. And you can also see that those linear scans aren’t that expensive either.

Futhermore, there’s some advantages to the code. It’s simple: there’s full visibility into the internal state, serialisation and deserialisation processes (unlike many ORMs, e.g. SQLAlchemy). It’s debuggable: it’s just JSON, so it’s incredibly easy to manually inspect (after pretty-printing with python -m json.tool) or write custom tools to analyse it. Schema migrations are easy: tell Thread.__init__ about the new key (providing a default value), and have Thread.to_json emit whatever schema you want stored: on the next database write, all record will be rewritten to the new schema.

This system performed adequately for the first year it was used. There were many schema changes, but it had no data corruption or was the cause of any outages. It’s performance was good enough: by the end, some filtering requests took just over a second to respond to, but it was never bad enough for users to notice.

The next year, I rewrote it to use SQLAlchemy backed by SQLite and added indices to optimise the most frequent filter requests. But I didn’t see that as admitting defeat: on the contrary: by starting with a terrible implementation, I learnt exactly what the bottlenecks were, and could then make informed decisions when designing the next version.

Perhaps you weren’t surprised I went with a “bad” solution first: the idea is certainly not new. It’s an interesting strategy, but you need to be careful with it: if you don’t make it bad enough, you won’t be motivated enough to rewrite it in the future, and if you make it too bad, then it’ll crash and burn before you’re ready to rewrite.

Even if this story hasn’t fully convinced you that deliberately choosing imperfect solutions is a good idea, I hope you can see it’s a viable strategy in some cases.

Introducing vipdf

2014-03-09T12:41:00+11:00

I mentioned in an earlier post (almost a year ago now, eep!) that I had some more software to release to the world, and as promised, this is the next one. The word “introducing” in the title is perhaps a bit misleading— I’ve written (but not released) several similar programs to this, and I’ve been using this particular program for a few years now, so it’s not really new.

But enough of that, let’s get on to what this thing is.

Rationale

The rationale for vipdf is that I’m unhappy with a lot of existing presentation software, for a variety of reasons, and I realised I could solve a lot of these problems with a vi-style keyboard mapping for navigating presentations.

The things which I wanted specifically were:

Moving to an arbitrary slide. If you get to the end of a presentation and then someone in your audience asks you a question about something on slide 15, you should be able to go directly to that slide— you shouldn’t have to move through all the intermediate slides.
Skipping over slides. Suppose you’re giving a talk, and you realise that you haven’t got time to talk about something in depth. No matter, your software should let you just skip over the next slide (or two), without flashing those slides at your audience.
Keyboard controls. If I have to use a mouse while giving a presentation, I’m going to accidentally click things I didn’t mean to and miss things that I meant to click on. Since I can touch-type, it makes sense to control everything by the keyboard, and by using familiar vi-style keybindings, I can reduce the cognitive overhead of remembering shortcuts.

At the moment, vipdf only supports PDFs (just in case the name didn’t make that obvious), which makes it great for pairing with beamer (as much as I dislike TeX, I haven’t found anything better yet), but you can easily export PDFs from most other presentation software (Microsoft Powerpoint, Keynote and LibreOffice Impress all have this functionality).

Yes, this will remove the gratuitous animations you had carefully wasted time setting up. No, there’s no support for embedded videos (in case you didn’t know, yes, you can do that with PDFs).

Controls

As I mentioned, there are vi-style keyboard controls:

<Left>: Move to the previous slide
N <Left>: Go back N slides (e.g. 3<Left> will move you back 3 slides)
<Right>: Move to the next slide
N <Right>: Go forward N slides (e.g. 2<Right> will skip forward over a slide)
N g: Goto slide N (e.g. 33g takes you to slide 33)
b: Toggle blackout

Yeah, I didn’t map hjkl. If you feel strongly about that, you should let me know in the comments what the mapping should be (should both h and j move back, or only h?)

There’s also controls for auto-advancing slides (t is for timer, btw):

t: Toggle auto-advance on or off.
N t: Start auto-advance (if not already started), and set the time interval to advance every N seconds. The default time interval is 5 seconds.
N [: Set the start slide to slide N, or unset if N is 0.
N ]: Set the end slide to slide N, or unset if N is 0. If auto-advance reaches the end slide, it will loop back to the start slide if one is set, otherwise it will turn off that auto-advance.

API

There’s also a DBUS api available for use, and there’s also a sample control script provided so you can write your own scripts to control your presentation:

$ vipdf-control list
# will give you a list of viewer ids, though at the moment there's just one
# you should take that and pass it into the following commands

$ vipdf-control next $viewerid
# advances by 1 slide

$ vipdf-control next $viewerid 2
# advances by 2 slides

$ vipdf-control prev $viewerid
# goes back 1 slide

$ vipdf-control goto $viewerid 10
# goto slide 10

Unfortunately, the API doesn’t make a lot of sense at the moment, as you can’t have more than one presentation open in a vipdf instance at a time.

Implementation

vipdf is written in Vala, using Poppler and Clutter. You can find the code here.

Quick build instructions for Debian:

$ sudo aptitude install valac libgee-dev libpoppler-glib-dev libclutter-1.0-dev
$ make
...
$ bin/vipdf /path/to/file.pdf

I’ve also written a web interface for it (which also integrates with pjlink).

Python: Wat

2014-03-08T19:18:00+11:00

You’ve all seen wat, right? Of course you have.

Of course, Python doesn’t have any Wats, right? Well, not so much.

Let’s talk about Python!

>>> 'toys"r"us'
'toys"r"us'

Makes sense. So if we change those outer quotes to double quotes, we should get a syntax error, right?

>>> "toys"r"us"
'toysus'

Wat.

Let’s talk about Python!

>>> 42**True
42
>>> 6 * False
0

This is because before True and False were available as builtins, people would write True = 1 and False = 0 at the top of their Python files. When they were introduced, they made bool a subclass of int, True == 1 and False == 0.

So of course, you can do arithmetic with booleans.

Wat.

Let’s talk about Python!

>>> data = {
...     1: 'one',
...     '1': 'won',
...     True: 'true',
... }

Right, it’s a dictionary with three items: the keys have different types, but the basic idea is that we map objects to some string representation of them. Right?

>>> data
{1: 'true', '1': 'won'}

There are three behaviours of dictionaries which come into play here. The first is that keys in dictionaries compare equal if they have the same hash value (__hash__), and compare equal (__eq__). The second is that dictionary literals are evaluated by starting with an empty dictionary, and inserting each item in order (left to right). The third is that if you are inserting an item into a dictionary and the key is already there, the value will be replaced, but not the key.

Which means that in the example above, this is what happens:

>>> data = {}
>>> data[1] = 'one'
# {1: 'one'}
>>> data['1'] = 'won'
# {1: 'one', '1': 'won'}
>>> data[True] = 'true'
# {1: 'true', '1': 'won'}

Wait, why did the last insertion replace the value for 1? Well because hash(True) == hash(1), and True == 1, of course.

Wat.

Enough making fun of languages that suck, let’s talk about JSON!

>>> import json
>>> data = {'1': 'won', 1: 'true'}
>>> json.dumps(data)
'{"1": "won", "1": "true"}'

This is valid JSON.

Wat.

Let’s talk about Python.

>>> 1 > 2 is False
False

Uh, wait, but 1 > 2 isn’t true, so it should be False, right? Like this:

>>> (1 > 2) is False
True

Hmmm, that works, maybe it’s just a precedence thing:

>>> 1 > (2 is False)
True

Nope (and note the re-occurence of False == 0 in this). This is actually a case of Python’s chained comparisons:

>>> 1 > 2 and 2 is False
False

Chained comparisons are actually awesome…

…but if you ever use it with disparate operators like this, Wat.

And let’s finish up with some Python!

This was fixed in Python 3, but there’s still plenty of Python 2 programmers just waiting to bump into this one.

>>> class GiantSpiders:
...    count = 0
...    legs = [
...        'leg %d' % count
...        for count in range(8)
...    ]
...
>>> GiantSpiders.count
7

Arrrghh! Help, spiders!

Wat.

Also, the one which didn’t qualify

Time objects representing midnight are falsey. Except, it's not quite as simple as that. Unfortunately, this only qualifies for an honourable mention: there’s an open bug for this issue, so it might be fixed.

Introducing gst-launch-dynamic

2013-04-05T23:15:00+11:00

I thought it would probably be a good idea to write posts about the random projects I do. As it happens, this particular post is about a tool which still could do with more work, but is in a reasonable enough state to present to the world. I’ve got another tool which I’ll write a post about after doing a little clean up.

So, on to the entertainment. GStreamer is a multimedia framework, which lets you build applications which do all kinds of strange and wonderful things with multimedia (audio / video / anything you can stream).

I like to think of it as UNIX pipelines for multimedia: a GStreamer pipeline is built out of many elements (processes), and each element has a number of pads (stdin / stdout / stderr / other fds) which can be joined to other elements. Indeed, there’s even a nice tool called gst-launch which lets you express the pipeline in a UNIX-like syntax— in fact, it’s better, since it lets you connect any element to any other element, which is not possible (as far as I can tell) in the POSIX shell language.

What is it?

Great, so that’s what GStreamer is, what does my tool do? Well, it’s almost the same as gst-launch, except for one little thing: it lets you modify the pipeline at runtime. I think this is both amazing and slightly insane: it greatly lowers the barrier for ad-hoc experimentation with pipelines, and allows for easy scripting of pipelines.

How does it work?

It’s very simple. To run it, call it in the same way you would normally call gst-launch:

$ gst-launch-dynamic videotestsrc ! autovideosink

It will construct the pipeline, and start playing it, in this case showing the SMPTE colour bars. But, to let you change the pipeline, it also watches standard input for commands, so we can type this:

videotestsrc0.pattern = 1

This sets the pattern property on the videotestsrc0 element (since we didn’t give it a name, it gets automatically numbered) to 1, which corresponds to "snow" (i.e., black & white noise). As soon as you press enter, the property is set, and the video displayed on your screen will change.

It’s probably worth noting a bug (well, more of an inconsistency) here: the thing on the right hand side of the equals sign is interpreted as a Python object (using ast.literal_eval), which isn’t exactly the same syntax that the command-line arguments (and gst-launch) uses.

Other commands

Aside from changing properties, you can control the state of the pipeline with these commands:

play
pause
stop

You can rewire the pipeline by adding, removing, linking and unlinking elements. This is a little unstable at the moment, since it’s just translating commands into GStreamer API calls without any sanity checking. It is possible to replace an element with another at runtime, but you need to be careful about unlinking and relinking the pads.

Link an element to another: element.sink -> element.src (in this case, both pad names are optional)

Unlink an element from another: element.sink x> element.src

Add an element: + type key=value …

Remove an element: - element

Scripting

One immediate possibility is to write a program which sends these commands. A side note: during development, FIFO files are great for keeping the pipeline open while you modify your script. For example:

$ mkfifo commands.fifo
$ gst-launch-dynamic … < commands.fifo
$ ./my-random-script >> commands.fifo

As an example of how this can be completely useless, but very entertaining, here’s how to make any video psychedelic.

psychedelic.py

import time

STEP = 0.01
DELAY = 0.01

def drange(start, stop, step):
    r = start
    while r < stop:
        yield r
        r += step

while True:
    for value in drange(-1.0, 1.0, STEP):
        print 'vb.hue =', value
        time.sleep(DELAY)

Usage:

$ python -u psychedelic.py | \
  gst-launch-dynamic filesrc location=gitannex.ogv ! \
    decodebin2 ! videobalance name=vb ! autovideosink

Finale

After I wrote this tool, I realised that nothing I had built so far really utilised its potential, so I really hope that someone will find this and build something cool with it.

Of course, you can’t do that if I don’t point you at the code!

Blackmagic DeckLink, ConsoleKit and TTYs

2013-02-19T07:11:00+11:00

Regular readers may skip this blog post, it’s being posted so people find the solution when the google it, not because it’s interesting.

I had a computer with a Blackmagic capture card, and discovered that it wouldn’t shut down properly. After tracing my way through PolicyKit, I discovered that it thought that my login session wasn’t local, but was a remote session.

Looking at the output of ck-list-sessions, ConsoleKit didn’t have a x11-display-device entry for my session (it’s normally something like /dev/tty7). Confusingly though, it did have an x11-display (which was :0 as expected).

ConsoleKit determines the x11-display-device using a helper program, which on my system is located at /usr/lib/ConsoleKit/ck-get-x11-display-device.

This helper program parses /proc/tty/drivers, into a struct in src/ck-sysdeps-linux.c:92. I discovered the name in /proc/tty/drivers was more than 16 characters, and hence overflowed the name array, causing the program to segfault.

Since ConsoleKit’s project page indicates that there’s no longer any active development on it, I decided on the hacky solution to solve my problem, rather than fixing the actual problem and bumped the size of the name array up by a couple of bytes.

Index: consolekit-0.4.5/src/ck-sysdeps-linux.c
===================================================================
--- consolekit-0.4.5.orig/src/ck-sysdeps-linux.c    2012-07-14 14:08:02.298395776 +1000
+++ consolekit-0.4.5/src/ck-sysdeps-linux.c 2012-07-14 14:11:07.282389984 +1000
@@ -93,7 +93,7 @@
         guint major_number;
         guint minor_first;
         guint minor_last;
-        char name[16];
+        char name[24];
         char devfs_type;
 } tty_map_node;

Shell startup scripts

2013-02-17T05:50:00+11:00

Update (2020-03-29)

When I originally wrote this blog post, my intent was to describe how things worked on a typical system at the time. Unfortunately, the operation of bash startup scripts is dependent on patches added by OS distributions, compile-time options and if bash thinks it was invoked by sshd or rshd. I have no desire to add complexity to this post, so instead please read this as a rough approximation of the truth. If you want to know for certain how it works on your system, the only answer is test it yourself.

If you’re a regular shell user, you’ve almost certainly got a .bash_profile or .bashrc script in your home folder, which usually contains various tweaks, such as setting environment variables (adding that directory to $PATH), telling your shell to do clever things (like set -o noclobber) and adding various aliases to commands (like alias please=sudo).

(If you’re really organised, you’ll have all your dotfiles in a repository somewhere so that you can keep your settings synchronised across all the machines you work on.)

Anyhow, I suspect that few people know when things like .bash_profile and .bashrc actually get executed. When I started, I just followed people’s advice of putting stuff in .bashrc, and then when it didn’t work, into .bash_profile. I could stop here and describe just the bash startup process (as silly as it is), but there’s a complication in that I switched to zsh a few years ago (and haven’t looked back), but occasionally use bash on machines which don’t have zsh installed.

In order to handle this nicely then, I need to be able to specify things which are specific to bash or zsh in their own files, and then to specify things which any POSIX-compliant shell (like aliases and environment variables) can understand in a common startup file.

My solution to this problem is to define some new dotfile folders, one for each shell (.bash/, .zsh/ and .sh/), and one for the shell-independent files (.shell/):

.bash/
    env
    interactive
    login
    logout
.sh/
    env
    interactive
    login
.shell/
    env
    interactive
    login
    logout
.zsh/
    env
    interactive
    login
    logout

“But!”, you say, [1] “what do the different files in here do?” Ah, well I’m glad you asked. There are two kinds of shells:

[non-]interactive shells (you type into them / shell scripts)
[non-]login shells (the shell run when you first login / subshells)

All shells will first run env, then login shells will run login, then interactive shells will run interactive. Once finished, login shells will run logout.

Where to put stuff

It all depends on when it needs to be run.

If it’s setting / modifying environment variables, it should go in login. If it’s alias or a terminal-specific environment variable (e.g., GREP_COLOR), it should go in interactive. In my .shell/env file, I have my umask set, and also define some useful functions for modifying colon-separated path environment variables (like $PATH).

Even if you don’t adopt anything else from my scheme, I’d recommend you take a look at the what my functions are doing which differs from something like export PATH=$PATH:/path/to/dir.

That particular pattern is way too common, and is very dangerous if you consider the case when $PATH (or whatever your variable is, like $LD_LIBRARY_PATH) isn’t set. Then, the value will be :/path/to/dir, which usually means both /path/to/dir and the current directory, which is usually both unexpected behaviour and a security concern.

With my implementation (see .shell/env_functions), you can append, prepend and remove directories from any colon-separated environment variable, and when appending or prepending, you are guaranteed that directory will only then appear in that variable once.

As a side note, I’m very disappointed in my indirect_expand() function, so if you have a better solution, please let me know (or send me a pull request).

Implementation

In order to implement this, you first need an understanding of which startup files are run in each case. Of course, this isn’t hard, since all the shells have the same, sensible system. Ahahaha, no.

Fortunately, I’ve read the man pages for you, and drawn a pretty diagram. To read it, pick your shell, whether it's a login shell, whether it's interactive, and follow the same colour through the diagram. When the arrows split out to multiple files, it means that the shell will try to read each one in turn (working left to right), and will use the first one it can read.

An important point here is that if you have a non-login, non-interactive shell running a plain POSIX shell (dash, bash in sh compatibility mode), it won’t fulfil our contract in that it won’t read .sh/env (and hence .shell/env).

The other special case which exists is Bash’s remote shell mode (which is rather warped), where it tries to detect if it’s running under ssh or rsh (I assume by looking at the process name of its parent), and if so, follows a different path, which is indicated on the diagram.

Except that diagram shows what happens according to the man page, and not what happens when you actually try it out in real life. This second diagram more accurately captures the insanity of bash:

See how remote interactive login shells read /etc/bash.bashrc, but normal interactive login shells don’t? Sigh.

Finally, here’s a repository containing my implementation and the graphviz files for the above diagram. If your POSIX-compliant shell isn’t listed here, or if I’ve made a horrible mistake (or just a tiny one), please send me a pull request or make a comment below, and I’ll update this post accordingly.

[1]	and since I’m writing this, I can make you say whatever I want for the purposes of narrative.

Controlling Projectors with PJLink

2012-11-30T22:41:00+11:00

I regularly need to control a data projector, and previously the only way to do this has been using a web interface created by the manufacturer. Of course the interface is, while usable, poorly designed, and somewhat sluggish.

Fortunately, there’s a protocol for controlling projectors, PJLink, but unfortunately, there aren’t many implementations (I did find a perl module in my search), so I decided to write my own implementation in Python.

So I did, and it took me just a single morning, which I was quite impressed with. The specification is reasonable simple, and aside from one or two stupid mistakes, implementing it was straight forward. In addition to the Python API, I also wrote a simple command line utility for communicating with the projector, which is probably more useful than the API (since you can throw it into your own shell scripts).

If you want to try it out, you can install it from PyPI using pip.

$ pip install --user pjlink
# or if you're in a virtualenv
$ pip install pjlink

The location of projectors can be specified in ~/.config/pjlink/pjlink.conf (on Linux, see appdirs for your platform):

[default]
host = myprojector
port = 4212
password = JBMIAProjectorLink

[other]
host = bob

Note this stores your passwords in plain text: this is an indication of what I think of the authentication protocol which PJLink uses. I couldn’t find a specific vulnerability, but I still wouldn’t trust it.

Then, you can use -p to reference the projector in the config file. You can also just specify a host and port manually (but not a password).

# get power state of myprojector:4212 (using password)
$ pjlink -p default power
# get power state of bob:4352 (default port, no password)
$ pjlink -p other power

# for the "default" projector, you can omit -p
# this is the same as the first command
$ pjlink power

# get available inputs
$ pjlink inputs

# get current input
$ pjlink input

# switch to input RGB-1
$ pjlink input RGB 1

And there are many more commands available: everything in the specification is implemented as plainly as possible, so if the projector can do it, my tool supports it.

And for the record, I disclaim any responsibility for inappropriate use of this tool. (because I can think of a few!)

Awesome + GNOME Configuration

2012-11-28T10:31:00+11:00

I was recently prompted to publish my configuration for integrating GNOME and awesome. At the time I wrote this setup, the awesome wiki was recommending a configuration which didn’t correctly autostart applications, nor integrate well with anything expecting to find a session manager. However, it now contains instructions for a similar setup, so you should have a look at that too.

The first thing you need is an entry to tell your display manager about a new session type, so put this in /usr/share/xsessions/awesome-gnome.desktop:

[Desktop Entry]
Name=awesome-gnome
Comment=This session logs you into GNOME using the Awesome WM
Exec=gnome-session --session=awesome-gnome
TryExec=gnome-session
Icon=
Type=Application

Then you need to tell gnome-session about the "awesome-gnome" session. You might want to modify the notification daemon here. Put this in ~/.config/gnome-session/sessions/awesome-gnome.session:

[GNOME Session]
Name=awesome-gnome
RequiredComponents=gnome-settings-daemon;
RequiredProviders=windowmanager;
DefaultProvider-windowmanager=awesome
DefaultProvider-notifications=notification-daemon
FallbackSession=gnome-fallback
DesktopName=GNOME

Then, GNOME expects to find a XDG application entry for "awesome", so put this in ~/.local/share/applications/awesome.desktop

[Desktop Entry]
Type=Application
Name=Awesome
Comment=Window manager
Exec=sh -c '/usr/bin/awesome &'
NoDisplay=true
X-GNOME-WMName=awesome
X-GNOME-Autostart-Phase=WindowManager
X-GNOME-Provides=windowmanager
X-GNOME-Autostart-Notify=true
X-GNOME-AutoRestart=false

Then you should be able to logout, and immediately see the awesome-gnome session which you can log into. Your display manager usually remembers the last session you logged into, but using this method, you can easily switch between GNOME shell, GNOME fallback and awesome-gnome.

Prettify script for PC^2

2012-11-17T17:23:00+11:00

When I was doing programming competitions at uni, I wrote this script, which takes the default HTML output from PC^2 and add some colour to it to indicate passed / failed problems. Since it was written / enhanced on the day of the competition, there’s definitely some hacky code in there, but I’m publishing it so future generations can steal it for their own nefarious purposes.

This code is released into the public domain.

# prettify.py

import re
import string
import sys

from lxml import html, etree

doc = etree.parse(open('prettify.xsl', 'rU'))
xslt = etree.XSLT(doc)

doc = html.parse(sys.stdin)

passed = re.compile('^[0-9]*/[1-9][0-9]*$')
failed = re.compile('^.*[1-9]/[-0]*$')

for node in doc.xpath('//td'):
    text = node.text or ''
    if passed.match(text):
        node.attrib['class'] = 'passed'
    elif failed.match(text):
        node.attrib['class'] = 'failed'

problems = dict(zip(string.uppercase, [
    'An Industrial Spy',
    'Simple Polygon',
    'High Score',
    'Rankings',
    'Wormholes',
    'Fractal',
    'Rewards for Math!',
    'Divisible Subsequences',
]))

for node in doc.xpath('//th/strong/u'):
    node.text = problems.get(node.text, node.text)

doc = xslt(doc)

#body = doc.xpath('//html/body')
#d = etree.Element('div')
#d.attrib['class'] = 'timer'
#
#import sys
#from datetime import datetime
#n = datetime.now()
#t = datetime(2011, 9, 3, 17, 0, 0, 0)
#print>>sys.stderr, n - t

print html.tostring(doc)

<!-- prettify.xsl -->

<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

    <xsl:variable name="stage" select="0" />

    <xsl:output method="xml" indent="yes"
        doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"
        doctype-public="-//W3C//DTD XHTML 1.0 Transitional//EN" />

    <!-- Document structure -->
    <xsl:template match="/">
        <html>
            <xsl:apply-templates select="/html/head" />
            <xsl:apply-templates select="/html/body" />
        </html>
    </xsl:template>

    <!-- Head for styling, etc. -->
    <xsl:template match="/html/head">
        <head>
            <xsl:copy-of select="/html/head/node()" />
            <script type="text/javascript" src="http://cornify.com/js/cornify.js"></script>
            <!--<meta http-equiv="refresh" content="5" />-->
            <link href="http://fonts.googleapis.com/css?family=Questrial"
                rel="stylesheet" type="text/css" />
            <style type="text/css">
                html, body { margin: 0; padding: 0; }
                a { color: #729fcf; }
                body {
                    font-family: 'Questrial', sans-serif;
                    color: #eeeeec;
                    background: #2e3436;
                }
                table { font-size: 20pt; }
                .passed, .failed { font-weight: bold; }
                .passed {
                    background: #8ae234;
                    color: black;
                }
                .failed {
                    background: #ef2929;
                    color: white;
                }

                h1 { text-align: center; margin: 0; }

                thead th { font-size: 12pt; }
                th.team-name {
                    font-weight: normal;
                    text-align: left;
                }

                th:nth-child(2) { width: 50px; }
                td:nth-child(4) { width: 3.5em; }
                td:nth-child(12) { width: 3.5em; }
            </style>
        </head>
    </xsl:template>

    <!-- Body -->
    <xsl:template match="/html/body/table[1]">
        <table id="scoreboard">
            <thead>
                <tr>
                <xsl:for-each select="tr[1]/th">
                    <th scope="col"><xsl:value-of select="strong/u" /></th>
                </xsl:for-each>
                </tr>
            </thead>
            <tbody>
                <xsl:for-each select="tr[position() > 2]">
                    <tr>
                        <!-- Rank -->
                        <td class="rank"><xsl:value-of select="td[1]" /></td>
                        <!-- Name -->
                        <th class="team-name" scope="row"><xsl:value-of select="td[2]" /></th>
                        <xsl:copy-of select="td[position() > 2]" />
                    </tr>
                </xsl:for-each>
            </tbody>
        </table>
    </xsl:template>

    <xsl:template match="/html/body">
        <xsl:if test="$stage >= 1">
            <h1>Competition is over! Congratulations to all!!!</h1>
        </xsl:if>
        <xsl:apply-templates select="table[1]" />
        <p>Prettify script by Peter Ward</p>
        <xsl:if test="$stage >= 2">
        <script type="text/javascript">
            <![CDATA[
            var count = 0;
            var delay = 10000;
            function magical() {
                cornify_add();
                count++;
                if (count > 60) {
                    var child = null;
                    var nodes = document.body.children;
                    for (var i = 0; i < nodes.length; i++) {
                        if (nodes[i].tagName == "DIV") {
                            child = nodes[i];
                            break;
                        }
                    }
                    document.body.removeChild(child);
                }
                window.setTimeout(magical, delay * (Math.random() / 2.0 + 0.75));
                if (delay > 10)
                    delay *= 0.9;
            }
            window.onload = magical;
            ]]>
        </script>
        </xsl:if>
    </xsl:template>
</xsl:stylesheet>

End of Honours

2012-11-15T09:57:00+11:00

Well, I’ve handed in my thesis (Providing Useful Feedback to Beginner Programmers), and done a presentation for it (twice, for logistical reasons), so I’ve now finished everything I need to do for uni! I’m very happy about this, and I’ve been enjoying the little bubble of happiness about its completion the past few days.

We also continued the tradition of doing a prank on my supervisor’s office, by turning it into a fish tank / sea themed room. Photos from the setup and the final office are in this photo album.

And for the few people who haven’t already caught up with this news, I start working full-time at Google on Monday.

Box specific bar numbers in Lilypond

2010-11-05T04:30:00+11:00

Just a bit of Lilypond / Scheme I wrote last night — if you want to have boxed bar numbers at certain bar numbers (which don't fit every nth bar - see the docs for that), you can use this code to do it:

\override Score.BarNumber #'break-visibility = #end-of-line-invisible
\override Score.BarNumber #'self-alignment-X = #0
\override Score.BarNumber #'font-size = #2
\override Score.BarNumber #'stencil = #(make-stencil-boxer 0.1 0.25 ly:text-interface::print)
\set Score.barNumberVisibility = #(lambda (x) (not (eq? (member x '(4 9 16 25)) #f)))

This will put boxed numbers at bars 4, 9, 16 and 25.

Mouse activity

2010-05-12T00:41:00+10:00

As seen on Planet Debian.

“IOGraph” is a small Java program that tracks and draws the status of your mouse over a period of time. Lines are movement, circles are places where the mouse has stopped… The bigger the circle, the longer it was there.

I suspect the large density of horizontal lines in the center is due to my habit of always selecting the text I'm currently reading. It's interesting to see the areas of the screen which I don't really use (yes, this only tracks mouse activity, but my eyes usually follow my mouse).

Productivity

2009-12-10T10:20:00+11:00

Any resemblance to reality is purely coincidental.