flowblok’s blog

Galactic Puzzle Hunt 2019: Reflection

2019-04-08T00:00:00+10:00

I recently organized and participated in a team for the 2019 edition of the Galactic Puzzle Hunt. This post is my reflections on that process.

Organizing a team

Teams for the puzzle hunt can be up to 10 people [1], so the first step was to attempt to find other people who are at least as insane as I am. I put out a tweet linking to a Google Form (and also messaged a few friends), asking (amongst a few other things) for a time commitment, with the intent that I’d select team members most happy to spend time on it.

As it happened, I only got seven signups (including myself), so that was my team formed. I thought that was a reasonable number, but it did have a lot of people only able to make a small time commitment, and I should have kept searching for more people. My perception was we had ~2.5 daytime solvers on average.

There were a few people who gave us some help during the hunt, so I think that means I need to promote signups more next year.

Having brought together some people, we needed a team name. A bunch of ideas we had (some of which followed the Antarctica theme for the puzzle hunt):

Production Freeze
By accepting this team name you agree to our EULA
Fizz puzz
Ice climbers
TE-901
UNTITL~1.GPH

But “How about ☃ repeated as many times as the registration form will allow?” was most liked. Confession time: I didn’t actually do this. Instead, I went for three rows of snowmen (as displayed on my laptop). So while we didn’t have the longest team name, we did have the most vertical space in the team list.

Perhaps next year we’ll figure out how to consume even more vertical space with our team name?

Team coordination

We used Slack and Google Docs to coordinate. In the Slack workspace, there was a #coordination channel which had login details, a periodically updated “best puzzles to work on” post, celebrations of solved puzzles, and occasional calls for help. Puzzle solving didn’t happen in that channel: instead, we created a channel per puzzle.

It’s hard to predict what technology will be best to solve a puzzle, but generally we always want a document to describing what we’ve tried and with partial progress (which eventually morphs into a “how to solve the puzzle” doc). All of these documents got put into a shared Google Drive folder, and each puzzle channel in Slack had a topic linking to the relevant doc for easy access.

A few of the puzzles needed more than a doc. We also used:

Photos shared in Slack (for a physical jigsaw-like puzzle)
Images shared in Slack
Google My Maps in Drive (super-useful for visualising latitude/longitude pairs and looking for patterns)
Google Spreadsheets in Drive (although a few times I started to create one, and then replaced it with a table in a Google Doc)
Google Colaboratory in Drive (omg sharable Python code with notes!)

I was generally happy with the technology choices, although I need to figure out how to use Colaboratory in a way that involves less clicking and makes better use of my vim muscle memory.

Puzzle solving

Something I keep saying about puzzle hunts is that it’s important to have lots of people on the team, even if they aren’t spending gross amounts of time solving puzzles. What’s really important is that there’s a continuous stream of “have you tried <thing>?” ideas and “that looks kinda like <other thing>” observations: otherwise, you get stuck trying to solve a puzzle and then get frustrated at not having ideas, which isn’t a great mental state to be producing ideas with.

I’m very grateful to my team members for contributing their time: I think I spent less time in that stuck state than I have in previous years, but there’s definitely still room for improvement there.

There were also a few times where I think we had trouble because we didn’t quite know what “rules” the puzzle authors were playing by, so to speak. In one case, I wrote a bunch of Python code looking for patterns in some data because of a perceived hint in the puzzle title, which turned out to be a wild goose chase. There was also at least one hint spent on something where we should have just been able to get it out by being more rigorous.

Results

Our team solved the first 12 puzzles, which put us in 239th place.

Out of 717 registered teams: top 33%
Out of 505 teams who solved at least one puzzle: top 47%

I’m pretty happy with how we went: I only wish we’d managed to finish our half-solved metapuzzle in time!

For next year’s Galactic Puzzle Hunt (or any other ones for that matter), I’ve been thinking lately about doing some training sessions (where you solve a puzzle, but there’s zero-cost hints as soon as you get stuck). If this sounds interesting, hit me up?

[1]	Although the rules also say “You may use any external sources for help, including other people, as long as they aren’t helping other teams and aren’t actively participating in the hunt.”

Python C Extension Module Corruption

2018-06-23T00:00:00+10:00

If you follow me on Twitter, you might have seen this recent thread, in which I spent a few days metaphorically banging my head against a wall trying to figure out why my Python program would occasionally segfault (with no apparent trigger).

This post is an attempt to write up the cause (and my workaround) in a clearer form, and hopefully make it Google-able for the next poor soul to encounter this (if it's not fixed before then).

Dramatis Personæ

The program in question here is a Python web application, written in Flask, and run using gunicorn (using a few sync workers). It also uses SQLAlchemy (via Flask-SQLAlchemy) to access a sqlite3 database. It’s all running on a Google Compute Engine VM running Debian Linux.

Most of those details aren’t relevant, but I mention them just to give a sense of the scope: each of them was a suspect at some stage or another during my investigation.

The Evidence

Aside from seeing 5xx responses from my web server, the only information I had was in syslog:

kernel: [1234.567890] gunicorn[1234]: segfault at ad0 ip 0000000000000ad0 sp 00007fffc0401448
                                      error 14 in python3.5[55f0889ee000+3ef000]

The gunicorn logs noted that it had started a new worker to replace the segfaulted one, but nothing more. The syslog entry is really quite odd: the at ad0 and ip 0000000000000ad0 parts indicate that the program tried to jump to (and hence execute) code at the address 0xad0, which is quite suspicious: I don't know offhand what's at that address, but it’s not usually program code.

Clearly I needed more information to debug this, so I tried to get a core dump. For reasons I’m still not entirely sure of, getting a core dump didn’t just work out-of-the-box—I believe Linux is meant to write a file named core to the process’s working directory (provided ulimit is set correctly).

Some quick Googling revealed that systemd-coredump is a thing, so I installed that, and then my program didn’t segfault for an hour, so I tweeted thinking that this was a Heisenbug. It turned out not to be the case: the segfault only happened a few times an hour, and I was just unlucky.

Anyhow, I later came back and used coredumpctl list to see when the program had segfaulted, and coredumpctl gdb to launch gdb with the core dump loaded. For all the hate systemd seems to get for reinventing the wheel, I will say that systemd-coredump seems quite good at its job.

$ coredumpctl gdb
...
Core was generated by `/usr/bin/python3'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000000ad0 in ?? ()
(gdb)

Great: there's the 0xad0 address from my syslog entry, so the crash is the same as the one I’m seeing! This might seem like a minor point, but fervently verifying the world is as you expect it to be is critically important in debugging. Let’s look at a backtrace to see what the program was doing when it segfaulted:

(gdb) bt
#0  0x0000000000000ad0 in ?? ()
#1  0x0000556dce8116df in PyCFunction_Call () at ../Objects/methodobject.c:109
#2  0x0000556dce7d16e9 in call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e1e0) at ../Python/ceval.c:4720
#3  PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#4  0x0000556dce7d193f in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffd2fc7e310, func=<optimized out>) at ../Python/ceval.c:4818
#5  call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e310) at ../Python/ceval.c:4745
#6  PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#7  0x0000556dce7d193f in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffd2fc7e440, func=<optimized out>) at ../Python/ceval.c:4818
#8  call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e440) at ../Python/ceval.c:4745
#9  PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#10 0x0000556dce7d6286 in _PyEval_EvalCodeWithName () at ../Python/ceval.c:4033
#11 0x0000556dce7d24c9 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffd2fc7e650, func=<optimized out>) at ../Python/ceval.c:4828
#12 call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e650) at ../Python/ceval.c:4745
#13 PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#14 0x0000556dce7d193f in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffd2fc7e780, func=<optimized out>) at ../Python/ceval.c:4818
#15 call_function (oparg=<optimized out>, pp_stack=0x7ffd2fc7e780) at ../Python/ceval.c:4745
#16 PyEval_EvalFrameEx () at ../Python/ceval.c:3251
#17 0x0000556dce7d70df in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=<optimized out>, argcount=<optimized out>,
    args=<optimized out>, locals=<optimized out>, globals=<optimized out>, _co=<optimized out>) at ../Python/ceval.c:4033
#18 PyEval_EvalCodeEx () at ../Python/ceval.c:4054
#19 0x0000556dce8135d3 in function_call.lto_priv () at ../Objects/funcobject.c:627
#20 0x0000556dce85a647 in PyObject_Call () at ../Objects/abstract.c:2166
#21 0x0000556dce77a94e in method_call.lto_priv () at ../Objects/classobject.c:330
#22 0x0000556dce85a647 in PyObject_Call () at ../Objects/abstract.c:2166
#23 0x0000556dce81aebf in slot_tp_iter () at ../Objects/typeobject.c:6214
#24 0x0000556dce85a4a3 in PyObject_GetIter () at ../Objects/abstract.c:2755
#25 0x0000556dce84cd56 in listextend.lto_priv.683 (
    b=<BaseQuery(_group_by=False, _params={<_anonymous_label at remote 0x7f26c3adbd48>: 'flowblok.id.au'}, _entities=[<_MapperEntity(entity_zero=<Mapper(self_and_descendants=<WeakSequence(_storage=[<weakref at remote 0x7f26c3a9bf98>]) at remote 0x7f26c3aae898>, inherits=None, version_id_generator=<function at remote 0x7f26c3e837b8>, confirm_deleted_rows=True, _readonly_props=set(), local_table=<Table(name=<quoted_name at remote 0x7f26c3e8c1c8>, description=<...>, _cloned_set={<...>}, constraints={<ForeignKeyConstraint(name='domain_admin_group_constraint', initially=None, dialect_kwargs=<_DialectArgView(obj=<...>) at remote 0x7f26c82b5160>, parent=<...>, _referred_table=<Table(name=<quoted_name at remote 0x7f26c3e8cbc8>, description=<...>, _cloned_set={<...>}, constraints={<ForeignKeyConstraint(name=None, initially=None, parent=<...>, _referred_table=<...>, columns=<ColumnCollection at remote 0x7f26c3e22cc0>, ondelete=None, elements=[<ForeignKey(name=None, column=<Column(_cloned_set={<...>}, constraints=set(), forei...(truncated),
    self=0x7f26c3433608) at ../Objects/listobject.c:831
...
(truncated for your sanity)

This isn’t terribly useful: it’s full of Python interpreter stuff. Knowing the title of the blog, you might notice that the calling function is PyCFunction_Call, which you might start to get ideas from: but that could be from anything. Fortunately, gdb has amazing Python integration, so I can do this instead:

(gdb) py-bt
Traceback (most recent call first):
  <error reading variable: Cannot access memory at address 0xe90>
  File "/…/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1024, in _execute_clauseelement
    distilled_params = _distill_params(multiparams, params)
  File "/…/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/…/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2958, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2935, in __iter__
    return self._execute_and_instances(context)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2864, in one_or_none
    ret = list(self)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2894, in one
    ret = self.one_or_none()
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/loading.py", line 250, in load_on_pk_identity
    return q.one()
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 962, in _get_impl
    return db_load_fn(self, primary_key_identity)
  File "/…/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 887, in get
    ident, loading.load_on_pk_identity)
  File "/…/app.py", line 47, in setup_context
...
(truncated for your sanity)

The last frame of that backtrace is my application code, trying to run a SQLAlchemy query. The rest of the backtrace is SQLAlchemy trying to execute it. But the first frame is most interesting: SQLAlchemy is calling a function _distill_params, but we don't have the source for that call.

If you look at the source for sqlalchemy/engine/base.py, you’ll see that _distill_params gets imported from sqlalchemy/engine/util.py, which has this:

try:
    from sqlalchemy.cutils import _distill_params
except ImportError:
    globals().update(py_fallback())

That is, _distill_params comes from a C extension module, unless it’s not importable, in which case there’s a pure Python version as a fallback.

C Extension Modules

At this point, I’ve figured out the first part of what’s gone wrong: something is causing Python to think that the address of the _distill_params function is 0xad0, instead of whatever it’s meant to actually be.

For the next part, we need to delve a little into how C extension modules work. Here’s part of the source for sqlalchemy/cextension/utils.c, which provides the sqlalchemy.cutils module imported above.

static PyMethodDef module_methods[] = {
    {"_distill_params", distill_params, METH_VARARGS,
     "Distill an execute() parameter structure."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

This C file gets compiled into a shared library: on my system it’s a file named cutils.cpython-35m-x86_64-linux-gnu.so. The CPython interpreter loads extension modules in Python/importdl.c, which uses the _PyImport_FindSharedFuncptr function in Python/dynload_shlib.c, which uses dlopen to load that .so file into memory and get a function pointer.

Most of those details weren’t important: the key part is that we have some C code, being loaded into the process’s memory at runtime. So, let’s attach GDB to a running process, and have at look at that module_methods array:

$ gdb -p 1234
...
(gdb) p 'sqlalchemy/cextension/utils.c'::module_methods
$1 = {{ml_name = 0x7f66abdfde90 "_distill_params", ml_meth = 0x7f66abdfdad0 <distill_params>,
       ml_flags = 1, ml_doc = 0x7f66abdfdef8 "Distill an execute() parameter structure."},
      {ml_name = 0x0, ml_meth = 0x0, ml_flags = 0, ml_doc = 0x0}}

That’s more or less what I’d expect to see: ml_name has the Python name of the function, and ml_meth is a pointer to the distill_params C function. What happens if we try the same thing in our core dump?

$ coredumpctl gdb
...
(gdb) p 'sqlalchemy/cextension/utils.c'::module_methods
$1 = {{ml_name = 0xe90 <error: Cannot access memory at address 0xe90>, ml_meth = 0xad0,
       ml_flags = 1, ml_doc = 0xef8 <error: Cannot access memory at address 0xef8>},
      {ml_name = 0x0, ml_meth = 0x0, ml_flags = 0, ml_doc = 0x0}}

Well. That’s not right.

But wait, those hex numbers look kinda similar to the correct numbers…

>>> print('{:48b}'.format(0x7f66abdfde90))
 11111110110011010101011110111111101111010010000
>>> print('{:48b}'.format(0xe90))
                                    111010010000

>>> print('{:48b}'.format(0x7f66abdfdad0))
 11111110110011010101011110111111101101011010000
>>> print('{:48b}'.format(0xad0))
                                    101011010000

Huh? Sometime between being a perfectly happy process chugging along, and it segfaulting, the module_methods struct in a C extension module gets all its numbers truncated to 12 [*] bits‽

Time and Space

Thinking that the problem was something writing to the memory to corrupt it, I spent quite some time looking through strace and ltrace outputs to try and find something touching the relevant addresses in memory, but to no avail.

My breakthrough came when I compared /proc/$PID/smaps from a running process and a almost-segfaulted process. To capture the latter, I attached GDB, and then typed in shell cat /proc/1234/smaps > 1234.smaps and hit enter, so it would get run when the program had segfaulted (but before Linux forgot the process state). I also attempted using a breakpoint, but in hindsight I don’t think that worked. Regardless, I found the section of memory in which the module_methods struct resided, and found this difference:

Before:

7f26c44e2000-7f26c44e3000 rw-p 00002000 08:10 666070 /…/cutils.cpython-35m-x86_64-linux-gnu.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac sd

After:

7f26c44e2000-7f26c44e3000 rw-p 00002000 08:10 666070 /…/cutils.cpython-35m-x86_64-linux-gnu.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB

The key difference here is that this section of memory went from having 4 kB of Private_Dirty data to having 4 kB of Private_Clean data.

Combined with the knowledge that the numbers inside that section of memory seemed to be being “truncated”, I realised what was happening: this section of memory was being reloaded from the shared library file. I’m glossing over some details here, but on disk the shared library has various pointers to code and data within that shared library, but when it gets loaded into a process’s memory, it’ll be loaded at some offset, and all the pointers within the shared library need to be adjusted in order to work.

So on disk, inside the module_methods struct, the ml_name pointer is actually 0xe90, but when it initially gets loaded into memory, it gets adjusted to 0x7f66abdfde90 so it’s pointing at the location where the string "_distill_params" got loaded into memory.

The Piping Gun

Knowing that I was looking for something which was causing the shared library file to be reloaded from disk, I remembered that I had a bug in my automation which was unnecessarily reinstalling all Python libraries using pip. I didn’t think it was a huge problem: the downloads are cached locally, and reinstalling a package with the same version should just be a no-op, right?

Turns out not so much. When pip installs a wheel, it replaces all the files using shutil.copyfile. This opens the file for writing, truncates it, and then copies the content in.

Because this is changing the same file which is also memory mapped into a running process, Linux notices that section of data has been re-written (even with exactly the same bytes), and invalidates all unwritten changes in active processes (this is what makes the memory mapping go from Private_Dirty to Private_Clean), thus undoing the offset adjustments the process made when loading the shared library.

Honestly, I’m not sure where the bug is here. Probably after loading a shared library, Python should unassociate the data section from the file it was loaded from: though I’m not sure if the C functions to do that even exist. I’d also say pip should do atomic file replacements to avoid the problem, but this is more of a bandaid than an actual fix.

My Workaround

After figuring out what was happening, I had a realisation that this problem could be mitigated in the application, with a pure Python solution.

Instead of trying to unassociate the memory mapping, I can atomically replace the file it loaded with a copy of itself, thus making the memory mapping in the process point at data which no longer exists in the filesystem.

So here is a Python module which you can import, which will periodically find C extension modules loaded into the Python process and perform shenanigans on them to make them resilient against this failure mode. This potentially costs memory, but I doubt it’ll ever be an amount you care about.

# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
'''
Module sopin implements "pinning" of shared libraries.

This is a workaround for the problem where rewriting a Python C extension module
on disk corrupts the state of any Python process which has previously imported
it: for more details, see
https://blog.flowblok.id.au/2018-06/python-c-extension-module-corruption.html

Usage:
    import sopin
    sopin.start()
'''
import os
import shutil
import sys
import tempfile
import threading
import warnings

SUFFIXES = ('.so',)

# Modules previously pinned.
_pinned = set()
# Modules which we couldn't pin (so shouldn't reattempt).
_blacklisted = set()

def pin(filename):
    if filename in _pinned or filename in _blacklisted:
        return
    try:
        # Atomically create a unique tempfile.
        # Don't delete, because we'll move it over the old file.
        with tempfile.NamedTemporaryFile(delete=False) as f:
            # Copy the content+permissions of the old file into our new file.
            shutil.copy2(filename, f.name)
            # Move the new file over the old file.
            os.rename(f.name, filename)
    except IOError:
        warnings.warn('could not pin %s' % filename)
        _blacklisted.add(filename)
    else:
        _pinned.add(filename)

def pin_all():
    for name, module in sys.modules.items():
        try:
            filename = module.__file__
        except AttributeError:
            continue
        if filename.endswith(SUFFIXES):
            pin(filename)

def start(period_seconds=10):
    threading.Timer(period_seconds, pin_all).start()

[*]	Twitter followers may note this is not the 13 or 14 that I tweeted: I was actually debugging `processors.c` at that point, so these details are slightly different.

Synchronized Music on micro:bits

2018-02-24T22:02:00+11:00

It’s been a while since I last posted. So, uh, sorry about that. I’m going to make it up to y’all with this post. It’s in two parts: the video demonstrates the project in action, and the rest of this blog post details all the technical details. I’d recommend starting with the video to see the cool demo, and if you make it to the end, read the rest of the blog post.

Introduction

For many years now, I’ve tutored at the National Computer Science School, which teaches computer science and programming to Australian high school kids. The school is divided into two parts: a web stream, which builds a website using Python, and an embedded stream, which (nowadays) uses MicroPython on the micro:bit.

Last year I tutored the web stream, and so it was only during the closing ceremony while watching videos demonstrating all the embedded students’ projects that I started wondering about the potential of the micro:bit, and whether I could build a mesh network so they could play music synchronized across a large area. So when work started getting quieter over the Christmas / New Year break, and this year’s NCSS started approaching, I decided to buy a couple of micro:bits and start hacking.

Breaking it down

I broke down the project into a number of distinct components:

network protocol: for sending structured data as micro:bit radio packets.
time sync: getting all micro:bits on the network to share a common time.
triggers: getting all micro:bits to do an action at a given time.
music: making the micro:bits play notes on a piezo buzzer.

Writing micro:bits all the time is too hard, so I’m going to refer to them from here on as nodes.

Time Sync

Time is hard. It’s a little easier in this application, because we don’t care what the actual wall-clock time is, just that all the nodes have the same time.

The first step in figuring out how to synchronize the nodes was to head off to Wikipedia and do some reading on the way I already knew computers synchronized time, Network Time Protocol (NTP). After very little reading, it seemed fairly obvious that NTP was far too complicated for this application: I didn’t need to deal with leap seconds, and polling multiple servers and doing filtering on the answers seemed like too much effort.

Fortunately it took little effort to find the Precision Time Protocol (PTP). PTP is a really nice simple protocol, though the Wikipedia page does have much more detail than is really needed. It’s designed to work on multiple network segments (hey, that sounds kinda like the network topology we have), and on each segment, there’s a master which all the other nodes synchronize to. Each node periodically pings (called Delay_Req and Delay_Resp, but I like ping better) the master to approximate their Round Trip Time (RTT), and the master periodically sends out a Sync message with its current time. Add in some maths, and hey presto, you have synchronized time.

In order to adapt the algorithm for a mesh network, there were a few changes I needed to make. PTP has a best master clock algorithm for selecting the master, but since my nodes might drop in and out of network connectivity all the time, my system needed to continuously adapt which nodes were sending timestamps to synchronize to, based on which nodes could talk to them. I decided to term those nodes thought leaders (partially because I really disliked the "master" terminology, partially because I like being stupid 😛).

Network Protocol

All nodes in my network are assigned an id and level. Given the scale of my network, both of these are unsigned bytes (i.e. 0–255). The id is chosen randomly at startup. Astute readers will note this falls victim to the Birthday Problem, so after 20 nodes there’s a 50% chance two nodes will share an id. This turns out not to actually be a problem in practice: the presence of duplicate ids on the network is only an issue if another node can see both of them (cough and the nodes are sufficiently unreliable that they restart occasionally anyway cough).

Levels: One node is designated as the "root" (by pressing buttons on the micro:bit), and is assigned level 0. All other nodes start with level 31 (chosen because it can be easily displayed in binary on the micro:bit LEDs). The level is an estimate of how close the node is to the root, and nodes will only synchronize with nodes closer to the root than it. As nodes do so, they decrease their own level (but never more than one more than the level of the node it’s synchronizing with). If the node hasn’t managed to synchronize for a few seconds, its level increases by one, ensuring that nodes always choose good nodes to synchronize with.

Here’s an example network:

Diagram showing several nodes with levels, with arrows indicating which nodes synchronize from each other.

The level 1 and 4 nodes can both synchronize from the level 0 node (since it’s less than 1 and 4). The level 4 doesn’t synchronize from the level 2 node in this example, just because it can’t see it over the radio network (but it would if it could). As the level 4 node becomes more in sync, it will reduce to a level 3 node, then a level 2 node, and finally a level 1 node—at which point it would stop synchronizing from the other level 1 node.

Pings: All nodes periodically send PING_REQUEST messages on the network, which contain:

req_node: the id of the node requesting the ping
req_level: the node’s level
ping_id: a unique (16 bit) id for this ping request
votes: a list of node ids this node would like to sync from.

The votes list is just the ids of nodes which the node has seen PING_REQUEST messages from, where req_level was less than its own: this may be out-of-date, but if so, the node can just ignore the SYNC message later (and it won't send that node id next time).

Any node which sees a PING_REQUEST message from another node responds back with a PING_RESPONSE message, which contains:

req_node: copied from the PING_REQUEST
resp_node: the id of the node responding to the ping
resp_level: the node’s level
ping_id: copied from the PING_REQUEST
req_end_timestamp: the time at which the PING_REQUEST was received, according to the responding node’s clock

Syncs: Periodically, nodes use the votes it has seen from other nodes to see if it’s the best node to send a SYNC message. If it is, it does, which contains:

node: the node sending the SYNC
level: the node’s level
timestamp: the node’s timestamp at time of sending
triggers: a list of trigger ids and timestamps (sent as deltas relative to timestamp)

When a node receives a PING from a node with a lower level, it uses the same formula from PTP to adjust its own clock. Assuming:

$T_1$ is the time the node sent a PING_REQUEST to the node sending a SYNC
$T'_1$ is the req_end_timestamp from the corresponding PING_RESPONSE
$T_2$ is the timestamp in the SYNC message
$T'_2$ is the time the SYNC message was received

Then the offset $õ$ between the two nodes’ clocks is:

\begin{equation*} õ = \frac{T'_1 - T_1 - T'_2 + T_2}{2} \end{equation*}

In practice, I use exponential smoothing on this process, which reduces the impact of outliers. Earlier I mentioned that nodes reduce their level as they become "more in sync": nodes actually reduce their level every time they do an offset adjustment where the adjustment is small enough ($|õ| < \varepsilon $, where $\varepsilon =$ 10ms).

Side note: Originally, I intended to use MicroPython on the micro:bit to implement this project. However, after just trying to implement something to decode the network protocol messages into in-memory data structures, I ran out of memory using MicroPython. I briefly toyed with the idea of having some code to generate the MicroPython code (so I could write sensible high-level code, and have it spit out stupid code), but that was far too much effort, so I switched to using the micro:bit C++ API.

Simulation

There’s a lot of detail in the previous section, without much explanation of why I designed it that way. This section isn’t going to help much, but it will explain the process a little better. When I started, I realised that attempting to develop the time synchronization protocol on the actual micro:bits would be very slow and tedious, so I started by writing a Python simulation.

The simulation let me quickly iterate on the design of the protocol, testing various number of nodes with different network connection topologies, and meant that when the network didn’t converge to a common time, I could debug the algorithm by adding things into my Python code, rather than messing around with micro:bit serial cables.

Unfortunately, I didn’t save all the diagrams along the way, but here’s the last one I generated, which shows 13 nodes all converging at different times to the same time (within an accuracy of 20ms).

A diagram showing latencies of various nodes, and lots of coloured circles. It’s not entirely clear what it represents.

The 0.1ms line in the diagram is a lie: it should be 0, but that doesn’t work so well with the logarithmic y-axis. The coloured dots on the lines indicate the level of the node at that point in time. Well. That’s what my notes claim, at least.

Triggers

So far, we’ve managed to get a mesh network of nodes synchronized to the same time, but we haven’t managed to get them to do anything useful yet. This is where triggers come in.

A trigger is just an event with an id (saying what you’d like to happen), and a timestamp in the future (when you’d like it to happen). Every SYNC message contains a list of triggers: the root node can schedule triggers, and every node in the network propagates on that list (while the triggers’ timestamps are still in the future) to all the other nodes.

In order to get triggers into the network, the root node listens on the serial port for a six-character hex number: the first two characters are a trigger id, and the remaining four are in how many milliseconds from now to schedule the trigger (for example, 2a0fa0 schedules trigger #42 to happen in 4 seconds).

Just before a trigger is scheduled to happen, the node checks if it’s reasonably in sync (the last adjustment $õ < \varepsilon $): if not, it skips the trigger (on the assumption that not playing some music is better than playing out of time).

Music

When a trigger fires, I want to play a segment (roughly five seconds long) of music. This means that if the node gets out of sync, it’ll only be silent for five seconds. Also, if the time synchronization is off by a noticable amount, it’ll only sound weird for five seconds.

Since I want the micro:bits to play music, I need some way to input music into the program. I started with Lilypond, an excellent music engraving program that I've used before (it’s kinda like LaTeX for music). As an example for this section, I’ve written Beethoven’s Ode to Joy in Lilypond:

\version "2.18.2"

\include "articulate.ly"

music = \new Staff \relative c'' {
  \tempo 4=160
  \time 4/4
  \key c \major
  e4 e f g |
  g f e d |
  c c d e |
  e4. d8 d2 |
  \bar "|."
}

\score {
  \articulate \music
  \midi {}
}

\score {
  \music
  \layout {}
}

This Lilypond file results in this PDF output:

It also generates a MIDI file (in case you’re wondering: the \articulate command just makes the MIDI output sound a lot nicer), which I can conveniently read using in Python using the Mido library, and thus generate some C++ arrays representing the data. These look like this:

typedef struct {
    uint16_t period_us;
    uint16_t duration_ms;
} music_event_t;

typedef struct {
    size_t start_event;
    size_t n_events;
} music_segment_t;

typedef struct {
    const music_event_t* events;
    size_t n_events;
    const music_segment_t* segments;
    size_t n_segments;
}

const music_event_t _song_events[] = {
    {.period_us = 1517, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1517, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1432, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1276, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1276, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1432, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1517, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1703, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1911, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1911, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1703, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1517, .duration_ms = 328},
    {.period_us = 0, .duration_ms = 47},
    {.period_us = 1517, .duration_ms = 492},
    {.period_us = 0, .duration_ms = 70},
    {.period_us = 1703, .duration_ms = 164},
    {.period_us = 0, .duration_ms = 23},
    {.period_us = 1703, .duration_ms = 656},
};

const music_segment_t _song_segments[] = {
    {.start_event = 0, .n_events = 26},
    {.start_event = 26, .n_events = 3},
};

const music_data_t song_data = {
    .events = _song_events,
    .n_events = sizeof(_song_events) /
                sizeof(_song_events[0]),
    .segments = _song_segments,
    .n_segments = sizeof(_song_segments) /
                  sizeof(_song_segments[0]),
};

The resulting song_data has two segments, which index into the events array. The intention there was that if music has repeating sections, the segments could reference the same events, saving program space: as it turned out, I had plenty of program space and finding repeating sections is hard, so this was a bit of a premature optimization.

Then all that remained was to hook up trigger n to play segment n of the music, using PWM on the piezo buzzer I’d attached to pin 0 of my micro:bits.

Reliability

Having written all this code, I found that on the micro:bits, the program would occasionally get stuck: either the CPU would freeze up entirely, or the radio module would get into a bad state and not receive / transmit messages, or other weirdness would happen.

Instead of trying to fix these, I used the watchdog timer of the nRF51 (the CPU which runs the program): this lets me configure up to 8 "reload requests", and a period. If my application code doesn’t trigger all the enabled reload requests within the period, the CPU will reset itself.

I configured the period to two seconds, and set up two reload requests: one in the main loop that I use for display (as a "the program has just stopped" check), and one inside the time sync logic, which triggers the reload request iff the time has synced (again, $|õ| < \varepsilon $) within the last 30 seconds. This second check is really useful: if the network gets into a weird state where none of the nodes can sync, the nodes will all be obviously displaying this by resetting themselves every 30 seconds.

The other huge reliability concern was the root node in the network: if that dies, all the other nodes lose their sync. As a way to reduce this impact, I made it so any node could become the root node (by holding down some buttons): that way, if I noticed the root node die, I could promote a standby node, and since it already had almost the same time, the other nodes wouldn’t lose their synchronization.

Epilogue

So while this project was really cool and fun to build, I didn’t really have a use for it until NCSS started. We run a programming competition at the camp, which this year was Hitchhiker’s Guide to the Galaxy themed. I did the theming for the scoreboard, and had a little intro animation (the words “Don’t Panic” zooming in over a background of starts), and had the micro:bits play Journey of a Sorcerer (the theme music for HG2G) during that.

I think it went reasonably well, though the big unforeseen issue was that the piezo buzzers on most of the micro:bits weren’t as loud as the ones I had been testing with, so the effect was somewhat, uh, muted. 😛

If I was to do more on this project, there’s a few things I would improve:

The time sync algorithm doesn’t actually need the SYNC packets: it could be done as part of the PING_RESPONSE packets, which would simplify it somewhat.
Send the music data across the network: compiling and flashing all the nodes every time I want to play a different song is very tedious.
Relatedly, move the song data to flash storage: that way the music data from the network would be persisted across reboots.
Use something other than piezo buzzers. I was watching the Wintergatan marble machine (well, actually the videos for the construction of its replacement), and it’s made me really want to do something where the micro:bit triggers physical actions to make noise (maybe plucking a violin string)?
Support multiple channels of MIDI. There’s actually very little stopping the micro:bits from playing polyphonic sounds: if I had compiled the nodes with different song data (but matching segment lengths), they could already do it pretty easily.

Source code: is in this Mercurial repository. It’s open source, under the Apache 2.0 license.

Finding Good Answers Quickly

2015-10-08T00:05:00+11:00

I recently had a difficult problem to solve: assigning 94 students into teams of four. The first reason it’s difficult is that you end up with 23.5 teams, and since we can’t chop students in half, we have to go with 22 teams of four, and two teams of three.

But we’re not done yet: the competition should be fair, in the sense that teams should be roughly equally matched: we don’t want to have one really good team that easily wins. So we get some estimations of students’ abilities— let’s say they get a score of 1 (worst), 2 or 3 (best). Then we can define a team’s strength as the sum of the students’ ability scores, and we want teams strengths to be roughly equal.

But we might even still have further requirements: maybe we want students to not be in teams with students they’ve already worked with (so they meet more people), or not put students who have done the competition together (as experience isn’t quite the same thing as ability).

Now this problem isn’t difficult because it’s hard to solve, but because it’s difficult to figure out exactly what the requirements are. The way I’d like to solve this problem is by specifying a small part of the problem, having something give me a possible solution, and then iterating to further define the problem (to avoid the bad cases I see).

I’ve had to solve a few problems of this nature, and after writing almost the same Python script each time, I refactored out the common code into Hufflepuff, an open-source project I released today. Like any good UNIX tool, it’s not especially interesting or complex itself, but lets you glue together other simple tools to produce something useful.

Let’s see how it works with a cut-down version of the example above. First, create a working directory, install a Python virtualenv and install Hufflepuff from pypi, and test it works:

$ mkdir -p ~/tmp/teams; cd ~/tmp/teams
$ virtualenv .
$ . bin/activate
(teams)$ pip install hufflepuff
(teams)$ hufflepuff -h
usage: hufflepuff [-h] [-c CONFIG_FILE] [--initial-binary INITIAL_BINARY]
                  [--initial-file INITIAL_FILE]
                  [--mutate-binary MUTATE_BINARY]
                  [--score-binary SCORE_BINARY] [--beam-width BEAM_WIDTH]
                  [--expand-mantissa EXPAND_MANTISSA]
                  [--expand-multiplier EXPAND_MULTIPLIER] [--target TARGET]
                  [--num-iterations NUM_ITERATIONS]
                  [--parallelism PARALLELISM]
...

We’ll also need some test data to play with, and for simplicity, we’ll just have a few students:

(teams)$ python
>>> import json, random
>>> with open('students.json', 'w') as f:
...   json.dump([
...     {
...       'name': name,
...       'skill': random.randint(1, 3),
...       'experience': random.random() < 0.3,
...     }
...     for name in 'ABCDEFGHIJKLMN'
...   ], f)
...
>>> [Ctrl-D]

You should now have a students.json file with some students, each with a unique name, skill level, and experience indicating whether they’ve done the competition before or not.

Now we need to provide three things to Hufflepuff: an initial assignment of students to teams, a program to mutate those teams, and a program to score a team assignment.

For the initial assignment, we need to specify the team sizes, and provide a valid assignment: so let’s just put the students in order. Here’s initial.py, which goes into your working directory:

import json

# For 14 students, two teams of four, and two of three.
team_sizes = [4] * 2 + [3] * 2

with open('students.json') as f:
    students = json.load(f)
assert sum(team_sizes) == len(students)

print json.dumps([
    [students.pop(0)['name'] for _ in range(team_size)]
    for team_size in team_sizes
])
assert len(students) == 0

Now when you run it, you should get the following output. It should be fairly obvious: the first team has four people (A, B, C, and D).

(teams)$ python initial.py
[["A", "B", "C", "D"], ["E", "F", "G", "H"], ["I", "J", "K"], ["L", "M", "N"]]

The next thing to provide to Hufflepuff is a way of mutating the state. In this example, the easiest thing to do is to swap a pair of students in different teams. We need to be carefuly to only produce valid states (every student should be in exactly one team), and ideally we shouldn’t duplicate states— so since teams are a set of students, let’s keep them in sorted order within the team.

Here’s mutate.py, also to be placed in the working directory:

import json
import random
import sys

# Read lines from stdin (ending cleanly on EOF):
for line in iter(sys.stdin.readline, ''):
    teams = json.loads(line)

    # Swap between 1-3 pairs.
    for _ in range(random.randint(1, 3)):
        # Select two teams to swap between.
        team_a, team_b = random.sample(teams, 2)

        # Choose a person from each team.
        person_a = random.choice(team_a)
        person_b = random.choice(team_b)

        # Swap them.
        team_a[:] = sorted(
            person_b if name == person_a else name
            for name in team_a
        )
        team_b[:] = sorted(
            person_a if name == person_b else name
            for name in team_b
        )

    print json.dumps(teams)

You should be able to run it as below. The first JSON list is input, the second is output, and it should continue accepting input until you send EOF (by pressing Ctrl-D).

(teams)$ python mutate.py
[["A", "B", "C", "D"], ["E", "F", "G", "H"], ["I", "J", "K"], ["L", "M", "N"]]
[["B", "C", "J", "M"], ["E", "F", "G", "L"], ["D", "I", "K"], ["A", "H", "N"]]

You’ll have noticed that it swaps a few pairs of people around. The initial state will be pretty bad, so we want to make a lot of swaps, to quickly head towards the better state. Towards the end of the search, lots of swaps will usually be bad, and we want single swaps to settle on the local optimum. There are more sophisticated things you could do, but I’ve found this is easy and works well enough in practice.

The last thing we need to provide is some way of scoring the state— nicely congregating all your experimental decisions of what to optimise into a simple problem: giving a score for a team assignment.

In this case, it’s easiest to specify the score as a “cost”, and say how bad an assignment is. We’ll inflict penalties for teams with more than one experienced student, and for the difference between a team’s strength and the average team strength. Here’s score.py, also to be placed in the working directory:

import json
import sys

with open('students.json') as f:
    students_by_name = {
        student['name']: student
        for student in json.load(f)
    }

def experience_penalty(teams, penalty=100):
    score = 0
    for team in teams:
        n_experienced = sum(1 for student in team if student['experience'])
        if n_experienced > 1:
            score += penalty
    return score

def skill_penalty(teams):
    strengths = [
        sum(student['skill'] for student in team)
        for team in teams
    ]

    average_strength = sum(strengths) / float(len(strengths))

    return sum(
        abs(average_strength - strength)
        for strength in strengths
    )

for line in iter(sys.stdin.readline, ''):
    teams = json.loads(line)

    # Replace students' names with their full info.
    teams = [
        [students_by_name[name] for name in team]
        for team in teams
    ]

    score = experience_penalty(teams) + skill_penalty(teams)
    print score

If you run it, and provide it the initial team JSON, it should print out some score (which will vary based on your students.json file):

(teams)$ python score.py
[["A", "B", "C", "D"], ["E", "F", "G", "H"], ["I", "J", "K"], ["L", "M", "N"]]
104.0

A side note: it’s handy to put a “debug mode” into the scoring program, so you can give it a solution, and quickly see how well it does on each of the conditions. This can be useful for distinguishing between multiple equally-scoring solutions, and also for telling if the search found the optimum.

Finally, we’ll write a search.cfg file to specify what we want Hufflepuff to do (we could also just pass the parameters as command line args, but this is nicer):

initial-binary = python initial.py
mutate-binary = python mutate.py
score-binary = python score.py
target = min

Then, all we need to do is to run Hufflepuff, which will take all those things, and run a beam search. As it runs, it prints out the scores of the items in it’s current search beam to stderr, and finally prints out [score, state] JSON-encoded lists to stdout of the best states it ever saw.

(teams)$ hufflepuff -c search.cfg
...
[2.0, [["A", "C", "D", "K"], ["B", "G", "J", "L"], ["F", "H", "I"], ["E", "M", "N"]]]
[2.0, [["A", "C", "D", "K"], ["B", "G", "J", "L"], ["F", "H", "N"], ["E", "I", "M"]]]
[2.0, [["A", "C", "E", "G"], ["D", "H", "J", "M"], ["B", "I", "L"], ["F", "K", "N"]]]
[2.0, [["A", "C", "G", "L"], ["I", "J", "K", "M"], ["D", "F", "N"], ["B", "E", "H"]]]
[2.0, [["A", "C", "L", "N"], ["I", "J", "K", "M"], ["D", "F", "G"], ["B", "E", "H"]]]

As you can see, it found a bunch of possible solutions, all of which had score 2.0. I can then manually inspect those states, determine if there’s any major problems with them, and refine the scoring program if necessary.

A cute little trick is that you can take those states (without the score), put them into best.json, and then run hufflepuff -c search.cfg --initial-binary='cat $initial-file' --initial-file=best.json to start the search from those states (and you can make it less ugly by defining initial-binary in search.cfg since that’s the default).

Pull requests, bugs and feature requests are welcome, and I’m also happy to answer questions in the comments below. And if case you were wondering, here’s the inspiration for the name.

Solving Problems Badly

2015-10-03T14:02:00+10:00

Perfect is the enemy of good.

I have an interesting story to tell today, and I’m sure the title of this post probably surprised you. Well, it’s not satire or schadenfreude, but actually serious advice. This is the story of some code I wrote (back in 2013) to solve a problem, where I deliberately chose a “bad” solution.

One of the things I’ve done for many years now is tutor high school students to learn programming as part of the NCSS Challenge. Each week, for five weeks, students are given a set of problems to solve, accompanied by notes explaining all the Python features and concepts they need to solve them. Many high school computing teachers sign up their class (sometimes using it as an assessment task), but many students also sign up themselves. In addition to the notes, students can also chat and discuss the problems (but not share code) on forums, or seek help privately from our volunteer tutors.

The challenge uses Discourse for the forums, which works well, but also for the private messaging (with some custom modifications), which works less well. There are two main problems, which the following sequence illustrates:

Student 1 sends a message (sending a notification to all tutors).
Tutor A starts writing a response.
Tutor B starts writing a response
Tutor A posts their response.
Tutor B either doesn’t notice (meaning the student gets two responses, which might attempt to solve the problem in different ways), or does notice and is annoyed that their time has been wasted.
Tutor A goes back to the notifications, and starts clicking through the unread posts, but they’ve all been answered by other tutors.

To solve the problem of multiple tutors answering the same post, we set up an IRC channel that tutors joined, so they could announce which threads they were going to work on. Various tutors wrote IRC bots to automate bits of this, and then I combined all the functionality into a single bot for keeping track of what state (unreplied/replied) threads were in, and which tutor was currently assigned to answer it.

And now you have enough background to get onto what my bad solution was. One of the parts of this IRC bot was the “threads backend”, which stores thread metadata: author, title, unreplied/replied state, and the tutor assigned to it. Now the obvious answer to this is to use a database: SQLite if it’s a toy problem, and PostgreSQL if it’s big enough.

Instead of doing that, however, I went with something far more crude:

import os

class Thread(object):
    def __init__(self, title=None, ...):
        self.title = title
        ...

    def to_json(self):
        return {
            'title': self.title,
            ...
        }

def load_threads():
    with open('threads.json', 'r') as f:
        threads = json.load(f)
    return {
        key: Thread(**kwargs)
        for key, kwargs in threads.iteritems()
    }

def save_threads(threads):
    with open('threads.json.new', 'w') as f:
        # If this crashes, we end up with an empty threads.json.new file,
        # but threads.json is still intact.
        json.dump({
            key: thread.to_json()
            for key, thread in threads.iteritems()
        }, f)
    os.rename('threads.json.new', 'threads.json')

...

def request_handler(request):
    threads = load_threads()

    if request.action == 'get_post':
        return get_post(threads, request)

    elif request.action == 'new_post':
        response = new_post(threads, request)
        save_threads(threads)
        return response
    ...

It’s terrible! On every request, the entire database is read from disk, and whenever a thread gets modified, the entire database has to be written back to disk (not just the parts which changed). There’s some other problems too: it doesn’t let you handle requests concurrently, and it doesn’t let you create secondary indices to speed up queries. It’s also missing an os.fsync() call to ensure the data is actually written to disk, but let’s not talk about that one. 😉

The secondary indices isn’t just a throwaway comment either: one of the operations this backend supports is filtering threads by various attributes. Obviously, the only way this backend can support that is by doing a linear scan over all threads, and returning the ones which match.

But actually, despite the number of raised eyebrows, it’s not actually quite as terrible as it first seems. Firstly, it’s a backend to an IRC bot, which means that any response it gives is going to be seen on an IRC channel, which is (sort of) a serialised output: so not handling concurrent requests isn’t a big deal. It also doesn’t have to handle a lot of traffic (much less than one request per second), and even a 1s latency wouldn’t be obviously bad on IRC.

The other thing to consider is that the database isn’t actually that big! In the first year, there were ~500 threads: even with a generous 1KiB of metadata per thread, that’s still less than a mebibyte, which is easily written to disk in less than a second. It’s also small enough to fit in the OS buffer cache, so reading the database from “disk” isn’t expensive either. And you can also see that those linear scans aren’t that expensive either.

Futhermore, there’s some advantages to the code. It’s simple: there’s full visibility into the internal state, serialisation and deserialisation processes (unlike many ORMs, e.g. SQLAlchemy). It’s debuggable: it’s just JSON, so it’s incredibly easy to manually inspect (after pretty-printing with python -m json.tool) or write custom tools to analyse it. Schema migrations are easy: tell Thread.__init__ about the new key (providing a default value), and have Thread.to_json emit whatever schema you want stored: on the next database write, all record will be rewritten to the new schema.

This system performed adequately for the first year it was used. There were many schema changes, but it had no data corruption or was the cause of any outages. It’s performance was good enough: by the end, some filtering requests took just over a second to respond to, but it was never bad enough for users to notice.

The next year, I rewrote it to use SQLAlchemy backed by SQLite and added indices to optimise the most frequent filter requests. But I didn’t see that as admitting defeat: on the contrary: by starting with a terrible implementation, I learnt exactly what the bottlenecks were, and could then make informed decisions when designing the next version.

Perhaps you weren’t surprised I went with a “bad” solution first: the idea is certainly not new. It’s an interesting strategy, but you need to be careful with it: if you don’t make it bad enough, you won’t be motivated enough to rewrite it in the future, and if you make it too bad, then it’ll crash and burn before you’re ready to rewrite.

Even if this story hasn’t fully convinced you that deliberately choosing imperfect solutions is a good idea, I hope you can see it’s a viable strategy in some cases.

Introducing vipdf

2014-03-09T12:41:00+11:00

I mentioned in an earlier post (almost a year ago now, eep!) that I had some more software to release to the world, and as promised, this is the next one. The word “introducing” in the title is perhaps a bit misleading— I’ve written (but not released) several similar programs to this, and I’ve been using this particular program for a few years now, so it’s not really new.

But enough of that, let’s get on to what this thing is.

Rationale

The rationale for vipdf is that I’m unhappy with a lot of existing presentation software, for a variety of reasons, and I realised I could solve a lot of these problems with a vi-style keyboard mapping for navigating presentations.

The things which I wanted specifically were:

Moving to an arbitrary slide. If you get to the end of a presentation and then someone in your audience asks you a question about something on slide 15, you should be able to go directly to that slide— you shouldn’t have to move through all the intermediate slides.
Skipping over slides. Suppose you’re giving a talk, and you realise that you haven’t got time to talk about something in depth. No matter, your software should let you just skip over the next slide (or two), without flashing those slides at your audience.
Keyboard controls. If I have to use a mouse while giving a presentation, I’m going to accidentally click things I didn’t mean to and miss things that I meant to click on. Since I can touch-type, it makes sense to control everything by the keyboard, and by using familiar vi-style keybindings, I can reduce the cognitive overhead of remembering shortcuts.

At the moment, vipdf only supports PDFs (just in case the name didn’t make that obvious), which makes it great for pairing with beamer (as much as I dislike TeX, I haven’t found anything better yet), but you can easily export PDFs from most other presentation software (Microsoft Powerpoint, Keynote and LibreOffice Impress all have this functionality).

Yes, this will remove the gratuitous animations you had carefully wasted time setting up. No, there’s no support for embedded videos (in case you didn’t know, yes, you can do that with PDFs).

Controls

As I mentioned, there are vi-style keyboard controls:

<Left>: Move to the previous slide
N <Left>: Go back N slides (e.g. 3<Left> will move you back 3 slides)
<Right>: Move to the next slide
N <Right>: Go forward N slides (e.g. 2<Right> will skip forward over a slide)
N g: Goto slide N (e.g. 33g takes you to slide 33)
b: Toggle blackout

Yeah, I didn’t map hjkl. If you feel strongly about that, you should let me know in the comments what the mapping should be (should both h and j move back, or only h?)

There’s also controls for auto-advancing slides (t is for timer, btw):

t: Toggle auto-advance on or off.
N t: Start auto-advance (if not already started), and set the time interval to advance every N seconds. The default time interval is 5 seconds.
N [: Set the start slide to slide N, or unset if N is 0.
N ]: Set the end slide to slide N, or unset if N is 0. If auto-advance reaches the end slide, it will loop back to the start slide if one is set, otherwise it will turn off that auto-advance.

API

There’s also a DBUS api available for use, and there’s also a sample control script provided so you can write your own scripts to control your presentation:

$ vipdf-control list
# will give you a list of viewer ids, though at the moment there's just one
# you should take that and pass it into the following commands

$ vipdf-control next $viewerid
# advances by 1 slide

$ vipdf-control next $viewerid 2
# advances by 2 slides

$ vipdf-control prev $viewerid
# goes back 1 slide

$ vipdf-control goto $viewerid 10
# goto slide 10

Unfortunately, the API doesn’t make a lot of sense at the moment, as you can’t have more than one presentation open in a vipdf instance at a time.

Implementation

vipdf is written in Vala, using Poppler and Clutter. You can find the code here.

Quick build instructions for Debian:

$ sudo aptitude install valac libgee-dev libpoppler-glib-dev libclutter-1.0-dev
$ make
...
$ bin/vipdf /path/to/file.pdf

I’ve also written a web interface for it (which also integrates with pjlink).

Python: Wat

2014-03-08T19:18:00+11:00

You’ve all seen wat, right? Of course you have.

Of course, Python doesn’t have any Wats, right? Well, not so much.

Let’s talk about Python!

>>> 'toys"r"us'
'toys"r"us'

Makes sense. So if we change those outer quotes to double quotes, we should get a syntax error, right?

>>> "toys"r"us"
'toysus'

Wat.

Let’s talk about Python!

>>> 42**True
42
>>> 6 * False
0

This is because before True and False were available as builtins, people would write True = 1 and False = 0 at the top of their Python files. When they were introduced, they made bool a subclass of int, True == 1 and False == 0.

So of course, you can do arithmetic with booleans.

Wat.

Let’s talk about Python!

>>> data = {
...     1: 'one',
...     '1': 'won',
...     True: 'true',
... }

Right, it’s a dictionary with three items: the keys have different types, but the basic idea is that we map objects to some string representation of them. Right?

>>> data
{1: 'true', '1': 'won'}

There are three behaviours of dictionaries which come into play here. The first is that keys in dictionaries compare equal if they have the same hash value (__hash__), and compare equal (__eq__). The second is that dictionary literals are evaluated by starting with an empty dictionary, and inserting each item in order (left to right). The third is that if you are inserting an item into a dictionary and the key is already there, the value will be replaced, but not the key.

Which means that in the example above, this is what happens:

>>> data = {}
>>> data[1] = 'one'
# {1: 'one'}
>>> data['1'] = 'won'
# {1: 'one', '1': 'won'}
>>> data[True] = 'true'
# {1: 'true', '1': 'won'}

Wait, why did the last insertion replace the value for 1? Well because hash(True) == hash(1), and True == 1, of course.

Wat.

Enough making fun of languages that suck, let’s talk about JSON!

>>> import json
>>> data = {'1': 'won', 1: 'true'}
>>> json.dumps(data)
'{"1": "won", "1": "true"}'

This is valid JSON.

Wat.

Let’s talk about Python.

>>> 1 > 2 is False
False

Uh, wait, but 1 > 2 isn’t true, so it should be False, right? Like this:

>>> (1 > 2) is False
True

Hmmm, that works, maybe it’s just a precedence thing:

>>> 1 > (2 is False)
True

Nope (and note the re-occurence of False == 0 in this). This is actually a case of Python’s chained comparisons:

>>> 1 > 2 and 2 is False
False

Chained comparisons are actually awesome…

…but if you ever use it with disparate operators like this, Wat.

And let’s finish up with some Python!

This was fixed in Python 3, but there’s still plenty of Python 2 programmers just waiting to bump into this one.

>>> class GiantSpiders:
...    count = 0
...    legs = [
...        'leg %d' % count
...        for count in range(8)
...    ]
...
>>> GiantSpiders.count
7

Arrrghh! Help, spiders!

Wat.

Also, the one which didn’t qualify

Time objects representing midnight are falsey. Except, it's not quite as simple as that. Unfortunately, this only qualifies for an honourable mention: there’s an open bug for this issue, so it might be fixed.

Learning Go with AppEngine

2013-04-06T18:24:00+11:00

I’ve been meaning to learn Go for a while, and I had a nice opportunity today to do that. But before I talk about Go, let me explain why I was writing a Go program in the first place.

Last night, I saw that Alex (a friend of mine) had written a little website for unicode faces called textfac.es. At the moment, you need to ask him to add new faces, but there’s a dynamic voting system: every time you click on a face, that’s counted as a vote.

Of course, with such a simple voting system, it’s pretty obvious that it’s going to be exploitable in various ways, so I set out to explore some of them. The first thing to do is to have a look at how clicks translate into votes in the browser. Looking at the JavaScript, I saw (essentially) this:

var faces = $("button.facebtn");

clip.on("mousedown", function(client, args) {
    var id = $(this).attr("face-id")
    $.ajax("/increment?id=" + id);
});

Which means that when you click it, it sends a GET request to a URL like http://textfac.es/increment?id=37.

The attack I didn’t do

The obvious thing to do would be to run a brute-force attack to increment the count as much as possible, like this:

$ while true; do wget -qO - http://textfac.es/increment?id=37 &>/dev/null; done

Bor-ing.

Not only that, it’s also very easy to block with a per-ip rate limit (which was put in place). I wanted to be more subtle about it, and make it more difficult to identify whether it was spam or not.

My first attack

Having voting happen by a GET request is clearly a mistake. According to RFC2616 (Section 9.1.1), GET requests should not change any state on the server:

In particular, the convention has been established that the GET and
HEAD methods SHOULD NOT have the significance of taking an action
other than retrieval. These methods ought to be considered "safe".

What this means is that it’s fairly easy to get a browser to send a GET request for you, since the browser "knows" it’s a harmless request.

And so, I added a little bit of HTML to the bottom of all the pages of my blog:

<img src="http://textfac.es/increment?id=37" style="display:none" />

Then I needed to generate some traffic, so I wrote a blog post (that I had been meaning to write for a while) and posted it to Google+, Twitter and Facebook.

So if you were one of the nice people who visited my blog since last night, your browser helpfully loaded that "image" with a GET request, thus casting a vote using your IP. Why thank you, random stranger!

My second attack, with Go!

Great, this generated some traffic, and I was sitting at #5 for a while. But I wanted to get higher up, and so I had the thought: you know, it would be really nice if I could use Google’s network to spam the voting system for me…

I knew that it was possible to make outbound web requests with AppEngine, and I’d been meaning to write something in Go for a while, so this seemed like a good excuse.

I just needed to write a simple application to take a url and a number, and send that many GET requests to the url. Easy, peasy.

So I wrote that, and then did some testing, and discovered that there was still only a fairly limited number of IPs which it would generate traffic from, though I could vary the IPs by changing where my request to AppEngine came from.

My final variation was to hard-code the url and number, and to use that in an img tag on my blog, so that the requests to AppEngine would be from a diverse IP range. I have no idea if that’s actually working or not!

Finally he gets onto actually talking about Go…

I only ran into two interesting things with Go. The first one was when I was doing authentication: I wanted to restrict it to a set of valid logins, which basically meant testing to see if a list contained a certain item.

I was very surprised that Go doesn’t have a builtin function for this, I don’t really expect to have to write linear search in 2013. I did find someone’s recipe for this on the golang-nuts mailing list, which I shamelessly stole^Wappropriated.

On this topic, it’s quite possible that the Go language does has this built in, and I just didn’t find it: while the documentation does appear to be complete, I didn’t find it anywhere near as readable or searchable as the Python documentation (hmmm, I think Python spoils me).

The other thing which I found interesting was that Go doesn’t have a map() primitive using goroutines. While it’s not overly difficult to express the same thing with channels, I felt that my code would have looked cleaner.

Here’s a contrived example using explicit goroutines and channels:

func square(x int, ch chan int) {
    ch <- x * x
}

func main() {
    n := 10

    ch := make(chan int)

    for i := 0; i < n; i++ {
        go square(i, ch)
    }

    sum := 0
    for i := 0; i < n; i++ {
        sum += <-ch
    }
}

And here’s what the equivalent code would look like with a concurrent map function:

func square(x int) int {
    return x * x
}

func main() {
    values := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}

    results := go_map(values, square)

    sum := 0
    for _, v := range results {
        sum += v
    }
}

Of course, I could have left that last example as pseudocode, but I didn’t like that, so I wrote out an implementation of go_map. Unfortunately, my go-fu isn’t at the point where I can write a generic version for any type. Perhaps someone will write one, and leave a note in the comments. Even better, perhaps someone will point out that this already exists in the standard library somewhere?

func go_map_wrap(x int, fn func(int) int, ch chan int) {
    ch <- fn(x)
}

func go_map(values []int, fn func(int) int) []int {
    results := make([]int, len(values))

    ch := make(chan int)

    for _, v := range values {
        go go_map_wrap(v, fn, ch)
    }

    for i := 0; i < len(values); i++ {
        results[i] = <-ch
    }

    return results
}

Epilogue

By the way, that last complaint wasn’t because I’m a functional programming nut or anything, it’s because I wanted to send off N concurrent web requests, and it seemed like a pain to have to manually manage the channel.

On the whole I do like Go, and in my book it’s up with Vala and Scheme as nice languages. Python is still vastly more usable, but it’s likely a language which I’ll use again in the future.

P.S: Alex is not an awful developer! He did realise the hole when he initially coded it, and just didn’t get around to fixing it until I started messing around. So don’t blame him, he’s awesome.

Introducing gst-launch-dynamic

2013-04-05T23:15:00+11:00

I thought it would probably be a good idea to write posts about the random projects I do. As it happens, this particular post is about a tool which still could do with more work, but is in a reasonable enough state to present to the world. I’ve got another tool which I’ll write a post about after doing a little clean up.

So, on to the entertainment. GStreamer is a multimedia framework, which lets you build applications which do all kinds of strange and wonderful things with multimedia (audio / video / anything you can stream).

I like to think of it as UNIX pipelines for multimedia: a GStreamer pipeline is built out of many elements (processes), and each element has a number of pads (stdin / stdout / stderr / other fds) which can be joined to other elements. Indeed, there’s even a nice tool called gst-launch which lets you express the pipeline in a UNIX-like syntax— in fact, it’s better, since it lets you connect any element to any other element, which is not possible (as far as I can tell) in the POSIX shell language.

What is it?

Great, so that’s what GStreamer is, what does my tool do? Well, it’s almost the same as gst-launch, except for one little thing: it lets you modify the pipeline at runtime. I think this is both amazing and slightly insane: it greatly lowers the barrier for ad-hoc experimentation with pipelines, and allows for easy scripting of pipelines.

How does it work?

It’s very simple. To run it, call it in the same way you would normally call gst-launch:

$ gst-launch-dynamic videotestsrc ! autovideosink

It will construct the pipeline, and start playing it, in this case showing the SMPTE colour bars. But, to let you change the pipeline, it also watches standard input for commands, so we can type this:

videotestsrc0.pattern = 1

This sets the pattern property on the videotestsrc0 element (since we didn’t give it a name, it gets automatically numbered) to 1, which corresponds to "snow" (i.e., black & white noise). As soon as you press enter, the property is set, and the video displayed on your screen will change.

It’s probably worth noting a bug (well, more of an inconsistency) here: the thing on the right hand side of the equals sign is interpreted as a Python object (using ast.literal_eval), which isn’t exactly the same syntax that the command-line arguments (and gst-launch) uses.

Other commands

Aside from changing properties, you can control the state of the pipeline with these commands:

play
pause
stop

You can rewire the pipeline by adding, removing, linking and unlinking elements. This is a little unstable at the moment, since it’s just translating commands into GStreamer API calls without any sanity checking. It is possible to replace an element with another at runtime, but you need to be careful about unlinking and relinking the pads.

Link an element to another: element.sink -> element.src (in this case, both pad names are optional)

Unlink an element from another: element.sink x> element.src

Add an element: + type key=value …

Remove an element: - element

Scripting

One immediate possibility is to write a program which sends these commands. A side note: during development, FIFO files are great for keeping the pipeline open while you modify your script. For example:

$ mkfifo commands.fifo
$ gst-launch-dynamic … < commands.fifo
$ ./my-random-script >> commands.fifo

As an example of how this can be completely useless, but very entertaining, here’s how to make any video psychedelic.

psychedelic.py

import time

STEP = 0.01
DELAY = 0.01

def drange(start, stop, step):
    r = start
    while r < stop:
        yield r
        r += step

while True:
    for value in drange(-1.0, 1.0, STEP):
        print 'vb.hue =', value
        time.sleep(DELAY)

Usage:

$ python -u psychedelic.py | \
  gst-launch-dynamic filesrc location=gitannex.ogv ! \
    decodebin2 ! videobalance name=vb ! autovideosink

Finale

After I wrote this tool, I realised that nothing I had built so far really utilised its potential, so I really hope that someone will find this and build something cool with it.

Of course, you can’t do that if I don’t point you at the code!

Blackmagic DeckLink, ConsoleKit and TTYs

2013-02-19T07:11:00+11:00

Regular readers may skip this blog post, it’s being posted so people find the solution when the google it, not because it’s interesting.

I had a computer with a Blackmagic capture card, and discovered that it wouldn’t shut down properly. After tracing my way through PolicyKit, I discovered that it thought that my login session wasn’t local, but was a remote session.

Looking at the output of ck-list-sessions, ConsoleKit didn’t have a x11-display-device entry for my session (it’s normally something like /dev/tty7). Confusingly though, it did have an x11-display (which was :0 as expected).

ConsoleKit determines the x11-display-device using a helper program, which on my system is located at /usr/lib/ConsoleKit/ck-get-x11-display-device.

This helper program parses /proc/tty/drivers, into a struct in src/ck-sysdeps-linux.c:92. I discovered the name in /proc/tty/drivers was more than 16 characters, and hence overflowed the name array, causing the program to segfault.

Since ConsoleKit’s project page indicates that there’s no longer any active development on it, I decided on the hacky solution to solve my problem, rather than fixing the actual problem and bumped the size of the name array up by a couple of bytes.

Index: consolekit-0.4.5/src/ck-sysdeps-linux.c
===================================================================
--- consolekit-0.4.5.orig/src/ck-sysdeps-linux.c    2012-07-14 14:08:02.298395776 +1000
+++ consolekit-0.4.5/src/ck-sysdeps-linux.c 2012-07-14 14:11:07.282389984 +1000
@@ -93,7 +93,7 @@
         guint major_number;
         guint minor_first;
         guint minor_last;
-        char name[16];
+        char name[24];
         char devfs_type;
 } tty_map_node;

Shell startup scripts

2013-02-17T05:50:00+11:00

If you’re a regular shell user, you’ve almost certainly got a .bash_profile or .bashrc script in your home folder, which usually contains various tweaks, such as setting environment variables (adding that directory to $PATH), telling your shell to do clever things (like set -o noclobber) and adding various aliases to commands (like alias please=sudo).

(If you’re really organised, you’ll have all your dotfiles in a repository somewhere so that you can keep your settings synchronised across all the machines you work on.)

Anyhow, I suspect that few people know when things like .bash_profile and .bashrc actually get executed. When I started, I just followed people’s advice of putting stuff in .bashrc, and then when it didn’t work, into .bash_profile. I could stop here and describe just the bash startup process (as silly as it is), but there’s a complication in that I switched to zsh a few years ago (and haven’t looked back), but occasionally use bash on machines which don’t have zsh installed.

In order to handle this nicely then, I need to be able to specify things which are specific to bash or zsh in their own files, and then to specify things which any POSIX-compliant shell (like aliases and environment variables) can understand in a common startup file.

My solution to this problem is to define some new dotfile folders, one for each shell (.bash/, .zsh/ and .sh/), and one for the shell-independent files (.shell/):

.bash/
    env
    interactive
    login
    logout
.sh/
    env
    interactive
    login
.shell/
    env
    interactive
    login
    logout
.zsh/
    env
    interactive
    login
    logout

“But!”, you say, [1] “what do the different files in here do?” Ah, well I’m glad you asked. There are two kinds of shells:

[non-]interactive shells (you type into them / shell scripts)
[non-]login shells (the shell run when you first login / subshells)

All shells will first run env, then login shells will run login, then interactive shells will run interactive. Once finished, login shells will run logout.

Where to put stuff

It all depends on when it needs to be run.

If it’s setting / modifying environment variables, it should go in login. If it’s alias or a terminal-specific environment variable (e.g., GREP_COLOR), it should go in interactive. In my .shell/env file, I have my umask set, and also define some useful functions for modifying colon-separated path environment variables (like $PATH).

Even if you don’t adopt anything else from my scheme, I’d recommend you take a look at the what my functions are doing which differs from something like export PATH=$PATH:/path/to/dir.

That particular pattern is way too common, and is very dangerous if you consider the case when $PATH (or whatever your variable is, like $LD_LIBRARY_PATH) isn’t set. Then, the value will be :/path/to/dir, which usually means both /path/to/dir and the current directory, which is usually both unexpected behaviour and a security concern.

With my implementation (see .shell/env_functions), you can append, prepend and remove directories from any colon-separated environment variable, and when appending or prepending, you are guaranteed that directory will only then appear in that variable once.

As a side note, I’m very disappointed in my indirect_expand() function, so if you have a better solution, please let me know (or send me a pull request).

Implementation

In order to implement this, you first need an understanding of which startup files are run in each case. Of course, this isn’t hard, since all the shells have the same, sensible system. Ahahaha, no.

Fortunately, I’ve read the man pages for you, and drawn a pretty diagram. To read it, pick your shell, whether it's a login shell, whether it's interactive, and follow the same colour through the diagram. When the arrows split out to multiple files, it means that the shell will try to read each one in turn (working left to right), and will use the first one it can read.

An important point here is that if you have a non-login, non-interactive shell running a plain POSIX shell (dash, bash in sh compatibility mode), it won’t fulfil our contract in that it won’t read .sh/env (and hence .shell/env).

The other special case which exists is Bash’s remote shell mode (which is rather warped), where it tries to detect if it’s running under ssh or rsh (I assume by looking at the process name of its parent), and if so, follows a different path, which is indicated on the diagram.

Except that diagram shows what happens according to the man page, and not what happens when you actually try it out in real life. This second diagram more accurately captures the insanity of bash:

See how remote interactive login shells read /etc/bash.bashrc, but normal interactive login shells don’t? Sigh.

Finally, here’s a repository containing my implementation and the graphviz files for the above diagram. If your POSIX-compliant shell isn’t listed here, or if I’ve made a horrible mistake (or just a tiny one), please send me a pull request or make a comment below, and I’ll update this post accordingly.

[1]	and since I’m writing this, I can make you say whatever I want for the purposes of narrative.

PSA: Hotmail is Borken

2013-02-09T15:49:00+11:00

I’ve just debugged an issue where someone was trying to send an email to my domain, and got back an error message, which they took to mean that the email address no longer existed.

Of the very little debugging information I managed to get out, was this snippet (some information anonymised):

Reporting-MTA: dns;blu0-omc2-s22.blu0.hotmail.com
Received-From-MTA: dns;BLU168-W101
Arrival-Date: Fri, 8 Feb 2013 19:04:02 -0800

Final-Recipient: rfc822;bob@example.net
Action: failed
Status: 5.5.0
Diagnostic-Code: smtp;550 relay not permitted

If you search for "hotmail 550 relay not permitted", you will find this post, which describes the almost unbelievable problem.

Turns out hotmail will try to send mail to port 25 of the hosts configured as your A records first, and if it manages to make a connection, will completely ignore your MX records.

What.

I checked the mail server logs on my web server (I do run a mail server there, for other reasons unrelated to that domain), and discovered, yes, there was an attempted mail delivery:

2013-02-09 14:04:26 H=blu0-omc2-s22.blu0.hotmail.com [65.55.111.97] F=<someone@hotmail.com> rejected RCPT <bob@example.net>: relay not permitted

How anyone could think this was a good idea defies belief. I think I’m still in shock.

Controlling Projectors with PJLink

2012-11-30T22:41:00+11:00

I regularly need to control a data projector, and previously the only way to do this has been using a web interface created by the manufacturer. Of course the interface is, while usable, poorly designed, and somewhat sluggish.

Fortunately, there’s a protocol for controlling projectors, PJLink, but unfortunately, there aren’t many implementations (I did find a perl module in my search), so I decided to write my own implementation in Python.

So I did, and it took me just a single morning, which I was quite impressed with. The specification is reasonable simple, and aside from one or two stupid mistakes, implementing it was straight forward. In addition to the Python API, I also wrote a simple command line utility for communicating with the projector, which is probably more useful than the API (since you can throw it into your own shell scripts).

If you want to try it out, you can install it from PyPI using pip.

$ pip install --user pjlink
# or if you're in a virtualenv
$ pip install pjlink

The location of projectors can be specified in ~/.config/pjlink/pjlink.conf (on Linux, see appdirs for your platform):

[default]
host = myprojector
port = 4212
password = JBMIAProjectorLink

[other]
host = bob

Note this stores your passwords in plain text: this is an indication of what I think of the authentication protocol which PJLink uses. I couldn’t find a specific vulnerability, but I still wouldn’t trust it.

Then, you can use -p to reference the projector in the config file. You can also just specify a host and port manually (but not a password).

# get power state of myprojector:4212 (using password)
$ pjlink -p default power
# get power state of bob:4352 (default port, no password)
$ pjlink -p other power

# for the "default" projector, you can omit -p
# this is the same as the first command
$ pjlink power

# get available inputs
$ pjlink inputs

# get current input
$ pjlink input

# switch to input RGB-1
$ pjlink input RGB 1

And there are many more commands available: everything in the specification is implemented as plainly as possible, so if the projector can do it, my tool supports it.

And for the record, I disclaim any responsibility for inappropriate use of this tool. (because I can think of a few!)

Awesome + GNOME Configuration

2012-11-28T10:31:00+11:00

I was recently prompted to publish my configuration for integrating GNOME and awesome. At the time I wrote this setup, the awesome wiki was recommending a configuration which didn’t correctly autostart applications, nor integrate well with anything expecting to find a session manager. However, it now contains instructions for a similar setup, so you should have a look at that too.

The first thing you need is an entry to tell your display manager about a new session type, so put this in /usr/share/xsessions/awesome-gnome.desktop:

[Desktop Entry]
Name=awesome-gnome
Comment=This session logs you into GNOME using the Awesome WM
Exec=gnome-session --session=awesome-gnome
TryExec=gnome-session
Icon=
Type=Application

Then you need to tell gnome-session about the "awesome-gnome" session. You might want to modify the notification daemon here. Put this in ~/.config/gnome-session/sessions/awesome-gnome.session:

[GNOME Session]
Name=awesome-gnome
RequiredComponents=gnome-settings-daemon;
RequiredProviders=windowmanager;
DefaultProvider-windowmanager=awesome
DefaultProvider-notifications=notification-daemon
FallbackSession=gnome-fallback
DesktopName=GNOME

Then, GNOME expects to find a XDG application entry for "awesome", so put this in ~/.local/share/applications/awesome.desktop

[Desktop Entry]
Type=Application
Name=Awesome
Comment=Window manager
Exec=sh -c '/usr/bin/awesome &'
NoDisplay=true
X-GNOME-WMName=awesome
X-GNOME-Autostart-Phase=WindowManager
X-GNOME-Provides=windowmanager
X-GNOME-Autostart-Notify=true
X-GNOME-AutoRestart=false

Then you should be able to logout, and immediately see the awesome-gnome session which you can log into. Your display manager usually remembers the last session you logged into, but using this method, you can easily switch between GNOME shell, GNOME fallback and awesome-gnome.

Prettify script for PC^2

2012-11-17T17:23:00+11:00

When I was doing programming competitions at uni, I wrote this script, which takes the default HTML output from PC^2 and add some colour to it to indicate passed / failed problems. Since it was written / enhanced on the day of the competition, there’s definitely some hacky code in there, but I’m publishing it so future generations can steal it for their own nefarious purposes.

This code is released into the public domain.

# prettify.py

import re
import string
import sys

from lxml import html, etree

doc = etree.parse(open('prettify.xsl', 'rU'))
xslt = etree.XSLT(doc)

doc = html.parse(sys.stdin)

passed = re.compile('^[0-9]*/[1-9][0-9]*$')
failed = re.compile('^.*[1-9]/[-0]*$')

for node in doc.xpath('//td'):
    text = node.text or ''
    if passed.match(text):
        node.attrib['class'] = 'passed'
    elif failed.match(text):
        node.attrib['class'] = 'failed'

problems = dict(zip(string.uppercase, [
    'An Industrial Spy',
    'Simple Polygon',
    'High Score',
    'Rankings',
    'Wormholes',
    'Fractal',
    'Rewards for Math!',
    'Divisible Subsequences',
]))

for node in doc.xpath('//th/strong/u'):
    node.text = problems.get(node.text, node.text)

doc = xslt(doc)

#body = doc.xpath('//html/body')
#d = etree.Element('div')
#d.attrib['class'] = 'timer'
#
#import sys
#from datetime import datetime
#n = datetime.now()
#t = datetime(2011, 9, 3, 17, 0, 0, 0)
#print>>sys.stderr, n - t

print html.tostring(doc)

<!-- prettify.xsl -->

<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

    <xsl:variable name="stage" select="0" />

    <xsl:output method="xml" indent="yes"
        doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"
        doctype-public="-//W3C//DTD XHTML 1.0 Transitional//EN" />

    <!-- Document structure -->
    <xsl:template match="/">
        <html>
            <xsl:apply-templates select="/html/head" />
            <xsl:apply-templates select="/html/body" />
        </html>
    </xsl:template>

    <!-- Head for styling, etc. -->
    <xsl:template match="/html/head">
        <head>
            <xsl:copy-of select="/html/head/node()" />
            <script type="text/javascript" src="http://cornify.com/js/cornify.js"></script>
            <!--<meta http-equiv="refresh" content="5" />-->
            <link href="http://fonts.googleapis.com/css?family=Questrial"
                rel="stylesheet" type="text/css" />
            <style type="text/css">
                html, body { margin: 0; padding: 0; }
                a { color: #729fcf; }
                body {
                    font-family: 'Questrial', sans-serif;
                    color: #eeeeec;
                    background: #2e3436;
                }
                table { font-size: 20pt; }
                .passed, .failed { font-weight: bold; }
                .passed {
                    background: #8ae234;
                    color: black;
                }
                .failed {
                    background: #ef2929;
                    color: white;
                }

                h1 { text-align: center; margin: 0; }

                thead th { font-size: 12pt; }
                th.team-name {
                    font-weight: normal;
                    text-align: left;
                }

                th:nth-child(2) { width: 50px; }
                td:nth-child(4) { width: 3.5em; }
                td:nth-child(12) { width: 3.5em; }
            </style>
        </head>
    </xsl:template>

    <!-- Body -->
    <xsl:template match="/html/body/table[1]">
        <table id="scoreboard">
            <thead>
                <tr>
                <xsl:for-each select="tr[1]/th">
                    <th scope="col"><xsl:value-of select="strong/u" /></th>
                </xsl:for-each>
                </tr>
            </thead>
            <tbody>
                <xsl:for-each select="tr[position() > 2]">
                    <tr>
                        <!-- Rank -->
                        <td class="rank"><xsl:value-of select="td[1]" /></td>
                        <!-- Name -->
                        <th class="team-name" scope="row"><xsl:value-of select="td[2]" /></th>
                        <xsl:copy-of select="td[position() > 2]" />
                    </tr>
                </xsl:for-each>
            </tbody>
        </table>
    </xsl:template>

    <xsl:template match="/html/body">
        <xsl:if test="$stage >= 1">
            <h1>Competition is over! Congratulations to all!!!</h1>
        </xsl:if>
        <xsl:apply-templates select="table[1]" />
        <p>Prettify script by Peter Ward</p>
        <xsl:if test="$stage >= 2">
        <script type="text/javascript">
            <![CDATA[
            var count = 0;
            var delay = 10000;
            function magical() {
                cornify_add();
                count++;
                if (count > 60) {
                    var child = null;
                    var nodes = document.body.children;
                    for (var i = 0; i < nodes.length; i++) {
                        if (nodes[i].tagName == "DIV") {
                            child = nodes[i];
                            break;
                        }
                    }
                    document.body.removeChild(child);
                }
                window.setTimeout(magical, delay * (Math.random() / 2.0 + 0.75));
                if (delay > 10)
                    delay *= 0.9;
            }
            window.onload = magical;
            ]]>
        </script>
        </xsl:if>
    </xsl:template>
</xsl:stylesheet>

End of Honours

2012-11-15T09:57:00+11:00

Well, I’ve handed in my thesis (Providing Useful Feedback to Beginner Programmers), and done a presentation for it (twice, for logistical reasons), so I’ve now finished everything I need to do for uni! I’m very happy about this, and I’ve been enjoying the little bubble of happiness about its completion the past few days.

We also continued the tradition of doing a prank on my supervisor’s office, by turning it into a fish tank / sea themed room. Photos from the setup and the final office are in this photo album.

And for the few people who haven’t already caught up with this news, I start working full-time at Google on Monday.

gvfs-gphoto2-volume-monitor Memory Usage

2012-11-11T15:25:00+11:00

I’ve noticed over the past few weeks that occasionally there was a gvfs-gphoto2-volume-monitor process taking an unreasonable amount of memory. Since my computer just ran out of memory, I was prompted to investigate why, and here’s what I found.

I’m not sure how much of this is relevant, but I’m running the awesome window manager with a partial GNOME desktop setup. The particular problem seems to be that when my Samsung Galaxy S3 is plugged in as a MTP device, if I have rhythmbox running with the MTP plugin, it will load the device in rhythmbox but (for some unknown reason) disconnect it again. Then, a moment later, it will load the device again, and disconnect again, and so on.

I have no idea what gvfs-gphoto2-volume-monitor is doing, but at a guess, each time it loads the device, it causes that process to do some work, reading stuff into memory, and then not freeing it again when the devices gets reloaded.

Since I’m not using it regularly, my solution was simply to disable the MTP plugin in Rhythmbox. Hopefully this information will help someone else figure out why they have no memory, and maybe follow up the full story and fix it.

Restarting this blog

2012-11-07T18:40:00+11:00

Welcome to my blog! Apparently I think these things are still useful for something (I may be slightly delusional on this point).

I’m using Pelican, and have imported all the content from my old blog(s), but there are still some formatting issues with the import which I need to manually fix up.

In other news, I submitted my honours thesis today (“Providing useful feedback to beginner programmers”), which means I have only a presentation to go before I have completely finished doing everything I need to for my Bachelor of Information Technologies degree. Happy face!

Keys

2011-08-07T04:35:00+10:00

Just a quick note for the people who care about trivial things like privacy. ;)

I’m decommissioning my four year old 1024-bit DSA key (1024D/D1ED1C9F), and replacing it with a new 4096-bit RSA key (4096R/3906C4AF). The fingerprint of the new key is 410D 28F0 1DFB 0DC4 0E0C D663 E060 B7CD 3906 C4AF, and it has been signed by the old key.

I’ve set the old key to expire at Christmas this year.

Semester 1, 2011

2011-07-11T10:58:00+10:00

Another year, another semester, another set of results. This time, a little worse than usual, for reasons I’ll explain shortly. However, let me first give you an update of what I’ve been up to since I last posted on my blog (more than six months ago :O).

First up was the work I did with CHAI over the holidays—I integrated WebKit into the Cruiser tabletop framework, and built a tagging interface to simplify finding content in large collections of images. There is a video demonstrating the Cruiser framework, which shows some of my work—my stuff starts about 80% of the way through, look for spiders. ;)

Straight after completing my CHAI work, I went back into uni mode! I took four courses:

Object Oriented Design (info3220),
Project Management in IT (info3402),
Artificial Intelligence (comp3608) and
a project unit with James Curran (info3911).

In the project unit, I was working on reproducing the Tagme system for use in linking "mentions" (useful snippets of text) in documents to their correct Wikipedia articles. In addition to these units, I was also tutoring the second-year databases course, info2120.

Now, let me tell you a little about info3220. The first thing about this unit is that it purports to teach Object Oriented Design, however in reality, the main focus of the unit was on teaching C++’s implementation of classes. It seems no-one had the idea that OO might actually have been implemented in more than one language? And that it might not have been implemented in exactly the same way in each language? Personally, I would love to see OO integrated into the Programming Languages and Paradigms unit (and the compilers part of that course should be split out into a separate unit to make room), where you can show all the wonderful ways of expressing the same operation in different ways (some of which are more maintainable than others).

Putting that aside, however, at the start of semester, we were offered the choice of doing standard assignments (which would be writing a vending machine-like program, and sounded boring), or doing an advanced assignment. Well, I made a very silly decision, and let @carlozancanaro talk me into doing an advanced project, which was building an introspection system for C++. Well, that was a bad idea, and we didn’t get very far. So, we scraped through with minimal marks on the assignment, and then played catch-up with the exam.

Anyhow, enough with the excuses already! I did enjoy the artificial intelligence course, which had two good assignments. Anyhow, I got a pass (info3220), a distinction (info3402) and two high distinctions. Exact marks are here.

It’s now July, and I’m currently working four days a week with CHAI, developing applications for two projects they’re working on—one is for Sydnovate, the other is a university project for a kinect-controlled display for the SIT building. I’m looking forward to next semester (with some hopefully better marks!), with the all-important project unit. ;)

Finally, I’m going to participate in NaNoWriMo, which means that in November, I’ll be trying to write a 50,000 word novel (in whatever spare time I have left lying around the place). Why, you ask? Well, I do a lot of programming. And I think that the best program is one with very little code which achieves a lot (e.g., Python & UNIX). Which means that most of my time is spent avoiding writing code. And then, when it comes around to documenting the systems I’ve build, I naturally tend to write terse descriptions of what it does, very factually correct and precise. BOR-ING. So one reason I’m participating in NaNoWriMo is to improve my documentation writing skills. But, the actual reason is far less complicated. I’m doing it because I think it’ll be fun. ;)

Anyhow, this is the public announcement so that I won’t be able to forget about it, or make excuses that “I don’t have time”, etc. :P

Semester 2 Results

2010-12-04T07:58:00+11:00

An update to my previous post - I did reasonably well this semester, with 2 HDs in COMP3109 (compilers) and INFO2912, a D in COMP2907 (algorithms), and a C in MATH2988 (number theory & crypto). I would have preferred a HD in algorithms, but I guess the exam brought it down. Also: yay, I survived maths again! ;) The complete gory details can be found here.

In other news, I’m enjoying my scholarship work with CHAI — I’ve already made something useful, and I’m slowly making progress. I do get the feeling that redesigning Cruiser will be a side project. ;)

I’m also excited about NCSS — I’m sure it’s going to be a heap of fun this (well, actually, next) year!

Box specific bar numbers in Lilypond

2010-11-05T04:30:00+11:00

Just a bit of Lilypond / Scheme I wrote last night — if you want to have boxed bar numbers at certain bar numbers (which don't fit every nth bar - see the docs for that), you can use this code to do it:

\override Score.BarNumber #'break-visibility = #end-of-line-invisible
\override Score.BarNumber #'self-alignment-X = #0
\override Score.BarNumber #'font-size = #2
\override Score.BarNumber #'stencil = #(make-stencil-boxer 0.1 0.25 ly:text-interface::print)
\set Score.barNumberVisibility = #(lambda (x) (not (eq? (member x '(4 9 16 25)) #f)))

This will put boxed numbers at bars 4, 9, 16 and 25.

Life update

2010-11-02T01:27:00+11:00

Hurrah, I’ve reached the end of semester! Unfortunately, this does not mean that I can stop working… sigh. Exams are coming up, speaking of which, grumble at the person who organised my exam timetable. :(

Here’s my predictions for marks: high distinction in comp3109 (a great course, btw), a pass or a credit for math2988 (given it’s advanced, and my past record of maths), distinction or high distinction in comp2907 (my assignment 2 was a little iffy, and we’ll see how the exam goes), and probably a HD in info2912 (provided that I get my reports done in time).

Now, if you’re currently reading the RSS feed (or Planet NCSS), you should definitely come and look at the fantastic theme I designed (and either admire it, or tell me that I should give up design for life!). Because I have plenty of work to be doing at the moment (info2912 reports, exam study, etc.), I decided to redesign my websites (the old theme was from OSWD, and I’d seen it on a few other websites). Fortunately, due to some software I wrote doing this sort of redesign is relatively painless, and doesn’t require remembering how Trac and Mercurial do templating. ;) I’d also noticed a few bugs with WSGIProxy, so I’ve written my own proxy and put it in Themere (see themere.proxy).

So, now that you’ve managed to survive reading through my rambling this far, I’ll continue rambling a bit more about what I’m going to be doing over the summer break. Item one is a Summer Scholarship with SIT, working on Surface Computing and Tabletops with the CHAI group, which should be fun.

I’m also going to offer to be a tutor at NCSS again. So, if you’re an Australian year 10/11/12 student interested in computing, go put in your application now! Who knows, you might even see me there. ;)

Semester 1 marks

2010-07-11T12:22:00+10:00

Another year, another set of marks — I’m quite happy with them, especially maths. I wasn’t sure about doing advanced level and didn’t think I did particularly well in the exam, so getting a Credit was especially fortunate.

So, here they are:

COMP2129 — Operating Systems and Machine Principles: 93 (High Distinction)
INFO2820 — Database Systems 1 (Advanced): 92 (High Distinction)
INFO2911 — IT Special Project 2A: 92 (High Distinction)
MATH2969 — Discrete Mathematics & Graph Theory Adv: 65 (Credit)

I went last week to ISCF Conference 2010, where not only did I have an amazing time and met some amazing people, but also got to do some user testing with Presenter! ;) The two things which were especially needed were backgrounds and keyboard shortcuts. So, today, with some very much appreciated assistance from Jamie, these features have been added! If you’re game, grab the latest version from the mercurial repository. Alternatively, you can wait until I release Presenter 0.8, which will hopefully be sometime in the next two weeks.

Semester 2 marks

2010-07-11T11:46:00+10:00

Avast! Here be (dungeons and?) dragons! Actually, it's just my marks for Semester 2 (which I've been rather slow in actually posting).

ELEC1601 - Foundations of Computer Systems: 90 (High Distinction)
INFO1905 - Data Structures (Advanced): 86 (High Distinction)
MATH1003 - Integral Calculus and Modelling: 71 (Credit)
MATH1004 - Discrete Maths: 80 (Distinction)
PHIL1012 - Introductory Logic: 92 (High Distinction)

A Merry Christmas to you all!

(on a side note, Presenter is coming along quite nicely, and has gained support for song lyrics and images - unfortunately, video support is still a little buggy...)

Mouse activity

2010-05-12T00:41:00+10:00

As seen on Planet Debian.

“IOGraph” is a small Java program that tracks and draws the status of your mouse over a period of time. Lines are movement, circles are places where the mouse has stopped… The bigger the circle, the longer it was there.

I suspect the large density of horizontal lines in the center is due to my habit of always selecting the text I'm currently reading. It's interesting to see the areas of the screen which I don't really use (yes, this only tracks mouse activity, but my eyes usually follow my mouse).

WSGI Resources

2010-04-09T01:02:00+10:00

Getting Started with WSGI
PEP 333, the official WSGI spec
Python Paste, a useful WSGI library

Tips:
- Start a PasteDeploy-based WSGI project: paster create -t paste_deploy $APP_NAME
- Serve the webapp: (Note: I usually rename docs/ to config/) cd $APP_NAME paster serve docs/devel_config.ini
If you're going the framework route, look at Pylons or Django.
If you're going the microframework route, you might be interested in Bottle.

Hello Planet NCSS!

2010-04-01T10:32:00+11:00

print "Hello Planet NCSS!"

Thanks to Tim for making this happen so quickly!

I want to stab this error in the face.

2010-03-27T03:13:00+11:00

The program 'presenter' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadShmSeg (invalid shared segment parameter)'.
  (Details: serial 351 error_code 158 request_code 132 minor_code 19)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the --sync command line
   option to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)

It's really annoying to have an application which does what it's meant to, but crashes all the time.

Mercurial hook for notify.io

2010-02-07T14:06:00+11:00

Update (9/2/10): Fixed bug when pushing multiple commits at once, silenced wget's output.

Here is a Mercurial hook to send notifications to notify.io whenever it gets new changesets pushed to it. It requires python, wget, bash, mercurial, internet access and an uncommon sense of humour (the latter is only needed for bug reports).

To use, put something like this into your hgrc (either for a specific repo, or user or system-wide):

[hooks]
incoming.notifyio = /path/to/script

And then put this into a shell script, where you referenced /path/to/script before:

#!/bin/bash

# User data (change this)

email="YOUREMAILHERE"
apikey="YOURAPIKEYHERE"

# Work out api url to send to.

emailhash=`echo -n $email | md5sum | cut -f 1 -d" "`
url="http://api.notify.io/v1/notify/$emailhash"

# Quotes urls.
alias quote='python -c "import urllib, sys; print urllib.quote_plus(sys.stdin.read())"'

# Get repo information
desc=`hg log -r $HG_NODE --template "{desc}"`
author=`hg log -r $HG_NODE --template "{author}" | sed 's/ &lt;.*//'`
text="$desc ($author)"

name=`pwd | awk 'BEGIN{FS="/"}{print $NF}'`

title="Mercurial changeset added"
if [ -n $name ]; then
 title="$title in $name"
fi

# text - The notification text. Only required field.
# title - The title or subject of the notification.
# icon - The URL for an icon to use for the notification.
# link - The URL for a link you want to associate with a notification.
# sticky - Set this to 'true' if you want the notification to be sticky.
# api_key - If you don't want to set it in the query string

text=`echo -n $text | quote`
title=`echo -n $title | quote`

data="text=$text&amp;title=$title&amp;api_key=$apikey"
#echo wget -O - --post-data $data $url
wget -q -O - --post-data $data $url

I take no responsibility for any loss of life, possessions or injuries sustained through the use of this hack. Please direct correspondence, thanks and hate mail to B.Posters, Devilish Lane, Cranbrook.

Productivity

2009-12-10T10:20:00+11:00

Any resemblance to reality is purely coincidental.

Test post

2009-11-11T03:31:00+11:00

Congratulations, you've won a prize! Nope...nothing to see here, move along.

Deliverance

2009-11-01T06:46:00+11:00

I've been trying out deliverance, and it is a really nice way of theming websites.

My motivation to use it is this: I want a website, which has various apps (in my case, Trac, HGWeb, SVN, and more), and I want to give all the pages a common theme. Instead of learning each of those application's theming languages (which would all be different, and some don't even have theme support), I want something to work as a "filter" at the web-server level...which is what Deliverance does, basically.

Unfortunately, it has a few things which annoys me. For one, it implements it's own web-server (deliverance-proxy uses paste.httpserver), and doesn't let you choose how to deploy it (e.g.: mod_wsgi on apache would be nice for me). Also, only half of it has been written as WSGI middleware (well, let's make that easily accessible WSGI middleware) - the bit which takes a theme and rules and transforms the content is available as deliverance.middleware.DeliveranceMiddleware. Unfortunately, link rewriting, response and request modifications aren't included in that, so you have to use deliverance-proxy to get them. :(

I've drawn two diagrams to illustrate my thoughts of how Deliverance should work, the first being how I currently understand it to be, and the second showing how I would like it to be. A disclaimer: representing Deliverance in a diagram was tricky, so it's not 100% accurate. Please don't take my lack of pedantry in drawing diagrams for a lack of understanding of how Deliverance works.

How deliverance works (according to me):

How I think Deliverance should work:

Some notes on things I'd like to see changed:

Deliverance becomes completely usable as a WSGI application - any server can run all the components of Deliverance, instead of just deliverance-proxy.
Deliverance wraps another application, which then does any proxying. This way, XML configuration files are replaced with Paste Deploy ini files (using URLMap and Proxy for previous behaviour).
Allow different rules to use different themes (I've coded up some preliminary support - see #17 for more details).

Links to stuff

2009-10-04T06:56:00+11:00

The Daily WTF failing, my iRobot Simulation winning, my iRobot program failing.

In other news, NCSS Summer School applications are open!

Update time

2009-09-22T01:06:00+10:00

Things I've done recently:

Participated in the ACM South-Pacific regional with Carlo & Ronny - we were the top 1st year team!
Wrote Huffman compression utility for INFO1905 assignment - Carlo & I worked out a really tiny way of storing the header. Now, it's just back to writing unit tests and documenting... (incidentally, bits & bytes in Java != fun)
Got an identi.ca account - getting real-time updates about stuff people are doing with Python is fun.
NCSS Challenge has officially finished - the Python streams finished a week ago, and the Embedded stream finished on Sunday, but there's an extra challenge for the Embedded stream - and a nice prize to got with it!
Change the name of this blog - again. We can't have it being consistent, can we?

Things to do:

Work on Presenter - this will probably be one of the things to do in my mid-semester break. The list of things to do has been slowly lengthening...
Complete everything on my TODO list... :(

Something I discovered today: ZIP File Quine. I'm wondering if I can do the something clever like this with my INFO1905 assignment... :P

Busy, busy, busy.

2009-09-01T00:12:00+10:00

Uni assignments, NCSS Challenge, math quizzes (urgh!), Presenter, and also ACM training. Wow there's heaps to do. I'm trying to uses Getting Things GNOME! to organise myself, and it works, but only when I use it. :( Clearly every organisation system should work even when I don't use it. ;)

In other news, I got myself an account on bitbucket (yay for OpenID!), and published Presenter on it. Thanks to that, I noticed that I hadn't updated the README before releasing 0.2. Whoops! I've also been thinking about the UI for adding / removing files from the runsheet, and I've got an idea which should be pretty, logical and intuitive - I just need to find the time to implement it! Also, a terminal handler will be coming as soon as I get some time... (which should be awesome for teaching Python with)

Also, PyPy is one of the more interesting things to compile, thanks to it's fractal drawing. I should be bothered to write one of them these days, shouldn't I?

The NCSS Challenge introduced an Embedded Stream, where the students get some modified Arduino boards to play with - lots of fun, especially the accelerometer. Unfortunately, the online marker is somewhat buggy and sensitive, so answering questions on the forum is usually not that much fun. :(

Uni is going fairly well, except for my Integral Calculus course. I had a rather odd assignment for INFO1905, where they were assessing graph-searching (disguised as a degrees of separation problem which needed Breadth-First Search), when they haven't even covered graphs in the lectures, let alone graph searching. I know we're an advanced class, but seriously - that just means we'll be able to learn faster, not know it already (even though I did...).

Finally, if you're using IDLE on Windows, note that len(string.lowercase) != 26, unlike every other Python environment on the planet. Stupid, stupid, stupid. Fine, we'll use string.ascii_lowercase instead.

Google Reader

2009-08-18T00:37:00+10:00

It's been nice knowing you, but I'm giving you up.

No more reading RSS feeds during my computer labs, no more worrying about whether Google is spying on me by working out what kind of subscriptions I read.

Goodbye Google Reader, Hello undisclosed client-side RSS reader.

Presenter Prerelease

2009-08-08T01:42:00+10:00

In the spirit of "release early, release often", here is another release of Presenter. This is an Alpha release, so it's still got plenty of bugs (the obvious one is with Video playback).

I've provided a .deb package, so if you're a Debian/Ubuntu user, it should be relatively easy to install. If you don't install any of the recommended packages (python-evince, python-gnome2-desktop, python-gst0.10), you'll find it rather boring.

There are two main problems with the .deb package: it explicitly depends on python2.5, and the icon for the menu item doesn't seem to appear unless you logout/login your session. These are known problems, directly related to how I created the package - I'm working on fixing them before the beta release.

Also, if you can be stuffed, it is possible to get Presenter working on Windows, and video playback works - you'll need PyGTK and PyGST (gstreamer bindings for Python).

You can get it from PyPI. Deb package Tar Gzip Egg Win32 Installer

Leaving Semester 1, Entering Semester 2

2009-08-01T00:43:00+10:00

So, I finished Semester 1, and got my results back. They're not brilliant, but they're enough...

Professional Engineering and IT: 71 (Credit) Fundamentals of Elec and Electronic Engineering: 59 (Pass) Differential Calculus: 71 (Credit) Linear Algebra: 74 (Credit) Informatics (Advanced): 95 (High Distinction)

And the subjects which I'm taking this semester are: Data Structures (Advanced), Discrete Mathematics, Introductory Logic, Integral Calculus and Modelling, Foundations of Computer Systems.

In other news, I set up Deliverance on aurora, make apache on aurora a lot faster, and also did some work on Presenter.

IWSGAY

2009-06-22T11:10:00+10:00

Oh, dear. Why did I waste my time with this, again?

Latex - inserting a box with no content

2009-05-04T06:36:00+10:00

Or, an empty box! Working out that query on Google took me some time, so I'll put the answer here:

\fbox{\begin{minipage}{5in}\hfill\vspace{3in}\end{minipage}}

RealLife (tm) Update: I'm enjoying University, and I'm thinking of moving into C/C++ via Vala.

Update on Presenter

2009-04-18T02:23:00+10:00

I'm building a different, better idea for Presenter.

Details here: http://flowblok.selfip.net:8001/trac/presenter

How to do plugins using eggs

2009-04-17T04:57:00+10:00

(Just so that I don't forget, and because some of you may not have seen this before.)

This is the magic Python code which automagically gets eggs with the entry point "my.plugin.namespace" declared.

for egg_plugin in pkg_resources.iter_entry_points('my.plugin.namespace'):
    print egg_plugin.name

New Machines (aka my interwebs are slow)

2009-04-13T03:47:00+10:00

So, 2 weekends ago, I got a new computer! It's been named "polaris". It was built from parts by my brother (since I don't generally like descending down into the hardware level).

Stats: Processor: 2x Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz Memory : 4GB (but not being utilized, as I'm using the wrong kernel atm) Operating System: Debian GNU/Linux squeeze/sid Display Resolution: 1680x1050 pixels

My mother also recieved a similar machine (3.16 GHz processer instead), and is running Windows XP Professional. (It's name is "m2", as her old one was "m1") I also took what I could from my mother's old machine, and put them into my old machine, and named is "aurora". (answer to last post: it's my new development server)

Unfortunately, this meant installing 3 operating systems from the internet (XP was CD-based, but it needed SP{2,3} and all the various upgrades). Also, I didn't think to download a real Debian CD/DVD (instead of my netinst install CD), so it also meant downloading 3 operatings systems.

Thus I am currently on dial-up speed internet.

Oh well, at least I've got a development server to play with (I'll set up version control stuff, Trac, and anything else I can think up).

Hey look - I've deviated from the original point of this post, so I'd better stop typing soon - but, before I go, which of these do people think I should work on:

Improved Sprite and Scene System: on http://pygame.org/wiki/gsoc2009ideas (I can't do GSOC this year, due to the age limit) - it looks like I could rewrite Bezel Engine to become part of PyGame. (which would be awesome)
Presenter - the control freak's presentation tool.
Web development - make my super-awesome WSGI app for developing web applications (yeah, sounds recursive). (and hard!)

So people, tell me what I should focus on - I'll continue doing small disjointed development on all of them for the moment, but if people tell that they really want to see me make one of these, I'll put a lot of time and effort into it.

Erm, whoops!

2009-04-12T22:32:00+10:00

flowblok@aurora:~git/cgit-0.8.2.1 Z make
zsh: command not found: make

Clearly, I should have installed build-essentials a long time ago! ;-) More about what this "aurora" machine is later.

Presentation software implementation

2009-02-24T05:51:00+11:00

WARNING: This article talks about pre-alpha code (i.e.: still in development). Use at your own risk.

Just over a week ago, I posted my idea for presentation software.

Since then, I've put together a PyGTK[1] application for this purpose.

Features:

Presenter console: Provides a window for easy navigation of slides, just by double-clicking.
Presentation window: Dual-head support, with monitor detection and switching at runtime.
Application: Keyboard shortcuts for all features.
Document format: ZIP file of images for each of the slides.
Document format: PDF - linux only, using the python bindings for poppler (Debian/Ubuntu: python-poppler)

I've uploaded it to the Python Package Index: http://pypi.python.org/pypi/Presenter/. Please be aware of the dependency on PyGTK (Debian/Ubuntu: python-gtk2)

Let me know if you have any installation problems (relating to my software, not it's dependencies), and any bugs you encounter.

Also, I haven't quite put a license on it definitively for the final release, however, this particular release is licensed under the terms of the GNU General Public Licence, version 3.

[1] Sorry to those of you who don't like GTK, or use Macs - whilst I would like to support you, this is just a prototype with a GUI toolkit I know.

Presentation software idea

2009-02-13T10:47:00+11:00

I now very much dislike how OpenOffice.org Impress was built in Java.

It's slow to load.
It's unresponsive.
It has an amazing ability to crash, on the most simple actions.

However, I do like the OpenDocument format. Also, presentations are to be presented, not just to be edited. And everyone presents with dual-monitor setups nowadays, and if not, they should.

So, I want presentation software which:

is Free Open Source Software.
works cross-platform.
reads in .odp (OpenDocument Presentation) files
displays a controlling window on one desktop, and a viewing window on a second desktop
allows for non-linear viewing of slides (can switch to any slide at any time)
(future) allows for editing / creation of slides on-the-fly
(future) allows for a presentation to be made up of various presentation files, added and removed on-the-fly.

The first step would be to just write a viewer, and the second step would be to write a nice editor. Then I wouldn't have to deal with so many program crashes.

Incidently, this is not intended to replace OpenOffice.org completely. Impress has various features which I have not addressed here, the point of this program is to have something which even your mother could use to do a presentation safely.

In terms of implementation, it doesn't really matter, as long as it is responsive to users, and stable. If I get around to implementing it (after some of my current projects are out of a development stage), then I'd probably use Python, wxPython and odfpy.

University

2009-02-06T22:43:00+11:00

By the way, I have enrolled at the University of Sydney for a Bachelor of Computer Science and Technology.

But you knew that already, didn't you?

A magical network client

2009-01-20T00:27:00+11:00

Game Engine:

from bezel.networking.engine import NetworkedGame, exposeclass GameEngine(NetworkedGame):
    def __init__(self, *args, **kwargs):
        super(GameEngine, self).__init__(*args, **kwargs)
        self.count = 0

    @expose
    def say(self, something):
        return 'Simon says "%s"' % something

    @expose
    def add(self, num):
        self.count += num
        return self.count

Client:

#!/usr/bin/env python
from bezel.networking.client import JSONGameClient
client = JSONGameClient('', 8602)
engine = client.engine
print engine.say('blah')
print 'count + 1 =', engine.add(1)

Missing is the network server (the code for that is fairly standard, but I've changed a few things, which may confuse people) - suffice to say that it's a threading TCP server with automatic Avahi publishing.

Example usage (from the root of the bezel-engine repository):

$ demos/network/main.py &
[1] 9422
$ demos/network/client.py
Simon says "blah"
count + 1 = 1
$ demos/network/client.py
Simon says "blah"
count + 1 = 2

Not yet done:

Load testing for the server.
Client (player) authentication.
A class to discover the servers through avahi.

Life update: Main round offers are coming out tomorrow, so I'm hoping to get one for Sydney Uni. It probably won't be a BIT or a BCST (Adv), but as long as I get in there and do well, I should be able to transfer fairly easily.

class FailEpicallyError(Exception): pass

2009-01-16T01:56:00+11:00

Error #1: Me failing epically at recursive callbacks:

RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeError^C
Traceback (most recent call last):
  File "src/bin/ua-sidebar", line 71, in do_forall
    def do_forall(self, internal, callback, data):
KeyboardInterrupt
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError^C
Traceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
^CRuntimeErrorRuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeErrorRuntimeErrorRuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 71, in do_forall
    def do_forall(self, internal, callback, data):
KeyboardInterrupt
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeError^C^C^C^C^C^C^CRuntimeError
RuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 71, in do_forall
    def do_forall(self, internal, callback, data):
KeyboardInterrupt
RuntimeErrorRuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeErrorRuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeError^CRuntimeErrorRuntimeErrorRuntimeError
RuntimeErrorRuntimeError^CRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeError^C
Traceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeErrorRuntimeError^CRuntimeErrorRuntimeError
RuntimeErrorRuntimeErrorRuntimeError^CRuntimeError
RuntimeErrorRuntimeErrorRuntimeErrorRuntimeError
RuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeErrorRuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeErrorRuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 71, in do_forall
    def do_forall(self, internal, callback, data):
KeyboardInterrupt
^CRuntimeErrorRuntimeErrorRuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 71, in do_forall
    def do_forall(self, internal, callback, data):
KeyboardInterrupt
RuntimeErrorRuntimeErrorRuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeError^CTraceback (most recent call last):
  File "src/bin/ua-sidebar", line 75, in do_forall
    for child in self.get_children():
KeyboardInterrupt
RuntimeError^C^C^C^C^C^CRuntimeError^Z

Error #2: My shell stopped responding, so I drew some artwork in it.

^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^Z
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^Z^Z^Z^Z^Z^Z
^C^C^C^C^C^C^C^C^C^C^C^C^C^Z^Z^Z^Z^Z^C^C^C^C^C
^C^C^C^C^C^C^C^C^C^C^C^C^C^Z^C^C^C^C^C^C^C^C^C
^C^C^C^C^C^Z^Z^Z^C^C^C^C^C^Z^C^C^C^C^C^C^C^C^C
^C^C^C^C^C^Z^C^Z^C^C^C^C^C^Z^C^C^C^Z^Z^Z^C^C^C
^C^C^C^C^C^Z^C^Z^Z^Z^Z^Z^Z^Z^C^C^C^Z^C^Z^C^C^C
^C^C^C^C^C^Z^C^C^C^C^C^C^C^C^C^C^C^Z^C^Z^Z^Z^C
^C^C^C^C^C^Z^C^C^C^C^C^C^C^C^C^C^C^Z^C^C^C^Z^C
^C^C^C^C^C^Z^C^C^C^C^C^C^C^C^C^C^Z^Z^C^C^C^Z^C
^C^C^C^C^C^Z^Z^Z^Z^Z^Z^C^C^C^C^C^Z^C^Z^Z^Z^Z^C
^C^C^C^C^C^C^C^C^C^C^Z^Z^Z^Z^Z^Z^Z^C^Z^C^C^C^C
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^Z^C^C^C^C
^C^C^C^C^C^C^C^C^C^C^C^Z^Z^Z^Z^Z^Z^Z^Z^C^C^C^C
^C^C^C^C^C^Z^Z^Z^Z^Z^Z^Z^C^C^C^C^C^C^C^C^C^C^C
^C^Z^Z^Z^Z^Z^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
^Z^Z^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
^Z^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C

HSC Results & UAI

2008-12-17T22:30:00+11:00

Units	Course Name	Examination Mark	Assessment Mark	HSC Mark	Performance Band
2	Business Studies	79/100	84/100	82	5
2	Chemistry	72/100	69/100	71	4
2	English (Advanced)	76/100	81/100	79	4
2	Mathematics	92/100	87/100	90	6
1	Mathematics Extension 1	46/50	40/50	43	E3
2	Physics	74/100	72/100	73	4
2	Software Design and Development	95/100	94/100	95	6

And my UAI is 88.75.

Contrast in Gmail Planets Theme

2008-12-04T03:58:00+11:00

If you're reading this on an LCD screen, you may need to tilt it (or move your head) to see this clearly.

You're doing it wrong!

2008-12-01T23:07:00+11:00

And I apologise for the correct spelling and grammar of the title.

On his blog, Brian Jones points out (amongst other things) that you shouldn't use Excel (or any other spreadsheet product) as a replacement for a real database.

But this isn't all that is wrong in the world:

Don't use one single big table in Word - using Excel or another spreadsheet program.
Don't use absolute positioning in Word - just get some real Desktop Publishing Software!
Don't use Access for website databases. Here's one which I have done, and the reason why it's a bad idea is because it means reading a file (often a big file) from the filesystem, typically every page request. It's also near impossible to use it reliably on a platform other than Windows.
Don't use any presentation software, especially Powerpoint, unless you know how to give an interesting talk without it. (I might talk about presentation software and presentations on this blog in the future - stay tuned!)
FrontPage is still considered by some as a real HTML editor. Don't use it! It was bad when I first used it, and it hasn't changed. I still write HTML by hand, and when I need a graphical HTML editor, my preference is WYMeditor.

There's certainly other failures in this area, and we could spend all day talking about failures of software, but I think we've all got better things to do, don't we?

Oh, and this isn't all Microsoft's fault. I'm just using their product names as they happen to be the most common nowadays. These faults are just as easily replicated with Free Open Source Software.

Programming Meme - Fantasy VM 2

2008-11-26T01:53:00+11:00

Those of you who participated in the NCSS Challenge this year, or attended NCSS at the start of the year may have encountered Fantasy VM, a rather silly Virtual Machine with direct hardware support for variables!

Some people have started a programming meme, so I thought I'd join in.

For this program however, it soon became apparent that the simple Fantasy VM would not come up to the job, as it had no support for strings. However, with a little bit of hacking, I've created Fantasy VM 2! The specifications for the VM are similar, it's fully backwards compatible, but adds a few instructions to make working with strings possible. The silliest part which I've added is the support of the ADD operator for strings, but considering that it's a very silly language, I'm regarding it as a feature, not a bug.

So, here's the program in FVM2:

SET name_prompt "Please enter your name:"
SET age_prompt "Please enter your age:"
SET counter 0
SET test 0
SET one 1
SET start ") Hello, "
READ$ name name_prompt
READ$ age age_prompt
INT age age
SUB age age one
SUB test age counter # pretest
JGE test 13 # skip
JGE one 19
STR counter_s counter # skip
ADD greeting start name
ADD greeting counter_s greeting
PRINT greeting
ADD counter counter one
JGE one 10 #pretest
NOP

And here is an interpreter for the language, written in Python. To run, just save the above program to a file, and call "python fvm2.py filename.vm".

After the HSC

2008-11-18T00:41:00+11:00

It's been a while since my last HSC exam, and I thought I'd blog about what's been happening since.

I've got my Year 12 formal tonight, which promises to be a heap of fun, lack of sleep and hopefully provide enough blackmail material for the rest of my classmates for the rest of the century! (mwahahaha!) Or maybe not. :(

I've been working on my application for the Microsoft Research Asia Scholarship in IT for the past week or two (and been thinking about working on it for far longer than that). Writing documents isn't my favourite bit of work to do, so it's been slow work, but hopefully I'll be shortlisted and get an interview.

I had thought that my desk would get a lot cleaner after I got all my school stuff of it. I really should have known better. Maybe it'll get cleaner once I get all my scholarship stuff of it. I doubt it.

On the programming front, I've been working on Universal Applets, the successor to Screenlets. I can't say the code's nice, the style seems to be half C and half BASIC, but I'm working through the core modules, and cleaning as I fix stuff. We should have a release out next year.

And on a completely different note, we've got a large pile of mulch in our driveway. I wonder what people walking by think of it. It'll take a while to move into places around our backyard, so I might get a photo sometime in the next week and post it.

I'm going off to finish my application. Also, some of my future posts may disallow comments, depending on whether I want commentary or not. I'm also going to rename the blog sometime within the next month or two.

Google is reading my mind?!

2008-11-14T08:32:00+11:00

I wanted to look up the past tense of "lead" (not Pb) today, and so typed it into the search box on my Google homepage. Those of you who have been paying attention will have noticed that autocomplete is now more promintent among the Google services, so I wasn't suprised by the little suggestions box appearing below my search.

What caused me the shock was that the first item in the list was "past tense of lead" - after only typing "past tense "!

I'm wondering if we're going to be seeing "Google MindReader" coming out soon... maybe that's what the guys who made MentalPlex have been working on...

HSC is finally over!

2008-11-06T09:03:00+11:00

YAY!!!!

(Yes, that's all.)

kthnxbai

Software which makes my life easier

2008-11-03T09:38:00+11:00

Firefox 3.0

The super-duper cool address bar - search within URLs and titles. I think that it's sorted by a 'popularity', based on if you've bookmarked it, and how many times you've visited the page.
Save tabs. If you've got a window with multiple tabs open, you can save them so that you get them opened when you next start Firefox. This feature was originally just a crash recovery feature, and I started using it by killing Firefox, and then asking it to restore my tabs, but now it's a proper feature. It's so much easier to save the tabs, shutdown my computer and go to bed, wake up the next morning, and open Firefox ready to continue reading whatever blog/technical manual/story/whatever I was reading last night.
All the plugins. Everything from Ubiquity to TabRenamizer (yes, I am paranoid).

Xournal

Take a look. Annotation support may come along sometime for Evince, but until then, I've been using this to annotate PDFs. Just it's PDF support and highlighter make it a worthwhile application.

OpenOffice.org

Sure, it's not perfect: there's a heap of stuff which it icky to use, and there are plenty of features which I never use.

But the main features which I use are generally a lot better than Microsoft Office. Formatting is actually nice to use - it uses hierarchial styles, so to change all your body text to Comic Sans MS means only changing the "Default" or "Text Body" style, depending on how much of your text you want to change.

I've already blogged about the Presenter Screen, and it's a great step forward.

Workspaces

I generally have at least 2 windows open at a time, and generally they correspond to two different tasks. For example, I might have Pidgin and OpenOffice.org both open - I'm chatting with a friend in one window, and typing an assignment in the other.

These tasks have nothing to do with each other, so why should they be grouped together by my computer? Workspaces (called Workspaces in Linux, Spaces in Mac OS X Leopard, and Virtual Desktops in Windows) let you put windows into seperate areas.

Less than 90x20 pixels of screen space on my computer to show me a representation of my workspaces is a wonderful trade-off for the ability to organise my windows in a logical and sensible way.

Wrapup

So, there's a few applications which makes my life easier - feel free to mention any which you use regularly in the comments.

I might expand on some of the smaller little 'bits and pieces' of my desktop later in another blog post, but for now here's two - iGoogle and Mail Notification.

I've got 2 HSC exams tomorrow: Physics and SDD, and then Chemistry on Thursday. It's still a little scary to think that in 3 days it'll all be over!

OpenOffice 3.0 available + HSC Progress

2008-10-17T06:57:00+11:00

http://www.openoffice.org/news/ http://download.openoffice.org/

Get it while it's hot!

On another note, my Business Studies and English Paper 1 have been done - B.S. went fairly well, I'm not so sure about English.

Analysis of the HSC courses

2008-10-10T03:25:00+11:00

See my last post for a link to my full data set.

Here's a short list of the HSC courses (Male subjects at top, Female subjects at bottom).

Software Design and Development
Physics
Information Processes and Technology
Mathematics Extension 2
Mathematics Extension 1
Chemistry
Mathematics
English (Standard)
Modern History
English (Advanced)
Biology
English Extension 1
English Extension 2
Drama

The most interesting trend to me is the English / Maths trend. As you move up the Extension courses, Maths moves towards more males, and English moves towards more females. Also, the sciences are spread throughout the list, which is interesting, because at my school I haven't noticed any major differences between gender ratios in science classes.

And finally a note on my HSC study - it was going well until I opened my computer. :(

From the depths of HSC...

2008-10-03T10:01:00+10:00

Study, study and ... ohnoimnotdoingenoughstudy ifeelreallyguiltyaboutplayingalltheseflashgames damnyourandallgetoutofmyheadand oohlookuniversalappletsisgoingtobereleasednextweekand oohbugsformetosolveand gameenginestowriteand...

So, if you managed to survive that, allow me to present the last half hour's efforts: Male / Female ratios of the HSC courses. Source is the BOS' media guide. Highlighted rows are merely courses which were relevant to me (i.e.: I have a friend who does course x).

And my SP3 update just failed because "Permission Denied" (Yes, I am logged in as an administrative user). It's nice to know that Microsoft didn't give themselves permission to run an update on my system when I ask for it. Ahhhhh. That'll show them who's in control!

Freezing in Sydney

2008-08-07T05:03:00+10:00

I've suspended my quest for a Python website system, in favour of Drupal. Drupal is an free open-source website engine. It's a modular design, so you can grab a .tar.gz or similar from their website (or anywhere, really), extract it to the modules (or themes) directory, and then run the update script.

The only things I'd like to see in it are an automatic updater (so that I don't have to FTP my files to my server), and some pretty looking graphs.

You can take a look at it on a site developed for my local concert band. Until I find something this good with Python, Drupal will definitely be one of my 'liked' bits of software.

On a completely different note, my HSC trials are almost over (despite the fact that we're just less than halfway time-wise), with Maths Extension 1 tomorrow (oh dear...), and Physics next Wednesday (meh!).

And as to the title, when I started to type this post, it started to hail. And of course, all my Trial exams are in our school hall, whose temperature is proportional to the number of people inside the hall. (Yes, it has heaters, but since when did that matter.)

Programming Competitions

2008-07-27T07:45:00+10:00

We got through to the finals of ProgComp (results here), which I was very happy about, and the NCSS Challenge is starting tomorrow. Yay!

Also, I've been trying to embed a Python Console into presentation software. I've spent most of today creating a Java applet which runs the Python process and has a terminal applet to interact with it. The brick wall here has been OpenOffice.org, which refused to let me insert a Java applet. Looking around, there's a bunch of QA issues with this, so I suspect that it just doesn't work. Looking at my OO.o 3.0 release for Windows shows that they've removed it from the GUI, so no luck there. I'll see how I go with NeoOffice tomorrow.

Note: I saw Chris Neugebauer's excellent use of Bruce to achieve this, but I would prefer it to be done in OO.o/NeoOffice, because it's already installed on my home computer and the computers I'm presenting on.

And lastly a message to Chatswood High students: if you're interested in programming, and you'd consider doing the NCSS Challenge, which doesn't require you to know anything about programming, please come along to Ms Ho's introduction session on Monday lunch (Room 43 or 46), and/or my training session Friday after school (Room 43).

FRisT P0S+!!!1!!!1!

2008-06-20T08:13:00+10:00

Sorry, I couldn't resist that one.

I'm just trying out how to blog using gnome-blog. It lets me type my blog posts from my panel. It's not as pretty as using the wordpress html editor, but it's a lot simpler and more responsive (on my slow, low memory computer).

Presenter Screen in OpenOffice 3.0

2008-05-25T07:54:00+10:00

Last night, I downloaded the beta of OpenOffice 3.0 (+ en-GB language pack). Then, I got Presenter Screen, and installed it all.

The result is, that OpenOffice 3.0 has a super-cool presenter mode that surpasses (IMHO) Microsoft's Presenter Tools for PowerPoint.

The extension works whenever you run a presentation with multiple monitors, and it displays a console on the main screen for presenters to use. There are 3 views:

Show large previews of the current and next slide.
Show small previews of the current and next slide, along with the (scalable) notes for the slide.
Show all slides, which can be clicked on to switch to any slide.

All 3 views also display "# of #" for which slide you're on, the current time (for knowing when morning tea is), and the time since the presentation was started (for knowing how far overtime you've gone).

Also provided is a small help window, which details all the keyboard shortcuts. There really are some insightful ideas for these - typing a slide number then enter (e.g: "20[Enter]") will switch you to that slide number.

Ever since I came across the notes feature in PowerPoint, I wondered what the point of it was, since you'd have to print it out anyway. The 'Notes' view in Presenter Screen is great, since you can see the notes for the current slide, and zoom them so that you can stand far away from the laptop (and secretly read your speech out!)

However, the feature which really makes me love this is the 'Slides' view. I produce presentations for church services every Sunday, and there's a high probability that the order of events which was given to us on Saturday is completely wrong, and songs, talks and other presentation stuff will get randomly swapped.

In order to manage this confusion, I wrote 3 macros which went onto buttons in PowerPoint 2000, NextSlide, PrevSlide and ChangeTo (essentially Int(InputBox()) ), and then used the Slide Sorter view to monitor slides.

With Presenter Screen, this task is made so much easier, as it's faster (my macros took 1/2 second to actually do stuff), and prettier (always a good thing!). The only problem I've found has been losing the ability to edit slides on-the-fly. I can still switch back to my main window, but OpenOffice doesn't like me editing while in Live Mode. Fortunately, though, I rarely need to do this, and so the benefits out-weigh the cons.

Main Presenter Screen:

Notes View:

Slides View:

Music players for GNOME

2008-04-16T23:20:00+10:00

Apologies for the short sections, and weird layouting stuff that you don't usually find in blog posts. Or maybe not. After all, this isn't a blog post!

Introduction

After installing Debian Etch on my computer, I started looking at music players for GNOME. The default for GNOME is Rythymbox, however, because it was so featureless and ugly, I also tried out a few others.

Later, I upgraded to Debian Lenny, and GNOME 2.22. Yay! So, I then had access to more music players, and I launched a second investigation.

The music players I looked at were Banshee, Quod Libet, Exaile, Listen and Muine.

Analysis

In no particular order, here is my thoughts on each of these players.

Exaile

Exaile claims to be Amarok, but for GTK. It appeared to be a good idea, and it's interface was pretty (though perhaps a little too pretty for my liking), and is a good player to try out. However, it didn't appear to play Last.fm streams.

Quod Libet

This was my first choice after discovering the problems in Rythymbox, and is a good organiser of music, and a generally good player. I've only had a few crashes with it, which I believe to be the fault of it's notification icon. However, it also didn't have Last.fm stream integration.

Listen

Listen had a lot of the features I wanted, but it's GUI didn't feel nice enough for me, and it's Last.fm stream support was dodgy.

Muine

Muine was too featureless for me, and crashed whilst importing my music library. Repeatedly.

And the winner is...

Rhythmbox!

Wait a moment... but didn't you just say that it was featureless and ugly?

Well, yes, but that was before GNOME 2.22. It now is a really nice player, which manages to import my disorganised music library with no problems, and has support for things like Podcasts (which I don't use), and it's Last.fm stream integration is almost always brilliant. (Occasionally it will fail to get the song metadata for the current song)

It hasn't crashed, and is part of GNOME, so I'm happy.

The future?

Well, the more astute of you will have realised that I've left out Banshee. That's because it deserves a special section to itself. Banshee was able to do all the things that I've mentioned about Rhythmbox, including the nice GUI (although it's Last.fm stream integration is a little more dodgy), and it plays video!

So, I think that it's very possible that Banshee will take on the features from Rythymbox, or vice-versa, and one of them will emerge as the winner of the GNOME desktop music player.

Which one? How would I know?!

Exams - Almost the end

2008-04-05T10:02:00+11:00

My exams are almost over - done all except Physics. Apart from Chemistry and Maths Extension 1, I reckon I've done fairly well.

I've been generally unable to work on my Python web framework-like-thing, but have been able to have a think about some ideas for changes.

This isn't really a post (like this really isn't a blog) - I'll have a little more time to write something useful next week. Yay for my 4-day weekends!

π and a T-shirt

2008-03-14T05:30:00+11:00

Well, today was π day! And for those of you who can't see that symbol, get some better software! (Of course, it could be my encoding or something useless, but I doubt it.)

So, what did I do? Not much. I've got a teacher involved, so that when we get to 22/7, we'll do something. As I said, don't worry about missing pi day, the 22nd of July is close enough! (Insert groans here)

Also, when I got home from school, I found my T-shirt from Google - I participated in the GHOP contest, and just told them where I live, my bank information (just joking!), and they sent me out a nice T-shirt for taking part in the contest - pretty good deal, if you ask me!

Robin Hood - A Critisism

2008-03-13T05:39:00+11:00

Those of you who have been watching ABC1 TV on Sunday nights will be familiar with the BBC production of Robin Hood. I've been rather impressed with the originality of the program, and I think that it's been well-adapted, even if the episodes do have the unfortunate effect of almost always ending how they started.

However. The last episode (Sunday 9th March), which I believe was titled "For England!", had a rather interesting point. Robin sneaks into the Sheriff's meeting with the Black Knights, and throws daggers at each of them, only to discover that they have nice little blocks (of wood?) protecting them. Funny that they predicted where he'd throw the knives, and that they knew that he would use knives. He could have shot each of them in the head, and so on.

But my main problem is when Robin managed to take the clothes off one of the Black Knights. How come, when he stripped the clothes off himself, did he miss noticing the block of wood? Surely he would have noticed it when taking off his outer clothes! In fact, the whole scene felt a little too much like "something happens, *magic occurs* and everything has just been reset".

That, is my main problem with episodal (my word for shows which are based around episodes) TV programs - they always finish off where they started. I've seen it in Doctor Who*, Firefly and now Robin Hood. It's really getting quite annoying.

* I haven't been too annoyed with the last 2 series of Doctor Who for this reason, but most of the earlier episodes had this problem. Mind you, my problems with the new version of Doctor Who could fill up another few posts!

Spellcaster Documentation

2008-02-26T05:36:00+11:00

Spellcaster Documentation for D&U:

Project Statement (done)
IPO Charts (done)
Storyboard (done)
DFDs (done)
Screen Designs (done)
Limited Prototype (done)
Data Dictionary (done)
Gantt Chart (done)
Hierarchy Chart (done)

As you can see, this isn't a really verbose post, but enough to see where I'm at, struggling through the uselessness of SDD documentation.

[Edit]Sorry about the abbr tag left open! It's now fixed.[/Edit]

SDD Walkthrough

2008-02-13T07:16:00+11:00

So I did my SDD Walkthrough for Spellcaster today, and passed successfully along with (I believe) all the other members of my Software class.

I've already done the Initial Prototype, so that's one step from D&U out of the way, but there's all the nasty documentation to get done …

PCMS Update

2008-02-11T08:06:00+11:00

Well, development of PCMS is coming along slowly but I'm quite happy with progress.

I've separated my modules and templates, which would make it feasible in the future to change to a different templating system.

Each module is responsible for a XML node, which itself can include subnodes (i.e. other modules), or just data in it's attributes.

My current templating system is based on just matching each tag name with a template in the current theme's folder. It was originally designed to be used with XSLT, but since Python 2.4/2.5 doesn't come with a XSLT processor included by default, I decided to seperate the templating side of things, and let it use whatever system you decide to implement.

This change was responsible for breaking most of the code in PCMS, which has caused some backtracking, but it's not that much, and the migration to the new XML system is really quite easy to implement.

--Update--

Just tonight I finished off support for viewing articles (which are generic posts/information bits), and also added in the support for styles (which are stylesheets used by themes). Next is moving the styles into the themes folder.

Once again, if I'm online, you can look at my development version.

Today's lessons

2008-02-04T08:51:00+11:00

Well, today I did a number of things.

One useful thing was giving a mini-lecture on CGI (with a direction towards Python), and Python 3k to my software class. They really don't understand the usefulness of one-liners in programming. Hmm. Maybe 'usefulness' is the wrong word.

In other news, I've named my 2nd USB disk partition "The Horse". It makes absolutely no sense unless you mount it (Massive groans from the not audience). (1st partition is a persistent filesystem for my Ubuntu Live CD - it still is not working well, but I haven't really tinkered with it yet)

And today's final topic is my software project, of which I have to present a practical way of implementing my game in a 10 minute interview with my teacher. Unfortunately, our computing rooms are exclusively Mac OS X, and thus I have a limited idea of how to demo my work on them. I know that I can use PyGTK on Windows & Linux, but unfortunately the only real way for Mac is to rewrite the GTK libraries wrapping Tkinter. Suggestions welcome!

And I'm going to try and link up this not blog to some blogs of friends & people I know.

Projects

2008-02-02T05:49:00+11:00

OK, here's an update on some of my projects.

I'm currently working on a Python-based CMS. It's going slowly at the moment, due to my slow algorithm design. The latest version is available on my private server, which probably isn't online at the moment.

I've also started work on my HSC SDD personal project work, which is a PyGTK implementation of Spellcaster (aka Waving Hands, Firetop Mountain, Warlocks). I highly recommend you take a look at one of the implementations (Warlocks is the latest), and have a try of the game.

Not a first post

2008-02-02T04:50:00+11:00

This is obviously not the first post of my ^(blog). It's where I say ^(FRSIT!!111!!!!!!!)

So. Firstly, about myself. If you're not aware, my name is Peter Ward, and I'm a (coder|programmer), who went to NCSS 2008 recently.

I like w3c valid XHTML / CSS, and have a problem with my "x" key on my keyboard, which makes typing words with "x" very difficult.

Rumours that I am starting a blog are untrue and completely unfounded.