Urgh, so it turns out that the webmentions plugin doesn't work with github.

Good enough reason to migrate to self hosted ASAP.