Yes it work. I've had to implement SaslServer, SaslServerFactory and Provider. Then using `Sasl.addProvider()` and `SASLAuthentication.addMechanism()` and everything works fine. There is even complete error handling implemented in Openfire for custom mechanisms, I don't have any idea why these fixed-code check is added there and there seem to be no reason why not removing it.
I've added the whole stuff to a plugin and tested dynamic adding/removing for enable/disable plugin too. I've also checked how the server behaves when using non-existing (=auth failed+one INFO debug line of someone tried to use XYZ for auth), broken (=auth failed+stack trace in log) and blocking (ends in timeout for the client session) auth mechanism and everything seems to get handled like I would have expected it.
When implementing a custom SaslServer the easiest way is to look how ClearspaceSaslServer is implemented and do exactly the same.
But since I don't like to use customized Openfire versions and instead like to use plugins for such stuff I hope the check will get removed in the next update of Openfire since there seems to be any reason anyway.