Additional info:
I traced the error to the file /openfire/plugins/admin/webapp/WEB-INF/lib/admin-jsp.jar.
The first time I copied it from my backup, it works fine. But the second time I copy it, it doesn't work.
If I take a fresh copy of OpenFire 3.9.1, and use the file from that package, then it works fine, no matter how many time I copy that file.
I'm thinking it might have been an issue with downgrade from 3.9.3, and that I copied the whole plugin folder from there, and received that file, but it seems strange then that it would work the first time it got copied, but not the second time.