I found out that some MySQL settings really change Openfire behavior. I set innodb_buffer_pool_size = 800M and since that Openfire uses only 60-65% of its memory with 5000 concurent users.
I can't increase heap - the server has only 4GB of RAM.
We're going to migrate to Ubuntu.