I figured this out by reading a bit more on Google and the forums here. Here is an answer to my questions --
There are 2 things to configure correctly --
1. Server Name: Each server name in the hazelcluster should have the same server name, from my earlier example - chat,domain.com. This is also the same name as my ELB under DNS and this is the same name the user will use.
2. Hostname: This is the backend server names. From my example server-a.domain.com and server-b.domain.com.
So, as long as all openfire servers have the same "Server Name", you can use this name as a single openfire instance. Adding users at user@chat.domain.com works too. No matter if users are connected to server-a or server-b, users always appear as user@chat.domain.com.
Anand