How do you add users. Are you using LDAP integration? Do you use shared groups, so users get the needed contacts automatically? Or do your users add them manually?
If you are using LDAP, maybe you are hitting this bug 
OF-830 . There is no solution so far, other than going back to 3.9.1 version. Though Steve seems to have found a possible cause (weird characters in the names of users in LDAP, he explains his findings in the comments).