While adding to your My Links from the Portal or HR portal and error would popup stating 409 Conflict error. I modified the Alternate Access Mappings to point to http for the default and https for Intranet. This seemed to resolve the issue.

Thanks to one of my Colleague for this one. Discussing with him helped me research why it worked from the Groups site and not from the others