The answer to your first question is basically the answer to your second question with associated risk analysis (i.e., if yes, is it worth it?).
So to answer your second question, yes it does seem likely that most plastic containers release compounds that mimic estrogen under some conditions. Furthermore, many of the compounds that are released from plastics have not been fully tested, so the absence of specific information on their risk is not evidence of an absence of risk.
Since it does appear that these plastics can release harmful compounds under certain conditions then the answer to your first question of is it safe, boils down to "are you replicating an environment where enough endocrine mimics are released to cause harm?"
Given the lack of research on all of the ways in which plastics leach and the fact that there is evidence that endocrine disruptors do not follow normal dose-response curves, I personally would conclude that the risk exceeds the inconvenience of having to find a non-plastic container to transport your tea.