This was interesting and I must read in more detail:

https://www.sciencedirect.com/science/article/pii/S0006322314007094?via%3Dihub

Essentially
longitudinal imaging in MtF and FtM under HRT. I _think_ what they
conclude are testosterone increases serotonin binding (so goes up under T
HRT and down on anti-androgens) which may affect mood. Sufficient E
supplementation may counter, meaning in some sense one wants to avoid
being low on both hormones. Again, need to read more.