Finally found the source of the mildew smell, there was mildew build up in the Toilet Air Vent, I removed the vent and gave it a good clean, I was surprised to see how much slimy crap had build up and was not visible by just looking at the vent.