In short, fungi are much more closely related evolutionarily speaking to animals than bacteria are. It’s harder to find substances that are toxic to fungal cells but not to animal cells because of the similarity between the two. Even our current anti fungals tend to have high rates of adverse affects and a lower therapeutic index as it is (meaning the line between helpful vs poisonous is pretty close for many of those drugs).