I tried this out like 12-18 months ago before deciding to just run my own Mastodon instance. I recall, like poVoq mentioned, having to play with my nginx config a bit to get webfinger junk behaving as expected.

But even after I got to working it was pretty bare-bones and I’m not sure it’s gotten any better since then.

Look into your .well-known and webfinger configuration of Nextcloud I would guess.