Doesn’t get any more secure than a battle-tested web server hosting simple MP3 files and a text file.
Convenience might be a thing, though. I’m in the Apple ecosystem so their Podcasts app shows that feed on all devices and tracks listening progress, etc.
If I didn’t have that, I’m still a lifetime customer with PocketCasts and PocketCasts Web. So, that’s that. But if you don’t have anything similar in place, a self-hosted streaming server might be the best way to go, yes.
If you’re happy with how Apple Password works for you, I can recommend StrongBox. It keeps all data in a KeePass2 database and integrates into Apple’s AutoFill API. That means it feels almost native when using it. No browser plugin needed. (At least not for Safari.) And you can decide how you sync the database file.