There was this RPG I played way back in maybe late 90s ish. I obviously can’t remember the name but I do remember that there was a tower where you lived (forgot if you owned from the start or purchased) and there were a LOT of PCs you could find in the world that would join your group, if they didn’t join right away you would later meet them again in your tower. They only got XP/levels when in a group out in the world.
If I remember more I will add it. Some questions may bring more memories of it back as well.
EDIT: Thank you to @[email protected] @[email protected] and @[email protected] that is the game


The Shining Force games (the first two were Genesis/Megadrive) did this as well - your inactive party members showed up in a tavern-like space in the basement of your home castle. (Unless they were currently dead, in which case they appeared as purple flames.)
Shining force two was amazing