Walking around the world in Mount and Blade II: Bannerlord feels a lot better when you can do it with a massive army behind you. When you first start the game, you can't have too many troops under your command because you haven't increased your value, and few troops will trust you reliably. However, over time, you can regularly add more troops to your clan, and you can lead them into battle. Growing the size of your Bannerlord group does take time, however.
The size of your party is related to your clan ranking and your character as your clan quartermaster and your flight attendant skills. Let's focus on leveling the clan first.
The higher your Clan Rating in the game, the more troops you can control and have under your command. To check out your clan page and see how you're doing, you can click the little crown icon at the bottom left of your screen to view it, or you can click the L button to view it. Once you've done that, you should be able to see your clan level at the top right, and then you can see your current progress right next to it. If you want to level up your clan, you need to expand your fame, and there are several ways to do it, such as participating in massive battles, winning, tournaments, and becoming a mercenary.
As your clan's Quartermaster with a high Quartermaster skill, you can regularly increase the number of troops under your command. It takes a bit of time to develop this skill, especially at the start of the game. To level it up without focusing too much on it during your initial character creation, you need to boost the morale of your troops by giving them power. variety, improve the prosperity of your colony and spend time in your colony and not travel as much. Providing your army with a varied diet and boosting morale is probably the most natural first step. You can do this by having a variety of consumable foods for them to eat every day and win battles.
It does take a bit of time to develop your army group size, so don't focus too much on it. The best way to gradually level up your clan is to go through the main story and improve your character's steward skills whenever you can while playing.