Mount&Blade/Neutral

In addition to the various factions there are also some party types that aren't affected by faction relations.

Mercenaries
Troops from the mercenary line are most easily found in taverns, but they can also be acquired by attacking caravans and then convincing the surviving Caravan Guards to join you. You can also rescue them from parties that have destroyed caravans. Faction parties will regularly attack any caravans that belong to their enemies, so it is quite common to find a faction party carrying a few Caravan Guards.

Bandits
Bandit parties are found roaming various parts of the map. They will attack any party smaller than their own, except for other bandits. Some bandits do not upgrade at all, while others can eventually upgrade into Mercenaries if you manage to.

Manhunters
Manhunters stay close to Looter spawn areas, but they will also attack Sea Raiders and other bandit parties. All troops in the Manhunter line are only equipped with blunt weapons, so if you want to take lots of captives they are perfect for this purpose.

Sword Sisters
Despite slowly evolving from initially vulnerable women, this line results in Sword Sisters, who are extremely powerful fighters on a par with the Hired Blades. Refugees and Peasant Women are very difficult to acquire, so for most players this troop line will be very rare.

If you intend to develop female fighters it is crucial that you keep them out of combat at all costs as they will only be able to hold their own from Camp Huntress onwards (anything lower will be quickly butchered by most other troop types). Once they reach Camp Defender or Sword Sister level they can hold their own against a great many troop types.