Riding a horse gives you an instant advantage against foes on foot, and adds another level of depth to gameplay.
Weight isn't an issue
The most important factor to remember is that the weight of your equipment has no real effect on the horse's speed, and so it is only the speed of the weapons and shields themselves that slows you down. This means you can fight efficiently while wearing full black armor, whereas if you fought on foot you'd move very slowly unless you put spent some skill points on Athletics.
Controls
The horse controls a little differently to being on foot. W A S D control the horse's speed and direction (if you bring the horse to a stop and then continue holding S you can reverse, although very slowly) while 
controls the camera and your character's aim. There is a blind spot behind your left shoulder where you cannot aim—when aiming into this blind spot with a ranged weapon the targeting crosshair will disappear, although you can fire blindly and still manage to score some hits.
Directional swings
Just like on foot, with a one- or two-handed weapon you can choose what side to swing to simply by looking at that side. If you keep missing or swinging from the wrong side you can swing your weapon back early, but bear in mind that this leaves you open to attack. Also, more of your weapon is exposed when on your right side than your left, so if you have a choice it's best to pass to the left of an enemy while swinging from the right for the best chance of dealing damage.
If you are using a polearm the only attack you can use is a thrust. This can be difficult to score hits with so it is best to either charge the attack ahead of time and then release as you pass the target or else slow down slightly.
Weapon choices
When using ranged weapons from horseback there is an accuracy penalty. This penalty is indicated by the targeting crosshair growing larger. You can decrease this penalty by putting points in the Horse Archery skill, or by slowing down before firing (the faster you go, the greater the penalty is). For all other weapons, horseback combat has no real disadvantages.
Unlike on foot, the length of your weapon is absolutely crucial. While fighting with a short sword you may have to get dangerously close to your enemies, which can result in damage to you and/or your horse. If you have a choice between a short weapon with good damage and a long weapon with mediocre damage, the long weapon is probably the safest choice.
The overall best one-handed sword for horseback use is the Bastard Sword. It is very long, deals great damage, and has decent speed. The Bastard Sword can also be wielded as a two-hander (simply put your shield away), but for general two-handed use the Sword of War is more powerful and so is an even better choice. You can also wield pole-based weapons such as Glaives and axes; some of these deal more damage and are longer than the Bastard Sword or Sword of War, but the longer the weapon the slower its attacks are. Many of them are two-handed, meaning you can't use a shield without changing weapons.
On horseback, the main difference between one- and two-handed weapons is whether you want a shield. While a two-hander is slightly longer and deals a good deal more damage the shield provides extra protection (especially against archers), and since you're only swinging the two-hander with one hand there are speed and damage penalties which further narrow the gap.
If you want to do both, try building up your proficiency wielding two-handed weapons with some points in one-handed as well, and then using a Bastard Sword and a shield. For most killing put your shield away, but whenever you need it just pull it out again. This way you will fight with a slight disadvantage when using the shield and also cannot access the greater damage of the Sword of War, but you still get the advantages of having a shield quickly accessible. Unless you want to raise the skills through use you will have to put some points into Weapon Master to keep your one-handed skill at a comparable level to your two-handed skill, although once you reach around 150 in each most enemies will go down quickly regardless of which you are using.
Polearm couching
On horseback, polearms provide a huge advantage. If you ride your horse as fast as possible your polearm will eventually drop into a horizontal position (the weapon's speed determines how fast this happens) and your character will "couch" it under under the shoulder. In this position, if the point hits an enemy it will deal significant damage, far greater than if you simply used 
to thrust at the target. When the tip hits a horseman it may damage the horse, the rider, or even both. Bear in mind that mounted enemies equipped with spears and lances can do the same to you!
If your horse is going too slowly (such as when going uphill, or if you haven't held W until it has reached its top speed) the polearm will go vertical again, making couching impossible. By increasing your Riding skill you can boost your horse's speed, which means there are fewer times when your horse is going too slowly for the polearm to couch.
Longer and "no shield" polearms deal more damage but they will lower more slowly. The advantage is that you can hit targets without having to get as close as with shorter polearms, which can help keep you out of danger. Another consideration is how effective a polearm will be when you are unhorsed; lance-style polearms move very slowly when on foot (meaning you will have a harder time fighting) but spear-style polearms can be used very effectively. Spear-style polearms can be used to block if you have no shield, but lance-style polearms cannot. Spear-style polearms can also be swung like a two-handed weapon if you aren't using a shield.
The damage a couched polearm deals can be further increased by getting a polearm with more speed or a better damage rating and by increasing your Polearms proficiency. Increasing your Riding skill makes your horse go faster, which increases the speed bonus that determines damage bonuses and penalties. Once you have raised your skills enough and are travelling at top speed you should be able to take out any target with a single touch of your polearm tip regardless of what the polearm's base damage is.
Horse choices
If you like the idea of running your enemies down with a lance or outstretched sword then what you're looking for in a horse is speed above all else. Speed has a direct effect on damage; the absolute best horse for this is a Courser (preferably Spirited), however it doesn't have much armour and may stop if it runs into enemies at lower speeds. A horse of a Heavy variety will deal more charge damage and also has more armour, but is slower than the Spirited equivalent.
If you would prefer durability over speed-based damage go for a Warhorse or Charger. These horses are extremely expensive, but they have an immense amount of armour and may be even harder to kill than you! Whereas other horses will often rear up and stop when they hit a target, Warhorses and Chargers will run straight over anything other than obstacles, horses, or friendly forces, making them perfect for dashing into dangerous throngs. If they do stop, however, their slow speed means it can take a while to escape from the situation.
The other horses fall somewhere between these two extremes. These particular three horse types are very expensive (especially the Charger) so if you start losing them in combat you may want to try different ones until you find one that matches your fighting style. The manoeuvrability of the horse is of less importance than its speed, as with a fast horse like the Courser you can slow down, do a U-turn, and then head off again, only being in danger for a very short period.