Root Rotting is a technique used by classes who get the Root line of spells. Basically, you root the monster, rendering it immobile and then proceed to damage it via spells. This is most effective with damage over time spells (DoTs) rather than direct damage spells, as DD spells tend to break the root and the monster will come running to kill you. If that happens, you cast your root spell again and continue to do damage until the monster is dead.
Handy spells to use in conjunction with root rotting are snare spells and slowing spells. Snares reduce the movement rate of the monster. These are helpful when the root breaks (which inevitably happens), as they slow the monster down so that you have a chance to either run away or recast your root spell. Slows are spells which increase the time in between swings that the monster takes at you ie. instead of being able to hit you once every 4 seconds (or quad you once every four seconds), he can only have a chance to hit you once every 6 seconds or 8 seconds, depending on the level of the monster and the slow. Once again, this gives you additional time to move out of the way, cast a spell or re-root the monster.