After defeating Varalys in the Forbidden Cave and collecting the Status of Varalys, return to the City of Illusion. Make your way to the right side to visit the prison.
Most of the prisoners will say the same thing they said to you before, with only one exception. The man occupying the cell in the upper left corner of the prison will realize you have the Statue of Varalys, and indicates that it is the key to entering the Door of Time, which is located north east of the city. You won't be able to leave without a special talisman, which the prisoner gives to you. The prisoner explains that he once went through the door with a fairy, but the fairy was captured by Kaizack. Then he gives you a modest hint about Kaizack's weakness, explaining that you should watch his movements.
There's one last hint that you need to open the Door of Time. Not many more of the citizens of this town will be talkative, but an old man in the center of town explains that the door can only be opened at midnight. With this knowledge and the Exit Talisman in hand, head out of the city.
As the prison said, you want to make your way north east of the city until you reach the coast of the large island. Then follow the coast as it heads south, which should lead you to a peninsula, with a rock formation in the center. Approach the rocks from the bottom and step up into middle, where you should hear a chime indicating that you should search. If you search and it says that it looks like some rocks have been moved, you're in the right place.
The entrance doesn't only open exactly at midnight. It happens to open whenever it's dark. The problem with waiting for darkness is that it can take a long time, and you will get tired and be unable to fight if you go that long without rest. Fortunately, all four classes can and should have learned the Time spell, which can advance time ahead by 12 hours. You can change a bright early morning into a dark evening with one cast of that spell. Then use the Statue of Varalys, and you will pass through the Time Door.