There are several glitches in Super Mario 64 and Super Mario 64 DS, which can be useful or just strange. Please note that they most likely were fixed in the DS remake.
Super Mario 64 Glitches[edit]
Shortcut on the Second Floor Steps[edit]
If you've opened the door to the castle's upper floors, on the spiral staircase leading to them, there's an unintended shortcut. Jump along the edge of the wall just before the left turn. Mario will grab onto an edge that shouldn't be there. You can pull him up through the wall, and he'll be right there at the door that faces the painting for Wet-Dry World. A black room of death (near the door to the second floor) can be found by doing this glitch.
The Rabbit Trick[edit]
If you grab the rabbit in the basement at the right angle, you will warp slightly so you are in front of the rabbit in the right position to grab it without changing which way you face. This can be exploited to warp through the star door without 30 stars by placing the rabbit in the right position and grabbing so you are on the other side of the door. This is used for beating the game with only 16 Stars.
Some other methods[edit]
This trick is used for "16 Star Speedruns" and there are seemingly multiple ways in doing it, the easiest would be to angle Mario facing the door you wish to pass through so that the rabbit looks as though it's inside the wall, on the corner of the door, dropping with 
and sliding in front of the rabbit. Just as you drop it you'll then be pushed through the wall, door and all, appearing half-way through it, grab (NOTE: This can be easier said than done at times if you happen to face the wrong direction) your lucky rabbit and continue to your next destination. As for the Star Door, this one can be done more quickly by dropping the rabbit in front of the door (yes, at the same angle) and doing a long jump into the center of the door, so that you would normally land where the rabbit is, but instead, be pushed forward through the door.
Both methods do require multiple attempts. You're extremely lucky to have this happen on your first attempt. This third method also works, and has been confirmed. Place your rabbit in front of the door, now, walk through the door normally, and walk through again, just as you walk through, run back and crouch, timed right you'll easily slide right into the door. You'll be halfway through. IF the rabbit is in the right place, you can safely grab him. However, proceed with caution, as you're "treading on ice" and any sudden movements will shove you through either side of the door. Overall the 1st one may be the safest if you plan on attempting this over, and over, and over. The 2nd attempt is used in speed runs and can be performed (after practice) very quickly. The third one is simply another method that may work for you if you have trouble getting the 1st one right.
Bypass The Last Star Door And The Endless Stairs[edit]
If you long jump backwards on the stairs at the right angle rapidly, you will go through the star door or climb the endless stairs, depending on which stairs you are doing it on. In order to long jump backwards, do a regular long jump, then move the control stick backwards. Keep holding down the and press the 
as fast as possible while on the stairs to fly through the star door or the endless stairs. This allows you to beat the game with as little as 16 Stars. The guys doing these videos use the rabbit glitch and the stair trick: [1]
Other Way to Kick[edit]
This is simple. When you press and 
at the same time, Mario will be able to kick without jumping.
Super Mario 64 DS Glitches[edit]
Suicidal Chill Bully[edit]
Go to Snowman's Land and grab the crate. Go over to the ? Block, line yourself up with the tree, and throw the box onto the platform with the Chill Bully on it. Chill Bully will walk onto the box and fall into the ice cold water.
Frozen Snowman's Land[edit]
This glitch works in the European version. In Snowman's Land, set the star to Snowman's Silver Star, collect all five Silver Stars, then go into the igloo. You will find that all the stars have disappeared. Now, go back outside the igloo and you will find out that the stars are back. If you collect one, everything except your character will freeze. Also, the enemies will disappear completely. To get out of the level, get yourself killed by the freezing water, press 
, 
, 
, and 
at the same time to return to the title screen, or simply turn the power off.
Fall Through the Stairs[edit]
In the room with the entrance to Tick Tock Clock, go over to the staircase and go onto the second step. Do a slide kick against the wall and you will fall through the staircase and die. Rarely, however, you will get shot up into the sky and can guide yourself to Bowser in The Sky. This means you can beat the game with as little as 50 stars.
Access the Castle Roof with Only 8 Stars[edit]
You will need to be Mario, Luigi, or Wario to do this glitch. Near the cannon outside the castle, slide head first into one of the trees. Half of your body will be above ground, and the other half will be under the ground. Now swim over to the cannon and shoot yourself to the roof. This glitch only works if the water hasn't been drained from the moat.
Access the Castle Basement with Only 8 Stars[edit]
You will need to be Mario, Luigi, or Wario to do this glitch. Do the Access the Castle Roof with Only 8 Stars glitch, but this time don't go into the cannon. Instead, swim deep under the water (below the moat) and enter the basement door. Make sure you enter it from behind. The screen will go white as if you entered a painting. You will appear in the castle basement. If you haven't completed Bowser in the Dark World, going up to the basement door will just result in a message box appearing, saying that you need the key. If you do this glitch with 30 Power Stars, you can complete Bowser in the Fire Sea without completing Bowser in the Dark World.