This paper describes a real time 3D object recognition approach based on exploiting spatio-temporal constraints on a continuous set of views. We obtain a compact representation of an image sequence by extracting scale-space local keypoints and tracking them over the sequence. Our recognition is strongly based on a two-step matching strategy that emphasizes spatially and temporally adjacent groups of features. On-line recognition is achieved by incrementally updating the test model and reducing the amount of keypoints detection by fully exploiting temporal continuity. We utilize such a system to discuss the importance of spatio-temporal information obtained observing an object for some time and possibly from slightly different view-points.