In this paper we present an algorithm for structure and motion (SM) recovery under affine projection from video sequences. The algorithm tracks the motion of a single structure, be it an object or the entire scene itself, allowing for any type of camera motion. This could be used for example to track the motion of a vehicle in a warehouse (single object, static camera) or for visual navigation from a moving platform (track scene from moving camera). The algorithm requires a set of features to be detected in each frame, and that at least four features are correctly matched between each three consecutive frames. Compared to previous algorithms this novel algorithm has a lower computational cost, making it attractive for real time applications. Our algorithm also provides dynamical detection of outliers and allows for previously lost features to reappear in the sequence.