The full deployment of service robots in daily activities will require the robot to adapt to the needs of non-expert users, particularly, to learn how to perform new tasks from “natural” interactions. Reinforcement learning has been widely used in robotics, however, traditional algorithms require long training times, and may have problems with continuous spaces. Programming by demonstration has been used to instruct a robot, but is limited by the quality of the trace provided by the user. In this paper, we introduce a novel approach that can handle continuous spaces, can produce continuous actions and incorporates the user’s intervention to quickly learn optimal policies of tasks deﬁned by the user. It is shown how the continuous actions produce smooth trajectories and how the user’s intervention allows the robot to learn signiﬁcantly faster optimal policies. The proposed approach is tested in a simulated robot with very promising results.