This paper describes a virtual reality system that realizes both an avatar motion control and a virtual object manipulation without avatar representation. Our goal is to do seamless mapping of human motion in the real world into virtual environments. We hope that the idea of direct human motion sensing will be used on future intelligent user interfaces. Our method can generate realistic avatar motion from the sensing data. We use virtual scene context as a priori knowledge. We assume that virtual environments provide action information for the avatar. In the virtual object manipulation without avatar representation, the user behaviors through physical-virtual interaction are interpreted as the object manipulation tasks. The behaviors which the user performs in the real world are converted into the scene events to manipulate the interesting virtual object. KEYWORDS image processing, motion capture, perceptual user interfaces