The doorbell camera 106 has a field of view of an entryway of the smart home environment 100. The motion stream is a video stream depicting movement or information regarding amount of motion in a scene. The server 164 determines based on an analysis of the motion stream that a visitor is approaching the entryway, and performs a facial recognition operation based on one or more frames of the motion stream. The server 164 then determines based on an outcome of the facial recognition operation whether the person is known to the smart home environment. A time window is initiated in response to the determination that a visitor is approaching, and the time window is optionally predefined to correspond to a reasonable amount of time for the visitor to complete approaching/reaching the entryway. During the time window, the server 164 obtains contextual information from one or more sensors of the smart home environment 100, and determines whether an action from the visitor e.