We Track Everything Except What Actually Moves Us