Microphone-equipped Internet of Things devices, and smart voice assistants specifically, offer the promise of great convenience, yet pose grave privacy challenges. The aim of our research is to understand the privacy implications of voice as a sensitive data source and develop techniques to help users protect their privacy from these always-listening devices. More specifically, our goal is for users to be able to specify restrictions on these devices—what should they be able to hear and what is off-limits?—and for our system to be able to enforce their preferences. As part of our research, we will study people’s expectations for these devices, design natural interfaces for specifying these constraints, investigate techniques for enforcing these preferences, and evaluate the effectiveness and usability of the proposed approaches.