Task oriented dialogue systems rely heavily on specialized dialogue state tracking (DST) modules for dynamically predicting user intent throughout the conversa- tion. State-of-the-art DST models are typically trained in a supervised manner from manual annotations at the turn level. However, these annotations are costly to obtain, which makes it difficult to create accurate dialogue systems for new do- mains. To address these limitations, we propose a method, based on reinforcement learning, for transferring DST models to new domains without turn-level super- vision. Across several domains, our experiments show that this method quickly adapts off-the-shelf models to new domains and performs on par with models trained with turn-level supervision. We also show our method can improve mod- els trained using turn-level supervision by subsequent fine-tuning optimization toward dialog-level rewards.