Using finite state machines for evaluating spoken dialog systems
Abstract
Development of spoken dialog systems (SDSs) can be facilitated by better evaluation methods. Previous methods seldom consider the efficiency of the system, which is important to users. We study the problem of evaluating SDSs and propose a new framework by generalizing states from utterances of dialogs to build finite state machine (FSM). These states can be regarded as efficiency measurement of SDSs. The FSM framework models dialogs as paths in an FSM to combine efficiency measurement with regression models. The proposed FSM framework can be applied in conjunction with regression models to improve evaluation accuracy. We compare our FSM framework combined with three regression models in several experiments. We obtain promising results on a collection of dialogs from the “Let's Go!” system, with our approach outperforming regression models.
Authors: Yi Zhu, Zhaojun Yang, Helen M. L. Meng, Baichuan Li, Gina Levow, Irwin King
Published in: IEEE Spoken Language Technology Workshop (SLT) (2010)