Crowd-talk yields great answers, says university team
Source: Nancy Owano
The Chorus system. User requests are forwarded to crowd workers, who then submit and vote on responses. Once sufficient agreement is reached, responses are made visible to users. The crowd’s working memory is updated by workers selecting lines from the conversation or summarizing important facts. Credit: Walter S. Lasecki et al.
Move over, Siri. Some researchers from the University of Rochester in collaboration with a University of California, Berkeley, mathematician/crowdsourcing entrepreneur, have come up with a killer personal assistant approach. "We introduce Chorus, a system that enables realtime, two-way natural language conversation between an end user and a crowd acting as a single agent." So begins their paper, "Speaking with the Crowd," suggesting the ideal artificial chat partner is the partner that is actually the work of contributions from many crowdsourced workers. The researchers propose a crowd-powered chat system that behaves as an online collaborative interface. They believe it one-ups existing systems because it can take on more complex tasks.
Walter S. Lasecki, Rachel Wesley, and Jeffrey P. Bigham from the University of Rochester worked with Anand Kulkarni, the cofounder of the crowdsourcing company MobileWorks, to create Chorus. They sought to demonstrate that the power of crowdsourcing might be able to go beyond simple tasks into the complex. "What we're really interested in is when a crowd as a collective can do better than even a high-quality individual," said co-author Bigham.
How the Chorus system works: People talk to Chorus with an instant messaging window. User requests are forwarded to crowd workers, who submit and vote on responses. When agreement is reached, responses are made visible to users. The crowd's working memory is updated by workers selecting lines from the conversation or summarizing important facts. According to the co-authors of the paper, "Chorus is capable of maintaining a consistent, on-topic conversation with end users across multiple sessions, despite constituent individuals perpetually joining and leaving the crowd. This is enabled by using a curated shared dialogue history."
Put to the test, the researchers found that Chorus was able to keep up consistent conversations between a single user and large numbers of crowd participants. They said the talk was kept on focus. Also, Chorus was capable of retaining meaningful long-term conversational memory across multiple sessions, even as individual users changed. As for accuracy, workers were able to answer 84.6 percent of user queries correctly.
In all, say the authors, "These findings suggest that Chorus is a robust interface for allowing disparate members of the crowd to represent a single individual during natural language conversations as an alternative to software agents."
How robust these systems become as technologies evolve remains to be seen. What is known is that personal assistants such as Apple's Siri are very useful but do not come up to par with the conversational skills of a real person. As the researchers note, robust two-way conversations with software agents remain a challenge. Existing dialogue-based systems generally rely on a fixed input vocabularies or restricted phrasings, have a limited memory of past interactions, and use a fixed output vocabulary.
As for where Chorus can fit in the real world, the authors say that "In the future, we expect Chorus will have utility as a conversational partner and as a natural-language dialogue interface to existing systems."
| }
|