Siri on the Street: Test Measures Comprehension at 83%, Accuracy at 62%

I love Siri, but I have to admit, she has a ways to go… Now an analyst and his team have conducted a “Siri-es” of tests with Siri. (See what I did there?) They say Apple’s personal assistant isn’t quite reliable enough to entirely replace text input with Google.

AppleInsider:

The closer look at Siri and its capabilities was conducted by analyst Gene Munster of Piper Jaffray. To test Siri, Munster and his team gave the software 800 queries in a quiet room, with no headphones, and 800 queries on a busy Minneapolis street through Skull Candy microphone headphones.

The test measured Siri comprehension on the street at 83 percent, with accuracy at 62 percent. In the quiet room, Siri fared slightly better, comprehending 89 percent of queries and accurately answering 68 percent of answers.

Siri didn’t fare as well when compared to searching Google, with queries manually typed in. Google provided an estimated 86 percent answer accuracy, when asked the same queries.

“We believe the most likely substitute for Siri is Google text input on the iPhone, not the Google voice search app,” Munster explained.

Of the 83 percent of queries Siri could comprehend, 21 percent were answered inaccurately. Some of the questions that stumped Siri were:

• “What team does Peyton Manning play for?” Siri would respond with the answer to the previous query.
• “Where is Elvis buried?” Siri looked for a person named “Elvis Buried.”
• “Where am I?” Sometimes a map pin would be dropped in the wrong place.
• “When did the movie Cinderella come out?” Siri would attempt to find current theater movie times via Yelp.
• “How do I get from Boston to New York?” Siri responded: “I can only give directions from your current location. I can’t give you directions to a place you are not in.”

As far as scoring goes, Munster gave Google an A+ for comprehension and B+ for accuracy, while Apple’s Siri was graded with a B for comprehension and a D for accuracy.

“While Siri is two-plus years behind Google today, we are optimistic about its future,” Munster wrote. “With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google.”