The engine covers all the easy verticals (books, DVDs, logos, landmarks, products, text, etc). The recognition quality is very good, though the landing pages are often a bit useless. It will take a bit of living with it to see how much use it is as a tool rather than a tech demo.
The major worry is that it may end up being too broad-but-shallow. For example, they do wine recognition, but the landing pages are generic. Perhaps visual wine recognition would be better built into Snoot or some other dedicated iPhone wine app. Or Google could take the route Bing recently took with recipes, and build rich landing pages for each vertical. Because of the nature of current visual search technology, Goggles is essentially a number of different vertical searches glued together, so this is more feasible than it would be for web search.

Certainly an interesting week for visual search!

The main thing that surprised me is that the interviewer mentioned response times of less than a second. I find that somewhat unlikely, and I think it’s probably a case of a journalist misunderstanding some fine (but crucial) distinctions.
In terms of the actual visual search engine, there’s no problem. Our own visual search engine can serve searches in a couple of hundred milliseconds, even for millions of images. The issue is transmitting the user image to the server over the mobile phone network. Cell networks are still pretty slow, and have high latency even to transmit 1 byte. Even after aggressive compression of the image and after playing some other fancy tricks, we see typical user waiting times between 4 and 5 seconds over a typical 3G connection. On a 2G connection, the system is basically unusably slow . I strongly suspect the Google app will perform similarly. In many cases it’s still much faster than pecking away on an on-screen keyboard, though because the user is waiting, it can feel longer. It would almost be worth giving the user a pointless maths puzzle to solve, tell them it’s powering the recognition, and they would probably be happier!

In any case, the Google demo is an interesting, if not unexpected, development in the visual search space. While we’re waiting to see the final version of Google Visual Search, Android users can try out our PlinkArt app, which does visual search for famous paintings. It’s live right now!

The big question now is how this changes the playing field for other companies in the visual search space. I would assume Amazon will move SnapTell’s focus away from their enhanced print advertising service and concentrate on image recognition for books, CDs, DVD, etc. (Up to now, Amazon has been doing this with human-powered image recognition, which was nuts.) While this makes perfect sense for Amazon, it’s going to mean more rather than less opportunities for companies still focused on the general visual search market.

So  I guess this is an ideal point to mention the open secret that I’m currently co-founding Plink, a new visual search engine similar in capability to SnapTell. While our demo shows some familiar use cases, we’re working on taking the technology in some entirely new directions. Visual search is very young, there’s a whole lot still to do! Anyone interested in visual search, feel free to contact me.

Today Amazon released an iPhone app with a feature called “Amazon Remembers”. You take a picture of some product you’re interested in and the app uploads it to Amazon. You can then revisit your list later from a browser, e.g. to buy the item. The interesting bit is that Amazon attempts to generate a link to the product page based on your image. Examples here. This isn’t instant, and may take anywhere from five minutes to 24 hours.

Fascinatingly, the back end is apparently not computer vision based. It uses Mechanical Turk, where Amazon is paying people $0.10 per image to generate the links. See here. This is quite amazing to me. I have no idea if Mechanical Turk is deep enough to support this app if it truly becomes popular, but I suppose in that case Amazon could set up a dedicated call-centre type operation to handle the image recognition.

So, visual search companies now have a very direct measure of the value of a correct image search. Judging by the current set of images on Mechanical Turk, a fully automated solution is not possible. However, a hybrid system where some easy categories like books are recognised automatically and harder cases are farmed out to Mechanical Turk would clearly translate into significantly lower costs.
(Of course it’s possible Amazon are already doing this, though I did see several books in the Mechanical Turk requests).

“Nokia is committed to owning the visual search space and has committed a staff of 30 to build the business and further develop the technology. The business area has the buy-in of Nokia senior execs and “quite large” funding from the company“

The technology comes from an acquisition of a valley startup called Pixto just over a year ago. Nokia’s service is apparently  due for launch soon, initially recognising movie posters only.

Right now is the perfect moment for this technology. New platforms like iPhone and Android are offering the first compelling mobile web experiences, and have built-in cameras of acceptable quality. Coincidentally the computer vision research on large scale object recognition has just recently reached the necessary level of maturity. Three search engines launched in the past three months, and a few more appear to be in the works. Some have been available for a while via mms or email, but it’s only as iPhone apps that they are beginning to be seriously useful.

Visual search is still very small. Right now I expect it’s generating almost no cash flow, so every company in the field is also pursuing alternative uses of the technology. Some of these are interesting businesses in their own right. The most popular application is enhanced print advertising, where newspaper or magazine readers are encouraged to take a picture of an advertisement and send it to a shortcode. The image is then recognised, and the user gets back extra content or is entered into a prize draw, etc. Advertisers get enhanced engagement, and crucially they also get a way to measure the effectiveness of their offline advertising spend. There is serious advertiser hunger for this, and the services are already growing quickly. A recent Kooaba campaign took the idea a step further.
Attractive as these applications are, I think they’re mainly good to pay the bills and raise awareness in the short term. Longer term I think more typical search applications are going to dominate.

So, the players:

Update, October 2009:

Since writing this post, I have co-founded my own visual search company, Plink. Plink was spun out of the computer vision and robotics research groups at Oxford University, and we have some of the best technology and and best visual search engineers anywhere on the planet. Our Developer API gives anyone access to the system, and entry level usage is free.
Our own first app built on the platform is called  PlinkArt. The app lets you identify famous paintings just by taking a photo, and we are trialing it as a replacement for traditional audio guides in museums. The app was selected as one of the $100,000 prize winners in Android Developer Challenge, and an iPhone version is coming soon.

My original, unedited thoughts from 2008 are preserved below. The main news since then is that SnapTell were acquired by Amazon, and that in the last few days Google pre-announced their internal Google Visual Search or “Google Googles” project. You can read all my posts on the subject here.

Original post continues:

SnapTell

SnapTell arguably leads the field at the moment. I believe they currently have the largest search index at 2.5 million items. This includes most book covers, games, CDs and DVDs.  Their recognition technology is first rate, even with poor images. Despite their search engine being so impressive, their main focus seems to be on their enhanced print advertising business, where they have a number of high profile clients like Sports Illustrated and Rolling Stone. This focus shows – they have slick marketing materials aimed at magazine advertisers, but rather poor landing pages for their search engine. However, their search engine app just got a rave review on TechCrunch, so perhaps search will be moving up their agenda as traffic goes up. Venture funding from NEA.

Kooaba

Kooaba are another company with first rate recognition technology (check out some examples). They’re a spinout from ETH Zurich, and are run by some well known computer vision researchers. As with SnapTell, their main current revenue stream is from enhanced print advertising, and some interest extensions of the concept. They also have an iPhone app for movie search. The number of items indexed is still quite small, though no doubt it will be expanding soon. Their technology certainly scales to millions of indexed items.
For other companies looking to acquire visual search capability, kooaba is a peach.

TinEye Mobile

Idée are a Canadian company involved in several types of image search. Their TinEye Mobile app for iPhone launched late last month, and currently recognises CD covers. Their main current revenue stream is from another product called PixID, which monitors print and online publications for unauthorized use of copyright images from clients like the European Pressphoto Agency. You can try the same system for web searches at TinEye.com, where they have a little over a billion images indexed. They also have a cute colour-based Flickr search.
The technology behind TinEye is quite different to the Kooaba/SnapTell approach. It can jump through hoops in its intended application of large scale near-duplicate detection for PixID, however, it’s not quite so good at dealing with some of the challenges of camera phone images, particularly geometric distortions. However, Idée strike me as a company with excellent corporate DNA and good PR management. They had 23 employees and were entirely privately funded as of last year. So while their technology is not quite as good as SnapTell or Kooaba, I think they’re one to watch.

Evolution Robotics

Evolution have a number of vision based products, notably for robot navigation and theft prevention in shops.  They released a video preview of an iPhone visual search engine back in April, but so far it’s not available on devices. A different version of their software is apparently big in Japan, (can’t resist), though it’s hard for me to assess that.
In addition to their own play, Evolution are taking a parallel tack as a technology provider. They are the primary licensee of David Lowe’s SIFT patent, which is a key ingredient of many of these search engines. Other players such as Mobot and possibly others have licensed from them. They recently took a chunk of investment from Vodaphone, which is suggestive.

Pongr

Pongr are a company I only came across a few days ago. They offer price comparison, optionally driven by image recognition. I haven’t had a chance to try their software yet, but based on their video demo their recognition technology is reasonable but not fantastic. Possibly licenced from somewhere? The image database is currently small, only thousands of products. However, where they might win is with a focus on UI, Facebook integration and all those other ingredients that make up a good experience.

SnapNow

SnapNow are another player in the enhanced advertising space. My impression is that they don’t have the necessary technology to roll out a full search engine. They’ve been concentrating on Japan since 2005, and have just launched in the UK, US and China. Judging by their patents, their recognition technology is out of date, but that’s not to say that it doesn’t work. Unless they’ve changed technology, they will likely have some trouble scaling to large index sizes. Without a smart phone client, the experience is also pretty clunk – you have to send a picture by mms (expensive) and get back a link in an SMS message (slow). However, they have a fairly established presence in Japan, and have already signed up a number of initial magazine clients in the UK.

Vodafone

Now we come to the 800-pound gorillas. Vodafone demoed a visual search engine called Otello at CeBIT in March, where it was amusingly panned by the German Chancellor for failing to recognise her. (In fairness, none of these engines can do face recognition). I must admit that I don’t know anything else about Otello other than what’s in the Engadget post. Nokia also have some dubious patents in the area. I’m highly skeptical that any major carrier can really pull off a project like this. If they want to own this space they’ll be acquiring start-ups. Nokia have a $250 million fund for supporting exactly that.

Microsoft

Microsoft research is exceptionally strong in computer vision – they employ many of the top people leaving academia. While there is a long history of research outfits where cool things are invented and then gather dust, MS Research have taken several ideas all the way to shipped product, the main example being Photosynth. They also have a prototype visual search engine called Lincoln. There is an article discussing the system here. Whether Microsoft has the motivation to make this a hit, I’m not sure, but they certainly have the setup to follow quickly if visual search takes off.

Google

Google are as usual a huge unknown. They haven’t announced anything, but they certainly have many good computer vision people from their acquisition of Neven Vision back in 2006. Neven Vision even hold a patent on mobile visual search engines, and YouTube has some pretty strong related technology for detecting uploads of copyrighted video. They also hired Henrik Stewénius, who published one of the seminal papers on visual search. On top of that, Eric Schmidt recently had some suggestive things to say about interactive magazine advertising, talking about how “young audiences will want a more interactive experience in print.” Make of that what you will.

Others

Several other players out there at early stages or with lesser offerings.

IQ Engines has a good demo, but is still pre-release. Based on their research papers, their technology is quite a different approach to anyone above. It’s an interesting direction in computer vision at the moment, but I wouldn’t have guessed it would be ready for prime time. The demos look good though.

Thrrum/23half claim to do general visual search based on text extraction, though I’m skeptical to what extent their technology actually works. Mobot do enhanced print advertising using the Evolution Robotics engine. No smart phone app at present, so the experience is not much fun. Mobile Acuity are another similar outfit, I am not sure that they are still active.

Whew, long post. As far as I know the above is a complete list of companies in this space. The really significant players are SnapTell, Kooaba, Idée and to some extent Evolution. To put my cards on the table, I’m thinking seriously about joining someone on that list or starting a visual search company myself. I think this is going to be a big hit, and there are still many niches unfilled.

If anyone knows of a company I’ve missed, please drop a comment below.

I spent a while “teasing”  the search engine with badly taken photos, and the recognition is very robust. It has no problems with blur, rotation, oblique views, background clutter or partial occlusion of the object. Below is a real recognition example:

I did find the service pretty slow, despite having a WiFi connection. Searches took about five seconds.  I had a similar experience with kooaba earlier.  There are published visual search algorithms that would let these services be as responsive as Google, so I do wonder what’s going on here. It’s possible the speed issue is somewhere else in the process, or possibly they’re using brute-force descriptor comparison to ensure high recognition rate. For a compelling experience, they desperately need to be faster.

While the recognition rate was generally excellent, I did manage to generate a few incorrect matches. One failure mode is where multiple titles have similar cover design (think “X for Dummies”)  – a picture of one title randomly returns one of the others. I saw a similar problem with a CD mismatching to another title because both featured the same record company logo. Another failure mode that might be more surprising to people who haven’t worked with these systems was mismatching on font. A few book searches returned completely unrelated titles which happened to use the same font on the cover. This happened particularly when the query image had a very plain cover, so there was no other texture to disambiguate it. The reason this can happen is that the search method relies on local shape information around sets of interest points, rather than attempt to recognise the book title as a whole by OCR.

My overall impression, however, is that this technology is very much ready for prime time. It’s easy to see visual search becoming the easiest and fastest way to get information about anything around you.

If you haven’t got an iPhone, you can try it by sending a picture to fun@snaptell.com.

I really think this technology is going to take off big-time in the near future.  The marketing intelligentsia are aware of this too. There is an adoption challenge, but SnapTell in particular are already running an excellent high profile education/promotion campaign with print magazines. They’re not messing about either: The campaign is running in Rolling Stone, GQ, Men’s Health, ESPN, Wired and Martha Stewart Weddings. In short, publications that reach a substantial chunk of the reading public. Whether the message will carry over that the technology is good for more than signing up for free deoderant samples is something I’m a little skeptical about, but in the short term it’s a ready revenue stream for the startup, and a serious quantity of collateral publicity.

Usage report later when I can get hold of an iPhone.

Update: I just got to try it, and it’s really rather good. First impressions here.

When people first started using the internet, the idea of search engines didn’t need much promotion. People were very clearly lost, and needed some tool to find the interesting content. Adopting search engines was reactive, rather than active.

Visual search is not like that. If kooaba or others do succeed in building a tool that lets you snap a picture of any object or scene and get information, well, people may completely ignore it. They’re not lost – visual search is a useful extra, not a basic necessity. The technology may never reach usage levels seen by search engines. That said, it’s clearly very useful, and I can see it getting mass adoption. It’ll just need education and promotion. Shazam is great example of a non-essential search engine that’s very useful and massively popular.

So, promotion, and lots of it. What’s the best way? Well, most of the different mobile visual search startups are currently running trail campaigns involving competitions and magazine ads (for example this SnapTell campaign).  Revenue for the startups, plus free public education on how to use visual search. Not a bad deal, easy to see why all the companies are doing it. The only problem is that it may get the public thinking that visual search is only about cheap promotions, not useful for anything real. That would be terrible for long-term usage. I rather prefer kooaba’s demo based on movie posters – it reinforces a real use case, plus it’s got some potential for revenues too.

At the moment it only recognises movie posters. Basically it’s current form is more of a technology demo than something really useful. Plans are to expand to recognise other things like books, DVDs, etc. I think there’s huge potential for this stuff. Snap a movie poster, see the trailer or get the soundtrack. Snap a book cover, see the reviews on Amazon. Snap an ad in a magazine, buy the product. Snap a resturant, get reviews. Most of the real world becomes clickable. Everything is a link.

The technology is very scalable – The internals use an inverted index just like normal text search engines. In my own research I’m working with hundreds of thousands of images right now. It’s probably going to be possible to index a sizeable fraction of all the objects in the world -  literally take a picture of anything and get search results. The technology is certainly fast enough, though how the recognition rate will hold up with such large databases is currently unknown.

My only question is – where’s the buzz, and why has it taken them so long?

Update: I gave the app a spin today on a friend’s iPhone, and it basically works as advertised. It was rather slow though – maybe 5 seconds per search. I’m not sure if this was a network issue (though the iPhone had a WiFi connection), or maybe kooaba got more traffic today than they were expecting. The core algorithm is fast – easily less than 0.2 seconds (and even faster with the latest GPU-based feature detection).  I am sure the speed issue will be fixed soon. Recognition seemed fine, my friend’s first choice of movie was located no problem. A little internet sleuthing shows that they currently have 5363 movie posters in their database. Recognition shouldn’t be an issue until the database gets much larger.