Archive for December, 2015

Pointy

Two and a half years ago I left Google and set out to build a new kind of search engine. This may sound a little crazy, but all the best things are like that :-)

We’ve been avoiding the tech press and trying to build things quietly, but this week we’re launched our user-facing app. I’m really proud of what the team has built, so it’s exciting to finally be able to say a bit more about it.

The problem we’ve been working on is finding specific items locally. For example, a light bulb just broke and it’s a strange fitting, where’s the nearest place you can get a new one?  Or you’re half way through a recipe and realise you’re missing an ingredient – where do you get it?

Existing search engines do a really bad job of answering questions like this. The reason is that, in order to provide a good answer, you need to know what products are stocked in all the local shops. It turns out that nobody has this data – not even the shops themselves in many cases. It’s kind of strange to think that you can search the entire internet in a fraction of a second, but the contents of the shop around the corner remains a mystery unless you go there in person. But that’s the state of the world in 2015. At Pointy we’ve been working on solving that problem.

The core challenge is data collection – building a scalable way to index the contents of every local shop. Enter the Pointy box:

D16929-0051_clipped_rev_1

The Pointy box is a piece of hardware we designed, which sits inline between a shop’s barcode scanner and Point of Sale system. Whenever the shop scans something, we intercept the barcode information and transmit it back to our servers over a built-in cellular connection. From this data we can figure out what products the shop stocks, and get a pretty good estimate of stock levels. How it all works is illustrated on our retailer signup page.

Technically, the magic is in how we interface (or don’t interface) with the POS system. We pull the data directly off the wire between the barcode scanner and the POS. Since we get it at such a low level, we don’t have to worry about integrating with every piece of POS software in the world. There’s still a lot of work to do, but it makes the problem much more tractable. At this stage we can support basically anything we find in the wild1, everything from ancient cash registers that look like they belong in a Western, right up to iPad based systems.

Pointy box

From the retailers’ point of view, it’s extremely simple. They just plug in the box and within a few minutes they have a nice website for their shop, which automatically lists everything they sell. They’re also part of the Pointy local search app. There’s no extra work and no configuration, it just fits in with their existing systems.

app_overview

Happily, retailers seem to love this. We started to roll it out widely in June this year, and by December roughly 1 in 8 of all shops in our launch city (Dublin, Ireland) are using the system.

map

There’s a vast variety of shops now on the platform, basically the whole range of local shops: bike stores, pharmacies, hardware, convenience, pet shops, delis, supermarketswine stores, toy shops, book stores, garden centres, even horse supply shops. There’s a huge data challenge in identifying the right name and picture to go with a barcode, and that actually occupies a big chunk of our engineering team, but that’s a topic for another post.

app_product

Infrastructure

Our system is built on Google Cloud Platform, which has let us scale quickly without having to spend time on non-core problems. We use a mixture of services including App Engine, Compute Engine and Cloud Storage. Services like Task Queues, Cron and Logs give us some great tools for building reliable services with minimal effort.

Processing all of the point of sale transaction data at the level of cities or countries is no small task. We need to deal with everything in real time in order to rapidly detect out-of-stock events and keep the search results accurate. We use Cloud Datastore for logging all of our transaction data. With a little bit of initial design work, we have a system that should scale pretty much indefinitely without us having to worry about it. Server load is concentrated around busy lunchtime and evening shopping hours, so being able to dynamically scale with demand is also helpful.

We use Compute Engine to manage our IoT device deployment, including over-the-air firmware updates and remote debugging. Our product search engine is also hosted on Compute Engine. We do a lot of batch processing of our transaction data logs to extract ranking signals, and process lots of external web data for additional product attributes and ranking signals. Everything is backed up nightly from Datastore to Google Cloud Storage for disaster recovery.

I built my last startup on AWS, so it was a little bit of a change to use Google Cloud this time around. However, it’s been a really great choice. It gives us a beautiful combination of scale and agility. We deploy to production often multiple times per day, which is extremely easy with the GCP tools. This lets us iterate rapidly, and focus on our product rather than system administration.  I suppose when you’re building a search engine, using Google’s infrastructure seems like an obvious choice :-)

What’s Next

It’s been a great experience so far, but we’re not close to the end. There’s still a long way to go to index every shop on the planet, after all. We’re getting there, and having fun along the way. If you’re interested, we’re always looking for good people.

Footnotes
  1. There’s always a few exotic ones that aren’t worth the trouble, but for practical purposes it might as well be 100% coverage.

Startups

Imagine you’re a road engineer and you’re designing an access road for a new town. The town will soon be built in a previously uninhabited area. You’ve managing the construction project, but unfortunately no one can tell you what the population of the town will be.

Taking your job seriously, you sit down to design the best road that you can build. You settle on constructing a seven lane highway with regular flyovers to minimize traffic. The road will be fully lit with a state-of-the-art LED lighting system. You add crash barriers and regularly spaced emergency telephones. After much consideration you decide to also include a rest area with parking and toilets. This involves designing a self-contained water and sewerage system, but it’s obviously worth it.

With three months to go until launch day, you discover problems with road drainage. After the panic subsides, the construction team agrees to work around the clock to refit a completely new system for surface water management. By a minor miracle, the work is completed on time.

Opening day finally arrives and the excitement is intense. Everyone agrees the finished product is an engineering marvel. The new town will have the best road in the world.

Unfortunately, it turns out that the town is a remote settlement with a population of 57. The road is mainly used by an old man and a donkey.

—————

The next year, you are again given a road construction project for another new town. Having learned your lesson, you build a modest single lane road. It’s well constructed but nothing special.

Opening day comes again, and it’s revealed that this time the “town” is in fact a major city with a population of 14 million. There are 50 mile tailbacks for six years before a larger road can be built. Your face appears on wanted posters throughout the nation, and you flee the country in disgrace.

—————

Twitter, I forgive you the Fail Whale. And I hope to always walk the middle *ahem* road.