Wildcard launched our first product a few weeks ago – a card-based mobile browser. It’s a new way of browsing the internet on your smart phone, and it’s just the beginning. A group of incredibly talented people came together, designed and built the app over the past 12 months. I want to share a few things under the hood for those who are curious.
Browser
We started our journey with the goal of a thin client on iOS. Instead of making a runtime that interprets programming languages and specification of behaviors, we opted to build an app that recognizes “types” of cards and focus on the data. For example, the browser understands the a “product” card, or an “article” card. We did this because of the speed of implementation, and the risk of increasing complexity for a “card language”.
The effort involved, however, is far from the sound of a “thin client”. It’s carefully designed and built by our product / iOS team. Significant efforts has gone into things like custom animation, custom gesture, local caching, etc.
The Wildcard browser is written in Objective-C. We chose this because of implementation speed and user experience. Cross-compiling technologies like PhoneGap or RubyMotion break down quickly when you want to have custom UX. Even though this means we have to start from scratch for Android, I expect the Android UX to be quite different that code-sharing would be very difficult regardless.
Platform
When we started our journey, the mobile card ecosystem was in its nascent state. We started by decomposing webpages into their most essential components – which eventually evolved into our own concept of “mobile cards”. A platform was created to convert generic webpages into cards.
For example, a product page on Lululemon might have a search bar, the product details, a checkout button, and a locator to find the closest store. We would create a “product search” card, a “product” card that allows users to checkout, and a store locator card.
Many iterations have gone into the platform, and it has evolved into a framework to execute commands like “extract image url at this XPATH” or “Parse this piece of string into a proper U.S Address”. Many of the iterations have to do with getting to the right level of abstraction, so that humans can manipulate the data pipeline as well as computers (through layers of abstraction). These commands also became building blocks for more complex algorithms – and we are working on machine learning techniques to make the platform more robust.
The platform acts as a blackbox that turns inputs like an URL or a piece of structured data, into a mobile card. It’s configurable by humans and algorithms, it’s parallelizable across machines, and it’s at the core of our technical infrastructure. We chose Java and Dropwizard for a good trade off between performance and ease of implementation. The renaissance in the JVM stack gives us confidence in our technology choice.
Data Services
Wildcard works on many large-scale data problems. As the complexity of our system grew, we grouped together services that directly manipulate data. This set of services includes things like RSS feed readers, small web crawlers, one-off scripts for data processing, and GUI tools to allow direct data manipulation. This is a high-touch/offline environment, where a lot of “raw data” gets processed into a state that can be fed into the platform through a simple API.
Having an offline environment gave us the ability to move quickly without worrying about the stability of our platform. Things can get a little messy if you are not careful, so following good software engineering practices is extra important here. We use a mixture of different technologies in this system, with the dominant languages being Ruby and Javascript.
Search Engine
As we built more and more cards, we started to realize the importance of search. Search is often the best way to expose a large variety of experiences to users. In our case, it has become an essential component of our product.
Building a search engine from scratch is no easy task. We started out with a vertical approach – focusing on e-commerce first. We used techniques like LSA to map queries onto commerce brands, rank the brands, and automatically kick off search within the brands. This is a powerful feature because users will always get the most up-to-date search results from the brands, and we wouldn’t have to do a massive/costly web crawl.
Since then, we have added many more result types into our search index. We have also built a scalable and configurable crawler (across the brands that use mobile cards). Search is the fastest evolving service in our stack, and we are constantly trying to expand our capacity / use cases.
We currently use a combination of Redis, Elasticsearch and different python scripts for search. As the index sizes continue to grow, more scalable solutions will be needed.
Integration Systems
One of the goals of Wildcard is to make our mobile card platform “writable”. This means any content publisher, retailer, service provider is able to take advantage of the system and control their own cards. To make this as easy as possible, we’ve created a product just for that. It allows anyone to publish to Wildcard, Twitter and Pinterest.
We built a schema translation layer in Java to support this product. We also built tools like this to help developers validate and visualize their own card implementations.
Dev Ops
As our engineering team grew, automation became more and more important. We started out with using Chef, Jenkins on top of AWS, and have since adapted an concoction of automation tools.
At the core, we use AWS as the base layer of our infrastructure. We use Chef for server configuration, terraform for orchestration, docker for resource management, graphite/keen for metrics and monitoring, consul for async jobs (like deployment), github/circleci for continuous integration, hubot to tie everything together, and probably many more tools I forgot to mention.
Automation has been a game changer in our daily engineering lives. It’s a significant investment – and it’s necessary. When done well, it can be a significant productivity booster for the engineering team.
As our 15-person product/engineering team continue to grow, our technology will continue to evolve and adapt to new needs. If you are interested in anything discussed here, shoot me a line and I’m happy to chat more.