Notes From The Field: CTO Weighs In On Big Data

Want to best understand the big data opportunities within reach of your portfolio companies? Then ask some of the world’s biggest enterprises about their most pressing needs.

This was my goal in attending this year’s Sierra Ventures CIO Summit, which began Wednesday afternoon in Silicon Valley. The event drew technology executives from companies such as PepsiCo, Samsung, Facebook and SAP.

Among the most interesting to offer his thought on big data was Mark Carges, CTO of eBay. Carges’ task is to mine business intelligence from the 250 million consumer queries and 75 billion database calls his site generates a day.

To do so, he has built a hadoop cluster, something many of his peers haven’t yet attempted. An informal survey at the event found roughly three quarters of companies doing early work with hadoop, but only a small handful with applications in production.

Carges said its efforts have generated a return on investment along with a better understanding what customers on the eBay site want. But while “I tell you it is worth the journey,” it is expensive, he says. It has meant storing hordes of behavioral data essentially forever in an attempt to improve site algorithms. Reducing cost is obviously one opportunity for young companies.

Another opportunity is developing software tools. EBay is constantly looking for tools and in some instances building tools for to further analysis on its hadoop cluster, says Carges.

A final one is finding expertise. Perhaps the biggest bottleneck so far is the availability of data scientists, says Carges. As a result, eBay is still in the market for small talent-driven acquisitions. That might include your company.

At eBay, studying transactional data is useful, but so is combing through behavioral data, such as how long a customer remains on a Web page before making a first click, or what prompted that first click, says Carges. It also is something many other American corporations will be doing in the coming years.

Photo courtesy of Shutterstock.