Data at Scale

Data at Scale

Dmitriy Ryaboy from Twitter discusses the challenges of scale.

Twitter’s fail whale days may be a thing of the past but it’s still fun to reminisce. For Dmitriy Ryaboy, head of Twitter’s 40-person Data Platform team, the 2010 World Cup was a particularly memorable and trying experience: “We would all sit in this one room and watch the game on one monitor and all of the internal Twitter metrics on another screen. Everyone was praying that neither team would score a goal.” Fast forward four years and Dmitriy’s organization – responsible for building all of the tools that Twitter’s data scientists, product managers, and engineers use to collect data inside Twitter – has come a long way, supporting real-time data analysis and powering new data driven products and new ways to provide recommendations to Twitter users.

In describing his experience helping Twitter navigate and outgrow the fail whale years, Dmitriy highlighted two key challenges of scale. The first, managing more and more data, is “reasonably straightforward” and the more “predictable scale problem.” The second challenge – scaling human capital – took Dmitriy “by surprise” and is “way harder than scaling petabytes.” He added, “When you grow to dozens of teams and hundreds of people you need to create systems and services so that data is discoverable and you don’t need to find the right person to get what you need…From the beginning we wanted everyone in the company to be able to access the data that they need to do their job.”

On the technical side, Dmitriy cited Twitter’s shift from Ruby on Rails to JVM, along with his organization’s switch to Scala, as a huge enabler. Dmitriy made the change “so that our engineers were all using the same language” and could access data directly without having to learn specialized technical skills. He also stressed that when building tools for internal usage, “It’s OK to have a product that’s more complex and takes longer to figure out. The tradeoff is that you can answer really hard questions and our focus is on making things that are impossible become possible.” That same approach doesn’t work as well on the consumer side, Dmitriy noted: for small businesses or general users, the questions tend to be “less complicated but what people want is very high SLAs.”

Turning to human capital challenges, Dmitriy focuses on building a culture of continuous learning and hiring people who are really smart, rather than people who are experts at one specific technology “because that technology will likely be outdated within a year.” With Twitter University – the company’s in-house training organization – Twitter has “a very structured way of onboarding people and teaching people new skills.” Dmitriy also touted Twitter’s significant investment in open source technologies: “We use open source technologies, we open source our own stuff, and when we hire new people they are already trained on our technology.”

Dmitriy made one surprising – at least to this correspondent – revelation when asked about the scale of Twitter’s analytic platform: “99.9% of data is stuff other than Tweets and the follow graph. Data related to how people interact with the product, the results of A/B experiments, etc., absolutely dwarf things like the social graph and Tweets.”

At the close of the session, Dmitriy fielded a final question from the audience and left us with this: “Elastic search is one technology that I wish we used more of and figured out how to add to our stack.” To the members of the Big Boulder community, we can’t help but ask, who’s working on that?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s