Dmitriy Ryaboy from Twitter discusses the challenges of scale.
Twitter’s fail whale days may be a thing of the past but it’s still fun to reminisce. For Dmitriy Ryaboy, head of Twitter’s 40-person Data Platform team, the 2010 World Cup was a particularly memorable and trying experience: “We would all sit in this one room and watch the game on one monitor and all of the internal Twitter metrics on another screen. Everyone was praying that neither team would score a goal.” Fast forward four years and Dmitriy’s organization – responsible for building all of the tools that Twitter’s data scientists, product managers, and engineers use to collect data inside Twitter – has come a long way, supporting real-time data analysis and powering new data driven products and new ways to provide recommendations to Twitter users.
In describing his experience helping Twitter navigate and outgrow the fail whale years, Dmitriy highlighted two key challenges of scale. The first, managing more and more data, is “reasonably straightforward” and the more “predictable scale problem.” The second challenge – scaling human capital – took Dmitriy “by surprise” and is “way harder than scaling petabytes.” He added, “When you grow to dozens of teams and hundreds of people you need to create systems and services so that data is discoverable and you don’t need to find the right person to get what you need…From the beginning we wanted everyone in the company to be able to access the data that they need to do their job.”
On the technical side, Dmitriy cited Twitter’s shift from Ruby on Rails to JVM, along with his organization’s switch to Scala, as a huge enabler. Dmitriy made the change “so that our engineers were all using the same language” and could access data directly without having to learn specialized technical skills. He also stressed that when building tools for internal usage, “It’s OK to have a product that’s more complex and takes longer to figure out. The tradeoff is that you can answer really hard questions and our focus is on making things that are impossible become possible.” That same approach doesn’t work as well on the consumer side, Dmitriy noted: for small businesses or general users, the questions tend to be “less complicated but what people want is very high SLAs.”
Turning to human capital challenges, Dmitriy focuses on building a culture of continuous learning and hiring people who are really smart, rather than people who are experts at one specific technology “because that technology will likely be outdated within a year.” With Twitter University – the company’s in-house training organization – Twitter has “a very structured way of onboarding people and teaching people new skills.” Dmitriy also touted Twitter’s significant investment in open source technologies: “We use open source technologies, we open source our own stuff, and when we hire new people they are already trained on our technology.”
Dmitriy made one surprising – at least to this correspondent – revelation when asked about the scale of Twitter’s analytic platform: “99.9% of data is stuff other than Tweets and the follow graph. Data related to how people interact with the product, the results of A/B experiments, etc., absolutely dwarf things like the social graph and Tweets.”
At the close of the session, Dmitriy fielded a final question from the audience and left us with this: “Elastic search is one technology that I wish we used more of and figured out how to add to our stack.” To the members of the Big Boulder community, we can’t help but ask, who’s working on that?