Open Sourcing Twitter Heron
Last year we announced the introduction of our new distributed stream computation system, Heron. Today we are excited to announce that we are open sourcing Heron under the permissive Apache v2.0 license. Heron is a proven, production-ready, real-time stream processing engine, which has been powering all of Twitter’s real-time analytics for over two years. Prior to Heron, we used Apache Storm, whic
Hadoop filesystem at Twitter
Twitter runs multiple large Hadoop clusters that are among the biggest in the world. Hadoop is at the core of our data platform and provides vast storage for analytics of user actions on Twitter. In this post, we will highlight our contributions to ViewFs, the client-side Hadoop filesystem view, and its versatile usage here. ViewFs makes the interaction with our HDFS infrastructure as simple as a
Diffy: Testing services without writing tests
Today, we’re excited to release Diffy, an open-source tool that automatically catches bugs in Apache Thrift and HTTP-based services. It needs minimal setup and is able to catch bugs without requiring developers to write many tests. Service-oriented architectures like our platform see a large number of services evolve at a very fast pace. As new features are added with each commit, existing code is
日本にエンジニアチームをつくります
2015年夏、Twitterは日本で次のステージに進みます。この動きのひとつが、日本にエンジニアチームをつくることです。日本のエンジニアチームについて、このチームの責任を負う、サンフランシスコ本社エンジニア部門のシニアディレクター アカッシュ・ガーグ (@akshgarg) に聞いてみました。 Q：日本にエンジニアチームをつくる理由 A：日本はTwitterにとって大きな市場のひとつです。2006年に始まったTwitterが日本語化されたのは2年後の2008年。日本語は、英語の次に初めてサポートを始めた言語です。2011年の震災直前に始まった日本法人は、米国外に初めてつくったオフィスでもあります。おかげさまでずっと好調に日本での利用者数を増やすことができていますが、一方で、まだまだ日本の皆さんに満足いただけるサービスにはなっていません。そこで、より満足いただけるよう、日本にエンジニアチーム
Building a complete Tweet index
Today, we are pleased to announce that Twitter now indexes every public Tweet since 2006. Since that first simple Tweet over eight years ago, hundreds of billions of Tweets have captured everyday human experiences and major historical events. Our search engine excelled at surfacing breaking news and events in real time, and our search index infrastructure reflected this strong emphasis on recency.
Using Twitter to measure earthquake impact in almost real time
Using Twitter to measure earthquake impact in almost real time At Twitter, we know that Tweets can sometimes travel as fast as an earthquake. We were curious to know just how accurate such a correlation might be, so we collaborated with Stanford researchers to model how Tweets can help create more accurate ShakeMaps, which provide near-real-time maps of ground motion and shaking intensity followin
Netty 4 at Twitter: Reduced GC Overhead
At Twitter, Netty (@netty_project) is used in core places requiring networking functionality. For example: Finagle is our protocol agnostic RPC system whose transport layer is built on top of Netty, and it is used to implement most services internally like Search TFE (Twitter Front End) is our proprietary spoon-feeding reverse proxy which serves most of public-facing HTTP and SPDY traffic using Ne
Observability at Twitter
As Twitter has moved from a monolithic to a distributed architecture, our scalability has increased dramatically. Because of this, the overall complexity of systems and their interactions has also escalated. This decomposition has led to Twitter managing hundreds of services across our datacenters. Visibility into the health and performance of our diverse service topology has become an important d
Streaming MapReduce with Summingbird
we’re thrilled to open source @summingbird, streaming mapreduce with @scalding and @stormprocessor #hadoop https://t.co/cV3LkCdCot — Twitter Open Source ( @TwitterOSS) September 3, 2013 Summingbird is a library that lets you write streaming MapReduce programs that look like native Scala or Java collection transformations and execute them on a number of well-known distributed MapReduce platforms li
New Tweets per second record, and how!
Recently, something remarkable happened on Twitter: On Saturday, August 3 in Japan, people watched an airing of Castle in the Sky, and at one moment they took to Twitter so much that we hit a one-second peak of 143,199 Tweets per second. (August 2 at 7:21:50 PDT; August 3 at 11:21:50 JST) To give you some context of how that compares to typical numbers, we normally take in more than 500 million Tw
Mesos Graduates from Apache Incubation
The Apache Software Foundation (ASF) has announced the graduation of Apache Mesos, the open source cluster manager that is used and heavily supported by Twitter, from its incubator. Mesos, which was released last year, acts as a layer of abstraction between applications and pools of servers, helping avoid the necessity of creating separate clusters to run individual frameworks and instead making i
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
