Introducing and Open Sourcing Ambry
Introducing and Open Sourcing AmbryLinkedIn’s New Distributed Object Store Media content has become ubiquitous around the web and almost all of Linkedin's new features interact with media in some form or the other. Profile photos, email attachments, logos, and influencer posts are a few examples of where photos, videos, PDFs, and other media types get uploaded and displayed to the end user. These
Kafka Ecosystem at LinkedIn
Apache Kafka is a highly scalable messaging system that plays a critical role as LinkedIn’s central data pipeline. Kafka was developed at LinkedIn back in 2010, and it currently handles more than 1.4 trillion messages per day across over 1400 brokers. Kafka’s strong durability and low latency have enabled us to use Kafka to power a number of newer mission-critical use cases at LinkedIn. These incl
Kafka at LinkedIn: Current and Future
The LinkedIn engineering team has developed and built Apache Kafka into a powerful open source solution for managing streams of information. We use Kafka as the messaging backbone that helps the company’s applications work together in a loosely coupled manner. LinkedIn relies heavily on the scalability and reliability of Kafka and a surrounding ecosystem of both open source and internal components
Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)
Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines) I wrote a blog post about how LinkedIn uses Apache Kafka as a central publish-subscribe log for integrating data between applications, stream processing, and Hadoop data ingestion. To actually make this work, though, this "universal log" has to be a cheap abstraction. If you want to use a system as a central data hub
The Log: What every software engineer should know about real-time data's unifying abstraction | LinkedIn Engineering
The Log: What every software engineer should know about real-time data's unifying abstraction I joined LinkedIn about six years ago at a particularly interesting time. We were just beginning to run up against the limits of our monolithic, centralized database and needed to start the transition to a portfolio of specialized distributed systems. This has been an interesting experience: we built, dep
Auto-Scaling with Apache Helix and Apache YARN
At LinkedIn, many individual services integrate together to deliver a reliable and consistent end-user experience. Although each service handles a specialized set of responsibilities, they all share a common set of required features such as load-balancing, dynamic reconfiguration, health monitoring, and fault-detection. Last year we introduced Apache Helix, an open-source generic cluster managemen
Optimizing Linux Memory Management for Low-latency / High-throughput Databases
Optimizing Linux Memory Management for Low-latency / High-throughput Databases Co-author: Cuong Tran Table of Contents Introduction Setting up the context Reproducing and understanding Linux's zone reclaim behavior NUMA memory rebalancing also triggers direct page scans Lessons learned Introduction GraphDB is the storage layer of LinkedIn's real-time distributed social graph service. Our service h
SNA Projects Blog : Tech Talk: Michael Deerkoski (Flickr) — “Continuous Deployment at Flickr”
LinkedIn operates the world’s largest professional network with more than 645 million members in over 200 countries and territories. This team builds distributed systems that collect, manage and analyze this digital representation of the world's economy, while our AI experts, data scientists and researchers conduct applied research that fuel LinkedIn’s data-driven products and provide insights tha
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Dynamometer: Scale Testing HDFS on Minimal Hardware with Maximum Fidelity
Project Falco: Decoupling Switching Hardware and Software