サクサク読めて、アプリ限定の機能も多数!
トップへ戻る
掃除・片付け
engineering.linkedin.com
How LinkedIn customizes Apache Kafka for 7 trillion messages per day Co-authors: Jon Lee and Wesley Wu Apache Kafka is a core part of our infrastructure at LinkedIn. It was originally developed in-house as a stream processing platform and was subsequently open sourced, with a large external adoption rate today. While many other companies and projects leverage Kafka, few—if any—do so at LinkedIn’s
The AI Behind LinkedIn Recruiter search and recommendation systems Co-authors: Qi Guo, Sahin Cem Geyik, Cagri Ozcaglar, Ketan Thakkar, Nadeem Anjum, and Krishnaram Kenthapadi LinkedIn Talent Solutions serves as a marketplace for employers to reach out to potential candidates and for job seekers to find career opportunities. A key mechanism to help achieve these goals is the LinkedIn Recruiter prod
Co-authors: Mars Lan, Seyi Adebajo, Shirshanka Das Editor’s note: Since publishing this blog post, the team open sourced DataHub in February 2020. You can read more on the journey of open sourcing the platform here. As the operator of the world’s largest professional network and the Economic Graph, LinkedIn’s Data team is constantly working on scaling its infrastructure to meet the demands of our
The pursuit of our mission to connect the world’s professionals to make them more productive and successful is deeply dependent on the technology and infrastructure we build and maintain. Ten years ago, we had 50 million members. Fast forward five years and that number jumped to 300 million. Today, we have more than 645 million members, 20 million jobs and someone is hired using LinkedIn every eig
Open sourcing Brooklin: Near real-time data streaming at scale Editor's note: This blog has been updated. Brooklin—a distributed service for streaming data in near real-time and at scale—has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, we are pleased to announce the open-sourcing of Brooklin and that the source c
Editor’s note: The use of AI in LinkedIn products has been the subject of multiple press articles and research papers (some highlighted on this blog). With the release of a new LinkedIn Learning course about AI at LinkedIn, we asked our Head of AI, Deepak Agarwal, for a brief overview of what AI is and how it works, geared towards people who are interested in this growing field. In this post, we d
At LinkedIn, Kafka is the de-facto messaging platform that powers diverse sets of geographically-distributed applications at scale. Examples include our distributed NoSQL store (Espresso), stream processing framework (Samza), monitoring infrastructure (InGraphs), and derived data serving platform (Venice). Given these use cases, it’s not surprising that Kafka usage at LinkedIn has grown exponentia
CSS at Scale: LinkedIn’s New Open Source Projects Take on Stylesheet Performance Browsers use Cascading Style Sheets (CSS) to control the appearance of websites. From borders, fonts, and colors to layout, images, and animations, there are roughly 500 different style properties that can be declared with CSS. These properties are what make the visual diversity of the internet possible for your handh
Lighter than Lightweight: How We Built the Same App Twice with Preact and Glimmer.js The beauty of the web is that there is no “install” step. Someone, somewhere taps a link to your site, and moments later it appears instantly in front of them. At least, that’s the idea—but not all devices and networks are created equally. Sites that feel fast on a desktop computer with broadband can feel downrigh
Co-authors: Bruno Connelly and Bhaskaran Devaraj Being a Site Reliability Engineer (SRE) means having to talk about hard problems. Site outages, complex failure scenarios, and other technical emergencies are the things we have to be prepared to deal with every day. When we’re not dealing with problems, we’re discussing them. We regularly perform post-mortems and root cause analyses, and we general
LinkedIn recently released Project Voyager, our codename for the new version of our flagship application for Android, iOS, and mobile web. Voyager is the result of more than a year of product development work by over 250 engineers. We took the opportunity to rethink the LinkedIn experience from the ground up, not only from a product perspective, but on the engineering side as well. Before Voyager,
Application Pauses When Running JVM Inside Linux Control GroupsCauses and Solutions Linux cgroups-based solutions (e.g., Docker, CoreOS) are increasingly being used to host multiple applications on the same host. We have been using cgroups at LinkedIn to build our own containerization product called LPS (LinkedIn Platform as a Service) and to investigate the impact of resource-limiting policies on
Instant Messaging at LinkedIn: Scaling to Hundreds of Thousands of Persistent Connections on One Machine Coauthor: Cliff Snyder We recently introduced Instant Messaging on LinkedIn, complete with typing indicators and read receipts. To make this happen, we needed a way to push data from the server to mobile and web clients over persistent connections instead of the traditional request-response par
Don’t Let Linux Control Groups Run UncontrolledAddressing Memory-Related Performance Pitfalls of Cgroups Coauthors: Cuong Tran and Jerry Weng Summary The Linux kernel feature of cgroups (Control Groups) is being increasingly adopted for running applications in multi-tenanted environments. Many projects (e.g., Docker and CoreOS) rely on cgroups to limit resources such as CPU and memory. Ensuring th
Introducing {py}gradle, an open source Python plugin for Gradle Image Credit: Yiying Lu, Copyright: Gradle Inc. Gradle is a build automation tool that supports many programming languages. It coordinates the process of building software between multiple different code repositories and automates a number of important related tasks, like checking dependencies and warning programmers if something they
Open Sourcing Test ButlerReliable Android Testing, at Your Service Automated testing is a key component to LinkedIn’s 3x3 strategy for releasing mobile applications. As we developed the new LinkedIn Android app, we found that our tests had a major problem: our testing environment was unreliable, so our tests failed intermittently. We needed a solution that would let us rely on our tests to inform
Fast performance is a key feature of LinkedIn’s mobile applications. So when we first released the new LinkedIn iOS app, and we learned that our members were experiencing noticeable delays when viewing certain profiles, we immediately investigated the issue to find a solution. Performance profiling revealed that the main thread was spending a significant amount of time running Auto Layout. Auto La
We live in an age where we want to know relevant things happening around the world as soon as they happen; an age where digital content is updated instantly based on our likes and dislikes; an age where credit card fraud, security breaches, device malfunctions and site outages need to be detected and remedied as soon as they happen. It is an age where events are captured at scale and processed in
Open Sourcing Photon MLLinkedIn’s Scalable Machine Learning Library for Spark Machine learning is a key component of LinkedIn’s relevance-driven products. We use machine learning to train the ranking algorithms for our feed, advertising, recommender systems (such as People You May Know), email optimization, search engines, and more. For an in-depth example, check out these posts (part one and two)
Introducing and Open Sourcing AmbryLinkedIn’s New Distributed Object Store Media content has become ubiquitous around the web and almost all of Linkedin's new features interact with media in some form or the other. Profile photos, email attachments, logos, and influencer posts are a few examples of where photos, videos, PDFs, and other media types get uploaded and displayed to the end user. These
Designing SSD-Friendly ApplicationsFor Better Application Performance and Higher IO Efficiency These days, solid state drives (SSD) are being increasingly adopted to alleviate the I/O performance bottlenecks of applications. Numerous measurement results have showcased the performance improvement brought by SSD as compared with hard disk drives (HDD). However, in most deployment scenarios, a SSD is
Apache Kafka is a highly scalable messaging system that plays a critical role as LinkedIn’s central data pipeline. Kafka was developed at LinkedIn back in 2010, and it currently handles more than 1.4 trillion messages per day across over 1400 brokers. Kafka’s strong durability and low latency have enabled us to use Kafka to power a number of newer mission-critical use cases at LinkedIn. These incl
Open Sourcing Dr. ElephantSelf-Serve Performance Tuning for Hadoop and Spark We are proud to announce today that we are open sourcing Dr. Elephant, a powerful tool that helps users of Hadoop and Spark understand, analyze, and improve the performance of their flows. We first presented Dr. Elephant to the community last year during the eighth annual Hadoop Summit, a leading conference for the Apache
In modern data-driven businesses, the complexity that arises from fast-paced analytics, data mining and ETL processes makes metadata increasingly important. In this blog post, we share our own journey and a new open source effort that aims to boost productivity and data provenance. WhereHows, a project of the LinkedIn Data team, works by creating a central repository and portal for the processes,
Eliminating Large JVM GC Pauses Caused by Background IO Traffic Coauthor: Cuong Tran, Systems Architect In our production environments, we have repeatedly seen that applications running in JVM (Java Virtual Machine) occasionally experience large STW (Stop-The-World) application pauses due to JVM’s GC logging being blocked by background IO traffic (e.g., OS page cache writeback). During such STW
Open-sourcing PalDB, a lightweight companion for storing side data LinkedIn’s data products give our members recommendations, analytics and insights. As we continue to invent more products that leverage data for our members, we need to push the envelope in our data processing capabilities and do more with less. One issue that often comes up is what to do to improve the usability and memory efficie
Background LinkedIn’s FeatureFu project is a new open source toolkit designed to enable creative and agile feature engineering for most machine learning tasks such as statistical modeling (classification, clustering, and regression) and rule-based decision engines. In this blog post, we will detail the design and implementation of Expr in FeatureFu, provide examples of how feature engineering is b
LinkedIn started in 2003 with the goal of connecting to your network for better job opportunities. It had only 2,700 members the first week. Fast forward many years, and LinkedIn’s product portfolio, member base, and server load has grown tremendously. Today, LinkedIn operates globally with more than 350 million members. We serve tens of thousands of web pages every second of every day. We've hit
At LinkedIn, we are always looking for the best software development frameworks and tools to build great products. During our 11-year history we have adopted many web frameworks for UI development - among them Grails, Frontier (Linkedin's internal web framework), and most recently, Play. We love Play and are excited to increase our adoption of it across the company. As we evolve and scale our inte
次のページ
このページを最初にブックマークしてみませんか?
『LinkedIn Engineering』の新着エントリーを見る
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く