Table of ContentThanos Coding Style Guide #This document details the official style guides for the various languages we use in the Thanos project. Feel free to familiarize yourself with and refer to this document during code reviews. If something in our codebase does not match the style, it means it was missed or it was written before this document. Help wanted to fix it! (: Generally, we care abo
Retrieval Augmented Generation at scale — Building a distributed system for synchronizing and…
Disclaimer: We will go into some technical and architectural details of how we do this at Neum AI — A data platform for embeddings management, optimization, and synchronization at large scale, essentially helping with large-scale RAG. As we’ve shared in other blogs in the past, getting a Retrieval Augmented Generation (RAG) application started is pretty straightforward. The problem comes when tryi
I have been thinking a lot about Envoy Proxy control planes recently so I thought it would be useful to share some of my conclusions about the industry status quo and where I envision the state of the art progressing over the next few years. The current lay of the landAlthough I am continuously talking to Envoy users, our now yearly EnvoyCon/KubeCon has become an important check-in point for me as
Shibboleths are historically a word or phrasing that indicate membership in a particular group or culture. I was introduced to the term in the West Wing where the President needed to verify the veracity of a person’s claims of religious persecution. I am still a relatively new engineer in the field of distributed systems, having only studied and worked in the field for around a decade, but in that
GitHub - linkedin/school-of-sre: At LinkedIn, we are using this curriculum for onboarding our entry-level talents into the SRE role.
Site Reliability Engineers (SREs) sits at the intersection of software engineering and systems engineering. While there are potentially infinite permutations and combinations of how infrastructure and software components can be put together to achieve an objective, focusing on foundational skills allows SREs to work with complex systems and software, regardless of whether these systems are proprie
GitHub - timlrx/tailwind-nextjs-starter-blog: This is a Next.js, Tailwind CSS blogging starter template. Comes out of the box configured with the latest technologies to make technical writing a breeze. Easily configurable and customizable. Perfect as a re
Demo Blog - this repo My personal blog - modified to auto-generate blog posts with dates Karhdo's Blog - Karhdo's Blog - Karhdo's Coding Adventure (source code) ben.codes blog - Benoit's personal blog about software development (source code) tsix blog - A front-end engineer is used to record some knowledge points in work and study 中文 SOTO's Blog - A more personalized personal website upgraded from
Cover Photo by Alexander Shatov on Unsplash Netflix accounts for about 15% of the world's internet bandwidth traffic. Serving over 6 billion hours of content per month, globally, to nearly every country in the world. Building a robust, highly scalable, reliable, and efficient backend system is no small engineering feat but the ambitious team at Netflix has proven that problems exist to be solved.
Chronofold is a replicated data structure for versioned text. It is designed for use in collaborative editors and revision control systems. Past models of this kind either retrofitted local linear orders to a distributed system (the OT approach) or employed distributed data models locally (the CRDT approach). That caused either extreme fragility in a distributed setting or egregious overheads in l
論文翻訳: In Search of an Understandable Consensus Algorithm (Extended Version) Abstract Raft は複製されたログを管理するためのコンセンサスアルゴリズムである。これは (Multi-) Paxosと同等の結果を生み出し Paxos と同程度に効率的だが、その構造は Paxos とは異なる; Raft によって Paxos よりも理解しやすく実用的なシステムを構築するためのより良い基盤が提供される。わかりやすさを高めるために Raft はリーダー選挙、ログ複製、安全性といった合意の重要な要素を分離し、考慮すべき状態の数を減らすためにより強力な一貫性を強制する。ユーザ調査の結果から Raft は Paxos よりも習得しやすいことを示している。Raft にはクラスタメンバーシップを変更するための新しいメカニズム
On distributed systems broadly defined and other curiosities. The opinions on this site are my own. I talked about the importance of reading foundational papers last week. To followup, here is my compilation of foundational papers in the distributed systems area. (I focused on the core distributed systems area, and did not cover networking, security, distributed ledgers, verification work etc. I e
2019年振り返り
2019年のアウトプットとインプットを簡単に振り返っておく． Working 業務でのチームとしてのアウトプットはMercari Microservices Platformの進捗（2019年）にまとめた．前年に引き続きPlatformの開発と運用を続けている． 昨年はAPI gatewayの開発など自分で手を動かすことが多かったが，今年は自分が具体的なプロジェクトを持ち自ら手を動かすことは意識的に少なくし，Tech leadとしてチームのアウトプットをどのように最大化にするか?ということを常に考えていた．技術的な視点や意思決定も時間的に影響範囲的により広く見るように意識し始めた（インプットも組織やチームに関連するものが多くなった）．見えやすいアウトプットは少ないが，プロジェクトを進めつつ，これまで曖昧だったPlatformのMissionは何かを明確に定義し，チームが拡大しても皆が同じ方
FoundationDB: A Distributed Key-Value Store – Communications of the ACM
CACM Web Account Membership in ACM includes a subscription to Communications of the ACM (CACM), the computing industry's most trusted source for staying connected to the world of advanced computing. Sign In Sign Up FoundationDB is an open-source transactional key-value store created more than 10 years ago. It is one of the first systems to combine the flexibility and scalability of NoSQL architect
最初に結論。 Example のコード そもそもログの目的とは何なのか。 W3C Trace Context Trace Context の HTTP ヘッダとフォーマット Traceparent Header version version-format trace-id parent-id trace-flags 用語 Distributed trace 要するに。 System.Diagnostics.Activity ConsoleFormatterOptions.IncludeScopes HttpClient 内での Activity のイベントを検知してログに出す。 まとめ References 最初に結論。 この記事自体はアレコレ書いていますが、結論に至るまでが長いので、先に結論というか言いたい事から入ります。 とりあえず覚えておいて欲しいのはたった1つです。 それは「Co
GitHub - featureform/featureform: The Virtual Feature Store. Turn your existing data infrastructure into a feature store.
Featureform is a virtual feature store. It enables data scientists to define, manage, and serve their ML model's features. Featureform sits atop your existing infrastructure and orchestrates it to work like a traditional feature store. By using Featureform, a data science team can solve the following organizational problems: Enhance Collaboration Featureform ensures that transformations, features,
CSE 138: Distributed Systems
From the beginning of this year, I started to take lecture courses of undergrad distributed systems course at UC Santa Cruz (CSE 138) by Lindsey Kuper. It consists of 23 lectures (you can see the schedule of topics from here) and recently I’ve finished all of them. I’m not a student at UCSC but due to the COVID-19 situation, all these lectures were delivered online and are available on YouTube. So
the morning paper a random walk through Computer Science research, by Adrian Colyer Made delightfully fast by strattic Meaningful availability, Hauer et al., NSDI’20 With thanks to Damien Mathieu for the recommendation. This very clearly written paper describes the Google G Suite team’s search for a meaningful availability metric: one that accurately reflected what their end users experienced, and
The Twelve-factor app is a methodology for building software-as-a-service apps that was first formulated by developers associated with Heroku. It's been ten years since the first presentation of this methodology. Despite the criticism that it is only applicable to Heroku and similar webapp services, it remains a relevant yard stick for software-as-a-service development. Some of its tenets have bee
2024/02/07A Distributed Systems Reading ListThis document contains various resources and quick definition of a lot of background information behind distributed systems. It is not complete, even though it is kinda sorta detailed. I had written it some time in 2019 when coworkers at the time had asked for a list of references, and I put together what I thought was a decent overview of the basics of
Microservice and Transaction Management3 : 合意理論からみる2Phase CommitとMicroservice - Qiita
Microservice and Transaction Management3 : 合意理論からみる2Phase CommitとMicroservicemicroservicesTransactionCAPSAGApaxos はじめに 日立製作所クラウドビジネス推進センタの西谷淳平です。今までの連載の中では、分散トランザクションのACID特性と、その背後に潜む分散トランザクションの原則であるCommitment Orderingに着目し、Microservicesで流行っているSagaデザインパタンの整合性担保能力を語りました。その中で、SagaデザインパタンはIsolationだけではなく、A/Cの特性が失われているという事実を指摘し、その結果としてデータ破壊が起こり得るということを説明しました。 そして、Sagaにおける並行制御機能の欠落を埋めるには2Phase Lockの仕組みを導
Photo by Ryan Parker / Unsplash At Narrator we support many data warehouses, including Postgres. Though it was designed for production systems, with a little tweaking Postgres can work extremely well as a data warehouse. For those that want to cut to the chase, here's the tl;dr don't use the same server as your production system upgrade to pg 12+ (or avoid common table expressions in your queries)
Database Fundamentals
About a year ago, I tried thinking which database I should choose for my next project, and came to the realization that I don't really know the differences of databases enough. I went to different database websites and saw mostly marketing and words I don't understand. This is when I decided to read the excellent books Database Internals by Alex Petrov and Designing Data-Intensive Applications by
CeleryのMessage Priorities機能を利用した処理遅延の低減 - Nealle Developer's Blog
こんにちは、SREチームの宮後(@miya10kei)です。 バイクに乗っていて気持ちが良い季節になってきましたね🌸 メッセージキューを利用した非同期タスクを扱っていて、誰しも優先度順にタスクを処理したいなと思ったことがあるのではないでしょうか？ 今回はCeleryの機能を利用して実現することができたので紹介したいと思います。 Celeryってなに？ Celeryは分散メッセージキュー機能を提供するPythonベースのOSSです。 メッセージキューのBrokerとしてRedisやRabbitMQ、Amazon SQSなどを使用でき、分散環境での非同期タスクの実行を実現しています。 公式サイトを引用すると次の説明になりますね。 Celery is a simple, flexible, and reliable distributed system to process vast amou
Monitoring resource usage in a cloud data warehouse | Google Cloud Blog
Marco TranquillinCustomer Engineering Manager for Data Analytics Google BigQuery is a fully managed serverless solution for your enterprise data warehouse workloads. Nothing could be easier than that: Just upload your data via batch or streaming and start running your queries. The underlying system will seamlessly take care of providing the infrastructural resources needed to complete your differe
週刊Railsウォッチ（20210126後編）Google Cloud FunctionsがRubyをサポート、Ruby 3のパターンマッチングでポーカーゲームほか｜TechRacho by BPS株式会社
2021.01.26 週刊Railsウォッチ（20210126後編）Google Cloud FunctionsがRubyをサポート、Ruby 3のパターンマッチングでポーカーゲームほか こんにちは、hachi8833です。 各記事冒頭には⚓でパーマリンクを置いてあります: 社内やTwitterでの議論などにどうぞ 「つっつきボイス」はRailsウォッチ公開前ドラフトを（鍋のように）社内有志でつっついたときの会話の再構成です👄 お気づきの点がありましたら@hachi8833までメンションをいただければ確認・対応いたします🙇 TechRachoではRubyやRailsの最新情報などの記事を平日に公開しています。TechRacho記事をいち早くお読みになりたい方はTwitterにて@techrachoのフォローをお願いします。また、タグやカテゴリごとにRSSフィードを購読することもできます
Containerize and deploy a gRPC application on AWS Fargate | Amazon Web Services
AWS Open Source Blog Containerize and deploy a gRPC application on AWS Fargate These days, building an application distributed among processes, hosts, or even networks is much easier. This is partially due to an evolution in the protocols used to negotiate between different components of an application or service. This evolution is largely the result of the popularity of both the TCP and HTTP prot
Queues may lead to bottlenecks, both for sheep and background jobs.Over the past year at Gusto, we’ve drastically changed how we approach Sidekiq in our main Rails monolith. Our changes have reduced costs, reduced the number of outages, made it simpler for product engineers to use Sidekiq, and improved the customer experience. This blog post will go through the different aspects of our Sidekiq tra
EngineeringMeasuring Git performance with OpenTelemetryUse our new open source Trace2 receiver component and OpenTelemetry to capture and visualize telemetry from your Git commands. When I think about large codebases, the repositories for Microsoft Windows and Office are top of mind. When Microsoft began migrating these codebases to Git in 2017, they contained 3.5M files and a full clone was more
Data-Oriented Architecture
There’s a little-known pattern in software architecture that deserves more attention. Data-Oriented Architecture was first described by Rajive Joshi in a 2007 whitepaper at RTI, and again in 2017 by Christian Vorhemus and Erich Schikuta at the University of Vienna in this iiWAS paper. DOA is an inversion of the traditional dichotomy between a monolithic binary and data store (monolithic architectu
Archived Reliability Pillar AWS Well-Architected Framework This paper has been archived. The latest version is now available at: https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html Archived Reliability Pillar AWS Well-Architected Framework Reliability Pillar: AWS Well-Architected Framework Copyright © 2020 Amazon Web Services, Inc. and/or its affiliates. All rights res
AWS Open Source Blog Innovating with Rust The Rust programming language is an open source project started by Mozilla Research more than a decade ago. Since then, more than 5,000 people have contributed to the Rust project, and the language has had a resounding impact on technology. Technology companies, large and small, are using and benefiting from Rust. The Rust language builds on the superpower
Go support for AWS X-Ray now available in AWS Distro for OpenTelemetry | Amazon Web Services
AWS Open Source Blog Go support for AWS X-Ray now available in AWS Distro for OpenTelemetry In this blog post, AWS interns Wilbert Guo and Kelvin Lo share their experience in enhancing the OpenTelemetry Go SDK to support sending traces to AWS X-Ray. These enhancements are also available in the AWS Distro for OpenTelemetry. AWS X-Ray is a service that collects data and provides tools that allow us
Time is the only real currency we have Suggestions on spending it on things that matter and avoiding things that don't A Note on ‘Software Engineering Stuff’ Stressed And Struggling For Time Things That Matter (Time Savers) Learn to type Fast TurboCharge your Dev Environment Automate Know your infrastructure Things That Don’t Matter (Time Sinks) My Language is the best (Or, your language sucks) ‘X
Valley of the Temples, Agrigento, AG, Italy. Photo by Dario Crisafulli on Unsplash.In the past few years, much has been talked and written about the “three pillars of observability”: metrics, logs, and traces. A Google search for the phrase brings up over 7,000 results, with almost every observability vendor having a blog post or an e-book on the topic. Recently, the term MELT started showing up t
Approximately a year ago, I had the opportunity to work on Sigma — a large, distributed system that protects Facebook users from spam and other kinds of abuse. One reason it was a pretty unique experience is that Sigma is almost entirely a Haskell codebase. It was the first time I got to work with the language in a professional setting, so I was eager to see how it performs in a real-world, produc
Monitoring Amazon DynamoDB for operational awareness | Amazon Web Services
AWS Database Blog Monitoring Amazon DynamoDB for operational awareness Amazon DynamoDB is a serverless database, and is responsible for the undifferentiated heavy lifting associated with operating and maintaining the infrastructure behind this distributed system. As a customer, you use APIs to capture operational data that you can use to monitor and operate your tables. This post describes a set o
In Search of an Understandable Consensus Algorithm (Extended Version) Diego Ongaro and John Ousterhout Stanford University Abstract Raft is a consensus algorithm for managing a replicated log. It produces a result equivalent to (multi-)Paxos, and it is as efficient as Paxos, but its structure is different from Paxos; this makes Raft more understandable than Paxos and also provides a better foundat
NDSS2020で「スマホアプリは会話を盗聴できる」と発表された。音声復元の精度は驚異の90%｜中国情報局＠北京オフィス
アメリカのサンディエゴで開催されたサイバーセキュリティ分野でとても評価が高い国際会議「NDSS（ISOC Network and Distributed System Security Symposium）2020」において、浙江大学、カナダのMcGill大学、トロント大学の科学者で構成される研究チームの発表が中国で話題になってます。 発表によるとこうです。 ... they discovered a new way of attack on a smartphone. Specifically speaking, an app in a smartphone can employ its built-in accelerometer to eavesdrop on the speaker by recognizing the speech emitted by the speaker a
This is an introductory course in Distributed Systems. Distributed systems is the study of how to build a computer system where the state of the program is divided over more than one machine (or "node"). This course is in active development. At the moment, it consists of a series of short videos. The intention is to create a complete set of video lectures and then add additional content (such as m
Kubernetes README
Kubernetes READMEPlease submit for books, tutorials, or other assets that would be useful to folks using Kubernetes. PHIPPY AND FRIENDSBy: Matt Butcher, Karen Chu & Bailey Beougher The characters Phippy, Captain Kube, Goldie, and Zee and the two books are owned by The Linux Foundation, on behalf of the Cloud Native Computing Foundation, and licensed under the Creative Commons Attribution License (
Reverse Proxy, HTTP Keep-Alive Timeout, and sporadic HTTP 502sHow HTTP Keep-Alive can cause TCP race condition These mysterious HTTP 502s happened to me already twice over the past few years. Since the amount of service-to-service communications every year goes only up, I expect more and more people to experience the same issue. So, sharing it here. TL;DR: HTTP Keep-Alive between a reverse proxy a
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Thanos - Highly available Prometheus setup with long term storage capabilities
On the state of Envoy Proxy control planes
Distributed Systems Shibboleths | Joey Lynch's Site
Netflix System Design- Backend Architecture
Chronofold: a data structure for versioned text
論文翻訳: In Search of an Understandable Consensus Algorithm (Extended Version) - MOXBOX #Raft
Foundational distributed systems papers
【C#】ASP.NET Core と W3C Trace Context とお手軽ロギング。 - ねののお庭。
Meaningful availability | the morning paper
Twelve-factor app anno 2022 | Xenit
A Distributed Systems Reading List
Using PostgreSQL as a Data Warehouse
Scaling Sidekiq at Gusto
Measuring Git performance with OpenTelemetry
Reliability Pillar AWS Well-Architected Framework
Innovating with Rust | Amazon Web Services
Time is the only real currency we have
TEMPLE: Six Pillars of Observability
A Haskell retrospective
In Search of an Understandable Consensus Algorithm
Distributed Systems Course
How HTTP Keep-Alive can cause TCP race condition