GitHub - juicedata/juicefs: JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a high-performance POSIX file system released under Apache License 2.0, particularly designed for the cloud-native environment. The data, stored via JuiceFS, will be persisted in Object Storage (e.g. Amazon S3), and the corresponding metadata can be persisted in various compatible database engines such as Redis, MySQL, and TiKV based on the scenarios and requirements. With JuiceFS, mass
【書き起こし】Payment distributed transaction case study – Rui Gao【Merpay Tech Fest 2021】 | メルカリエンジニアリング
【書き起こし】Payment distributed transaction case study – Rui Gao【Merpay Tech Fest 2021】 Merpay Tech Fest 2021は、事業との関わりから技術への興味を深め、プロダクトやサービスを支えるエンジニアリングを知れるお祭りで、2021年7月26日（月）からの5日間、開催しました。セッションでは、事業を支える組織・技術・課題などへの試行錯誤やアプローチを紹介していきました。 この記事は、「Payment distributed（分散）transaction case study」の書き起こしです。 Rui Gao氏：皆さん、こんにちは。本日は「Payment distributed（分散）transaction case study」というテーマで発表させていただきます。 まず、簡単に自己紹介させてください
GitHub - trinodb/trino: Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
An In-Depth Analysis of Distributed Transaction Solutions
This article summarizes five distributed transaction solutions and analyzes their features. By Daosu This article summarizes five distributed transaction solutions and analyzes their features: XA Specification, TCC, Saga, Local-Message-Based Distributed Transactions, and Transactional-Message-Based Distributed Transactions. Considering its length, I recommend that you add it to your favorites and
Distributed Load Testing on AWS は、大規模および負荷時のソフトウェアアプリケーションテストを自動化して、リリース前に性能上の潜在的なパフォーマンスの問題を特定することのできるソリューションです。このソリューションは、一定のペースでトランザクションレコードを生成する数多くの接続ユーザーを作成およびシミュレートします。サーバーをプロビジョニングする必要はありません。 更新されたソリューションには、以前に実行したテストのデータを表示する機能が含まれます。データには、テスト設定、テストデータ、および Amazon CloudWatch で視覚化された結果履歴が含まれます。更新には、既存の Amazon Virtual Private Cloud (Amazon VPC) を活用できる機能、および複数のアベイラビリティーゾーンで AWS Fargate タスクを起動す
GitHub - Cysharp/MessagePipe: High performance in-memory/distributed messaging pipeline for .NET and Unity.
MessagePipe is a high-performance in-memory/distributed messaging pipeline for .NET and Unity. It supports all cases of Pub/Sub usage, mediator pattern for CQRS, EventAggregator of Prism(V-VM decoupling), IPC(Interprocess Communication)-RPC, etc. Dependency-injection first Filter pipeline better event sync/async keyed/keyless buffered/bufferless singleton/scoped broadcast/response(+many) in-memory
Building globally distributed MySQL applications using write forwarding in Amazon Aurora Global Database | Amazon Web Services
AWS Database Blog Building globally distributed MySQL applications using write forwarding in Amazon Aurora Global Database AWS released Amazon Aurora Global Database in 2018. Aurora Global Database enables two primary use cases. The first use case is supporting a disaster recovery solution that can handle a full regional failure with a low recovery point objective (RPO) and a low recovery time obj
C++ Creator Bjarne Stroustrup Weighs in on Distributed Systems, Type Safety and Rust
Join our community of software engineering leaders and aspirational developers. Always stay in-the-know by getting the most important news and exclusive content delivered fresh to your inbox to learn more about at-scale software development.
GitHub - hibiken/asynq: Simple, reliable, and efficient distributed task queue in Go
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
GitHub - mgechev/is-esm: 🌳 CLI tool which checks if a package is distributed in ECMAScript module format. Helps you reason if the package is tree-shakable.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
AWS Step Functionsで大規模な並列処理を実施できるDistributed Mapを試してみた #reinvent | DevelopersIO
こんにちは、CX事業本部 IoT事業部の若槻です。 先日開催されたAWS re:Invent 2022で、AWS Step Functionsで大規模な並列処理を実施できるStep Functions Distributed Mapが発表されました。 AWS Step Functions launches large-scale parallel workflows for data processing and serverless applications Step Functions Distributed Map – A Serverless Solution for Large-Scale Parallel Data Processing | AWS News Blog S3 Bucket内のオブジェクトをもとに大規模なMap処理が行えるとのことで、実際に現地会場でKeynote
GitHub - tigerbeetle/tigerbeetle: The distributed financial transactions database designed for mission critical safety and performance.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
Introducing Services: Build Composable, Distributed Applications on Cloudflare Workers
Introducing Services: Build Composable, Distributed Applications on Cloudflare Workers Loading... First, there was the Worker script. It was simple, yet elegant. With just a few lines of code, you could rewrite an HTTP request, append a header, or make a quick fix to your website. Though, what if you wanted to build an entire application on Workers? You’d need a lot more tools in your developer to
We are taught to avoid coupling like a plague. Over the years we’ve introduces numerous design and architectural patterns that decouple systems. But what exactly is coupling? What types of coupling are out there? Are all of them equally bad? Or maybe there is a “good” coupling? In this talk, I’ll share my thoughts on the subject of coupling in distributed systems, and will answer all of the above
You’re seeing information for Bulgaria . To see local features and services for another location, select a different city. Show more Engineering, BackendMySQL to MyRocks Migration in Uber’s Distributed DatastoresSeptember 1, 2022 / Global Introduction Uber uses MySQL as the underlying database engine for Schemaless and Docstore, our distributed databases. By default, MySQL uses the most popular In
Distributed Data for Microservices — Event Sourcing vs. Change Data Capture
This article is a dive into the realms of Event Sourcing, Command Query Responsibility Segregation (CQRS), Change Data Capture (CDC), and the Outbox Pattern. Much needed clarity on the value of these solutions will be presented. Additionally, two differing designs will be explained in detail with the pros/cons of each. So why do all these solutions even matter? They matter because many teams are b
複数GPUで学習するときのDP(Data Parallel)とDDP(Distributed Data Parallel)の違い - Qiita
複数GPUで学習するときのDP(Data Parallel)とDDP(Distributed Data Parallel)の違いPythonDeepLearningPyTorch PyTorch とかで複数GPUを使って学習する場合に DP と DDP って何が違うのか毎回調べてたのでまとめました。 本記事は下記記事やドキュメントをさらに噛み砕いたものとなっています。 特に一つ目の記事は少し前の記事ですがわかりやすいのでおすすめです。 Distributed data parallel training using Pytorch on AWS – Telesens DataParallel — PyTorch 1.9.0 Distributed Data Parallel — PyTorch 1.9.0 Q. DP と DDP どっち使えばよいの？ A. DDP 使え（DDPの方が速いし
This research presents VPN⁰, the first distributed virtual private network offering a privacy preserving traffic authorization and validation mechanism. This research was conducted by Dr. Matteo Varvello (Performance Researcher at Brave), Iñigo Querejeta-Azurmendi (intern at Brave), Dr. Panagiotis Papadopoulos (Security Researcher at Brave), Gonçalo Pestana (Research Engineer at Brave) and Dr. Ben
Distributed Tracing for Ruby on Rails Microservices with OpenCensus/OpenTelemetry (part 2)
Photo by Matt Howard on UnsplashThis series of articles is based on a talk I gave at RailsConf 2019, titled “Troubleshoot Your RoR Microservices with Distributed Tracing.” In part 2 of this series, I will introduce OpenCensus, which provides not only distributed tracing, but also other features to improve observability. I will show how to use OpenCensus in your Rails applications. Part 1 is here f
A minimal distributed key-value database with Hashicorp's Raft library | notes.eatonphil.com
A minimal distributed key-value database with Hashicorp's Raft library When I wrote the "build a distributed PostgreSQL proof of concept" post I first had to figure out how to use Hashicorp's Raft implementation. There weren't any examples I could find in the Hashicorp repo itself. And the only example I could find was Philip O'Toole's hraftd. It's great! However, I have a hard time following mult
[re:Inforce] 組織によるセキュリティレベルの向上事例 | APS201 How AWS and MongoDB raise the security bar with distributed ownership セッションレポート #AWSreInforce | DevelopersIO
[re:Inforce] 組織によるセキュリティレベルの向上事例 | APS201 How AWS and MongoDB raise the security bar with distributed ownership セッションレポート #AWSreInforce AWS re:Inforce 2023のセッション、How AWS and MongoDB raise the security bar with distributed ownershipのレポートです。 こんにちは、菊池です。 AWS re:Inforce 2023参加中です。この記事は、セッション APS201 How AWS and MongoDB raise the security bar with distributed ownershipのレポートです。 セッション概要（公式より引用） In this se
![[re:Inforce] 組織によるセキュリティレベルの向上事例 | APS201 How AWS and MongoDB raise the security bar with distributed ownership セッションレポート #AWSreInforce | DevelopersIO](https://cdn-ak-scissors.b.st-hatena.com/image/square/c599b8acb6609a50aa4e503b6805c190c33b588e/height=288;version=1;width=512/https%3A%2F%2Fdevio2023-media.developers.io%2Fwp-content%2Fuploads%2F2023%2F06%2Freinforce-APS201-001.jpg)
#95: CockroachDB: The Resilient Geo-Distributed SQL Database
ちょっとやそっとで死なない分散 RDB の論文を向井が読みました。感想などはハッシュタグ #misreading か hello@misreading.chat にお寄せください。iTunes のレビューや星も歓迎です。 CockroachDB: The Resilient Geo-Distributed SQL Database | Proceedings of the 2020 ACM SIGMOD International Conference on Management of DataSpanner: Google’s Globally Distributed Database: ACM Transactions on Computer Systems: Vol 31, No 3CockroachDB: The Definitive Guide — Learn to Build
GitHub - ballista-compute/ballista: Distributed compute platform implemented in Rust, and powered by Apache Arrow.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
Instrument your applications just once, then send metrics and traces to multiple AWS monitoring solutions. Speed up performance troubleshooting and reduce mean time to resolution with increased visibility into your AWS resources. AWS support helps optimize performance, lower costs, and innovate faster, while testing provides quantifiable and predictable performance overhead.
GitHub - opensearch-project/OpenSearch: 🔎 Open source distributed and RESTful search engine.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
ABCI上でpytorch distributed data parallelによるマルチノード学習 - Qiita
なんの記事? pytorchのDistributedDataParallelについての日本語記事があまりにもなかったため，素人がまとめました． 並列処理がわからない人による，わからない人のための，とりあえず使えればいいや的なDDPの解説です． 基本的にABCIでの実行を前提に書かれていますが，それ以外の環境の人たちにも参考になれば幸いです． はじめに おなじみの機械学習フレームワークであるpytorch．気軽にDataParallelで並列処理の学習もできます． ですがfacebookなどの一流の機械学習エンジニアたちはDistributedDataParallelなるものを使った実装がちらほらみられます． そこでpytorchの解説記事を読むわけですが，これがびっくりするほどわからない． というわけで，ABCI上でのDistributedDataParallel(以下DDP)の使い方を自
GitHub - hatchet-dev/hatchet: A distributed, fault-tolerant task queue
Hatchet replaces difficult to manage legacy queues or pub/sub systems so you can design durable workloads that recover from failure and solve for problems like concurrency, fairness, and rate limiting. Instead of managing your own task queue or pub/sub system, you can use Hatchet to distribute your functions between a set of workers with minimal configuration or infrastructure: What Makes Hatchet
Temporal Python 1.0.0 – A Durable, Distributed Asyncio Event Loop
We are excited about the GA release of the Temporal Python SDK. Python is now a fully-supported workflow language in Temporal, and our use of native asyncio constructs makes it a perfect fit for Python developers looking to write durable workflows. Here are some links to read more about Temporal Python: Repository Application Development Guide Code Samples API Documentation Like essentially all of
This guide is a collaboration between:Katie Womersley, VP of engineering at Buffer. When I’m writing, you’ll see this: ● Juan Pablo Buriticá, VP of engineering at Splice. When I’m writing, you’ll see this: ▲ When we’re both writing, you’ll see this: ■ ▲ All teams past a certain size become distributed, whether across rooms, floors, buildings, cities, or continents. But tech is only starting to exp
Access on-demand workspaces, manage your own office space, orchestrate events, and make data driven decisions
8メートルの距離のチームを実装するリモートモブプログラミング #distributed_agile_team / remote-mobprogramming within within 8 meters
2020年4月21日「テレワークでのアジャイルチームってどうよ」のLTにて。
GitHub - vearch/vearch: A distributed system for embedding-based vector retrieval
A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
The moment we added our second server, distributed systems became the way of life at Amazon. When I started at Amazon in 1999, we had so few servers that we could give some of them recognizable names like “fishy” or “online-01”. However, even in 1999, distributed computing was not easy. Then as now, challenges with distributed systems involved latency, scaling, understanding networking APIs, marsh
Goptuna Distributed Bayesian Optimization Framework at Go Conference 2019 Autumn
Building Real Time Systems on MongoDB Using the Oplog at Stripe
Googleのエッジ／オンプレミスはAWS Outposts型に移行？ 「Google Distributed Cloud」発表の狙い
Googleのエッジ／オンプレミスはAWS Outposts型に移行？ 「Google Distributed Cloud」発表の狙い：Google Cloud Next '21で発表 Google Cloudが、オンプレミス／エッジで「AWS Outposts」のような製品を発表した。 これまではソフトウェアオンリーの製品を展開してきた同社。 ハードウェア／ソフトウェア一体型の製品で、方向転換をするのだろうか。 Google Cloudは2021年10月12日（現地時間）、年次イベント Google Cloud Next ‘21で「Google Distributed Cloud」を発表した。これは、ソフトウェア、サービスをサーバハードウェアに収め、Google Cloud Platformを企業のデータセンターや社内、エッジに持ち込めるようにするものだ。Amazon Web Servi
This is an introductory course in Distributed Systems. Distributed systems is the study of how to build a computer system where the state of the program is divided over more than one machine (or "node"). This course is in active development. At the moment, it consists of a series of short videos. The intention is to create a complete set of video lectures and then add additional content (such as m
Distributed Load Testing on AWS helps you easily simulate thousands of users connecting to your application so that you can better understand your application performance under load. The solution launches and configures containers on AWS Fargate to generate a specified number of transactions per second without having to provision servers. This update adds support for JMeter plugins and JMeter inpu
Evolving Clock Sync in Distributed Databases | YugabyteDB
Distributed clock synchronization is critical for many applications, including distributed SQL databases. Clock synchronization needs to keep up with the other demands in our modern infrastructure, such as: Applications that have increasing performance requirements while distributing data across different geographic regionsNetwork infrastructure and computing power that is improving constantlyA di
About Grafana TempoGrafana Tempo lets you scale tracing as far as possible with minimal operational cost and less complexity than ever before. The easiest way to get started is with Grafana Cloud, our fully composable observability stack. Introduction to Grafana TempoGrafana Tempo is an open source, easy-to-use, and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only ob
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Distributed Load Testing on AWS v2.0.0 の紹介
Balancing Coupling in Distributed Systems
MySQL to MyRocks Migration in Uber's Distributed Datastores
VPN⁰: A Privacy-Preserving Distributed Virtual Private Network | Brave
Distributed Tracing - AWS Distro for OpenTelemetry - AWS
A guide to distributed teams – Increment: Teams
Flex space and office management for distributed teams | Gable
Challenges with distributed systems
Distributed Systems Course
Introducing Distributed Load Testing v1.2
Grafana Tempo OSS | Distributed tracing backend
F5がSaaS型セキュリティ「F5 Distributed Cloud WAAP」を発表、DDoS対策やWAFなどを統合して提供
