Cloudflare Queues: globally distributed queues without the egress fees
Cloudflare Queues: globally distributed queues without the egress fees09/27/2022 Developers continue to build more complex applications on Cloudflare’s Developer Platform. We started with Workers, which brought compute, then introduced KV, Durable Objects, R2, and soon D1, which enabled persistence. Now, as we enable developers to build larger, more sophisticated, and more reliable applications, i
We are happy to announce the open source release of Waltz. Waltz is a distributed write-ahead log. It was initially designed to be the ledger of money transactions on the WePay system and was generalized for broader use cases of distributed systems that require serializable consistency. Waltz is similar to existing log systems like Kafka in that it accepts/persists/propagates transaction data prod
GitHub - jakekgrog/GhostDB: GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
GhostDB stemmed from a University project. Due to the nature of these projects (time constraints etc.), we feel some corners were cut. For example, we opted for the memcached model of distribution to save on time as it was easier to implement. However, this wasn't the original vision of GhostDB. Myself and Connor also started new jobs and these took up a good chunk of our time. This combined with
GitHub - microsoft/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
GitHub - wasmerio/ate: Distributed immutable data store with strong encryption and authentication
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
GitHub - frameable/aviary.sh: Minimal distributed configuration management in bash
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
Distributed Deep Learning with Chainer and Hadoop
EM完全に理解した と思ったけど、 やっぱり何も分からなかった話 / EM Night Fukuoka #1
GitHub - erikgrinaker/toydb: Distributed SQL database in Rust, written as a learning project
Distributed SQL database in Rust, written as a learning project. Most components are built from scratch, including: Raft-based distributed consensus engine for linearizable state machine replication. ACID-compliant transaction engine with MVCC-based snapshot isolation. Pluggable storage engine with BitCask and in-memory backends. Iterator-based query engine with heuristic optimization and time-tra
We’re thrilled to announce a new open-source package for the Swift on Server ecosystem, Swift Distributed Actors, a complete server-oriented cluster library for the upcoming distributed actor language feature! This library provides a complete solution for using distributed actors in server use-cases. By open-sourcing this project early, alongside the ongoing work on the language feature, we hope t
Reinventing virtualization with the AWS Nitro System - All Things Distributed
All Things Distributed Werner Vogels' weblog on building scalable and robust distributed systems. Running a business at the scale of Amazon, we often have to solve problems that no other company has faced before. The disadvantage of this is that there is no “how to” guide for us—a lot is unknown. However, the advantage is that when we solve a new problem, it’s an opportunity to reinvent our servic
大規模な負荷テストを実行可能。「Distributed Load Testing on AWS」 を試してみる - builders.flash☆ - 変化を求めるデベロッパーを応援するウェブマガジン | AWS
皆さん、こんにちは！テクニカルソリューションアーキテクトの山澤良介です。 皆さんは「AWS ソリューション」というページをご存知でしょうか？ AWS ソリューションでは、世界中のユーザーが直面する一般的な問題の解決策を提供しております。既に 60 を超える多数のソリューションが公開されており、お客様は AWS ソリューションを使用することで実装にかかる時間と労力を節約することができます。 このように非常に便利なものではあるのですが、AWS ソリューションの存在を知っていた方であっても、「どんなソリューションがあるのかわからない」「ソリューションの概要や使い方がわからない」などの理由で実際に活用したことがある方は少ないのではないでしょうか ? そのような背景から、今回の記事では、皆さんに活用頂けそうな AWS ソリューションを 1 つ、使い方などを含めて紹介していこうと思います ! 今回紹介
Index Compression Using Byte-Aligned ANS Coding and Two-Dimensional Contexts
オラクル、シャーディングを利用したグローバルな分散DBの新サービス「Oracle Globally Distributed Autonomous Database」正式サービス開始
オラクル、シャーディングを利用したグローバルな分散DBの新サービス「Oracle Globally Distributed Autonomous Database」正式サービス開始 オラクルはデータベースを複数のシャードに分割し、それぞれをグローバルに分散させつつ全体を1つの論理的なデータベースとして扱えるOracle Cloudの新サービス「Oracle Globally Distributed Autonomous Database」を正式サービスとして提供開始しました。 Find out how you can more easily address data sovereignty requirements and achieve the highest possible scale and availability with @Oracle Globally Distribute
クラウドネイティブ時代の分散トレーシング - Distributed Tracing in a Cloud Native Age
クラウドネイティブ時代の分散トレーシング - Distributed Tracing in a Cloud Native Age
GitHub - vdaas/vald: Vald. A Highly Scalable Distributed Vector Search Engine
Vald is a highly scalable distributed fast approximate nearest neighbor (ANN) dense vector search engine. Vald is designed and implemented based on Cloud-Native architecture. Vald has automatic vector indexing and index backup, and horizontal scaling which made for searching from billions of feature vector data. Vald is easy to use, feature-rich and highly customizable as you needed. It uses the f
YDBYDB is a versatile open source Distributed SQL Database that combines high availability and scalability with strong consistency and ACID transactions. It accommodates transactional (OLTP), analytical (OLAP), and streaming workloads simultaneously.
Cloudstate is no longer actively developed Cloudstate was an open source protocol and reference implementation exploring ideas for stateful serverless, and was originally developed by Lightbend. The project is no longer active, since 2021. An open source alternative is Eigr. A continuation of the ideas can be found in Lightbend's platform-as-a-service Akka Serverless.
Improving Distributed Caching Performance and Efficiency at Pinterest
IntroductionPinterest’s distributed caching system, built on top of open source technologies memcached and mcrouter, is a critical component of the production infrastructure stack. Pinterest’s cache-as-a-service platform is responsible for driving down application latency across the board, reducing the overall cloud cost footprint, and ensuring adherence to strict sitewide availability targets. To
Parapet - A purely functional library to build distributed and event-driven systems
A purely functional library to build distributed and event-driven systems It's not a secret that writing distributed systems is a challenging task that can be logically broken into two main aspects: implementing distributed algorithms and running them. Parapet plays the role of execution framework for distributed algorithms - it can be viewed as an intermediate layer between a low-level effect lib
Google Cloud、Google Distributed Cloudにオンプレミス用の「Anthos on-premises」を「GDC Virtual」として統合
Google Cloud、Google Distributed Cloudにオンプレミス用の「Anthos on-premises」を「GDC Virtual」として統合 Google Cloudは、これまで同社のハイブリッドクラウド戦略を実現する製品としてオンプレミス用に提供してきた「Anthos on-premises」の製品名を「Google Distributed Cloud Virtual」（GDC Virtual）に変更し、Google Distributed Cloudブランド傘下の製品として位置づけ直すことを明らかにしました。 New update to our Google Distributed Cloud portfolio Google Distributed Cloud Virtual: a software-only solution that brings o
[速報]Step Functionsで大量データの並列処理が可能に！Step Functions Distributed MapがGAされました！# reInvent | DevelopersIO
速報です！ 本日の Werner Vogels キーノートで、Step Functionsで大規模な並列処理を可能にする、「Step Functions Distributed Map」が発表されました！ これまで Step FunctionsにはもとよりMap並列処理を提供機能がありましたが、並列処理の同時実行数は最大40回という制限がありました。 Step Functions Distributed Mapで何ができるのか Mapは上記の制限により、数千件以上の項目を並列でタイムリーに処理することは困難でしたが、「Step Functions Distributed Map」により大規模な並列ワークロードの処理が可能になりました！！ 「Step Functions Distributed Map」では、最大1万回の並列ワークロードを起動することができます。並列で実行できる処理には、St
![[速報]Step Functionsで大量データの並列処理が可能に！Step Functions Distributed MapがGAされました！# reInvent | DevelopersIO](https://cdn-ak-scissors.b.st-hatena.com/image/square/0811654a8dac484dafd58339e445d07a684c2164/height=288;version=1;width=512/https%3A%2F%2Fdevio2023-media.developers.io%2Fwp-content%2Fuploads%2F2022%2F12%2Fstep-function-destributed-map-reinvent1.png)
GitHub - surrealdb/surrealdb: A scalable, distributed, collaborative, document-graph database, for the realtime web
SurrealDB is an end-to-end cloud-native database designed for modern applications, including web, mobile, serverless, Jamstack, backend, and traditional applications. With SurrealDB, you can simplify your database and API infrastructure, reduce development time, and build secure, performant apps quickly and cost-effectively. Key features of SurrealDB include: Reduces development time: SurrealDB si
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
Docs Index Get Started with LiteFS Toggle Get Started with LiteFS section LiteFS overview Speedrun: Adding LiteFS to your app Getting Started with LiteFS on Fly.io Getting Started with LiteFS in Docker How LiteFS Works FAQ LiteFS Cloud Toggle LiteFS Cloud section Using LiteFS Cloud for Backups Restoring from a LiteFS Cloud Backup Disaster Recovery from LiteFS Cloud Using LiteFS Toggle Using LiteFS
Reliable by Design: Applying Formal Methods to Distributed Systems
Search and analytics, data ingestion, and visualization – all at your fingertips
Distinguished Engineer Ensar’s primary areas of interest are distributed data, replication, consistency, and storage. He has more than seven years of hands-on expertise in designing, developing, and testing distributed algorithms, with solid experience in concurrency. He has authored a number of articles on distributed data and stream processing, and is a frequent speaker at industry conferences o
Building a large-scale distributed storage system based on Raft | Cloud Native Computing Foundation
Guest post by Edward Huang, Co-founder & CTO of PingCAP In recent years, building a large-scale distributed storage system has become a hot topic. Distributed consensus algorithms like Paxos and Raft are the focus of many technical articles. But those articles tend to be introductory, describing the basics of the algorithm and log replication. They seldom cover how to build a large-scale distribut
Debugging Incidents in Google’s Distributed Systems – Communications of the ACM
CACM Web Account Membership in ACM includes a subscription to Communications of the ACM (CACM), the computing industry's most trusted source for staying connected to the world of advanced computing. Sign In Sign Up Google has published two books about Site Reliability Engineering (SRE) principles, best practices, and practical applications.1,2 In the heat of the moment when handling a production i
Akamai To Acquire Linode to Provide Businesses with a Developer-friendly and Massively-distributed Platform to Build, Run and Secure Next Generation Applications | Akamai
Compute Build, release, and scale faster with VMs for every workload
Announcing Grafana Tempo, a massively scalable distributed tracing system | Grafana Labs
Solutions All end-to-end solutions Opinionated solutions that help you get there easier and faster
At the QCon in March, I gave a talk on the evolution of distributed systems with Kubernetes. First, I want to start with a question, what comes after microservices? I'm sure you all have an answer to that, and I have mine too. You'll find out at the end what I think that will be. To get there, I suggest we look at what are the needs of the distributed systems. And how those needs have been evolvin
GitHub - microsoft/retina: eBPF distributed networking observability tool for Kubernetes
Retina is a cloud-agnostic, open-source Kubernetes network observability platform that provides a centralized hub for monitoring application health, network health, and security. It provides actionable insights to cluster network administrators, cluster security administrators, and DevOps engineers navigating DevOps, SecOps, and compliance use cases. Retina collects customizable telemetry, which c
GitHub - losfair/mvsqlite: Distributed, MVCC SQLite that runs on FoundationDB.
Full feature-set from SQLite: mvSQLite integrates with SQLite as a custom VFS layer. It is a layer "below" SQLite, and all of SQLite's features are available. Time travel: Checkout the snapshot of your database at any point of time in the past. Lock-free, scalable reads and writes: Optimistic fine-grained concurrency with BEGIN CONCURRENT-like semantics. mvSQLite inherits FoundationDB's lock-free
CMSの第四の波 Distributed CMS (Drupal Developer Days Transylvania 2019) | ANNAIマガジン
2019 年 6 月にルーマニアで開催された「Drupal Developer Days Transylvania 2019」（DDD2019）における Gatsby 社 Preston So 氏の CMS の最新動向についての基調講演「The fourth wave of the content management system」をご紹介します。 なお DDD2019 については別記事をご参照下さい。 講演者の So 氏は Acquia 社で Director of Research and Innovation という役職に就いていましたが、2019 年の 3 月頃に Gatsby に移籍した Decoupled CMS の第一人者です。 現状の課題 コンテンツの利用形態の爆発的な多様化 従来は PC・タブレット・スマートフォンにとどまっていたコンテンツの利用形態は、今では Appl
Retrieval Augmented Generation at scale — Building a distributed system for synchronizing and…
Disclaimer: We will go into some technical and architectural details of how we do this at Neum AI — A data platform for embeddings management, optimization, and synchronization at large scale, essentially helping with large-scale RAG. As we’ve shared in other blogs in the past, getting a Retrieval Augmented Generation (RAG) application started is pretty straightforward. The problem comes when tryi
Announcing Ballista - Distributed Compute with Rust, Apache Arrow, and Kubernetes
Andy Grove Just another software engineer with a famous name and a blog. Eighteen months ago, I started the DataFusion project with the goal of building a distributed compute platform in Rust that could (eventually) rival Apache Spark. Unsurprisingly, this turned out to be an overly ambitious goal at the time and I fell short of achieving that. However, some very good things came out of this effor
Shibboleths are historically a word or phrasing that indicate membership in a particular group or culture. I was introduced to the term in the West Wing where the President needed to verify the veracity of a person’s claims of religious persecution. I am still a relatively new engineer in the field of distributed systems, having only studied and worked in the field for around a decade, but in that
ゆるふわ分散トレースはじめました / start casual distributed trace
【BASE社合同勉強会】コネヒトマルシェオンライン「事業を支えるWeb開発」 https://connehito.connpass.com/event/176890/ スライド資料内のリンクなどはこちらにまとめている。 https://budougumi0617.github.io/2020/06/04/connehito_marche/
MITの分散システムの講義 6.824: Distributed Systems が面白かった件 - Qiita
これは何？ 分散システム初学者が、MITの院生向け講義 6.824: Distributed Systems を一通りやってみたので、その内容をまとめる。 分散システムなる分野に入門したいと思った際の一つの選択肢として個人的にオススメできるコンテンツだったので、この場で広く推すために書いている。 Lab3で扱うRaftのアーキテクチャ図より引用1 きっかけ 元々のモチベーションとしては、お仕事周りでAurora2やDynamo3、EBS4といった文献に目を通す中で、そもそも根底にある分散システムという分野を全然知らない。俺たちは雰囲気でクラウドをやっている(画像略)5…という気分になったのがきっかけだった。 ちょうど年末の空き時間に分散システムのオススメコンテンツを探している中で、推されていたこのコンテンツに出会ったのが12月末。その後は仕事の後など空き時間にちまちまやって、結局Lab3ま
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Waltz: A Distributed Write-Ahead Log
File systems unfit as distributed storage backends: lessons from ten years of Ceph evolution | the morning paper
Introducing Swift Distributed Actors
Understanding distributed processing in Python
YDB — an open source Distributed SQL Database
Distributed State Management for Serverless - Cloudstate
GitHub - runabol/tork: A distributed workflow engine
LiteFS - Distributed SQLite
Distributed Locks are Dead; Long Live Distributed Locks!
The Evolution of Distributed Systems on Kubernetes
Distributed Systems Shibboleths | Joey Lynch's Site