S3に保存したログファイルをストリーム処理するサーバーレスアプリケーションの紹介 - クックパッド開発者ブログ
インフラストラクチャー部セキュリティグループの水谷(@m_mizutani)です。 クックパッドでは現在セキュリティ監視の高度化に取り組んでおり、その一環としてセキュリティ関連のログ収集およびその分析に力を入れています。ログ収集の部分では可用性などの観点からAWSのオブジェクトストレージサービスであるS3に一部のサービスやサーバのログをまず保存し、後から保存されたファイルを読み込んで分析などに利用しています。分析のためにS3に保存したファイルを前処理する方法としてAWS Glueなどを用いたバッチ処理がありますが、到着したログをなるべくストリームデータのように扱いたい場合もあります。特にセキュリティ関連のログでは以下のようなユースケースで利用しています。 アラートの検出: ログを検査してその中から危険度の高いと考えられるログを探し出し、アラートとして発報します。アラートの具体的な例としては
PipelineDB—The Streaming SQL Database
Bermain slot gacor di Toto88 bisa menjadi pengalaman yang menggembirakan dan menguntungkan. Dengan berbagai pilihan permainan, fitur yang menarik, dan peluang menang besar, Toto88 telah menjadi destinasi utama bagi para pemain slot online. Artikel ini akan membantu Anda memahami bagaimana memaksimalkan pengalaman Anda saat bermain slot gacor di Toto88. Toto88 menawarkan pengalaman bermain slot onl
Streams: a new general purpose data structure in Redis. - <antirez>
Until a few months ago, for me streams were no more than an interesting and relatively straightforward concept in the context of messaging. After Kafka popularized the concept, I mostly investigated their usefulness in the case of Disque, a message queue that is now headed to be translated into a Redis 4.2 module. Later I decided that Disque was all about AP messaging, which is, fault tolerance an
Introducing KSQL: Streaming SQL for Apache Kafka | Confluent
ksqlDB is the successor to KSQL. Read the announcement to learn more. To get started with ksqlDB in Confluent Cloud, you can sign up for fully managed Apache Kafka as a service and receive $400 of free usage during your first 60 days, plus an additional $60 of free usage when you use the promo code CL60BLOG.* I’m really excited to announce KSQL, a streaming SQL engine for Apache Kafka®. KSQL lower
Streaming Systems
Read it now on the O’Reilly learning platform with a 10-day free trial. O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers. Streaming data is a big deal in big data these days. As more and more businesses seek to tame the massive unbounded data sets that pervade our world, streaming systems have finally rea
Using logs to build a solid data infrastructure (or: why dual writes are a bad idea) | Confluent
This is an edited transcript of a talk I gave at the Craft Conference 2015. The video and slides are also available. How does your database store data on disk reliably? It uses a log. How does one database replica synchronise with another replica? It uses a log. How does a distributed algorithm like Raft achieve consensus? It uses a log. How does activity data get recorded in a system like Apache
Twitter Heron: Stream Processing at Scale | the morning paper
the morning paper a random walk through Computer Science research, by Adrian Colyer Made delightfully fast by strattic Twitter Heron: Stream Processing at Scale – Kulkarni et al. 2015 It’s hard to imagine something more damaging to Apache Storm than this. Having read it through, I’m left with the impression that the paper might as well have been titled “Why Storm Sucks”, which coming from Twitter
Esper
Where Complex Event Processing meets Open Source: Esper & NEsper Make no compromise on enterprise readiness: discover EsperHA, EsperJDBC and our support services. Embed mainstream CEP in your products and deployments: Learn about our licensing options. Event Stream Intelligence with Esper and NEsper Esper is a component for CEP and ESP applications, available for Java as Esper, and for .NET as
Norikra: Stream processing with SQL for everybody
Schema-less Stream Processing with SQL Norikra is a open source server software provides "Stream Processing" with SQL, written in JRuby, runs on JVM, licensed under GPLv2. Schema-less event streams (called as 'target') Input/Output event streams as JSON objects, which can contain any fields with a target name. SQL processing Norikra's query is SQL with window specifier support (It's actually Esper
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Streaming Systems Book
Acquisitions