yassan0627のブックマーク - はてなブックマーク

Building a Data Lake on PB scale with Apache Spark

yassan0627 2023/05/26

リンク

Integrating Airflow with Slack for Daily Reporting | by Mandy Gu | Towards Data Science

Photo by Austin Distel on UnsplashTech Stack: Python 3.7, Airflow (1.10.10), Docker GitHub link: All of the code can be found here. Airflow + Slack Slack is an increasingly popular chat app used in the workplace. Apache Airflow is an open source platform for orchestrating workflows. One of the biggest advantages to using Airflow is the versatility around its hooks and operators. Hooks are interface

yassan0627 2020/06/12

リンク

Airflow: how and when to use it (Advanced)

yassan0627 2020/05/23

リンク

How to use Jupyter on a Google Cloud VM

Note: The recipes in this article will still work, but I recommend that you use the notebook API now. Do: gcloud beta notebooks --help The simplest way to launch a notebook on GCP is to go through the workflow from the GCP console. Go to AI Platform and click on Notebook Instances. You can create a new instance from the user interface: Create a new notebook instance from the UIOnce the instance is

yassan0627 2019/07/08

リンク

Google Coral Edge TPU Board Vs NVIDIA Jetson Nano Dev board — Hardware Comparison

Both NVidia and Google recently released dev board targeted towards EdgeAI and also at a cost point to attract developers, makers and hobbyists. Both the dev boards are primarily for inference, but support limited transfer learning re-training. The Edge TPU supports transfer learning training using weight imprinting technique. Both of the dev kits consists of a SOM (System-on-Module) connected to

yassan0627 2019/04/24

TPU

リンク

KubernetesExecutor for Airflow

In the 1.10 release, Airflow introduced a new executor to run workers at scale: the Kubernetes executor. In this article we’ll look into: What is Airflow and which probl em it solvesThe Kubernetes executor and how it compares to the Celery executorAn example deployment on minikubeTL;DRAirflow has a new executor that spawns worker pods natively on Kubernetes. There’s a Helm chart available in this g