投稿時間:2023-03-23 05:26:00 RSSフィード2023-03-23 05:00 分まとめ(30件)

カテゴリー等	サイト名等	記事タイトル・トレンドワード等	リンクURL	頻出ワード・要約等/検索ボリューム	登録日
海外TECH	DEV Community	Self managed Apache Airflow with Data on EKS	https://dev.to/aws/self-managed-apache-airflow-with-data-on-eks-4jc7	Self managed Apache Airflow with Data on EKS data on eksI have written in the past about how you can get started with Apache Airflow using the AWS managed service Managed Workflows for Apache Airflow But what if you want to self managed Apache Airflow When I speak with developers there are sometimes reasons why a managed service might not fit their needs Some of the common things that come up include whether you need the increase level of access a greater level of control of the configuration of Apache Airflowhave the need to have the very latest versions or features of Apache Airflowif you have the need to run workflows that use more resources that managed services provide for example need significant compute Total Cost Ownership One thing to consider when assessing managed vs self managed is the cost of the managed service against the total costs of you having to do the same thing It is important to assess a true like for like and we often see just the actual compute and storage resources being compared without all the additional things that you need to make this available If the answer to these is yes then it is likely that using a managed service may frustrate you Self Managed Apache AirflowThere are many ways you can deploy your own Apache Airflow but I want to share with you a really cool project that is going to help simplify your life Based on EKS Blueprints Data on EKS or DoEKS for short is a comprehensive tool that allows you to build scalable data platforms on Amazon EKS The repository provides you with a wealth of resources including Infrastructure as Code templates guidance on best practices for deploying solutions on Amazon EKS and more One of the supported solutions they provide is a self managed Apache Airflow In this blog post I want to share my journey in deploying a self managed Apache Airflow using DoEKS The project is well documented and we are going to be referencing the document Self managed Apache Airflow deployment for EKS throughout this post Note Currently I could not get this working from my Cloud development desktop I am currently looking at why and then documenting a fix It is something to do with the kms key generation which fails from C but works fine when running from my local Macbook Pre reqsTo get started with this you are going to need AWS account non Prod with sufficient privilegesYou will need to make sure you have enough capacity to deploy a new VPC by default you can deploy VPCs in a region If you are already at your limit you will need to increase that limit or clean up one of your existing VPCsTerraform installed I was using Terraform v on darwin amd The following tools git Note The code has been developed on a Linux machine and tested working on a Mac It should work on a Windows machine with the Windows Subsystem for Linux WSL installed although I have not tested this If you do encounter issues I recommend that you spin up an Amazon Cloud IDE environment and run through the code there InstallationSo what exactly does the DoEKS install The Self Managed Airflow pattern deploys the following starter pattern An Amazon EKS clusterAn EFS shared volume that will be mounted by the EKS cluster for persistent storage of logs and DAGsAn S bucket for Apache Airflow loggingAWS Secrets Manager for storing passwords for the PostgreSQL database used by Apache Airflow for the metastore databaseAn Amazon RDS PostgreSQL instanceKMS keys to ensure everything is encryptedAWS CloudWatch logging groups for Apache Airflow and Amazon RDSDeployment of the official Apache Airflow helm chartDeployment of the git sync sidecar to synchronise DAGs from a test GitHub repositoryAs the source code is open you can explore how each of these components is configured and suit to your own needs The documentation provides the steps to install so I will be borrowing heavily from the docs over the next few steps We start off by grabbing the code from GitHub This will actually clone all the other deployment patterns but we are only interested in the self managed airflow git clone cd data on eks schedulers terraform self managed airflowBefore proceeding it is worth checking and updating the variables tf file to suit your particular setup This was how I set mine up changing the default region but other than that keeping the same defaults in the original file variable name description Name of the VPC and EKS Cluster type string default self managed airflow variable region description region type string default eu north variable eks cluster version description EKS Cluster version type string default variable vpc cidr description VPC CIDR type string default We can now initialise Terraform which will begin downloading any required Terraform modules needed as defined within the configuration files export AWS REGION eu north terraform initWhich will generate the following outputInitializing modules Downloading git for airflow irsa airflow irsa in terraform modules airflow irsa modules irsaDownloading registry terraform io terraform aws modules s bucket aws for airflow s bucket airflow s bucket in terraform modules airflow s bucketDownloading registry terraform io terraform aws modules rds aws for db db in terraform modules db Terraform has been successfully initialized You may now begin working with Terraform Try running terraform plan to seeany changes that are required for your infrastructure All Terraform commandsshould now work If you ever set or change modules or backend configuration for Terraform rerun this command to reinitialize your working directory If you forget othercommands will detect it and remind you to do so if necessary This will take about minutes Once finished hopefully without errors you can run the Terraform plan terraform planThis will generate a lot of ouput and is worth checking out before proceeding as it will save you from cleaning up for any obvious typos you might have made module eks blueprints kubernetes addons module cluster autoscaler module helm addon module irsa kubernetes service account v irsa will be created resource kubernetes service account v irsa automount service account token true default secret name known after apply id known after apply metadata annotations known after apply generation known after apply name cluster autoscaler sa namespace kube system resource version known after apply uid known after apply Plan to add to change to destroy Changes to Outputs configure kubectl known after apply ーNote You didn t use the out option to save this plan so Terraform can t guarantee to take exactly these actions if you run terraform apply now Once satisfied you are now ready to deploy using terraform applyterraform applyAfter about minutes and a lot of screen output you will be prompted to proceed After reviewing the configuration information enter yes if you want to go ahead Do you want to perform these actions Terraform will perform the actions described above Only yes will be accepted to approve Enter a value yesNow time to grab a cup of tea and come back in around minutes When if finishes you should be greeted by a message similar to this one module eks blueprints kubernetes addons module aws coredns aws eks addon coredns Still creating ms elapsed module eks blueprints kubernetes addons module aws ebs csi driver aws eks addon aws ebs csi driver Still creating ms elapsed module eks blueprints kubernetes addons module aws coredns aws eks addon coredns Creation complete after ms id self managed airflow coredns module eks blueprints kubernetes addons module aws ebs csi driver aws eks addon aws ebs csi driver Still creating ms elapsed module eks blueprints kubernetes addons module aws ebs csi driver aws eks addon aws ebs csi driver Creation complete after ms id self managed airflow aws ebs csi driver Apply complete Resources added changed destroyed Outputs configure kubectl aws eks region eu north update kubeconfig name self managed airflow To test this has worked ok run the following command to grab the Kubernetes ingress urlkubectl get ingress n airflowwhich should output something likeYou can then pop this into a browser and then login using the Airflow default user admin admin You should then see the Apache Airflow UI and also have a couple of workflows DAGs that have been populated from the git repository configured in our code Congratulations you have completed your setup How easy was that Improving the setupThis setup is pretty good but there is still some stuff that you should look at enabling TLS the example code does not setup a certificate or dns for the load balancer but this can be easily configured using the existing codeconfiguring a private git repository the current demo uses a sample read only GitHub repo to pull in the sample DAGs You can configure the values yaml to point to your private repoconfiguring a specific Airflow docker image typically you will have your own requirements and will not use the vanilla Airflow image so configuring a pipeline to build your own Airflow container image and then use this by updating values yamlI will come back to this blog post in the future and update with additional sections to show you how you can implement these I am interested in what you think is missing or perhaps you would like to see Let me know in the comments and I will add it to the list Cleaning upOnce you have deployed this tutorial you should delete all the resources to ensure you do not incur any further costs To do that we can use these commands terraform destroy target module db auto approveterraform destroy target module eks blueprints kubernetes addons auto approveterraform destroy target module eks blueprints auto approveterraform destroy auto approveConclusionIn this short blog post I introduced you to another way you can deploy Apache Airflow using Data on EKS DoEKS patterns DoEKS makes it a breeze to install popular open source data tools and I encourage you to explore some of the other open source projects you can deploy Let me know if you found this quick tutorial useful and what you would like to see me add to this TroubleshootingWhen I was putting this post together I encountered a number of errors When these came up trying to carry on installation after resolving the issues ended up causing me more problems and what I found worked was to completely do a clean up delete all resources before trying again I like cataloging the errors and issues that I came across incase anyone else bumps into these too So the following section provides a postscript of the issues I ran into and some of the things that helped me solve them Running within CloudI got the following error module vpc aws route private nat gateway Creation complete after s id r rtb aebfd ╷│Error creating IAM Role self managed airflow cluster role InvalidClientTokenId The security token included in the request is invalid│status code request id fa def af e bfcaf││with module eks blueprints module aws eks aws iam role this │on terraform modules eks blueprints aws eks main tf line in resource aws iam role this │ resource aws iam role this │╵╷│Error creating IAM Role airflow metastore InvalidClientTokenId The security token included in the request is invalid│status code request id bc a af cbcd││with module db module db instance aws iam role enhanced monitoring │on terraform modules db modules db instance main tf line in resource aws iam role enhanced monitoring │ resource aws iam role enhanced monitoring │╵╷│Error creating KMS Key MalformedPolicyDocumentException Policy contains a statement with one or more invalid principals ││with module eks blueprints module kms aws kms key this │on terraform modules eks blueprints modules aws kms main tf line in resource aws kms key this │ resource aws kms key this │╵Not finding resourcesIn one deployment it looked like it was creating resources in us west for some reason Even though I had set the environment variable AWS REGION it looked as if you also have to update the variables tf filevariable region description region type string default eu north Non unique S bucketI also ran into this error │Error creating Amazon S Simple Storage Bucket airflow logs ExpiredToken The provided token has expired │status code request id DVKHWQNPFGJQZ host id HxyYMENSjXGNGDpoEh EuiShPxTkYXKXOTuzrmlzw dL XmvQHdSxRb fdXYQumIoQ ││with module airflow s bucket aws s bucket this │on terraform modules airflow s bucket main tf line in resource aws s bucket this │ resource aws s bucket this │╵which was the result of a duplicate S bucket being created Once I removed and waited I was able to redo this Hack I modified the main tf logic and changed the name for the S bucket name to save myself some time Resources not available in the AWS region I was deploying intoI also got this error when I was deploying in eu north module eks blueprints kubernetes addons module aws ebs csi driver aws eks addon aws ebs csi driver Creation complete after ms id self managed airflow aws ebs csi driver ╷│Error creating RDS DB Instance airflow InvalidParameterValue Invalid DB Instance class db mi xlarge│status code request id dc b fa ba edff││with module db module db instance aws db instance this │on terraform modules db modules db instance main tf line in resource aws db instance this │ resource aws db instance this And taking a look within the RDS console confirmed this instance type was unavailable so I switched it to a similar db md xlarge vpu with gb ram by updating the main tf file instance class db mi xlarge instance class db m xlarge Not cleaning up resources before installingI also encountered the following error whilst I was getting this code working │Error cannot re use a name that is still in use││with module eks blueprints kubernetes addons module prometheus module helm addon helm release addon │on terraform modules eks blueprints kubernetes addons modules kubernetes addons helm addon main tf line in resource helm release addon │ resource helm release addon I got this error when I tried to carry on deploying an interrupted installation I found that deleting cleaning up all the resources and then doing everything from the beginning resolve the issue If you do have to re start the installation the destroy does not clean up all the resources Make sure you look and manually delete the following CloudWatch log groups aws eks self managed airflow cluster anything with self managed airflow that you did not have there before	2023-03-22 19:35:02
Apple	AppleInsider - Frontpage News	FlexiSpot E7 Pro Plus Standing Desk review: Simple design with smooth height transitions	https://appleinsider.com/articles/23/03/22/flexispot-e7-pro-plus-standing-desk-review-simple-design-with-smooth-height-transitions?utm_medium=rss	FlexiSpot E Pro Plus Standing Desk review Simple design with smooth height transitionsThe FlexiSpot E Pro Plus Standing Desk offers a compact design that is easy to use and understand FlexiSpot E Pro Plus Standing DeskThe FlexiSpot E Pro Plus Standing Desk meets your standing desk needs by offering a beautiful design with easy to operate controls at the helm of the overall experience It can fit into nearly any office setup with a choice of multiple options Read more	2023-03-22 19:31:30
海外TECH	Engadget	Ethernet co-inventor Bob Metcalfe wins the Turing Award, the 'Nobel Prize of computing'	https://www.engadget.com/bob-metcalfe-wins-turing-award-ethernet-195342698.html?src=rss	Ethernet co inventor Bob Metcalfe wins the Turing Award the x Nobel Prize of computing x Even if you re not plugging an Ethernet cable into the wall somewhere along the chain you re still relying on that technology to get online You ve got Bob Metcalfe and the late David Boggs to thank for that Together at Xerox s renowned Palo Alto Research Institute better known as Xerox PARC the pair developed Ethernet and set the stage for a networking revolution Yesterday Metcalfe received the Association for Computing Machinery ACM A M Turing Award which is often pegged the Nobel Prize of computing Thanks to funding from Google that prize is now worth million and it s yet another highlight for Metcalfe s storied career After leaving Xerox PARC he founded com a networking equipment company that brought Ethernet to the mainstream “It is dangerous to accept an award for developing Ethernet which turns on May Metcalfe told the University of Texas where he now serves as a professor emeritus “Over Ethernet s years hundreds of people have earned some claim of inventorship Join me in saying to these folks Thank you Despite its notoriety as a cable technology Metcalfe told the New York Times that he originally envisioned it to be more like the Wi Fi we know today “We wanted to make it wireless he said “But we couldn t have zero wires It would have been too slow and too expensive This article originally appeared on Engadget at	2023-03-22 19:53:42
海外TECH	Engadget	Epic will share 40 percent of 'Fortnite' purchase revenue with creators	https://www.engadget.com/epic-will-share-40-percent-of-fortnite-purchase-revenue-with-creators-192924916.html?src=rss	Epic will share percent of x Fortnite x purchase revenue with creatorsEpic Games will has a new way to encourage more custom Fortnite islands promise a steadier stream of cash The company has launched a Creator Economy system that will proportionately hand out percent of net revenue from most real money purchases to creators including Epic If your island is especially popular or keeps people coming back you ll get a larger cut of Item Shop spending and related transactions Until now Fortnite creators received special codes they had to share with fans if they wanted a five percent slice of purchases However that approach only worked if players knew to use the code That left some creators using heavy handed tactics to promote the code such as frequent promotion during livestreams or even designing islands around the concept Creators can sign up for Economy through a portal People with existing islands who joins on or before April st will be paid for engagement dating back to March st To qualify individuals only need to be years old and have a Fortnite account at least days old Companies are also eligible The news comes as Epic is introducing a free public beta of Unreal Engine for Fortnite As the name implies it will eventually bring most of Unreal Engine s tools to creators making islands games and other experiences inside the shooter You can edit maps the environment and game mechanics Effectively this should lead to more distinctive and more polished third party projects The change to payouts isn t a shock According to Activeplayer io Fortnite s average monthly player counts have gradually declined since peaks in and Economy could help Epic by rewarding creators of in demand islands To some degree this is also about fending off competition from Roblox Many companies are building islands in the rival platform to attract younger users ーEpic s move could keep those brands and gamers onboard This article originally appeared on Engadget at	2023-03-22 19:29:24
海外科学	NYT > Science	How Do Volcanic Smoke Rings Form? New Clues in the Vapor.	https://www.nytimes.com/2023/03/21/science/volcanoes-smoke-rings-vapor.html	gaseous	2023-03-22 19:40:22
海外科学	NYT > Science	William A. Wulf, Pioneering Computer Scientist, Dies at 83	https://www.nytimes.com/2023/03/22/technology/william-a-wulf-dead.html	William A Wulf  Pioneering Computer Scientist Dies at One of the first people to receive a graduate degree in the field Dr Wulf helped to adapt an early Pentagon communications web into the network that eventually grew into the internet	2023-03-22 19:27:48
ニュース	BBC News - Home	Partygate: MPs clash with Boris Johnson over denials	https://www.bbc.co.uk/news/uk-politics-65039661?at_medium=RSS&at_campaign=KARANGA	boris	2023-03-22 19:47:03
ニュース	BBC News - Home	Trail strikes planned in March and April suspended	https://www.bbc.co.uk/news/business-64896128?at_medium=RSS&at_campaign=KARANGA	april	2023-03-22 19:02:04
ニュース	BBC News - Home	Mosque fire suspect arrested over second attack	https://www.bbc.co.uk/news/uk-england-65045660?at_medium=RSS&at_campaign=KARANGA	london	2023-03-22 19:21:35
ニュース	BBC News - Home	Coppola Jr becomes film family's latest star, with viral TikTok clip	https://www.bbc.co.uk/news/entertainment-arts-65038887?at_medium=RSS&at_campaign=KARANGA	media	2023-03-22 19:04:10
ニュース	BBC News - Home	Fingerprints and a mugshot: What happens if Trump is arrested	https://www.bbc.co.uk/news/world-us-canada-65012096?at_medium=RSS&at_campaign=KARANGA	indictment	2023-03-22 19:09:47
ニュース	BBC News - Home	Lyon 0-1 Chelsea: Blues get crucial European win at holders Lyon	https://www.bbc.co.uk/sport/football/65029776?at_medium=RSS&at_campaign=KARANGA	france	2023-03-22 19:50:18
ニュース	BBC News - Home	Miami Open 2023 results: Emma Raducanu loses to Bianca Andreescu	https://www.bbc.co.uk/sport/tennis/65040668?at_medium=RSS&at_campaign=KARANGA	Miami Open results Emma Raducanu loses to Bianca AndreescuBritish number one Emma Raducanu is out of the Miami Open after losing to Canada s Bianca Andreescu in an enjoyable battle between two former teenage US Open champions	2023-03-22 19:12:57
ニュース	BBC News - Home	Johnson on Partygate: We avoided physical contact but passed drinks	https://www.bbc.co.uk/news/uk-politics-65043759?at_medium=RSS&at_campaign=KARANGA	contact	2023-03-22 19:38:43
ビジネス	ダイヤモンド・オンライン - 新着記事	東洋建設がインフロニアTOBで「密約」の証拠メモ入手！任天堂創業家との対立で深まる疑惑 - Diamond Premium News	https://diamond.jp/articles/-/319794	東洋建設がインフロニアTOBで「密約」の証拠メモ入手任天堂創業家との対立で深まる疑惑DiamondPremiumNews海洋土木の東洋建設と、同社に買収提案する任天堂創業家の資産運用会社の対立が激化する中、“密約が争点に浮上している。	2023-03-23 04:57:00
ビジネス	ダイヤモンド・オンライン - 新着記事	アクセンチュア日本法人の財務を徹底解剖！推計時価総額は野村総研、電通に比肩!? - コンサル大解剖	https://diamond.jp/articles/-/319892		2023-03-23 04:55:00
ビジネス	ダイヤモンド・オンライン - 新着記事	脂質異常症で処方患者数の多い「人気薬」ランキング！トップ3の共通点とは？ - 選ばれるクスリ	https://diamond.jp/articles/-/317916	脂質異常症	2023-03-23 04:50:00
ビジネス	ダイヤモンド・オンライン - 新着記事	【無料公開】税理士志望なら必見！税理士試験の科目を免除できる大学院の選び方【入試難易度ランキング付き】 - Diamond Premiumセレクション	https://diamond.jp/articles/-/319752	【無料公開】税理士志望なら必見税理士試験の科目を免除できる大学院の選び方【入試難易度ランキング付き】DiamondPremiumセレクション税理士になるには、通常、税理士試験で合計科目の合格が求められるが、そのうち科目を占める税法科目の合格を一部免除できる方法がある。	2023-03-23 04:45:00
ビジネス	ダイヤモンド・オンライン - 新着記事	信越化学、日本ペイント、旭化成…化学5社「天国と地獄」最高益から8割超の最終減益まで - ダイヤモンド 決算報	https://diamond.jp/articles/-/319891	信越化学、日本ペイント、旭化成…化学社「天国と地獄」最高益から割超の最終減益までダイヤモンド決算報新型コロナウイルス禍に円安、資源・原材料の高騰、半導体不足など、日本企業にいくつもの試練が今もなお襲いかかっている。	2023-03-23 04:40:00
ビジネス	ダイヤモンド・オンライン - 新着記事	塩野義製薬は営業利益2.4倍、エーザイは8割減益…「新薬」話題の2社で格差の訳 - ダイヤモンド 決算報	https://diamond.jp/articles/-/319890	塩野義製薬は営業利益倍、エーザイは割減益…「新薬」話題の社で格差の訳ダイヤモンド決算報新型コロナウイルス禍に円安、資源・原材料の高騰、半導体不足など、日本企業にいくつもの試練が今もなお襲いかかっている。	2023-03-23 04:35:00
ビジネス	ダイヤモンド・オンライン - 新着記事	「構造的賃上げ」は金融政策では無理、賃金が抑えられる仕掛けの数々 - 政策・マーケットラボ	https://diamond.jp/articles/-/319882	金融政策	2023-03-23 04:30:00
ビジネス	ダイヤモンド・オンライン - 新着記事	「AIに仕事を奪われる！」と不安な人が思いも及ばない“準備の流儀”とは - 「40代で戦力外」にならない！新・仕事の鉄則	https://diamond.jp/articles/-/319533		2023-03-23 04:25:00
ビジネス	ダイヤモンド・オンライン - 新着記事	なぜひとびとは悪意を抱きつづけるのか？「進化的悪意」が科学の進歩に大きな貢献をした - 日々刻々 橘玲	https://diamond.jp/articles/-/319849	罵詈雑言	2023-03-23 04:20:00
ビジネス	ダイヤモンド・オンライン - 新着記事	協和キリンに「リストラの予兆」…グローバル化に向けた“人事改革”の中身とは - 医薬経済ONLINE	https://diamond.jp/articles/-/319902	online	2023-03-23 04:15:00
ビジネス	ダイヤモンド・オンライン - 新着記事	キヤノンやグンゼが国内生産回帰、サプライチェーン「途絶リスク」対策が本格化 - 物流専門紙カーゴニュース発	https://diamond.jp/articles/-/319846	キヤノンやグンゼが国内生産回帰、サプライチェーン「途絶リスク」対策が本格化物流専門紙カーゴニュース発ロシア・ウクライナ戦争の長期化、米中対立の先鋭化を背景に、経済効率性を重視したサプライチェーンの見直しが本格化してきた。	2023-03-23 04:10:00
ビジネス	ダイヤモンド・オンライン - 新着記事	FRBが0.25ポイント利上げ、早期の利上げ終了示唆 - WSJ発	https://diamond.jp/articles/-/319957	終了	2023-03-23 04:08:00
ビジネス	ダイヤモンド・オンライン - 新着記事	WBC優勝の裏に「稲盛和夫の教え」、栗山監督が“直筆の手紙”に記した感謝の言葉とは？ - News&Analysis	https://diamond.jp/articles/-/319893	newsampampanalysiswbc	2023-03-23 04:07:00
ビジネス	ダイヤモンド・オンライン - 新着記事	WBC優勝が皮肉にも「野球離れ」に拍車をかけかねない理由 - 情報戦の裏側	https://diamond.jp/articles/-/319889	侍ジャパン	2023-03-23 04:05:00
ビジネス	東洋経済オンライン	U-NEXTとParavi統合で｢国内ドラマ｣何が起こる ｢大胆予想｣テレビ局系サービスは1つになる？ | インターネット | 東洋経済オンライン	https://toyokeizai.net/articles/-/659473?utm_source=rss&utm_medium=http&utm_campaign=link_back	配信サービス	2023-03-23 05:00:00
ビジネス	東洋経済オンライン	各地で新幹線が進化｢春のダイヤ改正｣の見どころ 在来線は東西で“新駅"開業､利便性向上した？ | 新幹線 | 東洋経済オンライン	https://toyokeizai.net/articles/-/660946?utm_source=rss&utm_medium=http&utm_campaign=link_back	東洋経済オンライン	2023-03-23 04:30:00