投稿時間:2021-10-29 00:38:47 RSSフィード2021-10-29 00:00 分まとめ(39件)

カテゴリー等	サイト名等	記事タイトル・トレンドワード等	リンクURL	頻出ワード・要約等/検索ボリューム	登録日
IT	ITmedia 総合記事一覧	[ITmedia News] ニコン「Z 9」正式発表、メカシャッター排除　8Kを125分連続撮影　約70万円	https://www.itmedia.co.jp/news/articles/2110/28/news179.html	itmedia	2021-10-28 23:05:00
python	Pythonタグが付けられた新着投稿 - Qiita	straight skeleton (scikit-geometry 利用)	https://qiita.com/kkdd/items/f7210619d908d7e82db8	straightskeletonscikitgeometry利用こんにちは。	2021-10-28 23:26:27
python	Pythonタグが付けられた新着投稿 - Qiita	【個人用メモ】Pythonよく使うモジュール機能まとめ	https://qiita.com/uwasanoyosida/items/539e5125915faa9b82ad	目次モジュールカテゴリ内容BeautifulSoupスクレイピングページのテキスト情報全取得ページのリンク全取得pandasdf操作df作成dfの差分比較csv操作dfをcsvに保存csvを読み込んでdf作成BeautifulSoup↑ページのテキスト情報全取得↑testpyimportrequestsfrombsimportBeautifulSoupurlrym日本不審者情報センターのページhtmlrequestsgeturlHTML取得soupBeautifulSouphtmltexthtmlparser上のgetメソッドで取得したデータのままだと扱い辛いので、BeautifulSoupでHTML解析を行う。	2021-10-28 23:15:05
js	JavaScriptタグが付けられた新着投稿 - Qiita	obnizを楽器化してみたかった。。	https://qiita.com/khara-xedge/items/60288f3361a7320dc68f	実は・・今回室温の値をもとに音階を決定してスピーカーから出力していますが、聞いていただいた通り楽器とは言えません。	2021-10-28 23:34:16
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	Zeitwerk::NameErrorを解決したい	https://teratail.com/questions/366720?rss=all	ZeitwerkNameErrorを解決したい発生した問題RailsアプリをHerokuにデプロイしたところ、ZeitwerkNameErrorが発生してしまい、アプリがクラッシュしてしまいました。	2021-10-28 23:52:19
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	JSON.parseするとエラーが発生する	https://teratail.com/questions/366719?rss=all	jsonparse	2021-10-28 23:48:26
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	外部APIへのHTTPリクエストのとき、CORS Policyに反してブロックされてしまう	https://teratail.com/questions/366718?rss=all	外部APIへのHTTPリクエストのとき、CORSPolicyに反してブロックされてしまう前提・実現したいことJavascriptで外部のAPIを使うときに、HTTPリクエストを送りたい。	2021-10-28 23:45:32
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	Unityでスワイプ処理を今まで使っていたスクリプトにうまく合わせたい	https://teratail.com/questions/366717?rss=all	Unityでスワイプ処理を今まで使っていたスクリプトにうまく合わせたいUnityでiOS用のDアクションゲームを作っている初心者です。	2021-10-28 23:29:41
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	ファイルオブジェクトかどうかを判別する方法	https://teratail.com/questions/366716?rss=all		2021-10-28 23:29:08
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	スプレッドシート関数	https://teratail.com/questions/366715?rss=all	スプレッドシート関数宜しくお願い致しますスプレッドシートにB列からK列まで画像URLが入っているのですが、入っていないセルもあります。	2021-10-28 23:18:36
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	【GAS】スプレッドシートを読み込み、金額表記（1000円であれば、カンマありの1,000円出力）するにはどのようにすればよいか教えてください	https://teratail.com/questions/366714?rss=all		2021-10-28 23:11:19
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	drawerが表示されない	https://teratail.com/questions/366713?rss=all	drawerが表示されない前提・実現したいこと表題の通り、humbergerをクリック時に、下記サイトのようなnavを表示させたい。	2021-10-28 23:09:43
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	SQLでカラムの値によってJOINするテーブルを切り分けたい	https://teratail.com/questions/366712?rss=all	mysql	2021-10-28 23:08:03
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	JavaScript csvファイルを読み込みたい	https://teratail.com/questions/366711?rss=all	JavaScriptcsvファイルを読み込みたい前提・実現したいことcsvファイルを読み込んで二次元配列にし、それを関数で使いたいです。	2021-10-28 23:07:25
Ruby	Rubyタグが付けられた新着投稿 - Qiita	文字列と数値を使う！	https://qiita.com/TAKAHIRO__k/items/405d416215719e8b0dc2	文字列と数値を使う①irbコマンドターミナルを用いたプログラムの記述と実行をしますプログラムを実行する際、「逐一ファイルに記述し、実行」という手順は踏まないこととします代わりに、irbというコマンドで起動する簡易プログラム実行機能を利用しますirbコマンドは、ターミナルから直接Rubyのプログラムを動かすことができる機能を起動しますちなみに、irbは「InteractiveRuby」の略です今回はirbコマンドという機能を使って、ファイル無しでプログラムを実行します例ターミナルirbを起動irbirbではエンターキーを打つたびに、そこまでの記述を実行して結果を表示するということが分かりますirbを終了するときは、exitと記述して実行しますirbを終了irbmaingtexit問題なく終了すれば、プロンプトが表示される値を簡単に確認するときはirbを使います②Rubyにおける値数や文字をはじめ、プログラムで扱うデータを「値」と言いますプログラミングでは値の種類を明確に分けて扱います種類によってそれぞれ書き方が異なり、実行できる命令も異なります③プログラムにおける文字の操作・文字列文字列は、プログラミングの中で文字を扱うための値です文字列を生成するには、文字をダブルクォーテーションまたはシングルクォーテーションで囲みます以下の例を見てください例ターミナルThisisstringThisisstringtooこのように、文字列は簡単に生成できますirbでRubyの文字列を記述し、実行してみましょう例ターミナル文字列を書いてエンターキーで実行irbmaingtHelloこんにちは続けてこのように表示されれば成功gtHelloこんにちは全角スペースや日本語が含まれているとプログラムはエラーが生じますが、文字列の値として扱う場合は、エラーにはなりません④文字列の連結文字列同士は、お互いを連結できます文字列を連結させるには、文字列同士をプラスで繋ぎます。	2021-10-28 23:02:59
GCP	gcpタグが付けられた新着投稿 - Qiita	BigQueryを使った簡単なデータの前処理	https://qiita.com/IwasakiYuuki/items/20880e021b6d2f8a2eba		2021-10-28 23:40:19
Azure	Azureタグが付けられた新着投稿 - Qiita	Azureでアーキテクチャ設計する際に参考にすると良いサイト	https://qiita.com/Shin-27/items/1ebf7c24be440bda5363	Azureでアーキテクチャ設計する際に参考にすると良いサイトはじめにAzureでアーキテクチャ設計をする際、よく参考にしているサイトを記載します。	2021-10-28 23:27:53
Git	Gitタグが付けられた新着投稿 - Qiita	GitHub Universeで公開された新機能まとめ	https://qiita.com/ozora/items/df93c624132e169bc4a9	CommandPalettefromgithubcom上でCMDKを押すとMacでいうサーチバーのようなものが現れ、レポジトリ、Issue、PRなどの検索が行える。	2021-10-28 23:29:50
Git	Gitタグが付けられた新着投稿 - Qiita	GitlabのSAST（Semgrep）を動かす時のメモ	https://qiita.com/dyamaguc/items/b61186acc2344b917309	ジョブ自体はJobsucceededとなるのですが、WARNSemgrepTZNomatchinbuildsdyamaguctestgroupWebGoatと表示されていて、実際にはSemgrepが実行されておらず、結果的にレポートのglsastreportjsonが出力されていない旨のERRORが表示されています。	2021-10-28 23:26:20
Git	Gitタグが付けられた新着投稿 - Qiita	ローカルのファイルをGithubのリポジトリにアップロードする方法	https://qiita.com/chorota/items/a9cca0c81c5938776063	gitcommitmコメント名④ローカルのリポジトリをGithub上のリモートリポジトリ上に紐づけします。	2021-10-28 23:09:05
技術ブログ	Developers.IO	StepFunctionsからAmazon SNSを使用してSMSのメッセージ送信はできなかった	https://dev.classmethod.jp/articles/it-was-not-possible-to-send-sms-messages-using-amazon-sns-from-stepfunctions/	amazonsns	2021-10-28 14:54:44
技術ブログ	Developers.IO	Coc.nvimの拡張を環境変数経由でNeoVim起動時に自動インストールされるようにしてみた	https://dev.classmethod.jp/articles/coc-extension-install-by-init-settings/	airdrop	2021-10-28 14:23:55
海外TECH	DEV Community	7 Common MISTAKES made by C# developers (+ How to avoid them)	https://dev.to/dotnetsafer/7-common-mistakes-made-by-c-developers-how-to-avoid-them-1na4	Common MISTAKES made by C developers How to avoid them Developers will always make mistakes no one is impeccable Many of them particularly C developers learn and grow via trial and error I prefer to refer to it as transitioning from junior to senior developer Although this technique of trial and error works if a developer does not identify errors they might cause several efficiency and quality issues in the developed application In this article I ve compiled a list of the most frequent and not so common mistakes made by C developers Usage of String ConcatenationString Concatenation Functions are really simple to use When something is added to the string a new address is immediately created in memory The previous string is transferred to the new section which now has a different location this is not efficient at all Let s see an example List dotnetsaferTools new List Shield Secrets Linker string tools string Empty foreach var tool in dotnetsaferTools tools tool Creates a copy of the string in memory SolutionThe way to solve this is simple you have to use StringBuilder object instead of string aggregation which will keep the same position in memory without any copy task This simplifies and speeds up the string concatenation process allowing us to smoothly add all of the actions we require This is how it would look like StringBuilder toolsBuilder new StringBuilder foreach var tool in dotnetsaferTools toolsBuilder Append tool Iterating with Values instead of with LINQThis problem happens when you attempt to iterate through a list of records using a loop which is not optimum foreach Customer customer in CustomerList if customer State ES tax customer Balance SolutionInstead of doing this we should simply use LINQ which will allow us to query objects easily tax CustomerList Where customer gt customer State Equals ES Sum c gt c Balance Note This is more efficient than a foreach loop Not using Yield ReturnThis is for when you need to create an object enumeration for some other caller Using this function will greatly improve performance SolutionTo prevent making this error simply do not attempt to build a return collection The main benefit of utilizing yield return is that the complete collection does not need to be held in memory This will allow you to have control after each iteration and you will only process the results that you need and that are necessary Deferred Execution in LINQThe main one of LINQ is to facilitate querying data in case we use foreach loops an example is nested if blocks To get a list of clients I recommend using LINQ to SQL public IEnumerable GetCustomer using var context new DBContext return from c in context Customer where c Balance gt select c Everything appears to be in order doesn t it Only until we try to count the items in the collection it will return an exception LINQ will not conduct any queries until we try to enumerate the results SolutionThe solution is simple just convert all LINQ queries to a ToArray By doing this LINQ should evaluate the queries before anything else Accessing Virtual Members within a ConstructorAlthough this error is not one of the first it is one of the most common although it may not seem so When an overridden method is called directly from the constructor of a base class this error will appear because it comes from executing code before its time public class Parent public Parent Console WriteLine Parent Ctor Method public virtual void Method Console WriteLine Parent method public class Child Parent public Child Console WriteLine Child Ctor public override void Method Console WriteLine Child method SolutionAlthough it may seem very complex it is actually very easy We simply have to mark that class as sealed By doing this we make sure that when we call the virtual method we will not receive any warning Not knowing the importance of USING for Object DisposalAs partech nl says many C developers are unfamiliar with the concept that the using keyword is not only used as a directive for adding namespaces but is also very beneficial for object removal SolutionIf you are really sure that an object should be deleted when performing certain actions simply use the using statement to make sure that the object has been deleted correctly Let s see an example using DisposableClass DisposableObject new DisposableClass DisposableObject DoTheJob Does the same as DisposableClass DisposableObject new DisposableClass try DisposableObject DoTheJob finally DisposableObject Dispose Using Weak Cryptographic AlgorithmsHere we agree with Stackify this error is simply based on the fact that many developers use obsolete or vulnerable encryption algorithms which is very easy to avoid In this case we have for example SHA or RIPEMD which do not offer a current and very good level of security using System Security Cryptography var hashAlg SHA Create SolutionThe solution to this is to use stronger encryption algorithms for example SHA instead of SHASHA instead of RIPEMDAES instead of TripleDES ConclusionSoftware development is the process of developing applications that can be shared with others As software developers we are responsible for creating the code that makes up the application as well as designing and testing it We are also responsible for updating or adding new features to an existing application when necessary The world of software development is changing it s no longer just about coding but also about solving problems understanding user behavior and much more That s why I hope you found this article useful	2021-10-28 14:33:47
海外TECH	DEV Community	Build Universal Web Components with LitElement	https://dev.to/codesphere/build-universal-web-components-with-litelement-2kam	Build Universal Web Components with LitElementIt feels like every day a new web framework appears fighting for your attention With all these different frameworks however you might find yourself rewriting code to get your components working right on a new framework If you re looking to create a universal component library look no further than LitElement a simple base class for creating cross library components with their own embedded styling and javascript In this tutorial I ll be showing you how to create your first LitElement component and render it in a vanilla HTML file SetupTo get started we re going to create a new npm project withnpm initWe are then going to install lit element and Polymer with npm i lit element polymerPolymer isn t necessary but will just allow us to test this component easily Finally let s create an index html and a card jstouch index html card js CodeOur index html can be as simple as Note that all we need is the inclusion of the script tags to our custom component as well as the component itself of course Our Card js can then look as follows And that s all it takes We can then serve this page live with polymer serverAnd lo and behold Using Lit Element AnywhereWe can then use this exact same LitElement class in React Vue Angular or whatever web framework you or your fellow developers prefer All that needs to be done is to install Litelement and then access the custom tag you created So what do you think Do you see a benefit of using LitElement or is this just an extraneous layer Let us know down below As always happy coding from your friends at Codesphere the most intuitive cloud provider on the market	2021-10-28 14:13:10
海外TECH	DEV Community	Performance capabilities of data warehouses and how Cube can help	https://dev.to/cubejs/performance-capabilities-of-data-warehouses-and-how-cube-can-help-257d	Performance capabilities of data warehouses and how Cube can helpDon t you really hate latency Yeah I hate it too Today I want to talk about building applications on top of data warehouses I want to discuss how to achieve low latency if your app is consuming data from BigQuery Snowflake Redshift or any other cloud based data warehouse My goal is to give you a crash course into data warehouse performance explain how to understand the performance of data APIs and highlight a few tools that help build responsive apps on top of data warehouses What are Data Warehouses Broadly speaking data warehouses are central locations for data from multiple sources Call it a single source of truth It s up to you to set up a process to extract transform and load ETL data from source systems based on a schedule or a set of events Then you usually run reporting and data analysis to get some business insights Data warehouses are usually optimized for online analytical processing OLAP It means running a relatively low volume of complex analytical queries with heavy joins of billion row tables The queries will often include numerical calculations using aggregate functions like AVG COUNT DISTINCT PERCENTILE CONT and many more like them As a direct consequence data warehouses are often designed as auto scaling and heavily distributed systems with columnar storage engines Snowflake Data Cloud Google BigQuery and Amazon Redshift are all good examples of such data warehouses and the most used and popular choice for storing huge amounts of data If your company has a data warehouse in use chances are it s one of these behemoths OLAP is often opposed to OLTP which stands for online transaction processing It means running a high volume of not so complex queries where the data is mostly inserted or updated rather than being read However the spectrum is much wider and there are more dimensions to it pun intended As companies collect and transfer crazy volumes of data into their data warehouses they want insight into this data Your internal users become more interested in getting on demand reports instead of weekly printouts Your business becomes less tolerant of BI tools with time to insight comparable to how long it takes to make a cup of coffee rather than the time it takes to think about making one You as an application developer suddenly need to figure out how to build responsive and performant apps on top of data warehouses while keeping your business and users happy But what exactly does responsive and performant mean Let s figure that out Data Warehouse Performance from an App Developer s PerspectiveAs application developers we d like our users to maintain the feeling that they operate directly on the data regardless of an operation data volume or amount of other users acting in parallel Studies say that an app reacts instantaneously if it responds to user actions within ms and second is the limit above which an app risks interrupting their flow of thought Here we come to query latency LatencyQuery latency is the amount of time it takes to execute a query and receive the result Now let s take the speed of light into account Seriously let s imagine that a user from California sends a request to your app deployed in a popular region e g us east on AWS With the roundtrip of twice the distance of miles and the speed of light of mi s the minimum delay between sending the request and getting the response would be ms Wait and what about our overseas users You can actually use massively distributed CDNs like CloudFront or Netlify Edge to bring your app closer to users but how practical is it for the data warehouses that your app interacts with It means that any request that your app makes to a data warehouse should take less than second Actually way less than second if we d like to account for the speed of light network delays query execution time time to run our business logic on top of the data etc What affects the query execution time Primarily it s the amount of data that needs to be scanned With larger datasets these scans take more time To distribute load data warehouses use multiple nodes which introduces delays for inter node communications Now let s see what real world data warehouses can offer Query Latency in BigQuery Let s start with BigQuery a serverless big data warehouse available as a part of the Google Cloud Platform It s highly scalable meaning that it can process tiny datasets as well as petabytes of data in seconds using more cloud capacity as needed You re able to manage its performance by choosing the pricing model on demand by default flat rate available as an option which affects how BigQuery allocates slots its virtual compute units with CPUs memory and temporary storage used to execute queries BigQuery automatically calculates how many slots are required by each query depending on query size and complexity It s worth noting that every query competes for the slots with other queries within a GCP project and also with other projects if you stick with the default on demand pricing model as many do Let s use BigQuery s interactive console in GCP to estimate possible latencies After importing TPC H data with million rows into my own data set I wanted to figure out a few sample queries I d need for my analytics First of all I d want to see a list of all orders per day of certain status Here s the SQL query I ran in BigQuery The query takes seconds to run and it processed GB If you run it twice you ll see that the second run latency is well under a second because the results of the first run were cached However if you change the query maybe add or update a filter in the WHERE clause you ll still experience the same query latency of around a second and a half Why The answer is caching BigQuery will cache the response of a query so the subsequent identical query gets a much quicker response time The BigQuery docs explain in more detail what cached results are But what you need to know is that to retrieve data from the cache the duplicate query text must be exactly the same as the original query All query results are cached in temporary tables for approximately hours The pricing of cached queries is confusing You are not charged for queries that use cached results but these queries are subject to the BigQuery quota policies BigQuery is distributed by nature and its compute units are by default shared between users That s why in BigQuery query latency includes not only query execution time but also initialization time which is spent to build a query plan check quotas and limits and allocate slots That s why it s unrealistic to expect BigQuery to provide sub second query latency Okay but what about Snowflake Query Latency in Snowflake ️Unlike BigQuery Snowflake doesn t share its compute resources between users It processes queries using so called virtual warehouses and each virtual warehouse contains multiple dedicated compute nodes allocated from a cloud provider for your project So each virtual warehouse does not share compute resources with other virtual warehouses and has no impact on the performance of other virtual warehouses However your own queries will indeed compete for the resources of your virtual warehouses Snowflake s query latency is improved by having pre allocated virtual warehouses However everything comes at a price pun intended Running a virtual warehouse will cost you something from to credits per hour This ends up being between and USD per day according to Snowflake s usage based pricing model Nobody likes to throw money down the drain for nothing so Snowflake supports auto suspending idle virtual warehouses after a specified amount of time A suspended warehouse doesn t consume credits but when a query comes it should be re provisioned Snowflake claims that warehouse provisioning generally takes or seconds however depending on the size of the warehouse and the availability of compute resources it can take longer It means that with Snowflake we should either pay a significant premium or expect intermittent delays when a suspended virtual warehouse is being provisioned Let s use Snowflake s interactive console to check our intuition I picked the TPC H public dataset and used the built in sample database provided by Snowflake This data is provided in several schemas in the SNOWFLAKE SAMPLE DATA shared database based on the exponent of increasing from the base data set of around a few several million elements The schema I used is the TPCH SF schema with million rows The data I want is an exact match for the query I ran against BigQuery in the example above I want to see the total amount of orders grouped by day and status in a period between the years and The query ran for seconds It returned rows from a data set containing million elements I used the X Small compute instance in Snowflake for this test Of course you can scale up the compute instances But how much money are you willing to spend So it puts us in a world where getting a sub second query latency from a data warehouse is something like a magic unicorn And I ve only been talking about latency so far We can all agree there are many more factors in play Let me continue by explaining a few of them below Latency vs ConcurrencyAs application developers we build products used by hundreds thousands and millions of users Rarely is it ever a lone user executing a single query at a given time Your app should be able to run multiple queries against a data warehouse in parallel The issue is that these queries will compete for available resources The question is how would these parallel competing queries impact query latency We know that BigQuery can provide an approximate second response time for a query that processes vast amounts of data possibly petabytes What would the response times be for simultaneous queries Or maybe during Black Friday even a few hundred queries Query concurrency is the amount of actively co executing parallel queries The actively part is important here because data warehouses can queue queries over a certain limit and run them only when previous queries are completed In BigQuery according to its quotas and limits concurrency is capped at queries per project That s a relatively high number just beware of that per project part It means that the quota is shared between all apps that interact with the same GCP project Queries with results that are returned from the query cache are also subject to the quota The reason is that BigQuery needs to determine that it is a cache hit However you re not charged money for queries that use cached results Because BigQuery is such a black box the best course of action is to improve query performance Luckily there s a detailed explanation in the BigQuery docs on how to achieve this Here s a quick rundown You should limit the input data and data sources the fewer bytes your query reads the better Never use SELECT as it will scan all columns in the data set Next improve communication between slots by reducing data before using a GROUP BY or JOIN clause Use ORDER BY and LIMIT only in the outermost query to improve the computation performance of queries and manage the query output Remember keeping the output bytes low is also important Because BigQuery is capped at concurrent queries per project using these best practices to improve query performance is definitely a must Snowflake is a bit different It can use node sizes anywhere from to credits in a warehouse Here s an explanation of how credits are charged A warehouse can scale horizontally like a cluster if you select adding warehouses to a multi cluster warehouse Resizing a warehouse can improve query performance particularly for larger more complex queries However warehouse resizing is not intended for handling concurrency issues instead use additional warehouses to handle the workload or use a multi cluster warehouse Keep in mind that larger warehouses are not faster for smaller more basic queries Small queries do not need a larger warehouse because they won t benefit from the additional resources regardless of the number of queries being processed concurrently In general you should try to match the size of the warehouse to the expected size and complexity of the queries to be processed by the warehouse So if we expect multiple queries to run in parallel we should research and be aware of query latency degradation that happens when the concurrency grows Let s benchmark BigQuery to know for sure Latency vs Concurrency in BigQueryI decided to run a few load test benchmarks with k io to measure the response time percentile when querying BigQuery directly by using the SDK through a Node js API Here s the source code for the load tests I decided to run a set of randomly generated queries BigQuery is incredibly scalable and elastic however only up to concurrent users First of all I ran a benchmark with concurrent user for seconds I can already see the percentile latency is unacceptable concurrent userp sp sI then decided to run the same benchmark with concurrent users for seconds concurrent usersp sp sThen a load test with concurrent users concurrent usersp sp sAs you can see the latency stays almost the same as BigQuery is autoscaling to handle the load It looks almost the same even with concurrent users concurrent usersp sp sHowever the issues pop up with more than concurrent users Request iterations started failing due to the concurrency limit High concurrency with sub second latency Here comes the real question How to get acceptable latency with high concurrency Does this unicorn even exist For this to become reality you need materialized views It s a set of pre computed results from queries You also need OLAP cubes Remember at the beginning of the article I mentioned OLAP Well here s why it s important OLAP cubes are pre computed multi dimensional datasets By using OLAP cubes you can filter a dataset by one or more dimensions and aggregate values along select dimensions By using materialized views with OLAP cubes you can generate a condensed version of the source data by specifying attributes from the source itself This simple yet powerful optimization can reduce the size of the data set by several orders of magnitude and ensures subsequent queries can be served by the same condensed data set if any matching attributes are found What is Cube Cube is an API server for making sense of huge datasets It doesn t get any simpler than that It s the de facto Analytics API for Building Data Apps Why is Cube so cool Let me tell you It s open source with more than stars on GitHub Cube also integrates with every major database on the market today With Cube you can create a semantic API layer on top of your data manage access control cache and aggregate data Cube is also visualization agnostic It s up to you to use any front end visualization library to build your own dashboards Building an Analytics APII ll run a Cube instance to build the actual analytics API It ll generate all the queries I need to run against BigQuery Note Keep in mind you can set up Cube with any database of your choice including Snowflake Redshift or any other data warehouse from the list here Configuring Cube with Docker Compose is the simplest way to get started Note This example will run a single cube instance For production I recommend running multiple Cube instances They include an API a refresh worker and Cube Store for caching I ll talk about production ready configs a bit further down in the article First up create a new directory Give it a name and make a docker compose yaml file Copy this code into it version services cube image cubejs cube latest ports Cube js API and Developer Playground Dashboard app if created environment CUBEJS DEV MODE true CUBEJS DB TYPE bigquery CUBEJS DB BQ PROJECT ID your project id CUBEJS DB BQ KEY FILE path to your key json CUBEJS API SECRET somesecret volumes cube conf Configuring Access to BigQueryIn my GCP account I added a service account for accessing BigQuery Here are the permissions I needed I then created a key for this service account and downloaded it to my local machine This means I can programmatically access this dataset once I start building my analytics API Yes Make sure to set the CUBEJS DB TYPE to bigquery The CUBEJS DB BQ PROJECT ID value should match the Project ID you created above And the CUBEJS DB BQ KEY FILE is the JSON key you create for your Service Account What I tend to do is paste the key json in the Cube directory Referencing it in the docker compose yaml is simpler that way Next up start Docker Compose Open a terminal window in the Cube directory and run docker compose upThis will start the Cube API server and the Developer Playground The Playground is used to test queries create schemas generate SQL and so much more Open up localhost in your browser Navigate to the Schema page Here I generated a Schema from the orders table in the TPC H data set You ll see a file show up under Files This is the autogenerated Schema file that will contain measures and dimensions for all analytics values I did need to edit a time dimension however to CAST it to a TIMESTAMP oOrderdate sql CAST CUBE O ORDERDATE AS TIMESTAMP type time Next up I ran the same query against the TPC H data set through the Playground I get the same result set which means the config works You can also see the generated SQL query I m running this query without pre aggregations meaning it will almost take as long as querying BigQuery directly This might be fine for testing but not for running in production There are two ways you can go about running Cube in production First you can use the hosted Cloud deployment of Cube Or run a production ready setup yourself Pre aggregations will cache your data and make it available for quicker querying It s the bread and butter of Cube and what makes it so powerful for building apps on top of Data Warehouses I ll explain it in more detail in the section below Performance Improvement with Pre AggregationsIn simple English a pre aggregation is a condensed version of source data A pre aggregation specifies attributes from the source which Cube uses to condense the data This optimization can reduce the size of the data set by several orders of magnitude and ensures subsequent queries can be served by the same condensed data set if any matching attributes are found When you run a query in the Playground you ll see a button show up called Query was not accelerated with pre aggregation Clicking on it opens this popup below This is a guide on how to enable a pre aggregation for this query In the schema folder in my Cube app I opened the Orders js file and added this piece of code in the pre aggregations block Note The Orders js file is located under the Schema tab in Cube Cloud cube Orders preAggregations main measures Orders count dimensions Orders oOrderstatus refreshKey every hour updateWindow day incremental true partitionGranularity month timeDimension Orders oOrderdate granularity day Here I specify what to pre aggregate I want a rollup on the count measure and the oOrderstatus dimension After adding the pre aggregation I ran the same query again This time it ran in under ms I consider this a win Raw Data Warehouse vs Cube Performance BenchmarksI ran a few load tests with k io to measure the response time percentile differences between querying BigQuery directly by using the SDK through a Node js API versus using Cube through a Node js API Here s the source code for the load tests The tests themselves generate random queries that hit the database with the sole purpose of avoiding the query cache This will benchmark the raw performance of the database itself Let me remind you that with concurrent users querying the BigQuery SDK for seconds I see percentiles between and seconds which is astonishingly far from ideal concurrent users BigQuery SDKp sp sWith Cube I see percentiles around a half of a second with the same amount of concurrent users concurrent users Cubep msp msHowever I see a significant performance difference when running with more than concurrent users concurrent users Cubep msp msThat s significantly quicker than it would take to query BigQuery directly with even one concurrent user Wrapping UpIn this blog post I ve tried making sense of data warehouse performance I ve also explored their theoretical and practical limits in terms of query latency and query concurrency You ve learned how to find the magic unicorn of high concurrency and sub second latency by creating an analytics API with Cube that adds a powerful caching layer on top of any data warehouse I hope you re now confident that it s possible to drop latency to below a second while getting a consistent concurrency of above queries per second The cool part is that the p latency with Cube at queries per second will stay below the p latency of BigQuery at queries per second I d love to hear your feedback about these benchmarks in the Cube Community Slack Click here to join Until next time stay curious and have fun coding Also feel free to leave Cube a on GitHub if you liked this article ️	2021-10-28 14:05:40
Apple	AppleInsider - Frontpage News	Best deals Oct. 28: $80 1TB external SSD, $99 Apple Pencil 2, more!	https://appleinsider.com/articles/21/10/28/best-deals-oct-28-80-1tb-external-ssd-99-apple-pencil-2-more?utm_medium=rss	Best deals Oct TB external SSD Apple Pencil more Thursday s best deals include big Halloween sales from accessory manufacturers Belkin and Satechi off a Crucial X TB SSD and both versions of the Apple Pencil on sale Best deals for October There are a lot of sales each day but only a handful are worth pursuing So rather than sifting through miles of advertisements we ve hand picked a bunch just for the AppleInsider audience Read more	2021-10-28 14:10:46
Apple	AppleInsider - Frontpage News	Nikon unveils new mirrorless flagship Z9 camera & additional Nikkor lenses	https://appleinsider.com/articles/21/10/28/nikon-unveils-new-mirrorless-flagship-z9-camera-additional-nikkor-lenses?utm_medium=rss	Nikon unveils new mirrorless flagship Z camera amp additional Nikkor lensesNikon has officially unveiled the Nikon Z the company s first mirrorless flagship camera alongside a number of new lens options Credit NikonThe Nikon Z features a professional body speedy autofocus performance and K video resolution at up to frames per second The company says it s a well rounded camera that s suited for both filmmakers and still photographers Read more	2021-10-28 14:43:30
海外TECH	Engadget	Ninja's 10-in-1 multicooker is 41 percent off at Amazon today	https://www.engadget.com/ninja-foodi-10-in-1-multicooker-sale-142500132.html?src=rss	Ninja x s in multicooker is percent off at Amazon todayDo you want a do everything multicooker but don t need an Instant Pot You re in luck Amazon is selling Ninja s OS Foodi in for a percent discount on the regular price as part of a one day sale This may be the best Ninja cooker deal yet ーthere was a Prime Day deal last year for a less capable if larger in model that still cost more Buy OS Foodi in at Amazon It s easy to see the appeal for this model It handles typical multicooker tasks like pressure cooking slow cooking steaming and yogurt but it also includes air frying not always present in multicookers and a nesting broil rack that lets you broil or steam on different levels The quart capacity is also just slightly higher than the six quarts you find in comparable Instant Pot models so Ninja s offering is worth considering if you re worried about space The drawbacks mainly stem from your choice of ecosystem Instant Pot s fandom is large enough that you ll find recipes and other guides for it where you won t with Ninja s cookers If you re just interested in preparing a tasty dish though there s no reason you have to follow the crowd ーespecially not at a price like this Follow EngadgetDeals on Twitter for the latest tech deals and buying advice	2021-10-28 14:25:00
海外TECH	Engadget	Apple is the first public partner to join sustainable chip initiative	https://www.engadget.com/imec-ssts-sustainable-chip-program-141057882.html?src=rss	Apple is the first public partner to join sustainable chip initiativeIt may not be a household name but one of the more important players in the semiconductor industry is Imec It s a nonprofit research organization that has been around since and frequently works with companies like ASML currently the only manufacturer in the world that makes the EUV lithography machines necessary for making nm nm and nm chips nbsp On Thursday the organization announced the creation of the Sustainable Semiconductor Technologies and Systems program As part of the initiative Imec says it will work with the semiconductor industry to reduce its carbon footprint by helping partners analyze and anticipate the environmental impacts of their manufacturing decisions With today s announcement the program also has its first partner Apple If the program is to have a positive effect on the environment Imec will need to attract more partners but attracting Apple is a good first step The company may not produce its own silicon but there s no denying it s on the bleeding edge of chip design More importantly it s among the most powerful buyers of manufacturing capacity in the industry It frequently monopolizes the top end processes of TSMC the most important chip foundry in the world Working together Imec and Apple are well positioned to inform other companies how to make their manufacturing more sustainable	2021-10-28 14:10:57
Cisco	Cisco Blog	Cisco Knowledge Network: Extend visibility into the Internet, Cloud, and SaaS with Cisco SD-WAN and ThousandEyes	https://blogs.cisco.com/networking/cisco-knowledge-network-extend-visibility-into-the-internet-cloud-and-saas-with-cisco-sd-wan-and-thousandeyes	Cisco Knowledge Network Extend visibility into the Internet Cloud and SaaS with Cisco SD WAN and ThousandEyesJoin us for our next Cisco Knowledge Network webinar Extend visibility into the Internet Cloud and SaaS with Cisco SD WAN and ThousandEyes on Wednesday November at am PST GMT	2021-10-28 14:11:02
金融	RSS FILE - 日本証券業協会	株券等貸借取引状況（週間）	https://www.jsda.or.jp/shiryoshitsu/toukei/kabu-taiw/index.html	貸借	2021-10-28 15:30:00
金融	金融庁ホームページ	金融審議会「ディスクロージャーワーキング・グループ」（第３回）議事次第を公表しました。	https://www.fsa.go.jp/singi/singi_kinyu/disclose_wg/siryou/20211029.html	金融審議会	2021-10-28 15:00:00
ニュース	BBC News - Home	Covid: All countries to be removed from England's travel red list	https://www.bbc.co.uk/news/uk-59063818?at_medium=RSS&at_campaign=KARANGA	england	2021-10-28 14:11:49
ニュース	BBC News - Home	Covid: Moscow shops and restaurants shut in partial lockdown	https://www.bbc.co.uk/news/world-europe-59076152?at_medium=RSS&at_campaign=KARANGA	covid	2021-10-28 14:02:39
ニュース	BBC News - Home	Heavy rain sees homes flooded and travel disrupted in Cumbria	https://www.bbc.co.uk/news/uk-england-cumbria-59066324?at_medium=RSS&at_campaign=KARANGA	weather	2021-10-28 14:27:53
ニュース	BBC News - Home	Yorkshire to take no disciplinary action over Rafiq racism allegations	https://www.bbc.co.uk/sport/cricket/59080881?at_medium=RSS&at_campaign=KARANGA	Yorkshire to take no disciplinary action over Rafiq racism allegationsYorkshire will not take disciplinary action against any player employee or executive after a report found Azeem Rafiq had been a victim of racial harassment and bullying at the club	2021-10-28 14:37:52
ニュース	BBC News - Home	Dominant Raducanu reaches quarter-finals in Romania - but Norrie beaten in Vienna	https://www.bbc.co.uk/sport/tennis/59071114?at_medium=RSS&at_campaign=KARANGA	Dominant Raducanu reaches quarter finals in Romania but Norrie beaten in ViennaBritain s Emma Raducanu produces a composed display to beat Romania s Ana Bogdan in straight sets and reach the Transylvania Open quarter finals	2021-10-28 14:37:09
ニュース	BBC News - Home	Travel rules: What tests do I need and what about the red list?	https://www.bbc.co.uk/news/explainers-52544307?at_medium=RSS&at_campaign=KARANGA	covid	2021-10-28 14:46:53
北海道	北海道新聞	米、大型歳出２００兆円案　大統領、当初規模を半減	https://www.hokkaido-np.co.jp/article/605494/	気候変動	2021-10-28 23:12:00