投稿時間:2022-06-23 23:33:29 RSSフィード2022-06-23 23:00 分まとめ(38件)

カテゴリー等	サイト名等	記事タイトル・トレンドワード等	リンクURL	頻出ワード・要約等/検索ボリューム	登録日
python	Pythonタグが付けられた新着投稿 - Qiita	歪度、尖度の計算(Python)	https://qiita.com/yellow_detteiu/items/06ffa27b415eff76ee38	機械学習	2022-06-23 22:24:50
python	Pythonタグが付けられた新着投稿 - Qiita	全日本大学バスケットボール新人戦（プレ大会）競技日程のPDFをCSVに変換	https://qiita.com/barobaro/items/cc67134186784659513b	wgethttpsj	2022-06-23 22:12:43
js	JavaScriptタグが付けられた新着投稿 - Qiita	【GAS】GAS+スプレッドシートが思ったより辛かったので、EntityとRepositoryを導入する	https://qiita.com/taimax/items/db2b859f851be5ded53b	entity	2022-06-23 22:51:57
js	JavaScriptタグが付けられた新着投稿 - Qiita	WebSocket Byte、Stringの送受信の検証	https://qiita.com/dlstjq7685/items/3926fcddc1f00405d780	arraybuffer	2022-06-23 22:16:19
Linux	Ubuntuタグが付けられた新着投稿 - Qiita	cronって、シンボリックリンクを認識しないの？	https://qiita.com/jennyx220/items/ba483afda6be5f82bf84	homeoredocumentspysoftpy	2022-06-23 22:31:34
AWS	AWSタグが付けられた新着投稿 - Qiita	スナップショット とは？EBSとAMIの違いも見てみた	https://qiita.com/miyuki_samitani/items/4c3cb765c682c8f42ee3	違い	2022-06-23 22:43:56
Docker	dockerタグが付けられた新着投稿 - Qiita	7.6.10 Equality operators [expr.eq] C++N4910:2022 (65) p143.cpp	https://qiita.com/kaizen_nagoya/items/892e5783752decb3c981	draft	2022-06-23 22:56:50
Docker	dockerタグが付けられた新着投稿 - Qiita	7.6.4 Pointer-to-member operators [expr.mptr.oper] C++N4910:2022 (64) p139.cpp	https://qiita.com/kaizen_nagoya/items/fb693da56e224c5a731d	draft	2022-06-23 22:25:30
Docker	dockerタグが付けられた新着投稿 - Qiita	7.6.3 Explicit type conversion (cast notation) [expr.cast] C++N4910:2022 (63) p138.cpp	https://qiita.com/kaizen_nagoya/items/640c0d2dcbba8c20aa67	conversion	2022-06-23 22:20:18
Docker	dockerタグが付けられた新着投稿 - Qiita	7.6.2.8 New [expr.new] C++N4910:2022 (62) p132.cpp	https://qiita.com/kaizen_nagoya/items/fe0e93b830b88846e77b	draft	2022-06-23 22:16:12
golang	Goタグが付けられた新着投稿 - Qiita	WebSocket Byte、Stringの送受信の検証	https://qiita.com/dlstjq7685/items/3926fcddc1f00405d780	arraybuffer	2022-06-23 22:16:19
Ruby	Railsタグが付けられた新着投稿 - Qiita	経験ゼロの新人が1ヶ月でウェブアプリを作らされるヤバい研修 part 3	https://qiita.com/Terao-Takumi/items/dade773928422ee7052b	partpart	2022-06-23 22:29:12
海外TECH	MakeUseOf	How to Untag Yourself From an Instagram Post	https://www.makeuseof.com/how-to-untag-instagram-post/	instagram	2022-06-23 13:45:14
海外TECH	MakeUseOf	All of the Color Correcting Options Available for Final Cut Pro, Explained	https://www.makeuseof.com/final-cut-pro-color-correcting-options-explained/	All of the Color Correcting Options Available for Final Cut Pro ExplainedColor correcting is an essential step for neutralizing a video scene Here are all the color correcting tools you need to know in Final Cut Pro	2022-06-23 13:30:14
海外TECH	DEV Community	Can googling be considered a must-have skill for developers?	https://dev.to/krishnaagarwal/can-googling-be-considered-a-must-have-skill-for-developers-33o4	commentsi	2022-06-23 13:44:42
海外TECH	DEV Community	4Bn rows/sec query benchmark: Clickhouse vs QuestDB vs Timescale	https://dev.to/questdb/4bn-rowssec-query-benchmark-clickhouse-vs-questdb-vs-timescale-i9g	Bn rows sec query benchmark Clickhouse vs QuestDB vs TimescaleThis article is originally published on questdb io by Andrey Pechkurov We introduced JIT Just in Time compiler for SQL filters in our previous version QuestDB As we mentioned last time the next step would be to parallelize the query execution when suitable to improve the execution time even further and that s what we re going to discuss and benchmark today QuestDB enables JIT compiled filters by default and what s even more noticeable includes parallel SQL filter execution optimization allowing us to reduce both cold and hot query execution times quite dramatically Prior to diving into the implementation details and running some before after benchmarks for QuestDB we ll be having a friendly competition with two popular time series and analytical databases TimescaleDB and ClickHouse The purpose of the competition is nothing more but an attempt to understand whether our parallel filter execution is worth the hassle or not Comparing with other databasesOur test box is a ca xlarge AWS VM running Ubuntu Server bit In practice this means vCPU and GB RAM The attached storage is a TB gp volume configured for MB s throughput and IOPS Apart from that we ll be using QuestDB with the default settings which means both parallel filter execution and JIT compilation being enabled In order to make the benchmark easily reproducible we re going to use TSBS benchmark utilities to generate the data We ll be using so called IoT use case tsbs generate data use case iot seed scale timestamp start T Z timestamp end T Z log interval s format influx gt tmp data The above command generates six months of per minute measurements for truck IoT devices This yields almost billion records stored in a table named readings Loading the data is as simple as tsbs load questdb file tmp dataNow when we have the data in the database we re going to execute the following query on the readings table SELECT FROM readings WHERE velocity gt AND latitude gt AND latitude lt AND longitude gt AND longitude lt This kinda synthetic query aims to find all measurements sent from fast moving trucks in the given location Apart from that it has a filter on three DOUBLE columns and doesn t include analytical clauses like GROUP BY or SAMPLE BY which is exactly what we need Our first competitor is TimescaleDB running on top of PostgreSQL As the official installation guide suggests we made sure to run timescaledb tune to fine tune TimescaleDB for better performance We generate the test data with the following command tsbs generate data use case iot seed scale timestamp start T Z timestamp end T Z log interval s format timescaledb gt tmp data That s the same command as before but with the format argument set to timescaledb Next we load the data tsbs load timescaledb pass your pwd file tmp dataBe prepared to wait for quite a while for the data to get in this time We observed x ingestion rate difference between QuestDB and two other databases in this particular environment Yet that s nothing more but a note for anyone who wants to repeat the benchmark If you d like learn more on the ingestion performance topic check out this blog post Finally we re able to run the first query and measure the hot execution time Yet if we do it it would take more than minutes for TimescaleDB to execute this query At this point experienced TimescaleDB amp PostgreSQL users may suggest us to add an index to speed up this concrete query So let s do that CREATE INDEX ON readings velocity latitude longitude With an index in place TimescaleDB can execute the query much much faster in around seconds To get the full picture let s include one more contestant The third member of our competition is ClickHouse Just like with TimescaleDB the command to generate the data stays the same with only the format argument being set to clickhouse Once the data is generated it can be loaded into the database tsbs load clickhouse file tmp dataWe re ready to do the benchmark run The above chart shows that QuestDB is an order of magnitude faster than both TimescaleDB and ClickHouse in this specific query Interestingly an index based scan doesn t help TimescaleDB to win the competition This is a nice illustration of the fact that a specialized parallelism friendly storage model may save you from having to deal with indexes and paying the additional overhead during data ingestion As the next step let s give another popular type of query a go In the world of time series data it s common to query only the latest rows based on a certain filter QuestDB supports that elegantly through negative LIMIT clause values If we were to query ten latest measurements sent from fast moving yet fuel efficient trucks it would look like the following SELECT FROM readingsWHERE velocity gt AND fuel consumption lt LIMIT Notice the LIMIT clause in our query it basically asks the database to return the last rows that correspond to the filter Thanks to the implicit ascending order based on the designated timestamp column we also didn t have to specify the ORDER BY clause In TimescaleDB this query would look more verbose SELECT FROM readingsWHERE velocity gt AND fuel consumption lt ORDER BY time DESCLIMIT Here we had to specify descending ORDER BY and LIMIT clauses When it comes to ClickHouse the query would look just like for TimescaleDB with the exception of another column being used to store timestamps created at instead of time How do databases from our list deal with such query Let s measure and find out This time surprisingly or not TimescaleDB does a better job than ClickHouse That s because just like QuestDB TimescaleDB filters the data starting with the latest time based partitions and stops the filtering once enough rows are found We could also add an index on the velocity and fuel consumption columns but it won t change the result That s because TimescaleDB doesn t use the index and does a full scan instead for this query Thanks to such behavior both QuestDB and TimescaleDB are significantly faster than ClickHouse in the exercise Needless to say that both TimescaleDB and ClickHouse are great pieces of engineering Your mileage may vary and the performance of your particular application depends on a large number of factors So as with any benchmark take our results with a grain of salt and make sure to measure things on your own That should be it for our comparison and now it s time to discuss the design decisions behind our parallel SQL filter execution How does it work First let s quickly recap on QuestDB s storage model to understand why it supports efficient multi core execution The database has a column based append only storage model Data is stored in tables with each column stored in its own file or multiple files in case when the table is partitioned by the designated timestamp When a SQL filter i e the WHERE clause is executed the database needs to scan the files for the corresponding filtered columns As you may have already guessed when the column files are large enough or the query touches multiple partitions filtering the records on a single thread is inefficient Instead the file s could be split into contiguous chunks we call them page frames Then multiple threads could execute the filter on each page frame utilizing both CPU and disk resources in a much more optimal way We already had this optimization in place for some of the analytical types of queries but not for full or partial table scan with a filter So that s basically what we ve added in version As usual there are edge cases and hidden reefs so the implementation is not as simple as it may sound Say what if your query has a filter and a LIMIT clause just like in our recent benchmark Then the database should execute the query in parallel fetch the last records and cancel the remaining page frame filtering tasks so that there is no useless filtering done by other worker threads A similar cancellation should take place in the face of a closed PGWire or HTTP connection or a query execution timeout So as you already saw in the above comparison we made sure to handle all of these edge cases If you re interested in the implementation details go check this lengthy pull request From the end user perspective this optimization is always enabled and applies to non JIT and JIT compiled filters But how does it improve QuestDB s performance Let s find out Speed up measurementsWe ll be using the same benchmark environment as above while using a slightly different query to keep things simple SELECT count FROM readingsWHERE velocity gt AND fuel consumption lt This query counts the total number of measurements sent from fast moving yet fuel efficient trucks First we focus on cold execution time i e situation when the column files data is not in the OS page cache Multi threaded runs use QuestDB while single threaded ones use version of the database That s because JIT compilation is only available in when parallel filter execution is on starting from The database configuration is kept default except for the JIT being disabled or enabled in the corresponding measurements Also notice while this given query supports JIT compilation there is a number of limitations for the types of the queries supported by the JIT compiler The below chart shows the cold execution times What s that Parallel filter execution is only two times faster More than that enabled JIT compiled filters have almost no effect on the end result The thing is that the disk is the bottleneck here Let s try to make some sense out of these results It takes around seconds for QuestDB to execute the query when the data is only on disk The query engine has to scan two groups of column files partitions each having two MB files This gives us around GB of on disk data and around MB s disk read rate That s lower than the configured maximum in our EBS volume but we should keep in mind allowed fluctuations from the maximum throughput and what s even more important individual limits for EBS optimized instances Our instance type is ca xlarge and according to the AWS documentation it s limited with MB s on KiB I O which is very close to our back of the envelope calculation Long story short we re maxing out the disk with multi threaded query execution while single threaded execution time in version stays the same With this in mind further instance type and volume improvements would lead to better performance Things should get even more exciting when it comes to hot execution scenario so there we go In the next and all of the subsequent benchmark runs we measure the average hot execution time for the same query On this particular box default QuestDB configuration leads to threads being used for the shared worker pool So both runs execute the filter on multiple threads speeding up the query when compared with the runs Another observation is almost x difference between JIT compiled and non JIT filters on So even with many cores available to parallel query execution it s a good idea to keep JIT compilation enabled You might have noticed a weird proportion in the above chart Namely the difference between the execution times when JIT compilation is disabled QuestDB takes seconds to finish the query with a single thread while it takes only roughly seconds on That s x improvement and it s impossible to explain it only with parallel processing remember we run the filter on threads So what may be the reason The thing is that parallel filter execution uses the same batch based model as JIT compiled filter functions This means that the filter is executed in a tight CPU friendly loop while the resulting identifiers for the matching rowsare stored in an intermediate array For instance if we restrict parallel filter engine to run on a single thread which is as simple as adding shared worker count database setting the query under test would execute in around seconds Thus in this very scenario batch based filter processing done on a single thread allows us to cut down of the query execution time Obviously multiple threads available to the engine let it run even faster Refer to this blog post for more information on how we do batch based filter processing in our SQL JIT compiler There is one more optimization opportunity around the query we used here Namely in case of queries that select only simple aggregate functions like count or max and no column values we could push down the functions into the filter loop As an example the filter loop will be incrementing the count function s counter in place rather than doing a more generic accumulation of the filtered row identifiers You could say that such queries are rather niche but they could be met in various dashboard applications Thus it s something that we definitely consider adding in future What s next Certainly parallel SQL filter execution introduced in is not the final point in our quest As we ve mentioned already we have multi threading in place for aggregate queries like SAMPLE BY or GROUP BY but only for certain shapes of them Aggregate functions push down is another potential optimization So stay tuned for further improvements As always we encourage our users to try out release on your QuestDB instances and provide feedback in our Slack Community You can also play with our live demo to see how fast it executes your queries And of course open source contributions to our project on GitHub are more than welcome	2022-06-23 13:30:34
Apple	AppleInsider - Frontpage News	iPhone dominates premium smartphone market with 62% of sales in Q1 2022	https://appleinsider.com/articles/22/06/23/iphone-dominates-premium-smartphone-market-with-62-of-sales-in-q1-2022?utm_medium=rss	iPhone dominates premium smartphone market with of sales in Q Apple s iPhone took of Q global market share for devices costing over with the iPhone lineup driving the sales The iPhone lineup sales drove Apple s market share growthThe smartphone market is declining but Apple s share within that market is growing It grew from premium market share in Q to in Q Read more	2022-06-23 13:50:47
Apple	AppleInsider - Frontpage News	Apple TV+ releases trailer for 'Trying' season 3	https://appleinsider.com/articles/22/06/23/apple-tv-releases-trailer-for-trying-season-3?utm_medium=rss	Apple TV releases trailer for x Trying x season Ahead of the comedy s third season premiere on July Apple TV has unveiled a trailer showing what new problems the couple will face with their children Rafe Spall left and Esther Smith in Trying The eight part third season sees Nikki Esther Smith and Jason Rafe Spall waking up as new parents to two children they are still getting to know Before they get much chance to bond as a new family though the trailer shows how hard that is going to be Read more	2022-06-23 13:45:39
Apple	AppleInsider - Frontpage News	Daily deals June 23: $570 for M1 Mac mini, 24% off 2 bay NAS, 21% off Dell 27-inch 4K monitor, more	https://appleinsider.com/articles/22/06/23/daily-deals-june-23-570-for-m1-mac-mini-24-off-2-bay-nas-21-off-dell-27-inch-4k-monitor-more?utm_medium=rss	Daily deals June for M Mac mini off bay NAS off Dell inch K monitor moreThursday s best deals include off a Radeon RX XT graphics card for TP Link Mesh WiFi pack off a HyperX SoloCast USB microphone and much more Best deals June Every day AppleInsider scours online retailers to find offers and discounts on Apple devices hardware accessories smart TVs and other products The best discoveries are put into our daily deals post for you to enjoy Read more	2022-06-23 13:03:54
ニュース	BBC News - Home	Hundreds of British Airways staff vote to strike over pay	https://www.bbc.co.uk/news/business-61906236?at_medium=RSS&at_campaign=KARANGA	holidays	2022-06-23 13:49:27
ニュース	BBC News - Home	Afghanistan quake: Many children feared dead in disaster	https://www.bbc.co.uk/news/world-asia-61900260?at_medium=RSS&at_campaign=KARANGA	disasterover	2022-06-23 13:21:07
ニュース	BBC News - Home	Rwanda asylum plan: I will defend scheme to Prince Charles - PM	https://www.bbc.co.uk/news/uk-politics-61907884?at_medium=RSS&at_campaign=KARANGA	wales	2022-06-23 13:11:17
ニュース	BBC News - Home	Timed Out - magazine issues final print edition after 54 years	https://www.bbc.co.uk/news/uk-england-london-61907597?at_medium=RSS&at_campaign=KARANGA	edition	2022-06-23 13:14:33
ビジネス	ダイヤモンド・オンライン - 新着記事	インスタグラム、子どもの年齢確認する新機能導入へ - WSJ発	https://diamond.jp/articles/-/305414	確認	2022-06-23 22:17:00
北海道	北海道新聞	ソニー生命社員、１億円詐取　無断で貸し付け請求や解約	https://www.hokkaido-np.co.jp/article/697362/	神奈川県	2022-06-23 22:45:00
北海道	北海道新聞	校内いじめで母に相談　旭川中２死亡　入学直後に複数回	https://www.hokkaido-np.co.jp/article/697360/	旭川市内	2022-06-23 22:38:00
北海道	北海道新聞	全国都市緑化北海道フェア　札幌市も２５日から関連イベント	https://www.hokkaido-np.co.jp/article/697219/	関連	2022-06-23 22:35:56
北海道	北海道新聞	道内に訪日ツアー客　香港から第１陣	https://www.hokkaido-np.co.jp/article/697359/	新型コロナウイルス	2022-06-23 22:32:00
北海道	北海道新聞	道産花に笑顔咲く　白い恋人パークで２５日から販売会	https://www.hokkaido-np.co.jp/article/697358/	札幌市西区	2022-06-23 22:33:00
北海道	北海道新聞	米経常赤字、過去最大に　１～３月期、３９兆円	https://www.hokkaido-np.co.jp/article/697353/	過去	2022-06-23 22:24:00
北海道	北海道新聞	船内火災想定、避難誘導を確認　洞爺湖汽船など訓練	https://www.hokkaido-np.co.jp/article/697342/	避難	2022-06-23 22:22:30
北海道	北海道新聞	トヨタ電気自動車リコール　２車種、タイヤ脱落恐れ	https://www.hokkaido-np.co.jp/article/697349/	電気自動車	2022-06-23 22:20:00
北海道	北海道新聞	ＮＹ円、１３５円前半	https://www.hokkaido-np.co.jp/article/697347/	外国為替市場	2022-06-23 22:09:00
北海道	北海道新聞	十勝ワイン飲み比べを　セットの限定販売開始	https://www.hokkaido-np.co.jp/article/697346/	十勝ワイン	2022-06-23 22:06:00
北海道	北海道新聞	鮮やかキルト、２００枚大集合　中札内で２４日からフェスティバル	https://www.hokkaido-np.co.jp/article/697345/	鮮やか	2022-06-23 22:06:00
北海道	北海道新聞	バラの香りに酔いしれ　函館・旧イギリス領事館庭園	https://www.hokkaido-np.co.jp/article/697344/	函館市元町	2022-06-23 22:03:00
北海道	北海道新聞	犯罪被害者条例「住民の安心に」　せたな町が４月施行、道南３例目　見舞金支給や住宅あっせん	https://www.hokkaido-np.co.jp/article/697343/	犯罪被害者	2022-06-23 22:02:00
仮想通貨	BITPRESS（ビットプレス）	[日経] 逆境ブロックチェーン（上）仮想通貨、ドル連動の虚構	https://bitpress.jp/count2/3_9_13267	連動	2022-06-23 22:13:32