投稿時間:2021-07-20 08:28:58 RSSフィード2021-07-20 08:00 分まとめ(42件)

カテゴリー等	サイト名等	記事タイトル・トレンドワード等	リンクURL	頻出ワード・要約等/検索ボリューム	登録日
TECH	Engadget Japanese	観察・写真・録画を全部これ1台で。昼夜両用単眼望遠鏡・暗視スコープ一体型「MILESEEY NV20」	https://japanese.engadget.com/mileseey-nv-20-224154210.html	昼夜両用単眼望遠鏡・暗視スコープ一体型「MILESEEYNV」MILESEEYマイルシーNV主な特徴写真ビデオを撮影・録音可能mタイムリーに記録も可能ナイトビジョン搭載で赤外線を利用して夜間でもはっきり見えるパソコンと接続可能、ディスプレイは接眼レンズGBmicroSDカードが付属光学倍とデジタル倍ズーム搭載で遠くの被写体も写せる「MILESEEYNV」は昼夜兼用の単眼鏡・暗視スコープ一体型の望遠装置です。	2021-07-19 22:41:54
IT	ITmedia 総合記事一覧	[ITmedia News] 小規模企業向けクラウドPCは浸透するか　Webブラウザから使えるマシン「Windows 365」登場で考えた	https://www.itmedia.co.jp/news/articles/2107/20/news045.html	itmedia	2021-07-20 07:20:00
IT	ITmedia 総合記事一覧	[ITmedia ビジネスオンライン] NY株、一時800ドル下げ　コロナ再流行で売り加速	https://www.itmedia.co.jp/business/articles/2107/20/news058.html	itmedia	2021-07-20 07:08:00
IT	ITmedia 総合記事一覧	[ITmedia エグゼクティブ] 村上隆らのドラえもんが街を彩る、六本木アートナイトが開催	https://mag.executive.itmedia.co.jp/executive/articles/2107/20/news012.html	itmedia	2021-07-20 07:06:00
IT	ITmedia 総合記事一覧	[ITmedia Mobile] iOS 14.7の配信開始　「MagSafeバッテリーパック」のサポートなど	https://www.itmedia.co.jp/mobile/articles/2107/20/news057.html	apple	2021-07-20 07:02:00
AWS	AWS Japan Blog	Amazon ECS での Bottlerocket の始め方 〜 コンテナ向けのセキュアな Linux ディストリビューション	https://aws.amazon.com/jp/blogs/news/getting-started-with-bottlerocket-and-amazon-ecs/	なぜBottlerocketなのかお客様はワークロードの実行にコンテナを採用し続けており、AWSはこれらのコンテナ化されたアプリケーションを実行するために設計、最適化されたLinuxディストリビューションの必要性を感じていました。	2021-07-19 22:40:38
python	Pythonタグが付けられた新着投稿 - Qiita	1. Pythonで学ぶ統計学　2-3. 正規分布の基本	https://qiita.com/y_itoh/items/9e75cea0c8b91f67729d		2021-07-20 07:42:08
js	JavaScriptタグが付けられた新着投稿 - Qiita	package.jsonにfilesを書かないあなたは、誰かを少しだけ不幸にしています	https://qiita.com/masato_makino/items/656f4fbb1595cbcdc23d	このフィールドはnpmパッケージがインストールされたとき、どのファイルをnodemodulesにコピーするかを設定します。	2021-07-20 07:40:06
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	Powershell ISEからgit statusなどを実行すると日本語部分が文字化けする	https://teratail.com/questions/350372?rss=all	PowershellISEからgitstatusなどを実行すると日本語部分が文字化けする事象PowershellnbspISEからgitnbspstatusなどを実行するとパスの中で日本語になっている部分が文字化けします。	2021-07-20 07:51:41
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	Twitter4j: Relationship の isSourceBlockingTarget()が正しく動かない	https://teratail.com/questions/350371?rss=all	TwitterjRelationshipのisSourceBlockingTargetが正しく動かないTwitterjを利用してAさんがBさんをブロックしているか確認するコードを作りたいのですが、TwittertwitterTwitterFactorygetSingletonStringsourcesourceのユーザーIDStringtargettargetのユーザーIDRelationshiprelationshiptwittershowFriendshipsourcetargetSystemoutprintlnisSourceBlockingTargetrelationshipisSourceBlockingTargetこれがブロックしている場合でも常にfalseとなり、正常に動いてくれません。	2021-07-20 07:48:54
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	パワーシェルのエラー	https://teratail.com/questions/350370?rss=all	パワーシェルのエラー困っていることパワーシェルでavrdudecavrdudeconfvpatmegaucavrPCOMUflashwJoystickhexiと打ち込んでもavrdude用語avrdudeは、コマンドレット、関数、スクリプトファイル、または操作可能なプログラムの名前として認識されません。	2021-07-20 07:20:25
海外TECH	Ars Technica	Dish switching network to AT&T after calling T-Mobile anticompetitive	https://arstechnica.com/?p=1781474	anticompetitive	2021-07-19 22:12:26
海外TECH	Ars Technica	The surprising connection between a mockingbird’s song and Kendrick Lamar	https://arstechnica.com/?p=1781239	distinct	2021-07-19 22:05:58
海外TECH	Ars Technica	Pandemic of unvaccinated rages with delta’s spread; cases up in all 50 states	https://arstechnica.com/?p=1781469	variant	2021-07-19 22:05:03
海外科学	NYT > Science	Carbon Border Tax Is Proposed by Democrats	https://www.nytimes.com/2021/07/19/climate/democrats-border-carbon-tax.html	ambitious	2021-07-19 22:16:24
金融	金融総合：経済レポート一覧	主要銀行貸出動向アンケート調査（2021年7月）	http://www3.keizaireport.com/report.php/RID/462414/?rss	主要銀行貸出動向アンケート調査	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	FX Daily(7月16日)～ドル円、110円台前半まで上昇	http://www3.keizaireport.com/report.php/RID/462416/?rss	fxdaily	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	中央銀行による気候変動への対応：井上哲也のReview on Central Banking	http://www3.keizaireport.com/report.php/RID/462417/?rss	reviewoncentralbanking	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	気候変動対応オペで日本銀行のリスク回避姿勢：木内登英のGlobal Economy ＆ Policy Insight	http://www3.keizaireport.com/report.php/RID/462418/?rss	lobaleconomypolicyinsight	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	デジタル人民元発行近づく：中国人民銀行が白書を公表～デジタル人民元の個人ウォレットは人口の約1.5％に広がる...：木内登英のGlobal Economy ＆ Policy Insight	http://www3.keizaireport.com/report.php/RID/462419/?rss	lobaleconomypolicyinsight	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	銀行によるゼロゼロ融資の利用動向から読み取れること～政策効果か、モラルハザードか	http://www3.keizaireport.com/report.php/RID/462424/?rss	大和総研	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	為替相場展望2021年7月号～ドル円：米利上げ織り込みでドル高 / ユーロ：金融政策格差で当面はユーロ安基調	http://www3.keizaireport.com/report.php/RID/462425/?rss	日本総合研究所	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	日本銀行が示した気候変動対応の方向性～資金供給制度の骨子素案と包括的な取り組み方針を発表：リサーチ・アイ No.2021-023	http://www3.keizaireport.com/report.php/RID/462427/?rss	取り組み	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	みずほ経済・金融ウィークリー 2021年7月19・26日合併号～先週の内外経済・金融市場動向・評価＆今週の注目点	http://www3.keizaireport.com/report.php/RID/462428/?rss	金融市場	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	人間参加型のAI活用(Human-in-the-loop)：イノベーションジャーナル	http://www3.keizaireport.com/report.php/RID/462429/?rss	humanintheloop	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	日本株は内需待ち～欧米とのギャップ解消が待たれる：Market Flash	http://www3.keizaireport.com/report.php/RID/462431/?rss	marketflash	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	経済・物価情勢の展望（2021年7月、全文）～BOX：国際商品市況の上昇が企業収益に及ぼす影響...	http://www3.keizaireport.com/report.php/RID/462434/?rss	日本銀行	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	アジア主要通貨・株価の動き（7月16日まで）	http://www3.keizaireport.com/report.php/RID/462443/?rss	国際金融情報センター	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	SPACが変えるイノベーション（２）アジアにも広がるSPACの波～20年前のネットバブル紳士も蠢く：JCER 中国・アジアウォッチ	http://www3.keizaireport.com/report.php/RID/462452/?rss	日本経済研究センター	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	大きすぎて潰せない自然～金融主体に求められる速やかな対応	http://www3.keizaireport.com/report.php/RID/462458/?rss	pwcjapan	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	自律分散型社会がもたらす、個々がつながり輝く世界～自律分散型社会がもたらすものと金融のこれからのあり方...	http://www3.keizaireport.com/report.php/RID/462459/?rss	Detail Nothing	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	米国の景気・金利・株価の見方～金利・株価ともに上昇する業績相場が続く見通し：マーケットレター	http://www3.keizaireport.com/report.php/RID/462467/?rss	投資信託	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	グローバルREITウィークリー 2021年7月第3週号	http://www3.keizaireport.com/report.php/RID/462468/?rss	日興アセットマネジメント	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	楽読 Vol.1725～出遅れの巻き返しに期待したい日本株式	http://www3.keizaireport.com/report.php/RID/462469/?rss	巻き返し	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	週間市場レポート（2021年7月12日～7月16日）～日本の株式・債券市場、米国の株式市場、外国為替市場	http://www3.keizaireport.com/report.php/RID/462470/?rss	債券市場	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	【注目検索キーワード】スポーツツーリズム	http://search.keizaireport.com/search.php/-/keyword=スポーツツーリズム/?rss	検索キーワード	2021-07-20 00:00:00
金融	金融総合：経済レポート一覧	【お薦め書籍】小さなメーカーが生き残る経営術 独自市場のつくり方	https://www.amazon.co.jp/exec/obidos/ASIN/486367662X/keizaireport-22/	経営	2021-07-20 00:00:00
ビジネス	ダイヤモンド・オンライン - 新着記事	米ズーム初の大型買収、法人ビジネスに本腰 - WSJ発	https://diamond.jp/articles/-/277404	買収	2021-07-20 07:20:00
サブカルネタ	ラーブロ	麺屋 やまひで 塚本店。。	http://feedproxy.google.com/~r/rablo/~3/YaxiJDVWxpc/single_feed.php	配信	2021-07-19 23:05:20
北海道	北海道新聞	ＮＹ円、１０９円前半	https://www.hokkaido-np.co.jp/article/569014/	外国為替市場	2021-07-20 07:14:00
ビジネス	東洋経済オンライン	自動車整備士版｢Uber｣が旧弊から見いだす勝機 整備ベンチャーからみた業界に横たわる課題 | インターネット | 東洋経済オンライン	https://toyokeizai.net/articles/-/441540?utm_source=rss&utm_medium=http&utm_campaign=link_back	東洋経済オンライン	2021-07-20 07:30:00
GCP	Cloud Blog	Scaling deep learning workloads with PyTorch / XLA and Cloud TPU VM	https://cloud.google.com/blog/topics/developers-practitioners/scaling-deep-learning-workloads-pytorch-xla-and-cloud-tpu-vm/	Scaling deep learning workloads with PyTorch XLA and Cloud TPU VMIntroductionMany deep learning advancements can be attributed to increases in data size and computational power Training deep learning models with larger datasets can be extremely beneficial for model training Not only do they help stabilize model performance during training but research shows that for moderate to large scale models and datasets model performance converges as a power law with training data size meaning we can predict improvements to model accuracy as the dataset grows Figure Learning curve and dataset size for word language models source In practice this means as we look to improve model performance with larger datasets we need access to hardware accelerators such as GPUs or TPUs and we need to architect a system that efficiently stores and delivers this data to the accelerators There are a few reasons why we may choose to stream data from remote storage to our accelerator devices Data size data can be too large to fit on a single machine requiring remote storage and efficient network accessStreamlined workflows transferring data to disk can be time consuming and resource intensive we want to make fewer copies of the data Collaboration disaggregating data from accelerator devices means we can more efficiently share accelerator nodes across workloads and teamsStreaming training data from remote storage to accelerators can alleviate these issues but it introduces a host of new challenges Network overhead Many datasets consist of millions of individual files randomly accessing these files can introduce network bottlenecks We need sequential access patternsThroughput Modern accelerators are fast the challenge is feeding them fast enough to keep them fully utilized We need parallel I O and pipelined access to dataRandomness vs Sequential The optimization algorithms in deep learning jobs benefit from randomness but random file access introduces network bottlenecks Sequential access alleviates network bottlenecks but can reduce the randomness needed for training optimization We need to balance these How do we architect a system that addresses these challenges at scale Figure Scaling to larger datasets more devices In this post we will cover The challenges associated with scaling deep learning jobs to distributed training settingsUsing the new Cloud TPU VM interfaceHow to stream training data from Google Cloud Storage GCS to PyTorch XLA models running on Cloud TPU Pod slicesYou can find accompanying code for this article in this GitHub repository  Model and datasetIn this article we will train a PyTorch XLA ResNet model on a v TPU Pod slice where training data is stored in GCS and streamed to the TPU VMs at training time ResNet is a layer convolutional neural network commonly used for computer vision tasks and machine learning performance benchmarking To demonstrate an end to end example we will use the CIFAR dataset The original dataset consists of x color images divided into classes each class containing images We have upsampled this dataset creating a training and test set of and images respectively CIFAR is used because it is publicly accessible and well known however in the GitHub repository we provide guidance for adapting this solution to your workloads as well as larger datasets such as ImageNet Cloud TPUTPUs or Tensor Processing Units are ML ASICs specifically designed for large scale model training As they excel at any task where large matrix multiplications dominate they can accelerate deep learning jobs and reduce the total cost of training If you re new to TPUs check this article to understand how they work  The v TPU used in this example consists of TPU v cores and GiB of total TPU memory This TPU Pod slice consists of TPU Boards a Board has TPU cores Each TPU Board is connected to a high performance CPU based host machine for things like loading and preprocessing data to feed to the TPUs Figure Cloud TPU VM architecture source We will access the TPU through the new Cloud TPU VMs When we use Cloud TPU VMs a VM is created for each TPU board in the configuration Each VM consists of vCPUs and GB of memory and comes preinstalled with the latest PyTorch XLA image Because there is no user VM we ssh directly into the TPU host to run our model and code This root access eliminates the need for a network VPC or firewall between our code and the TPU VM which can significantly improve the performance of our input pipeline For more details on Cloud TPU VMs see the System Architecture PyTorch XLAPyTorch XLA is a Python library that uses the XLA Accelerated Linear Algebra deep learning compiler to connect PyTorch and Cloud TPUs Check out the GitHub repository for tutorials best practices Docker Images and code for popular models e g ResNet and AlexNet Data parallel distributed trainingDistributed training typically refers to training workloads which use multiple accelerator devices e g GPU or TPU In our example we are executing a data parallel distributed training job with stochastic gradient descent In data parallel training our model fits on a single TPU device and we replicate the model across each device in our distributed configuration When we add more devices our goal is to reduce overall training time by distributing non overlapping partitions of the training batch to each device for parallel processing Because our model is replicated across devices the models on each device need to communicate to synchronize their weights after each training step In distributed data parallel jobs this device communication is typically done either asynchronously or synchronously Cloud TPUs execute synchronous device communication over the dedicated high speed network connecting the chips In our model code we use PyTorch XLA s optimizer step optimizer to calculate the gradients and initiate this synchronous update Figure Synchronous all reduce on Cloud TPU interconnectAfter the local gradients are computed the xm optimizer step function synchronizes the local gradients between cores by applying an AllReduce SUM operation and then calls the PyTorch optimizer step optimizer which updates the local weights with the synchronized gradients On the TPU the XLA compiler generates AllReduce operations over the dedicated network connecting the chips Ultimately the globally averaged gradients are written to each model replica s parameter weights ensuring the replicas start from the same state in every training iteration We can see the call to this function in the training loop  Input pipeline performanceAs previously mentioned the challenge with TPUs is feeding them the training data fast enough to keep them busy This problem exists when we store training data on a local disk and becomes even more clear when we stream data from remote storage Let s first review a typical machine learning training loop Figure Common machine learning training loop and hardware configurationIn this illustration we see the following steps Training data is either stored on local disk or remote storage The CPU requests and reads the data augments it with various transformations batches it and feeds it to the model Once the model has the transformed batched training data the accelerator takes over The accelerator a computes the forward pass b loss and c backwards pass After computing the gradients the parameter weights are updated the learning  And we repeat the cycle over again While this pattern can be adapted in several ways e g some transformations could be computed on the accelerator the prevailing theme is that an ideal architecture seeks to maximize utilization of the most expensive component the accelerator And because of this we see most performance bottlenecks occurring in the input pipeline driven by the CPU To help with this we are going to use the WebDataset library WebDataset is a PyTorch dataset implementation designed to improve streaming data access for deep learning workloads especially in remote storage settings Let s see how it helps WebDataset formatWebDatasets are just POSIX tar archive files and they can be created with the well known tar command They don t require any data conversion the data format is the same in the tar file as it is on disk For example our training images are still in PPM PNG or JPEG format when they are stored and transferred to the input pipeline The tar format provides performance improvements for both small and large datasets as well as data stored on either local disk or remote storage such as GCS Let s outline three key pipeline performance enhancements we can achieve with WebDataset Sequential I OGCS is capable of sustaining high throughput but there is some network overhead when initiating a connection If we are accessing millions of individual image files this is not ideal Alternatively we can achieve sequential I O by requesting a tar file containing our individual image files Once we request the tar file we get sequential reads of the individual files within that tar file which allows for faster object I O over the network This reduces the number of network connections to establish with GCS and thus reduces potential network bottlenecks  Figure Comparing random and pipelined access to data files Pipelined data access With file based I O we randomly access image files which is good for training optimization but for each image file there is a client request and storage server response Our sequential storage achieves higher throughput because with a single client request for a tar file the data samples in that file flow sequentially to the client This pattern gives us pipelined access to our individual image files resulting in higher throughput   ShardingStoring TBs of data in a single sequential file could be difficult to work with and it prevents us from achieving parallel I O Sharding the dataset can help us in several ways Aggregate network I O by opening shards in parallel Accelerate data preprocessing by processing shards in parallelRandomly access shards but read sequentially within each shard Distribute shards efficiently across worker nodes and devicesGuarantee equal number of training samples on each deviceBecause we can control the number of shards and the number of samples in those shards we can distribute equal sized shards and guarantee each device receives the same number of samples in each training epoch Sharding the tar files helps us balance the tradeoff between random files access and sequential reads Random access to the shards and in memory shuffling satisfy enough randomness for the training optimization The sequential reads from each shard reduce network overhead  Distributing shards across devices and workersAs we are essentially creating a PyTorch IterableDataset we can use the PyTorch DataLoader to load data on the devices for each training epoch Traditional PyTorch Datasets distribute data at the sample level but we are going to distribute at the shard level We will create two functions to handle this distribution logic and pass them to the splitter and nodesplitter arguments when we create our dataset object All these functions need to do is take a list of shards and return a subset of those shards To see how the following snippets fit into the model script check out test train mp wds cifar py in the accompanying GitHub repository We will split shards across workers with We will split shards across devices with With these two functions we will create a data loader for both train and validation data Here is the train loader Here is an explanation of some of the variables used in these snippets xm xrt world size is the total number of devices or TPU coresFLAGS num workers is the number of subprocesses spawned per TPU core for loading and preprocessing dataThe epoch size specifies the number of training samples each device should expect for each epochshardshuffle True means we will shuffle the shards while shuffle shuffles samples inline batched batch size partial True explicitly batches data in the Dataset by batch sizeand partial True handles partial batches typically found in the last shardOur loader is a standard PyTorch DataLoader Because our WebDataset Dataset accounts for batching shuffling and partial batches we do not use these arguments in PyTorch s DataLoaderPerformance comparisonThe table in Figure compares the performance between different training configurations for a PyTorch XLA ResNet model training on the ImageNet dataset Configuration A provides baseline metrics and represents a model reading from local storage randomly accessing individual image files Configuration B uses a similar setup as A except the training data is sharded into POSIX tar files and the WebDataset library is used to sample and distribute shards to the model replicas on Cloud TPU devices Configuration C uses the same sampling and distribution logic as B but sources training data from remote storage in GCS The metrics represent an average of each configuration over five epoch training jobs Figure Training performance comparisonComparing configurations A and B these results show that simply using a sharded sequentially readable data format improves pipeline and model throughput average examples per second by They also show that we can take advantage of remote storage without negatively impacting model training performance Comparing configurations A and C we were able to maintain pipeline and model throughput training time and model accuracy To highlight the impacts of sequential and parallel I O we held many configuration settings constant There are still several areas to investigate and improve In a later post we will show how to use the Cloud TPU profiler tool to further optimize PyTorch XLA training jobs End to end exampleLet s walk through a full example To follow this example you can use this notebook to create a sharded CIFAR dataset Before you beginIn the Cloud Shell run the following commands to configure gcloud to use your GCP project install components needed for the TPU VM preview and enable the TPU API For additional TPU VM setup details see these instructions Connecting to a Cloud TPU VMThe default network comes preconfigured to allow ssh access to all VMs If you don t use the default network or the default network settings were edited you may need to explicitly enable SSH access by adding a firewall rule Currently in the TPU VM preview we recommend disabling OS login to allow native scp required for PyTorch XLA Pods Creating a TPU VM sliceWe will create our TPU Pod slice in europe west a because this region supports both TPU VMs and v TPU Pod slices TPU NAME name of the TPU nodeZONE location of the TPU nodeACCELERATOR TYPE find the list of supported accelerator types hereRUNTIME VERSION for PyTorch XLA use v alpha for single TPUs and TPU pods This is a stable version for our public preview release PyTorch XLA requires all TPU VMs to be able to access the model code and data Using gcloud we will include a metadata startup script which installs the necessary packages and code on each TPU VM  This command will create a v TPU Pod slice and VMs one dedicated to each TPU board  To ssh into a TPU VM we will use the gcloud ssh command below By default this command will connect to the first TPU VM worker denoted with w To ssh into any other VM associated with the TPU Pod append worker WORKER NUMBER in the command where the WORKER NUMBER is based See here for more details on managing TPU VMs   Once in the VM run the following command to generate the ssh keys to ssh between VM workers on a pod PyTorch trainingCheck to make sure the metadata startup script has cloned all the repositories After running the following command we should see the torchxla tpu directory To train the model let s first set up some environment variables BUCKET name of GCS bucket storing our sharded dataset We will also store training logs and model checkpoints here see guidelines on GCS object names and folders split SHARDS train val shards using brace notation to enumerate the shardsWDS split DIR uses pipe to run a gsutil command for downloading the train val shardsLOGDIR location in GCS bucket for storing training logsOptionally we can pass environment variables for storing model checkpoints and loading from a previous checkpoint file When we choose to save model checkpoints a checkpoint file will be saved at the end of each epoch if the validation accuracy improves Each time a checkpoint is created the PyTorch XLA xm save utility API will save the file locally overwriting any previous file if it exists Then using the Cloud Storage Python SDK we will upload the file to the specified LOGDIR overwriting any previous file if it exists Our example saves a dictionary of relevant information like this Here is the function that uses the Cloud Storage SDK to upload each model checkpoint to GCS If we want to resume training from a previous checkpoint we use the LOAD CHKPT FILE variable to specify the GCS object to download and the LOAD CHKPT DIR variable to specify the local directory to place this file Once the model is initialized we deserialize the dictionary with torch load load the model s parameter dictionary with load state dict and move the model to the devices with to device  Here is the function that uses the Cloud Storage SDK to download the checkpoint and save it to a local directory We can use other information from our dictionary to configure the training job such as updating the best validation accuracy and epoch If we don t want to save or load these files we can omit them from the command line arguments Details on saving and loading PyTorch XLA checkpoint files can be found here  Now we are ready to train restart tpuvm pod server restarts the XRT SERVER XLA Runtime and is useful when running consecutive TPU jobs especially if that server was left in a bad state Since the XRT SERVER is persistent for the pod setup environment variables won t be picked up until the server is restarted test train mp wds cifar pyclosely follows the PyTorch XLA distributed multiprocessing script but is adapted to include support for WebDataset and CIFARTPUs have hardware support for Brain Floating Point Format which can be used by setting XLA USEBF During training output for each step looks like this refers to the IP address for this VM worker refers to VM worker Recall there are VM workers in our exampleTraining Device xla refers to the TPU core In our example there are TPU cores so you should see up to xla since they are based Rate refers to the exponential moving average of examples per second for this TPU coreGlobalRate refers to the average number of examples per second for this core so far during this epochAt the end of each epoch s train loop you will see output like this Replica Train Samplestells us how many training samples this replica processedReduced GlobalRateis the average GlobalRate across all replicas for this epochOnce training is complete you will see the following output The logs for each VM worker are produced asynchronously so it can be difficult to read them sequentially To view the logs sequentially for any TPU VM worker we can execute the following command where the IP ADDRESS is the address to the left of our We can convert these to a txt file and store them in a GCS bucket like this Cleaning upWe can clean up our TPU VM resources in one simple command First disconnect from the TPU VM if you have not already done so In the Cloud Shell use the following command to delete the TPU VM resources If you wish to delete the GCS bucket and its contents run the following command in the Cloud Shell terminal What s next In this article we explored the challenges of using remote storage in distributed deep learning training jobs We discussed the advantages of using sharded sequentially readable data formats to solve the challenges with remote storage access and how the WebDataset library makes this easier with PyTorch We then walked through an example demonstrating how to stream training data from GCS to TPU VMs and train a PyTorch XLA model on Cloud TPU Pod slices  ReferencesCloud TPUsCloud TPU VM architecturePyTorch XLA GitHub repositoryWebDataset GitHub repositoryGitHub repository for this codeIn the next installment of this series we will revisit this example and work with Cloud TPU Tools to further optimize our training job We will demonstrate how variables such as shard size shard count batch size and number of workers impact the input pipeline resource utilization examples per second accuracy loss and overall model convergence   Have a question or want to chat Find the authors here Jordan and Shane  Special thanks to Karl Weinmeister Rajesh Thallam and Vaibhav Singh for their contributions to this post as well as Daniel Sohn Zach Cain and the rest of the PyTorch XLA team for their efforts to enhance the PyTorch experience on Cloud TPUs Related ArticleHow to use PyTorch Lightning s built in TPU supportHow to start training ML models with Pytorch Lightning on TPUs Read Article	2021-07-19 22:30:00