投稿時間:2023-07-22 05:32:46 RSSフィード2023-07-22 05:00 分まとめ(34件)

カテゴリー等	サイト名等	記事タイトル・トレンドワード等	リンクURL	頻出ワード・要約等/検索ボリューム	登録日
AWS	AWS	AWS Innovation with Genome Institute of Singapore | Innovation Ambassadors	https://www.youtube.com/watch?v=LTi3xSnnSlM	AWS Innovation with Genome Institute of Singapore Innovation AmbassadorsOn this episode we re showcasing the journey of the Genome Institute of Singapore GIS a national initiative with a global vision Their mission is to harness the power of genomic sciences for remarkable advancements in human health and public prosperity Our host Sara Armstrong delves into GIS s pivotal role as the trusted producer custodian and curator of Singapore s genomic data You ll also hear about their collaboration with the AWS prototyping team which has enabled seamless integration and access to vast amounts of genomic data within a secure and scalable framework AWS AmazonWebServices CloudComputing	2023-07-21 19:42:26
AWS	AWS	Datadog used Graviton to deliver more value while keeping costs constant | Amazon Web Services	https://www.youtube.com/watch?v=hEsf8ACaaUM	Datadog used Graviton to deliver more value while keeping costs constant Amazon Web ServicesDatadog a cloud monitoring and analytics platform company needed to offer more product features and serve more customers They were looking for new innovation that they can adopt to deliver more value for our customers Datadog adopted AWS Graviton based Amazon EC instances to pass savings to their customers by delivering more features per compute cycles and serving more customers with an identical number of cores Learn more at Subscribe More AWS videos More AWS events videos Do you have technical AWS questions Ask the community of experts on AWS re Post ABOUT AWSAmazon Web Services AWS is the world s most comprehensive and broadly adopted cloud platform offering over fully featured services from data centers globally Millions of customers ーincluding the fastest growing startups largest enterprises and leading government agencies ーare using AWS to lower costs become more agile and innovate faster AWS AmazonWebServices CloudComputing	2023-07-21 19:04:56
AWS	AWS	How to Hire and Develop Security Assurance Talent | Amazon Web Services	https://www.youtube.com/watch?v=nIwvIQAVZoI	How to Hire and Develop Security Assurance Talent Amazon Web ServicesA big part of any security leader s role is to hire and develop the next generation of great security leaders In this interview with Jessie Skibbe a privacy and security assurance leader at AWS Jessie shares her criteria for hiring security assurance talent Watch now to get her perspective on how great leaders are made not born Learn more at Subscribe More AWS videos More AWS events videos Do you have technical AWS questions Ask the community of experts on AWS re Post ABOUT AWSAmazon Web Services AWS is the world s most comprehensive and broadly adopted cloud platform offering over fully featured services from data centers globally Millions of customers ーincluding the fastest growing startups largest enterprises and leading government agencies ーare using AWS to lower costs become more agile and innovate faster EnterpriseSecurity Compliance HiringAdvice AWS AmazonWebServices CloudComputing	2023-07-21 19:04:52
AWS	AWS	How to Pass Your Compliance Audit With AWS | Amazon Web Services	https://www.youtube.com/watch?v=mRU6HO1t0vg	How to Pass Your Compliance Audit With AWS Amazon Web ServicesIn this conversation with Jessie Skibbe Senior Practice Manager of AWS Security Assurance we discuss the odds and ends of security compliance and what it takes to pass an audit Discover how AWS came to be a Qualified Security Assessor and what that means for customers seeking compliance advice and guidance Learn more at Subscribe More AWS videos More AWS events videos Do you have technical AWS questions Ask the community of experts on AWS re Post ABOUT AWSAmazon Web Services AWS is the world s most comprehensive and broadly adopted cloud platform offering over fully featured services from data centers globally Millions of customers ーincluding the fastest growing startups largest enterprises and leading government agencies ーare using AWS to lower costs become more agile and innovate faster EnterpriseSecurity Compliance AWS AmazonWebServices CloudComputing	2023-07-21 19:04:48
Git	Gitタグが付けられた新着投稿 - Qiita	git-flowについて	https://qiita.com/ooyy0121/items/81e9a908b2c399eaa6cb	gitflow	2023-07-22 04:18:13
海外TECH	MakeUseOf	How to Fix Adobe Error 16 in Windows 10 & 11	https://www.makeuseof.com/adobe-error-16-windows-10-11/	adobe	2023-07-21 19:16:21
海外TECH	DEV Community	Data Modeling in DynamoDB: What You Need to Know for Peak Performance	https://dev.to/brandondamue/data-modeling-in-dynamodb-what-you-need-to-know-for-peak-performance-kgo	Data Modeling in DynamoDB What You Need to Know for Peak PerformanceData modelling and querying in NoSQL databases represent a fascinating landscape that challenges traditional relational database concepts and opens up a world of possibilities for modern data driven applications Unlike traditional SQL databases NoSQL databases offer a flexible and schema less approach to data modelling allowing developers to adapt their data structures to ever changing application requirements Data modelling and querying in DynamoDB is an art of precision and ingenuity What I mean by this is it requires a creative and thoughtful approach to design data structures and create queries in a way that enhances the database s performance and efficiency What I intend to do within this article is to help you learn how to craft efficient DynamoDB queries that can make all the difference in your applications I will start with an overview all the way down to monitoring and troubleshooting common issues So buckle up and let s explore this together Data Modelling OverviewData modelling refers to the art of designing the structure of your database to efficiently store and retrieve data based on your application s access patterns In DynamoDB it revolves around designing the primary keys sort keys and composite keys that define the structure of your database The primary key is crucial as it uniquely identifies each item in the table and it can be either a partition key or a combination of partition key and sort key The partition key distributes data across multiple partitions for scalability while the sort key allows for more complex querying patterns enabling range queries and item sorting By carefully selecting the appropriate keys based on the application s access patterns developers can optimize query performance and minimize data retrieval costs in DynamoDB Understanding these fundamentals empowers developers to create efficient data models that cater to diverse use cases and unleash the true potential of DynamoDB s flexible and scalable nature Indexes and Query OptimizationGlobal Secondary Indexes GSI and Local Secondary Indexes LSI play a critical role in optimizing query performance by enabling alternative ways to access and retrieve data from DynamoDB GSIs offer additional indexes for attributes other than the primary key allowing more diverse query paths while LSIs provide secondary sort keys within the same partition key to support additional querying options These index types expand the range of attributes that can be used for querying reducing the need for costly scans and filtering operations and enhancing overall query performance and efficiency Some best practices and tips for using these indexes include Distribute queries evenly across partitions to avoid creating hot partitions Uneven data distribution can lead to throttling and reduced performance particularly when using GSIs Regularly monitor the performance of your GSIs and LSIs to identify potential bottlenecks or underutilized indexes Adjust provisioned throughput and data modelling as needed to achieve optimal performance Select attributes for your indexes that align with common query patterns GSIs should focus on attributes frequently used in different access patterns while LSIs should support secondary sort keys that enhance specific queries within a partition key Keep in mind that GSIs consume their own read and write capacity units Be cautious with the provisioned throughput to avoid overprovisioning or underprovisioning which can impact overall database performance and costs To optimize performance consider using GSIs in combination with partition keys to efficiently access data across multiple partitions This approach can significantly enhance the database s query capabilities DynamoDB Streams can be invaluable when using GSIs as they provide change notifications when items are added modified or deleted Streams allow you to react to changes in real time and maintain data consistency across different indexes Query PatternsIn DynamoDB various query patterns cater to different data retrieval needs Point queries efficiently fetch a specific item using its unique primary key ensuring fast and predictable access For composite keys range queries come into play enabling retrieval of items within a specific sort key range ideal for ordered data or time based queries However scan operations examine every item in the table returning matching items based on criteria While scans provide flexibility they should be used sparingly due to their resource intensive nature making them less suitable for large datasets or frequent use Understanding the nuances of each query pattern helps developers optimize data access and performance in DynamoDB selecting the appropriate pattern based on the specific access patterns and query requirements for their application Partitioning and Provisioned ThroughputData partitioning is a fundamental aspect of DynamoDB s impressive scalability and performance A table s data is divided into partitions and each partition is independently stored and managed across multiple servers This distribution allows DynamoDB to efficiently handle large volumes of data and traffic making it well suited for applications with varying workloads The partition key plays a central role in data partitioning as it determines how items are distributed across partitions DynamoDB uses an internal hashing algorithm to map partition key values to specific partitions Consequently items with the same partition key value reside in the same partition while those with different values may be stored in different partitions By distributing data across multiple partitions DynamoDB achieves a high degree of parallelism for read and write operations enabling it to handle a massive number of requests simultaneously This design leads to low latency responses and high throughput even under heavy workloads However careful consideration must be given to the choice of partition key to ensure even data distribution and avoid hot partitions Hot partitions can occur when a specific partition receives an excessive number of requests leading to throttling and reduced performance To prevent this it s essential to select a partition key with a wide range of values and a relatively uniform data distribution By leveraging proper data partitioning strategies developers can fully harness DynamoDB s scalability and achieve optimal performance for their applications To optimize provisioned throughput in DynamoDB based on the workload it s essential to understand your application s access patterns and carefully select a well designed partition key to evenly distribute data across partitions Avoid hot partitions by choosing a partition key with a wide range of values Utilize composite keys when needed ensuring they complement the workload and sorting requirements Consider leveraging DynamoDB s On Demand Capacity Mode for workloads with unpredictable or varying traffic and implement caching mechanisms to reduce read operations hitting DynamoDB Monitor provisioned throughput using CloudWatch metrics and adjust as necessary to meet changing demand Adaptive capacity can help maintain performance during sudden spikes in traffic By following these insights you can efficiently utilize resources control costs and achieve optimal performance for your DynamoDB applications Consistency ModelsI have spoken about the consistency models that exist in DynamoDB in a previous article In DynamoDB there are two consistency models strong consistency and eventual consistency Strong consistency ensures that after a write operation any subsequent read operation will immediately reflect the latest changes This means all read operations see the most up to date data providing a linearizable and globally consistent view of the database While strong consistency guarantees data accuracy it may slightly impact performance as it requires coordination between data replicas On the other hand eventual consistency offers lower latency and higher throughput by relaxing immediate consistency After a write operation there may be a short delay before changes are propagated to all data replicas Consequently read operations performed immediately after a write may not reflect the latest data but eventually all replicas converge to the same state Eventual consistency is ideal for scenarios where real time consistency is not strictly required and the application can tolerate a brief period of inconsistency DynamoDB allows developers to choose the consistency model on a per operation basis providing the flexibility to tailor data access patterns based on specific application requirements Data Modeling for Time Series DataTo efficiently manage time series data in DynamoDB specific data modelling strategies are essential One effective approach is time window partitioning where data is partitioned based on time intervals such as days or hours using a timestamp as the partition key This ensures even distribution of data across partitions reducing the risk of hot partitions and maintaining query performance Additionally utilizing composite keys with the timestamp as the sort key allows for efficient range queries enabling retrieval of time series data within specific time periods Another valuable technique is leveraging Time to Live TTL to automatically expire time series data after a predefined period TTL eliminates the need for manual data cleanup and optimizes storage utilization by automatically removing old data Implementing aggregation and rollups is also beneficial to reduce the volume of data retrieved during queries Pre aggregating time series data at specific intervals such as hourly or daily reduces the number of individual data points and enhances query performance Additionally employing compression techniques like delta encoding or lossless compression can further optimize storage efficiency without sacrificing data accuracy By combining these strategies and carefully considering the unique characteristics of time series data developers can effectively manage and query time series data in DynamoDB ensuring high performance cost effectiveness and scalability for time based applications Troubleshooting and MonitoringTo effectively monitor DynamoDB performance and troubleshoot data modelling and query related issues follow these key practices Utilize Amazon CloudWatch to monitor key DynamoDB metrics such as read and write capacity utilization throttled requests and latency and set up alarms for immediate notifications Enable DynamoDB Streams to capture changes made to the table and trigger downstream processes or analyze changes for troubleshooting Keep track of common and resource intensive query patterns optimize data models and indexes to align with access patterns and monitor data distribution across partitions to avoid hot partitions Watch for throttled read write requests and errors in CloudWatch and adjust provisioned capacity or revise the data model if needed Track the usage of Global Secondary Indexes GSI and Local Secondary Indexes LSI and remove or optimize underutilized indexes to reduce write overhead and storage costs Utilize query profilers and performance analysis tools to identify slow or resource intensive queries and continuously review and optimize data access patterns to support the most common and critical queries Monitor provisioned throughput utilization to avoid capacity issues or unnecessary over provisioning By following these practices you can maintain high performance scalability and efficiency in DynamoDB while promptly addressing any issues that may arise Final ThoughtsIt is important to keep in mind that data modelling is not a one size fits all approach it requires thoughtful analysis and iterative refinement to strike the right balance between read and write performance cost optimization and data access patterns As you venture into your DynamoDB journey embrace the spirit of experimentation and continue fine tuning your data models based on evolving application needs Whether you are handling time series data designing efficient querying strategies or optimizing provisioned throughput DynamoDB provides the canvas for innovation and empowers you to build applications that scale with ease Embrace the challenge explore its myriad features and unlock the full potential of DynamoDB in your next data driven venture Happy modelling and querying in DynamoDB ReferenceExploring the NoSQL Powerhouse Amazon DynamoDBNavigating the AWS Database Landscape Which One is Right for You	2023-07-21 19:20:30
海外TECH	DEV Community	Telling a Story in an Interview	https://dev.to/jcsmileyjr/telling-a-story-in-an-interview-2aio	Telling a Story in an InterviewTelling your story in an interview is a great way to control the interview in a way that highlights your strengths Your resume has to tell enough of your story to get you called but leave enough out that you can tell more stories in the interview Every single question in the interview is an opportunity to expand a vision or add to a positive narrative If you can tie the answers together with some type of theme or related events then the story feels more complete and they feel more like they know you well You are the hero in your story and this is your chance to show that hero s actions and results from beginning to end It s an opportunity to downplay negatives and upgrade positives People love remember and share great stories with others Finally stories sound better then random facts Telling your story also gives you a chance to anchor the conversation at the beginning It allows you to shine a spotlight on what matters to you and where you excel framing the conversation In interviews all you re doing is talking about your successes and what you learned along the way Even failures are successes as long as you learned something There are several key aspects of story telling in an interview Planning aka preparation Every story has a beginning characters and theme You have to pick which work experiences that flows together to tell a memorable story Confidence Your speech and body language has to convey that you believe in the persona you are sharing Passion Everything you say and do have to scream “I want this job and have overcome similar challenges before Reading the room Every interview from the interviewer to the company culture is different Communication The interviewer has to be able to hear you understand you and enjoy the moment Showcasing the Authentic you The story you are telling needs to be unique to just you Tips to prepare for interviews Make a habit of weekly mock interviews with a friend or mentor Practice telling your story over and over until it feel natural A cheat sheet of highlights and experiences you can quickly scan and then use My personal tactic is to “rehearse the hour before the interview If you have tips on crafting a story during the interview please leave a comment	2023-07-21 19:18:53
海外TECH	DEV Community	Good bye and thanks to "typescript-is" (ancestor of "typia", 20,000x faster validator)	https://dev.to/samchon/good-bye-typescript-is-ancestor-of-typia-20000x-faster-validator-49fi	Good bye and thanks to quot typescript is quot ancestor of quot typia quot x faster validator SummaryThis story is about a precursor library typescript is There had been an ancesor library of typia named typescript isOnly one line requiredAoT Ahead of Time compilation through TypeScript Compiler APIMuch convenient than any other validator libraries like zod and class validatorI had another library typescript jsonBoost up JSON serialization speedJSON schema generationPerforms AOT compilation like typescript isI had developed nestia by combining both of themNestJS helper libraryValidators by typescript isJSON booster and Swagger generation by typescript jsontypescript is has stopped maintanence since two years agoI ve changed typescript json to cover typescript is featuresAdded validator features like typescript isEnhanced by M LOC test codesRenamed to typiaToday I requested typescript is author to uphold typiaAuthor of typescript is agreedFrom now on typescript is is unified with typia formallyThanks for pioneering challenge of typescript is and say good byeRelated Repositories typescript is typia nestia Good bye issue typescript is Featuresexport function is lt T gt input unknown input is T export function assertType lt T gt input unknown T export function validate lt T gt input unknown ValidationResult lt T gt TypeScript Source Fileimport is from typescript is is lt number gt Compiled JavaScript Fileimport is from typescript is input gt if typeof input number return false return true This is the AoT compilation typescript is analyzes the TypeScript source codes and transforms to optimal JavaScript codes for the target type T number Long time ago there had been a great validator library named typescript is It had performed AoT Ahead of Time compilation through TypeScript Compiler API As it requires only one line with pure TypeScript type like above it was much convenient than any other validator libraries like zod or class validator In my case I d developed backend server through NestJS framework By the way the NestJS developer recommends to using class validator one of the most horrible library than what I ve ever experienced It enforces user to define quadruple duplicated structure and extremely slow validation speed is extremely slow MB s It even has enormous bugs that saying there s no problem for wrong data and maintainer seems to be not interested in fixing those bugs CLASS VALIDATOR REQUIES DUPLCIATED DEFINITIONSexport class BbsArticle ApiProperty type gt AttachmentFile nullable true isArray true description List of attached files Type gt AttachmentFile IsArray IsOptional IsObject each true ValidateNested each true files AttachmentFile null TYPESCRIPT IS OKAY WITH PURE TYPESCRIPT TYPEexport interface IBbsArticle List of attached files files IAttachmentFile null class validator tests only casestypescript is tests about casestypia tests K cases with M LOC codesInstead I d loved to using typescript is in NestJS projects It was much convenient and safer than class validator Only one line was required and need not to be suffered from enourmous bugs typescript is also could not validate union or complicate types either but not as serious as class validator Using typescript is for the NestJS backend projects I d admired the genius idea of typescript is author and always thankful that it frees me from the nightmare of class validator If you feel like you ve heard this story somewhere you re right typescript is is an ancestor library of typia and is a project that implemented a validator based on AOT compilation before typia However maintenance had been stopped and broken since years ago typescript json JSON SCHEMA GENERATORexport function application lt Types extends unknown Purpose extends swagger ajv gt IJsonSchema JSON SERIALIZATION BOOSTERexport function stringify lt T gt input T string Around the same time I had made another library named typescript json It performs AoT Ahead of Time compliation like typescript is but it was not for runtime validation but for JSON schema generation About JSON serialization boosting typescript json had utilized fast json stringify by automatically generated JSON schema For reference purpose of typescript json was to accomplish below nestia generating Swagger Documents with pure TypeScript type nestiaUsing typescript is in NestJS made backend developments were much easier and convenient However in that case it was not possible to generate Swagger Documents It s because swagger generator of NestJS could not analyze TypeScript source code in the compilation level Instead it required quadruple duplicated definitions through decorator function calls At that time I d thought it would better to make an alternative swagger generator for NestJS project Although it needs complicated logics like analyzing NestJS utilization TypeScript source codes in the compilation level I thought that it is much worthy work instead of rolling back to the terrible class validator I wanted to progress rather than regress Therefore I d made a new library named nestia It had performed validation through typescript is and had generated Swagger Documents through typescript json Also after succeeded to anayzing NestJS source codes I made more features like SDK Software Development Kit library generator like below My team members especially frontend developers were very happy with it and looking at them I d thought that I did a right decision As a result of avoiding class validator which I really don t want to use and pursuing beautiful typescript is efficiency has more than doubled even including frontend developers Left is NestJS server code and right is client frontend code utilizing SDK In nowadays nestia can generate Mock Simulator and even possible to build those SDK library and Mockup Simulator through a swagger json file It means that you can build SDK library and Mockup Simulator from every languages and frameworks SDK a collection of fetch functions with type definitionsMockup simulator embedded backend simulator in SDK Death of typescript istypescript is author has stopped maintenance and it was a critical problem for nestia The moment when cheered for the right choice and had fun with my teammates typescript is has been died suddenly In actually typescript is was already out of maintenance even before starting nestia development Unfortunately at that moment TypeScript had been break changed a lot in compiler API for several times and typescript is had been broken since years ago It was very embarrassing that typescript is suddenly died not long after making nestia and tasting its fruits At that time I thought for a while that should I turn back to the terrible and hopeless class validator or not Developing nestia I had consumed about an year and the one year is not a short time Therefore I considered it for a while However my final decision was still same with before Let s make one thing more and one year more Rather than using that horrible class validator again it makes sense to spend another year again and it would be a great opportunity for studying to me typia RUNTIME VALIDATORSexport function is lt T gt input unknown input is T returns booleanexport function assert lt T gt input unknown T throws TypeGuardErrorexport function validate lt T gt input unknown IValidation lt T gt detailedexport const customValidators CustomValidatorMap for customization JSON FUNCTIONSexport namespace json export function application lt T gt IJsonApplication JSON schema export function assertParse lt T gt input string T type safe parser export function assertStringify lt T gt input T string safe and faster PROTOCOL BUFFER NOT YET BUT SOON export namespace protobuf export function message lt T gt string Protocol Buffer message export function assertDecode lt T gt buffer Buffer T safe decoder export function assertEncode lt T gt input T UintArray safe encoder RANDOM GENERATORexport function random lt T gt g Partial lt IRandomGenerator gt T Evolved typescript json to cover typescript is features and renamed to typia As typescript is had stopped maintenance and broken by update of TypeScript compiler and it was a critical problem for my project nestia I d decided to enhance typescript json to cover typescript is features Also I d renamed typescript json to typia because that library is no more designed only for JSON related functions Also as typescript json had designed to generate JSON schema from TypeScript types it must have well designed metadata definition for TypeScript types Besides typescript is had not designed metadata structure and just wrote a lot of hard coded if statements for TypeScript types Such different made typia to have much more LOC Line of Codes than typescript is but typia could be much stable than before typescript is Furthermore I ve enhanced typia through LOC test codes Considering characteristics of typia using pure TypeScript type typia must support every TypeScript type cases Also expression power of TypeScript about type is the much more powerful and wider than any other languages Therefore to support every TypeScript types safely I had to write enormous test codes about one million LOC It was a hard work for me during one year but such endurance evolved typia to be only one validator libary which supports every TypeScript types typia tests K cases with M LOC codestypescript is tests about casesclass validator tests only casesComponentstypiats istypeboxajvio tszodc v Easy to useObject simple Object hierarchical Object recursive Object union implicit Object union explicit Object additional tags Object template literal types Object dynamic properties Array rest tuple Array hierarchical Array recursive Array recursive union Array R U implicit Array repeated Array repeated union Ultimate Union TypeData structure of typia for TypeScript types For reference as typescript is doesn t have such well structured metadata definition and it is the reason why typescript is cannot validate complicate type However to make an excuse for typescript is it was the most stable of the validator libraries at that time and even still much more stable than the terrible class validator ts is means typescript isc v means class validatorAlso as typia performs AoT Ahead of Time compilation skill and nestia is utilizing such typia features I could enhance NestJS performance enormously Current nestia can enhance NestJS server performance like below x faster validation speed than class validatorx faster JSON serialization speed than class transformerComposite NestJS server performance be x upBenchmark Resultstypia benchmark R Core TM i G GHznestia server benchmark R Core TM i G GHz Good bye typescript isA year has passed since the successful development of typia And in the meantime typescript is is still out of maintenance Therefore I ve written an issue to typescript is repo requesting derepcate typescript is and uphold typia instead While writing the issue I also left a message of appreciation for typescript is s pioneering ideas and past dedications that helped me to escape from terrible class validator Thanks for his great idea and I could learn lots of things by motivated from his project At past week typescript is author accepted my suggestion and typescript is has started upholding typia From now on typescript is is unified with typia formally Thanks for pioneering challenge of typescript is and say good bye Next Story Theoretical ExplanationRecently I ve written some articles about AoT compilation By the way I ve only told you that AoT compilation is much faster than dynamic key accessing logics but have not explained from a theoretical point of view The next article would be about that Let s study the reason why AoT compilation makes your code faster Behind Story of NestiaI dislike class validator due to bad experience When developing a NestJS backend server about insurance service I d used class validator but suffered from its extremely slow performance class validator can validate only MB data per a second and unfortunately each contract of insurance easily exceeds MB At that time my backend server can get only connections per a second and it was a critical problem for business It was the reason why I abandoned class validator and used typescript is instead And as you could see from here article such bad experience made me to develop typia and nestia The next article would be about this story If you can read Korean language you can read the story from here right now If not wait my next article please ouid amp rtpof true amp sd true	2023-07-21 19:15:56
海外TECH	DEV Community	Data Integration: Google BigQuery with Mage	https://dev.to/mage_ai/data-integration-google-bigquery-with-mage-461p	Data Integration Google BigQuery with MageGuest blog by Shashank Mishra Data Engineer Expedia TLDRThis article outlines the integration between Mage and Google BigQuery a serverless data warehousing service We ll discuss the integration process its benefits and how it aids businesses in making data driven decisions OutlineIntroduction to MageOverview of Google BigQueryStep by step process to integrate Google BigQuery with MageConclusion Introduction to MageIn an age where data is the new oil efficient and reliable data management tools are essential Mage is a platform committed to simplifying data integration and analytics Designed for seamless data transformation and loading Mage is transforming how businesses approach data management Here are its key features Automated Data Pipeline Mage automates data extraction transformation and loading ETL processes It can extract data from multiple sources transform it to a desirable format and load it into a data warehouse Data Connectors Mage offers various data connectors to widely used data sources like Shopify Facebook Ads Google Ads Google Analytics etc This makes it easier to import data from these platforms Easy Integration Mage provides easy integration with popular data warehouses including Google BigQuery Amazon Redshift and Snowflake Pre built SQL Models Mage comes with pre built SQL models for popular e commerce platforms like Shopify and WooCommerce These models simplify the process of data analysis Incremental Loading Mage supports incremental loading which means only new or updated data is loaded into the data warehouse This saves storage space and improves efficiency Data Transformations Mage performs automatic data transformations converting raw data into a more usable format This process makes the data ready for analysis and reporting Scheduled Refresh Data refreshes can be scheduled in Mage ensuring that the data in the warehouse is always up to date Data Security Mage places a high emphasis on data security ensuring data privacy and compliance with GDPR and other data protection regulations Source Giphy Overview of Google BigQueryGoogle BigQuery is a highly scalable serverless data warehouse offered by Google as part of its Google Cloud Platform GCP It is designed to streamline and simplify the processing of big data Serverless Architecture BigQuery operates on a serverless model which means users don t need to manage any servers or infrastructure This means you can focus more on analysis and less on maintenance It allows you to query massive datasets in seconds and get insights in real time without needing to worry about resource provision Real Time Analytics BigQuery is engineered for real time analytics It allows users to analyze real time data streams instantly With its ability to run SQL queries on petabytes of data it delivers speedy results on real time data analytics enabling businesses to make timely decisions Google BigQuery with its serverless architecture and real time analytics serves as a robust platform to handle analyze and draw insights from massive datasets with ease Source Giphy Step by step process to migrate Google BigQuery with MageBefore we begin we ll need to create a service account key Please read Google Cloud s documentation on how to create that Once we are finished following these steps Create a new pipeline or open an existing pipeline Expand the left side of the screen to view the file browser Scroll down and click on a file named io config yamlEnter the following keys and values under the key named default we can have multiple profiles add it under whichever is relevant for us Note we only need to add the keys under GOOGLE SERVICE ACC KEY or the value for key GOOGLE SERVICE ACC KEY FILEPATH both are not simultaneously required version default GOOGLE SERVICE ACC KEY type service account project id project id private key id key id private key BEGIN PRIVATE KEY nyour private key n END PRIVATE KEY client email your service account email auth uri token uri auth provider x cert url client x cert url GOOGLE SERVICE ACC KEY FILEPATH path to your service account key json Using SQL blockCreate a new pipeline or open an existing pipeline Add a data loader transformer or data exporter block Select SQL Under the Data provider dropdown select BigQueryUnder the Profile dropdown select default or the profile we added credentials underneath Next to the Database label enter the database name we want this block to save data to Next to the Save to schema label enter the schema name we want this block to save data to Under the Write policy dropdown select Replace or Append please see SQL blocks guide for more information on write policies Enter in this test query SELECT Run the block Using Python blockCreate a new pipeline or open an existing pipeline Add a data loader transformer or data exporter block the code snippet below is for a data loader Select Generic no template Enter this code snippet note change the config profile from default if we have a different profile from mage ai data preparation repo manager import get repo pathfrom mage ai io bigquery import BigQueryfrom mage ai io config import ConfigFileLoaderfrom os import pathfrom pandas import DataFrameif data loader not in globals from mage ai data preparation decorators import data loader data loaderdef load data from big query kwargs gt DataFrame query SELECT config path path join get repo path io config yaml config profile default return BigQuery with config ConfigFileLoader config path config profile load query Run the block Source Giphy ConclusionIntegrating Mage with Google BigQuery provides your team with a potent combination of automated data pipeline management and robust data warehousing This partnership not only simplifies data extraction transformation and loading but also provides a seamless pathway for data analysis and insight generation As we ve demonstrated in this step by step guide the integration process is straightforward making it an accessible option for businesses of all sizes By leveraging this integration you can unlock the full potential of your data streamline operations and drive data informed decisions Link to the original blog	2023-07-21 19:01:33
Apple	AppleInsider - Frontpage News	Apple's M2 Max MacBook Pro with 64GB RAM gets $300 price drop, plus $80 off AppleCare	https://appleinsider.com/articles/23/07/21/apples-m2-max-macbook-pro-with-64gb-ram-gets-300-price-drop-plus-80-off-applecare?utm_medium=rss	Apple x s M Max MacBook Pro with GB RAM gets price drop plus off AppleCareA price war has emerged on Apple s latest inch MacBook Pro with a loaded M Max configuration packing GB of RAM and TB of storage now off in addition to off AppleCare To take advantage of the exclusive savings head over to Adorama com and shop with promo code APINSIDER during checkout instructions showing where to enter the coupon code can be found below Use code APINSIDER Read more	2023-07-21 19:08:43
Apple	AppleInsider - Frontpage News	How to use iPad as a portable screen for your Nintendo Switch	https://appleinsider.com/inside/ipados-17/tips/how-to-use-ipad-as-a-portable-monitor-for-your-nintendo-switch?utm_medium=rss	How to use iPad as a portable screen for your Nintendo SwitchA feature in iPadOS enables iPads to work with USB C capture cards Here s how to use the feature to make your large screen iPad work as an external monitor for your Nintendo Switch Use iPad as a monitorApple briefly mentioned external webcam support when it revealed iPadOS during WWDC This support uses a system that brings video recorded from a device into a compatible camera app which coincidentally applies to devices like capture cards Read more	2023-07-21 19:23:35
海外TECH	Engadget	OpenAI's trust and safety lead is leaving the company	https://www.engadget.com/openais-trust-and-safety-lead-is-leaving-the-company-190049987.html?src=rss	OpenAI x s trust and safety lead is leaving the companyOpenAI s trust and safety lead Dave Willner has left the position as announced via a Linkedin post Willner is staying on in an “advisory role but has asked Linkedin followers to “reach out for related opportunities The former OpenAI project lead states that the move comes after a decision to spend more time with his family Yes that s what they always say but Willner follows it up with actual details “In the months following the launch of ChatGPT I ve found it more and more difficult to keep up my end of the bargain he writes “OpenAI is going through a high intensity phase in its development ーand so are our kids Anyone with young children and a super intense job can relate to that tension He continues to say he s “proud of everything the company accomplished during his tenure and noted it was “one of the coolest and most interesting jobs in the world Of course this transition comes hot on the heels of some legal hurdles facing OpenAI and its signature product ChatGPT The FTC recently opened an investigation into the company over concerns that it is violating consumer protection laws and engaging in “unfair or deceptive practices that could hurt the public s privacy and security The investigation does involve a bug that leaked users private data which certainly seems to fall under the purview of trust and safety Willner says his decision was actually a “pretty easy choice to make though not one that folks in my position often make so explicitly in public He also states that he hopes his decision will help normalize more open discussions about work life balance nbsp There s growing concerns over the safety of AI in recent months and OpenAI is one of the companies that agreed to place certain safeguards on its products at the behest of President Biden and the White House These include allowing independent experts access to the code flagging risks to society like biases sharing safety information with the government and watermarking audio and visual content to let people know that it s AI generated This article originally appeared on Engadget at	2023-07-21 19:00:49
海外TECH	WIRED	‘Now I Am Become Death, the Destroyer of Worlds.’ The Story of Oppenheimer’s Infamous Quote	https://www.wired.com/story/manhattan-project-robert-oppenheimer/	Now I Am Become Death the Destroyer of Worlds The Story of Oppenheimer s Infamous QuoteThe line from the Hindu sacred text the Bhagavad Gita has come to define Robert Oppenheimer but its meaning is more complex than many realize	2023-07-21 19:56:20
ニュース	BBC News - Home	The Open: Brian Harman leads Tommy Fleetwood by five at Royal Liverpool	https://www.bbc.co.uk/sport/golf/66267262?at_medium=RSS&at_campaign=KARANGA	liverpool	2023-07-21 19:41:59
ビジネス	ダイヤモンド・オンライン - 新着記事	【無料公開】JR5社の「勝負ホテル」が外資系マリオット一色に染まる理由【世界メジャー徹底解剖】 - Diamond Premiumセレクション	https://diamond.jp/articles/-/326328	diamond	2023-07-22 04:55:00
ビジネス	ダイヤモンド・オンライン - 新着記事	「出世したい欲」が強い人の“残念すぎる末路” - 「静かな人」の戦略書	https://diamond.jp/articles/-/326008	動機づけ	2023-07-22 04:50:00
ビジネス	ダイヤモンド・オンライン - 新着記事	【考えすぎて眠れない…】人間関係の悩みから一瞬で解放される考え方とは？ - 無意識さんの力でぐっすり眠れる本	https://diamond.jp/articles/-/326370	人間関係	2023-07-22 04:47:00
ビジネス	ダイヤモンド・オンライン - 新着記事	【投資のプロが教える】日本の小型株に投資するおすすめファンド2本 - インフレ・円安からお金を守る最強の投資	https://diamond.jp/articles/-/326481	【投資のプロが教える】日本の小型株に投資するおすすめファンド本インフレ・円安からお金を守る最強の投資インフレ・円安の時代に入った今、資産を預金だけで持つことはリスクがあり、おすすめできない。	2023-07-22 04:44:00
ビジネス	ダイヤモンド・オンライン - 新着記事	【名医が教える】ケトン食の有効性が、特に期待できるがんとは？ - 糖質制限はやらなくていい	https://diamond.jp/articles/-/326473	【名医が教える】ケトン食の有効性が、特に期待できるがんとは糖質制限はやらなくていい「日食では、どうしても糖質オーバーになる」「やせるためには糖質制限が必要」…。	2023-07-22 04:41:00
ビジネス	ダイヤモンド・オンライン - 新着記事	【子育ての悩み】てぃ先生が答える！「今度にしようね」を納得してもらうには？ - カリスマ保育士てぃ先生の子育てのみんなの悩み、お助け中！	https://diamond.jp/articles/-/324820	【子育ての悩み】てぃ先生が答える「今度にしようね」を納得してもらうにはカリスマ保育士てぃ先生の子育てのみんなの悩み、お助け中【YouTube万人、Twitter万人、Instagram万人】今どきのママパパに圧倒的に支持されているカリスマ保育士・てぃ先生の子育てアドバイス本第弾『子どもにもっと伝わるスゴ技大全カリスマ保育士てぃ先生の子育てのみんなの悩み、お助け中』ができましたテレビやSNSで大人気、今どきのママパパに圧倒的に支持されている現役保育士・てぃ先生。	2023-07-22 04:38:00
ビジネス	ダイヤモンド・オンライン - 新着記事	【『世界一受けたい授業』で話題】 真面目な人ほど気をつけたい「オール・オア・ナッシングの罠」 - 10年後、後悔しない体のつくり方	https://diamond.jp/articles/-/324701	【『世界一受けたい授業』で話題】真面目な人ほど気をつけたい「オール・オア・ナッシングの罠」年後、後悔しない体のつくり方【大反響Amazonベストセラー・ベスト】「近ごろ体を動かす機会がメッキリ減ってしまった」なんて人は多いはず。	2023-07-22 04:35:00
ビジネス	ダイヤモンド・オンライン - 新着記事	「数字に弱い」コンプレックスの解消法ベスト1 - 小学生がたった1日で19×19までかんぺきに暗算できる本	https://diamond.jp/articles/-/326418	解消	2023-07-22 04:32:00
ビジネス	ダイヤモンド・オンライン - 新着記事	明治大学の学生にリアルな就活について話を聞いてみた【学生のコメント付き！】 - 大学図鑑！2024 有名大学82校のすべてがわかる！	https://diamond.jp/articles/-/326507		2023-07-22 04:29:00
ビジネス	ダイヤモンド・オンライン - 新着記事	職場にいる「本当に仕事ができる人」と「仕事ができそうでできない人」の決定的な差とは - 1秒で答えをつくる力 お笑い芸人が学ぶ「切り返し」のプロになる48の技術	https://diamond.jp/articles/-/326503		2023-07-22 04:26:00
ビジネス	ダイヤモンド・オンライン - 新着記事	税金による吸い上げに一矢報いる秘策とは？ - 40代からは「稼ぎ口」を2つにしなさい	https://diamond.jp/articles/-/326546	新刊『代からは「稼ぎ口」をつにしなさい年収アップと自由が手に入る働き方』では、余すことなく珠玉のメソッドを公開しています。	2023-07-22 04:23:00
ビジネス	ダイヤモンド・オンライン - 新着記事	会社を変えようとする経営者が「絶対にやってはいけない」たった1つのこと（前編） - 新装版　売上2億円の会社を10億円にする方法	https://diamond.jp/articles/-/325760	会社を変えようとする経営者が「絶対にやってはいけない」たったつのこと前編新装版売上億円の会社を億円にする方法コツコツと業績を伸ばしてきた経営者が直面する「売上の壁」。	2023-07-22 04:20:00
ビジネス	ダイヤモンド・オンライン - 新着記事	【精神科医が教える】 “異なる意見を言うと怒り出す人”へのたった1つの対処法 - 精神科医Tomyが教える 40代を後悔せず生きる言葉	https://diamond.jp/articles/-/325956	【精神科医が教える】“異なる意見を言うと怒り出す人へのたったつの対処法精神科医Tomyが教える代を後悔せず生きる言葉【大好評シリーズ万部突破】誰しも悩みや不安は尽きない。	2023-07-22 04:17:00
ビジネス	ダイヤモンド・オンライン - 新着記事	知っておくといざという時に役立つ「仮説思考の7つ道具」 - １位思考	https://diamond.jp/articles/-/325594		2023-07-22 04:14:00
ビジネス	ダイヤモンド・オンライン - 新着記事	【まんが】相手の気持ちを試してしまう…「試し行動」をやめられない人の「心の闇」が驚くほど晴れる、たった1つの質問＜心理カウンセラーが教える＞ - あなたはもう、自分のために生きていい	https://diamond.jp/articles/-/326291	【まんが】相手の気持ちを試してしまう…「試し行動」をやめられない人の「心の闇」が驚くほど晴れる、たったつの質問あなたはもう、自分のために生きていいTwitterで人気の人間関係、親子問題、機能不全家族専門カウンセラーが、生きづらさを抱え、すべての原因は自分にあると思い込んで生きてきた人たちに、本当の原因は何なのかを明らかにし、傷ついた心をラクにする方法を伝えます。	2023-07-22 04:11:00
ビジネス	ダイヤモンド・オンライン - 新着記事	フランスの自動車会社が「自社の利益」より大切にするものとは？ - 発想の回路	https://diamond.jp/articles/-/326520	フランスの自動車会社が「自社の利益」より大切にするものとは発想の回路「アイデアが思いつかない」「企画が通らない」「頑張っても成果が出ない」このように悩んだことはないでしょうか。	2023-07-22 04:08:00
ビジネス	ダイヤモンド・オンライン - 新着記事	100万円もらったらすぐに使う？ 貯金する？「幸福度が高い人」のお金の使い方の共通点 - DIE WITH ZERO	https://diamond.jp/articles/-/326465	diewithzero	2023-07-22 04:05:00
ビジネス	ダイヤモンド・オンライン - 新着記事	グループディスカッションを通過するための「正しい準備」3選 - 絶対内定	https://diamond.jp/articles/-/326471	杉村太郎	2023-07-22 04:02:00
ビジネス	東洋経済オンライン	東武･獨協大学前､マンモス団地を支えた駅の変身 かつての｢松原団地駅｣は学園都市の玄関に | 駅･再開発 | 東洋経済オンライン	https://toyokeizai.net/articles/-/688017?utm_source=rss&utm_medium=http&utm_campaign=link_back	埼玉県草加市	2023-07-22 04:30:00