投稿時間:2021-08-13 02:23:59 RSSフィード2021-08-13 02:00 分まとめ(24件)

カテゴリー等	サイト名等	記事タイトル・トレンドワード等	リンクURL	頻出ワード・要約等/検索ボリューム	登録日
AWS	AWS Startups Blog	Next Generation Data Management for Clinical Trials & Research Built on AWS	https://aws.amazon.com/blogs/startups/next-generation-data-management-for-clinical-trials-research-built-on-aws/	Next Generation Data Management for Clinical Trials amp Research Built on AWSGuest post by Steve Bilawey Co Founder amp Chief Business Officer Precision Digital Health Data and Automation in Drug Development The life sciences industry is absorbing a deluge of Real World Data RWD nbsp For the players who embrace technology and automation this can revolutionize the clinical trials process reducing time to market for life changing treatments Sponsors …	2021-08-12 16:54:34
AWS	AWS Startups Blog	Understanding and Optimizing ML Models with DarwinAI	https://aws.amazon.com/blogs/startups/understanding-optimizing-ml-models-darwinai/	Understanding and Optimizing ML Models with DarwinAIArtificial intelligence may be the future but of AI models developed today don t make it into production DarwinAI has set out to solve that problem by enabling organizations to understand and optimize models making it easier to build what matters	2021-08-12 16:34:23
AWS	AWS - Webinar Channel	Explore Cloud Native Observability with AWS - AWS Virtual Workshop	https://www.youtube.com/watch?v=UW7aT25Mbng	Explore Cloud Native Observability with AWS AWS Virtual WorkshopIn this session we will guide users through the AWS observability suite and demystify tools we offer that enable collection analytics and insights into your logs metrics and transaction traces Users will gain deep insights from proactive alerting mature automated root cause detection and application performance and health Explore how these tools will reduce your time to identify production issues and how to expose your dashboard presentation capabilities to a wider audience within your organization Learning Objectives What cloud native observability tools are available from AWS Understand the techniques of logs metrics and traces for monitoring application performance and health Determine the services that will enable you to break down data silos gain system wide visibility and quickly resolve issues	2021-08-12 16:41:07
js	JavaScriptタグが付けられた新着投稿 - Qiita	［JavaScript］fetch メソッド	https://qiita.com/tanakadaichi_1989/items/0a67b43b00fbcd03a4f7		2021-08-13 01:33:36
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	JavaScript transitionEndイベント	https://teratail.com/questions/354041?rss=all	JavaScripttransitionEndイベントtransitionEndイベントでは、アニメーションで変化する要素の回数だけ実行されます。	2021-08-13 01:59:16
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	横文字で固定したいのに画面が縮小すると文字が動く	https://teratail.com/questions/354040?rss=all	横文字で固定したいのに画面が縮小すると文字が動く枚目のお手本のように文字の横に線があってセンターに配置、画面サイズを拡大縮小しても変わらないようにしたいのですができませんでした。	2021-08-13 01:21:53
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	jupyter notebook上でanacondaのpythonが使えない	https://teratail.com/questions/354039?rss=all	jupyternotebook上でanacondaのpythonが使えない前提・実現したいことjupyternbspnotebook上でanacondaのpythonが使えない為、各種ライブラリが使えず困っております。	2021-08-13 01:05:53
Program	[全てのタグ]の新着質問一覧｜teratail（テラテイル）	python３でjsonがインストールできない場合	https://teratail.com/questions/354038?rss=all	pythonでjsonがインストールできない場合PCnbspwindowsエディターvscodepythonnbspバージョンnbsprenderingpyというファイルにこちらを書いたところインストールされて無いようでしたのでターミナルを開きpipnbspinstallnbspjsonを打ち込みましたがこうエラーが出てしまいます。	2021-08-13 01:05:38
海外TECH	Ars Technica	Apple and Google seem spooked by bill requiring more app stores and sideloading	https://arstechnica.com/?p=1786775	party	2021-08-12 16:38:32
海外TECH	Ars Technica	Star Trek: Lower Decks still understands what makes Trek tick	https://arstechnica.com/?p=1786380	decks	2021-08-12 16:17:21
海外TECH	DEV Community	Structuring a Node.js and Express Backend	https://dev.to/samarmohan/structuring-a-node-js-and-express-backend-23h	Structuring a Node js and Express Backend IntroductionThere are so many ways to structure a Node js and Express API and each one has its own pros and cons In this post I ll detail the two main ways MVC and modules Note I m not going to keep adding Easy to understand and related pros becausethis is all based on opinion pick one and stick to it The structures Modules Inspired by Django s appsOne way to structure an app is with modules Each module contains all the code for a certain part of the app like auth or posts Each folder will have the controllers routes entities and middleware An example of the auth module is above You structure the auth module your own way this is just mine You could use an MVC pattern inside the module and some people may want to strip out the middleware and entities Pros Easy to work on a certain feature everything is in one place Cons You may not want everything bundled up like that You ll have to use fileName typeOfFile extension instead of just fileName extension MVC Used in Phoenix Rails and ASP NET CoreThis is by far the most common way to structure a backend project and many popular frameworks use it You have your controllers logic handling requests views ui and models database schemas business logic An example of MVC in action The auth controller will have the register login and logout methods The model will have the database entity fields and the view will be what the end user sees Pros You know how everything interactsCons Multiple foldersRepetitiveYou need to understand the why behind MVCThanks for reading I hope this cleared up some things for you	2021-08-12 16:55:01
海外TECH	DEV Community	How to Burst the "Tutorial Phase" When Becoming a Developer	https://dev.to/michaelmangial1/how-to-burst-the-tutorial-phase-when-becoming-a-developer-gk	How to Burst the quot Tutorial Phase quot When Becoming a DeveloperTeaching is essentially a transfer of knowledge In this sense every developer is a teacher Every developer has to instruct transfer knowledge to the computer so that an application does what it is supposed to Now imagine if a teacher s knowledge was limited to only knowing the right textbook to read and follow without any awareness of the reasoning behind the subject Picture a chemistry teacher who doesn t grasp the fundamental concepts The same is true for developers Many new aspiring developers become so fixated on understanding a particular framework following tutorials video courses etc without grasping the fundamental concepts After doing a video course they can make a React movie app but they don t know where to begin if asked to make a Twitter clone Now I don t mean to sound discouraging I ve been there Impostor syndrome creeps in You begin to think that you must not be cut out for the field etc etc That s tough But I think pointing all of this out works towards encouragement Here s why I think that there are tangible practical ways to get out of the tutorial phase and understand the concepts behind the particular discipline of software development that you are trying to learn So what are those practical ways to grasp the fundamental concepts and move beyond the tutorial phase Teach the material to othersThis could go for anything that you are learning but it is especially helpful for those who are just learning to code As we ve mentioned good teaching requires understanding the fundamental concepts So to write an article on a subject for example you will be forced to understand it in such a manner that you can effectively communicate it to someone else It s impossible to teach and not learn in the process Take whatever you re trying to learn and post articles videos etc explaining it to someone else Not only will this help you learn but you communicate in a manner that is more sympathetic and comprehensible to those who are also in the middle of learning something new Sure It takes time But I m persuaded it will take less time than the vicious cycle of the tutorial phase Oh and by the way you can also make some profit from your time I applied the teaching method earlier this year when I wanted to learn how to build a Shopify like design system from scratch using design tokens I wrote blog posts as I was learning After mastering the subject through my teaching I rewrote all my thoughts on the subject into an ebook I don t make a lot from it but it s a nice bonus Use the Source LukeI once recorded a podcast episode called Use the Source Luke that explains this concept to Star Wars fans The gist is that the way to take it to the next level as a developer is to look at the source code of frameworks and libraries that you are using After all a library s documentation is just an author of the library s attempt to explain their code Sometimes reading the code itself gives clues to the original intent as to how things work Libraries are created to solve problems If you read the code of a library you tracing the breadcrumbs as to how the problem was solved If you read carefully you will understand how it solves the problem and be able to employ similar ideas to solve your own problems The point of all of this is to build confidence so that when you get stuck in writing code you will know how to get to the bottom of it Sometimes this requires looking at source code Even if it doesn t looking at source code enhances the same debugging mindset Assign yourself a projectToo often aspiring developers start doing what tutorials tell them to do i e a Twitter Slack Twitch clone Those are good projects to be sure However the issue is when you are more focused on following a tutorial than learning the concepts A good way to learn is to force yourself to have to do it on your own So you should assign yourself a project Also doing a project that someone else finds motivating won t be too motivating for yourself To put it positively doing a project that you are super interested in will provide the stamina to continue through the troubles that arise when trying to build something ConclusionIn conclusion tutorials are a good start for aspiring developers I still use them today However they can be a stumbling block Consider following any of these three means that I have provided to get out of the tutorial phase	2021-08-12 16:49:46
海外TECH	DEV Community	Implementing Reactivity from scratch	https://dev.to/siddharthshyniben/implementing-reactivity-from-scratch-51op	Implementing Reactivity from scratchReactivity is at the heart of many web interfaces It makes programming robust and interactive web apps much much easier Although most frameworks have reactivity as a built in feature there will always be a point when you need reactivity in plain JavaScript So here I will show you how to implement reactivity in JavaScript Wait What is reactivity There are a bunch of explanations out there the best one so far being this But here I ll show you a code sample which is easier to understand Suppose you have this const who Siddharth document querySelector h innerText who Later you change who who Somebody But the content in the H does not change until we call document querySelector h innerText who again This is where reactivity comes in It automatically reruns the code in our case document querySelector h innerText who when the referred variables change So when we change the variable the change is automatically reflected in the code The engineNote to keep this tutorial simple and fun I won t implement error handling objects and all the boring checks The next parts of this tutorial will go in detail on some of them First let s build an object which we need to react to let data name John Doe age One way to make it reactive would be to have setters getters to listen for events and react to that A quick note on setters getters Getters and setters are functions which are called when an object s property is called set Here s a simple example const obj data get foo return this data join set foo val this data push val obj foo obj foo obj foo obj foo gt Setters and getters are really helpful when building reactivitySo we would need to change the object to be like this let data name John Doe get name return this name set name val this name name TODO notify And code using it would look like this const data new Reactive name John Doe age data listen name val gt console log name was changed to val data contents name Siddharth gt name was changed to SiddharthSo let s first build the Reactive class class Reactive constructor obj TODO listen prop TODO the constructor is quite simple just set the data and start observing constructor obj this contents obj this listeners Will be explained later this makeReactive obj Now we ll implement makeReactive makeReactive obj Object keys obj forEach prop gt this makePropReactive obj prop Now we ll implement makePropReactive makePropReactive obj key let value obj key Cache Object defineProperty obj key get return value set newValue value newValue this notify key Here we use Object defineProperty to set getters on an the object Next thing to do is set up a notifier and an listener The listener is pretty simple listen prop handler if this listeners prop this listeners prop this listeners prop push handler Here we set listeners on an object as values in an array Next to notify notify prop this listeners prop forEach listener gt listener this contents prop And that s the end Here s the full code class Reactive constructor obj this contents obj this listeners this makeReactive obj makeReactive obj Object keys obj forEach prop gt this makePropReactive obj prop makePropReactive obj key let value obj key Gotta be careful with this here const that this Object defineProperty obj key get return value set newValue value newValue that notify key listen prop handler if this listeners prop this listeners prop this listeners prop push handler notify prop this listeners prop forEach listener gt listener this contents prop Simple isn t it Here s a repl Setup codeclass Reactive constructor obj this contents obj this listeners this makeReactive obj makeReactive obj Object keys obj forEach prop gt this makePropReactive obj prop makePropReactive obj key let value obj key Gotta be careful with this here const that this Object defineProperty obj key get return value set newValue value newValue that notify key listen prop handler if this listeners prop this listeners prop this listeners prop push handler notify prop this listeners prop forEach listener gt listener this contents prop const data new Reactive foo bar data listen foo change gt console log Change change data contents foo baz Thanks for reading In the next parts we ll get a bit more into how we can enhance this	2021-08-12 16:09:52
Apple	AppleInsider - Frontpage News	How to use Hide My Email in iOS 15	https://appleinsider.com/articles/21/08/12/how-to-use-hide-my-email?utm_medium=rss	How to use Hide My Email in iOS For an iOS feature that s meant to be incredibly easy to use Hide My Email comes with a lot of options Here s what you can and as yet can t do You re going to be using Hide My Email a lot although you ll rarely go to this particular screenThe forthcoming Hide My Email feature in iOS is not meant to replace Sign In with Apple but it s a clear evolution of that idea Rather than give a company your actual email address you give them one that works perfectly well ーbut which you can switch off when you need Read more	2021-08-12 16:55:42
Apple	AppleInsider - Frontpage News	Apple TV+ original sci-fi film 'Finch' starring Tom Hanks debuts on Nov. 5	https://appleinsider.com/articles/21/08/12/apple-tv-original-sci-fi-film-finch-starring-tom-hanks-debuts-on-nov-5?utm_medium=rss	Apple TV original sci fi film x Finch x starring Tom Hanks debuts on Nov Apple has announced that its Tom Hanks led post apocalyptic sci fi original film Finch will make its debut on Apple TV on Friday Nov Credit AppleThe film follows a man a robot and a dog who form an unlikely family in a a powerful and moving adventure of one man s quest to ensure that his beloved canine companion will be cared for after he s gone Read more	2021-08-12 16:31:05
Apple	AppleInsider - Frontpage News	'iPhone 13' to launch in September with A15, bigger batteries, researchers say	https://appleinsider.com/articles/21/08/12/iphone-13-to-launch-in-september-with-a15-bigger-batteries-trendforce-says?utm_medium=rss	x iPhone x to launch in September with A bigger batteries researchers sayApple is likely to release all four models of its new iPhone lineup in September with larger batteries an updated chipset and expanded mmWave G support according to research firm TrendForce Credit Andrew O Hara AppleInsiderTrendForce on Thursday outlined its expectations for the device which it says will return to a typical September release Driven by the new models the research firm believes that total iPhone shipments could surge year over year in the third quarter of of Read more	2021-08-12 16:20:02
Apple	AppleInsider - Frontpage News	Billie Eilish and Apple Music team up on Spatial Audio short film	https://appleinsider.com/articles/21/08/12/billie-eilish-and-apple-music-team-up-on-spatial-audio-short-film?utm_medium=rss	Billie Eilish and Apple Music team up on Spatial Audio short filmA short film made by Apple Music promotes its new Spatial Audio with Dolby Atmos feature using Billie Eilish s new album Happier Than Ever Billie Eilish promotes her new album in Spatial AudioThe second short film is among a group of short interview clips promoting the album Apple s marketing push for Spatial Audio in Apple Music has seen many short ads and featured playlists on the service Read more	2021-08-12 16:15:51
Cisco	Cisco Blog	Grandson of FISMA: Why We Desperately Need New Cybsersecurity Legislation from the 117th Congress	https://blogs.cisco.com/security/grandson-of-fisma-why-we-desperately-need-new-cybsersecurity-legislation-from-the-117th-congress	Grandson of FISMA Why We Desperately Need New Cybsersecurity Legislation from the th CongressCongress is considering reform of FISMA legislation and this blog is intended to provide insights and recommendations and inform	2021-08-12 16:25:41
海外科学	NYT > Science	For Many, Hydrogen Is the Fuel of the Future. New Research Raises Doubts.	https://www.nytimes.com/2021/08/12/climate/hydrogen-fuel-natural-gas-pollution.html	For Many Hydrogen Is the Fuel of the Future New Research Raises Doubts Industry has been promoting hydrogen as a reliable next generation fuel to power cars heat homes and generate electricity It may in fact be worse for the climate than previously thought	2021-08-12 16:54:12
ニュース	BBC News - Home	Afghanistan: Taliban take 11th provincial capital as Ghazni and Herat fall	https://www.bbc.co.uk/news/world-asia-58184202	cities	2021-08-12 16:36:27
ニュース	BBC News - Home	Actress Una Stubbs dies aged 84	https://www.bbc.co.uk/news/entertainment-arts-58190446	sherlock	2021-08-12 16:24:16
ニュース	BBC News - Home	Leeds United: Marcelo Bielsa signs one-year contract extension	https://www.bbc.co.uk/sport/football/58193100	Leeds United Marcelo Bielsa signs one year contract extensionLeeds United head coach Marcelo Bielsa signs a new one year contract extension ahead of Saturday s Premier League opener at Manchester United	2021-08-12 16:16:34
ニュース	BBC News - Home	Covid-19 in the UK: How many coronavirus cases are there in my area?	https://www.bbc.co.uk/news/uk-51768274	cases	2021-08-12 16:15:55
GCP	Cloud Blog	BigQuery Admin reference guide: Data governance	https://cloud.google.com/blog/topics/developers-practitioners/bigquery-admin-reference-guide-data-governance/	BigQuery Admin reference guide Data governanceHopefully you ve been following along with our BigQuery Admin series and are well on your way to getting ramped up with BigQuery Now that you re equipped with the fundamentals let s talk about something that s relevant for all data professionals data governance  What does data governance mean Data governance is everything you do to ensure your data is secure private accurate available and usable inside of BigQuery With good governance everyone in your organization can easily find and leverage the data they need to make effective decisions All while minimizing the overall risk of data leakage or misuse and ensuring regulatory compliance  BigQuery security featuresBecause BigQuery is a fully managed service we take care of a lot of the hard stuff for you Like we talked about in our post on BigQuery Storage Internals BigQuery data is replicated across data centers to ensure reliability and availability Plus data is always encrypted at rest By default we ll manage encryption keys for you However you have the option to leverage customer managed encryption keys by using Cloud KMS to automatically rotate and destroy encryption keys  You can also leverage Google Virtual Private Cloud VPC Service Controls to restrict traffic to BigQuery When you correctly apply these controls unauthorized networks can t access BigQuery data and data can t be copied to unauthorized Google Cloud projects Free communication can still occur within the perimeter but communication is restricted across the perimeter Aside from leveraging BigQuery s out of the box security features there are also ways to improve governance from a process perspective In this post we ll walk you through the different tactics to ensure data governance at your organization  Dataset onboarding Understanding amp classifying data Data governance starts with dataset onboarding Let s say you just received a request from someone on your eCommerce team to add a new dataset that contains customer transactions The first thing you ll need to do is understand the data You might start by asking questions like these What information does this contain How will it be used to make business decisions Who needs access to this data Where does the data come from and how will analysts get access to it in BigQuery Understanding the data helps you make decisions on where the new table should live in BigQuery who should have access to this data and how you ll plan to make the data accessible inside of BigQuery e g leveraging an external table batch loading data into native storage etc   For this example the transactions live in an OLTP database Let s take a look at what information is contained in the existing table in our database Below we can see that this table has information about the order when it was placed who purchased it any additional comments for the order and details on the items that were purchased the item ID cost category etc Click to enlargeNow that we have an idea of what data exists in the source and what information is relevant for the business we can determine which fields we need in our BigQuery table and what transformations are necessary to push the data into a production environment Classifying informationData classification means that you are identifying the types of information contained in the data and storing it as searchable metadata  By properly classifying data you can make sure that it s handled and shared appropriately and that data is discoverable across your organization  Since we know what the production table should look like we can go ahead and create an empty BigQuery table with the appropriate schema that will house the transactions  As far as storing metadata about this new table we have two different options  Using labelsOn the one hand we can leverage labels Labels can be used on many BigQuery resourcesincluding Projects Datasets and Tables They are key value pairs and can be used to filter data in Cloud Monitoring or can be used in queries against the Information Schema to find data that pertains to specific use cases Click to enlargeAlthough labels provide logical segregation and management of different business purposes in the Cloud ecosystem they are not meant to be used in the context of data governance Labels cannot specify a schema and you can t apply them to specific fields in your table  Labels cannot be used to establish access policies or track resource hierarchy  It s pretty clear that our transactions table may contain personally identifiable information PII Specifically we may want to mark the email address column as “Has PII True Instead of using labels on our new table we ll leverage Data Catalog to establish a robust data governance policy incorporating metadata tags on BigQuery resources and individual fields Using data catalog tagsData Catalog is Google Cloud s data discovery and metadata management service As soon as you create a new table in BigQuery it is automatically discoverable in Data Catalog Data Catalog tracks all technical metadata related to a table such as name description time of creation column names and datatypes and others In addition to the metadata that is captured through the BigQuery integration you can create schematized tagsto track additional business information For example you may want to create a tag that tracks information about the source of the data the analytics use case related to the data or column level information related to security and sharing Going back to that email column we mentioned earlier we can simply attach a column level governance tag to the field and fill out the information by specifying that email address is not encrypted it does contain PII and more specifically it contains an email address While this may seem like a fairly manual process Data Catalog has a fully equipped API which allows for tags to be created attached and updated programmatically With tags and technical metadata captured in a single location data consumers can come to Data Catalog and search for what they need Ingesting amp staging dataWith metadata for the production table in place we need to focus on how to push data into this new table As you probably know there are lots of different ways to pre process and ingest data into BigQuery Often customers choose to stage data in Google Cloud Services to kick off transformation classification or de identification workflows There are two pretty common paths for staging data for batch loading Stage data in a Google Cloud storage bucket Pushing data into a Google Cloud storage bucket before directly ingesting it into BigQuery offers flexibility in terms of data structure and may be less expensive for storing large amounts of information Additionally you can easily kick off workflows when new data lands in a bucket by using PubSub to trigger transformation jobs However since transformations will happen outside of the BigQuery service data engineers will need familiarity with other tools or languages Blob storage also makes it difficult to track column level metadata Stage data in a BigQuery staging container Pushing data into BigQuery gives you the opportunity to track metadata for specific fields earlier in the funnel through BigQuery s integration with Data Catalog When running scan jobs with Data Loss Prevention we ll cover this in the next section you can leave out specific columns and store the results directly in the staging table s metadata inside of Data Catalog Additionally transformations to prepare data for production can be done using SQL statements which may make them easier to develop and manage  Identifying and de identifying sensitive information One of the hardest problems related to data governance is identifying any sensitive information in new data Earlier we talked through tracking known metadata in Data Catalog but what happens if we don t know if data contains any sensitive information This might be especially useful for free form text fields like the comments field in our transactions With the data staged in Google Cloud there s an opportunity to programmatically identify any PII or even remove sensitive information from the data using Data Loss Prevention DLP  DLP can be used to scan data for different types of sensitive information such as names email addresses locations credit card numbers and others You can kick off a scan job directly from BigQuery Data Catalog or the DLP service or API DLP can be used to scan data that is staged in BigQuery or in Google Cloud Additionally for data stored in BigQuery you can have DLP push the results of the scan directly into Data Catalog You can also use the DLP API to de identify data For example we may want to replace any instances of names email addresses and locations with an asterisk “ In our case we can leverage DLP specifically to scan the comments column from our staging table in BigQuery save the results in Data Catalog and if there are instances of sensitive data run a de identification workflow before pushing the sanitized data into the production table Note that building a pipeline like the one we re describing does require the use of some other tools  We could use a Cloud Function to make the API call and an orchestration tool like Cloud Composer to run each step in the workflow trying to decide on the right orchestration tool check out this post You can walk through an example of running a de identification workflow using DLP and composer in this post Data sharingBigQuery Identity Access ManagementGoogle Cloud as a whole leverages Identity Access Management IAM to manage permissions across cloud resources With IAM you manage access control by defining who identity has what access role for which resource BigQuery like other Google Cloud resources has several predefined roles Or you can create custom roles based on more granular permissions When it comes to granting access to BigQuery data many administrators chose to grant Google Groups representing your company s different departments access to specific datasets or projects so policies are simple to manage You can see some examples of different business scenarios and the recommended access policies here  In our retail use case we have one project for each team Each team s Google Group would be granted the BigQuery Data Viewer role to access information stored in their team s project However there may be cases where someone from the ecommerce team needs data from a different project like the product development team project One way to grant limited access to data is through the use of authorized views Protecting data with authorized viewsGiving a view access to a dataset is also known as creating an authorized view in BigQuery An authorized view allows you to share query results with particular users and groups without giving them access to the underlying source data So in our case we can simply write a query to grab the pieces of information the ecommerce team needs to effectively analyze the data and save that view into the existing ecommerce project that they already have access to Column level access policiesAside from controlling access to data using standard IAM roles or granting access to query results through authorized views you also can leverage BigQuery s column level access policies For example remember that email address column we marked as containing PII earlier in this post We may want to ensure that only members with high security level clearance have access to query those columns We can do this by First defining a taxonomy in Data Catalog including a “High policy tag for fields with high security level clearanceNext add our group of users who need access to highly sensitive data as Fine Grained Access Readers to the High resourceFinally we can set a policy tag on the email columnYou can find some tips on creating column level access policies in our documentation on best practices Row level access policiesAside from restricting access to certain fields in our new table we may want to only grant users access to rows that are relevant to them One example may be if analysts from different business units only get access to rows that represent transactions for that business unit In this case the Google Group that represents the Activewear Team should only have access to orders that were placed on items categorized as “Active In BigQuery we can accomplish this by creating a row level access policy on the transactions table You can find some tips on creating row level access policies in our documentation on best practices When to use what for data sharingAt the end of the day you can achieve your goal of securing data using one or more of the concepts we discussed earlier Authorized Views add a layer of abstraction to sharing data by providing the necessary information to certain users without giving them direct access to the underlying dataset For cases where you want to transform e g pre aggregate before sharing authorized views are ideal While authorized views can be used for managing column level access it may be preferable to leverage Data Catalog as you can easily centralize access knowledge in a single table s metadata and control access through hierarchical taxonomies Similarly leveraging row level access policies instead of authorized views to filter out rows may be preferable in cases where it is easier to manage a single table with multiple access policies instead of multiple authorized views in different places  Monitoring data qualityOne last element of data governance that we ll discuss here is monitoring data quality The quality of your BigQuery data can drop for many different reasons maybe there was a problem in the data source or an error in your transformation pipeline Either way you ll want to know if something is amiss and have a way to inform data consumers at your organization Just like we described earlier you can leverage an orchestration tool like Cloud Composer to create pipelines for running different SQL validation tests Validation tests can be created in a few different ways One option is to leverage open source frameworks like this one that our professional services team put together Using frameworks like these you can declare rules for when validation tests pass or failSimilarly you can use a tool like Dataform which offers the ability to leverage YAML files to declare validation rules Dataform recently came under the Google Cloud umbrella and will be open to new customers soon join the waitlist here Alternatively you could always roll your own solution by programmatically running queries using built in BigQuery functionality like ASSERT if the assertion is not valid then BigQuery will return an error that can inform the next step in your pipelineBased on the outcome of the validation test you can have Composer send you a notification using Slack or other built in notifiers  Finally you can use Data Catalog s API to update a tag that tracks the data quality for the given table  Check out some example code here  With this information added to Data Catalog it becomes searchable by data consumers at your organization so that they can stay informed on the quality of information they use in their analysis  What s next One thing that we didn t mention in this post but is certainly relevant to data governance is ongoing monitoring around usage auditing and access policies We ll be going into more details on this in a few weeks when we cover BigQuery monitoring as a whole  Be sure to keep an eye out for more in this series by following me on LinkedIn and Twitter Related ArticleBigQuery Admin reference guide Query processingBigQuery is capable of some truly impressive feats be it scanning billions of rows based on a regular expression joining large tables Read Article	2021-08-12 16:30:00