投稿時間:2022-05-11 04:43:18 RSSフィード2022-05-11 04:00 分まとめ(45件)

カテゴリー等	サイト名等	記事タイトル・トレンドワード等	リンクURL	頻出ワード・要約等/検索ボリューム	登録日
AWS	AWS Big Data Blog	Analyze Amazon SES events at scale using Amazon Redshift	https://aws.amazon.com/blogs/big-data/analyze-amazon-ses-events-at-scale-using-amazon-redshift/	Analyze Amazon SES events at scale using Amazon RedshiftEmail is one of the most important methods for business communication across many organizations It s also one of the primary methods for many businesses to communicate with their customers With the ever increasing necessity to send emails at scale monitoring and analysis has become a major challenge Amazon Simple Email Service Amazon SES is a cost effective …	2022-05-10 18:20:39
AWS	AWS Machine Learning Blog	Create video subtitles with Amazon Transcribe using this no-code workflow	https://aws.amazon.com/blogs/machine-learning/create-video-subtitles-with-amazon-transcribe-using-this-no-code-workflow/	Create video subtitles with Amazon Transcribe using this no code workflowSubtitle creation on video content poses challenges no matter how big or small the organization To address those challenges Amazon Transcribe has a helpful feature that enables subtitle creation directly within the service There is no machine learning ML or code writing required to get started This post walks you through setting up a no code …	2022-05-10 18:23:57
AWS	AWS	Set up Amazon VPC IP Address Manager for AWS Control Tower | Amazon Web Services	https://www.youtube.com/watch?v=YP69a9WRobI	Set up Amazon VPC IP Address Manager for AWS Control Tower Amazon Web ServicesIn this video you ll see how to set up Amazon VPC IP Address Manager Amazon VPC IPAM for AWS Control Tower With this integration you can set up IPAM in a multi account environment create VPCs while avoiding CIDR conflicts and ensure visibility of IP resources and utilization across your environment For more information on this topic please visit the resources below Subscribe More AWS videos More AWS events videos ABOUT AWSAmazon Web Services AWS is the world s most comprehensive and broadly adopted cloud platform offering over fully featured services from data centers globally Millions of customers ーincluding the fastest growing startups largest enterprises and leading government agencies ーare using AWS to lower costs become more agile and innovate faster AWSControlTower AWS AmazonWebServices CloudComputing	2022-05-10 18:45:26
js	JavaScriptタグが付けられた新着投稿 - Qiita	「観たかった映画が気が付いたら終わってる！」を無くす自動化	https://qiita.com/7ENDO/items/c4b9f1c5368ff5ad5469	ifttt	2022-05-11 03:39:24
js	JavaScriptタグが付けられた新着投稿 - Qiita	価格コムのキープ機能を比較表URLに変換	https://qiita.com/korotom/items/5fb27c5d7c9e7154fd8e	価格コム	2022-05-11 03:19:55
golang	Goタグが付けられた新着投稿 - Qiita	go: json.Unmarshal(data, slice)	https://qiita.com/yyoshiki41/items/edc793751574fdd1a042	cases	2022-05-11 03:25:40
海外TECH	Ars Technica	The Man in Black is back to ruin everyone’s perfect day in Westworld S4 teaser	https://arstechnica.com/?p=1853159	teaserthere	2022-05-10 18:45:52
海外TECH	Ars Technica	Netflix’s ad-supported tier will reportedly roll out sooner than expected	https://arstechnica.com/?p=1853315	netflix	2022-05-10 18:16:08
海外TECH	MakeUseOf	Fedora 36 Woos Developers With Desktop Overhaul, but Will It Please Linus?	https://www.makeuseof.com/fedora-36-released/	desktop	2022-05-10 18:50:23
海外TECH	MakeUseOf	11 Amazing Android Apps That'll Change How You Use Your Phone	https://www.makeuseof.com/tag/10-amazing-apps-will-change-use-android/	basis	2022-05-10 18:45:13
海外TECH	MakeUseOf	Apple Is Right to Discontinue the iPod, Unfortunate as It Is	https://www.makeuseof.com/apple-right-discontinue-ipod/	Apple Is Right to Discontinue the iPod Unfortunate as It IsApple officially discontinued the iPod marking the end for the year old device Unfortunate as it is Apple made the right decision Here s why	2022-05-10 18:34:54
海外TECH	MakeUseOf	Your Phone Is Secretly Always Recording: How to Stop Google From Listening	https://www.makeuseof.com/tag/stop-google-android-listening/	google	2022-05-10 18:15:14
海外TECH	MakeUseOf	5 Ways to Fix the Windows File Explorer When It’s Slow	https://www.makeuseof.com/ways-to-fix-windows-file-explorer-when-its-slow/	slowfile	2022-05-10 18:15:14
海外TECH	DEV Community	Multiplayer Logo Quiz? 😳	https://dev.to/anshsaini/multiplayer-logo-quiz-16hn	Multiplayer Logo Quiz Overview of My SubmissionEver played Logo Quiz Its a game where you see a logo and have to guess which brand owns that logo I ve played it a lot but have always felt that it would be more fun if I could play against my friends That s exactly what PvP Logo Quiz is Submission Category Wacky Wildcards Link to Code ansh saini pvp logo quiz Play Logo Quiz with Friends PVP Logo QuizProject setup guideInstall and run Appwrite Create a project on Appwrite Add a Web Platform to your project Head to API Keys section and create a new appwrite server key with the following permissionsThe command will fail if you have not provided the necessary permissions users read users write collections read collections write attributes read attributes write indexes read indexes write documents read documents write Clone this repositoryCreate a copy of env example file Rename it to env and fill out the variables You can find values for APPWRITE ENDPOINT APPWRITE PROJECT on the Appwrite dashboard itself APPWRITE SERVER API KEY is the server API key secret that we generated Rest of the variables will be generated by the setup script Run yarn setup It will create the required collectoins and provide you with their unique IDs Add them to your env file It will also add some logos to your database so you can get up and… View on GitHub Link to Website PvP Logo Quiz Play Logo Quiz with Friends pvp logo quiz vercel app Additional Resources Info Tech StackAppwrite Backend Next js Frontend Backend DigitalOcean Deployment Demo amp Screenshots Log in Sign up How to play Home Join Room Share Room Code Round in progress Result How it worksAfter creating an account or logging in on PvP Logo Quiz you can create a room and share the room code with your friends The game will start when all players have joined the room Players will be presented with questions logos The first player to get the most questions right wins You only get three seconds to answer the question After the game ends for all players you can see various stats like who won the game and how many questions each player got right and how much time they took overall Future ScopeMatchmaking This will allow players to play with anyone in the world without needing to share an invite code Ability to have more than players in a room Play as guest No need to signup Ability to set number of questions per game Ability to play again in the same room Categories filter Play a game with only a specific category of logos Mobile version so players can play on the go Project Setup GuideI have created a setup script Just run one command and everything will be set up for you See project README for more details Appwrite features usedDatabaseAuthenticationRealtimeUsersAccounts Thoughts about AppwriteThis was my first time using Appwrite The tool is pretty awesome It does so many things behind the scenes that would ve taken ages to set up from scratch Had an amazing time using Appwrite However there are a few things that I struggled with Please please please let me know if there was something that I missed in the documentation which would have helped me with my problems In no way I am trying to criticize Appwrite This is just my personal feedback from using Appwrite to build this project I would ve liked to have an option to list all documents of a collection The limit of seems like a good default But it would be nice to be able to override it beyond Support for JSON fieldHowever if I turn on Array mode for a string attribute I can send a JavaScript Object as payload It gets accepted but doesn t show up properly in Appwrite admin But when you retrieve it using the API the data comes as it is It was a bit unclear to me how to verify the user s identity on my Next js server Reached out on Appwrite discord and someone told me that I needed to generate a JWT for the user and send it to my backend then I can perform actions on behalf of the user I have personally never used this way for authorizing a user It felt a little cumbersome Can t use Query with Users API If I want to fetch data of two users I have to write get queries with their ID I can not use the list API I accidentally exceeded the size limit of a string field I got a very vague error Generic saying Internal Server Error from which I was not able to figure out that I exceeding the size limit I had to do some trial and error to figure out the root cause I was adding attributes to my collection from Appwrite admin I was not able to add many attributes with a high size This is not specified anywhere in the docs that I cannot add multiple high value size fields I got an error saying Attribute limit reached when I tried to add fields with size db getIndex function throws exception even when index exists The API Key does have indexes read and index write permission await database getCollection fc indexes Response of getCollection key name type unique status available attributes name orders ASC getIndex does not work on the same collectionawait database getIndex fc name Response of getIndexAppwriteException Error Index not found at Client call lt projectPath gt node modules node appwrite lib client js at processTicksAndRejections node internal process task queues at async Database getIndex lt projectPath gt node modules node appwrite lib services database js at async x lt projectPath gt setup js code type index not found response message Index not found code type index not found version Final ThoughtsThis was my first time participating solo in a hackathon Thank you so much Dev and Appwrite for this opportunity I learned a lot of new things while building out this project It was a lot of fun Looking forward to using Appwrite in future projects	2022-05-10 18:53:21
海外TECH	DEV Community	Tik Tik ToDo🌺 🍃	https://dev.to/kithminiii/tik-tik-todo-48dc	Tik Tik ToDo Overview of My Submission Created a simple ToDo application with AppWrite and React Native To Run the ProjectNow run the following commands and you should be good to go npm installFor iOS devices you ll need to link the relevant CocoaPods From the ios directory run pod installTo run on Android npx react native run androidTo run on iOS npx react native run ios Submission Category Wacky Wildcards Link to Code kithminiii TikTik ToDo TikTik ToDoTo Run the ProjectNow run the following commands and you should be good to go npm installFor iOS devices you ll need to link the relevant CocoaPods From the ios directory run pod installTo run on Android npx react native run androidTo run on iOS npx react native run ios View on GitHubHappy Coding ​​	2022-05-10 18:38:13
海外TECH	DEV Community	Asynchronous vs Synchronous Callbacks in JS	https://dev.to/melguachun/asynchronous-vs-synchronous-callbacks-in-js-hif	Asynchronous vs Synchronous Callbacks in JSCallbacks are a concept I struggled implementing when I was first introduced to them in bootcamp I understood their purpose but it s inevitable that not everything you learn will immediately stick the first time around Hopefully this dive into callbacks will help you understand their function and purpose What are callbacks in JavaScript The first thing you wanna know is that a callback is a function In JavaScript functions are first class objects meaning they can be stored in a variable object or array passed as an argument to a function returned from a function returned from a functionA callback is a function that is passed inside another function as an argument It is invoked inside the outer function or higher order function that calls back the callback function to complete the task What is a higher order function A higher order function is a function that takes another function as an input returns a function or does both When can callbacks be used Callbacks can be used for asynchronous and synchronous tasks within our code Let s look at an example of a synchronous callback console log Start function isEvenOrOdd num callbck callback function invoked callbck num callback function logicfunction callbck num if num console log num is even else console log num is odd calling outer function isEvenOrOdd callbck console log End To test our code and check out output I use the console in Google Chrome Observe the order of our output to full understand the code flow of this snippet I find using the google chrome console or a code sandbox to observe the order of the output Our output should console log Start tell us if our input is even or odd and console log End The logic that occurs is function isEvenOrOdd is an outer function that takes callbck and num our callback as an argument the callback function is invoked when we call callbck num function callbck num gives us the callback logic when it is invoked isEvenOrOdd callbck calls the outer function it invokes callbck num which calls on the callback logic Notice that in synchronous callbacks callbacks are executed immediately in their order where the output is initiated with the console log Start the functions are established invoked and logic is called followed by the console log End Synchronous callbacks are blocking which means the higher order function doesn t complete its execution until the callback is done executingNow let s look at an asynchronous callback console log Start function isEvenOrOdd num callbck callback function invoked inside Timeout setTimeout gt callbck num callback function logicfunction callbck num if num console log num is even else console log num is odd calling outer function isEvenOrOdd callbck console log End This code snippet is similar to our first code snippet except we incorporate the setTimeout function Let s take a look at our output Our asynchronous output has changed what s going on With this asynchronous callback the callback is enclosed within the setTimeout function By default the setTimeout function is an asynchronous function because it causes the code flow to delay until the task has been completed The asynchronous callback is executed after the execution of the higher order function he asynchronous way to invoke the callbacks The higher order function starts execution console log Start The higher order function completes its execution console log End The callback function executes after seconds is oddAsynchronous callbacks are non blocking which means the higher order function completes its execution without waiting for the callback The higher order function makes sure to execute the callback later on a certain event This is what causes us to see a difference in the order of our output Conclusion Understanding the code flow of callbacks is advantageous because you ll gain a deeper insight into how tasks are being performed manipulating data and debugging I get fuzzy on these concepts still fearing I m not using the right syntax so it always helps to have resources to refresh your knowledge References	2022-05-10 18:37:53
海外TECH	DEV Community	Sliding Window Technique	https://dev.to/user64bit/sliding-window-technique-5aha	Sliding Window Technique IntroductionThe sliding window technique is used for reducing some redundant calculations that slow down the program like it can reduce time complexity from O n to O n with O space complexity Why use it First of all it reduce time coplexity and also with O space complexity so let s understand with Example So we want to find maximum sum of k consecutive integer from array so brute force would look like this for k include lt iostream gt include lt vector gt using namespace std int main vector lt int gt arr int final max for int i i lt arr size i int temp max for int j i j lt i j temp max arr j if temp max gt final max final max temp max cout lt lt final max lt lt endl return But time complexity of the above program is O nk Brute Force ApproachAs per we can see in the above image brute approach checks every pattern of k length here k if you compare the above code with this image you will understand it here k so it won t make too much difference in O n and O nk but what if k is too big then it will impact the running time of the program so what to do now can we implement the above code to O n The answer is YES with the use of the sliding window we can reduce the time complexity of the above code O n How does Sliding Window Works So let s see how sliding window works let me give you simple visual with small array for k because k is too much to write gt first elements gt remove and add gt remove and add gt remove and add gt remove and add gt remove and add gt remove and add gt remove and add let s understand with second example k gt first gt remove and add and so on As we can see in the above image it moves one step at one time so this is how actually it works Let s CodeCode for a maximum sum of k consecutive integer from the array using the sliding window technique include lt iostream gt include lt vector gt using namespace std int sum of k ele vector lt int gt arr int k int sum for int i i lt k i sum arr i return sum int main vector lt int gt arr int k below function will be used only once for finding sum of first k digits int final sum sum of k ele arr k int temp sum final sum for int i k i lt arr size i temp sum temp sum arr i k temp sum temp sum arr i if temp sum gt final sum final sum temp sum cout lt lt final sum lt lt endl return When to use When you are looking for a subrange in a given string or array like highest or smallest value or targeted value it involves data structure which is iterable like string or array when there can be brute force solution with O n or n More Examples Find all anagrams in a string gt SolutionPermutation in string gt SolutionNote Solutions in above questions are just plain code ReferencesStackoverflowmediumThank You	2022-05-10 18:24:24
海外TECH	DEV Community	JavaScript Array and String common methods.	https://dev.to/yadabsd/javascript-array-and-string-common-methods-1oo0	JavaScript Array and String common methods JavaScript is an awesome language with a lot of built in objects Array Date Math and String and their build in methods split sort parseInt which make a developer s life easier Array gt Index Value pairString gt Also can work as Index Value character pairLet s see in details const jsString abc can be considered equivalent for one dimension array only to const jsArray a b c Example jsString indexOf a jsArray indexOf a or jsString ajsArray ajsString jsArray false because array pass reference a b c a b c true loose check Common Methods There are some useful methods those work on arrays as well as in strings in javascript So if you learn them can easily apply for both Those are slice indexOf lastIndexOf concat includes length toString valueOf strOrArray Like Remember A String always carries its value whereas an array always carries reference That s why there are some methods not all of the array that changes the actual value of the variable But for string there are no methods that can change its actual value Array Methods Here listed some methods for array that modifies the original array value directly These methods never can be applied for string splice push pop shift unshift reverse fill These array methods don t modify the array but used only for array map filter reduce String Methods Only for string substr similar to splice substring similar to slice charAt match trim If you want to use string methods for an array you can convert that array to a string and vice versa you can use array methods for string by converting that string to an array You can easily convert each other THANKS	2022-05-10 18:23:45
海外TECH	DEV Community	Debugging Deadlocks and Race Conditions	https://dev.to/codenameone/debugging-deadlocks-and-race-conditions-31eo	Debugging Deadlocks and Race ConditionsThread debugging has the reputation of being one of the most arduous tasks for developers I beg to differ Asynchronous debugging is so much worse It s supposed to solve the problems of threading and to some degree async helps…But it doesn t make debugging simpler I will get into that in the next post In the last two ducklings I talked about threading issues Shai Almog debugagent Duckling the th This week we delve into the world of thread debugging The thread view and all it s deep configuration Next week we ll discuss dealing with race conditions and deadlocks Also and possibly more importantly new shirt CodeNewbie SecondDucklings PM Apr Shai Almog debugagent Duckling the th Debug race conditions and deadlocks easily Method breakpoints make a triumphant return showing that in fact they have another great use case Threading issues are scary at first but they re actually very easy to find amp fix CodeNewbie SecondDucklings PM May Today we ll discuss the process of debugging threading issues dealing with deadlocks and race conditions in the debugger Multithreaded DebuggingDebugging in a multi threaded environment is often perceived as difficult because it s hard to know what s going on You place a breakpoint and a thread that might deadlock is suspended in the background As a result you can no longer reproduce the problem with a debugger Instead of modifying the debugging technique developers blame the tooling That s throwing the baby with the bathwater Debuggers have so many amazing tools to control their environment Once you learn how to master these resources things like deadlock detection will become trivial Thread ViewIf you ve used JetBrains IDEs such as IntelliJ you re probably familiar with the thread combo box that lives above the stack trace panel in the UI This widget lets us toggle the current thread and with it the stack that we re looking at It s a very effective tool but it also provides a very limited view It s very hard to gauge the state of a specific thread by looking at the combo box Additional details such as grouping location etc are unclear when looking at this widget only Luckily most IDEs support a view that s more oriented to heavily threaded apps The downside is that it s a bit more noisy by comparison I guess this is the reason it isn t the default UI But if the process that you re debugging has complex concurrency this might improve your experience noticeably To enable that mode we need to check the Threads option in the IDE in the debugger view This is off by default as the UX is difficult and most developers don t need this for typical apps But when we have a thread heavy application this view becomes a lifesaver The threads effectively become the top level element We can see the stack by expanding a particular thread e g File Watcher in this image Here we have full access to the stack as we had before but we can see all the threads If you have an app with a very high thread count this might be a problem e g with the coming project Loom this might become untenable We can further tune this view through settings this can enable more verbosity and hierarchy There are several interesting capabilities mentioned in the settings dialog but the most interesting one is grouping by thread groups Thread groups let us package a thread as part of a group As a result we can create common behavior for all the threads within E g a single catch handler etc Most threads you ll receive from a pool or a framework would already be grouped logically This means grouping should already be relatively intuitive and easy to grok Debugging a Deadlock SituationWikipedia defines a deadlock as In concurrent computing deadlock is any situation in which no member of some group of entities can proceed because each waits for another member including itself to take action such as sending a message or more commonly releasing a lock Deadlocks are a common problem in multiprocessing systems parallel computing and distributed systems because in these contexts systems often use software or hardware locks to arbitrate shared resources and implement process synchronization This sounds complicated but it isn t too bad Unfortunately if you place a breakpoint the problem will no longer occur so you can t even use the typical debugging tools for a deadlock situation The reason is that a breakpoint typically suspends the entire process when it stops and you won t see the problem occurring I won t talk about deadlock prevention which is a vast subject in its own right The nice thing is that it s pretty easy to debug once you reproduce it with a debugger running All we need to do is press pause in the debugger Once the application is suspended we can review the entries on the list Notice the two entries are stuck on MONITOR threads waiting for a monitor This effectively means they are probably stuck on a synchronized block or some other synchronization API call This might mean nothing but it s pretty easy to review this list and the stack to see the resource they re waiting for If one entry is waiting for the resource held by another That s probably a deadlock risk If both hold resources needed by the other this is a pretty obvious deadlock You can switch between threads and walk the stack In this screenshot the stack is one method deep so it isn t representative of real world cases However this is an easy way to detect such issues Debugging Race ConditionsThe most common issue with multi threading is race conditions Wikipedia defines race conditions as A race condition or race hazard is the condition of an electronics software or other system where the system s substantive behavior is dependent on the sequence or timing of other uncontrollable events It becomes a bug when one or more of the possible behaviors is undesirable This is a far more insidious problem since it s nearly impossible to detect I wrote about it I the past and about debugging it with Lightrun here Derrick also wrote about this in the Lightrun blog but he covered it a bit differently My technique is simpler in my opinion Method Breakpoints Done RightI had some harsh things to say about method breakpoints before They re inefficient and problematic But for this truck we need them They give us the type of control over the breakpoint location we need E g in this method public Set lt PetDTO gt findPetDTOSet Integer vetId List lt Visit gt visits visitRepository findByVetId vetId return visits stream distinct map visit gt Pet current petRepository findById visit getPetId return new PetDTO current getName current getOwner getLastName visitRepository findByPetId current getId collect Collectors toSet If we place a breakpoint on the last line we will miss the functionality of the method But if we place a method breakpoint that tracks method exit it will hit after everything in the method was executed Ideally we could track method entry and exit but then we won t be able to distinguish between them After we create a method breakpoint we set it to not suspend and enable logging We effectively created a tracepoint We can now log that we re exiting the method and log the thread name This will print every exit from the method Method Entry EventWe can do the same thing for method entry but here we can use a regular breakpoint Again we don t suspend the thread and use what is effectively a tracepoint This lets us see if we re a deadlock victim by reviewing the logs If they include two entry logs in a row It might be a race condition Since the threads aren t suspended things shouldn t be disturbed by the debugging process In some cases the output might be so verbose and from a single thread In that case we can use a simple conditional statement to filter out the noise We can also build a poor man s deadlock detector using a similar technique It can give us a sense of shared resource usage so we can properly evaluate deadlock potentials TL DRPossibility of deadlock code makes debugging a process pretty challenging A lock on resources can make things worse and the traditional usage of breakpoints just doesn t work Every time we run into an issue that we suspect of a race or deadlock in multitasking we need to stop Use these techniques to check for occurrences of deadlocks or races Multithreaded debugging isn t as hard as it s often made out to be You might not get errors that point you directly at the line but with the right concurrency control you can narrow things down considerably	2022-05-10 18:14:43
海外TECH	DEV Community	How to use Spark and Pandas to prepare big data	https://dev.to/mage_ai/how-to-use-spark-and-pandas-to-prepare-big-data-4efg	How to use Spark and Pandas to prepare big data source Lucas Films If you want to train machine learning models you may need to prepare your data ahead of time Data preparation can include cleaning your data adding new columns removing columns combining columns grouping rows sorting rows etc Once you write your data preparation code there are a few ways to execute it Download the data onto your local computer and run a script to transform itDownload the data onto a server upload a script and run the script on the remote serverRun some complex transformation on the data from a data warehouse using SQL like languageUse a Spark job with some logic from your script to transform the dataWe ll be sharing how Mage uses option to prepare data for machine learning models PrerequisitesApache Spark is one of the most actively developed open source projects in big data The following code examples require that you have Spark set up and can execute Python code using the PySpark library The examples also require that you have your data in Amazon S Simple Storage Service All this is set up on AWS EMR Elastic MapReduce We ve learned a lot while setting up Spark on AWS EMR While this post will focus on how to use PySpark with Pandas let us know in the comments if you re interested in a future article on how we set up Spark on AWS EMR source Nickelodeon OutlineHow to use PySpark to load data from Amazon SWrite Python code to transform data How to use PySpark to load data from Amazon S source History Channel PySpark is “an interface for Apache Spark in Python It not only allows you to write Spark applications using Python APIs but also provides the PySpark shell for interactively analyzing your data in a distributed environment We store feature sets and training sets data used to store features for machine learning models as CSV files in Amazon S Here are the high level steps in the code Load data from S files we will use CSV comma separated values file format in this example Group the data together by some column s Apply a Python function to each group we will define this function in the next section from pyspark sql import SparkSessiondef load data spark s location spark Spark session s location S bucket name and object prefix return spark read options delimiter header True inferSchema False csv s location with SparkSession builder appName Mage getOrCreate as spark Load data from S files df load data spark s feature sets users profiles v Group data by user id column grouped df groupby user id Apply function named custom transformation function we will define this function later in this article df transformed grouped apply custom transformation function Write Python code to transform dataLet s transform some data source Paramount Pictures Here are the high level steps in the code Define Pandas UDF user defined function Define schemaWrite code logic to be run on grouped data Define Pandas UDF user defined function Pandas is “a fast powerful flexible and easy to use open source data analysis and manipulation tool built on top of the Python programming language Pandas user defined function UDF is built on top of Apache Arrow Pandas UDF improves data performance by allowing developers to scale their workloads and leverage Panda s APIs in Apache Spark Pandas UDF works with Pandas APIs inside the function and works with Apache Arrow to exchange data from pyspark sql functions import pandas udf PandasUDFType pandas udf SCHEMA COMING SOON PandasUDFType GROUPED MAP def custom transformation function df pass Define schemaUsing Pandas UDF requires that we define the schema of the data structure that the custom function returns from pyspark sql functions import pandas udf PandasUDFTypefrom pyspark sql types import IntegerType StringType StructField StructType StructField arguments First argument column name Second argument column type Third argument True if this column can have null values SCHEMA COMING SOON StructType StructField user id IntegerType True StructField name StringType True StructField number of rows IntegerType True pandas udf SCHEMA COMING SOON PandasUDFType GROUPED MAP def custom transformation function df pass Write code logic to be run on grouped dataOnce your data has been grouped your custom code logic can be executed on each group in parallel Notice how the function named custom transformation function returns a Pandas DataFrame with columns user id date and number of rows These columns have their column types explicitly defined in the schema when decorating the function with the pandas udf decorator from pyspark sql functions import pandas udf PandasUDFTypefrom pyspark sql types import IntegerType StringType StructField StructType StructField arguments First argument column name Second argument column type Third argument True if this column can have null values SCHEMA COMING SOON StructType StructField user id IntegerType True StructField name StringType True StructField number of rows IntegerType True pandas udf SCHEMA COMING SOON PandasUDFType GROUPED MAP def custom transformation function df number of rows by date df groupby date size number of rows by date columns date number of rows number of rows by date user id df user id iloc return number of rows by date Putting it all togetherThe last piece of code we add will save the transformed data to S as a CSV file df transformed write option delimiter option header True mode overwrite csv s feature sets users profiles transformed v Here is the final code snippet that combines all the steps together from pyspark sql import SparkSessionfrom pyspark sql functions import pandas udf PandasUDFTypefrom pyspark sql types import IntegerType StringType StructField StructType StructField arguments First argument column name Second argument column type Third argument True if this column can have null values SCHEMA COMING SOON StructType StructField user id IntegerType True StructField name StringType True StructField number of rows IntegerType True pandas udf SCHEMA COMING SOON PandasUDFType GROUPED MAP def custom transformation function df number of rows by date df groupby date size number of rows by date columns date number of rows number of rows by date user id df user id iloc return number of rows by datedef load data spark s location spark Spark session s location S bucket name and object prefix return spark read options delimiter header True inferSchema False csv s location with SparkSession builder appName Mage getOrCreate as spark Load data from S files df load data spark s feature sets users profiles v Group data by user id column grouped df groupby user id Apply function named custom transformation function we will define this function later in this article df transformed grouped apply custom transformation function Save new transformed data to S df transformed write option delimiter option header True mode overwrite csv s feature sets users profiles transformed v Conclusion source Sony Studios This is how we run complex transformations on large amounts of data at Mage using Python and the Pandas library The benefit of this approach is that we can take advantage of Spark s ability to query large amounts of data quickly while using Python and Pandas to perform complex data transformations through functional programming You can use Mage to handle complex data transformations with very little coding We run your logic on our infrastructure so you don t have to worry about setting up Apache Spark writing complex Python code understanding Pandas API setting up data lakes etc You can spend your time focusing on building models in Mage and applying them in your product maximizing growth and revenue for your business	2022-05-10 18:07:08
海外TECH	DEV Community	I created the first interplanetary exchange platform.	https://dev.to/simerca/i-created-the-first-interplanetary-exchange-platform-61d	I created the first interplanetary exchange platform Good morning Yes very ambitious as a title but that s all Let s start with the beginning Some time ago I thought about an optimized commentary for a few points of a rocket launch specification After a few weeks of research I found a working topic regarding payloads in the rocket fairing head The difficulty today for space agencies and private launchers is optimizing this space and managing to fit useful volumes into a cylinder several meters high Short I worked on an optimization topic pitched my concept to someone in the business and presented a useful space optimization algorithm I was then invited to my first astronaut conference Then an idea came to me The first interplanetary e commerce platform based on the Stellar Lumens XLM blockchainMy ambitions were to facilitate exchanges between the various space service providers To do this I imagined a platform operating on the principle of the blockchain Who can any operator on Earth or from a space station or another planet to be able to order products from a partner It could take advantage of the spaces made available by private space companies that sell the payload space in their rocket fairing head A problem quickly arises what do I do when the connection does not exist I imagined an application installed on the client computer micro computer smartphone etc which is practically autonomous and which signs the transactions directly via the client Once the synchronization has been carried out again the signature is authenticated by the rest of the nodes of this chain and therefore is validated or not in the event of cheating by the blockchain This system allows all operators to be sure that the order has been made by the right customer Fraud is only possible if your secret key is stolen I decided to make this work Open Source under MIT license You can find details of my work here And the demo Glad to read you about this crazy idea lt	2022-05-10 18:00:54
Apple	AppleInsider - Frontpage News	Apple developers facing bug preventing them from distributing apps	https://appleinsider.com/articles/22/05/10/apple-developers-facing-bug-preventing-them-from-distributing-apps?utm_medium=rss	Apple developers facing bug preventing them from distributing appsApple developers are apparently running into an issue or bug that is preventing them from validating or distributing apps on the App Store Apple developer bugReports of the issue first surfaced early Tuesday morning when a developer posted about getting an error analyzing app version popup when attempting to distribute their app Read more	2022-05-10 18:56:37
Apple	AppleInsider - Frontpage News	'Lovely Little Farm' children's show comes to Apple TV+ on June 10	https://appleinsider.com/articles/22/05/10/lovely-little-farm-childrens-show-comes-to-apple-tv-on-june-10?utm_medium=rss	x Lovely Little Farm x children x s show comes to Apple TV on June The hybrid live action and animated Apple TV series Lovely Little Farm features talking animals created by Industrial Light and Magic and the first trailer gives a glimpse of them in action Lovely Little Farm coming to Apple TV on June Lovely Little Farm is a new children s show for Apple TV about two sisters raising animals on their family farm The trailer shows the live action series uses a blend of real animals and animated ones Read more	2022-05-10 18:31:28
Apple	AppleInsider - Frontpage News	Apple's iPod is the other device that saved Apple, and set the table for the iPhone	https://appleinsider.com/articles/22/05/10/apples-ipod-is-the-other-device-that-saved-apple-and-set-the-table-for-the-iphone?utm_medium=rss	Apple x s iPod is the other device that saved Apple and set the table for the iPhoneIt was the iMac that saved Apple Computer but it was the iPod that made the company into a consumer technology hit ーand it lasted for an amazing years From launch to discontinuation the iPod lasted days and across that time it went from limited appeal to incredible mass market hit and on to being practically forgotten Originally if only briefly just for Mac users the iPod somehow managed to be become both a status symbol and ubiquitous all at the same time We lured you here today Steve Jobs told assembled journalists on October with the promise of a breakthrough digital device that s not a Mac And that s exactly what we re going to do Read more	2022-05-10 18:30:30
Apple	AppleInsider - Frontpage News	Top B&H deals of the week: save up to $1,200 on Apple, digital cameras, TVs & more	https://appleinsider.com/articles/22/05/10/top-bh-deals-of-the-week-save-up-to-1200-on-apple-digital-cameras-tvs-more?utm_medium=rss	Top B amp H deals of the week save up to on Apple digital cameras TVs amp moreSpring has sprung and that means retailers like B amp H Photo are running a variety of sales on Apple electronics TVs and other tech goodies The best deals from B amp H Photo this week include discounts on Apple products and other tech goodies If you re looking for a new computer or particular tech product it s well worth seeing what B amp H Photo has to offer during its latest series of sales Read more	2022-05-10 18:22:34
海外TECH	Engadget	YouTube TV adds Spanish-only plans	https://www.engadget.com/google-youtube-spanish-language-plan-184520243.html?src=rss	YouTube TV adds Spanish only plansYouTube TV is adding two new Spanish language plans Google announced today The first of those the aptly named Spanish Plan will offer customers access to more than Spanish language channels including Univision and ESPN Deportes Google has priced the package at per month for the first six months and per month thereafter New YouTube TV customers can subscribe to the plan without paying for the platform s base per month package Now introducing our new Spanish Plan Spanish channels of live sports news telenovelas amp more at mo for the first months mo after Tag the first person who needs to know ️pic twitter com ctIZgpufSーYouTube TV YouTubeTV May For those who want to add to their existing subscription Google has also introduced a Spanish Plus add on Priced at per month for the first six months and then thereafter the package comes with access to more than Spanish language channels including Cine Latino and Discovery en Español Google has increasingly looked to Hispanic customers to grow YouTube TV s subscriber base Last year the company made Univison UniMás and Galavisión available to subscribers at “no extra cost Like many other streaming TV platforms YouTube TV has repeatedly increased the price of its service to offset the cost of rising content deals	2022-05-10 18:45:20
海外TECH	Engadget	Elon Musk says he would ‘reverse’ Donald Trump’s Twitter ban	https://www.engadget.com/elon-musk-says-he-would-reverse-donald-trump-twitter-ban-182237750.html?src=rss	Elon Musk says he would reverse Donald Trump s Twitter banElon Musk has finally confirmed what many have long suspected that he would allow Donald Trump back on Twitter In an interview with The Financial Times Musk said that he would reinstate Trump when his deal to acquire Twitter closes “I guess the answer is that I would reverse the perma ban Musk said in response to a question about whether he would allow the former president back on the platform “Obviously I don t own Twitter yet So this is not a thing that will definitely happen because what if I don t own Twitter Though Musk has long made it clear he disagrees with Twitter s decision to ban Trump it s the first time he has explicitly said he would “reverse it “I think bans just fundamentally undermine trust in Twitter as a town square where everyone can voice their opinion Musk said He added that permanent bans should be reserved for “accounts that are bots or spam scam accounts Whether Musk would bring back Trump has been a major question and the company s own executives have told employees they don t know what direction Musk will take the company Employees have been concerned that Musk could reverse many of the company s policies around trust and safety and reverse progress they have made in combatting online abuse and misinformation Musk said that accounts that are quot destructive to the world quot could still be punished with temporary suspensions or having individual tweets deleted But he said that he and Twitter co founder Jack Dorsey believe that permanent bans “should be extremely rare Twitter didn t immediately respond to a request for comment The company banned Trump in the wake of the January th insurrection saying that the president had broken its rules around incitement of violence Then CEO Dorsey said at the time he believed permanent bans could set a quot dangerous quot precedent but that the company had made the right decision in the face of quot extraordinary and untenable circumstance quot nbsp Musk called the ban a quot morally bad decision quot and cited the Trump backed Truth Social as proof Twitter s ban did not have the intended effect The former president has stated that he would not rejoin Twitter even if his ban is lifted nbsp quot He is now going to be on Truth Social as will a large part of the right in the in the United States quot Musk said quot Banning Trump from Twitter didn t end Trump s voice It will amplify it among the right and that is why it is morally wrong and flat out stupid quot	2022-05-10 18:22:37
Cisco	Cisco Blog	Wholesale Carriers Benefit from the Cisco/Qwilt Solution	https://blogs.cisco.com/sp/wholesale-carriers-benefit-from-the-cisco-qwilt-solution	Wholesale Carriers Benefit from the Cisco Qwilt SolutionThe Cisco Qwilt open caching solution is a CDN use case that can be taken full advantage of by both Communication Service Providers as well as Wholesale Carriers	2022-05-10 18:01:28
海外TECH	WIRED	How YouTube Can Rewrite the Past and Shape an Election	https://www.wired.com/story/youtube-philippines-election	historical	2022-05-10 18:02:57
海外ニュース	Japan Times latest articles	Elon Musk says he would reverse Twitter’s permanent ban on Trump	https://www.japantimes.co.jp/news/2022/05/11/business/musk-trump-twitter-ban/	Elon Musk says he would reverse Twitter s permanent ban on TrumpMusk who reached an agreement to acquire Twitter for roughly billion late last month has said that he believes the company has overstepped on	2022-05-11 03:42:03
ニュース	BBC News - Home	Queen's Speech: We can't spend our way out of trouble, says PM	https://www.bbc.co.uk/news/uk-politics-61391974?at_medium=RSS&at_campaign=KARANGA	families	2022-05-10 18:17:53
ニュース	BBC News - Home	Fifa: EA Sports to break away from football body	https://www.bbc.co.uk/news/entertainment-arts-61383672?at_medium=RSS&at_campaign=KARANGA	bodyelectronic	2022-05-10 18:50:58
ニュース	BBC News - Home	Elon Musk would reverse Donald Trump's Twitter ban	https://www.bbc.co.uk/news/business-61399483?at_medium=RSS&at_campaign=KARANGA	president	2022-05-10 18:38:29
ニュース	BBC News - Home	Apple discontinues the iPod touch after 21 years	https://www.bbc.co.uk/news/technology-61401626?at_medium=RSS&at_campaign=KARANGA	yearsthe	2022-05-10 18:16:27
ビジネス	ダイヤモンド・オンライン - 新着記事	中東で油田を開発したアラビア石油・山下太郎が説く「日本人の手で石油を掘る重要性」 - The Legend Interview不朽	https://diamond.jp/articles/-/302893	ところで、かつてサウジアラビアとクウェートに石油利権を持ち、実際に油田開発に成功した日本企業がある。	2022-05-11 03:50:00
ビジネス	ダイヤモンド・オンライン - 新着記事	銀行の自己資本比率規制「バーゼル3」、内部格付け手法の見直しを解説 - きんざいOnline	https://diamond.jp/articles/-/302906	online	2022-05-11 03:45:00
ビジネス	ダイヤモンド・オンライン - 新着記事	スタバの新戦略、「第3の場所」の魅力失う恐れも - WSJ PickUp	https://diamond.jp/articles/-/302976	wsjpickup	2022-05-11 03:40:00
ビジネス	ダイヤモンド・オンライン - 新着記事	コロナ快走後のIT業界、「息切れ」の深刻度は - WSJ PickUp	https://diamond.jp/articles/-/302977	wsjpickup	2022-05-11 03:35:00
ビジネス	ダイヤモンド・オンライン - 新着記事	台湾有事に備える保険、ウクライナ侵攻で需要増 - WSJ PickUp	https://diamond.jp/articles/-/302978	wsjpickup	2022-05-11 03:30:00
ビジネス	ダイヤモンド・オンライン - 新着記事	日本企業が想定外だった「ASEANリスク」を本気で考えざるを得ない理由 - ＤＯＬ特別レポート	https://diamond.jp/articles/-/302942	asean	2022-05-11 03:25:00
ビジネス	ダイヤモンド・オンライン - 新着記事	テレワーク・出社混合の若手育成計画を詳解、「人事必読」外資流ノウハウ - News&Analysis	https://diamond.jp/articles/-/302901	テレワーク・出社混合の若手育成計画を詳解、「人事必読」外資流ノウハウNewsampampAnalysisテレワークが当たり前になった一方で、出社比率を上げる企業も増えてきました。	2022-05-11 03:20:00
ビジネス	ダイヤモンド・オンライン - 新着記事	役所に残る「メールよりFAX」信仰、時代錯誤な住民の行政批判もDXの壁に - 酒井真弓のDX最前線	https://diamond.jp/articles/-/302975	時代錯誤	2022-05-11 03:15:00
ビジネス	ダイヤモンド・オンライン - 新着記事	日本人が間違えやすいカタカナ英語、「仕事」の英単語を使い分けられる？ - ニュース3面鏡	https://diamond.jp/articles/-/302962	使い分け	2022-05-11 03:10:00
ビジネス	ダイヤモンド・オンライン - 新着記事	ロッテ重光武雄、渾身の資本防衛策が二男の「クーデター」を生んでしまったという皮肉 - ロッテを奪われた男・重光武雄～なぜ事業承継に失敗したのか	https://diamond.jp/articles/-/303002	ロッテ重光武雄、渾身の資本防衛策が二男の「クーデター」を生んでしまったという皮肉ロッテを奪われた男・重光武雄なぜ事業承継に失敗したのかロッテグループの創業者であり、オーナーだったはずの武雄は二男の昭夫によっていとも簡単に、すべての地位を剥奪され、追放されてしまう。	2022-05-11 03:07:00
ビジネス	ダイヤモンド・オンライン - 新着記事	「ブラック校則」で下着の色を男性教師が確認、防寒着NG…ひどすぎる実態 - ニュース3面鏡	https://diamond.jp/articles/-/302250		2022-05-11 03:05:00