投稿時間:2022-04-17 02:14:40 RSSフィード2022-04-17 02:00 分まとめ(15件)

カテゴリー等	サイト名等	記事タイトル・トレンドワード等	リンクURL	頻出ワード・要約等/検索ボリューム	登録日
python	Pythonタグが付けられた新着投稿 - Qiita	【AtCoder】PythonでABC248のA, B, C, D, Eを解く	https://qiita.com/yassu/items/ae83817838c2c25e5a1d	rangecountquery	2022-04-17 01:01:56
Docker	dockerタグが付けられた新着投稿 - Qiita	error while creating mount source path '~~path~~'でやったこと	https://qiita.com/kumokumo/items/2e0ed7c5690c28d70624	mountsourcepathsomepath	2022-04-17 01:19:59
golang	Goタグが付けられた新着投稿 - Qiita	go言語のインポートについて（「Go言語でつくるインタプリタ」の1.3字句解析器のエラー対処）	https://qiita.com/OrukRed/items/a7d9b4a532583e572d0e	字句解析器	2022-04-17 01:41:45
海外TECH	Ars Technica	Steph Curry stars in new Nope teaser airing during NBA playoff games	https://arstechnica.com/?p=1848606	curry	2022-04-16 16:54:50
海外TECH	MakeUseOf	The 5 Best Privacy Add-Ons for Microsoft Edge	https://www.makeuseof.com/best-microsoft-edge-privacy-add-ons/	microsoft	2022-04-16 16:30:14
海外TECH	MakeUseOf	How to Set Up Your Windows PC for Seniors	https://www.makeuseof.com/windows-pc-seniors-set-up/	windows	2022-04-16 16:15:14
海外TECH	DEV Community	Reliability Restaurant – How to approach software reliability as a mindset	https://dev.to/anttispitkanen/reliability-restaurant-how-to-approach-software-reliability-as-a-mindset-3656	Reliability Restaurant How to approach software reliability as a mindsetPhotograph by Life Of Pix from Pexels What is software reliability ScienceDirect defines software reliability as follows Software reliability is the probability of failure free operation of a computer program for a specified period in a specified environment Reliability is a customer oriented view of software quality It relates to operation rather than design of the program and hence it is dynamic rather than static Another closely related term is resiliency as defined by Microsoft Reliability is the outcome cloud service providers strive for it s the result Resiliency is the ability of a service to withstand certain types of failure and yet remain functional from the customer perspective In other words reliability is the outcome and resilience is the way you achieve the outcome So for the sake of simplicity let s discuss reliability without specifying resiliency explicitly we can assume it to be the other side of the coin implicitly What about a restaurant Reliability like security has much less to do with using the correct frameworks and libraries than with asking the right questions and having the right mindset For the sake of discussing the principles rather than specifics let s use a metaphor of a pancake restaurant rather than some computer program Everyone knows what a restaurant does and can imagine what failure free operation looks like for one The same mindset can then be applied to software systems Why care about reliability Looking at the definition of reliability it seems like the thing to do achieve failure free operation is to avoid failure at any cost But anyone who s ever written software knows that even with careful writing reviewing and testing bugs still manage to find their way to production regularly More generally speaking unexpected things and mistakes of different kinds are always bound to happen regardless of the activity especially at scale It s safe to assume that our pancake restaurant will run into some problems and failures in its operations no matter the amount of planning and trying to avoid them If failure is inevitable why talk about failure free operation then The definition of failure matters Reliability is all about constructing the bigger system so that individual failures can be tolerated and recovered from without failure to the overall system From a business owner s perspective you want your system to be able to provide business value even if some component fails instead of the whole system being rendered useless If something fails you want to be able to work around it In the restaurant if someone drops and breaks a plate a failure you don t want to have to close the restaurant as a result Instead you want to be able to tolerate and work around such failures keep the doors open serve paying customers and have your business running Likewise you don t want every error to take your whole software system down Defining failure free operation for a pancake restaurantThe properties of an operational restaurant are pretty much obvious but worth listing explicitly for the sake of the example So let s say that the restaurant must be able to Take in customers and seat them in tablesTake orders from themPrepare the dishes as orderedServe the food to the customersCharge money from the customersLet s keep it simple and limit the requirements to these five things Looks kind of like a user story doesn t it Now it s fairly easy to map out the best case scenario or happy path A customer walks in to the restaurant and sits at a table The waiter goes and gets their order pancakes and takes that info to the kitchenThe chef prepares a plate of delicious pancakes all the ingredients are there readily available ‍The waiter takes the pancakes to the customer to eat After dining the customer orders the check pays for the food and leaves happy To serve the one customer only one waiter and one chef are needed so let s say there are no more employees around All seems good right Diverting from the happy path remaining operational under unexpected circumstancesThe happy path looks simple enough But what if something different happens Let s say that Elon Musk tweets about the best pancakes he ever had at your restaurant the customer was Elon Musk the whole time bet you didn t see that coming and the next day people want to come try those pancakes Now what if All the people rush in at the same time Someone orders something that the chef doesn t know how to make Some ingredient runs out in the kitchen Some ingredient deliveries get cancelled due to the supplier The waiter or chef is out sick People try to pay but your card reader device isn t working It s hard to imagine the happy path working under these circumstances so let s look at the things one by one Overflow of customersThe restaurant can hardly operate in a satisfactory way if people march in and try to seat themselves uncontrollably especially when the amount of people greatly exceeds the restaurant s seating capacity What can be done A simple reliability mechanism that the restaurant can apply is instructing the customers to wait to be seated and queue for that This way the customers wait at the door or lobby in an ordered queue based on the order they arrived and the restaurant in this example probably the waiter controls the way and rate in which that they get seated in tables Some delay can be added to make time for cleaning the tables after the previous customers resulting in much better customer experience This is analogous to throttling and queuing in software If there are simply too many customers coming in for the restaurant to handle the waiter can turn them away at the door and ask them to come back later Not the best customer experience for sure but this kind of expectation management is better for everyone than seating them and not being able to serve them By only taking in the amount of customers the restaurant can handle we can make sure to remain operational for those customers that we do take in This is analogous to rate limiting in software don t take in more requests than you can properly handle How to avoid having to turn people away at the door Enter table reservations This way the restaurant can control the amount and rate of customers ahead of time and expectation management is even better for the customers as it s nicer to be turned away at the phone rather than after already walking to the restaurant This is analogous to planned capacity or scheduling in software Note that it is possible and maybe even desirable to apply these same mechanisms also later on in the flow For example the kitchen can take in orders as a queue and work on the dishes one at a time to control the work pressure for the chef In general however it s good to apply these mechanisms as early in the flow as possible It s better to have the whole restaurant serve a manageable amount of customers at once than to have the dining room overflowing from hungry people while just the kitchen can work at a controlled pace Unexpected ordersNow that we have limited the amount of customers to serve at any given time all seems to be good But what if someone orders waffles instead of pancakes and our kitchen lacks the required gear ingredients or expertise to make waffles You can imagine what would happen to the kitchen s ability to produce dishes if the chef needs to start by looking up a recipe and going out to buy the needed ingredients and a waffle iron Contrast that to having the chef know the recipe already and having all that s needed for cooking it readily available in the kitchen So what s the reliability mechanism that can be applied here Having a menu and only serving dishes from the menu This is a way of managing the expectations for the chef and kitchen It s much easier to prepare with the necessary knowhow devices and ingredients when the dishes that can be ordered are listed ahead of time and no surprises can happen The software analogy here is a bit more vague but think allowlisting inputs to limit what kind of requests are taken in for handling Ingredients running outThe amount of customers and unexpected orders are handled but what happens if the kitchen runs out of flour during opening hours Surely that will limit the capability of making pancakes In the happy path example it might be good enough to go to the supermarket next door to buy the ingredients for a single serving of pancakes only when the order arrives the equivalent of lazy evaluation just in time computation or on demand fetching in software Let s make an assumption that this won t take more than a couple of minutes Not that this sounds like a reasonable thing for a restaurant s kitchen but it would be doable It starts to get less tolerable when the amount of customers and the expected throughput of pancakes in the kitchen is expected to grow At full service it might be catastrophic to the restaurant s operation remember being able to serve pancakes to paying customers in reasonable time to run out of an important ingredient So it makes sense to stock up with all the necessary ingredients in preparation for a day s work The software analogy is again not accurate but think caching Instead of having to repeatedly read data from a database or API or perform a repeated intensive computation caching the results in memory allows much faster serving of responses back downstream Supplier cancellingDue to various reasons such as limited storage space and ingredients expiring it s not possible to stock up indefinitely So regardless of preparations new ingredients are continuously needed maybe even during the opening hours What happens if the supplier calls to say that their delivery truck has a flat tire and they cannot deliver the much needed flour The kitchen will run out in one hour if new flour isn t received somehow To solve this problem the restaurant needs some kind of backup Maybe the chef will make a run to the supermarket next door as some delay in the kitchen is better than not being able to serve pancakes at all Or maybe the restaurant has a backup supplier that they can call in a situation like this The software analogy is backups replication and failover all parts of the wider concept of high availability HA If you only have one database instance and it shuts down your service cannot remain operational But if you have a system that can automatically failover to a working database instance the system can remain operational with minimal downtime Note that you would need data replication for this to be feasible Likewise you can run multiple parallel instances of your stateless application servers and balance the load between those so that one application server failing won t disrupt the system s overall operation Waiter or chef being out sickIf there s just one waiter and one chef either of them being out sick would be a major blocker for the restaurant s operation Without a waiter no pancakes can be served to customers and without a chef there are no pancakes to serve We can safely identify both the waiter and the chef as a single point of failure SPOF for the restaurant s operation The obvious solution is to have multiple waiters and chefs working or active backups for them that you can call in to work when needed think high availability We ll look at the financial implications at the end of this post but it s obvious that there s a limit to how many waiters and chefs the restaurant can afford to have in rotation So let s look at other potential solutions as well If the waiter is out sick maybe the restaurant can try a form of self service instead Have the customers come make orders at the kitchen counter and fetch the dishes from the kitchen when they are done instead of waiting to be served The payments can also be handled at the kitchen counter Looks a little like how most fast food restaurants operate If the chef is out sick maybe the restaurant can serve pre made pancakes that only need heating that can be handled by the waiter in a microwave Definitely not as good as fresh ones but perhaps better than nothing Both of these examples are naive and not something that would make for a good experience if the customers expect to be served fresh pancakes directly to the tables In reality both of these examples are also such that they would require planning and implementation ahead of time to be feasible But they both represent a way of working that s somehow less than the optimal but still working to some extent instead of being completely shut down The software analogy is partial graceful failure It depends on the case to what extent this is possible to implement but in general you want to limit the blast radius of any single point of failure so that a failure won t render the whole system unusable If nothing else you probably want to catch errors and show the users something human readable instead of blank screens or stack traces Or maybe you want to show cached data if fresh data cannot be fetched Or if your website cannot load images maybe you want to have some placeholders instead Payments not workingEverything has gone well for the customers the mood was on point service top notch and pancakes absolutely delicious All that s left is paying the bill but alas the card reader doesn t work While the customers might enjoy getting their food for free that would certainly be undesirable for the restaurant Remember the happy path requires being able to charge money from the customers otherwise the restaurant is hardly feasible business As for the actual card reader device malfunctioning maybe you can have a backup reader available Or if the problem is batteries running out at least have spare batteries at hand High availability again But maybe the problem is at the card processor s end meaning that no amount of actions from the restaurant s side could avoid or solve that failure In such a case you most likely want to have some backup way of collecting payments from the customers Maybe you can take payments in cash or even open a tab if you trust the customers to come back later to pay the bill In this example the card processor is a dependency and specifically an external dependency External meaning that you cannot affect the reliability of the dependency itself only how you build your system to depend on it In this case you can mitigate the risk of the dependency failing by introducing backup mechanisms cash or tabs The same concepts of high availability apply as described earlier Prepare for the unexpected by expecting itWhen something unexpected happens the restaurant staff can creatively improvise and adapt to the situation work around it and keep the business running Computers however despite some promising developments tend to be less capable of creativity or improvising and more capable of following precise orders given as the source code and other artefacts by the programmers So the burden of creativity must remain mostly on the programmer In order for your software to be able to handle unexpected situations you have to build it as such And in order to be able to build in the necessary reliability mechanisms for different situations you must be able to creatively think about the different situations that might occur Instead of trying to avoid failures altogether build your system to be able to tolerate failure and recover from them Avoid single points of failure reduce the blast radius of any potential pieces failing and don t forget high availability where it makes sense Keep the happy path in mind I hope that the restaurant metaphor can help to think about reliability holistically when designing your software systems Reliability is a spectrum SRE makes sense of the spectrumReliability is not a binary thing that you either have or don t but rather a spectrum It s largely a business decision to decide how reliable your system should be A pacemaker can t really afford to fail or a plane in flight system for that matter but most systems don t work under such high pressure Maybe it makes more financial sense to occasionally close our restaurant for a day than to keep extra staff on the payroll Or if there is high demand for pancakes maybe it makes sense to expand to a second location creating a high availability setup for the whole business Now if the other restaurant needs to be closed due to say a fire in the kitchen the business as a whole can still operate as the other restaurant can serve customers The Site Reliability Engineering SRE methodology aims to make reliability an explicit goal for the business and puts emphasis on the reliable enough part When explicit Service Level Objectives SLOs are set those can be used to drive the development efforts in a way that s reliable enough for the business but not more than that As discussed it doesn t make sense to invest more money in reliability than it well makes sense For our pancake restaurant we defined earlier that we want to be able to serve our customers their pancakes in a reasonable time Let s make reasonable time explicit and set an SLO we want of our customers to receive their pancakes within minutes of ordering Let s track this by measuring the time from ordering to the pancakes being served for every order If we are lagging badly behind our target we need to do something about it or re evaluate the target itself but let s assume the target is good for now We can speed up the orders in many ways for example by having two chefs and two waiters working at all times If this takes our SLO to the desired level we know that this is good enough By employing even more chefs and waiters we might be able to make the orders even faster but since we have reached our reliability target and assuming our target is sensible it doesn t make sense to invest more Wrapping upBuilding a reliable system is not about following a step by step guide but rather about being able to think creatively about your system and the different conditions it can face There s no clear generalizable one to one mapping between the examples given here and your system but with creative thinking you should be able to identify the chefs and waiters or your system and more The examples also serve as direction on how to approach applying the various kinds of reliability mechanisms Hopefully this helps you build your system as reliable and resilient	2022-04-16 16:52:21
海外TECH	DEV Community	Why 0.1 + 0.2 !== 0.3?	https://dev.to/noriller/why-01-02-03-1g3j	Why The short answer is because binary The longer answer is that computers store data as binary and while it s ok for integers it gets a little complicated for fractions You probably know this but in bits you would have Number base But with fractions The floating point you don t actually “store you tell the computer how many bits are for the integer part in this case and how many are for the fraction part in this case But since you have smaller and smaller parts some numbers aren t direct representations and you need an infinite or close to that number of bits that when added will give the number you want So the reason is that when you finish up with all the roundings and depending on how many bits you ve used you ll get a different result than you might expect It will be really close but not perfect Why not use base then Because binary Even if you can store it in base say with strings you then have to do the operations as you would in base but with base as an intermediary and that will chew on the performance Then again there must have other ways to do it that I have no idea about Little exerciseHow would you sum two numbers in base or how do you do it with pen and paper check the signs a plus added with a minus is a subtraction but two minuses are an addition check for fractions“align the numberssum each numberif one number overflows that is more than then you carry for the nextgo back and be sure if you re also adding the carried numbersprint the result All that just for sum that is the easiest one Enter noriller manual calculatorI wanted a “simple project just to get my mind off things and wouldn t you know it s a lot more complicated than I ve anticipated noriller manual calculatorSum subtraction multiplication and division All that is there The really tricky one is division even now in an infinite division the more digits you want it to have the growth is exponential k digits took almost minutes while digits take “only around ms Check a few runs I made here And or try yourself with these tests follow the README on how to Bugs and unexpected things aside at least you can trust those digits should be the same you would get if you calculated it by hand So this was my shameless promotion Check it out and probably forget it until you find yourself really wanting that or you want to use numbers that won t wrap and round to weird numbers like becoming or even n with the BigInt wrapper and of course if you need a division with thousands of digits albeit with some time cost Cover Photo by Michal Matlon on Unsplash	2022-04-16 16:31:23
海外TECH	DEV Community	Storing Growing Files Using Azure Blob Storage and Append Blobs	https://dev.to/david_whitney/storing-growing-files-using-azure-blob-storage-and-append-blobs-49aa	Storing Growing Files Using Azure Blob Storage and Append BlobsThere are several categories of problems that require data to be append only sequentially stored and able to expand to arbitrary sizes You d need this type of “append only file for building out a logging platform or building the data storage backend for an Event Sourcing system In distributed systems where there can be multiple writers and often when your files are stored in some cloud provider the “traditional approach to managing these kinds of data structures often don t work well You could acquire a lock download the file append to it and re upload but this will take an increasing amount of time as your files grow or you could use a database system that implements distributed locking and queuing which is often more expensive than just manipulating raw files Azure blob storage offers Append Blobs which go some way to solving this problem but we ll need to write some code around the storage to help us read the data once it s written What is an Append Blob An Append Blob is one of the blob types you can create in Azure Blob Storage Azure s general purpose file storage Append Blobs as the name indicates can only be appended to you append blocks to append blobs From Azure s Blob Storage documentation An append blob is composed of blocks and is optimized for append operations When you modify an append blob blocks are added to the end of the blob only via the Append Block operation Updating or deleting of existing blocks is not supported Unlike a block blob an append blob does not expose its block IDs Each block in an append blob can be a different size up to a maximum of MiB and an append blob can include up to blocks The maximum size of an append blob is therefore slightly more than GiB MiB X blocks There are clearly some constraints there that we must be mindful of using this technique Our total file size must be less than GiBEach block can be no bigger than MiBThere s a hard cap on blocks so if our block size is less than MiB the maximum size of our file will be less Still even with small blocks blocks should give us a lot of space for entire categories of application storage The Blob Storage SDKs allow us to read our stored files as one contiguous file or read ranges of bytes from any given offset in that file Interestingly we can t read the file block by block only by byte offset and this poses and interesting problem if we store data which has any kind of data format that isn t just plain text e g JSON XML literally any data format and we want to seek through our file there is no way we can ensure we read valid data from our stored file even if it was written as a valid block when first saved Possible Solutions to the Read ProblemIt s no good having data if you can t meaningfully read it especially when we re using a storage mechanism specifically optimised for storing large files There are a few things we could try to make reading from our Append Only Blocks easier We could maintain an index of byte offset to block numbersWe could pad our data to make sure block sizes were always consistentWe could devise a read strategy that understands it can read partial or seemingly malformed dataThe first solution maintaining a distinct index may seem appealing at first but it takes a non trivial amount of effort to maintain that index and make sure that it s keep both in track and up to data with our blob files This introduces the possibility of a category of errors where those files drift apart and we may well be in a situation where data appears to get lost even if it s in the original data file because our index loses track of it The second solution is the “easiest as it gives us a fixed block size that we can use to page back through our blocks but storing our data becomes needlessly more expensive Which really leaves us with our final option making sure the code that reads arbitrary data from our file understands how to interpret malformed data and interpret where the original write blocks were Scenario A Chat ApplicationOne of the more obvious examples of infinite append only logs are chat applications where messages arrive in a forwards only sequence contain metadata and to add a little bit of spice must be read tail first to be useful to their consumers We ll use this example to work through a solution but a chat log could be an event log or a list of business events and metadata or really anything at all that happens in a linear fashion over time We ll design our fictional chat application like this A Blob will be created for every chat channel We ll accept that a maximum of messages can be present in a channel In a real world application we d create a subsequent blob once we hit this limit We ll accept that a single message can t be more than MiB in size because that d be silly In fact we re going to limit every single chat message to be a maximum of KiB this means that we know that we ll never exceed the maximum block size and each block will only contain a single chat message Each chat message will be written as its own distinct block including its metadata Our messages will be stored as JSON so we can also embed the sender timestamps and other metadata in the individual messages Our message could look something like this “senderId foo “messageId some guid “timestamp T Z “messageType chat message “data “text hello This is a mundane and predictable data structure for our message data Each of our blocks will contain data that looks roughly like this There is a side effect of us using structured data for our messages like this which is that if we read the entire file from the start it would not be valid JSON at all It would be a text file with some JSON items inside of it but it wouldn t be “a valid JSON array of messages there s no surrounding square bracket array declaration and there are no separators between entries in the file Because we re not ever going to load our whole file into memory at once and because our file isn t actually valid JSON we re going to need to do something to indicate our individual message boundaries so we can parse the file later It d be really nice if we could just use an open curly bracket and that d just be fine but there s no guarantee that we won t embed complicated object structures in our messages at a later point that might break our parsing Making our saved messages work like a chat historyChat applications are interesting as an example of this pattern because while the data is always append only and written linearly it s always read in reverse from the tail of the file first We ll start with the easy problem adding data to our file The code and examples here will exist outside of an application as a whole but we ll be using TypeScript and the azure storage blob Blob Storage client throughout and you can presume this code is running in a modern Node environment the samples here have been executed in Azure Functions Writing to our file thankfully is easy We re going to generate a Blob filename from our channel name suffixing it with “ json which is a lie it s “mostly JSON but it ll do and we re going to add a separator character to the start of our blob Once we have our filename we ll prefix a serialized version of our message object with our separator character create an Append Blob Client and call appendBlob with our serialized data import BlobServiceClient from azure storage blob export default async function channelName string message any const fileName channelName json const separator String fromCharCode const data separator JSON stringify message const blobServiceClient BlobServiceClient fromConnectionString process env AZURE STORAGE CONNECTION STRING const containerName process env ARCHIVE CONTAINER archive const containerClient blobServiceClient getContainerClient containerName await containerClient createIfNotExists const blockBlobClient containerClient getAppendBlobClient fileName await blockBlobClient createIfNotExists await blockBlobClient appendBlock data data length This is exceptionally simple code and it looks almost like any “Hello World Azure Blob Storage example you could thing of The interesting thing we re doing in here is using a separator character to indicate the start of our block What is our Separator Character It s a nothing So the wonderful thing about ASCII as a standard is that it has a bunch of control characters that exist from a different era to do things like send control codes to printers in the s and they ve been enshrined in the standard ever since What this means is there s a whole raft of character that exist as control codes that you never see never use and are almost fathomably unlikely to occur in your general data structures ASCII is my one According to ASCII the th character code which you can see in the above sample being loaded using String fromCharCode is “RECORD SEPARATOR C and C control codes Wikipedia “Can be used as delimiters to mark fields of data structures If used for hierarchical levels US is the lowest level dividing plain text data items while Record Separator Group Separator and File Separator are of increasing level to divide groups made up of items of the level beneath it That ll do Let s use it By prefixing each of our stored blocks with this invisible separator character we know that when it comes time to read our file we can identify where we ve appended blocks and re convert our “kind of JSON file into a real array of JSON objects Whilst these odd control codes from the s aren t exactly seen every day this is a legitimate use for them and we re not doing anything unnatural or strange here with our data Reading the Chat HistoryWe re not going to go into detail of the web applications and APIs that d be required above all of this to present a chat history to the user but I want to explore how we can read our Append Only Blocks into memory in a way that our application can make sense of it The Azure Blob Client allows us to return the metadata for our stored file public async sizeof fileName string Promise lt number gt const blockBlobClient await this blobClientFor fileName const metadata await blockBlobClient getProperties return metadata contentLength private async blobClientFor fileName string Promise lt AppendBlobClient gt this ensureStorageExists const blockBlobClient this containerClient getAppendBlobClient fileName await blockBlobClient createIfNotExists return blockBlobClient private async ensureStorageExists TODO Only run this once await this containerClient createIfNotExists It s been exposed as a sizeof function in the sample code above by calling getProperties on a blobClient you can get the total content length of a file Reading our whole file is easy enough but we re almost never going to want to do that sort of downloading the file for backups We can read the whole file like this public async get fileName string offset number count number Promise lt Buffer gt const blockBlobClient await this blobClientFor fileName return await blockBlobClient downloadToBuffer offset count If we pass as our offset and the content length as our count we ll download our entire file into memory This is a terrible idea because that file might be GiB in size and nobody wants those cloud vendor bills Instead of loading the whole file we re going to use this same function to parse backwards through our file to find the last batch of messages to display to our users in the chat app RememberWe know our messages are a maximum of KiB in sizeWe know our blocks can store up to MiB of dataWe know the records in our file are split up by Record Separator charactersWhat we re going to do is read chunks of our file from the very last byte backwards in batches of KiB to get our chat history Worst case scenario We might just get one message before having to make another call to read more data but it s far more likely that by reading KiB chunks we ll get a whole collection of messages because in text terms KiB is quite a lot of data This read amount really could be anything you like but it makes sense to make it the size of a single data record to prevent errors and prevent your app servers from loading a lot of data into memory that they might not need lt summary gt Reads the archive in reverse returning an array of messages and a seek position to continue reading from lt summary gt public async getTail channelName string offset number maxReadChunk number kb const blobName this blobNameFor channelName const blobSize await this repository sizeof blobName let position blobSize offset maxReadChunk let reduceReadBy if position lt reduceReadBy position position const amountToRead maxReadChunk reduceReadBy const buffer await this repository get blobName position amountToRead In this getTail function we re calculating the name of our blob file and then calculating a couple of values before we fetch a range of bytes from Azure The code calculates the start position by taking the total blob size subtracting the offset provided to the function and then again subtracting the maximum length of the chunk of the file to read After the read position has been calculated data is loaded into an ArrayBuffer in memory const buffer await this repository get blobName position amountToRead const firstRecordSeparator buffer indexOf String fromCharCode const wholeRecords buffer slice firstRecordSeparator const nextReadPosition position firstRecordSeparator const messages this bufferToMessageArray wholeRecords return messages messages position nextReadPosition done position lt Once we have KiB of data in memory we re going to scan forwards to work out where the first record separator in that chunk of data is discarding any data before it in our Buffer because we know that from that point onwards because we are strictly parsing backwards through the file we will only have complete records As the data before that point has been discarded the updated “nextReadPosition is returned as part of the response to the consuming client which can use that value on subsequent requests to get the history block before the one returned This is similar to how a cursor would work in a RDBMS The bufferToMessageArray function splits our data chunk on our record separator and parses each individual piece of text as if it were JSON private bufferToMessageArray buffer Buffer const messages buffer toString utf return messages split String fromCharCode filter data gt data length gt map m gt JSON parse m Using this approach it s possible to “page backwards thought our message history without having to deal with locking file downloads or concurrency in our application and it s a really great fit for storing archive data messages and events where the entire stream is infrequently read due to it s raw size but users often want to “seek upwards ConclusionThis is a fun problem to solve and shows how you could go about building your own archive services using commodity cloud infrastructure in Azure for storing files that could otherwise be “eye wateringly huge without relying on third party services to do this kind of thing for you It s a great fit for chat apps event stores or otherwise massive stores of business events because blob storage is very very cheap In production systems you d likely want to implement log rotation for when the blobs inevitably reach their block limits but that should be a simple problem to solve It d be nice if Microsoft extended their block storage SDKs to iterate block by block through stored data as presumably that metadata exists under the hood in the platform	2022-04-16 16:18:08
ニュース	BBC News - Home	Dovercourt woman's bid to house Ukrainian family in caravan rejected	https://www.bbc.co.uk/news/uk-england-essex-61127404?at_medium=RSS&at_campaign=KARANGA	ukrainian	2022-04-16 16:04:33
ニュース	BBC News - Home	Billie Jean King Cup: Great Britain lose to Czech Republic in Prague	https://www.bbc.co.uk/sport/tennis/61120358?at_medium=RSS&at_campaign=KARANGA	Billie Jean King Cup Great Britain lose to Czech Republic in PragueGreat Britain are unable to clinch their place in the Billie Jean King Cup finals as the Czech Republic win the decisive doubles match in the best of five tie	2022-04-16 16:34:23
ニュース	BBC News - Home	FA Cup semi-final: Man City 2-3 Liverpool - Sadio Mane double helps Jurgen Klopp's side through	https://www.bbc.co.uk/sport/football/61120882?at_medium=RSS&at_campaign=KARANGA	FA Cup semi final Man City Liverpool Sadio Mane double helps Jurgen Klopp x s side throughSadio Mane scores twice as Liverpool beat Manchester City in a classic FA Cup semi final at Wembley	2022-04-16 16:47:51
ニュース	BBC News - Home	Manchester United 3-2 Norwich City: Cristiano Ronaldo nets superb hat-trick	https://www.bbc.co.uk/sport/football/61048518?at_medium=RSS&at_campaign=KARANGA	Manchester United Norwich City Cristiano Ronaldo nets superb hat trickCristiano Ronaldo scores a stunning hat trick as Manchester United boost their top four hopes with a thrilling win over Norwich	2022-04-16 16:24:50
北海道	北海道新聞	アフガン国境の村空爆４１人死亡　パキスタン軍機	https://www.hokkaido-np.co.jp/article/670502/	空爆	2022-04-17 01:28:00
北海道	北海道新聞	大量殺処分開始　感染拡大懸念「悲しい」　白老、網走　鳥インフル	https://www.hokkaido-np.co.jp/article/670476/	感染拡大	2022-04-17 01:16:08