Or why manual throttling is mandatory when using async parallelism in Node.js

A valuable lesson from scientific computing that also applies to anyone doing I/O, networking or number-crunching

Photo by shun idota on Unsplash

I am running a very specialized weather site for paragliding and sailplane flying that does scientific computing in a web context.

Since discovering that the need for C++ code for high performance is constantly diminishing in “In 2021, is there still a huge performance difference between JavaScript and C++ for CPU-bound operations”, I set to finalize the migration of some of the API code from my old GDAL/C++ implementation to a more…

If you think that this is a joke, then this story is for you

Photo by Asyrafunk RKTW on Unsplash

I published very recently a new plugin for the JS bindings of GDAL that I currently maintain for importing and exporting N-dimensional arrays to scijs. scijs, while still far behind its Python namesake, is currently gaining more and more popularity, driven by the general success of JavaScript and V8.

I always knew that Node.js I/O performance was unbeatable compared to other scripting languages. Then, last year, I completely gave up using Python an year ago after I was unable to beat my own mono-threaded JS implementation of a linear optimization algorithm in multi-threaded Python.

Still, being used to the node-gdal-async

Recently I had to make some examples for using my new cartography library for React. For every example, I wanted to be able to display both the component output and the code itself, like this:

(the code loading the examples discussed here is in https://github.com/mmomtchev/rlayers/blob/master/examples/App.tsx)

As some of the examples were complex and constantly evolving, it was all natural to assume that this was to be all automatic with no maintenance required after the initial setup.

My first reflex was to use React itself, trying to leverage the JSX and to transform it to HTML. …

They told me it couldn’t be done, but I refused to listen

With well-designed JS, there is always room for one more DOM element

About an year ago I set to remake the profile page on my soaring weather website. That page is a very good exercise in data visualization and web page performance, so I decided to share some of the valuable experience I learned from that single page.

If you go to that page and click the GFS model from the models menu on the top left, your browser will have to deal with a monstrosity. …

Some common pitfalls and design patterns when using Promises and async functions

Photo by marcos mayer on Unsplash

Single-threaded event-driven asynchronous I/O is one of the most interesting features of the JS language. It is not a new concept, in fact, more than 20 years ago, there was one very influential paper in the world of network programming called “The C10K problem”. It was an eye opening experience for one whole generation of system developers and it spawned the arguably fastest and one of the most widely used low-level I/O frameworks in the world — libevent. …

Reading and parsing a large CSV file in Node.js doesn’t have to be slower than the equivalent compiled C code… that is if you are willing to give up the comfort of the automatic memory management

Photo by Waldemar Brandt on Unsplash

Recently I had to process a very large (1.6G) CSV-like file in Node.js. After going through several iterations, some reading, and quite a few a-ha moments, I decided to write yet another story on this subject.

Reading large files in Node.js is a subject that comes back very often. In fact, there are already a few Medium and dev.to stories and some very informative…

How many times you have been confronted with the classical problem of parallelizing download loops in crawlers and scrapers

Photo by Marvin Ronsdorf on Unsplash

Consider the following typical crawler or scraper code:

Thanks to async / await this is as simple and readable as it can get.

But what if you have thousands of URLs? This would take ages as each iteration waits for the previous one.

Luckily, JS has a solution:

Here we are launching all the downloads without await. The beauty of promise chaining allows us to create new promises out of succession of asynchronous operations. We push all those promises…

Momtchil Momtchev

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store