https://codeinfocus.com/categories/postmortem/Recent content in Postmortem onHugoen-usWed, 07 Apr 2021 09:24:02 +0300https://codeinfocus.com/blog/2021-04/the-tale-of-misbehaving-node.js-microservice-postmortem/Wed, 07 Apr 2021 09:24:02 +0300https://codeinfocus.com/blog/2021-04/the-tale-of-misbehaving-node.js-microservice-postmortem/<p><em>&ldquo;You don’t build production shit on your machine!&rdquo; - <a href="https://pushbuildtestdeploy.com/">Yuval Oren</a></em></p> <h2 id="the-problem">The problem</h2> <p>About a month ago we started getting weird issues with one of our Node.js services in change of extracting data from the database to our analytics platform. After a while, we discovered it did extract only partial data, producing no errors.<br> We spent long hours trying to catch the problem, which was difficult since we couldn’t recreate it neither on development machines, nor on our test clusters.<br> Furthermore, the issue would go away for a while after some tweaks and mysteriously come back after a while.</p>