Getting speed back, my Webpack debugging journey
I want to take you on a journey, through a land of slowness. Where everything lost its motivation to go fast, where hope seems lost. I want to take you through debugging the speed issues we were having with Webpack. Usually I'll try and write articles about processes, giving people advice and tips and get them to think about certain subject. For this article I'm trying a different direction, my goal is to share the process I went through while working on this problem, describing my thought process, but without any judgement. Just what I did, and why. Hoping to inspire and remove the illusion that experienced developers know everything. I'd love to hear your thoughts about the format.
Webpack is a widely used package, very widely, as in 4+ million downloads per week. While it has gotten somewhat easier, every time I have to work on it, it feels like going on a jungle expedition...on an island where some old guy managed to recreate some dinosaurs... which manage to escape... We've tried alternatives, but out of all of them Webpack matches our needs the most. The documentation is pretty good, there are a lot of tutorials and examples online, but at the same time a lot of them are outdated and no longer work. But for me the biggest issue may be that debugging can be pretty hard, fighting a lion with your hands tied behind your back hard.
When we finish a user story we first deploy our solution to our test environment, so our automated test suite can do it's magic. We had a story where we happened to find some issues while testing on this environment, resulting in having to do this process several times. But I started to notice that I took pretty long, longer than usual. When looking on Team City to confirm this feeling, I noticed our 'Run unit test & build' took up to 15 minutes, while this usually is around 7 / 8 max. When diving into the logs I found that our build step took between 8 and 10 minutes, at some moments even going up to 12. Our solution is not that big, it should not take this long. Even better, it didn't use to take that long. Something took its mojo!
Where to start?
My first instinct was to check the history on team city. It saves the information of the last couple of builds, so if the problem was introduced in one of the last few stories, I should be able to see a big increase in time here. But all of them were slow, so this wasn't caused recently.
What next? It could be a problem of our CI environment, so I want to check if it's also slow on my own machine. This was quite easy to confirm, building on my own machine takes 8+ minutes.
Could it be an update? We update all our packages at the beginning of our sprints. We do two week sprints, so I decided to take a look at the release notes of Webpack. There was no change in the last sprint that should have caused this. To be sure I tried downgrading it to a version of a few weeks ago, but still it stayed as slow as it was. I quickly checked out the change logs of other major packages we used, but there were no mentions of anything that made me suspect them. Since it was quite some work to revert all the packages to a earlier version I decided to skip this, and follow my feeling that this is probably some issue in Webpack itself.
Getting to the core
The next tool I was going to use was Google. First trying to find somebody with the same issue, looking up terms like "Webpack slow", "Webpack performance problem" and "Webpack speed". I found a lot of results, really a lot. Okay so now I know that a lot of people find Webpack slow, still that doesn't explain a sudden big drop in performance. I looked through some results, most of them being guides to increase performance through caching and pre-building, which wasn't what I was looking for. Since I didn't want a general performance guide, but a way to find the reason of our sudden bad performance I decided to change my search direction, going for "Webpack debug performance".
The first suggestion that took my attention was a plugin that would show you the build times of the different parts of the process. "Just what I need!" is what I thought. So I installed the plugin, and started building again. When it was finally done I hopefully scrolled up through the build result looking for the times. And what did it tell me, nothing... The total build time was shown as a little over 8 minutes, it showed a few build steps, but their combined time didn't even touch the 3 minutes. I went to google to check for some examples, maybe I used it wrong, but came up empty handed. Going back to my earlier searched for performance debugging help I found a discussion, somewhere on github where the person was showing the slowness by using the '--progress' flag in webpack. So I decided to give that I try and see what it does.
I'm not alone
When running webpack with the progress flag I got real-time information about what it was doing and instantly noticed I stayed at "92% Chunk asset optimization" for almost 90% of the build time. This has to be the problem! So I opened up Google again and searched for "webpack 92% chunk asset optimization". There were a lot of people having the same issue, finally!
Trial and error
I found massive discussions on github projects about this issue, some spanning more then a year and some with the last message in the last few days. So this issue has been in there for a long time, and it may be still not solved. But there were a lot of suggestions in the replies, people saying "This fixes it for me", and a lot of different things fixed the same issue for different people. My next instinct was to use trial and error by just trying out the solutions posted there, starting of with the most mentioned solutions. The most mentioned solutions were disabling source maps and disabling compression (and some other options) in UglifyJS. So I went back to basics with build mode, disabling source maps and installed a separate version of UglifyJS so I could disable the compression. My next build, was 80 seconds!
The first sign of hope
Tweaking and confirming
I was very close now, it worked. My next step was figuring out what was causing the slowness. Since the thing I read most was the compression flag on UglifyJS, I started by turning that off. The build time instantly jumped back to 8 minutes. Okay, so this causes the slowness, but why wasn't it an issue before? I think it just slowly got slower while the project grew, adding typescript as well eventually impacted it even more and slowly it reached this point. If it wasn't for having to deploy the same story multiple times to our test environment we probably wouldn't even have noticed this issue until it was even worse.
Can we make it even better?
Now that everything is working it's time to tweak it even more, see if we can get it to go even faster! First I slowly started to change settings back to what they were, so I ended up with only the settings on the UglifyJS part where the compression was disabled being the only real change made. Once this was confirmed to be working, I was at a build time of around 120 seconds. Remember the performance guides I found in my first searches? The ones I didn't need? Well, I was glad I found them now. Skimming back through these guides I saw one easy step that made a lot of sense to me. When using loaders the most efficient way is to use include paths, to make sure it only processes the files it needs to process. With this in mind I looked at the webpack configuration and saw that js / jsx and ts / tsx files were not limited to a folder. That could mean that every module file is put through the loader. I updated the settings to make sure paths were defined where possible. Time for one more build, and our new build time is...79 seconds!
Looking back at this process you'll notice a lot of different techniques. I like to follow my gut feeling, sometimes I just start trying solutions that I find even if I don't understand them. But I always try to understand the actual problem and solution in the end. I use Google, a lot, it would be insane, even impossible to figure this stuff out without using it. Other times I'll take a more structured approach, this all depends on the problem. No two issues are the same, no two developers work the same, but this was my way. I wrote this article to inspire, but also to see if people find these real world though processes on paper interesting, helpful or any thing else. I'd love to hear your thoughts about it.