This project has been interesting, to say the least. This was my first real deep dive into a topic that has fascinated me for years, it was just something I never quite had the time to go all-in on due to various reasons. In doing so, I discovered a niche in computer science that I didn’t get bored/burned-out of after spending months working on it on-and-off, and I learned a lot about the process of developing useful software in the real world. Ignoring the purely technical challenges, completing this project had three major challenges that I had to overcome to complete it successfully.

        The first challenge I faced is fairly simple: I needed to not fail to complexity. I was creating a fairly complex program from scratch, each piece of that program on its own could be the content of a semester-long class if the underlying problem it tried to solve was explored anywhere near fully, each piece had to work reasonably well in isolation, and each piece was highly sensitive to how the other pieces worked. Therefore, to avoid failing to complexity, I just needed to write modular, clear and non-brittle code. Unfortunately, that is much easier said than done. It’s difficult to write good code to solve a problem that you only fully understand weeks after you originally wrote the code to solve it, after all. Despite that, I did reasonably well at this overall, but it was absolutely not perfect. The messiest innards of my codebase ended up being something I wanted to just ignore and not change anymore for fear of everything turning into an unmanageable mess; complexity from those messier parts spread outwards and leaked into the rest of the codebase. Other parts started out being very clean and streamlined, and ended up getting messier as time went on when I inevitably realized I hadn’t considered an edge case and needed to rework something, and had to work around what I already had (as starting ‘from scratch’ and fixing the issue at the source would simply be too time-consuming). This was probably the most important lesson I got out of this project: learning how real software works and how “software entropy” slowly creeps in and makes everything more complex than it needs to be. Care needs to be taken at every step of development to ensure that any structural changes are well-thought-out and do not introduce issues in other seemingly unrelated parts of the codebase. Overall, my code definitely did its job. It can produce machine code given a text file with code written in the language I created for this project, and it does it extremely quickly (under a tenth of a second for most programs!). What I learned from all of this is how to make the next attempt even better, and hopefully how to better defend the next project I undertake from software entropy.

        The second major challenge I faced was trying to document my codebase. Doing so introduced me to another fundamental problem of software development: how do you explain the intricacies of a complex piece of software to someone else? And not just anyone else for that matter, someone who has absolutely no idea what assumptions are made, what the exact constraints are, and why it had to be made that way. I learned as I went back to try to better document the more obtuse parts of my code that they were the result of so many assumptions gained throughout development that it was nearly impossible to fully document every detail of those pieces without just explaining what every single line of code did and why. At its core, the issue was that the code was not designed to be independent: it was one part of a greater whole, therefore to properly understand that small piece of the whole, one needs to understand the whole (at least at a high level). I also learned about the dangers of hidden assumptions: when a large piece of code is not designed from the ground up to work under a set of constraints, it ends up having all sorts of hidden assumptions (about the rest of the codebase) that are nearly impossible to document after-the-fact. If I had instead written the documentation (and the assumptions) beforehand, I could have designed my code to work within those constraints and would have had something that I would have been able to document much better. While I would say the documentation I have does a reasonable job at documenting the codebase at a high level, what I could have done better on was documenting the details.

        The final major challenge of this project was how to properly articulate what exactly I’ve been doing for the last 4 months in a way that doesn’t look like I just threw a bunch of random technobabble at a screen and just left whatever stuck. In trying to do so, I discovered the two most powerful tools in existence for doing that: analogies, and simplification. Finding simple, understandable and reasonably coherent analogies for complex technical problems can drastically improve how well a problem can be explained, and in writing my “Compilers Crash Course” series I had to make use of several to explain concepts that really wouldn’t make sense otherwise without a sizable amount of background knowledge. In general, I found that the closer to the real world the analogy is, the better. I also had to get much better at cutting out the ‘noise’ of the problem and getting straight to the abstract idea that underpins it. After finding that root idea, I was able to take it and explain it in a (relatively) non-technical way. I also had to learn to simply let go of my desire to explain the solution to these problems with any amount of technical accuracy: while I may appreciate the solution, a reader who frankly doesn’t really care how the problem is solved will not. Basically, technical accuracy almost always impedes understanding, and I had to internalize that before I could properly write those non-technical articles.

        At the end of the day, I am proud of what I have created, and am very satisfied to have been able to do this project. I may not have done everything perfectly, but I learned an absolutely massive amount about the technical aspects of producing a usable programming language / compiler, and about how to properly develop software in general. I also learned a large amount about the non-technical aspects of software development: sharing your creation to people who are not paid to know what you’re talking about. It has been an extremely enriching experience overall.