Open source software has become an integral part of modern application development, enabling developers to accelerate their projects by leveraging pre-existing libraries and frameworks. Open source offers numerous benefits, yet it's not without its challenges.
Open source software has become an integral part of modern application development, enabling developers to accelerate their projects by leveraging pre-existing libraries and frameworks. Open source offers numerous benefits, yet it's not without its challenges. A significant hurdle is ensuring that dependencies remain up-to-date. In our latest blog post, we delve into the complexities associated with updating open source components. We'll discuss why updating to the most recent versions comes with its downsides. We'll also highlight ways on how you can leverage the benefits of open source while eliminating the constant need for updates and potentially breaking changes.
One of the most common issues encountered when upgrading open source software is that it can break existing functionality. This happens even when the goal is just to patch security vulnerabilities. Even though the intention is to enhance security, sometimes the fixes inadvertently introduce new bugs.
For example, most open source libraries follow Semantic Versioning as their "contract" with their users when a certain change is breaking. According to the standard, versions are numbered MAJOR.MINOR.PATCH and PATCH is incremented when you make backward compatible bug fixes. However, the widely used library passport-saml broke its code between 3.2.0 and 3.2.1 while trying to fix a security issue, leading to many frustrated developers.
In some instances, these issues can have serious consequences. A recent incident happened at DataDog, a popular monitoring platform. This incident cost them millions of dollars. According to them, a multi-region infrastructure connectivity issue arose after an upgrade, causing disruption to their services.
Another instance of potential damage occurred when Discord’s authentication service went down for approximately 50 minutes, preventing users from connecting, logging in, reading and sending messages, or doing much of anything on the platform. The incident happened due to an operation where they were upgrading the operating system on the underlying ScyllaDB cluster. Such incidents highlight the potential financial cost of upgrading even for companies with proper testing and validation procedures.
Furthermore, reputational damage is also a concern. Organizations dependent on open source software may face public backlash if upgrading results in unintended consequences. This risk has even led to the emergence of "protestware", where open source libraries are intentionally manipulated to convey messages, potentially causing unintended consequences or harm. One well known example is the attack on the color.js library, which due to its popularity affected the AWS CDK.
Even when breaking changes are properly documented, keeping open source dependencies up to date can be a major challenge.
For instance, packages that process data, like TensorFlow, an open source machine learning framework, may output different results for the same input after an upgrade. In its documentation, TensorFlow explicitly declares that floating point numerical details can change in backward incompatible ways at any point, which might render previously trained models useless. However, some versions of TensorFlow contain more than 100(!) highly rated vulnerabilities.
Additionally, libraries often drop support for older interpreter versions, creating a dilemma for developers who are unable to upgrade their codebase due to dependencies on outdated interpreters (updating from Python 2 to 3, or migrating to the latest version of the Node engine).
Major upgrades can also introduce breaking changes, leaving developers with the arduous task of modifying their code to accommodate the new version.
One example is the node ejs library. A critical vulnerability affecting versions 3.1.6 and below was discovered 2 years ago. However, version 3 of ejs dropped support of Nodes 4 and 6 leaving developers to choose between the tremendous effort of upgrading the Node engine or leaving a 9.8 vulnerability in their code. In fact, even as of today, the ejs version 2.7.4 gets about 2 million downloads (20% of the total) per week.
In an effort to streamline the upgrade process, some open source projects provide tools to assist with transitioning between versions. However, these tools are not always foolproof. For example, the React Native community published a tool that generates a huge diff file when upgrading, requiring significant manual effort to resolve conflicts. Similarly, Google has published a tool to update from CentOS 7 which is reaching its end of life to Rocky Linux. However, these tools have multiple open issues, causing frustration for those attempting the migration.
Keeping open source applications secure requires a constant race to update all the components. Besides the major engineering effort involved in that, this effort introduces operational risk.
To address these challenges, a unique approach like that offered by Seal Security can be a game-changer. By providing automated vulnerability management and continuous monitoring, Seal Security helps organizations keep their open source dependencies up-to-date, secure, and free from vulnerabilities. With the right tools and processes in place, developers can overcome the challenges, enabling them to focus more on innovation and less on the complexities of open source upgrades.
