Vulnerability Detection and Patching: A Survey Of The Enterprise Environment

Vulnerability Detection and Patching Linux

Detecting vulnerabilities and managing the associated patching is challenging even in a small-scale Linux environment. Scale things up and the challenge becomes almost unsurmountable. There are approaches that help, but these approaches are unevenly applied.

In our survey, State of Enterprise Vulnerability Detection and Patch Management, we set out to investigate how large organizations handle the dual, linked security concerns of vulnerability detection and patch management.

The results produced interesting insights into the tools that organizations depend on to effectively deal with vulnerability and patch management at scale, how these tools are used, and which restrictions organizations face in their battle against threat actors. Download the copy of the report here.

Vulnerability management is an enterprise responsibility

Before we dive into the results of our survey, let’s take a quick look at why vulnerability management operations matter so much in large organizations.

Vulnerabilities are widespread and a major cybersecurity headache. In fact, vulnerabilities are such a critical problem that laws and regulations are in place to ensure that covered organizations adequately perform vulnerability management tasks – because the failure to do so can hurt a company’s customers.

Each industry has different rules that apply to it – with organizations that handle personal data such as healthcare records and financial service firms operating under the strictest rules. It has an impact on day-to-day vulnerability management operations – some organizations must act much faster and more thoroughly than others.

This is one of the points we explored in the survey, trying to understand how different industry compliance requirements affect vulnerability operations on the ground.

The survey

Early in 2021, we kicked off a survey with the intention to study three key factors in vulnerability and patch management operations. We examined patch deployment practices, how maintenance windows are handled, and tried to get a view into the overall level of security awareness of the organizations that responded.

The survey was advertised publicly to IT professionals around the world and it continues to run, even though we have published the initial results.

Live Patching Requires Reproducible Builds – and Containers Are the Answer

Live patching a key threat management tool

We know that live patching has real benefits because it significantly reduces the downtime associated with frequent patching. But live patching is relatively difficult to achieve without causing other problems and for that reason live patching is not implemented as frequently as it could be. After all, the last thing sysadmins want is a live patch that crashes a system.

Reproducible builds are one of the tools that can help developers to implement live patching consistently and safely. In this article, I explain why reproducible builds matter for live patching, what exactly reproducible builds are, and how containers are coming to the rescue.

Live patching: a key threat management tool

Patching is a critical part of systems maintenance because patching fixes faulty and buggy code. More importantly, security teams rely on patching to plug security holes, and there is a real urgency to it. Waiting for a convenient maintenance window to patch is risky because it leaves an opportunity for hackers to take advantage of an exploit.

It creates a difficult conundrum: maintain high availability but run a security risk, or patch frequently but end up with frustrated stakeholders. Live patching bridges that gap. With live patching, the offending code is swapped out while a process is actively running, without restarting the application or service that depends on that process.

Implementing live patching isn’t easy

Live patching is not that straightforward to accomplish – the drop-in code must “fit” in a like-for-like manner, or all sorts of unwanted things can happen. Get it wrong, and the application – or entire server – will crash.

The code behind a running process usually comes from a binary executable file – a machine-readable block of code compiled from source code. A kernel, for example, has thousands of source files all compiled into a few binaries.

With live patching, the live patch code must fit in at an exact level. Yes, the binary file containing the patch code will be different from the binary file containing the bad code. Nonetheless, the new code must slot into place precisely and must depend on the same version of imported libraries. The live patch code must also be compiled using the same compiler options and flags. Bit endianness matters too – the binary file must be ordered in exactly the same way.

In principle, all this is achievable – but in practice, it is a challenge. For example, day-to-day system updates often impact libraries. These libraries could be slightly different, in turn producing binaries that are slightly different when compiling code.