Defensive Programming Tips-1: Bad URL Handling Patterns

We are starting a new series on defensive programming where we will share useful tips to mitigate common mistakes that can have severe consequences. We will be aiming to create content based on real world examples, real CVE’s and our own assessments.

In a recent security assessment, we have identified a common URL handling misusage which leads to a critical vulnerability. After doing some digging about this “programming error” we were surprised to see how common and yet so easy to find it was.

At first, the problem seemed to be a “sanitization” issue and actually it was. Developers were missing a very important point although they thought they were “sanitizing” the user input (which was the URL redirection) properly. As we noticed this was a common mistake among developers in different organizations, we decided to address this issue in the first episode.

So, let’s start with the following snippet:

Although at first glance this snippet seems to be okay, in reality there are two problems here. First, “target” is an uncontrolled input and second “govalidator.IsRequestURL” and “url.Parse” are not sanitizers. So, let’s add a hostname validation to make sure the function communicates to our services only.

Now our handler seems to be better because we are limiting user input with a predefined scope. The expected form value (target) is supposed to be something like https://foo.company.com/data or https://bar.company.com/users so that the handler function forwards relevant requests to an internal service. However if an attacker sends a payload like https://foo.company.com@attacker.com URL.Parse will be true, regex check will be true and the hostname will be attacker.com which will lead to a SSRF vulnerability.

To understand the problem better, let’s take a closer look at RFC3986[1]. According toSection 3 – Syntax Componentsa valid URI structure is shown as:

Which directly quotes `The authority component is preceded by a double slash (“//”) and is terminated by the next slash (“/”), question mark (“?”), or number sign (“#”) character, or by the end of the URI.

This means that a valid URI may have the following structure:

//<user>:<password>@<host>:<port>/<url-path>

Since at-sign (@) is a valid URL syntax, URL schema checks (url.Parse or govalidator.IsRequestURL…, etc.) will always be valid and such a pattern will become meaningless in terms of security.

This is a common pattern we encountered in different organizations multiple times. As nodes are supposed to communicate with different internal and external domains in microservice architectures, this type of vulnerability is more common than you would expect.

You can play around with the vulnerable code snippet at; https://play.golang.org/p/tIgiEBXUOW1

So, how should we handle URLs properly?

1- Sanitization/Filtering:

Unfortunately, user input sanitization is the general remediation advice for the majority of application security issues. You should always apply sanitization even in internal services considering that any input may be malformed.

For the examples above, even if the URL.Parse will filter most of the bad values, you should still be aware of the schema, authority and path of your URI.

If you are sure that basic authentication is not relevant, you can filter at sign (@) as well.

Pro-Tip: Consider developing an internal SafeURLParse function which could be useful in other parts of the project.

2- Whitelisting:

Whitelisting is the default approach in defensive programming when there is any filtering on the table. If your business logic allows you to predefine target URLs, the better and the correct approach is to always check for target.Hostname() within your domain whitelist.

With this approach, you can be sure that nothing funky will happen in your URL redirection.

3- Security Engineering:

Security Engineering is creating secure-by-default approaches in your organization. By secure by default, we mean engineering solutions that proactively prevent security vulnerabilities at the root cause.

For this specific scenario enforcing your microservices to use a service discovery solution/service registration or a custom DNS server may prevent a security issue even if there was a vulnerability in your code.