How do I create a custom rule to exclude URLs for a web assessment?

You can exclude specific URLs from a web assessment by creating a custom crawl exclusion rule using regular expressions (regex). Any URL that matches the defined rule will be skipped during the assessment.

Learn more about regular expressions (external link):

https://en.wikipedia.org/wiki/Regular_expression

Create a custom URL exclusion rule

1. Log in to your Security Center.

2. In the main navigation bar, hover over Assets.

3. From the dropdown menu, select Web applications & APIs.

4. Click Edit for the relevant web application.

5. In the configuration window, select Crawl exclusion list.

6. Under Blacklist, enable Regular expressions.

7. In the Regular expression field, enter one or more regex rules to exclude URLs.

   • Make sure to escape special characters such as ., ?, ^, $, [, ], (, ), and =.

8. Click OK.

The exclusion rule will be applied the next time the web application is assessed.

Common examples

Exclude all URLs below a folder

Exclude all URLs below a specific folder:

Regexp:

Example matches

http://www.holmsecurity.se/about-us/contact.html
http://www.holmsecurity.se/about-us/contact/us-office.php

Exclude URLs based on file type

Exclude all URLs that end with a specific file type (for example, all HTML files):

Regexp:

Example matches

http://www.holmsecurity.se/about-us/contact.html
http://www.holmsecurity.se/start.html

Exclude URLs based on a string

Exclude all URLs that contain a specific string:

Regexp: