How do I avoid scanning duplicate web pages?
This article describes how to use duplicate detection to skip web pages that are very similar to pages already scanned, and how to review the skipped pages in the scan report.
What is duplicate detection?
Duplicate detection is a web crawler feature in web application security. When the crawler finds a page that is very similar to a page it has already scanned, the page is skipped. The comparison covers both the page content and all its properties.
Choose a threshold setting
The threshold sets how similar a page must be to an already-scanned page before it is skipped.
| Setting: | A page is skipped when it is at least: |
|---|---|
| Off | Duplicate detection is disabled. All pages are scanned. |
| High | 95% similar |
| Medium | 90% similar |
| Low | 85% similar |
| Custom | Your own value between 85% and 100% |
A lower threshold skips more pages
A lower threshold means more pages count as duplicates. For example, Low skips more pages than High and can noticeably reduce the total number of pages scanned.
Set the threshold:
- Log in to Security Center.
- Hover over Assessments in the main menu.
- Click Scans > New web scan.
- Under Target, select the web application you want to scan from the Target dropdown.
- From the Duplicate detection dropdown, select High, Medium, Low, or Custom to set your own value. The default is Medium, which is the recommended setting.
- If you selected Custom, enter a value between 85 and 100, e.g., "92".
- Click Start Scan.
- Done!
Review skipped pages
Skipped pages are listed in the scan report under the vulnerability Page Skipped - Similar Attack Surface (HID: 3-1-10056).
For each skipped page, the report shows:
- The skipped URL
- The page it was considered similar to
Skipped pages share the same vulnerabilitiesThe comparison covers both content and properties, so you can treat each skipped page as having the same vulnerabilities as its matched page.