Content Discovery: Finding Hidden Directories and Endpoints
A beginner content discovery guide: map hidden directories, files and endpoints with passive sources first, then ffuf and gobuster, read status codes, filter soft 404s, and shut it all down.
Content discovery is the recon step where you map every directory, file and endpoint a web application exposes, including the ones it never links to from the homepage. The links in the navigation bar are the part the owner wants you to see. The interesting part is almost always the rest: an old admin panel, a forgotten /backup.zip, a staging API that skipped the login check, a .git folder someone copied straight to the web root. This guide walks the whole method from a beginner's first request to a repeatable pipeline, and it closes on the defender side, because the same habits that make a site quiet for an attacker make it safe.
Read it top to bottom once to build the mental model, then keep it open while you work a box. Every command here is aimed at a lab you own or a target you are authorized to test.
Run these techniques only against an application you own or have written permission to assess, such as a personal lab or the training host lab.0xpriv.dev. Brute forcing paths on a site you do not control is unauthorized access and is a crime in most places.
01What content discovery is and why hidden paths matter
Content discovery is the practice of finding the parts of a web app that are reachable but not advertised, and every hidden path you find widens the attack surface you can test. A web server will happily serve a file whether or not anything links to it. "Not linked" is not "not there", and it is certainly not "protected". If /admin/ responds, the fact that no button points to it changes nothing.
Each new path is another place a bug can live. A hidden endpoint might take parameters the public pages never send, run an older code path, or skip an authorization check the developers assumed no one would reach. Mapping those paths first is what turns a blank "here is a login form" into a real list of things to attack. This is the first phase OWASP groups under information gathering, and it pays for itself on nearly every assessment.
02Start passive before you make noise
Passive discovery means collecting paths that are already public before you send a single brute force request. It is faster, quieter, and often more productive than guessing, because real applications leak their own structure in a dozen predictable places. Spend the first few minutes here and your wordlist run later has a head start.
The obvious files come first. robots.txt lists paths the owner asked crawlers to skip, which is a free map of things they consider sensitive. sitemap.xml lists pages they want indexed. The /.well-known/ directory holds security and protocol metadata. After that, read the application's own JavaScript: modern front ends hard-code their API routes, and a quick grep over the bundled JS surfaces endpoints no page links to.
# 0xpriv.dev/recon passive sources, no brute force yet curl -s https://lab.0xpriv.dev/robots.txt curl -s https://lab.0xpriv.dev/sitemap.xml curl -s https://lab.0xpriv.dev/.well-known/security.txt # pull endpoint-looking strings out of the app's own JavaScript curl -s https://lab.0xpriv.dev/assets/app.js | grep -oE '"/[a-zA-Z0-9_/.-]+"' | sort -u
The Wayback Machine is the other free win. It stores snapshots of a site going back years, so it remembers paths that were removed but never actually taken offline. Its CDX API gives you every URL it has ever seen for a host in one request.
# every path the Internet Archive has recorded for the host curl -s "https://web.archive.org/cdx/search/cdx?url=lab.0xpriv.dev/*&output=text&fl=original&collapse=urlkey"
$ curl -s https://lab.0xpriv.dev/robots.txt User-agent: * Disallow: /admin/ Disallow: /backup/ Disallow: /api/internal/
Finally, let a crawler walk the site and follow every link, form and redirect it can reach from the pages you already know. A spider turns one known page into the full set of linked paths, which you then merge with everything passive gave you. Only once that list stops growing is it worth making noise.
03Brute forcing directories and files with ffuf and gobuster
Directory and file brute forcing is the active phase: you take a wordlist of likely names and request each one, then read the status code the server returns to decide what exists. Two tools dominate here. ffuf is a fast, flexible fuzzer built around a FUZZ keyword you drop anywhere in the request. gobuster is a focused directory and DNS brute forcer that is simple to drive. Learn one well and the other reads the same.
# ffuf: FUZZ marks where each wordlist entry is substituted ffuf -u https://lab.0xpriv.dev/FUZZ -w /usr/share/seclists/Discovery/Web-Content/common.txt -t 40 # gobuster: dir mode against the same host gobuster dir -u https://lab.0xpriv.dev -w /usr/share/seclists/Discovery/Web-Content/common.txt -t 40
The whole game is reading the response. A status code tells you how the server felt about the path, and a 404 is not the only answer worth having. A 403 means the path exists but access is blocked, which is still a finding. A 401 means it wants authentication. A 301 or 302 often redirects to a login or to the real location of a directory. A 405 means the endpoint is real but you used the wrong HTTP method.
| Status | What it usually means for discovery |
|---|---|
200 | The path exists and returned content. Confirm it is not a soft 404. |
301 / 302 | Redirect. Often points at a login, or at the canonical path of a directory. |
401 / 403 | The path exists but is protected. A real finding, not a dead end. |
405 | The endpoint exists but rejects your method. Try GET versus POST. |
500 | Server error. A real code path broke, sometimes leaking a stack trace or file name. |
By default ffuf shows everything, so you match the codes you care about with -mc. gobuster whitelists with -s (status codes) or blacklists with -b. Pointing both at the status codes you want keeps the output readable.
# ffuf: only show these status codes ffuf -u https://lab.0xpriv.dev/FUZZ -w common.txt -mc 200,204,301,302,307,401,403,405 # gobuster: whitelist the same set of codes gobuster dir -u https://lab.0xpriv.dev -w common.txt -s 200,204,301,302,307,401,403,405 -b ""
$ ffuf -u https://lab.0xpriv.dev/FUZZ -w common.txt -mc all -ac admin [Status: 301, Size: 0, Words: 1, Lines: 1] login [Status: 200, Size: 2411, Words: 140, Lines: 61] backup [Status: 403, Size: 281, Words: 20, Lines: 10] api [Status: 401, Size: 173, Words: 12, Lines: 7] server-status [Status: 403, Size: 299, Words: 21, Lines: 11]
04Choosing and tuning wordlists
Your wordlist decides what you find, so picking the right one matters more than raw speed. A huge list against a small app wastes time and makes noise; a tiny list misses the one path that mattered. The standard source is SecLists, a large collection of curated wordlists that ships with Kali and installs anywhere. Inside it, the Discovery/Web-Content folder is where you live.
| SecLists wordlist | When to reach for it |
|---|---|
common.txt | A fast first pass. Small, high hit rate, good for a quick map. |
raft-*-directories.txt | Directories ranked by how often they appear in the wild. |
raft-*-files.txt | File names, pair it with the right extensions. |
directory-list-2.3-medium.txt | A deeper sweep when the quick pass was thin. |
Two knobs turn a directory list into a file list. Extensions append a suffix to every word, so admin also tests admin.php and admin.bak. Match the extensions to the stack you fingerprinted: .php on a PHP app, .aspx on IIS, plus the backup and archive suffixes that get left behind, like .bak, .old, .zip and .tar.gz. Recursion then re-runs the search inside every directory you find, so /api/ turns into a search of /api/ itself.
# ffuf: add extensions and recurse into discovered directories ffuf -u https://lab.0xpriv.dev/FUZZ -w raft-medium-words.txt \ -e .php,.bak,.old,.zip,.txt \ -recursion -recursion-depth 2 -mc 200,301,302,401,403 # gobuster: the same extensions in dir mode gobuster dir -u https://lab.0xpriv.dev -w raft-medium-words.txt -x php,bak,old,zip,txt
Fingerprint the stack first with the response headers and the file extensions you already see. Running an ASP.NET wordlist against a PHP app, or recursing to depth five on a huge list, buys noise and little else. Start small, widen only where the hits are.
05Filtering out soft 404s
Soft 404s are pages that return a 200 OK for a path that does not really exist, and they will bury your real results if you let them. Plenty of apps answer every unknown URL with a friendly "page not found" template rendered at status 200, or redirect everything to the homepage. Match on status alone and every one of those shows up as a hit.
The fix is to filter on the shape of the response, not just its code. A fake page tends to be the same size, the same word count and the same line count every time, because it is one template. ffuf filters those with -fs (size), -fw (words), -fl (lines) and -fr (a regex over the body). The easy button is auto-calibration: -ac requests a few random paths that cannot exist, learns what the "not found" response looks like, and filters it for you.
# let ffuf learn the soft-404 shape automatically ffuf -u https://lab.0xpriv.dev/FUZZ -w common.txt -ac -mc all # or filter by hand once you know the junk page is 1256 bytes / 94 words ffuf -u https://lab.0xpriv.dev/FUZZ -w common.txt -fs 1256 -fw 94 # drop any response whose body matches a known not-found string ffuf -u https://lab.0xpriv.dev/FUZZ -w common.txt -fr 'Page not found'
gobuster handles the same problem with --wildcard, which tells it a server answers every request so it can account for the always-on response. Either way, the goal is the same: a result list short enough that every line is worth a human look.
06Finding API endpoints and parameters
API endpoint discovery is content discovery pointed at the parts of an app that return JSON instead of HTML, and it needs its own wordlists and its own eye. Modern apps put most of their logic behind routes like /api/v1/users or a single /graphql, and those routes rarely appear in a normal web-content list. SecLists carries API-specific wordlists for exactly this, and documentation files like swagger.json, openapi.json and /api-docs hand you the entire route map when they are left exposed.
# fuzz API routes with an API-focused wordlist, under a known prefix ffuf -u https://lab.0xpriv.dev/api/v1/FUZZ \ -w /usr/share/seclists/Discovery/Web-Content/api/api-endpoints.txt \ -mc 200,201,400,401,403,405 -ac # the free wins: a left-open spec documents every route for you curl -s https://lab.0xpriv.dev/openapi.json | grep -oE '"/[a-zA-Z0-9_/{}.-]+"'
Once you have an endpoint, the hidden inputs are the next target. An API route often accepts parameters the front end never sends, and a debug or admin flag is a common one. You discover them the same way: put FUZZ in the parameter name and watch for a response that changes. A 200 that suddenly becomes a 400 "missing value" or returns a different body is a parameter the app actually reads.
# parameter name discovery on a GET endpoint ffuf -u 'https://lab.0xpriv.dev/api/v1/user?FUZZ=1' \ -w /usr/share/seclists/Discovery/Web-Content/burp-parameter-names.txt \ -mc all -fw 12 # 12 words is the "unknown param" baseline response
For parameter mining at scale, arjun is purpose-built and reads the same wordlists. ffuf is enough to learn the idea and covers most jobs, so start there and add specialized tooling only when a target needs it.
07A worked pipeline against lab.0xpriv.dev
Here is the whole method as one repeatable pipeline against the lab host, so you can see how the phases hand off to each other. The order matters: cheap and quiet first, loud and thorough last, with a filter step before anything reaches your notes.
Collect passive paths
Pull robots.txt, sitemap.xml, the Wayback CDX list and the app's own JS. Save everything to one file of candidate paths.
Fingerprint the stack
Read the response headers and visible extensions to pick the right wordlist and the right extension list.
Quick brute force
Run common.txt with auto-calibration on, matching the status codes that matter. Note every 200, 301, 401 and 403.
Deepen and recurse
Re-run a larger list with extensions inside the directories you found, one or two levels deep.
Hit the API
Fuzz routes under any /api/ prefix and grab any exposed spec, then mine parameters on the endpoints you confirm.
Triage by hand
Open each survivor. A 403 on /backup/ and a 200 on /api/v1/debug are where the real testing starts.
# the active passes, strung together against the lab host ffuf -u https://lab.0xpriv.dev/FUZZ -w common.txt -ac -mc all -o quick.json -of json ffuf -u https://lab.0xpriv.dev/FUZZ -w raft-medium-directories.txt \ -e .php,.bak,.zip -recursion -recursion-depth 2 -ac -mc 200,301,401,403 ffuf -u https://lab.0xpriv.dev/api/v1/FUZZ -w api-endpoints.txt -ac -mc 200,401,403,405
$ cat quick.json | jq -r '.results[] | "\(.status) \(.url)"' 301 https://lab.0xpriv.dev/admin 403 https://lab.0xpriv.dev/backup 200 https://lab.0xpriv.dev/api/v1/debug 401 https://lab.0xpriv.dev/api/internal
08Being loud and staying in scope
Content discovery is noisy by nature, and pretending otherwise gets people in trouble, so handle the noise deliberately. A single run can send tens of thousands of requests in a minute. On a lab you own that is fine. On a client engagement it can tip over a fragile app, trip rate limits, or bury the real users, so you throttle on purpose and tell the client when you plan to run it.
ffuf rate-limits with -rate (requests per second) and spaces requests with -p (a delay between them). Keep the thread count sane on anything you do not own. And keep yourself honest about scope: only fuzz hosts and paths the engagement actually covers, because a wildcard wordlist does not know where the authorization ends.
# a considerate run: capped rate, small delay, modest threads ffuf -u https://lab.0xpriv.dev/FUZZ -w common.txt -rate 20 -p 0.1 -t 10 -mc 200,301,401,403
A path that resolves is not the same as a path you are allowed to touch. Stay inside the hosts, subdomains and directories named in writing, and stop the moment a run drifts toward something outside them.
09Detecting content discovery
Detecting a content discovery run is one of the easier jobs in defense, because the behavior has a loud signature in the logs. One client address generating hundreds or thousands of 404s in a short window, walking a predictable list of path names, is a brute force sweep and nothing else. A burst of 403s and 401s against paths no normal user visits tells the same story.
Alert on the behavior rather than any single request. A rule that fires on a high 404 rate per source IP, or on a spike of requests to paths that never appear in your own sitemap, catches ffuf and gobuster without needing to match a specific user agent (which any attacker can change in one flag). Feed those alerts into the same monitoring you already run and the sweep becomes obvious in real time.
10Shutting it down on the defender side
Shutting down content discovery is mostly about not leaving things lying around and making the server boring to probe. You cannot stop someone from guessing path names, but you can make sure every guess hits a wall and that the walls all look the same. The controls below are ordered from the one that matters most.
.git, .env, config files and archives out of the web root entirely. Deploy from a build artifact, not by copying your working directory, so the sensitive files never reach the public server.11FAQ
Is content discovery the same as directory brute forcing?FAQ
Brute forcing is one part of it. Content discovery is the whole goal of mapping hidden paths, and it starts with passive sources like robots.txt, sitemaps, JavaScript and the Wayback Machine before any guessing. Brute forcing with ffuf or gobuster is the active step you take once the free sources are exhausted.
ffuf or gobuster, which should a beginner learn first?FAQ
Either works, and they overlap heavily. gobuster is a little simpler to start with for plain directory brute forcing. ffuf is more flexible once you need parameter fuzzing, custom filters and auto-calibration, so most people settle on ffuf as their main tool and keep gobuster around.
Why do I get hundreds of 200s that are all fake?FAQ
The app is serving a soft 404: a "not found" page rendered at status 200. Filter on the response shape instead of the code. Turn on ffuf auto-calibration with -ac, or filter by size and word count with -fs and -fw, and the fake hits drop out.
Is a 403 worth anything or is it a dead end?FAQ
It is worth plenty. A 403 confirms the path exists but is blocked, which means there is something there to protect. Note every 401 and 403, because access controls are sometimes weaker than they look and the path is a real finding either way.
Which wordlist should I start with?FAQ
Start with common.txt from SecLists for a fast, high-value first pass, then widen to the raft lists or directory-list-2.3-medium.txt where you found hits. Match extensions to the stack you fingerprinted, and only reach for the largest lists when the smaller ones come back thin.
references
Founder of 0xpriv, covering offensive security, web application vulnerabilities, and practical security research through PHP, Python, and hands-on technical analysis.