Lazy directory searching for Pentesters

In many penetration tests the client will often just give you the IP of your target.  I tend to look for web applications as soon as I can. In my opinion if you want to gain access to a large organization's data then the fastest way is often via some security issue in one of their web applications.

On the topic of using Kali Linux, I use to think this was a good idea till an online friend of mine mentioned something interesting to me, this was not his exact words, but he said something like this:
"You don't need Kali, you can use any Linux distro or even BSD, just clone the tools that you need and familiarize yourself with your tools." I enjoy poking fun at people who make use of Kali even though there are loads of really talented infosec folk who make use of Kali. I could make list, but there are just so many people who make use of it. In an internal pentest it would often really make sense to make use of Kali as some of the tools used will take a long time to setup if you don't use them often.  Kali has a pretty cool set of tools, but I often like the set of tools over on the blackarch distro's list of tools far more, as it contains many more new unheard of tools. See the extensive list here: blackarch.org/tools.html , this list might keep you busy for a while.   I found on of my favourite(this is the correct spelling for the word in British english) tools for finding directories and files during web application assessments on this list of tools, I'll talk about it later, the tool is called dirsearch. https://github.com/maurosoria/dirsearch (This tool might be on Kali, I don't keep up with Kali enough to comment on that.)
If you do end up making use of Kali, especially in an internal pentest, then I suggest you lock it down, especially if you are running it in a vm and you are using ssh to login to your own vm. There is a topic that is not often discussed in the infosec community and that is the topic of attacking attackers and attacking attacker tools. We don't practice what we preach or we don't practice what we preach as much as we should.  I often see things that really disappoint me such as: pentesters with Burp's Root CA installed in their regular browser or guys being logged into facebook,twitter and gmail in the same browser that they are using to find cross site scripting on sites that could possibly already have other attacker's stored cross-site scripting payloads stored in that very page.

 If you can code, then I suggest that you modify your own tools or make helper tools even if they are just wrappers to automate running your different tools of choice.  If you have the time then go as far as to write your own Burp suite plugins, this can be very rewarding to if you do bug bounties.

If you like hacking web applications then you will hopefully know that you need a  really good directory or file searching tool. I know many people like the classic tool dir buster. My current favorite directory "busting" or searching tool is dirsearch, the author of dirsearch calls it a web path scanner.  It was originally called "dirs3arch". Dirsearch is written in python3 and follows a pretty structured coding style. (Maybe if you ever read my python code you would think that I know very little about writing beautiful and well structured code, so maybe my opinion doesn't matter in that respect.)
 Here is a link to a really cool post describing how to use dirsearch to find interesting URL's on a url shortening service. https://shubs.io/exploiting-url-shortners-to-discover-sensitive-resources-2/ . I would highly recommend reading over the entire blog of shubs.io, the guy has an interesting take on things and comes up with interesting ideas.  I like the angle he takes on things and I also like how he
comes up with new ideas.

The basic usage for dirsearch is as follows:

$ python3 dirsearch.py -u https://secure.site.com -e php  -w db/dirbuster/directory-list-2.3-medium.txt  -x 403 

You can output the stdout to a file and to stdout with:

$ python3 dirsearch.py -u http://secure.site.com -e php,html -t 5 -w db/dirbuster/directory-list-2.3-medium.txt -x 403 2>&1 | tee results1.txt

This brings me to my next point:
Where do you get wordlists for directory searching?
How to you make use of them in a way that is quick,easy,lazy and intuitive?
There isn't much written on this topic in my opinion.
Well let me show you what I prefer to do.  Daniel Miessler (cool guy with interesting opinions,also has a blog worth reading) maintains a very cool list which contains various files related to hacking web applications. I like to clone this list if I don't have it already, and if I already have it then I like to pull the latest changes. The repo has a directory for finding directories in web applications it can be found here:

$ ls Discovery/Web_Content/

A quick very hacky method we could use to create one wordlist to rule them all, would be to do the following: (The list will contain duplicates though)

$ cd ~/SecLists-master/Discovery/Web_Content/

$ cat *.txt > merged.txt

$ python3 dirsearch.py -u "http://site.com" -t 5 -w ~/SecLists-master/Discovery/Web_Content/merged.txt -e jsp,jspx -x 400 2>&1 | tee results1.txt

The problem is that we will have many duplicate entries in our file merged.txt. Thanks to the wonderful set of tools that open sorcerers have worked hard to create for us, we can remove the duplicate entries. See this example, this example is not related to finding directories yet.

$ cat /tmp/one.txt 
one 
two 
one 
three
four
four 
one
one
one
one
one
ten
eleven
five
four
six
eight
seven

We can remove the duplicate entries using the following:

$ sort /tmp/one.txt | uniq
eight
eleven
five
four
four 
one
one 
seven
six
ten
three
two 

This is obviously worth nothing if you don't save the output, let's save the output to a file so we have a file without the duplicate entries:

$ sort /tmp/one.txt | uniq > /tmp/two.txt

You can see that this file only contains the unique entries by looking at the file's contents:

$ cat /tmp/two.txt

This is just a very short post, but if you want to do some further reading on this topic then I suggest that you look at the tool cewl, it can be used to generate wordlists specific to a company or organization. I've certainly found great use with this tool, there are other tools like it out there too that might also be worth looking into.