The Problem with OSINT…
Something we as pentesters have to contend with on each of our engagements is recon. It is the nature of the beast with pentesting. Unlike Hugh Jackman, we cannot simply pull Hollywood magic out of our hats and break into networks on demand. If you want to successfully pull off the heist and get away with the loot, you need to do your homework properly first. Case the target location, gather usernames and perform email recon, credentials on rare occasions, and passively gatherer as much information on the network as possible.
Many pentesters don’t particularly enjoy this part of the engagement, save for a few rare occasions of OPSEC failure that leave us laughing and shaking our heads in disbelief (like finding cleartext credentials on a help-forum.) The process of thumbing through pages of Google search results for valuable OSINT is a dry and time-consuming process. It goes without saying that we like tools that make our lives easier.
And to make matter worse…
One thing that has grown increasingly hard with footprinting and recon is that scraping is becoming more difficult less effective. Because of a lack of support and interest, incredible tools like TheHarvester are now falling to the wayside. Search engines are aggressively patching out their traits that allow them to be scraped, and the lack of good resources continue to make our jobs harder. This has proven to be extremely frustrating as we start cobbling together our own web scrapers to substitute. Unfortunately, they don’t really compare to the old days where scraping was quite easy.
Bots and web scrapers are clearly being discriminated against. The giant spiders of the internet like Google spider, BingBot are clearly favored. Companies like LinkedIn are suing scrapers because they want to preserve their information for themselves. This is understandable, but that leaves us pentesters with little choice…
We are forced to do things…manually.
Recon Tools
We still have things like the ever-faithful Shodan and recon-ng tools to help us along the way. Unfortunately, everything we need is not always up on Shodan, and recon-ng relies on modules that expect you to give it good data. Additionally, recon-ng’s modules are mostly scrapers themselves while others require paid API tools. Once again, we must turn to scrapers or manually google-hacking information on our target. This is now how things should be.
We are hackers after all. There has to be a better way. Short answer: there is.
There are still tools out that pentesters can leverage, even if they were not designed with this purpose in mind. You can find some unconventional tools that do an excellent job of recon. These tools were not designed for a security-related purpose, but they have some extraordinarily handy applications in our field. One thing that comes to mind are sales and marketing. How do they always manage to find my email address and phone number? Well, part of it is they are really good at OSINT research. The other is, they have access to databases with all of our personal information in them.
Hey, that sounds like a pretty good idea!
Email Recon
First off, I wanted to just go ahead and mention that this was not solely my idea, and that the whole inspiration for this post goes to Rascuache for introducing me to the tool I’m about to describe. Before that, I had never even heard of this service.
Without further ado, I want to introduce Hunter.io, a marketing tool for harvesting, verifying, and generating email addresses based on domain or company name. Hunter has some pretty cool features, like email lookup and email convention guessing that will give you a pretty confident guess as to what usernames and email addresses look like within your target environment. They even go a step further by allowing you to verify these email addresses actually exist by sending a non-deliverable email and checking for a bounce.
Hunter does an excellent job of grabbing all of the relevant data on a specific target environment by the company name or domain, as I mentioned, including names, email addresses, job titles, and even social media accounts associated with that email. It even lets you export all of the output to a CSV, which you are welcome to do, but I much prefer command line to GUI for the sake of time and simplicity.
What Exactly is Hunter?
You may be thinking Hunter is a scraper, but it is not. It is actually a database of information gathered from elected sources. In most cases, Hunter has acquired this information from the email owner themselves. How they do this is a trade secret, but we can assume. As a marketing tool, Hunter is a way of finding names and contact information. This helps marketers find and target customers for sales.
Hunter.io also has a very, very nice API. The API wraps its data up in a cozy JSON package that code wizards can easily pars and dump into any file format they like. In fact, I was trying to develop a tool to do just that, until I discovered that a wrapper for this API was already available in Python, Ruby, Node.js, PHP, and Go code. You can find these wrappers on the bottom of their API page after you sign up right below your API keys.
Setting up the Script
Special thanks go to Nathan Wailes for his email_hunter.py script which does everything I was hoping to create. It is made even better for the fact that it can be easily installed using
pip3 install email-hunter-python
The usage for this tool is quite straight-forward as you can see below
After signing up for Hunter.io, you will acquire an API key. You can either throw that into a bash variable or paste directly into your command. Fair warning, if you decide to sign up for this service, you get over 200 queries per month free of charge, but you have to elect to give up your name and email. Sadly, public email service accounts are not accepted. Sorry about that.
After installing email_hunter.py, I am able to run any of the commands that are available to me on hunter.io. Except, I get a LOT more output in a hurry which I can format however I like. Hunter shows you the domain, all of the emails attached to it, the type of email it is (personal or generic), and the source hunter.io acquired it from.
As you can see, the output is a bit messy. We can clean that up with some clever one-line bash scripting. I piped the output from the email_hunter command to what will eventually become a nice, clean csv file.
my_api_key=abcdef1234567890a1b2c3d4f5
email_hunter –domain my_helpless_target.com search $my_api_key
cat target_emails.csv | cut -d ”” -f2 | uniq | sort > target_emails.csv
Sanity Check:
cat target_emails.csv | Print out Target data
| cut -f2 | cut by the default TAB delimiter, and select the second field (email addresses)
| uniq | sort | print unique items and sort alphabetically
> clean.csv | pipe and overwrite the results back into your csv (optional)
The output this generates is far prettier, grep-able, and more importantly importable output.
Wrapping Up
I can already imagine that those of you who do phishing campaigns and social engineering engagements are silently nodding your heads in approval as the use of this tool becomes immediately apparent. This tool is pretty robust, but the API for Hunter.io has a lot of functionality that this script does not cover. A few pythonic pentesters could easily take this and develop some pretty helpful scripts. I have found it extremely beneficial to use the automated mail verification and auto-importing the results them into more rigorous OSINT tools like Maltego and Recon-ng. After this step, you can port your results directly into your exploitation framework ie: Metasploit. It is equally useful to bring these results into KingPhisher if you are conducting any phishing assessments. If they weren’t already, I hope the wheels in your heads are beginning to turn.
I highly suggest looking into Hunter.io more in-depth when you get the chance. Since you’ll have to sign up to get an API key first, you might as well read up on how it works and learn some other cool things about Hunter.io.
Again, all hats off to Rascuache for suggesting Hunter.io to me and prompting my research, and to Nathan Wailes for making this handy little python wrapper for Hunter.io. I will gratefully look forward to testing it very thoroughly.
Originally authored by Mike
