Bigdata tools in the pentester's toolbox

For both penetration tester's and blackhat's it is important to always stay on top in terms of learning new things and staying ahead of the curb in this very competitive industry. Blackhat's tend to be better at staying on top of their ball game compared to penetration testers who are really just always a step behind (this is a rash generalization, but from my experience this is the reality). A trend that I've been seeing a great deal is that many companies are moving a great deal of their assets to the Cloud based hosting providers.  This is a challenge for pentesters who are used to environments where everything is hosted on premise or in conventional data centers. To stay on top of times I suggest that you start using some cloud related technologies in your penetration testing workflow. Also while we are seeing this huge move toward the "Cloud" we are also seeing a huge bubble in the Big Data world, the infosec world is slowly also moving towards being more cloud orientated.

What are some big data tools that you might want to have in your pentester toolbox and why?
1. Elasticsearch:

• For very obvious reasons Elasticsearch with an ingester plugin such as Logstash or Fluentd is very good for log management and analyzing logs for security related reasons.
•  Elasticsearch is a nice search engine for web application penetration testing, when used in combination with Burp Suite. See this article that I wrote as a ghostwriter for qbox on the topic: https://qbox.io/blog/elk-penetration-testing-workflow-elasticsearch-python
• Many people are using Elasticsearch these days in PHP web applications. As Elasticsearch is used more and more by developers we will see more and more occurences of noSQL injection attacks or what some people like to refer to as Elasticsearch injection. Elasticsearch has a very weird security model and at the time of writing ,unless you have the X-pack plugin installed you won't have the ability to restrict data in your index to specific users. (Correct me if I'm wrong on this one). Here are some interesting things to read regarding Elasticsearch injection: https://2016.zeronights.ru/wp-content/uploads/2016/12/Hacking-ElasticSearch.pdf also see: https://www.amazon.com/NoSQL-Injection-Elasticsearch-Gary-Drocella-ebook/dp/B00TG0KN7U 
• There have been several CVE's for Elasticsearch. They are all worth reading about.

2. Dynamodb:

• I can't say much about this, but Dynamodb has a very cool feature where you can "alert" on changes to your table. I'm not going to go into much detail on this topic, but tldr is that you need to enable streams and triggers on the stream. This could be an amazing feature to use for example if you are logging changes to your domain controller to a Dynamodb table. (Hope I'm not ruining anyone's startup idea right there.)
• Another interesting use case would be too "alert" every time a new IP tries to ssh into one of your boxes. 

3. Hadoop:

So before I carry on rambling take note that I havn't mentioned hadoop yet, which is strange since no Big Data conversation is complete without the words "hadoop" or "Map reduce" being mentioned. So far the topics I've covered are what are considered "analytics" Big data products. Hadoop is most certainly something useful to be familiar with, but in my opinion we might see something else come to replace the Hadoop ecosystem in the next five years. One thing I've already spotted an open source project that has a great deal of potential in terms of replacing the hadoop eco system. This is the project if you would like to have a look: http://pachyderm.io/. Hadoop is a cool topic because it is very widely used not only in the cloud , but also in one site deployments at many big financial deployments. There is very little research from the security community on the topic of Hadoop in my opinion, or atleast not that I am aware of.

4. MongoDB:

Mongodb is a really cool topic. I could write an entire blogpost just about Mongodb and interesting things that I've seen around mongodb. Where do I even start?