A given IOC comes to your hands, for instance a hash, URL, IP or suspicious domain. You need to find out some basic information. Is it malware? Is it in any repository? Since when? Whois? Source country? Is it in pastebin?
You start to open tabs, to enter passwords in the different services and launch queries. Hopefully you have an API shared with a co-worker and, after checking several systems, you open a TXT to recopy the data to the intelligence platform. Your co-worker, who you share those APIs and passwords with but who is on your computer elsewhere in the world, does the same because the same IOC has also reached their hands. This is over with TheTHE.
The Threat Hunting Environment
We recently presented this tool at the Black Hat 2019 in London, where it was well received among the target audience: researchers, SOCs, teams, security companies, CERTs, etc. TheTHE is an environment intended to help analysts and hunters over the early stages of their work to make it easier, faster and more unified. One of the major drawbacks when hunting or investigating IoCs is dealing with the initial collection of such amount of information coming from so many sources, public and private.
All this information is usually scattered and sometimes even volatile. Perhaps at a certain point there is no information on a particular IOC (Indicator of Compromise), but that situation may change within a few hours and become crucial for the investigation. Based on our experience on Threat Hunting, we have created a free and open source framework to make the early stages of the investigation simpler:
- IoCs are yours: never shared out of your platform.
- Free as in beer, free as in freedom: dockerized and totally yours.
- Server-client architecture: investigation may be shared with your team.
- Results are cached so no repeated API calls are used.
- Feeds better your Threat intelligence Platform: TheTHE enables performing easier and faster previous research.
- Easy plugins: whatever is needed, it may be easily embedded within the interface.
- Ideal for SOCs, CERTS or any team.
- APIkeys are stored in a database and may be shared by a team from a single point.
- Automation of tasks and searches.
- Rapid API processing of multiple tools.
- Unification of information in a single interface: so that screenshots, spreadsheets, text files, etc. are not scattered.
- Enrichment of collected data.
- Periodic monitoring of a given IOC in case new information or related movements appear.
- IOCs do never leave your environment!
TheTHE has a web interface where the analyst starts its work by entering IOCs that will be sent to a backend, where the system will automatically look up for such resource (via plugins) on the various configured platforms in order to obtain unified information from different sources and access related reports or data existing on them.
Details of this tool
TheTHE is an open source and modular framework developed in Python 3 that allows to locally consolidate and analyze information on a MongoDB database, without sharing such information with other platforms until it is appropriately organized, linked and synthesized, which allows the information to be analyzed later in any other platform (like a Threat Intelligence Platform) in the most enriched way possible. It is a unique tool within its category that helps analysts and hunters to to carry out their research tasks in a more agile and practical way.
TheTHE is a framework that runs locally in your own system or local server. It currently has passive modules for information collection as well as active modules, which in turn allow to:
- Obtain information automatically from multiple public and private sources (by configuring users’ own accounts and configurable APIs) such as Hunter.io, Maltiverse, Shodan, Sherlock, etc.
- Execute tests and consolidate information from other tools such as Cansina, etc.
How does it work?
TheTHE is based on projects. A project is a container of related IoCs, for example. It creates a project for a set of users or for a specific research.
In each project, there are six main menus based on the initial IoC you are working with. According to the IoC entered, TheTHE will try to classify it into the appropriate menu:
- Network: basically, IP addresses.
- Domain: domains only, any TLD.
- URL: if your domain has a path, then is an URL.
- Hash: any hash, MD5, SHA1 and SHA256.
- Emails
- Usernames: any string that is not in any other category will be treated as a username.
If you want to force something as a specific kind of IOC, choose it here.
Within each menu, the minimum information required to process the information will be loaded. Within it, you can choose the appropriate plugins for each category that may be applied to each IoC.
For Network:
For Domains:
For URL:
For Hash:
For Emails:
For Usernames:
When a plugin is being used, the task will be queued, and results will be displayed when the necessary information is retrieved. All tasks will be queued asynchronously so work can continue while the results are calculated. In case of using third-party services on the network that need to consume APIs with APIkeys, these will be stored on the main server and all users will be able to use them remotely. The results will be cached indefinitely so as not to use requests if they have already been made by another team member. The results may be refreshed on request.
Depending on the plugin output, a new tag will be created once the IoC is selected. Tags may be used to better categorize your work (with colors and names you can choose). Tags will be available for the same project once created.
Future work
- Monitoring specific IOCs programmatically under platforms in case new data may appear in the future.
- Monitoring changes in the infrastructures under investigation in case of failure or if new threats appear within.
- Keeping a local record of the investigations performed.
- Accessing information in a consolidated way on a local DB from a web interface.
- Storing securely the various APIKeys and the pre-configuration of queries from dozens of public and private platforms.
- Many more plugins!
The post TheTHE: The Threat Hunting Environment, our tool for researchers appeared first on Think Big.