In recent weeks ChatGPT by OpenAI started making waves around the world for its genuinely groundbreaking capabilities. It is a powerful language model capable of performing tasks that would take multiple people days of work in minutes or even seconds. All it needs are questions in the native human language of your choice. Researchers and public enthusiasts alike have been able to use this tool for anything from writing children’s fairy tales and tailored weight loss plans to merging excel sheets or coding entire apps. We are going to explore its potential for cybersecurity, both defence and offence and see if we need to worry about our jobs.
Once there were first headlines and rumours in the cybersecurity community about ChatGPT and its capabilities, CYFIRMA researchers also gave it a fair shake. The user interface of the free research preview is very simple and doesn’t require any learning curve. It welcomes you with a few words about what ChatGPT is and that it has been restricted for safety. Importantly it is ‘not connected’ to the Internet and unable to search for anything on it. All answers come from its learned knowledge, which was cut off in 2021, hence it doesn’t know anything that happened recently.
One of the safety guardrails is that it won’t provide any malicious code or otherwise dangerous content, such as personal information about people. While we didn’t explore the latter, we were able to get around the former. This is where the most powerful and impressive feature of ChatGPT comes in. It remembers everything within the thread and follows up on both user and its own output, which can be used against its safety restrictions.
For this exploration report, we set to achieve some defensive and offensive cybersecurity tasks. For defence, we wanted to see if it can give us short analysis tools and scripts. And if it knows cybersecurity industry standards like YARA or Snort. For offence, we simply tried to make it write basic malware for us.
And the takeaway is that it very easily can do it all.
We were so impressed with the results that we decided to let the AI speak for itself through a lot of lengthy screenshots. As you will see, it only takes simple English to get desired results.
We have reviewed the code ChatGPT wrote and most of the simple scripts worked on the first try. While the more complex code had some issues, the AI was able to address all of them and make the code work when asked about it. And since this is already a very long and screenshot-heavy report, we decided to omit the troubleshooting for clarity. We encourage you to explore reports from programmers and developers on this topic.
Every cybersecurity professional has a need to whip out quick scripts for things like parsing logs or automating tasks. We started with something basic like grep.
In seconds ChatGPT returns working syntax and an explanation of how the command works. However, grep and regex aren’t that complicated for machines to learn and provide ready-to-use short commands. So, let’s try something more complex, like a python script to run a very basic sample analysis on contained strings.
Once again, in a matter of seconds, we got a working script with a description of its functionality. However, in the case of larger files, this is not the most efficient approach. By simply asking the AI to give us something more advanced it recognized the efficiency issue and returned an updated version.
Now let us take it further and add some complexity. We asked AI to identify base64 encoding, decode it and then search for strings. And it was delivered right away. Later, after finishing this conversation thread, we returned to this code and played with it some more. We tried to add URL encoding and XOR and while we received the requested functions, it needed some back and forth with the AI and little Python knowledge to fix the code. So, it is not a magic solution for coding without any knowledge at all.
Note that this is where we are starting to receive warnings as ChatGPT has some safety guardrails in place. Also, we are running out of screen real estate for convenient screenshots, so text size is going to vary going forward.
As demonstrated, ChatGPT has no problem with writing simple but useful scripts. Let’s switch to some more direct cybersecurity stuff, starting with writing Snort rule.
It clearly understands what Snort rules are and how they are used in context. By simply asking for more examples using some of the most common patterns it returns an exhaustive list of scenarios. Note that there is a length limit per answer, so it was cut off. Thankfully ChatGPT remembers the entire thread, so you can tell it to continue to finish its answer.
This is impressive and incredibly useful for both learning and direct application by SOC analysts or security engineers.
Now to test SIGMA and YARA rules. For that, we wanted something more complex than simple outbound traffic and went with some Windows LOLBINs detection. Using scheduled tasks as an example, ChatGPT knows very well what it is and how it is used.
Also testing AI on its ability to work with the previous context we pointed to its own answer to see if it can identify the command, it mentioned and write SIGMA or YARA rules for it. Note that this is still a single thread and this exchange with AI is here in its entirety, just chopped into screenshots.
Another flawless victory for the AI. It was able to identify exactly what command we meant in it its previous answer and write SIGMA and YARA rules for it despite triggering a content warning for SIGMA. It understands potentially malicious commands and can break them down to create a reasonable detection logic.
Obviously, in practice, there are many variables to fine-tune, most commonly around false positive detections. We asked for adjustments to avoid false positives and AI instead of giving out a new version of the rule provided well written advice on how to go about it.
Reading through the advice it looks like ChatGPT really understands the nuances of false positive detection and that is where AI provides value over the usual tools.
Windows scheduled tasks are still easy enough. Every junior analyst should be able to see that and quickly write a simple detection rule for it. How about DLL side-loading? Apparently no problem.
Please note that we are still asking really simple questions in plain English. This AI language model has a powerful ability to follow up with increasingly specific and detailed queries and get down to working on complex tasks that require advanced skills and a lot of manhours. While it is far away from being able to automatically solve any complex problems and tasks, it is a tremendously powerful personal assistant that can help with a lot of tedious tasks or point in the right direction when stuck.
Before we jump into the offensive part, we need to point out one thing. ChatGPT has some limits in place to avoid giving out malicious code right away. However, we could talk around it by asking benign-looking questions exploring cybersecurity topics and concepts. The model learns from user input within the thread, and you can convince it to change its ‘mind’ even
for mathematical questions.
As per the above-mentioned safety in place, if asked directly to write malware, it will politely refuse to oblige. However, we can start a related conversation and then steer it into what we want.
Here we asked in a new conversation thread for malware.
Here we continued in our previous defence thread, and it had no problem to keep going.
As you can see it is missing the Free the DLL part and it is chopped. By simply asking it to use the code box, it will return a directly usable code.
With the same logic of starting a benign but related conversation, we initiated a chat about FIN7 group and what TTPs they use. And eventually, we were able to obtain complete basic Trojan malware code for Windows, Android and iPhone.
Once AI mentioned in our FIN7 conversation a Trojan virus and some WMI commands, we simply asked it to give us an example of Trojan code. It obliged with a warning and then happily provided a more advanced version of a Python-based Trojan that is ready to receive WMI commands. As a cherry on top, it was also able to provide WMI commands to use with the code right away.
Once we had a basic Python script for Windows, we asked for an Android version without any further specification, and we received a Java-based code in a matter of a minute.
With a base code in hand, we tested adding complexity. To do that we requested a keylogger function that will send the keystrokes to the attacker. As an unintended bonus, it had no issue with missed word “new” in the command and did what we really meant.
Happy with the result, we switched platforms once more and asked if we can have an iPhone version. The first response wasn’t satisfying but provided nice additional context to understand the challenges with iPhone along with a safety warning. Upon asking for a more advanced version, it was able to return the complete code.
Similarly, to the Android code scenario, we tested adding more complexity. This time we asked for stealing photos, videos and documents from the iPhone. Once more ChatGPT gave us advice on challenges with writing malicious code for iPhone. It is well-meant advice, but should we really want to make actual malware, it could be easily used to troubleshoot and work around these challenges. In other words, to write an exploit.
Note that at this point the code became too long, so for clarity and to avoid giving out entire potentially malicious code, we are showing only a snippet that fit the screen.
At this stage, we have proven what we set to do and more. ChatGPT is indeed capable of writing malware by only using simple plain English input to get there. Striking is the AI capability to instantly convert code not only into a different programming language but to retain the function of the code when changing platforms. All while providing helpful advice and context on challenges with the conversion and explaining how everything works.
To finish this exercise, we returned back to our Android Trojan and started specifying some functions to see how it will handle further building the code from a simple Trojan sample code to usable working code. And as per the snippet below, it had no issues to keep building the code.
Leaving the Skynet-driven end of the world aside and staying in the context of cybersecurity it is a tricky question to answer. It is a powerful tool for everybody and as history has shown, such tools do create an advantage for those who learn it over those who won’t or can’t.
Can it be pointed at a server to find and exploit vulnerabilities?
No, and yes.
It is ‘not connected’ to the internet and can’t be pointed at anything. As for finding vulnerabilities, exploiting them etc., based on what we have seen ourselves, it absolutely has that potential. But it still requires a skilled and experienced operator to get into that level of complexity.
Both sides of the never-ending cyber arms race have equal access to this tool at the moment, so neither has gained a great advantage over the other.
ChatGPT is an impressive and genuinely disruptive technology for years to come. While it still has its kinks and only the basic scripts worked on the first try, it was able to quickly fix the more complex ones, when we came back asking about errors returned.
Should we be worried about our jobs? Not at this point, but we should certainly start learning this tool to get it to help us at our jobs. And more importantly to make sure we are on top when it inevitably ends up in hands of malicious actors. While powerful, it still needs a human operator. It makes mistakes and is very confident about them. It is also naïve and trusting, making it prone to be bamboozled by intentionally misleading input to make it do things it wasn’t really supposed to.
The most amazing and scary part about this exercise is the fact this is only the beginning. AI’s learning curve is exponential, meaning it took a long time to get here, but it will take a fraction of the time to double its capacity. Imagine where it will be next year and then in 5 years.
As we saw, AI can drastically save time and increase individual productivity. With such power at our fingertips, the creative potential, used for good or bad, is truly mind-blowing.
