IA x pickle: vulnerability or feature?

Artificial intelligence is on a meteoric rise, revolutionizing fields as strategic as healthcare, finance and cybersecurity. It is now an integral part of our daily lives.
This rapid evolution inevitably attracts the covetousness of malicious actors eager to exploit its flaws.

From adversarial attacks designed to deceive models, to cyberattacks exploiting their vulnerabilities, to the misuse of AI to generate disinformation or sophisticated deepfakes, the threats are constantly growing. The more AI becomes part of our daily lives, the more crucial it becomes to strengthen its security in order to preserve its integrity.

Artificial intelligence models are the target of numerous threats that exploit their technical vulnerabilities.
Among the most widespread are:

• Adversarial attacks aim to disrupt their predictions by manipulating inputs. For example, misleading a facial recognition system with imperceptible modifications.
• Data poisoning attacks seek to alter their learning by injecting biased data, thus compromising their reliability.
• Model extraction attacks enable attackers to reproduce a model by interacting massively with it, threatening the confidentiality of the algorithms.
• Data leaks represent another major threat, where attackers exploit a model to retrieve sensitive information.
• Some models can also be hijacked for malicious purposes, such as generating sophisticated deepfakes or automating cyberattacks.

These threats underline the need to strengthen template security and deployment methods, and to improve detection of potential attacks.

The cyber world has also seen attacks evolve in recent years. The cyberthreat panorama published by ANSSI in 2022 predicted a drastic increase in blockchain attacks. A reality that is now affecting the field of artificial intelligence.

These Supply Chain Attacks in the context of AI models aim to compromise the model development, distribution or execution process.
One such attack technique is to exploit Pickle files, a Python serialization format commonly used to store and share AI models.

But how vulnerable is this functionality?

Like other protocols such as TCPIP or CANBUS of the time, the creation of the pickle file format didn’t take security by design into account, since it was designed above all to function, be open and easily exchangeable.
So it’s a very permissive format. The `.pickle` or `.pkl` can be used to save Python objects, including machine learning models. However, it directly executes the code contained in the file when deserialized, making it vulnerable to attack.

This is where Remote Code Execution (RCE) comes into play.
This is a first-rate tactic, since an attacker can inject malicious code into a pickle file, which is then automatically executed when the file is loaded (pickle.load()). Indeed, the call to pickle.load() instantiates and executes the file’s __reduce__() function.

At this point, there is no control over what is executed. This means that system commands can be launched to delete files, steal data or install malware such as stealer, trojan, loader or even ransomware.

Scenario: Distribution of a malicious model as a primary infection

An attacker targets companies wishing to integrate a pre-trained model to create a chat-bot feature, such as “meta-llama”.
Such a company might, for example, want to implement this type of service on-premise to protect itself from the data leakage inherent in the use of online products such as Deepseek chat or ChatGPT.

1/ Creating a malicious template

The attacker creates a fake template, with a similar name such as “faceboookLlama”, in scikit-learn (or other ML frameworks) and injects a payload into it.

This faceboookLlama.pkl file contains code that will be executed as soon as it is loaded with `pickle.load()`.

2/ Distribution of the infected model

The attacker has this package hosted on Hugging Face, or other popular platforms, and makes it attractive by:

• using a package name close to the legitimate one.
• copying the official documentation of the real project.
• adding keywords and tags to trick searchers.

3/ Operation: Loading by the victim

The user or company wishing to set up a local chatbot service downloads and loads the model:

⚠️ Consequences ⚠️

The faceboookLlama.pkl file triggers os.system().
A remote malicious script (in our example a simple reverse shell) is then downloaded and executed without the user’s knowledge.
The attacker takes control of the target machine and can exploit a privilege escalation vulnerability (TA0004) to perform persistence (TA0003) and lateralize (TA0008) his attack.
The attacker then has free rein to find other targets to deploy his final effects, such as stealers or ransomware.

A hybrid approach can be adopted to combine safety and performance.
We can try to implement some of these best practices.

Decouple model weights from business logic:

– Store weights in .npy or .h5 (avoids code execution).
– Store architecture and hyperparameters in JSON (secure format).
– Prefer ONNX or other standardized formats where compatible.

Enhance security with threat detection solutions like those we offer at GLIMPS :

– Advanced threat detection with GLIMPS Malware analysis solutions, to identify malicious payloads hidden in Pickle files.
– Restrict deserialization with a strict whitelist of accepted objects.
– Prohibit the execution of reduce() and any unintended code.

Automate threat detection in a CI/CD pipeline with GLIMPS Malware:

– Integration of GLIMPS solutions into CI/CD pipelines to automatically scan all files before they are used.
– Validation of templates via digital signatures and integrity checks, preventing unauthorized modification of template files.
– Run in a controlled environment with automated testing and sandboxing mechanisms to evaluate templates before deployment.

The combination of these strategies secures the entire ML model processing chain, from weight storage to production deployment, drastically reducing the risk of deserialization attacks.

Would you like to find out more? Contact our teams today by clicking on this link.