Code Mirage: How cyber criminals harness AI-hallucinated code for malicious machinations

Read Time:4 Minute, 30 Second

The content of this post is solely the responsibility of the author.  AT&T does not adopt or endorse any of the views, positions, or information provided by the author in this article. 

Introduction:

The landscape of cybercrime continues to evolve, and cybercriminals are constantly seeking new methods to compromise software projects and systems. In a disconcerting development, cybercriminals are now capitalizing on AI-generated unpublished package names also known as “AI-Hallucinated packages” to publish malicious packages under commonly hallucinated package names. It should be noted that artificial hallucination is not a new phenomenon as discussed in [3]. This article sheds light on this emerging threat, wherein unsuspecting developers inadvertently introduce malicious packages into their projects through the code generated by AI.

AI-hallucinations:

Artificial intelligence (AI) hallucinations, as described [2], refer to confident responses generated by AI systems that lack justification based on their training data. Similar to human psychological hallucinations, AI hallucinations involve the AI system providing information or responses that are not supported by the available data. However, in the context of AI, hallucinations are associated with unjustified responses or beliefs rather than false percepts. This phenomenon gained attention around 2022 with the introduction of large language models like ChatGPT, where users observed instances of seemingly random but plausible-sounding falsehoods being generated. By 2023, it was acknowledged that frequent hallucinations in AI systems posed a significant challenge for the field of language models.

The exploitative process:

Cybercriminals begin by deliberately publishing malicious packages under commonly hallucinated names produced by large language machines (LLMs) such as ChatGPT within trusted repositories. These package names closely resemble legitimate and widely used libraries or utilities, such as the legitimate package ‘arangojs’ vs the hallucinated package ‘arangodb’ as shown in the research done by Vulcan [1].

The trap unfolds:

When developers, unaware of the malicious intent, utilize AI-based tools or large language models (LLMs) to generate code snippets for their projects, they inadvertently can fall into a trap. The AI-generated code snippets can include imaginary unpublished libraries, enabling cybercriminals to publish commonly used AI-generated imaginary package names. As a result, developers unknowingly import malicious packages into their projects, introducing vulnerabilities, backdoors, or other malicious functionalities that compromise the security and integrity of the software and possibly other projects.

Implications for developers:

The exploitation of AI-generated hallucinated package names poses significant risks to developers and their projects. Here are some key implications:

Trusting familiar package names: Developers commonly rely on package names they recognize to introduce code snippets into their projects. The presence of malicious packages under commonly hallucinated names makes it increasingly difficult to distinguish between legitimate and malicious options when relying on the trust from AI generated code.
Blind trust in AI-generated code: Many developers embrace the efficiency and convenience of AI-powered code generation tools. However, blind trust in these tools without proper verification can lead to unintentional integration of malicious code into projects.

Mitigating the risks:

To protect themselves and their projects from the risks associated with AI-generated code hallucinations, developers should consider the following measures:

Code review and verification: Developers must meticulously review and verify code snippets generated by AI tools, even if they appear to be similar to well-known packages. Comparing the generated code with authentic sources and scrutinizing the code for suspicious or malicious behavior is essential.
Independent research: Conduct independent research to confirm the legitimacy of the package. Visit official websites, consult trusted communities, and review the reputation and feedback associated with the package before integration.
Vigilance and reporting: Developers should maintain a proactive stance in reporting suspicious packages to the relevant package managers and security communities. Promptly reporting potential threats helps mitigate risks and protect the wider developer community.

Conclusion:

The exploitation of commonly hallucinated package names through AI generated code is a concerning development in the realm of cybercrime. Developers must remain vigilant and take necessary precautions to safeguard their projects and systems. By adopting a cautious approach, conducting thorough code reviews, and independently verifying the authenticity of packages, developers can mitigate the risks associated with AI-generated hallucinated package names.

Furthermore, collaboration between developers, package managers, and security researchers is crucial in detecting and combating this evolving threat. Sharing information, reporting suspicious packages, and collectively working towards maintaining the integrity and security of repositories are vital steps in thwarting the efforts of cybercriminals.

As the landscape of cybersecurity continues to evolve, staying informed about emerging threats and implementing robust security practices will be paramount. Developers play a crucial role in maintaining the trust and security of software ecosystems, and by remaining vigilant and proactive, they can effectively counter the risks posed by AI-generated hallucinated packages.

Remember, the battle against cybercrime is an ongoing one, and the collective efforts of the software development community are essential in ensuring a secure and trustworthy environment for all.

The guest author of this blog works at www.perimeterwatch.com

Citations:

Lanyado, B. (2023, June 15). Can you trust chatgpt’s package recommendations? Vulcan Cyber. https://vulcan.io/blog/ai-hallucinations-package-risk
Wikimedia Foundation. (2023, June 22). Hallucination (Artificial Intelligence)1. Wikipedia. https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)
Ji Z, Lee N, Frieske R, Yu T, Su D, Xu Y, et al. Survey of hallucination in natural language generation. ACM Comput Surv. (2023 June 23). https://doi.org/10.1145/3571730

Read More

USN-6267-1: Firefox vulnerabilities

Read Time:52 Second

Multiple security issues were discovered in Firefox. If a user were
tricked into opening a specially crafted website, an attacker could
potentially exploit these to cause a denial of service, obtain sensitive
information across domains, or execute arbitrary code. (CVE-2023-4047,
CVE-2023-4048, CVE-2023-4049, CVE-2023-4051, CVE-2023-4053, CVE-2023-4055,
CVE-2023-4056, CVE-2023-4057, CVE-2023-4058)

Max Vlasov discovered that Firefox Offscreen Canvas did not properly track
cross-origin tainting. An attacker could potentially exploit this issue to
access image data from another site in violation of same-origin policy.
(CVE-2023-4045)

Alexander Guryanov discovered that Firefox did not properly update the
value of a global variable in WASM JIT analysis in some circumstances. An
attacker could potentially exploit this issue to cause a denial of service.
(CVE-2023-4046)

Mark Brand discovered that Firefox did not properly validate the size of
an untrusted input stream. An attacker could potentially exploit this issue
to cause a denial of service. (CVE-2023-4050)

Read More

php-8.1.22-1.fc37

Read Time:2 Minute, 3 Second

FEDORA-2023-c68f2227e6

Packages in this update:

php-8.1.22-1.fc37

Update description:

PHP version 8.1.22 (03 Aug 2023)

Build:

Fixed bug GH-11522 (PHP version check fails with ‘-‘ separator). (SVGAnimate)

CLI:

Fix interrupted CLI output causing the process to exit. (nielsdos)

Core:

Fixed oss-fuzz php#60011 (Mis-compilation of by-reference nullsafe operator). (ilutov)
Fixed use-of-uninitialized-value with ??= on assert. (ilutov)
Fixed build for FreeBSD before the 11.0 releases. (David Carlier)

Curl:

Fix crash when an invalid callback function is passed to CURLMOPT_PUSHFUNCTION. (nielsdos)

Date:

Fixed bug GH-11368 (Date modify returns invalid datetime). (Derick)

DOM:

Fixed bug GH-11625 (DOMElement::replaceWith() doesn’t replace node with DOMDocumentFragment but just deletes node or causes wrapping <> depending on libxml2 version). (nielsdos)

Fileinfo:

Fixed bug GH-11298 (finfo returns wrong mime type for xz files). (Anatol)

FTP:

Fix context option check for “overwrite”. (JonasQuinten)
Fixed bug GH-10562 (Memory leak and invalid state with consecutive ftp_nb_fget). (nielsdos)

GD:

Fix most of the external libgd test failures. (Michael Orlitzky)

Hash:

Fix use-of-uninitialized-value in hash_pbkdf2(), fix missing $options parameter in signature. (ilutov)

Intl:

Fix memory leak in MessageFormatter::format() on failure. (Girgias)

Libxml:

Fixed bug GHSA-3qrf-m4j2-pcrr (Security issue with external entity loading in XML without enabling it). (CVE-2023-3823) (nielsdos, ilutov)

MBString:

Fix GH-11300 (license issue: restricted unicode license headers). (nielsdos)

Opcache:

Fixed bug GH-10914 (OPCache with Enum and Callback functions results in segmentation fault). (nielsdos)
Prevent potential deadlock if accelerated globals cannot be allocated. (nielsdos)

PCNTL:

Fixed bug GH-11498 (SIGCHLD is not always returned from proc_open). (nielsdos)

PCRE:

Mangle PCRE regex cache key with JIT option. (mvorisek)

PDO:

Fix GH-11587 (After php8.1, when PDO::ATTR_EMULATE_PREPARES is true and PDO::ATTR_STRINGIFY_FETCHES is true, decimal zeros are no longer filled). (SakiTakamachi)

PDO SQLite:

Fix GH-11492 (Make test failure: ext/pdo_sqlite/tests/bug_42589.phpt). (KapitanOczywisty, CViniciusSDias)

Phar:

Add missing check on EVP_VerifyUpdate() in phar util. (nielsdos)
Fixed bug GHSA-jqcx-ccgc-xwhv (Buffer mismanagement in phar_dir_read()). (CVE-2023-3824) (nielsdos)

PHPDBG:

Fixed bug GH-9669 (phpdbg -h options doesn’t list the -z option). (adsr)

Session:

Removed broken url support for transferring session ID. (ilutov)

Standard:

Fix serialization of RC1 objects appearing in object graph twice. (ilutov)

SQLite3:

Fix replaced error handling in SQLite3Stmt::__construct. (nielsdos)

Read More