PRESENTED BY

Cyber AI Chronicle

By Simon Ganiere · 25^th February 2024

Welcome back! This week newsletter is a 15 minutes read.

Project Overwatch is a cutting-edge newsletter at the intersection of cybersecurity, AI, technology, and resilience, designed to navigate the complexities of our rapidly evolving digital landscape. It delivers insightful analysis and actionable intelligence, empowering you to stay ahead in a world where staying informed is not just an option, but a necessity.

Table of Contents

• What I learned this week

  • TLDR;

  • Adversarial Machine Learning: Third Party View

  • Specific Measures

• Worth a full read

  • Google: How AI can Strengthen Digital Security?

    • Key Takeaway

  • Sygnia: Annual Field Report

    • Key Takeaway

  • LLM Agent can Autonomously Hack Websites

    • Key Takeaway

• Some more reading

• Wisdom of the week

• Contact

What I learned this week

TLDR;

• Continue my learning on Adversarial Machine Learning, this week focusing on the third party angle. Summary is below.

• I continued to play around with the AI Agent tools that I mentioned last time. I continue to be impressed with the possibilities this is going to create! This is not about replacement this is about augmentation. I’m sometime wondering if we should not be talking about Augmented Intelligence rather than Artificial Intelligence.

• I’m reading quite a few books on AI. Books like Life 3.0, Our Final Invention, Human Compatible: AI and the Problem of Control, Superintelligence. Gotta say some of those books are not super positive about the future 😅 Let’s hope not all of those scenarios will play out and AI will actually improve the world!
  If you have any good book recommendation ping me a note!

Adversarial Machine Learning: Third Party View

Last week we introduce the topic of Adversarial Machine Learning (AML) with some basic taxonomy alignment. I was curious about the third party angle as it usually add a good level of complexity from a security perspective…so I decided to dig a little bit deeper. By no means this is a complete guide on either AI risk management or third-party management but it should give you some pointers.

Trying to frame the problem, let’s start by defining what we are talking about in the context of AI:

• Third-party components such as libraries, frameworks, APIs, and datasets in order to develop and run your own models; and

• Machine Learning as a Service (MLaaS), there are subcategories here as you can go for offering where you literally consume a read-to-go model via API or using an SDK. You can leverage a ML platform such as AWS SageMaker or Microsoft Azure Automated ML or Google AutoML. Last but not least you can have Machine Learning infrastructure, so dedicated compute power that has optimised for ML, up to you to develop and build on top of it. An example of this is AWS EC2 DL1 instances that relay on Gaudi accelerators from Habana Labs. The other cloud providers have similar offering.

Obviously, your setup, build vs. buy, will define your potential exposure. If you build your AI stack from scratch, then the focus is on vulnerabilities management. Those third-party libraries and framework obviously have bug and vulnerabilities that need to be managed. If you leverage other third-party components like third party datasets to train your model then data poisoning is on the list of threat and you should spend more time on data security and model validation. Last but not least, API security is critical. Either because you expose your own API or also if you consume a third party API. Those can get compromise and/or abuse and leading potentially to model stealing.

Specific Measures

At a high level, the overall approach should not change. The mitigation measure we are talking about here should be known and already being in used in any mature security environment. That being said, like with any new technology, whoever is driving those measures need to have a good understanding of the actual technology so they can adapt the approach as and when necessary.

1. Risk Management Framework: We are not covering that part in full here but you for sure need to adopt an overall framework from an AI perspective.

2. Model validation: If you rely on a third party model, you need to be in a position where you can validate that model for accuracy and robustness. This probably means you need to thoroughly test the model.

3. Vulnerability Scanning and Patch Management: Regularly scan third-party libraries and frameworks for known vulnerabilities using tools designed for dependency checking. Ensure that all third-party components are up to date and apply patches or updates as soon as they become available. This has to be part of your basic security controls!

4. Data Integrity Checks: Perform rigorous validation and integrity checks on third-party datasets before using them for training machine learning models. Employ techniques like anomaly detection to identify and mitigate data poisoning attempts.

5. API security: When using third-party APIs for model serving, ensure that the APIs are secure and implement adequate access control and authentication mechanisms. The whole push for AI Agents is going to make API security extremely important.

6. Third-party components assessment: Conduct thorough review of the third-party components used in machine learning workflows. This includes evaluating the security posture of third-party providers, especially in the case of MLaaS, and understanding the potential risks associated with it.

7. Contractual Agreements and Audits: Establish clear contractual agreements with third-party providers that outline security requirements and responsibilities. Regular audits and compliance checks can help ensure that third-party components meet security standards.

8. Incident Response Plan Adaptation: For those of you who are involved with incident response, you know that responding to a third-party breach adds a new level of complexity to an already complex situation. Top it with an AI stack and you have a perfect storm. Ensure your third party response plan is up to date and ensure you have a plan to response to AI stack incident. Last but not least, ensure your response team has a good understanding of how AI stack works and how threat materialised in such an environment.

Conclusion

Given the increasing complexity and interconnectedness of machine learning ecosystems, third-party components play a critical role in the security posture of machine learning applications. Whilst there are some specific you should be able to leverage your actual third-party risk management approach to cover a lot of the topics.

I did run a small survey on LinkedIn earlier this week. Lots of people focus on model validation - which make sense - but do not forget those APIs! (and glad nobody thought this is about some AI magic 😆 )

Worth a full read

Google: How AI can Strengthen Digital Security?

Key Takeaway

• AI represents a turning point in digital security, offering solutions to long-standing challenges like the "Defender's Dilemma."

• Strategic investments in AI infrastructure and startups are critical for building a secure digital ecosystem.

• The Secure AI Framework and open-source tools like Magika are pivotal in making AI security technologies robust and effective.

• Collaboration between industry, government, and academia is essential for advancing AI-powered cybersecurity research and development.

• A balanced regulatory framework is necessary to ensure AI's potential in cybersecurity is maximized while minimizing risks.

Note: I don’t disagree with most of the roadmap items (page 40 of the full report) however some of the key enablers are going to be a real challenge. Anyone who worked in a big corporate knows that “[…] rapid procurement and deployment[…] ” procurement is not rapid, neither is having “[…] international framework[…] ” or “[…] prevent opt-outs on security[…]”, etc. all of this is the same reason why big corporate struggle to patch, have too much legacy infra, exceptions everywhere for security controls or all sorts of cross-border data challenges (hello my lawyer friends) especially for regulated industries.

Sygnia: Annual Field Report

Key Takeaway

• Ransomware groups are shifting from encryption to data exfiltration and extortion for quicker monetization.

• Bypassing Multi-Factor Authentication (MFA) has become a common tactic among cybercriminals.

• Identity and cloud-based breaches are increasing, exploiting unmonitored IT areas for lateral movement.

• The simplicity of major cyberattacks in 2023 suggests a "return to basics" in cybercrime strategies.

• AI's role in cybersecurity is growing, potentially changing attack execution speed and targeting capabilities.

• Executives are advised on building security teams, prioritizing budgets, and measuring cybersecurity effectiveness.

• The legal landscape is evolving with the CISO's role, SEC disclosure requirements, and third-party attack trends.

• Practical defense strategies are emphasized over new technology investments for robust cybersecurity.

• The cryptocurrency industry faces increased attacks due to its monetary potential and regulatory gaps.

• Anticipated 2024 trends include sophisticated cloud and identity attacks, AI in cyber intrusions, and new regulations

LLM Agent can Autonomously Hack Websites

Key Takeaway

• GPT-4's advanced capabilities enable it to autonomously hack websites, showcasing a potential cybersecurity threat.

• The inability of open-source models to perform similar hacks suggests a gap that may close with further development.

• The economic feasibility of using LLM agents for hacking highlights a shift in the cybersecurity landscape.

• The study's responsible approach to experimentation and disclosure emphasizes the ethical considerations in cybersecurity research.

• The findings underscore the importance of cautious model deployment and the potential need for regulatory measures.

Some more reading

High vulnerabilities Disclosed in the AI Development supply chain…so traditional security is still relevant I guess 😉 » READ

In relation with the few weeks reports on deepfake Banking trojan collects face and other data for bank account hacking  » READ & » READ

This one read like a sales pitch but I really like the “risk based alerting” approach. At time of high workload, shortage of SOC analysts, alert fatigue, taking such approach is key » READ

Aligned with last week reporting from Microsoft and OpenAI, The Financial Times is coming back with more details on how North Korea is leveraging AI technologies » READ (behind paywall, you can use the usual solutions to go around it).

Clarity, which enables organisations to rapidly identify and immunise from Deepfake and synthetic media created by generative AI, raised $16M in seed funding. » READ

Anthropic, who is behind Claude takes steps to prevent election misinformation » READ

Softbank founder, Masayoshi Son, is seeking as much as $100 billion to launch a chip venture to compete with Nvidia » READ 

LockBit ransomware gang disrupted by international law enforcement operation…we know they will most probably come back but the numerous law enforcement actions raise the pressure on cyber criminals » READ

On a lighter note…seems iPhones are not eating rice for their lunch » READ

Wisdom of the week

Contact

Let me know if you have any feedback or any topics you want me to cover. You can ping me on LinkedIn or on Twitter/X. I’ll do my best to reply promptly!

Thanks! see you next week! Simon