{"id":8505,"date":"2023-11-22T12:00:00","date_gmt":"2023-11-22T10:00:00","guid":{"rendered":"https:\/\/blog.eset.ee\/et\/2023\/11\/22\/your-voice-is-my-password-the-risks-of-ai-driven-voice-cloning\/"},"modified":"2023-11-22T12:00:00","modified_gmt":"2023-11-22T10:00:00","slug":"your-voice-is-my-password-the-risks-of-ai-driven-voice-cloning","status":"publish","type":"post","link":"https:\/\/blog.eset.ee\/et\/en\/2023\/11\/22\/your-voice-is-my-password-the-risks-of-ai-driven-voice-cloning\/","title":{"rendered":"Your voice is my password \u2013 the risks of AI-driven voice cloning"},"content":{"rendered":"

The recent theft of my voice<\/a> brought me to a new fork in the road in terms of how AI already has the potential of causing social disruption. I was so taken aback by the quality of the cloned voice (and in that extremely clever, yet comedic, style by one of my colleagues) that I decided to use the same software for \u201cnefarious\u201d purposes and see how far I could go in order to steal from a small business \u2013 with permission, of course! Spoiler alert: it was surprisingly easy to carry out and took hardly any time at all.<\/p>\n

“AI is likely to be either the best or worst thing to happen to humanity.” \u2013 Stephen Hawking<\/p><\/blockquote>\n

Indeed, since the concept of AI became more mainstream in fictional films such as Blade Runner and The Terminator, people have questioned the relentless possibilities of what the technology could go on to produce. However, only now with powerful databases, increasing computer power, and media attention have we seen AI hit a global audience in ways that are both terrifying and exciting in equal measure. With technology such as AI prowling among us, we are extremely likely to see creative and rather sophisticated attacks take place with damaging results.<\/p>\n

Voice cloning escapade<\/h2>\n

My previous roles in the police force<\/a> instilled in me the mindset to attempt to think like a criminal. This approach has some very tangible and yet underappreciated benefits: the more one thinks and even acts<\/em> like a criminal (without actually becoming one), the better protected one can be. This is absolutely vital in keeping up to date with the latest threats as well as foreseeing the trends to come.<\/p>\n

So, to test some of AI\u2019s current abilities, I have once again had to take on the mindset of a digital criminal and ethically attack a business!<\/p>\n

I recently asked a contact of mine \u2013 let\u2019s call him Harry \u2013 if I could clone his voice and use it to attack his company. Harry agreed and allowed me to start the experiment by creating a clone of his voice using readily available software. Luckily for me, getting hold of Harry\u2019s voice was relatively simple \u2013 he often records short videos promoting his business on his YouTube channel, so I was able to stitch together a few of these videos in order to make a good audio test bed. Within a few minutes, I had generated a clone of Harry\u2019s voice, which sounded just like him to me, and I was then able to write anything and have it played back in his voice.<\/p>\n

To up the ante, I also decided to add authenticity to the attack by stealing Harry\u2019s WhatsApp account<\/a> with the help of a SIM swap attack<\/a> \u2013 again, with permission. I then sent a voice message from his WhatsApp account to the financial director of his company \u2013 let\u2019s call her Sally \u2013 requesting a \u00a3250 payment to a \u201cnew contractor\u201d. At the time of the attack, I knew he was on a nearby island having a business lunch, which gave me the perfect story and opportunity to strike.<\/p>\n

The voice message included where he was and that he needed the \u201cfloor plan guy\u201d paid, and said that he would send the bank details separately straight after. This added the verification from the sound of his voice on top of the voice message being added to Sally\u2019s WhatsApp thread, which was enough to convince her that the request was genuine. Within 16 minutes of the initial message I had \u00a3250 sent to my personal account.<\/p>\n

\"Figure_01\"<\/p>\n

I must admit I was shocked at how simple it was and how quickly I was able to dupe Sally into being confident that Harry\u2019s cloned voice was real.<\/p>\n

This level of manipulation worked because of a compelling number of connected factors:<\/p>\n

    \n
  1. the CEO\u2019s phone number verified him,<\/li>\n
  2. the story I fabricated matched the day\u2019s events, and<\/li>\n
  3. the voice message, of course, sounded like the boss.<\/li>\n<\/ol>\n

    In my debrief with the company, and on reflection, Sally stated she felt this was \u201cmore than enough\u201d verification needed to carry out the request. Needless to say, the company has since added more safeguards to keep their finances protected. And, of course, I refunded the \u00a3250!<\/p>\n

    WhatsApp Business impersonation<\/h2>\n

    Stealing someone\u2019s WhatsApp account<\/a> via a SIM swap attack could be a rather long-winded way to make an attack more believable, but it happens far more commonly than you might think. Still, cybercriminals don\u2019t have to go to such lengths to produce the same outcome.<\/p>\n

    For example, I have recently been targeted with an attack that, on the face of it, looked believable. Someone had sent me a WhatsApp message purporting to be from a friend of mine who is an executive at an IT company.<\/p>\n

    The interesting dynamic here was that although I am used to verifying information, this message arrived with the linked contact name instead of it showing up as a number. This was of special interest, because I did not have the number it came from saved in my contacts list and I assumed it would still show as a mobile number, rather than the name.<\/p>\n

    \"Figure<\/p>\n<\/p>\n

    \n
    \"Figure<\/a><\/figure>\n
    \"Figure<\/a><\/figure>\n<\/div>\n

    Apparently the way they finagled this was simply by creating a WhatsApp Business account, which enables adding any name, photo and email address you want to an account and make it immediately look genuine. Add this to AI voice cloning and voila, we have entered the next generation of social engineering<\/a>.<\/p>\n

    Fortunately, I knew this was a scam from the outset, but many people could fall for this simple trick that could ultimately lead to the release of money in the form of financial transactions, prepaid cards, or other cards such as Apple Card, all of which are favorites among cyberthieves.<\/p>\n

    With machine learning and artificial intelligence progressing by leaps and bounds and becoming increasingly available to the masses recently, we are moving into an age where technology is starting to help criminals more efficiently than ever before, including by improving all the existing tools that help obfuscate the criminals\u2019 identities and whereabouts.<\/p>\n

    Staying safe<\/h2>\n

    Going back to our experiments, here are a few basic precautions business owners should take to avoid falling victim to attacks leveraging voice cloning and other shenanigans:<\/p>\n