南洋才女,德艺双馨,孙燕姿本尊回应AI孙燕姿(基于Sadtalker/Python3.10)

孙燕姿果然不愧是孙燕姿，不愧为南洋理工大学的高材生，近日她在个人官方媒体博客上写了一篇英文版的长文，正式回应现在满城风雨的“AI孙燕姿”现象，流行天后展示了超人一等的智识水平，行文优美，绵恒隽永，对AIGC艺术表现得极其克制，又相当宽容，充满了语言上的古典之美，表现出了“任彼如泰山压顶，我只当清风拂面”的博大胸怀。

本次我们利用edge-tts和Sadtalker库让AI孙燕姿朗诵本尊的博文，让流行天后念给你听。

Sadtalker配置

之前我们曾经使用百度开源的PaddleGAN视觉效果模型中一个子模块Wav2lip实现了人物口型与输入的歌词语音同步，但Wav2lip的问题是虚拟人物的动态效果只能局限在嘴唇附近，事实上，音频和不同面部动作之间的连接是不同的，也就是说，虽然嘴唇运动与音频的联系最强，但可以通过不同的头部姿势和眨眼来反作用于音频。

和Wav2lip相比，SadTaker是一种通过隐式3D系数调制的风格化音频驱动Talking头部视频生成的库，一方面，它从音频中生成逼真的运动系数（例如，头部姿势、嘴唇运动和眨眼），并单独学习每个运动以减少不确定性。对于表达，通过从的仅嘴唇运动系数和重建的渲染三维人脸上的感知损失（唇读损失，面部landmark loss）中提取系数，设计了一种新的音频到表达系数网络。

对于程序化的头部姿势，通过学习给定姿势的残差，使用条件VAE来对多样性和逼真的头部运动进行建模。在生成逼真的3DMM系数后，通过一种新颖的3D感知人脸渲染来驱动源图像。并且通过源和驱动的无监督3D关键点生成扭曲场，并扭曲参考图像以生成最终视频。

Sadtalker可以单独配置，也可以作为Stable-Diffusion-Webui的插件而存在，这里推荐使用Stable-Diffusion插件的形式，因为这样Stable-Diffusion和Sadtalker可以共用一套WebUI的界面，更方便将Stable-Diffusion生成的图片做成动态效果。

进入到Stable-Diffusion的项目目录：

cd stable-diffusion-webui

启动服务：

python3.10 webui.py

程序返回：

Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]  
Version: v1.3.0  
Commit hash: 20ae71faa8ef035c31aa3a410b707d792c8203a3  
Installing requirements  
Launching Web UI with arguments: --xformers --opt-sdp-attention --api --lowvram  
Loading weights [b4d453442a] from D:\work\stable-diffusion-webui\models\Stable-diffusion\protogenV22Anime_protogenV22.safetensors  
load Sadtalker Checkpoints from D:\work\stable-diffusion-webui\extensions\SadTalker\checkpoints  
Creating model from config: D:\work\stable-diffusion-webui\configs\v1-inference.yaml  
LatentDiffusion: Running in eps-prediction mode  
DiffusionWrapper has 859.52 M params.  
Running on local URL:  http://127.0.0.1:7860

代表启动成功，随后http://localhost:7860

选择插件(Extensions)选项卡

点击从url安装，输入插件地址：github.com/Winfredy/SadTalker

安装成功后，重启WebUI界面。

接着需要手动下载相关的模型文件：

https://pan.baidu.com/s/1nXuVNd0exUl37ISwWqbFGA?pwd=sadt

随后将模型文件放入项目的stable-diffusion-webui/extensions/SadTalker/checkpoints/目录即可。

接着配置一下模型目录的环境变量：

set SADTALKER_CHECKPOINTS=D:/stable-diffusion-webui/extensions/SadTalker/checkpoints/

至此，SadTalker就配置好了。

edge-tts音频转录

之前的歌曲复刻是通过So-vits库对原歌曲的音色进行替换和预测，也就是说需要原版的歌曲作为基础数据。但目前的场景显然有别于歌曲替换，我们首先需要将文本转换为语音，才能替换音色。

这里使用edge-tts库进行文本转语音操作：

import asyncio  import edge_tts  TEXT = '''  As my AI voice takes on a life of its own while I despair over my overhanging stomach and my children's every damn thing, I can't help but want to write something about it.  My fans have officially switched sides and accepted that I am indeed 冷门歌手 while my AI persona is the current hot property. I mean really, how do you fight with someone who is putting out new albums in the time span of minutes.  Whether it is ChatGPT or AI or whatever name you want to call it, this "thing" is now capable of mimicking and/or conjuring,  unique and complicated content by processing a gazillion chunks of information while piecing and putting together in a most coherent manner the task being asked at hand. Wait a minute, isn't that what humans do? The very task that we have always convinced ourselves; that the formation of thought or opinion is not replicable by robots, the very idea that this is beyond their league, is now the looming thing that will threaten thousands of human conjured jobs. Legal, medical, accountancy, and currently, singing a song.   You will protest, well I can tell the difference, there is no emotion or variance in tone/breath or whatever technical jargon you can come up with. Sorry to say, I suspect that this would be a very short term response.  Ironically, in no time at all, no human will be able to rise above that. No human will be able to have access to this amount of information AND make the right calls OR make the right mistakes (ok mayyyybe I'm jumping ahead). This new technology will be able to churn out what exactly EVERYTHING EVERYONE  needs. As indie or as warped or as psychotic as you can get, there's probably a unique content that could be created just for you. You are not special you are already predictable and also unfortunately malleable.  At this point, I feel like a popcorn eater with the best seat in the theatre. (Sidenote: Quite possibly in this case no tech is able to predict what it's like to be me, except when this is published then ok it's free for all). It's like watching that movie that changed alot of our lives Everything Everywhere All At Once, except in this case, I don't think it will be the idea of love that will save the day.   In this boundless sea of existence, where anything is possible, where nothing matters, I think it will be purity of thought, that being exactly who you are will be enough.   With this I fare thee well.  '''  VOICE = "en-HK-YanNeural"  
OUTPUT_FILE = "./test_en1.mp3"  async def _main() -> None:  communicate = edge_tts.Communicate(TEXT, VOICE)  await communicate.save(OUTPUT_FILE)  if __name__ == "__main__":  asyncio.run(_main())

音频使用英文版本的女声：en-HK-YanNeural，关于edge-tts，请移步：口播神器,基于Edge,微软TTS(text-to-speech)文字转语音免费开源库edge-tts语音合成实践(Python3.10)，这里不再赘述。

随后再将音频文件的音色替换为AI孙燕姿的音色即可：AI天后,在线飙歌,人工智能AI孙燕姿模型应用实践，复刻《遥远的歌》，原唱晴子(Python3.10)。

本地推理和爆显存问题

准备好生成的图片以及音频文件后，就可以在本地进行推理操作了，访问 localhost:7860

这里输入参数选择full，如此会保留整个图片区域，否则只保留头部部分。

生成效果：

SadTalker会根据音频文件生成对应的口型和表情。

这里需要注意的是，音频文件只支持MP3或者wav。

除此以外，推理过程中Pytorch库可能会报这个错误：

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 6.00 GiB total capacity; 5.38 GiB already allocated; 0 bytes free; 5.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

这就是所谓的"爆显存问题"。

一般情况下，是因为当前GPU的显存不够了所导致的，可以考虑缩小torch分片文件的体积：

set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:60

如果音频文件实在过大，也可以通过ffmpeg对音频文件切片操作，分多次进行推理：

ffmpeg -ss 00:00:00 -i test_en.wav -to 00:30:00 -c copy test_en_01.wav

藉此，就解决了推理过程中的爆显存问题。

结语

和Wav2Lip相比，SadTalker(Stylized Audio-Driven Talking-head)提供了更加细微的面部运动细节（如眼睛眨动）等等，可谓是细致入微，巨细靡遗，当然随之而来的是模型数量和推理成本以及推理时间的增加，但显然，这些都是值得的。