AI视频生成与编辑完全教程（Sora/Runway/Kling）
一、概述：AI视频生成技术的发展历程
1.1 从静态图像到动态视频的演进
人工智能在视觉生成领域的突破，经历了从图像分类、目标检测到图像生成、视频生成的漫长演进。早期的深度学习模型主要聚焦于理解（Recognition），而近年来，生成式AI（Generative AI）的崛起彻底改变了内容创作的范式。
视频生成相比图像生成面临的核心挑战在于时间一致性（Temporal Consistency）。一段30fps的10秒视频包含300帧图像，每帧之间不仅需要视觉上的连贯，还需要在运动、光照、物体形变等方面保持物理合理性。这使得视频生成的计算复杂度和模型设计难度远超图像生成。
1.2 技术演进路线
第一阶段：基于GAN的视频生成（2016-2020）
生成对抗网络（GAN）是最早被应用于视频生成的技术。代表工作包括：
- VGAN（Video GAN, 2016）：将DCGAN扩展到时间维度，生成短视频片段
- MoCoGAN（2018）：将视频的运动和内容分离建模
- VideoGPT（2021）：结合VQ-VAE和自回归Transformer
GAN方法的局限性在于训练不稳定、模式坍缩严重，且生成的视频分辨率和时长都受到很大限制。
第二阶段：基于扩散模型的视频生成（2022-2023）
扩散模型在图像生成领域的成功（DALL-E 2、Stable Diffusion）自然延伸到了视频领域：
- Video Diffusion Models（Google, 2022）：首次将DDPM扩展到视频
- Make-A-Video（Meta, 2022）：利用图像扩散模型的先验知识生成视频
- Imagen Video（Google, 2022）：级联式视频扩散模型
第三阶段：Diffusion Transformer时代（2024-至今）
Transformer架构与扩散模型的结合催生了新一代视频生成模型：
- Sora（OpenAI, 2024）：基于DiT架构，支持长达60秒的高质量视频生成
- Stable Video Diffusion（Stability AI, 2023）：开源视频生成模型
- Runway Gen-3 Alpha（2024）：商业级视频生成工具
- Kling（快手, 2024）：国产视频生成模型，支持长达2分钟视频
- CogVideo（智谱AI, 2024）：开源文本到视频生成模型
1.3 当前技术格局
2024-2025年，AI视频生成已经从实验室走向商业化应用。主要参与者包括：
| 公司/团队 | 产品 | 特点 |
|---|---|---|
| OpenAI | Sora | 最长60秒，物理世界模拟 |
| Runway | Gen-3 Alpha | 精细控制，专业工作流 |
| 快手 | Kling | 国产标杆，长视频支持 |
| Pika | Pika 1.0 | 简单易用，风格化强 |
| Stability AI | SVD | 开源，可本地部署 |
| 智谱AI | CogVideoX | 开源，中文优化 |
| 阿里 | I2VGen-XL | 图生视频，阿里生态 |
二、主流工具对比与选择指南
2.1 Sora（OpenAI）
核心特性：
- 基于Diffusion Transformer（DiT）架构
- 支持文本到视频、图像到视频、视频到视频
- 最长生成60秒、1080p分辨率视频
- 强大的物理世界模拟能力
- 支持视频拼接、扩展、编辑
技术架构： Sora的核心创新在于将视频表示为时空Patch（Spacetime Patches），类似于ViT将图像表示为Patch序列。这种表示方式使得模型能够处理不同分辨率、时长和宽高比的视频。
视频 → 时空Patch化 → DiT处理 → 去噪 → 视频解码器 → 输出视频
适用场景： 电影预览、创意广告、概念可视化
局限性： 访问受限（仅对部分用户开放），成本较高
2.2 Runway Gen-3 Alpha
核心特性：
- 高度可控的运动和风格
- 支持文生视频、图生视频
- 专业的运动画笔（Motion Brush）功能
- 与Adobe等专业工具集成
- 逐帧一致性好
API使用示例：
import requests
import time
API_KEY = "your_runway_api_key"
BASE_URL = "https://api.runwayml.com/v1"
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
# 创建视频生成任务
def create_video_generation(prompt, duration=4, resolution="720p"):
payload = {
"model": "gen3a_turbo",
"promptText": prompt,
"duration": duration,
"resolution": resolution,
"watermark": False
}
response = requests.post(
f"{BASE_URL}/video_generations",
json=payload,
headers=headers
)
return response.json()
# 查询任务状态
def get_generation_status(task_id):
response = requests.get(
f"{BASE_URL}/video_generations/{task_id}",
headers=headers
)
return response.json()
# 使用示例
prompt = "A cinematic shot of a golden retriever running through a field of sunflowers at sunset, slow motion, 4K quality"
result = create_video_generation(prompt, duration=5)
task_id = result["id"]
# 轮询等待完成
while True:
status = get_generation_status(task_id)
if status["status"] == "SUCCEEDED":
video_url = status["output"][0]
print(f"视频生成完成: {video_url}")
break
elif status["status"] == "FAILED":
print(f"生成失败: {status.get('failure', 'Unknown error')}")
break
print(f"生成中... 状态: {status['status']}")
time.sleep(10)
2.3 Kling（快手可灵）
核心特性：
- 国产自研，中文提示词友好
- 支持长达2分钟的视频生成
- 1080p分辨率，30fps
- 支持文生视频、图生视频
- 运动幅度大，物理合理性强
- 提供API接口
API调用示例：
import requests
import json
import time
class KlingClient:
def __init__(self, access_key, secret_key):
self.access_key = access_key
self.secret_key = secret_key
self.base_url = "https://api.klingai.com/v1"
self.token = self._get_token()
def _get_token(self):
"""获取访问令牌"""
# Kling使用JWT鉴权，需要根据官方文档生成token
import jwt
import time
payload = {
"iss": self.access_key,
"exp": int(time.time()) + 1800,
"nbf": int(time.time()) - 5
}
token = jwt.encode(payload, self.secret_key, algorithm="HS256")
return token
def create_text_to_video(self, prompt, mode="std", duration="5",
aspect_ratio="16:9", model="kling-v1"):
"""文生视频"""
headers = {
"Authorization": f"Bearer {self.token}",
"Content-Type": "application/json"
}
payload = {
"model_name": model,
"prompt": prompt,
"mode": mode, # std: 标准, pro: 专家
"duration": duration, # 5 或 10 秒
"aspect_ratio": aspect_ratio,
"cfg_scale": 0.5 # 0-1，值越小越自由
}
response = requests.post(
f"{self.base_url}/videos/text2video",
json=payload,
headers=headers
)
return response.json()
def create_image_to_video(self, image_url, prompt="", mode="std",
duration="5", model="kling-v1"):
"""图生视频"""
headers = {
"Authorization": f"Bearer {self.token}",
"Content-Type": "application/json"
}
payload = {
"model_name": model,
"image": image_url,
"prompt": prompt,
"mode": mode,
"duration": duration
}
response = requests.post(
f"{self.base_url}/videos/image2video",
json=payload,
headers=headers
)
return response.json()
def query_task(self, task_id):
"""查询任务状态"""
headers = {"Authorization": f"Bearer {self.token}"}
response = requests.get(
f"{self.base_url}/videos/text2video/{task_id}",
headers=headers
)
return response.json()
# 使用示例
client = KlingClient("your_access_key", "your_secret_key")
result = client.create_text_to_video(
prompt="一只金毛犬在夕阳下的向日葵花田中奔跑，电影级画质，慢动作",
mode="pro",
duration="10"
)
print(json.dumps(result, indent=2, ensure_ascii=False))
2.4 Stable Video Diffusion（SVD）
核心特性：
- 完全开源，可本地部署
- 基于Stable Diffusion架构
- 支持图生视频（Image-to-Video）
- 社区活跃，插件丰富
- 可通过ControlNet等扩展控制
本地部署与使用：
# 安装依赖
# pip install torch diffusers transformers accelerate
import torch
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video
from PIL import Image
# 加载模型
pipe = StableVideoDiffusionPipeline.from_pretrained(
"stabilityai/stable-video-diffusion-img2vid-xt",
torch_dtype=torch.float16,
variant="fp16"
)
pipe.to("cuda")
# 启用内存优化
pipe.enable_model_cpu_offload()
# 加载参考图像
image = load_image("path/to/your/image.png")
image = image.resize((1024, 576))
# 生成视频
generator = torch.manual_seed(42)
frames = pipe(
image,
decode_chunk_size=8,
generator=generator,
num_frames=25, # 生成25帧
motion_bucket_id=127, # 运动强度 1-255
noise_aug_strength=0.02, # 噪声增强强度
).frames[0]
# 导出为视频文件
export_to_video(frames, "output.mp4", fps=7)
print("视频已保存到 output.mp4")
# 高级配置：使用调度器控制
from diffusers import EulerDiscreteScheduler
pipe.scheduler = EulerDiscreteScheduler.from_config(
pipe.scheduler.config
)
# 批量生成变体
seeds = [42, 123, 456, 789]
for seed in seeds:
gen = torch.manual_seed(seed)
frames = pipe(image, generator=gen, num_frames=25).frames[0]
export_to_video(frames, f"output_seed_{seed}.mp4", fps=7)
2.5 工具选择决策树
你需要AI视频生成？
├── 需要本地部署/隐私优先？
│ ├── 有GPU（≥16GB显存）→ Stable Video Diffusion / CogVideoX
│ └── 无GPU → 使用云API（Kling/Runway）
├── 需要最长视频？
│ └── Kling（2分钟）> Sora（60秒）> Runway（10秒）
├── 需要最佳画质？
│ └── Sora > Runway Gen-3 > Kling Pro
├── 需要精细控制？
│ └── Runway（Motion Brush）> Kling > Sora
├── 预算有限？
│ └── SVD（开源免费）> Kling（价格适中）> Runway > Sora
└── 中文提示词？
└── Kling > CogVideoX > 其他
三、文生视频原理与架构
3.1 Diffusion Transformer（DiT）架构
Sora的核心架构——Diffusion Transformer（DiT）代表了视频生成领域的范式转变。它将传统的U-Net扩散模型骨干替换为Transformer，带来了更好的可扩展性和生成质量。
DiT的核心思想：
传统扩散模型：噪声 → U-Net去噪 → 清洁图像
DiT模型： 噪声 → Transformer去噪 → 清洁图像/视频
DiT Block的结构：
import torch
import torch.nn as nn
import math
class DiTBlock(nn.Module):
"""Diffusion Transformer的基本块"""
def __init__(self, hidden_dim, num_heads, mlp_ratio=4.0):
super().__init__()
self.norm1 = nn.LayerNorm(hidden_dim, elementwise_affine=False)
self.attn = nn.MultiheadAttention(hidden_dim, num_heads, batch_first=True)
self.norm2 = nn.LayerNorm(hidden_dim, elementwise_affine=False)
mlp_dim = int(hidden_dim * mlp_ratio)
self.mlp = nn.Sequential(
nn.Linear(hidden_dim, mlp_dim),
nn.GELU(),
nn.Linear(mlp_dim, hidden_dim)
)
# AdaLN-Zero调制参数
self.adaLN_modulation = nn.Sequential(
nn.SiLU(),
nn.Linear(hidden_dim, 6 * hidden_dim)
)
def forward(self, x, t):
"""
x: 输入特征 [batch, seq_len, hidden_dim]
t: 时间步条件 [batch, hidden_dim]
"""
# 计算AdaLN调制参数
shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = \
self.adaLN_modulation(t).chunk(6, dim=-1)
# 自注意力 + AdaLN
x_norm = self.norm1(x)
x_norm = x_norm * (1 + scale_msa.unsqueeze(1)) + shift_msa.unsqueeze(1)
attn_out, _ = self.attn(x_norm, x_norm, x_norm)
x = x + gate_msa.unsqueeze(1) * attn_out
# FFN + AdaLN
x_norm = self.norm2(x)
x_norm = x_norm * (1 + scale_mlp.unsqueeze(1)) + shift_mlp.unsqueeze(1)
mlp_out = self.mlp(x_norm)
x = x + gate_mlp.unsqueeze(1) * mlp_out
return x
class PatchEmbed(nn.Module):
"""将视频编码为时空Patch序列"""
def __init__(self, patch_size=(2, 16, 16), in_channels=3, embed_dim=1152):
super().__init__()
self.patch_size = patch_size
self.proj = nn.Conv3d(
in_channels, embed_dim,
kernel_size=patch_size,
stride=patch_size
)
def forward(self, x):
"""
x: [B, C, T, H, W] 视频张量
输出: [B, T'*H'*W', embed_dim] Patch序列
"""
x = self.proj(x) # [B, embed_dim, T', H', W']
x = x.flatten(2).transpose(1, 2) # [B, N, embed_dim]
return x
class TimestepEmbedder(nn.Module):
"""时间步嵌入"""
def __init__(self, hidden_dim, frequency_dim=256):
super().__init__()
self.mlp = nn.Sequential(
nn.Linear(frequency_dim, hidden_dim),
nn.SiLU(),
nn.Linear(hidden_dim, hidden_dim)
)
self.frequency_dim = frequency_dim
@staticmethod
def timestep_embedding(t, dim, max_period=10000):
"""正弦余弦位置编码"""
half = dim // 2
freqs = torch.exp(
-math.log(max_period) * torch.arange(half, device=t.device) / half
)
args = t[:, None].float() * freqs[None]
embedding = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)
return embedding
def forward(self, t):
t_emb = self.timestep_embedding(t, self.frequency_dim)
return self.mlp(t_emb)
class DiT(nn.Module):
"""完整的Diffusion Transformer模型"""
def __init__(self, input_size=(16, 48, 80), patch_size=(2, 4, 4),
in_channels=4, hidden_dim=1152, depth=28, num_heads=16,
num_classes=1000, class_dropout_prob=0.1):
super().__init__()
self.patch_size = patch_size
self.num_patches = (
input_size[0] // patch_size[0],
input_size[1] // patch_size[1],
input_size[2] // patch_size[2]
)
# Patch嵌入
self.x_embedder = PatchEmbed(patch_size, in_channels, hidden_dim)
self.t_embedder = TimestepEmbedder(hidden_dim)
# 类别条件嵌入（可选）
self.class_dropout_prob = class_dropout_prob
if num_classes is not None:
self.y_embedder = nn.Embedding(num_classes + 1, hidden_dim) # +1 for null class
# 位置编码
num_patches_total = self.num_patches[0] * self.num_patches[1] * self.num_patches[2]
self.pos_embed = nn.Parameter(torch.zeros(1, num_patches_total, hidden_dim))
# Transformer块
self.blocks = nn.ModuleList([
DiTBlock(hidden_dim, num_heads) for _ in range(depth)
])
# 输出层
self.final_norm = nn.LayerNorm(hidden_dim, elementwise_affine=False)
self.final_linear = nn.Linear(hidden_dim, patch_size[0] * patch_size[1] * patch_size[2] * in_channels)
self.initialize_weights()
def initialize_weights(self):
# 初始化位置编码
nn.init.normal_(self.pos_embed, std=0.02)
# 初始化线性层
def _basic_init(module):
if isinstance(module, nn.Linear):
nn.init.xavier_uniform_(module.weight)
if module.bias is not None:
nn.init.zeros_(module.bias)
self.apply(_basic_init)
def unpatchify(self, x):
"""将patch序列还原为视频张量"""
c = self.x_embedder.proj.out_channels
p = self.patch_size
h = self.num_patches[1]
w = self.num_patches[2]
t = self.num_patches[0]
x = x.reshape(x.shape[0], t, h, w, p[0], p[1], p[2], c)
x = x.permute(0, 7, 1, 4, 2, 5, 3, 6)
x = x.reshape(x.shape[0], c, t * p[0], h * p[1], w * p[2])
return x
def forward(self, x, t, y=None):
"""
x: 噪声视频 [B, C, T, H, W]
t: 时间步 [B]
y: 类别标签 [B] (可选)
"""
x = self.x_embedder(x) + self.pos_embed
t_emb = self.t_embedder(t)
# 添加类别条件
if y is not None and hasattr(self, 'y_embedder'):
y_emb = self.y_embedder(y)
t_emb = t_emb + y_emb
# 通过Transformer块
for block in self.blocks:
x = block(x, t_emb)
# 输出
x = self.final_norm(x)
x = self.final_linear(x)
x = self.unpatchify(x)
return x
3.2 视频VAE（Variational Autoencoder）
视频VAE负责将高维视频数据压缩到潜空间，是视频生成pipeline中不可或缺的组件。
import torch
import torch.nn as nn
class CausalConv3d(nn.Module):
"""因果3D卷积 - 时间维度上只向前看"""
def __init__(self, in_channels, out_channels, kernel_size,
stride=1, padding=0):
super().__init__()
# 时间维度使用因果填充
self.time_padding = kernel_size[0] - 1 if isinstance(kernel_size, tuple) else 0
self.conv = nn.Conv3d(
in_channels, out_channels, kernel_size,
stride=stride,
padding=(0, padding, padding) if isinstance(padding, int)
else (0, padding[1], padding[2])
)
def forward(self, x):
# 在时间维度上填充（只向过去填充）
if self.time_padding > 0:
x = nn.functional.pad(x, (0, 0, 0, 0, self.time_padding, 0))
return self.conv(x)
class VideoVAEEncoder(nn.Module):
"""视频VAE编码器"""
def __init__(self, in_channels=3, latent_dim=4, hidden_dims=[128, 256, 512]):
super().__init__()
layers = []
for i, h_dim in enumerate(hidden_dims):
layers.extend([
CausalConv3d(
in_channels if i == 0 else hidden_dims[i-1],
h_dim, kernel_size=(3, 3, 3), stride=(2 if i > 0 else 1, 2, 2),
padding=1
),
nn.GroupNorm(8, h_dim),
nn.SiLU()
])
self.encoder = nn.Sequential(*layers)
self.fc_mu = nn.Conv3d(hidden_dims[-1], latent_dim, 1)
self.fc_var = nn.Conv3d(hidden_dims[-1], latent_dim, 1)
def forward(self, x):
"""
x: [B, C, T, H, W] 原始视频
输出: mu, logvar [B, latent_dim, T', H', W']
"""
h = self.encoder(x)
mu = self.fc_mu(h)
logvar = self.fc_var(h)
return mu, logvar
class VideoVAEDecoder(nn.Module):
"""视频VAE解码器"""
def __init__(self, latent_dim=4, out_channels=3, hidden_dims=[512, 256, 128]):
super().__init__()
layers = []
for i, h_dim in enumerate(hidden_dims):
layers.extend([
nn.ConvTranspose3d(
latent_dim if i == 0 else hidden_dims[i-1],
h_dim, kernel_size=(4, 4, 4),
stride=(2 if i < len(hidden_dims)-1 else 1, 2, 2),
padding=(1, 1, 1)
),
nn.GroupNorm(8, h_dim),
nn.SiLU()
])
layers.append(nn.Conv3d(hidden_dims[-1], out_channels, 3, padding=1))
self.decoder = nn.Sequential(*layers)
def forward(self, z):
return self.decoder(z)
3.3 时空注意力机制
视频生成中的核心挑战是如何同时建模空间和时间关系。现代视频生成模型通常采用**分解的时空注意力（Factorized Spatial-Temporal Attention）**策略：
class SpatialTemporalAttention(nn.Module):
"""分解的时空注意力"""
def __init__(self, hidden_dim, num_heads):
super().__init__()
self.spatial_attn = nn.MultiheadAttention(
hidden_dim, num_heads, batch_first=True
)
self.temporal_attn = nn.MultiheadAttention(
hidden_dim, num_heads, batch_first=True
)
self.norm_s = nn.LayerNorm(hidden_dim)
self.norm_t = nn.LayerNorm(hidden_dim)
def forward(self, x, t_emb=None):
"""
x: [B, T*H*W, D] 时空特征序列
"""
B, N, D = x.shape
T = 8 # 时间帧数
HW = N // T # 空间token数
# 空间注意力：对每帧独立做自注意力
x_spatial = x.reshape(B * T, HW, D)
x_spatial_norm = self.norm_s(x_spatial)
attn_s, _ = self.spatial_attn(x_spatial_norm, x_spatial_norm, x_spatial_norm)
x_spatial = x_spatial + attn_s
# 时间注意力：对每个空间位置跨帧做注意力
x_temporal = x_spatial.reshape(B, T, HW, D)
x_temporal = x_temporal.permute(0, 2, 1, 3).reshape(B * HW, T, D)
x_temporal_norm = self.norm_t(x_temporal)
attn_t, _ = self.temporal_attn(x_temporal_norm, x_temporal_norm, x_temporal_norm)
x_temporal = x_temporal + attn_t
# 恢复原始形状
x = x_temporal.reshape(B, HW, T, D).permute(0, 2, 1, 3).reshape(B, N, D)
return x
四、提示词工程：精确控制视频生成
4.1 提示词结构框架
高质量的视频提示词应该包含以下要素：
[镜头类型] + [主体描述] + [动作/运动] + [环境/场景] + [风格/氛围] + [技术参数]
详细分类：
| 要素 | 描述 | 示例 |
|---|---|---|
| 镜头类型 | 摄影机角度和运动 | 特写、广角、航拍、跟踪镜头 |
| 主体描述 | 视频中的主要对象 | 一位穿着红色连衣裙的女性 |
| 动作/运动 | 主体和摄影机的运动 | 缓慢转身、快速奔跑 |
| 环境/场景 | 背景和氛围 | 雨天的城市街道、阳光明媚的海滩 |
| 风格/氛围 | 视觉风格和情感 | 电影级、赛博朋克、梦幻 |
| 技术参数 | 画质和效果 | 4K、浅景深、慢动作 |
4.2 专业提示词模板
模板1：电影级场景
Cinematic wide shot of [主体] [动作] in [环境],
[光照描述], [镜头运动], [色调/氛围],
shot on [摄影机/镜头], [画质参数]
示例：
Cinematic wide shot of a lone astronaut walking across a vast red desert
on Mars, golden hour lighting with long shadows, slow dolly forward,
warm orange and red color palette, shot on ARRI Alexa 65,
anamorphic lens flare, 4K ultra HD, film grain
模板2：产品展示
Product showcase: [产品] [展示方式] against [背景],
[光照设置], [摄影机运动], [风格], [画质]
示例：
Product showcase: a sleek matte black smartphone rotating slowly
on a reflective glass surface, against a gradient dark blue background,
studio three-point lighting with rim light, 360-degree orbit shot,
modern minimalist style, 4K photorealistic, shallow depth of field
模板3：自然风光
[景别] of [自然场景], [时间/天气], [运动元素],
[摄影机运动], [色彩/氛围], [画质]
示例：
Aerial drone shot of a winding river through dense autumn forest,
golden hour with mist rising from the water, leaves gently falling,
smooth ascending camera movement revealing the vast landscape,
warm golden and amber tones, 8K ultra high definition,
hyperrealistic, National Geographic style
4.3 运动控制关键词
# 运动控制关键词库
MOTION_KEYWORDS = {
"摄影机运动": {
"平移": ["pan left", "pan right", "lateral tracking"],
"推拉": ["dolly in", "dolly out", "push in", "pull back"],
"升降": ["crane up", "crane down", "ascending shot", "descending"],
"环绕": ["orbit shot", "360-degree rotation", "circling around"],
"跟踪": ["tracking shot", "following shot", "steadicam follow"],
"手持": ["handheld camera", "shaky cam", "documentary style"],
"固定": ["static shot", "locked-off camera", "fixed position"]
},
"运动速度": {
"极慢": ["extremely slow motion", "time-lapse slow", "ultra slow-mo"],
"慢速": ["slow motion", "slowed down", "120fps slow motion"],
"正常": ["normal speed", "real-time", "standard pace"],
"快速": ["fast motion", "speed ramp", "time-lapse"],
"极快": ["hyper lapse", "extremely fast", "rapid movement"]
},
"运动风格": {
"流畅": ["smooth movement", "fluid motion", "seamless transition"],
"急促": ["quick cuts", "dynamic movement", "energetic"],
"优雅": ["graceful movement", "elegant motion", "flowing"],
"机械": ["mechanical precision", "robotic movement", "precise"]
}
}
# 镜头类型关键词
SHOT_TYPES = {
"景别": {
"大远景": ["extreme wide shot", "extreme long shot"],
"远景": ["wide shot", "long shot", "establishing shot"],
"全景": ["full shot", "medium wide"],
"中景": ["medium shot", "mid shot"],
"近景": ["medium close-up", "bust shot"],
"特写": ["close-up", "tight shot"],
"大特写": ["extreme close-up", "macro shot", "detail shot"]
},
"角度": {
"平视": ["eye level", "straight on"],
"俯视": ["high angle", "bird's eye", "overhead"],
"仰视": ["low angle", "worm's eye", "looking up"],
"斜角": ["Dutch angle", "tilted angle", "canted angle"]
}
}
# 风格关键词
STYLE_KEYWORDS = {
"电影风格": [
"cinematic", "film noir", "neo-noir", "golden age Hollywood",
"French New Wave", "Wes Anderson style", "Kubrick style",
"Tarantino style", "Wong Kar-wai aesthetic"
],
"艺术风格": [
"impressionist", "expressionist", "surrealist", "abstract",
"pop art", "art deco", "art nouveau", "minimalist"
],
"视觉效果": [
"anamorphic lens flare", "bokeh", "lens distortion",
"film grain", "vintage film look", "cross-process",
"double exposure", "glitch effect", "chromatic aberration"
],
"色彩方案": [
"warm tones", "cool tones", "monochrome", "desaturated",
"high contrast", "pastel colors", "neon colors",
"teal and orange", "sepia", "black and white"
]
}
4.4 负面提示词策略
大多数视频生成模型支持负面提示词（Negative Prompt），用于排除不想要的元素：
# 通用负面提示词模板
NEGATIVE_PROMPTS = {
"通用": "low quality, blurry, distorted, deformed, disfigured, "
"bad anatomy, extra limbs, mutated, watermark, text, logo",
"人物": "ugly, duplicate, morbid, mutilated, poorly drawn face, "
"mutation, extra limb, bad proportions, cloned face, "
"gross proportions, malformed limbs, missing arms, "
"missing legs, extra arms, extra legs, fused fingers, "
"too many fingers, long neck",
"风景": "oversaturated, overexposed, underexposed, flat lighting, "
"low resolution, pixelated, noisy, artifacts, banding",
"产品": "reflections of photographer, fingerprints, dust, scratches, "
"uneven lighting, color cast, lens distortion"
}
def build_prompt(positive, negative_type="通用"):
"""构建包含负面提示词的完整prompt"""
negative = NEGATIVE_PROMPTS.get(negative_type, NEGATIVE_PROMPTS["通用"])
return {
"positive": positive,
"negative": negative
}
五、图生视频技术
5.1 技术原理
图生视频（Image-to-Video, I2V）是将静态图像作为起点，生成动态视频的技术。其核心思想是：
- 保持首帧一致性：生成的视频第一帧必须与输入图像高度一致
- 合理运动推断：根据图像内容推断可能的运动方向和幅度
- 时间连贯性：后续帧之间保持平滑过渡
5.2 使用SVD进行图生视频
import torch
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video
from PIL import Image
import numpy as np
class ImageToVideoGenerator:
def __init__(self, model_id="stabilityai/stable-video-diffusion-img2vid-xt"):
self.pipe = StableVideoDiffusionPipeline.from_pretrained(
model_id,
torch_dtype=torch.float16,
variant="fp16"
)
self.pipe.to("cuda")
self.pipe.enable_model_cpu_offload()
def generate(self, image_path, num_frames=25, fps=7,
motion_bucket_id=127, noise_aug_strength=0.02,
seed=None):
"""
从图像生成视频
参数:
image_path: 输入图像路径
num_frames: 生成帧数 (14-25)
fps: 输出帧率
motion_bucket_id: 运动强度 (1-255, 值越大运动越明显)
noise_aug_strength: 噪声增强 (0-1, 影响与原图的偏离程度)
seed: 随机种子
"""
# 加载并预处理图像
image = load_image(image_path)
image = image.resize((1024, 576))
# 设置随机种子
generator = None
if seed is not None:
generator = torch.manual_seed(seed)
# 生成视频帧
frames = self.pipe(
image,
decode_chunk_size=8,
generator=generator,
num_frames=num_frames,
motion_bucket_id=motion_bucket_id,
noise_aug_strength=noise_aug_strength,
).frames[0]
return frames
def generate_with_motion_control(self, image_path, motion_vectors,
num_frames=25, fps=7):
"""
使用自定义运动向量控制视频生成
motion_vectors: [(dx, dy), ...] 每帧的运动偏移
"""
image = load_image(image_path)
image = image.resize((1024, 576))
# 通过调整noise_aug_strength来间接控制运动
# 较大的值会产生更多变化
motion_strength = np.mean([abs(dx) + abs(dy) for dx, dy in motion_vectors])
noise_aug = min(0.1, motion_strength * 0.01)
# 运动桶ID也可以根据运动幅度调整
motion_bucket = int(min(255, max(1, motion_strength * 10)))
frames = self.pipe(
image,
num_frames=num_frames,
motion_bucket_id=motion_bucket,
noise_aug_strength=noise_aug,
).frames[0]
return frames
def save_video(self, frames, output_path, fps=7):
"""保存视频"""
export_to_video(frames, output_path, fps=fps)
print(f"视频已保存: {output_path}")
# 使用示例
generator = ImageToVideoGenerator()
# 基础生成
frames = generator.generate(
"sunset.jpg",
num_frames=25,
motion_bucket_id=180, # 较高的运动强度
seed=42
)
generator.save_video(frames, "sunset_video.mp4")
# 多个运动强度变体
for motion in [50, 100, 150, 200]:
frames = generator.generate(
"portrait.jpg",
motion_bucket_id=motion,
seed=42
)
generator.save_video(frames, f"portrait_motion_{motion}.mp4")
5.3 ControlNet扩展
from diffusers import StableVideoDiffusionPipeline, ControlNetModel
import torch
# 使用ControlNet进行更精细的运动控制
# ControlNet可以接受光流、深度图等作为条件输入
class ControlledVideoGenerator:
def __init__(self):
# 加载基础pipeline
self.pipe = StableVideoDiffusionPipeline.from_pretrained(
"stabilityai/stable-video-diffusion-img2vid-xt",
torch_dtype=torch.float16
)
self.pipe.to("cuda")
def generate_with_depth(self, image_path, depth_map_path,
num_frames=25):
"""
使用深度图控制视频生成
深度图可以帮助模型理解3D空间关系
"""
from PIL import Image
import numpy as np
image = Image.open(image_path).resize((1024, 576))
depth_map = Image.open(depth_map_path).resize((1024, 576))
# 将深度图转换为灰度
depth_gray = depth_map.convert("L")
depth_array = np.array(depth_gray) / 255.0
# 基于深度信息推断运动
# 近处物体运动更明显
foreground_motion = depth_array.mean() * 100 + 50
frames = self.pipe(
image,
num_frames=num_frames,
motion_bucket_id=int(foreground_motion),
).frames[0]
return frames
六、视频编辑技术
6.1 风格迁移
视频风格迁移是将一个视频的视觉风格转移到另一个视频上，同时保持内容和运动不变。
import torch
import torch.nn as nn
from torchvision import transforms
from PIL import Image
class VideoStyleTransfer:
"""基于神经网络的视频风格迁移"""
def __init__(self):
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
def extract_frames(self, video_path):
"""从视频中提取帧"""
import cv2
cap = cv2.VideoCapture(video_path)
frames = []
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frames.append(Image.fromarray(frame_rgb))
cap.release()
return frames
def apply_style_diffusion(self, content_frames, style_image,
strength=0.75, num_inference_steps=50):
"""
使用Stable Diffusion进行视频风格迁移
参数:
content_frames: 内容帧列表
style_image: 风格参考图像
strength: 风格强度 (0-1)
num_inference_steps: 推理步数
"""
from diffusers import StableDiffusionImg2ImgPipeline
pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16
).to(self.device)
styled_frames = []
for i, frame in enumerate(content_frames):
# 对每帧应用风格迁移
result = pipe(
prompt="in the style of the reference image",
image=frame,
strength=strength,
num_inference_steps=num_inference_steps,
guidance_scale=7.5
).images[0]
styled_frames.append(result)
if i % 10 == 0:
print(f"已处理 {i+1}/{len(content_frames)} 帧")
return styled_frames
def temporal_consistency_loss(self, frame1, frame2):
"""
时间一致性损失 - 确保相邻帧之间平滑过渡
"""
t1 = transforms.ToTensor()(frame1).to(self.device)
t2 = transforms.ToTensor()(frame2).to(self.device)
# 光流一致性
diff = torch.abs(t1 - t2)
return diff.mean()
def smooth_styled_frames(self, styled_frames, alpha=0.3):
"""
使用指数移动平均平滑风格化帧序列
alpha: 平滑系数 (0-1)
"""
smoothed = [styled_frames[0]]
for i in range(1, len(styled_frames)):
prev_tensor = transforms.ToTensor()(smoothed[-1])
curr_tensor = transforms.ToTensor()(styled_frames[i])
smoothed_tensor = alpha * curr_tensor + (1 - alpha) * prev_tensor
smoothed_tensor = torch.clamp(smoothed_tensor, 0, 1)
smoothed_frame = transforms.ToPILImage()(smoothed_tensor)
smoothed.append(smoothed_frame)
return smoothed
def export_video(self, frames, output_path, fps=30):
"""导出视频"""
import cv2
import numpy as np
if not frames:
return
w, h = frames[0].size
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter(output_path, fourcc, fps, (w, h))
for frame in frames:
frame_cv = cv2.cvtColor(np.array(frame), cv2.COLOR_RGB2BGR)
out.write(frame_cv)
out.release()
print(f"视频已保存: {output_path}")
# 使用示例
transfer = VideoStyleTransfer()
frames = transfer.extract_frames("input_video.mp4")
styled = transfer.apply_style_diffusion(frames, "vangogh_style.jpg", strength=0.6)
smoothed = transfer.smooth_styled_frames(styled, alpha=0.4)
transfer.export_video(smoothed, "styled_output.mp4", fps=30)
6.2 物体替换与背景更换
import torch
from PIL import Image
from diffusers import StableDiffusionInpaintPipeline
import numpy as np
class VideoObjectReplacer:
"""视频中的物体替换"""
def __init__(self):
self.inpaint_pipe = StableDiffusionInpaintPipeline.from_pretrained(
"runwayml/stable-diffusion-inpainting",
torch_dtype=torch.float16
).to("cuda")
def replace_object(self, frame, mask, new_object_prompt,
negative_prompt="", num_inference_steps=50):
"""
替换视频帧中的物体
参数:
frame: 原始帧 (PIL Image)
mask: 掩码图像，白色区域为需要替换的部分
new_object_prompt: 新物体的描述
"""
# 确保尺寸为8的倍数
w, h = frame.size
new_w = (w // 8) * 8
new_h = (h // 8) * 8
frame = frame.resize((new_w, new_h))
mask = mask.resize((new_w, new_h))
result = self.inpaint_pipe(
prompt=new_object_prompt,
image=frame,
mask_image=mask,
negative_prompt=negative_prompt,
num_inference_steps=num_inference_steps,
guidance_scale=7.5
).images[0]
return result.resize((w, h))
def replace_background(self, frame, subject_mask, background_prompt,
negative_prompt=""):
"""
替换视频背景
参数:
frame: 原始帧
subject_mask: 主体掩码（白色为主体）
background_prompt: 新背景描述
"""
# 反转掩码（背景区域为白色）
bg_mask = Image.fromarray(255 - np.array(subject_mask))
return self.replace_object(
frame, bg_mask, background_prompt, negative_prompt
)
# 使用示例
replacer = VideoObjectReplacer()
# 逐帧处理视频
import cv2
cap = cv2.VideoCapture("input.mp4")
fps = cap.get(cv2.CAP_PROP_FPS)
frames = []
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frames.append(Image.fromarray(frame_rgb))
cap.release()
# 假设已有掩码帧序列
masks = load_mask_sequence("masks/")
# 替换每帧中的物体
replaced_frames = []
for frame, mask in zip(frames, masks):
result = replacer.replace_object(
frame, mask,
"a red sports car, photorealistic, high quality"
)
replaced_frames.append(result)
# 导出视频
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
h, w = np.array(replaced_frames[0]).shape[:2]
out = cv2.VideoWriter("replaced.mp4", fourcc, fps, (w, h))
for frame in replaced_frames:
out.write(cv2.cvtColor(np.array(frame), cv2.COLOR_RGB2BGR))
out.release()
七、长视频生成与拼接策略
7.1 逐段生成与无缝拼接
当前AI视频生成模型通常只能生成几秒到几十秒的片段。要生成长视频，需要采用分段生成+智能拼接的策略。
import torch
from PIL import Image
import numpy as np
class LongVideoGenerator:
"""长视频生成器 - 通过分段生成和智能拼接实现"""
def __init__(self, pipeline):
self.pipe = pipeline
self.overlap_frames = 4 # 重叠帧数
def generate_long_video(self, prompt, total_duration=60,
segment_duration=5, fps=24):
"""
生成长视频
参数:
prompt: 视频描述
total_duration: 总时长（秒）
segment_duration: 每段时长（秒）
fps: 帧率
"""
num_segments = int(np.ceil(total_duration / segment_duration))
all_frames = []
for i in range(num_segments):
print(f"正在生成第 {i+1}/{num_segments} 段...")
# 构建子提示词（包含时间进展）
sub_prompt = self._build_segment_prompt(prompt, i, num_segments)
# 生成当前段
if i == 0:
frames = self._generate_segment(sub_prompt, segment_duration, fps)
else:
# 使用上一段的最后一帧作为参考
last_frame = all_frames[-1]
frames = self._generate_segment_with_reference(
sub_prompt, last_frame, segment_duration, fps
)
# 处理重叠区域
if i > 0 and all_frames:
frames = self._blend_overlap(all_frames, frames)
all_frames.extend(frames)
return all_frames
def _build_segment_prompt(self, base_prompt, segment_idx, total_segments):
"""为每个片段构建适当的提示词"""
progress = segment_idx / total_segments
# 添加时间进展描述
time_descriptors = [
"beginning of the scene",
"developing action",
"continuing movement",
"approaching climax",
"final moments"
]
time_desc = time_descriptors[min(segment_idx, len(time_descriptors) - 1)]
return f"{base_prompt}, {time_desc}, smooth continuous motion"
def _generate_segment(self, prompt, duration, fps):
"""生成单个视频片段"""
num_frames = int(duration * fps)
frames = self.pipe(
prompt,
num_frames=min(num_frames, 25), # SVD最大25帧
).frames[0]
return frames
def _generate_segment_with_reference(self, prompt, reference_frame,
duration, fps):
"""使用参考帧生成片段"""
# 使用图生视频模式，确保与前一段的连续性
num_frames = int(duration * fps)
frames = self.pipe(
prompt,
image=reference_frame,
num_frames=min(num_frames, 25),
motion_bucket_id=100,
).frames[0]
return frames
def _blend_overlap(self, prev_frames, new_frames, blend_frames=4):
"""混合重叠帧"""
if len(prev_frames) < blend_frames or len(new_frames) < blend_frames:
return new_frames
blended = []
for i in range(blend_frames):
alpha = i / blend_frames # 从0到1渐变
prev_tensor = self._frame_to_tensor(prev_frames[-(blend_frames - i)])
new_tensor = self._frame_to_tensor(new_frames[i])
blended_tensor = (1 - alpha) * prev_tensor + alpha * new_tensor
blended.append(self._tensor_to_frame(blended_tensor))
# 替换新帧的前几帧为混合帧
return blended + new_frames[blend_frames:]
def _frame_to_tensor(self, frame):
"""PIL Image -> Tensor"""
return torch.from_numpy(np.array(frame)).float() / 255.0
def _tensor_to_frame(self, tensor):
"""Tensor -> PIL Image"""
return Image.fromarray((tensor.numpy() * 255).astype(np.uint8))
# 使用示例
# generator = LongVideoGenerator(svd_pipeline)
# frames = generator.generate_long_video(
# "A timelapse of a flower blooming in a garden",
# total_duration=30,
# segment_duration=4,
# fps=24
# )
7.2 Story2Video：基于故事线的视频生成
class StoryToVideo:
"""基于故事线自动生成长视频"""
def __init__(self, video_generator):
self.generator = video_generator
def generate_from_story(self, story_segments, style="cinematic"):
"""
从故事段落生成视频
story_segments: [
{
"scene": "场景描述",
"action": "动作描述",
"mood": "氛围",
"duration": 5 # 秒
},
...
]
"""
all_frames = []
for i, segment in enumerate(story_segments):
prompt = self._segment_to_prompt(segment, style, i)
print(f"生成场景 {i+1}: {segment['scene']}")
if i == 0:
frames = self.generator.generate(prompt, segment["duration"])
else:
last_frame = all_frames[-1]
frames = self.generator.generate_with_reference(
prompt, last_frame, segment["duration"]
)
all_frames.extend(frames)
return all_frames
def _segment_to_prompt(self, segment, style, index):
"""将故事段落转换为视频提示词"""
return (
f"{style} shot: {segment['scene']}. "
f"{segment['action']}. "
f"{segment['mood']} atmosphere. "
f"Smooth camera movement, high quality."
)
八、AI视频质量评估
8.1 评估指标体系
import torch
import torch.nn as nn
import numpy as np
from PIL import Image
class VideoQualityEvaluator:
"""AI生成视频的质量评估器"""
def __init__(self):
self.metrics = {}
def evaluate_all(self, generated_frames, reference_frames=None):
"""执行全面质量评估"""
results = {}
# 1. 帧间一致性（Temporal Consistency）
results["temporal_consistency"] = self.temporal_consistency(generated_frames)
# 2. 视觉质量（Visual Quality）
results["visual_quality"] = self.visual_quality_score(generated_frames)
# 3. 运动平滑度（Motion Smoothness）
results["motion_smoothness"] = self.motion_smoothness(generated_frames)
# 4. 如果有参考视频，计算相似度
if reference_frames is not None:
results["fvd"] = self.fvd_score(generated_frames, reference_frames)
results["ssim"] = self.average_ssim(generated_frames, reference_frames)
return results
def temporal_consistency(self, frames):
"""
计算帧间一致性分数
使用相邻帧的光流估计和平滑度
"""
if len(frames) < 2:
return 1.0
consistencies = []
prev_tensor = self._frame_to_tensor(frames[0])
for i in range(1, len(frames)):
curr_tensor = self._frame_to_tensor(frames[i])
# 像素级差异
diff = torch.abs(curr_tensor - prev_tensor)
# 结构相似性（简化版）
mean_diff = diff.mean().item()
# 一致性分数（差异越小越一致）
consistency = max(0, 1 - mean_diff)
consistencies.append(consistency)
prev_tensor = curr_tensor
return np.mean(consistencies)
def visual_quality_score(self, frames):
"""
视觉质量评分
基于图像清晰度、对比度等指标
"""
scores = []
for frame in frames:
tensor = self._frame_to_tensor(frame)
# 清晰度（拉普拉斯方差）
sharpness = self._laplacian_variance(tensor)
# 对比度
contrast = tensor.std().item()
# 亮度均衡
brightness = tensor.mean().item()
brightness_score = 1 - abs(brightness - 0.5) * 2
score = (sharpness * 0.4 + contrast * 0.3 + brightness_score * 0.3)
scores.append(score)
return np.mean(scores)
def motion_smoothness(self, frames):
"""
运动平滑度评估
检查运动加速度是否平滑
"""
if len(frames) < 3:
return 1.0
# 计算帧间差异序列
diffs = []
for i in range(1, len(frames)):
prev = self._frame_to_tensor(frames[i-1])
curr = self._frame_to_tensor(frames[i])
diff = (curr - prev).abs().mean().item()
diffs.append(diff)
# 计算加速度（差异的变化率）
accelerations = []
for i in range(1, len(diffs)):
accel = abs(diffs[i] - diffs[i-1])
accelerations.append(accel)
# 平滑度 = 1 - 平均加速度（归一化）
if accelerations:
mean_accel = np.mean(accelerations)
smoothness = max(0, 1 - mean_accel * 10)
else:
smoothness = 1.0
return smoothness
def _laplacian_variance(self, tensor):
"""计算拉普拉斯方差（清晰度指标）"""
# 简化版：计算相邻像素差异
if tensor.dim() == 3:
gray = tensor.mean(dim=0)
else:
gray = tensor
laplacian = torch.zeros_like(gray)
laplacian[1:-1, 1:-1] = (
gray[2:, 1:-1] + gray[:-2, 1:-1] +
gray[1:-1, 2:] + gray[1:-1, :-2] -
4 * gray[1:-1, 1:-1]
)
return laplacian.var().item()
def _frame_to_tensor(self, frame):
"""PIL Image -> Tensor"""
if isinstance(frame, Image.Image):
return torch.from_numpy(np.array(frame)).float() / 255.0
return frame
def average_ssim(self, generated, reference):
"""计算平均结构相似性"""
from skimage.metrics import structural_similarity as ssim
scores = []
for gen, ref in zip(generated, reference):
gen_arr = np.array(gen)
ref_arr = np.array(ref)
if gen_arr.shape != ref_arr.shape:
continue
score = ssim(gen_arr, ref_arr, multichannel=True, channel_axis=2)
scores.append(score)
return np.mean(scores) if scores else 0.0
def fvd_score(self, generated, reference):
"""
Frechet Video Distance (FVD) - 简化版
使用预训练特征提取器计算特征分布距离
"""
# 实际实现需要预训练的I3D或VideoMAE模型
# 这里返回一个占位符
print("FVD计算需要预训练的视频特征提取器")
return 0.0
# 使用示例
evaluator = VideoQualityEvaluator()
# 加载生成的视频帧
from PIL import Image
import glob
gen_frames = [Image.open(f) for f in sorted(glob.glob("generated/*.png"))]
ref_frames = [Image.open(f) for f in sorted(glob.glob("reference/*.png"))]
# 评估
results = evaluator.evaluate_all(gen_frames, ref_frames)
print("视频质量评估结果:")
for metric, score in results.items():
print(f" {metric}: {score:.4f}")
九、商业应用场景
9.1 广告与营销
AI视频生成在广告领域的应用最为广泛：
class AdVideoGenerator:
"""AI广告视频生成器"""
TEMPLATES = {
"产品展示": {
"intro": "Cinematic product reveal, {product} emerging from shadows, "
"dramatic lighting, studio setting",
"feature": "Close-up detail shot of {product}, highlighting {feature}, "
"macro lens, shallow depth of field",
"lifestyle": "{product} being used by {persona} in {setting}, "
"natural lighting, lifestyle photography style",
"cta": "Product shot with text overlay space, clean background, "
"brand colors {colors}"
},
"品牌故事": {
"opening": "Aerial establishing shot of {location}, golden hour, "
"cinematic drone movement",
"narrative": "Medium shot of {scene}, {mood} atmosphere, "
"professional cinematography",
"emotional": "Close-up of {subject}, conveying {emotion}, "
"soft focus background, warm tones",
"closing": "Wide shot with brand elements, {brand} aesthetic, "
"professional color grading"
}
}
def generate_ad_sequence(self, product, style="产品展示",
duration_per_scene=5, fps=24):
"""生成广告视频序列"""
template = self.TEMPLATES.get(style, self.TEMPLATES["产品展示"])
scenes = []
for scene_type, prompt_template in template.items():
prompt = prompt_template.format(
product=product,
feature="premium materials and craftsmanship",
persona="confident professional",
setting="modern urban environment",
colors="#FF6B35, #004E89",
location="scenic mountain landscape",
scene="artisan at work",
mood="inspirational",
subject="artisan's hands",
emotion="passion and dedication",
brand="innovative"
)
scenes.append({
"prompt": prompt,
"duration": duration_per_scene
})
return scenes
9.2 短视频内容创作
class ShortVideoCreator:
"""短视频内容自动生成"""
def generate_explainer_video(self, topic, key_points, style="modern"):
"""
生成解释性短视频
参数:
topic: 主题
key_points: 要点列表
style: 视觉风格
"""
scenes = []
# 开场
scenes.append({
"type": "title",
"prompt": f"Abstract visualization of {topic}, {style} style, "
f"dynamic motion graphics, bold typography space",
"duration": 3
})
# 每个要点一个场景
for i, point in enumerate(key_points):
scenes.append({
"type": "content",
"prompt": f"Visual metaphor for '{point}', {style} animation, "
f"clean design, smooth transitions, "
f"professional infographic style",
"duration": 5
})
# 结尾
scenes.append({
"type": "outro",
"prompt": f"Summary visualization of {topic}, {style} style, "
f"conclusion with call-to-action space, "
f"professional ending card",
"duration": 3
})
return scenes
9.3 电影预览与分镜
class FilmPrevisualization:
"""电影预览/分镜自动生成"""
def generate_storyboard(self, script_segments):
"""
从剧本片段生成分镜
script_segments: [
{
"scene": "INT. COFFEE SHOP - DAY",
"description": "A young woman sits alone, staring out the window",
"camera": "Medium close-up, slight low angle",
"mood": "melancholic, introspective"
},
...
]
"""
storyboard = []
for i, segment in enumerate(script_segments):
prompt = (
f"{segment['camera']}: {segment['description']}. "
f"Scene: {segment['scene']}. "
f"Mood: {segment['mood']}. "
f"Cinematic lighting, film grain, professional cinematography, "
f"shot on 35mm film, anamorphic"
)
storyboard.append({
"scene_number": i + 1,
"prompt": prompt,
"camera_notes": segment['camera'],
"duration": 4 # 每个镜头4秒
})
return storyboard
十、实战案例：用开源模型构建视频生成Pipeline
10.1 完整的视频生成Pipeline
"""
完整的AI视频生成Pipeline
集成文本处理、视频生成、后处理等功能
"""
import torch
import numpy as np
from PIL import Image
from pathlib import Path
import json
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class VideoGenerationPipeline:
"""端到端的视频生成Pipeline"""
def __init__(self, model_path="stabilityai/stable-video-diffusion-img2vid-xt",
device="cuda"):
self.device = device
self.model_path = model_path
self._load_models()
def _load_models(self):
"""加载所有必要的模型"""
from diffusers import StableVideoDiffusionPipeline
logger.info(f"加载视频生成模型: {self.model_path}")
self.video_pipe = StableVideoDiffusionPipeline.from_pretrained(
self.model_path,
torch_dtype=torch.float16,
variant="fp16"
).to(self.device)
# 启用内存优化
self.video_pipe.enable_model_cpu_offload()
logger.info("模型加载完成")
def generate_from_text(self, prompt, output_path,
num_frames=25, fps=7, seed=None):
"""
文生视频流程：
1. 文本 -> 图像（使用SD）
2. 图像 -> 视频（使用SVD）
"""
logger.info(f"开始文生视频: {prompt[:50]}...")
# 步骤1：文生图
logger.info("步骤1: 生成关键帧图像...")
key_frame = self._text_to_image(prompt)
# 步骤2：图生视频
logger.info("步骤2: 从图像生成视频...")
frames = self._image_to_video(
key_frame, num_frames, fps, seed
)
# 步骤3：后处理
logger.info("步骤3: 后处理...")
self._save_video(frames, output_path, fps)
logger.info(f"视频生成完成: {output_path}")
return output_path
def _text_to_image(self, prompt):
"""文本生成关键帧"""
from diffusers import StableDiffusionPipeline
if not hasattr(self, 'sd_pipe'):
self.sd_pipe = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16
).to(self.device)
image = self.sd_pipe(
prompt=prompt,
negative_prompt="blurry, low quality, distorted",
num_inference_steps=30,
guidance_scale=7.5
).images[0]
return image.resize((1024, 576))
def _image_to_video(self, image, num_frames, fps, seed=None):
"""图像生成视频"""
generator = None
if seed is not None:
generator = torch.manual_seed(seed)
frames = self.video_pipe(
image,
decode_chunk_size=8,
generator=generator,
num_frames=num_frames,
motion_bucket_id=127,
noise_aug_strength=0.02,
).frames[0]
return frames
def _save_video(self, frames, output_path, fps):
"""保存视频文件"""
from diffusers.utils import export_to_video
export_to_video(frames, output_path, fps=fps)
def batch_generate(self, prompts, output_dir, **kwargs):
"""批量生成视频"""
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
results = []
for i, prompt in enumerate(prompts):
output_path = output_dir / f"video_{i:04d}.mp4"
try:
path = self.generate_from_text(
prompt, str(output_path), **kwargs
)
results.append({"prompt": prompt, "path": path, "status": "success"})
except Exception as e:
logger.error(f"生成失败: {e}")
results.append({"prompt": prompt, "error": str(e), "status": "failed"})
return results
class PromptEnhancer:
"""提示词增强器"""
ENHANCEMENTS = {
"quality": [
"high quality", "professional", "4K", "ultra HD",
"photorealistic", "detailed"
],
"lighting": [
"dramatic lighting", "golden hour", "studio lighting",
"natural light", "volumetric lighting"
],
"camera": [
"cinematic", "shallow depth of field", "anamorphic",
"steady cam", "professional cinematography"
]
}
@classmethod
def enhance(cls, base_prompt, aspects=None):
"""增强提示词"""
if aspects is None:
aspects = ["quality", "lighting"]
enhancements = []
for aspect in aspects:
if aspect in cls.ENHANCEMENTS:
enhancements.extend(cls.ENHANCEMENTS[aspect][:2])
return f"{base_prompt}, {', '.join(enhancements)}"
@classmethod
def optimize_for_model(cls, prompt, model="svd"):
"""针对特定模型优化提示词"""
if model == "svd":
# SVD对图像描述性提示词效果好
return f"A high quality frame from a video: {prompt}"
elif model == "kling":
# Kling对中文友好
return prompt
return prompt
# ==================== 使用示例 ====================
def main():
"""主函数 - 演示完整的视频生成流程"""
# 初始化Pipeline
pipeline = VideoGenerationPipeline()
# 示例1：单个视频生成
prompt = PromptEnhancer.enhance(
"A beautiful sunset over the ocean with waves gently crashing on the shore",
aspects=["quality", "lighting", "camera"]
)
pipeline.generate_from_text(
prompt=prompt,
output_path="sunset_ocean.mp4",
num_frames=25,
fps=7,
seed=42
)
# 示例2：批量生成
prompts = [
"A cat playing with a ball of yarn in a cozy living room",
"Northern lights dancing over a snowy mountain landscape",
"A futuristic city at night with flying cars and neon lights",
"A steaming cup of coffee on a rainy window sill",
]
enhanced_prompts = [
PromptEnhancer.enhance(p, ["quality", "camera"])
for p in prompts
]
results = pipeline.batch_generate(
enhanced_prompts,
output_dir="batch_output",
num_frames=25,
fps=7
)
# 输出结果
for r in results:
print(f"状态: {r['status']} | 路径: {r.get('path', 'N/A')}")
if __name__ == "__main__":
main()
10.2 Gradio Web界面
"""
AI视频生成Web界面
使用Gradio构建交互式界面
"""
import gradio as gr
import torch
from pathlib import Path
# 假设pipeline已在其他文件中定义
# from pipeline import VideoGenerationPipeline, PromptEnhancer
def create_demo(pipeline):
"""创建Gradio界面"""
def generate_video(prompt, negative_prompt, num_frames, fps,
motion_strength, seed, progress=gr.Progress()):
"""生成视频的核心函数"""
try:
# 增强提示词
enhanced = PromptEnhancer.enhance(prompt)
# 生成视频
output_path = "output.mp4"
progress(0.1, desc="准备中...")
# 如果有参考图像，使用图生视频
frames = pipeline.generate_from_text(
enhanced, output_path,
num_frames=int(num_frames),
fps=int(fps),
seed=int(seed) if seed else None
)
progress(1.0, desc="完成!")
return output_path, "视频生成成功！"
except Exception as e:
return None, f"错误: {str(e)}"
# 构建界面
with gr.Blocks(title="AI视频生成器", theme=gr.themes.Soft()) as demo:
gr.Markdown("# 🎬 AI视频生成器")
gr.Markdown("输入文字描述，AI为你生成精彩的视频")
with gr.Row():
with gr.Column(scale=1):
prompt = gr.Textbox(
label="视频描述",
placeholder="描述你想要生成的视频...",
lines=3
)
negative_prompt = gr.Textbox(
label="排除内容（可选）",
placeholder="不想出现的元素...",
lines=2
)
with gr.Row():
num_frames = gr.Slider(
minimum=14, maximum=25, value=25,
step=1, label="帧数"
)
fps = gr.Slider(
minimum=5, maximum=24, value=7,
step=1, label="帧率"
)
with gr.Row():
motion_strength = gr.Slider(
minimum=1, maximum=255, value=127,
step=1, label="运动强度"
)
seed = gr.Number(
label="随机种子（留空为随机）",
value=None
)
generate_btn = gr.Button("🎬 生成视频", variant="primary")
with gr.Column(scale=1):
video_output = gr.Video(label="生成结果")
status = gr.Textbox(label="状态", interactive=False)
# 预设示例
gr.Markdown("## 💡 示例提示词")
examples = gr.Examples(
examples=[
["A golden retriever playing in autumn leaves, slow motion, cinematic", "", 25, 7, 127, 42],
["Northern lights over Iceland, time-lapse, 4K", "", 25, 7, 100, 123],
["A steaming cup of coffee, morning light, cozy atmosphere", "", 20, 7, 80, 456],
],
inputs=[prompt, negative_prompt, num_frames, fps, motion_strength, seed]
)
# 绑定事件
generate_btn.click(
fn=generate_video,
inputs=[prompt, negative_prompt, num_frames, fps,
motion_strength, seed],
outputs=[video_output, status]
)
return demo
# 启动
if __name__ == "__main__":
# pipeline = VideoGenerationPipeline()
# demo = create_demo(pipeline)
# demo.launch(server_name="0.0.0.0", server_port=7860)
pass
十一、最佳实践与技巧
11.1 性能优化
# 1. 使用半精度浮点数
pipe.to(torch.float16)
# 2. 启用xFormers内存高效注意力
pipe.enable_xformers_memory_efficient_attention()
# 3. 模型CPU卸载（适合显存不足时）
pipe.enable_model_cpu_offload()
# 4. 顺序CPU卸载（最低显存需求）
pipe.enable_sequential_cpu_offload()
# 5. VAE切片解码（减少显存占用）
pipe.vae.enable_slicing()
# 6. VAE切片解码（进一步优化）
pipe.vae.enable_tiling()
# 7. 调整chunk_size平衡速度和显存
frames = pipe(image, decode_chunk_size=4).frames[0]
11.2 常见问题与解决方案
| 问题 | 原因 | 解决方案 |
|---|---|---|
| 显存不足（OOM） | 模型过大或分辨率太高 | 启用CPU卸载，降低分辨率，减小batch_size |
| 视频闪烁 | 帧间一致性差 | 调整noise_aug_strength，使用时序平滑 |
| 运动幅度太小 | motion_bucket_id过低 | 增大该值（建议50-200） |
| 内容与提示不符 | 提示词不够具体 | 使用更详细的描述，添加负面提示词 |
| 生成速度慢 | 采样步数过多 | 减少steps（20-30步通常足够），使用DDIM调度器 |
| 首帧与原图差异大 | noise_aug_strength过高 | 降低该值（建议0.01-0.05） |
| 视频分辨率低 | 模型限制 | 使用超分辨率模型进行后处理 |
11.3 显存需求参考
| 模型 | 最低显存 | 推荐显存 | 分辨率 |
|---|---|---|---|
| SVD-XT | 12GB | 16GB+ | 576x1024 |
| CogVideoX | 16GB | 24GB+ | 480x720 |
| Kling API | N/A | N/A | 1080p |
| Runway API | N/A | N/A | 720p-1080p |
十二、总结与展望
12.1 技术总结
AI视频生成在2024-2025年取得了突破性进展：
- DiT架构成为主流，取代了传统的U-Net扩散模型
- 时空Patch表示法使模型能够处理不同时长和分辨率的视频
- 物理世界模拟能力显著提升，Sora等模型能够生成符合物理规律的视频
- 开源生态快速发展，SVD、CogVideoX等模型可本地部署
12.2 未来趋势
- 更长视频：从当前的几十秒扩展到分钟级甚至更长
- 更高控制性：精确控制每一帧的内容和运动
- 交互式编辑：实时修改视频内容
- 多模态融合：结合文本、图像、音频的一体化生成
- 3D一致性：生成具有3D空间一致性的视频
- 个性化定制：针对特定风格和角色的定制化生成
12.3 学习路径建议
初学者：
1. 学习Stable Diffusion基础
2. 掌握SVD图生视频
3. 练习提示词工程
进阶者：
1. 理解DiT架构原理
2. 使用API进行商业项目开发
3. 探索视频编辑技术
专家级：
1. 训练自定义视频生成模型
2. 优化推理pipeline性能
3. 开发创新的视频应用
参考资源
论文：
- "Scalable Diffusion Models with Transformers" (DiT)
- "Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets"
- "CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer"
开源项目：
- diffusers - HuggingFace的扩散模型库
- CogVideo - 智谱AI的开源视频生成
- AnimateDiff - 动画扩散模型
API文档：
- Runway API
- Kling API
- Replicate - 多种视频生成模型的统一API
本教程涵盖了AI视频生成与编辑的核心技术和实践方法。随着技术的快速发展，建议持续关注最新的模型和工具更新，保持技术敏感度，在实践中不断探索和创新。
