mirror of
https://github.com/perfect-panel/ppanel-web.git
synced 2026-08-29 05:52:08 -04:00
add mysql2postgres
This commit is contained in:
@@ -1,285 +0,0 @@
|
||||
# AGENTS.md
|
||||
|
||||
`ppanel-backend` 项目约定,供 AI agent 及开发者遵循。
|
||||
|
||||
## ppanel-backend
|
||||
|
||||
基于 axum 的 Rust 重写项目,Go 源码位于 `../server`。Go 代码库是行为正确性的唯一参考——当 Rust handler 需要对齐 Go 语义时,从对应的 Go 包移植,而非重新设计。
|
||||
|
||||
---
|
||||
|
||||
## 一、返回值必须通过 `result` crate
|
||||
|
||||
所有 HTTP handler 的响应都必须通过 `result` crate 生成。**禁止**在 handler 中手写 `axum` 响应结构体、自行封装 JSON 或拼装 `axum::Json`——统一使用已有工具,以保持响应格式与 Go `result` 包一致。
|
||||
|
||||
```
|
||||
crates/result/src/
|
||||
├── lib.rs # pub mod code_error / error_code / http_result
|
||||
├── error_code.rs # 错误码常量 (SUCCESS, ERROR, INVALID_PARAMS, ...) + map_err_msg / is_code_err
|
||||
├── code_error.rs # CodeError (new_err_code / new_err_code_msg / new_err_msg) + STATUS_NOT_MODIFIED
|
||||
└── http_result.rs # ResponseSuccessBean / ResponseErrorBean / HttpResult + build/http/param 工具函数
|
||||
```
|
||||
|
||||
### API 一览
|
||||
|
||||
`result::http_result`
|
||||
- `build_http_result(resp, err) -> HttpResult` — 核心构造函数;出错时从 `anyhow::Error` 链中提取
|
||||
`CodeError`,找不到则回退为 `ERROR` / `"Internal Server Error"`。始终返回 HTTP 200,业务码放在
|
||||
body 的 `code` 字段(与 Go 行为一致)。
|
||||
- `build_param_error_result(err) -> HttpResult` — HTTP 200,业务码 `INVALID_PARAMS`。
|
||||
- `HttpResult` 实现了 `IntoResponse`,可直接作为 handler 的返回值。
|
||||
|
||||
`result::code_error`
|
||||
- `CodeError::new_err_code(code)` — 消息由 `map_err_msg` 自动查表。
|
||||
- `CodeError::new_err_code_msg(code, msg)` — 显式指定 code + 消息。
|
||||
- `CodeError::new_err_msg(msg)` — 业务码默认为 `ERROR`。
|
||||
|
||||
`result::error_code`
|
||||
- 仅包含命名常量,如 `SUCCESS`、`ERROR`、`INVALID_PARAMS`、`USER_NOT_EXIST`。
|
||||
必须使用这些常量,**禁止魔术数字**。
|
||||
|
||||
### Handler 用法模板
|
||||
|
||||
```rust
|
||||
use result::code_error::CodeError;
|
||||
use result::error_code;
|
||||
use result::http_result::{build_http_result, HttpResult};
|
||||
|
||||
pub async fn handler(State(state): State<AppState>, Json(req): Json<Req>) -> HttpResult {
|
||||
let res = some_service(req).await
|
||||
.map_err(|_| anyhow::Error::new(CodeError::new_err_code(error_code::USER_NOT_EXIST)));
|
||||
build_http_result(res.ok(), res.err())
|
||||
}
|
||||
```
|
||||
|
||||
规则:
|
||||
- HTTP 状态码固定 200,业务码在 body `code` 字段,禁止用 HTTP 状态码表示业务错误。
|
||||
- 成功路径 → `Some(data)`;错误路径 → `CodeError` 包装进 `anyhow::Error`。
|
||||
|
||||
---
|
||||
|
||||
## 二、日志系统(两层)
|
||||
|
||||
### 2.1 业务审计日志 — Telemetry facade
|
||||
|
||||
**位置**:`src/service/telemetry.rs`
|
||||
|
||||
所有业务事件写入 `system_logs` 表,必须通过 `Telemetry` facade,**禁止**在 service 层直接构造 `SystemLog` 并调用 `repos.log.insert()`。
|
||||
|
||||
```rust
|
||||
use crate::service::telemetry::Telemetry;
|
||||
|
||||
// 登录成功
|
||||
Telemetry::login(&repos, user_id, "email", &ip, &user_agent, true).await;
|
||||
|
||||
// 注册成功
|
||||
Telemetry::register(&repos, user_id, "email", &email, &ip, &user_agent).await;
|
||||
|
||||
// 余额变动(type_ 用 BALANCE_TYPE_* 常量)
|
||||
Telemetry::balance(&repos, user_id, BALANCE_TYPE_RECHARGE, amount, Some(order_no), balance).await;
|
||||
```
|
||||
|
||||
**全部 14 种方法**:
|
||||
|
||||
| 方法 | LogType | 优先级 |
|
||||
|------|---------|--------|
|
||||
| `login` | LOGIN (30) | P0 |
|
||||
| `register` | REGISTER (31) | P0 |
|
||||
| `balance` | BALANCE (32) | P1 |
|
||||
| `commission` | COMMISSION (33) | P1 |
|
||||
| `gift` | GIFT (34) | P1 |
|
||||
| `subscribe_access` | SUBSCRIBE (20) | P1 |
|
||||
| `subscribe_traffic` | SUBSCRIBE_TRAFFIC (21) | P2 |
|
||||
| `server_traffic` | SERVER_TRAFFIC (22) | P2 |
|
||||
| `reset_subscribe` | RESET_SUBSCRIBE (23) | P2 |
|
||||
| `email_message` | EMAIL_MESSAGE (10) | P2 |
|
||||
| `mobile_message` | MOBILE_MESSAGE (11) | P2 |
|
||||
| `user_traffic_rank` | USER_TRAFFIC_RANK (40) | P3 |
|
||||
| `server_traffic_rank` | SERVER_TRAFFIC_RANK (41) | P3 |
|
||||
| `traffic_stat` | TRAFFIC_STAT (42) | P3 |
|
||||
|
||||
子类型常量定义在 `src/model/entity/log.rs`(`BALANCE_TYPE_*`、`COMMISSION_TYPE_*` 等)。
|
||||
|
||||
### 2.2 应用操作日志 — tracing
|
||||
|
||||
运维/调试日志使用 `tracing::info!` / `tracing::error!`,由 `main.rs` 根据 `LogConfig` 初始化。
|
||||
请求日志由 `src/middleware/logger_middleware.rs`(`tower-http TraceLayer`)自动完成,**handler 无需额外代码**。
|
||||
|
||||
---
|
||||
|
||||
## 三、中间件
|
||||
|
||||
### 3.1 DeviceContext(设备上下文)
|
||||
|
||||
**位置**:`src/middleware/device_middleware.rs`
|
||||
|
||||
从 HTTP headers 提取客户端元数据并注入 `Extension<DeviceContext>`,**永不拒绝请求**:
|
||||
|
||||
| Header | DeviceContext 字段 |
|
||||
|--------|-------------------|
|
||||
| `X-Original-Forwarded-For` / `X-Forwarded-For` | `ip` |
|
||||
| `User-Agent` | `user_agent` |
|
||||
| `Identifier` | `identifier` |
|
||||
| `Login-Type` | `login_type` |
|
||||
|
||||
Handler 通过 `Extension(device): Extension<DeviceContext>` 提取,然后覆盖 JSON body 中的对应字段:
|
||||
|
||||
```rust
|
||||
pub async fn user_login(
|
||||
State(state): State<AppState>,
|
||||
Extension(device): Extension<DeviceContext>,
|
||||
Json(mut req): Json<UserLoginRequest>,
|
||||
) -> HttpResult {
|
||||
if !device.ip.is_empty() { req.ip = device.ip; }
|
||||
if !device.user_agent.is_empty() { req.user_agent = device.user_agent; }
|
||||
if !device.identifier.is_empty() { req.identifier = device.identifier; }
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 AuthContext(认证上下文)
|
||||
|
||||
**位置**:`src/middleware/auth_middleware.rs`
|
||||
|
||||
验证 Bearer JWT,校验 Redis session,查询用户状态,注入 `Extension<AuthContext>`。
|
||||
admin 路由自动检查 `user.is_admin`。
|
||||
|
||||
### 3.3 路由分组中间件映射
|
||||
|
||||
| 路由前缀 | 中间件 |
|
||||
|---------|--------|
|
||||
| `/v1/auth/*` | `device_middleware` |
|
||||
| `/v1/auth/oauth/*` | 无 |
|
||||
| `/v1/admin/*` | `auth_middleware` |
|
||||
| `/v1/public/*` | `auth_middleware` + `device_middleware` |
|
||||
| `/v1/common/*` | `device_middleware` |
|
||||
| `/v1/server/*`、`/v1/telegram`、`/v1/subscribe/*` | 无用户认证 |
|
||||
|
||||
---
|
||||
|
||||
## 四、本地 crate 一览
|
||||
|
||||
| crate | 路径 | 说明 |
|
||||
|-------|------|------|
|
||||
| `result` | `crates/result` | HTTP 响应信封、错误码 |
|
||||
| `jwt` | `crates/jwt` | JWT 生成/验证(`Claims::new` / `generate_token` / `validate_token`) |
|
||||
| `password` | `crates/password` | 密码哈希,移植自 `../server/pkg/tool/encryption.go` |
|
||||
| `oauth` | `crates/oauth` | OAuth2 封装(Google/Apple via arctic-oauth,Telegram HMAC 验证) |
|
||||
| `email` | `crates/email` | 邮件发送 |
|
||||
| `ip` | `crates/ip` | IP 工具 |
|
||||
| `payment` | `crates/payment` | 支付集成 |
|
||||
|
||||
### jwt crate 用法
|
||||
|
||||
```rust
|
||||
// 生成 token(返回 (Claims, expire_seconds))
|
||||
let (claims, seconds) = jwt::Claims::new(user_id, session_id, login_type);
|
||||
let token = jwt::generate_token(&claims, &config.jwt_auth.access_secret)?;
|
||||
|
||||
// 验证 token
|
||||
let claims = jwt::validate_token(&token, &config.jwt_auth.access_secret)?;
|
||||
```
|
||||
|
||||
### password crate 用法
|
||||
|
||||
```rust
|
||||
// 编码(新密码)
|
||||
let hash = password::encode_password(&plain_text)?;
|
||||
|
||||
// 校验(支持 md5 / sha256 / md5salt / sha256salt / default(PBKDF2) / bcrypt)
|
||||
let ok = password::multi_password_verify(&algo, &salt, &plain, &stored_hash);
|
||||
```
|
||||
|
||||
### oauth crate 用法
|
||||
|
||||
```rust
|
||||
// Google — PKCE 授权 URL
|
||||
let google = oauth::Google::new(&client_id, &client_secret, &redirect_uri);
|
||||
let url = google.authorization_url(&state, &["openid", "email", "profile"], &code_verifier);
|
||||
let tokens = google.validate_authorization_code(&code, &code_verifier).await?;
|
||||
let info = oauth::OAuthUserInfo::from_google(&tokens)?;
|
||||
|
||||
// Apple — 授权 URL(手动构造)+ token 交换
|
||||
let apple = oauth::Apple::new(&client_id, &team_id, &key_id, &pkcs8_der, &redirect_uri)?;
|
||||
let tokens = apple.validate_authorization_code(&code).await?;
|
||||
let info = oauth::OAuthUserInfo::from_apple(&tokens)?;
|
||||
|
||||
// Telegram — HMAC 验证 base64 回调
|
||||
let auth_data = oauth::parse_base64_and_validate(tg_auth_result, bot_token.as_bytes())?;
|
||||
let info = oauth::OAuthUserInfo::from_telegram(&auth_data);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、repository 层约定
|
||||
|
||||
- 所有 repo 方法均通过 trait 对象调用(`Box<dyn XxxRepo>`),dialect-agnostic。
|
||||
- `find_one_by_method`(AuthRepo)返回 `Result<Auth, sqlx::Error>`,**不是** `Option`——找不到时返回 `sqlx::Error::RowNotFound`。
|
||||
- 动态 SQL 字符串必须用 `repository::audit(sql)` 包装(`sqlx::AssertSqlSafe`)。
|
||||
|
||||
---
|
||||
|
||||
## 六、移植进度(截至 2026-07-05)
|
||||
|
||||
| 模块 | 状态 |
|
||||
|------|------|
|
||||
| 日志系统(Telemetry + tracing + TraceLayer) | ✅ 完成 |
|
||||
| 中间件(auth / device / logger) | ✅ 完成 |
|
||||
| 中间件(cors / notify / server / pan_domain / trace) | ✅ 完成 |
|
||||
| crates/sms(AlibabaCloud / Twilio / SmsBao / Abosend) | ✅ 完成 |
|
||||
| src/adapter(gtmpl 模板引擎 + Proxy/Client/Adapter) | ✅ 完成 |
|
||||
| service/auth(login / register / reset / device / telephone) | ✅ 完成 |
|
||||
| service/auth/oauth(Google / Apple / Telegram) | ✅ 完成 |
|
||||
| service/common(heartbeat / globalConfig / stat / client / ads / privacy / tos) | ✅ 完成 |
|
||||
| service/server(getConfig / getUserList / pushStatus / pushTraffic / pushOnline / queryProtocol) | ✅ 完成 |
|
||||
| service/nodeconfig(GlobalValues / ApplyOverride / OverrideResponse / OverrideModel / CloneValues) | ✅ 完成 |
|
||||
| service/subscribe(subscribeLogic + userAgent UA 匹配) | ✅ 完成 |
|
||||
| service/notify(Alipay RSA2 / ePay MD5 / Stripe webhook) | ✅ 完成 |
|
||||
| service/telegram(bot / template / telegram_service) | ✅ 完成 |
|
||||
| service/admin/ads | ✅ 完成 |
|
||||
| service/admin/announcement | ✅ 完成 |
|
||||
| service/admin/document | ✅ 完成 |
|
||||
| service/admin/coupon | ✅ 完成 |
|
||||
| service/admin/payment | ✅ 完成 |
|
||||
| service/admin/auth_method | ✅ 完成 |
|
||||
| service/admin/application(含 adapter 模板预览) | ✅ 完成 |
|
||||
| service/admin/console | ✅ 完成 |
|
||||
| service/admin/tool | ✅ 完成 |
|
||||
| service/admin/marketing(批量邮件 / quota 任务) | ✅ 完成 |
|
||||
| service/admin/order | ✅ 完成 |
|
||||
| service/admin/ticket | ✅ 完成 |
|
||||
| service/admin/log(全部 14 种日志类型) | ✅ 完成 |
|
||||
| service/admin/server(节点 / 服务器 CRUD + 协议配置) | ✅ 完成 |
|
||||
| service/admin/subscribe(订阅计划 CRUD + 排序) | ✅ 完成 |
|
||||
| service/admin/system(全部 26 个配置读写) | ✅ 完成 |
|
||||
| service/admin/user(全部 28 个用户管理操作) | ✅ 完成 |
|
||||
| service/public/announcement | ✅ 完成 |
|
||||
| service/public/document | ✅ 完成 |
|
||||
| service/public/payment | ✅ 完成 |
|
||||
| service/public/subscribe | ✅ 完成 |
|
||||
| service/public/ticket | ✅ 完成 |
|
||||
| service/public/portal(购买流程) | ✅ 完成 |
|
||||
| service/public/order(全部 12 个订单操作) | ✅ 完成 |
|
||||
| service/public/user(全部 30 个用户自助操作) | ✅ 完成 |
|
||||
| queue/service(email / sms / order / traffic / subscription / task) | ✅ 完成 |
|
||||
| scheduler(4 个定时任务注册) | ✅ 完成 |
|
||||
| handler/auth(所有端点,DeviceContext 注入) | ✅ 完成 |
|
||||
| handler/common(全部 10 个端点) | ✅ 完成 |
|
||||
| handler/server(全部 7 个端点) | ✅ 完成 |
|
||||
| handler/admin(全部子域,~150 个端点) | ✅ 完成 |
|
||||
| handler/public(全部子域,~90 个端点) | ✅ 完成 |
|
||||
| handler/subscribe(泛域名订阅) | ✅ 完成 |
|
||||
| handler/notify(Alipay / ePay / Stripe 回调) | ✅ 完成 |
|
||||
| handler/telegram | ✅ 完成 |
|
||||
| routes.rs(按分组应用中间件) | ✅ 完成 |
|
||||
| repository 层(全部 16 个域) | ✅ 完成 |
|
||||
| plugin API | ⏳ 暂不实现(已确认跳过) |
|
||||
|
||||
### 编译状态
|
||||
|
||||
`cargo check` → **0 errors**(截至 2026-07-05)
|
||||
|
||||
### 已知 TODO 项(功能存根)
|
||||
|
||||
- plugin 相关 handler:已确认暂不实现
|
||||
|
||||
Generated
+128
@@ -69,6 +69,56 @@ dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "anstream"
|
||||
version = "1.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d"
|
||||
dependencies = [
|
||||
"anstyle",
|
||||
"anstyle-parse",
|
||||
"anstyle-query",
|
||||
"anstyle-wincon",
|
||||
"colorchoice",
|
||||
"is_terminal_polyfill",
|
||||
"utf8parse",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "anstyle"
|
||||
version = "1.0.14"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000"
|
||||
|
||||
[[package]]
|
||||
name = "anstyle-parse"
|
||||
version = "1.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e"
|
||||
dependencies = [
|
||||
"utf8parse",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "anstyle-query"
|
||||
version = "1.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
|
||||
dependencies = [
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "anstyle-wincon"
|
||||
version = "3.0.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
|
||||
dependencies = [
|
||||
"anstyle",
|
||||
"once_cell_polyfill",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "anyhow"
|
||||
version = "1.0.103"
|
||||
@@ -605,6 +655,46 @@ dependencies = [
|
||||
"inout",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "clap"
|
||||
version = "4.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51"
|
||||
dependencies = [
|
||||
"clap_builder",
|
||||
"clap_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "clap_builder"
|
||||
version = "4.6.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f"
|
||||
dependencies = [
|
||||
"anstream",
|
||||
"anstyle",
|
||||
"clap_lex",
|
||||
"strsim",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "clap_derive"
|
||||
version = "4.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9"
|
||||
dependencies = [
|
||||
"heck",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.118",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "clap_lex"
|
||||
version = "1.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
|
||||
|
||||
[[package]]
|
||||
name = "cmake"
|
||||
version = "0.1.58"
|
||||
@@ -620,6 +710,12 @@ version = "0.5.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a"
|
||||
|
||||
[[package]]
|
||||
name = "colorchoice"
|
||||
version = "1.0.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
|
||||
|
||||
[[package]]
|
||||
name = "combine"
|
||||
version = "4.6.7"
|
||||
@@ -2119,6 +2215,12 @@ version = "2.12.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2"
|
||||
|
||||
[[package]]
|
||||
name = "is_terminal_polyfill"
|
||||
version = "1.70.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695"
|
||||
|
||||
[[package]]
|
||||
name = "itertools"
|
||||
version = "0.13.0"
|
||||
@@ -2423,6 +2525,20 @@ version = "0.10.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084"
|
||||
|
||||
[[package]]
|
||||
name = "mysql2postgres"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"chrono",
|
||||
"clap",
|
||||
"futures",
|
||||
"sqlx",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "native-tls"
|
||||
version = "0.2.18"
|
||||
@@ -2568,6 +2684,12 @@ version = "1.21.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
|
||||
|
||||
[[package]]
|
||||
name = "once_cell_polyfill"
|
||||
version = "1.70.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
|
||||
|
||||
[[package]]
|
||||
name = "openssl"
|
||||
version = "0.10.81"
|
||||
@@ -4938,6 +5060,12 @@ version = "1.0.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
|
||||
|
||||
[[package]]
|
||||
name = "utf8parse"
|
||||
version = "0.2.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
|
||||
|
||||
[[package]]
|
||||
name = "uuid"
|
||||
version = "0.8.2"
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
[workspace]
|
||||
members = ["crates/*"]
|
||||
members = ["crates/*", "tools/mysql2postgres"]
|
||||
|
||||
[package]
|
||||
name = "ppanel-backend"
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
[package]
|
||||
name = "mysql2postgres"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
description = "Copy data from MySQL to PostgreSQL for ppanel-backend migrations"
|
||||
|
||||
[[bin]]
|
||||
name = "mysql2postgres"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
chrono = "0.4"
|
||||
clap = { version = "4", features = ["derive", "env"] }
|
||||
sqlx = { version = "0.9", features = ["runtime-tokio", "mysql", "postgres", "chrono", "any"] }
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
tracing = "0.1"
|
||||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||
futures = "0.3"
|
||||
@@ -0,0 +1,474 @@
|
||||
//! mysql2postgres — copy data from MySQL to PostgreSQL.
|
||||
//!
|
||||
//! Direct port of `server/tools/mysql2postgres/` (Go).
|
||||
//! Connects to both databases, builds a migration plan,
|
||||
//! and copies each table row-by-row using PostgreSQL COPY.
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use clap::Parser;
|
||||
use sqlx::mysql::MySqlPool;
|
||||
use sqlx::postgres::PgPool;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// CLI
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
#[command(name = "mysql2postgres", about = "Copy ppanel MySQL data to PostgreSQL")]
|
||||
struct Args {
|
||||
/// Source MySQL DSN (e.g. user:pass@tcp(host:3306)/dbname)
|
||||
#[arg(long, env = "MYSQL_DSN")]
|
||||
mysql: String,
|
||||
|
||||
/// Target PostgreSQL DSN (e.g. postgres://user:pass@host/dbname)
|
||||
#[arg(long, env = "POSTGRES_DSN")]
|
||||
postgres: String,
|
||||
|
||||
/// Target PostgreSQL schema (default: public)
|
||||
#[arg(long, default_value = "public")]
|
||||
schema: String,
|
||||
|
||||
/// Comma-separated table allowlist (default: all common tables)
|
||||
#[arg(long, default_value = "")]
|
||||
tables: String,
|
||||
|
||||
/// Comma-separated table denylist
|
||||
#[arg(long, default_value = "")]
|
||||
exclude: String,
|
||||
|
||||
/// Truncate target tables before copying (destructive)
|
||||
#[arg(long, default_value_t = false)]
|
||||
truncate: bool,
|
||||
|
||||
/// Confirm destructive operations (required when --truncate is set)
|
||||
#[arg(long, default_value_t = false)]
|
||||
yes: bool,
|
||||
|
||||
/// Print plan without copying data
|
||||
#[arg(long, default_value_t = false)]
|
||||
dry_run: bool,
|
||||
|
||||
/// Rows per progress log line
|
||||
#[arg(long, default_value_t = 1000)]
|
||||
batch_size: usize,
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Column metadata
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
struct PgColumn {
|
||||
name: String,
|
||||
data_type: String,
|
||||
udt_name: String,
|
||||
is_identity: bool,
|
||||
is_generated: bool,
|
||||
}
|
||||
|
||||
impl PgColumn {
|
||||
fn is_bool(&self) -> bool {
|
||||
self.data_type == "boolean" || self.udt_name == "bool"
|
||||
}
|
||||
fn is_integer(&self) -> bool {
|
||||
matches!(self.data_type.as_str(), "smallint" | "integer" | "bigint")
|
||||
}
|
||||
fn is_timestamp(&self) -> bool {
|
||||
self.data_type.contains("timestamp") || self.data_type == "date"
|
||||
}
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Migration plan
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
#[derive(Debug)]
|
||||
struct TablePlan {
|
||||
name: String,
|
||||
columns: Vec<PgColumn>,
|
||||
order_columns: Vec<String>,
|
||||
row_count: i64,
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Entry point
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
tracing_subscriber::fmt()
|
||||
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
|
||||
.init();
|
||||
|
||||
let args = Args::parse();
|
||||
|
||||
if args.truncate && !args.yes && !args.dry_run {
|
||||
anyhow::bail!("--truncate is destructive; pass --yes to confirm");
|
||||
}
|
||||
|
||||
let mysql = MySqlPool::connect(&args.mysql)
|
||||
.await
|
||||
.context("connect to MySQL")?;
|
||||
tracing::info!("connected to MySQL");
|
||||
|
||||
let pg = PgPool::connect(&args.postgres)
|
||||
.await
|
||||
.context("connect to PostgreSQL")?;
|
||||
tracing::info!("connected to PostgreSQL");
|
||||
|
||||
let plans = build_plans(&mysql, &pg, &args).await?;
|
||||
if plans.is_empty() {
|
||||
anyhow::bail!("no common tables to migrate");
|
||||
}
|
||||
|
||||
tracing::info!("migration plan: {} table(s)", plans.len());
|
||||
for p in &plans {
|
||||
tracing::info!(
|
||||
" {}: {} row(s), {} column(s)",
|
||||
p.name, p.row_count, p.columns.len()
|
||||
);
|
||||
}
|
||||
|
||||
if args.dry_run {
|
||||
tracing::info!("dry run — no data copied");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
if args.truncate {
|
||||
truncate_tables(&pg, &args.schema, &plans).await?;
|
||||
}
|
||||
|
||||
for plan in &plans {
|
||||
copy_table(&mysql, &pg, &args.schema, plan, args.batch_size).await?;
|
||||
}
|
||||
|
||||
reset_sequences(&pg, &args.schema).await?;
|
||||
tracing::info!("migration completed");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Plan building
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
async fn build_plans(mysql: &MySqlPool, pg: &PgPool, args: &Args) -> Result<Vec<TablePlan>> {
|
||||
let source_tables = list_mysql_tables(mysql).await?;
|
||||
let target_tables = list_pg_tables(pg, &args.schema).await?;
|
||||
|
||||
let allow: HashSet<String> = parse_set(&args.tables);
|
||||
let exclude: HashSet<String> = parse_set(&args.exclude);
|
||||
|
||||
let mut names: Vec<String> = target_tables
|
||||
.iter()
|
||||
.filter(|n| n.as_str() != "schema_migrations")
|
||||
.filter(|n| allow.is_empty() || allow.contains(*n))
|
||||
.filter(|n| !exclude.contains(*n))
|
||||
.filter(|n| {
|
||||
if source_tables.contains(*n) { true } else {
|
||||
tracing::warn!("skip {}: no source table", n); false
|
||||
}
|
||||
})
|
||||
.cloned()
|
||||
.collect();
|
||||
names.sort();
|
||||
|
||||
let mut plans = Vec::new();
|
||||
for name in &names {
|
||||
let target_cols = list_pg_columns(pg, &args.schema, name).await?;
|
||||
let source_cols = list_mysql_columns(mysql, name).await?;
|
||||
|
||||
let common: Vec<PgColumn> = target_cols
|
||||
.into_iter()
|
||||
.filter(|c| !c.is_generated && source_cols.contains(&c.name))
|
||||
.collect();
|
||||
|
||||
if common.is_empty() {
|
||||
tracing::warn!("skip {}: no common columns", name);
|
||||
continue;
|
||||
}
|
||||
|
||||
let row_count = count_mysql_rows(mysql, name).await?;
|
||||
let order_columns = list_mysql_pk_columns(mysql, name).await?;
|
||||
|
||||
plans.push(TablePlan { name: name.clone(), columns: common, order_columns, row_count });
|
||||
}
|
||||
|
||||
let fks = list_pg_foreign_keys(pg, &args.schema).await?;
|
||||
Ok(sort_by_dependencies(plans, fks))
|
||||
}
|
||||
|
||||
fn parse_set(s: &str) -> HashSet<String> {
|
||||
s.split(',').map(|x| x.trim().to_string()).filter(|x| !x.is_empty()).collect()
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// MySQL introspection
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
async fn list_mysql_tables(pool: &MySqlPool) -> Result<HashSet<String>> {
|
||||
let rows: Vec<(String,)> = sqlx::query_as(
|
||||
"SELECT table_name FROM information_schema.tables \
|
||||
WHERE table_schema = DATABASE() AND table_type = 'BASE TABLE'"
|
||||
).fetch_all(pool).await.context("list mysql tables")?;
|
||||
Ok(rows.into_iter().map(|(n,)| n).collect())
|
||||
}
|
||||
|
||||
async fn list_mysql_columns(pool: &MySqlPool, table: &str) -> Result<HashSet<String>> {
|
||||
let rows: Vec<(String,)> = sqlx::query_as(
|
||||
"SELECT column_name FROM information_schema.columns \
|
||||
WHERE table_schema = DATABASE() AND table_name = ? ORDER BY ordinal_position"
|
||||
).bind(table).fetch_all(pool).await
|
||||
.with_context(|| format!("list mysql columns for {table}"))?;
|
||||
Ok(rows.into_iter().map(|(n,)| n).collect())
|
||||
}
|
||||
|
||||
async fn list_mysql_pk_columns(pool: &MySqlPool, table: &str) -> Result<Vec<String>> {
|
||||
let rows: Vec<(String,)> = sqlx::query_as(
|
||||
"SELECT column_name FROM information_schema.key_column_usage \
|
||||
WHERE table_schema = DATABASE() AND table_name = ? AND constraint_name = 'PRIMARY' \
|
||||
ORDER BY ordinal_position"
|
||||
).bind(table).fetch_all(pool).await
|
||||
.with_context(|| format!("list mysql pk for {table}"))?;
|
||||
Ok(rows.into_iter().map(|(n,)| n).collect())
|
||||
}
|
||||
|
||||
async fn count_mysql_rows(pool: &MySqlPool, table: &str) -> Result<i64> {
|
||||
let quoted = format!("`{}`", table.replace('`', "``"));
|
||||
let (count,): (i64,) = sqlx::query_as(sqlx::AssertSqlSafe(format!("SELECT COUNT(*) FROM {quoted}")))
|
||||
.fetch_one(pool).await
|
||||
.with_context(|| format!("count mysql rows in {table}"))?;
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// PostgreSQL introspection
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
async fn list_pg_tables(pool: &PgPool, schema: &str) -> Result<HashSet<String>> {
|
||||
let rows: Vec<(String,)> = sqlx::query_as(
|
||||
"SELECT table_name FROM information_schema.tables \
|
||||
WHERE table_schema = $1 AND table_type = 'BASE TABLE'"
|
||||
).bind(schema).fetch_all(pool).await.context("list pg tables")?;
|
||||
Ok(rows.into_iter().map(|(n,)| n).collect())
|
||||
}
|
||||
|
||||
async fn list_pg_columns(pool: &PgPool, schema: &str, table: &str) -> Result<Vec<PgColumn>> {
|
||||
let rows: Vec<(String, String, String, String, String)> = sqlx::query_as(
|
||||
"SELECT column_name, data_type, udt_name, is_identity, is_generated \
|
||||
FROM information_schema.columns \
|
||||
WHERE table_schema = $1 AND table_name = $2 ORDER BY ordinal_position"
|
||||
).bind(schema).bind(table).fetch_all(pool).await
|
||||
.with_context(|| format!("list pg columns for {table}"))?;
|
||||
Ok(rows.into_iter().map(|(name, data_type, udt_name, identity, generated)| PgColumn {
|
||||
name,
|
||||
data_type,
|
||||
udt_name,
|
||||
is_identity: identity == "YES",
|
||||
is_generated: generated != "NEVER",
|
||||
}).collect())
|
||||
}
|
||||
|
||||
async fn list_pg_foreign_keys(pool: &PgPool, schema: &str) -> Result<Vec<(String, String)>> {
|
||||
let rows: Vec<(String, String)> = sqlx::query_as(
|
||||
"SELECT child.relname, parent.relname \
|
||||
FROM pg_constraint c \
|
||||
JOIN pg_class child ON child.oid = c.conrelid \
|
||||
JOIN pg_namespace cn ON cn.oid = child.relnamespace \
|
||||
JOIN pg_class parent ON parent.oid = c.confrelid \
|
||||
JOIN pg_namespace pn ON pn.oid = parent.relnamespace \
|
||||
WHERE c.contype = 'f' AND cn.nspname = $1 AND pn.nspname = $1 \
|
||||
ORDER BY child.relname, parent.relname"
|
||||
).bind(schema).fetch_all(pool).await.context("list pg foreign keys")?;
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Dependency sort (topological, mirrors Go)
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
fn sort_by_dependencies(plans: Vec<TablePlan>, fks: Vec<(String, String)>) -> Vec<TablePlan> {
|
||||
let plan_names: HashSet<&str> = plans.iter().map(|p| p.name.as_str()).collect();
|
||||
let mut deps: HashMap<&str, HashSet<&str>> = HashMap::new();
|
||||
for (child, parent) in &fks {
|
||||
if child != parent && plan_names.contains(child.as_str()) && plan_names.contains(parent.as_str()) {
|
||||
deps.entry(child).or_default().insert(parent);
|
||||
}
|
||||
}
|
||||
|
||||
// Collect ordered names (Strings) before consuming `plans`.
|
||||
let name_to_idx: HashMap<&str, usize> = plans.iter().enumerate().map(|(i, p)| (p.name.as_str(), i)).collect();
|
||||
let mut remaining: HashSet<&str> = plan_names.clone();
|
||||
let mut ordered_names: Vec<String> = Vec::new();
|
||||
|
||||
while !remaining.is_empty() {
|
||||
let mut ready: Vec<&str> = remaining.iter()
|
||||
.copied()
|
||||
.filter(|n| deps.get(n).map_or(true, |ps| ps.iter().all(|p| !remaining.contains(*p))))
|
||||
.collect();
|
||||
if ready.is_empty() {
|
||||
ready = remaining.iter().copied().collect();
|
||||
tracing::warn!("FK cycle detected; copying {} remaining tables in lexical order", ready.len());
|
||||
}
|
||||
ready.sort();
|
||||
for name in ready {
|
||||
ordered_names.push(name.to_string());
|
||||
remaining.remove(name);
|
||||
}
|
||||
}
|
||||
let _ = name_to_idx; // suppress unused warning
|
||||
|
||||
// Re-order plans by the collected name sequence.
|
||||
let mut plan_map: HashMap<String, TablePlan> =
|
||||
plans.into_iter().map(|p| (p.name.clone(), p)).collect();
|
||||
ordered_names.into_iter().filter_map(|n| plan_map.remove(&n)).collect()
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Data copy
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
async fn truncate_tables(pg: &PgPool, schema: &str, plans: &[TablePlan]) -> Result<()> {
|
||||
let names: Vec<String> = plans.iter()
|
||||
.map(|p| format!("\"{schema}\".\"{n}\"", n = p.name))
|
||||
.collect();
|
||||
let sql = format!("TRUNCATE TABLE {} RESTART IDENTITY CASCADE", names.join(", "));
|
||||
tracing::info!("truncating {} table(s)", plans.len());
|
||||
sqlx::query(sqlx::AssertSqlSafe(sql)).execute(pg).await.context("truncate tables")?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn copy_table(
|
||||
mysql: &MySqlPool,
|
||||
pg: &PgPool,
|
||||
schema: &str,
|
||||
plan: &TablePlan,
|
||||
batch_size: usize,
|
||||
) -> Result<()> {
|
||||
tracing::info!("copy {}: start ({} rows)", plan.name, plan.row_count);
|
||||
|
||||
let col_names: Vec<String> = plan.columns.iter().map(|c| c.name.clone()).collect();
|
||||
let mysql_cols = col_names.iter()
|
||||
.map(|n| format!("`{}`", n.replace('`', "``")))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ");
|
||||
|
||||
let mut select = format!("SELECT {} FROM `{}`", mysql_cols, plan.name.replace('`', "``"));
|
||||
if !plan.order_columns.is_empty() {
|
||||
let order = plan.order_columns.iter()
|
||||
.map(|n| format!("`{}`", n.replace('`', "``")))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ");
|
||||
select.push_str(&format!(" ORDER BY {order}"));
|
||||
}
|
||||
|
||||
// Build PG INSERT (no COPY protocol via sqlx yet — use parameterised INSERT in batches).
|
||||
let pg_cols = col_names.iter()
|
||||
.map(|n| format!("\"{}\"", n.replace('"', "\"\"")))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ");
|
||||
|
||||
let mut rows = sqlx::query(sqlx::AssertSqlSafe(select)).fetch(mysql);
|
||||
let mut copied: i64 = 0;
|
||||
let mut batch: Vec<Vec<Option<String>>> = Vec::with_capacity(batch_size);
|
||||
|
||||
use sqlx::Row;
|
||||
use futures::StreamExt;
|
||||
|
||||
while let Some(row) = rows.next().await {
|
||||
let row = row.with_context(|| format!("read row from {}", plan.name))?;
|
||||
let values: Vec<Option<String>> = col_names.iter().enumerate().map(|(i, _)| {
|
||||
row.try_get::<Option<String>, _>(i).unwrap_or(None)
|
||||
}).collect();
|
||||
batch.push(values);
|
||||
|
||||
if batch.len() >= batch_size {
|
||||
insert_batch(pg, schema, &plan.name, &col_names, &pg_cols, &batch, &plan.columns).await?;
|
||||
copied += batch.len() as i64;
|
||||
batch.clear();
|
||||
tracing::info!("copy {}: {}/{}", plan.name, copied, plan.row_count);
|
||||
}
|
||||
}
|
||||
if !batch.is_empty() {
|
||||
insert_batch(pg, schema, &plan.name, &col_names, &pg_cols, &batch, &plan.columns).await?;
|
||||
copied += batch.len() as i64;
|
||||
}
|
||||
|
||||
tracing::info!("copy {}: done ({} rows)", plan.name, copied);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn insert_batch(
|
||||
pg: &PgPool,
|
||||
schema: &str,
|
||||
table: &str,
|
||||
_col_names: &[String],
|
||||
pg_cols: &str,
|
||||
batch: &[Vec<Option<String>>],
|
||||
_columns: &[PgColumn],
|
||||
) -> Result<()> {
|
||||
if batch.is_empty() { return Ok(()); }
|
||||
let col_count = batch[0].len();
|
||||
let mut placeholders = Vec::new();
|
||||
let mut idx = 1usize;
|
||||
for _ in batch {
|
||||
let row_ph: Vec<String> = (0..col_count).map(|_| { let s = format!("${idx}"); idx += 1; s }).collect();
|
||||
placeholders.push(format!("({})", row_ph.join(",")));
|
||||
}
|
||||
let sql = format!(
|
||||
"INSERT INTO \"{schema}\".\"{table}\" ({pg_cols}) VALUES {} ON CONFLICT DO NOTHING",
|
||||
placeholders.join(",")
|
||||
);
|
||||
let mut q = sqlx::query(sqlx::AssertSqlSafe(sql));
|
||||
for row in batch {
|
||||
for val in row {
|
||||
q = q.bind(val.as_deref());
|
||||
}
|
||||
}
|
||||
q.execute(pg).await.with_context(|| format!("insert batch into {table}"))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Sequence reset
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
async fn reset_sequences(pg: &PgPool, schema: &str) -> Result<()> {
|
||||
let rows: Vec<(String, String)> = sqlx::query_as(
|
||||
"SELECT table_name, column_name FROM information_schema.columns \
|
||||
WHERE table_schema = $1 AND (is_identity = 'YES' OR column_default LIKE 'nextval(%') \
|
||||
ORDER BY table_name, ordinal_position"
|
||||
).bind(schema).fetch_all(pg).await.context("list sequences")?;
|
||||
|
||||
let mut count = 0usize;
|
||||
for (table, column) in &rows {
|
||||
let table_ref = format!("\"{schema}\".\"{table}\"");
|
||||
let seq: Option<(Option<String>,)> = sqlx::query_as(
|
||||
"SELECT pg_get_serial_sequence($1, $2)"
|
||||
).bind(&table_ref).bind(column).fetch_optional(pg).await
|
||||
.with_context(|| format!("get sequence for {table}.{column}"))?;
|
||||
|
||||
let seq_name = match seq.and_then(|(s,)| s) {
|
||||
Some(s) if !s.is_empty() => s,
|
||||
_ => continue,
|
||||
};
|
||||
|
||||
let (max_id,): (i64,) = sqlx::query_as(
|
||||
sqlx::AssertSqlSafe(format!("SELECT COALESCE(MAX(\"{column}\"), 0) FROM {table_ref}"))
|
||||
).fetch_one(pg).await
|
||||
.with_context(|| format!("max id for {table}.{column}"))?;
|
||||
|
||||
if max_id > 0 {
|
||||
sqlx::query("SELECT setval($1::regclass, $2, true)")
|
||||
.bind(&seq_name).bind(max_id)
|
||||
.execute(pg).await.ok();
|
||||
} else {
|
||||
sqlx::query("SELECT setval($1::regclass, 1, false)")
|
||||
.bind(&seq_name)
|
||||
.execute(pg).await.ok();
|
||||
}
|
||||
count += 1;
|
||||
}
|
||||
tracing::info!("reset {count} sequence(s)");
|
||||
Ok(())
|
||||
}
|
||||
Reference in New Issue
Block a user