chore: snapshot current aws standby and backup tooling work
This commit is contained in:
@@ -0,0 +1,286 @@
|
||||
# Hifast 新 RDS 生产切换清单（已完成归档）
|
||||
|
||||
本文档记录已完成的 `hifast-mysql-prod -> hifast-mysql-prod-v2` 生产切换过程，作为归档和回溯参考。
|
||||
|
||||
本次切换已经完成，当时的目标是避免继续在线硬改现有生产 RDS，而是：
|
||||
|
||||
1. 新建一台规范公网子网组的新 RDS
|
||||
2. 导入现有 `hifast` 数据
|
||||
3. 将 AWS 应用切到新 RDS
|
||||
4. 将 `104.238.220.230` 的外部从库改挂新 RDS
|
||||
5. 验收通过后再下线旧 RDS
|
||||
|
||||
## 1. 适用背景
|
||||
|
||||
当前现网已经可用，但还存在 1 个结构性问题：
|
||||
|
||||
- 为了让 `104` 通过公网白名单做 MySQL 复制，当时的 `hifast-mysql-prod` 仍依赖 `hifast-hk-private-1b` 临时挂公网路由
|
||||
|
||||
当前已准备好的收敛资源：
|
||||
|
||||
- 纯公网 DB subnet group:
|
||||
- `hifast-hk-rds-public-only-sgprep`
|
||||
- 安全组：
|
||||
- `hifast-hk-rds-core-sg`
|
||||
|
||||
## 2. 目标实例参数
|
||||
|
||||
建议新实例名称：
|
||||
|
||||
- `hifast-mysql-prod-v2`
|
||||
|
||||
建议参数如下：
|
||||
|
||||
- Region: `ap-east-1`
|
||||
- Engine: `MySQL`
|
||||
- Engine version: `8.4.8`
|
||||
- Templates: `Production`
|
||||
- DB instance class: `db.r7g.xlarge`
|
||||
- Storage type: `gp3`
|
||||
- Allocated storage: `200 GiB`
|
||||
- Provisioned IOPS: `3000`
|
||||
- Storage throughput: `125`
|
||||
- Multi-AZ: `No`
|
||||
- Publicly accessible: `Yes`
|
||||
- VPC: `vpc-09fc384522517debc`
|
||||
- DB subnet group: `hifast-hk-rds-public-only-sgprep`
|
||||
- VPC security group: `hifast-hk-rds-core-sg`
|
||||
- DB name: `hifast`
|
||||
- Master username: `admin`
|
||||
- Master password: 与现网保持一致
|
||||
- Backup retention: `7 days`
|
||||
- Performance Insights: `On`
|
||||
- Storage encryption: `On`
|
||||
- Deletion protection: `On`
|
||||
- Auto minor version upgrade: 建议保持与现网一致
|
||||
- Maintenance window: 可与现网同策略，建议维护窗口内切换
|
||||
|
||||
## 3. 切换前确认
|
||||
|
||||
切换前必须满足：
|
||||
|
||||
1. `104` 当前主从正常：
|
||||
- `Replica_IO_Running: Yes`
|
||||
- `Replica_SQL_Running: Yes`
|
||||
- `Seconds_Behind_Source: 0`
|
||||
2. AWS 应用机心跳正常：
|
||||
- `curl http://127.0.0.1:8080/v1/common/heartbeat`
|
||||
3. AWS Redis 主从正常：
|
||||
- AWS 主 `connected_slaves:1`
|
||||
- `104` 从 `master_link_status:up`
|
||||
4. 已确认当前应用真实配置文件：
|
||||
- `/opt/ppanel/configs/ppanel.yaml`
|
||||
5. 已确认当前应用重启方式：
|
||||
- `cd /opt/ppanel && docker compose -f docker-compose.cloud.yml up -d ppanel-server`
|
||||
|
||||
## 4. 创建新 RDS
|
||||
|
||||
已在 AWS 控制台创建 `hifast-mysql-prod-v2`，未对原 `hifast-mysql-prod` 做高风险在线子网调整。
|
||||
|
||||
创建完成后先确认：
|
||||
|
||||
1. 新 endpoint 已分配
|
||||
2. `Publicly accessible = Yes`
|
||||
3. SG 为 `hifast-hk-rds-core-sg`
|
||||
4. DB subnet group 为 `hifast-hk-rds-public-only-sgprep`
|
||||
5. 从 `104` 可以 TCP 连通 `3306`
|
||||
|
||||
连通性验证：
|
||||
|
||||
```bash
|
||||
nc -zv <NEW_RDS_ENDPOINT> 3306
|
||||
```
|
||||
|
||||
## 5. 导出旧主库
|
||||
|
||||
在一台可连旧 RDS 的机器执行：
|
||||
|
||||
```bash
|
||||
mysqldump \
|
||||
-h hifast-mysql-prod.cd6aey40m6ag.ap-east-1.rds.amazonaws.com \
|
||||
-u admin \
|
||||
-p \
|
||||
--single-transaction \
|
||||
--routines \
|
||||
--triggers \
|
||||
--events \
|
||||
--set-gtid-purged=OFF \
|
||||
hifast > hifast-full.sql
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- 当前业务库体量约 `187 MB`
|
||||
- 这条路线的成本低，且比现网主库在线改子网更稳
|
||||
|
||||
## 6. 初始化新 RDS
|
||||
|
||||
先连接新 RDS：
|
||||
|
||||
```bash
|
||||
mysql -h <NEW_RDS_ENDPOINT> -u admin -p
|
||||
```
|
||||
|
||||
建议先执行：
|
||||
|
||||
```sql
|
||||
CALL mysql.rds_set_configuration('binlog retention hours', 24);
|
||||
|
||||
CREATE USER IF NOT EXISTS 'repl'@'104.238.220.230' IDENTIFIED BY '<REPL_PASSWORD>';
|
||||
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl'@'104.238.220.230';
|
||||
FLUSH PRIVILEGES;
|
||||
```
|
||||
|
||||
如果库是空的，再导入：
|
||||
|
||||
```bash
|
||||
mysql -h <NEW_RDS_ENDPOINT> -u admin -p hifast < hifast-full.sql
|
||||
```
|
||||
|
||||
导入完成后确认：
|
||||
|
||||
```bash
|
||||
mysql -h <NEW_RDS_ENDPOINT> -u admin -p -e "USE hifast; SHOW TABLES;"
|
||||
```
|
||||
|
||||
## 7. 切换 AWS 应用到新 RDS
|
||||
|
||||
### 7.1 备份当前配置
|
||||
|
||||
在 AWS app EC2：
|
||||
|
||||
```bash
|
||||
cp /opt/ppanel/configs/ppanel.yaml /opt/ppanel/configs/ppanel.yaml.bak.$(date +%Y%m%d%H%M%S)
|
||||
```
|
||||
|
||||
### 7.2 修改数据库地址
|
||||
|
||||
编辑：
|
||||
|
||||
- `/opt/ppanel/configs/ppanel.yaml`
|
||||
|
||||
把：
|
||||
|
||||
```yaml
|
||||
MySQL:
|
||||
Addr: hifast-mysql-prod.cd6aey40m6ag.ap-east-1.rds.amazonaws.com:3306
|
||||
```
|
||||
|
||||
改成：
|
||||
|
||||
```yaml
|
||||
MySQL:
|
||||
Addr: hifast-mysql-prod-v2.cd6aey40m6ag.ap-east-1.rds.amazonaws.com:3306
|
||||
```
|
||||
|
||||
### 7.3 重启业务容器
|
||||
|
||||
```bash
|
||||
cd /opt/ppanel
|
||||
docker compose -f docker-compose.cloud.yml up -d ppanel-server
|
||||
```
|
||||
|
||||
### 7.4 应用验收
|
||||
|
||||
```bash
|
||||
curl -sf http://127.0.0.1:8080/v1/common/heartbeat
|
||||
docker compose -f /opt/ppanel/docker-compose.cloud.yml logs --tail=100 ppanel-server
|
||||
```
|
||||
|
||||
重点确认：
|
||||
|
||||
- 心跳返回 `200`
|
||||
- 日志中无 MySQL 连接异常
|
||||
|
||||
## 8. 改挂 `104` 从库到新 RDS
|
||||
|
||||
先在新 RDS 获取位点：
|
||||
|
||||
```sql
|
||||
SHOW MASTER STATUS;
|
||||
```
|
||||
|
||||
然后在 `104` 执行：
|
||||
|
||||
```sql
|
||||
STOP REPLICA;
|
||||
RESET REPLICA ALL;
|
||||
CHANGE REPLICATION SOURCE TO
|
||||
SOURCE_HOST='hifast-mysql-prod-v2.cd6aey40m6ag.ap-east-1.rds.amazonaws.com',
|
||||
SOURCE_PORT=3306,
|
||||
SOURCE_USER='repl',
|
||||
SOURCE_PASSWORD='<REPL_PASSWORD>',
|
||||
SOURCE_LOG_FILE='<MASTER_LOG_FILE>',
|
||||
SOURCE_LOG_POS=<MASTER_LOG_POS>,
|
||||
SOURCE_SSL=1;
|
||||
START REPLICA;
|
||||
SHOW REPLICA STATUS\G
|
||||
```
|
||||
|
||||
验收标准：
|
||||
|
||||
- `Replica_IO_Running: Yes`
|
||||
- `Replica_SQL_Running: Yes`
|
||||
- `Seconds_Behind_Source: 0`
|
||||
|
||||
## 9. 切换后验收
|
||||
|
||||
至少做下面这些检查：
|
||||
|
||||
1. AWS 应用心跳正常
|
||||
2. 管理后台可登录
|
||||
3. 新建一条测试数据后，主库可见
|
||||
4. `104` 从库能同步到该测试数据
|
||||
5. Navicat 可从白名单来源连接新 RDS
|
||||
6. Redis 主从仍正常，不受本次 MySQL 切换影响
|
||||
|
||||
推荐额外核对：
|
||||
|
||||
```bash
|
||||
docker compose -f /opt/ppanel/docker-compose.cloud.yml ps
|
||||
docker exec hifast-redis redis-cli -a '<REDIS_PASSWORD>' INFO replication
|
||||
```
|
||||
|
||||
## 10. 回滚方案
|
||||
|
||||
如果切换后 AWS 应用异常：
|
||||
|
||||
1. 立刻把 `/opt/ppanel/configs/ppanel.yaml` 中 `MySQL.Addr` 改回旧 endpoint
|
||||
2. 重启 `ppanel-server`：
|
||||
|
||||
```bash
|
||||
cd /opt/ppanel
|
||||
docker compose -f docker-compose.cloud.yml up -d ppanel-server
|
||||
```
|
||||
|
||||
3. 优先恢复主生产可用
|
||||
4. `104` 是否回切旧 RDS 复制，视恢复窗口决定
|
||||
|
||||
如果只是 `104` 复制改挂失败，但 AWS 应用已正常使用新 RDS：
|
||||
|
||||
- 不必立刻回滚应用
|
||||
- 先修好 `104 -> 新 RDS` 的白名单、位点、用户或网络
|
||||
|
||||
## 11. 切换完成后的收尾
|
||||
|
||||
全部验收通过后再做：
|
||||
|
||||
1. 已将新 endpoint 记录到正式运维文档
|
||||
2. 仍需根据回收窗口安排旧 RDS 下线
|
||||
3. 仍需按网络收敛计划处理 `private-1b` 路由语义
|
||||
4. 已更新外部复制 runbook 中的主库 endpoint
|
||||
|
||||
## 12. 当前建议的执行顺序
|
||||
|
||||
本次切换当时按以下顺序执行：
|
||||
|
||||
1. 创建 `hifast-mysql-prod-v2`
|
||||
2. 验证新 RDS 网络与参数
|
||||
3. 导出旧库
|
||||
4. 导入新库
|
||||
5. 创建 / 确认 `repl` 用户
|
||||
6. 切 AWS 应用到新 RDS
|
||||
7. 验证应用
|
||||
8. 改挂 `104` 到新 RDS
|
||||
9. 验证从库
|
||||
10. 收尾，并保留旧 RDS 待后续下线
|
||||
Reference in New Issue
Block a user