tunasync是清华大学TUNA镜像源目前使用的镜像方案,github地址:https://github.com/tuna/tunasync
前端页面代码来自:https://github.com/weyo/mirrors
tunasync服务启动停止脚本:https://github.com/whsir/tunasync-bin
CentOS源列表:https://www.centos.org/download/mirrors/
EPEL源列表:https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL
当前以CentOS7系统为例,搭建一个基于内网的镜像仓库,建议硬盘至少2T以上,同步速度与网络带宽、磁盘读写速度有关!
1、安装依赖
1 |
yum install rsync wget |
2、下载tunasync
1 2 3 4 |
wget https://github.com/tuna/tunasync/releases/download/v0.3.7/tunasync-linux-bin.tar.gz tar xf tunasync-linux-bin.tar.gz mv tunasync /usr/bin/ mv tunasynctl /usr/bin/ |
3、创建仓库目录
1 |
mkdir -p /data/mirrors |
4、创建日志目录
1 |
mkdir -p /data/logs/tunasync |
5、创建tunasync配置文件
1 |
mkdir /etc/tunasync |
6、创建manager.conf配置文件
1 |
vi /etc/tunasync/manager.conf |
1 2 3 4 5 6 7 8 9 10 11 12 |
debug = false [server] addr = "127.0.0.1" port = 12345 ssl_cert = "" ssl_key = "" [files] db_type = "bolt" db_file = "/etc/tunasync/manager.db" ca_cert = "" |
manager部分参数说明:
addr:manager服务监听地址
port:manager服务监听端口
ssl_cert和ssl_key:证书配置
db_file:数据库文件位置
7、创建worker.conf配置文件
1 |
vi /etc/tunasync/worker.conf |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
[global] name = "test_worker" log_dir = "/data/logs/tunasync/{{.Name}}" mirror_dir = "/data/mirrors" concurrent = 10 interval = 240 [manager] api_base = "http://127.0.0.1:12345" token = "" ca_cert = "" [cgroup] enable = false base_path = "/sys/fs/cgroup" group = "tunasync" [server] hostname = "localhost" listen_addr = "127.0.0.1" listen_port = 6000 ssl_cert = "" ssl_key = "" [[mirrors]] name = "centos" provider = "rsync" upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/centos/" use_ipv6 = false [[mirrors]] name = "epel" provider = "rsync" upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/epel/" use_ipv6 = false |
worker部分参数说明:
log_dir:tunasync日志路径
mirror_dir:镜像的路径
concurrent:线程数
interval:同步周期,单位分钟
api_base:manager地址
name:镜像名称
upstream:upstream同步地址
8、下载tunasync启动停止脚本
1 2 3 4 |
wget -P /etc/init.d/ https://raw.githubusercontent.com/whsir/tunasync-bin/master/tunasync-manager wget -P /etc/init.d/ https://raw.githubusercontent.com/whsir/tunasync-bin/master/tunasync-worker chmod +x /etc/init.d/tunasync-manager chmod +x /etc/init.d/tunasync-worker |
9、启动服务
1 2 |
/etc/init.d/tunasync-manager start /etc/init.d/tunasync-worker start |
10、查看同步状态
此处12345为manager的端口
1 |
tunasynctl list -p 12345 --all |
11、添加计划任务
1 |
*/1 * * * * wget -q http://127.0.0.1:12345/jobs -O /data/mirrors/jobs.json |
12、配置前端web
1 2 |
rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm yum install wnginx |
13、配置nginx
1 |
vi /usr/local/nginx/conf/vhost/demo.conf |
修改后的内容如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
server { listen 80 default_server; server_name _; root /data/mirrors; location ~ ^/(centos|epel) { index index.html index.htm; autoindex on; autoindex_exact_size off; autoindex_localtime on; } error_page 500 502 503 504 /50x.html; #include enable-php56.conf; #include enable-php70.conf; #include enable-php71.conf; #include enable-php72.conf; #include enable-php73.conf; #include enable-php74.conf; location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$ { expires 30d; } location ~ .*\.(js|css)?$ { expires 12h; } access_log /data/logs/nginx/demo_access.log; error_log /data/logs/nginx/demo_error.log; } |
14、设置前端web
1 2 |
cd /data/mirrors wget https://down.whsir.com/downloads/mirrors_web.tar.gz |
重启nginx
1 |
/etc/init.d/nginx restart |
15、访问测试
访问URL即可看到以下内容,由于这两个源很大,同步时间会非常长
附:我这里再举个例子,基于docker-ce镜像同步
镜像同步脚本:https://github.com/tuna/tunasync-scripts
1、创建镜像同步脚本目录
1 2 |
mkdir /etc/tunasync/scripts/ cd /etc/tunasync/scripts/ |
2、下载docker-ce同步脚本
1 2 |
wget https://raw.githubusercontent.com/tuna/tunasync-scripts/master/docker-ce.py chmod +x docker-ce.py |
3、在worker中新增docker-ce同步配置
1 |
vi /etc/tunasync/worker.conf |
1 2 3 4 5 6 |
[[mirrors]] name = "docker-ce" provider = "command" upstream = "https://download.docker.com/" command = "/etc/tunasync/scripts/docker-ce.py" docker_image = "tunathu/tunasync-scripts:latest" |
4、安装docker-ce同步时所需依赖
1 2 |
yum install python3 python-pip pip3 install requests pyquery |
5、启动服务
1 2 |
/etc/init.d/tunasync-manager start /etc/init.d/tunasync-worker start |
6、修改nginx
在location中继续增加一个docker-ce
1 |
vi /usr/local/nginx/conf/vhost/demo.conf |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
server { listen 80 default_server; server_name _; root /data/mirrors; location ~ ^/(centos|epel|docker-ce) { index index.html index.htm; autoindex on; autoindex_exact_size off; autoindex_localtime on; } error_page 500 502 503 504 /50x.html; #include enable-php56.conf; #include enable-php70.conf; #include enable-php71.conf; #include enable-php72.conf; #include enable-php73.conf; #include enable-php74.conf; location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$ { expires 30d; } location ~ .*\.(js|css)?$ { expires 12h; } access_log /data/logs/nginx/demo_access.log; error_log /data/logs/nginx/demo_error.log; } |
7、重启nginx
1 |
/etc/init.d/nginx restart |
访问URL即可看到以下内容
前端web都是基于html写的,如果继续添加其他的源,可以自行更改
写在最后
手动同步方式
1 |
rsync -aHvh --no-o --no-g --stats --exclude .~tmp~/ --delete --delete-after --delay-updates --safe-links --timeout=120 --contimeout=120 rsync://mirrors.tuna.tsinghua.edu.cn/epel/ /data/mirrors/epel |
1 |
rsync -aHvh --no-o --no-g --stats --exclude .~tmp~/ --delete --delete-after --delay-updates --safe-links --timeout=120 --contimeout=120 rsync://mirrors.tuna.tsinghua.edu.cn/centos/ /data/mirrors/centos |
1 |
python3 /etc/tunasync/scripts/docker-ce.py --working-dir /data/mirrors/docker-ce/ |
docker-ce同步不建议使用上面python脚本方式,也可以通过rsync增量方式来同步清华大学站的docker-ce
1 |
rsync -aHvh --no-o --no-g --stats --exclude .~tmp~/ --delete --delete-after --delay-updates --safe-links --timeout=120 --contimeout=120 rsync://mirrors.tuna.tsinghua.edu.cn/docker-ce/ /data/mirrors/docker-ce |
PS:如果同步后发现清华大学的centos源有个权限的报错
rsync: readlink_stat("7.7.1908/isos/x86_64/.CentOS-7-x86_64-Everything-1908.iso.RjFDl5" (in centos)) failed: Permission denied (13)
可以考虑将centos源更换香港这个源:rsync://mirror.hostlink.com.hk/centos/
原文链接:通过tunasync搭建镜像源仓库,转载请注明来源!