Hello selfhosted! Sometimes I have to transfer big files or a large amounts of small files in my homelab. I used rsync but specifying the IP address and the folders and everything is bit fiddly. I thought about writing a bash script but before I do that I wanted to ask you about your favourite way to achieve this. Maybe I am missing out on an awesome tool I wasn’t even thinking about.
Edit: I settled for SFTP in my GUI filemanager for now. When I have some spare time I will try to look into the other options too. Thank you for the helpful information.
yeah I also use SFTP using FileZilla. Or like everybody mentioned including yourself, rsync to sync files across computers. Or even scp.
What’s wrong with rsync? If you don’t like IP addresses, use a domain name. If you use certificate authentication, you can tab complete the folders. It’s a really nice UX IMO.
If you’ll do this a lot, just mount the target directory with sshfs or NFS. Then use rsync or a GUI file manager.
Just don’t run rsync as a daemon as that’s a security nightmare
Why would you do that? That sounds awful…
The daemon tracks file state, so the transfers start quicker because rsync doesn’t have to scan the filesystem.
Right, but if you’re transferring things that frequently, there are better solutions.
Not necessarily. Rsync deltas are very efficient, and not everything supports deltas.
It may very well be the correct tool for the job.
Anyway, problem fit wasn’t part of the question.
Yeah, there are probably a few perfect fits for it. I don’t rsync between machines very often, so the only use case I might have is backups, which is already well covered with a number of tools. Otherwise I just want to sync a few directories.
It is, rsync sends data in plain text. There is a optional password that is also sent in plain text.
I never even set up DNS for things that aren’t public facing. I just keep /etc/hosts updated everywhere and ssh/scp/rsync things around using their non-fqdn hostnames.
You could also use mDNS to the same effect.
People have already covered most of the tools I typically use, but one I haven’t seen listed yet that is sometimes convenient is
python3 -m http.serverwhich runs a small web server that shares whatever is in the directory you launched it from. I’ve used that to download files onto my phone before when I didn’t have the right USB cables/adapters handy as well as for getting data out of VMs when I didn’t want to bother setting up something more complex.Honestly, this is an easy way to share files with non-technical people in the outside world, too. Just open up a port for that very specific purpose, send the link to your friend, watch the one file get downloaded, and then close the port and turn off the http server.
It’s technically not very secure, so it’s a bad idea to leave that unattended, but you can always encrypt a zip file to send it and let that file level encryption kinda make up for lack of network level encryption. And as a one-off thing, you should close up your firewall/port forwarding when you’re done.
Rsync and NFS for me.
And me.
sftp
All my machines have my keys, nothing to set up, nothing to tear down.
Syncthing
rsync is indeed fiddly. Consider SFTP in your GUI of choice. I mount the folder I need in my file browser and grab the files I need. No terminal needed and I can put the folders as favorites in the side bar.
If you want to use the terminal though, there is
scpwhich is supported on both windows and Linux.Its just
scp [file to copy] [username]@[server IP]:[remote location]That’s essentially the same as rsync
Just slower if you already have some of the files there.
smb share if its desktop to desktop. If its from phone to PC, I throw it on nextcloud on the phone, then grab it from the web ui on pc.
Smb is the way to go if you have identity set up, since your PC auth will carry over for the connection to the smb share. Nextcloud will be less typing if not since you can just have persistent auth on the app / web.
Solid explorer on android is pretty useful too, it can access the SMB share. I use nextcloud for photo backup, but usually solid explorer for one off file transfers.
rclone. I have a few helper functions;
fn mount { rclone mount http: X: --network-mode } fn kdrama {|x| rclone --multi-thread-streams=8 --checkers=2 --transfers=2 --ignore-existing --progress copy http:$x nas:Media/KDrama/$x --filter-from ~/.config/filter.txt } fn tv {|x| rclone --multi-thread-streams=8 --checkers=2 --transfers=2 --ignore-existing --progress copy http:$x nas:Media/TV/$x --filter-from ~/.config/filter.txt } fn downloads {|x| rclone --multi-thread-streams=8 --checkers=2 --transfers=2 --ignore-existing --progress copy http:$x nas:Media/Downloads/$x --filter-from ~/.config/filter.txt }So I download something to my seedbox, then use
rclone lsd http:to get the exact name of the folder/files, and runtv "filename"and it runs my function. Pulls all the files (based on filter.txt) using multiple threads to the correct folder on my NAS. Works great, and maxes out my connection.WinSCP for editing server config
Rsync for manual transfers over slow connections
ZFS send/receive for what it was meant for
Samba for everything else that involves mounting on clients or other servers.
By “homelab”, do you mean your local network? I tend to use shared folders, kdeconnect, or WebDAV.
I like WebDAV, which i can activate on Android with DavX5 and Material Files, and i use it for Joplin.
Nice thing about this setup is that i also have a certificate secured OpenVPN, so in a pinch i can access it all remotely when necessary by activating that vpn, then disconnecting.
Magic wormhole is pretty dead simple https://magic-wormhole.readthedocs.io/en/latest/welcome.html#installation
I use this a lot at work for moving stuff between different test vms, as you don’t need to check IPs/hostnames
As I understand it, the establishing of the connection is reliant on a relay server. So this would not work on a local network without a relay server and would, by default, try to reach a server on the internet to make connections.
What do you mean by specifying IP address?
As a lazy person, I just prefer
sftpon thunar.rsync if it’s a from/to I don’t need very often
More common transfer locations are done via NFS
