In the next ~6 months I’m going to entirely overhaul my setup. Today I have a NUC6i3 running Home Assistant OS, and a NUC8i7 running OpenMediaVault with all the usual suspects via Docker.
I want to upgrade hardware significantly, partially because I’d like to bring in some local LLM. Nothing crazy, 1-8B models hitting 50tps would make me happy. But even that is going to mean a beefy machine compared to today, which will be nice for everything else too of course.
I’m still all over the place on hardware, part of what I’m trying to decide is whether to go with a single machine for everything or keep them separate.
Idea 1 is a beefy machine and Proxmox with HA in a VM, OMV or TrueNAS in another, and maybe a 3rd straight Debian to separate all the Docker stuff. But I don’t know if I want to add the complexity.
Idea 2 would be beefy machine for straight OMV/TrueNAS and run most stuff there, and then just move HA over to the existing i7 for more breathing room (mostly for Frigate, which could also separate to other machine I guess).
I hear a lot of great things about Proxmox, but I’m not sold that it’s worth the new complexity for me. And keeping HA (which is “critical” compared to everything else) separated feels like a smart choice. But keeping it on aging hardware diminishes that anyway, so I don’t know.
Just wanting to hear various opinions I guess.
You must log in or # to comment.
No.
In my opinion, Proxmox is worth it for two reasons:
-
Easy high-availability setup and control
-
Proxmox Backup Server
Those two are what drove me to switch from KVM, and I don’t regret it at all. PBS truly is a fantastic piece of software.
Upvoted for PBS alone. Incremental backups that are rock solid mean you can completely brick your server and have it back to normal in minutes
-
From an earlier post I made much like yours, I decided to go with incus. I’d be fully migrated if real life hadn’t kicked me in the taint for a few weeks.
Not sure what youre doing with OMV that couldn’t be done in proxmox, so feel free to elaborate there.
Almost all my servers are proxmox (some just Debian, though a few more specific work related solutions are lurking about). For docker I’d do an LXC, btw, I wouldn’t bother with a full VM.
My (excessive) setup is all proxmox, set up as a high availability cluster. HA runs in a VM, and my USB devices are passed through (technically its USB over IP extension, so the USB devices for various VMs continually pass through even if I have to shut a server down).
Its where Jellyfin, Audiobookshelf, homepage.dev, a bajillion stupid containers I mostly dont need, DNS, monitoring and analytics, mealie (recipe server), various websites I host, etc, etc all live. Nothing is by itself on a box except my workstations, but for non-linux use I have VMs I remote into (mostly industry specific software and random crap like an xp VM to use an old piece of hardware).
Can you quickly run me through how USB over IP is helping you out? I get it for devices that are physically distant, but how is the abstraction helping you for reboots? Isn’t it just the server you’re rebooting that talks to the USB device anyway?
I have a single.ip transmitter and multiple receivers, IP controllable and routable.
If VM1 uses USB device1 on RX1 from tx1, and host1 goes down, when VM1 is going to be run on host2, rx2 is switched as the receive from tx1, and VM1 still has access to the USB device.
For the record, icron 2304s I got because of work stuff (that accepts commands, which are the version they only oem now).
Ah, got it, it’s for VM migrations. That makes a lot more sense.
Ah yeah, sorry, didnt realize that wasn’t clear.
Only one machine at a time handles USB devices by design - OTA TV tuner, zigbee/zwave, USB to serial adapter, and an 8 channel relay.
For me, I’m Team Proxmox. It’s just easy to spin up containers for pretty much anything I need. No need for the resource overhead of a full-on virtual machine if I simply need to run a LAMP app. Anything you really have an issue transitioning from Docker to LXC can still be run inside a container with Docker installed. And if you need to set up a VM for Windows or pfSense or some other OS for whatever reason, it’s insanely easy to do.
Proxmox adds a lot of complexity and a nice GUI. If you are fine with using the terminal, there is really not much benefit from Proxmox and the potential issues from the added complexity are IMHO not worth it. I am not a Proxmox expert though, so take this advise with a grain of salt 😅
Is it decently easy to create and manage vm’s and containers with the terminal? I use proxmox at the moment. Should I switch to Ubuntu server?
Should I switch to Ubuntu server?
Thats a hard no IMO.
Even if you want to do something other than proxmox (just use Debian, fedora, or opensuse).
Its not bad from the CLI, you just need to know your commands.
virt-install --name=deb13-vm --vcpus=1 --memory=1024 --cdrom=/tmp/debian-13.0.0-amd64-netinst.iso --disk size=8 --os-variant=debian13Will get you 1 vcpu, 1GB ram, and an 8GB drive worth of debian. If you don’t specify a path, in home under .local/share/libvirt/images it will go!
You can also then
virsh edit deb13-vmAnd you’ll get the XML, where you can edit away.
Personally, I’d rather use the webgui for most things, but yeah its perfectly doable from the CLI.
I would have thought debian is better than ubuntu but I couldn’t find a server version of debian. Where do I find debian server or debian cli only?
Debian is by default suited for server. Just skip the desktop environment part in the installer.
Oh ok, I’ve never installed debian before so thats good to know.
With libvirt it is fairly easy yes. And you can also install a standalone web-gui like Cockpit or use the desktop app virt-manager over ssh to do it.
I’ve been using Ganeti for like 15 years now, and I’m not sure what proxmox offers besides a nice GUI. I know how Ganeti works and getting up to speed on a new one doesn’t seem super interesting to me. Is anyone here familiar with both?
Ganeti development is more or less dead. If you look at the github repo, it hasn’t seen a notable release in 4 years. All that’s been done is a small bugfix patch two months ago by the community.
The project being based on Haskell code also makes it less attractive for new devs.
Stable. :)
It’s great if you need what it offers. Otherwise, it’s simpler to set up something like Ubuntu Server.
I use Proxmox to run my email service, https://port87.com/, because I can have high-availability services that can move around the different Proxmox hosts. It’s great for production stuff.
I also use it to run my seedbox, because graphics in the browser through Proxmox is really easy.
For everything else (my Jellyfin, Nextcloud, etc), I have a server that runs Ubuntu Server and use a docker compose stack for each service.
I had never heard of Port87 before, how do you like it? And I assume you pay no monthly fee by hosting your own domain?
I meant that I made it. :) It’s my own email service, and I run it on Proxmox. So, take this with a grain of salt knowing that I wrote and run it, but I think it’s the best email service by far. I wrote an article about how it works really well for me here:
https://sciactive.com/2023/07/17/the-best-email-for-those-who-struggle-with-organization/
Feel free to sign up for free and try it out. :D
Interesting.
I have an old free email provider that’s just passed the email service to another provider
I’m looking to move because I used to be able to use <anything-at-all>@my-email.domain and I’m not sure I’ll be able to do that anymore
I basically do what you’re doing - using email prefixes for the site I’m registering with… I even caught a company out once when I suddenly started getting spam from that email address. They’d sold my details…
You should check out Port87. :) You wouldn’t need to change any of your addresses if you bring your domain on. Custom domains is $10/month though, so it would cost you more. Hopefully the features would be worth it for you, and if not, you can always migrate it again to a different provider. That’s something I love about email. If you have your own domain, you can completely avoid vendor lock in.
Don’t use Proxmox, use incus. It’s way easier to run and doesn’t give a care about your storage.
No backup utility like PBS though, thats why I haven’t switched.
Like I said, incus don’t care about your storage.
I’ve never used PBS, I’ve always just rolled my own. I currently keep 7 daily, 4 weekly and 4 monthly. My data mounts are all nfsv4.
Edit: isnt it possible to use pbs with non-proxmox systems?
Yeah it sounds nice but too much time investment for me.
I can install PBS client on any system but it requires manual setup and scheduling which I don’t want to do. When used with Proxmox that’s all handled for me.
Also I don’t think Proxmox cares about storage either, I just use ZFS which is completely standard under the hood.
Also I don’t think Proxmox cares about storage either,
Proxmox forces you to add a “storage area”, which is fine, except you must use their mount path of /mnt/pve/ and you must add NFS tuning switches via pve or they don’t work.
Proxmox is great, I used it for 8 years. But it is also opinionated and doesn’t like non-standard configs.
Oh I see what you mean yeah, I’ve never used NFS before with it.
I like Incus a lot, but it’s not as easy to create complex virtual networksnas it is with proxmox, which is frustrating in educational/learning environments.
Best thing to do is give it a go and see what shakes out OP. I absolutely love both my Proxmox boxes. In my humble opinion, Proxmox was an easier set up, and the possibilities are endless really. It’s a solid freemium product. Couple it with the extensive Helper Scripts, and Jack’s a doughnut, Bob’s your uncle.
I will always recommend Proxmox, not just because it’s really easy to add more stuff, but because it’s really safe to tinker with. You take a snapshot, start messing around, and if you break something you just revert to the snapshot
This. Even if you were going to run a bare metal server it’s almost always nicer to install Proxmox and just have a single VM
This is how I run my OPNsense router. Snapshots are great and rebooting is SO much faster!
Uh. OpnSense on bare metal can also do snapshots, if you set it up correctly…
I like ProxMox too, I’m quite happy that I dove in with it. Just one word of warning - if you mount a drive volume in a container, destroy the container and restore it from a backup, it wipes out the mounted drive. I, uh, lost a bunch of data that way. Not super important data, but still.
I’m still glad I went with ProxMox though. It makes spinning up something a breeze, and I also went with HA in a VM, and another Debian VM for Docker, and a bunch of random LXCs.
Is this separate from a bind mount? Cause that doesn’t happen with bind mounts.
Yeah, not a bind mount. There was a warning, but I was restoring a ton of LXCs and clicked through the warning too fast. My fault, I’m not super sore about it, just warning others as a service to prevent what happened to me!
Fair enough!
If you can replicate it, you should really file a bug report so that the next guy doesn’t lose data.
It tells you it will happen when you use the restore backup feature.
I’m running Proxmox and hate it. I still recommend it for what you are trying to do. I think it would work quite nicely. Three of my four nodes have llama.cpp VMs hosting OpenAI-compatible LLM endpoints (llama-server) and I run Claude Code against that using a simple translation proxy.
Proxmox is very opinionated on certain aspects and I much prefer bare metal k8s for my needs.
Agreed. Proxmox is not worth the complexity. Install truenas, you can put all the apps on that and you can have home assistant in VM. All you need is one machine. I actually have this setup. I only installed Proxmox on other machine to test it and to install OpenWRT and a bunch of networking software on that. If I feel confident, I will use this as my new router, but that’s long way to go. Oh, and by the way, Truenas in virtual machine is not recommended. I originally thought I will also install Truenas in Proxmox, but after reading plenty resources and things about pass-through I finally decided that I was stick to the recommendation and not use proxmox with truenas. I do not regret the decision.
Don’t add a layer of abstraction until you need it, or you have the free time to learn it well enough that it won’t cause you problems while you experiment.
