This is more or less just a note for myself in the future (for Windows 11 anyways, who knows how Windows 12 will work), but I figure if someone on a search engine stumbles across this, it might be helpful to them as well.
tl,dr; Go to OneDrive's Sync and Backup section in Settings, unsync all the "important folders" from your computer to OneDrive, then check below for the Microsoft guide on which registry keys to change.
How OneDrive worked before Windows 11
Before Windows 11, OneDrive functioned much like the rest of the Microsoft (or any standard software company's) ecosystem did: it was there, it was a convenient default, it was somewhat hard to totally remove, and it was totally optional. Specifically, by default its usage was separated out from critical system folders, and was in general the same as any other cloud storage solution.
What changed in Windows 11
If you're upgrading from a previous version of Windows to 11, not much has changed with regards to OneDrive. If, however, you're doing a fresh install or buying a new machine, one major annoying change is that by default, the Documents, Desktop, and Pictures folders built into Windows are automatically synced to your OneDrive account.
I can see why Microsoft did this, business case (tighter integration with OneDrive = more usage = more $$$) aside. While some people only want to selectively back up certain important files or otherwise organize their cloud storage how they please, a lot of people will tend to store most, if not all of their important files in a cloud backup just by default. Many people especially want to do this with their documents and photos, hence the natural inclusion of those two folders. As for the Desktop, we all know of many, many people who basically store all their files that they regular use or download in their Desktop area, because it's immediate and easily accessible.
So if you use OneDrive already, or if you don't want to think about which files to backup, this kind of default would generally work. There are a couple of scenarios where this absolutely sucks, however:
You don't pay for OneDrive storage. In this case, you'd run up against your storage limits pretty quickly if you put even a relatively small number of picture or documents in those folders.
You tend to use the Desktop area to temporarily transfer large files, like from a phone or if you're copying over a hard drive or thumb drive
You have multiple machines and you don't want to share everything across all of them (especially the Desktop icons for any applications you have installed)
The list can go on, but generally it makes more sense to not have this feature enabled than it does to have it enabled unless you are paying for OneDrive and don't want to do the bare minimum to manage your files. However, if you happen to be anywhere close to a power user or have used Windows for more than a few years, this default can be especially annoying.
Fixing the OneDrive syncing issue
This section is short because there are only really a few steps:
Click on the OneDrive icon, go to Settings, then Sync and backup.
Click on Manage Backup next to "Back up important PC folders to OneDrive", and uncheck all the folders.
Make sure that the Windows Registry keys are changed following this guide: https://support.microsoft.com/en-us/topic/operation-to-change-a-personal-folder-location-fails-in-windows-ffb95139-6dbb-821d-27ec-62c9aaccd720
And that's it. Now you should have actual local folders for your computer, separate from whatever you want to store in OneDrive