• 0 Posts
  • 40 Comments
Joined 2 years ago
cake
Cake day: November 5th, 2023

help-circle
  • greyfox@lemmy.worldtoSelfhosted@lemmy.worldRaid Z2 help
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    1 month ago

    Well I am not claiming to be a ZFS expert, I have been using it since 2008ish both personally and professionally. So I am fairly certain what I have said here is correct semantics aside.

    Both lz4 and zstd have almost no performance impact on modern hardware.

    So “almost” is zero in your mind? Why waste the CPU cycles on compressing data that is already compressed? I recognize that you might not care, but I sure do. And I wouldn’t say it would be wrong to think that way.

    compression acts on blocks in ZFS, therefore it is enabled at the pool level

    This is incorrect. You can zfs set compression=lz4 dataset (or off) on a per dataset basis. You can see your compression efficiency per dataset by running zfs get compressratio dataset, if your blocks were written to a dataset with compress=off you will see no compression for that dataset. You can absolutely mix compressed and uncompressed datasets in the same pool.

    OP added a -O option to set compression when he created the pool but that is not a pool level setting. If you look at the documentation for zpool-create you will see that -O are just properties passed to the root dataset verses -o options which are actual pool level parameters.

    You might be confusing compression and deduplication. Deduplication is more pool wide.

    ZFS does indeed need to allocate some space at the front and end of a pool for slop, metaslab, and metadata. I think you are confusing filesystem and datasets.

    Well yes and no here. You are right I should have been calling them datasets. Datasets are a generic term, and there are different dataset types, like file systems, volumes and snapshots.

    So yeah I maybe should have been more generic and called them datasets but unless OP is using block volumes we are probably talking about ZFS file systems here. Go to say the zfsprops man page and you will see file system mentioned about 60 times when discussing properties that can be set for file type datasets.

    I’m not sure what you’re trying to say about NFS and ZFS, here but this is completely false, even if you mean datasets.

    It sounds like you are unaware of the native NFS/SMB integrations that ZFS has.

    It is totally optional but instead of using your normal /etc/exports to set NFS settings ZFS can dynamically load export settings when your dataset is mounted.

    This is done with the sharenfs parameter zfs set sharenfs=<export options> dataset. Doing this means you keep your export settings with your pool instead of the system it is mounted on, that way if you say replicate your pool to another system those export settings automatically come with it.

    There are also sharesmb options for samba.

    My point was then that you should lay out your dataset hierarchy based on your expected permissions for NFS/SMB. You could certainly skip all of this and handle these exports manually yourself in which case you wouldn’t have to worry about separate filesystems and this point is moot.

    My post was less about compression and more about saying that you should consider splitting your datasets based on what is in them because the more separate they are the more control you have. You gain a lot of control, and lose very little since it all comes from the same pool.

    Some of the reasons I have these options are less important than they were a decade ago. i.e. doing a 20tb ZFS send before resuming a send was possible sucked. Any little problem and you have to start over. Having more smaller filesystems meant smaller sends. And yeah I was using ZFS before lz4 was even an option and CPU was more precious back then, but I don’t see any reason to waste CPU cycles when you can create a separate file system for your media and set compression to off on it.

    And most importantly I would want different snapshot policies for different data types. I don’t need years worth of retention for a movie collection, but I would like to have years worth of retention on my documents filesystem because it is relatively small so the storage consumed is minimal to protect against accidental deletion.


  • greyfox@lemmy.worldtoSelfhosted@lemmy.worldRaid Z2 help
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 month ago

    Yeah it won’t make much difference these days.

    I suppose my point was more so that because ZFS is a pool that can be split up with filesystems new users should be thinking a little differently than they would have been used to with traditional raid volumes/partitions.

    With a normal filesystem partitions are extremely limiting, requiring you to know how much space you need for each partition. ZFS filesystems just being part of the pool means that you can get logical separation between data types without needing that kind of pre-planning.

    So many settings with ZFS that you may want to set differently between data types. Compression, export settings, snapshot schedules, replicating particular data sets to other systems, quotas, etc.

    So I was mostly just saying “you should consider splitting those up so that you can adjust settings per filesystem that make sense”.

    There is also a bit of danger with a single ZFS filesystems if you have no snapshots. ZFS being a copy on write filesystem means that even deleting something actually needs space. A bit counter intuitive but deleting something means writing a new block first then updating the FS to point at the new block. If you fill the pool to 100% you can’t delete anything to free up space. Your only option is to delete a snapshot or delete entire filesystems to free up a single block so that you can cleanup. If you don’t have a snapshot to delete you have to delete the entire filesystem and if you only have one filesystem you need to backup+delete everything… ask me how I know this ;)

    If you have several filesystems you only need to backup and destroy the smallest one to get things moving again. Or better yet have some snapshots you can roll off to free up space or have quotas in place so that you don’t fill the pool entirely.



  • greyfox@lemmy.worldtoSelfhosted@lemmy.worldRaid Z2 help
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    1
    ·
    1 month ago

    You probably shouldn’t enable compression on the root filesystem of the pool. Since you mention movies/TV shows/music those are just going to waste cpu cycles compressing uncompressable data.

    Instead you should consider separate ZFS filesystems for each data type. Since ZFS is a pool you don’t have to pre-allocate space like partitions so there is no harm in having separate filesystems for each data type rather than single large filesystem for everything. You can then turn on compression only for those filesystems that benefit from it.

    Also remember that many permissions like nfs export settings are done on a per filesystem basis so you should lay out your filesystems according to your data type and according to what permissions you want to give out for that filesystem.

    i.e. if you are going to have a Navidrome server to stream your music you don’t want to give that server access to your entire pool, just your music.

    Separate filesystems also means you can have different snapshot schedules/retentions. Documents might need to snapshot more often, and be kept around longer than media snapshots.



  • +1 to this. Lots of talk in this thread about drivers, but the only driver involved here is the Bluetooth driver. Half of the point of Bluetooth is that peripherals don’t need their own drivers, they just provide various profiles which are standardized so the Bluetooth service can consume those profiles from any device.

    Not an expert in this area but I believe the implementation of most of those profiles is user space, so the proper place to be debugging is the Bluetooth service or in pulsesudio. So start your Bluetooth service logs they might give you some idea as to what is going on. Try to get a list of what profiles are supported by your OS and what profiles are supported by the device, maybe the device only supports some newer lossless profile that hasn’t been implemented in Linux yet.






  • I don’t use the WebOS app but generally default subtitles/audio languages are set on your profile and the apps pick up those settings.

    Try logging in to the web interface and going to your user profile. There is a “Playback” section where you can set your preferred languages. If this isn’t set it likely is taking the default language from your media files instead.





  • Unauthorized VPNs (non government approved) are illegal in China. If a business needs their own they can get approval but they have to apply for those exceptions.

    It isn’t really enforced, probably especially so for non citizens, but if you do something they don’t like it is something they could use against you.

    You would probably be less breaking the law to just directly open up SSH and access that instead of tunneling through a VPN. Even though SSH can do tunneling of its own.



  • Nope, the switch only keeps saves on the internal storage or synced to their cloud if you pay for it. When doing transfers between devices like this there is no copy option only a move and delete.

    There are some legitimate reasons they want to prevent this like preventing users from duplicating items in multiplayer games, etc. Even if you got access to the files they are encrypted so that only your user can use them.

    I think the bigger reason they do this is there are occasionally exploits that are done through corrupted saves. So preventing the user from importing their own saves helps protect the switch from getting soft modded.

    If you mod your switch you can get access to the save files and since it has full access it can also decrypt them, so that you can back them up. One of several legitimate reasons to mod your switch.



  • Named volumes are often the default because there is no chance of them conflicting with other services or containers running on the system.

    Say you deployed two different docker compose apps each with their own MariaDB. With named volumes there is zero chance of those conflicting (at least from the filesystem perspective).

    This also better facilitates easier cleanup. The apps documentation can say “docker compose down -v”, and they are done. Instead of listing a bunch of directories that need to be cleaned up.

    Those lingering directories can also cause problems for users that might have wanted a clean start when their app is broken, but with a bind mount that broken database schema won’t have been deleted for them when they start up the services again.

    All that said, I very much agree that when you go to deploy a docker service you should consider changing the named volumes to standard bind mounts for a couple of reasons.

    • When running production applications I don’t want the volumes to be able to be cleaned up so easily. A little extra protection from accidental deletion is handy.

    • The default location for named volumes doesn’t work well with any advanced partitioning strategies. i.e. if you want your database volume on a different partition than your static web content.

    • Old reason and maybe more user preference at this point but back before the docker overlay2 storage driver had matured we used the btrfs driver instead and occasionally Docker would break and we would need to wipe out the entire /var/lib/docker btrfs filesystem, so I just personally want to keep anything persistent out of that directory.

    So basically application writers should use named volumes to simplify the documentation/installation/maintenance/cleanup of their applications.

    Systems administrators running those applications should know and understand the docker compose well enough to change those settings to make them production ready for their environment. Reading through it and making those changes ends up being part of learning how the containers are structured in the first place.


  • For shared lines like cable and wireless it is often asymmetrical so that everyone gets better speeds, not so they can hold you back.

    For wireless service providers for instance let’s say you have 20 customers on a single access point. Like a walkie-talkie you can’t both transmit and receive at the same time, and no two customers can be transmitting at the same time either.

    So to get around this problem TDMA (time division multiple access) is used. Basically time is split into slices and each user is given a certain percentage of those slices.

    Since the AP is transmitting to everyone it usually gets the bulk of the slices like 60+%. This is the shared download speed for everyone in the network.

    Most users don’t really upload much so giving the user radios equal slices to the AP would be a massive waste of air time, and since there are 20 customers on this theoretical AP every 1mbit cut off of each users upload speed is 20mbit added to the total download capability for anyone downloading on that AP.

    So let’s say we have APs/clients capable of 1000mbit. With 20 users and 1AP if we wanted symmetrical speeds we need 40 equal slots, 20 slots on the AP one for each user to download and 1 slot for each user to upload back. Every user gets 25mbit download and 25mbit upload.

    Contrast that to asymmetrical. Let’s say we do a 80/20 AP/client airtime split. We end up with 800mbit shared download amongst everyone and 10mbit upload per user.

    In the worst case scenario every user is downloading at the same time meaning you get about 40mbit of that 800, still quite the improvement over 25mbit and if some of those people aren’t home or aren’t active at the time that means that much more for those who are active.

    I think the size of the slices is a little more dynamic on more modern systems where AP adjusts the user radios slices on the fly so that idle clients don’t have a bunch of dead air but they still need to have a little time allocated to them for when data does start to flow.

    A quick Google seems to show that DOCSIS cable modems use TDMA as well so this all likely applies to cable users as well.