Before acquiring an Oracle ZFS Appliance
As a first post in the series, it makes sense to just briefly set the scene for what lead up to us using this particular system. Maybe some of this can be useful to others in the same situation as we were.
Before actually acquiring these systems you need to consider whether they are the right option of course. What we needed and what we felt that the Oracle ZFS Appliances could deliver, are:
- Proper iSCSI storage delivered over Ethernet to Hyper-V or SQL Server systems
- High performance NFS suitable for both mail and web hosting as well as vSphere
- Easy and simple administration with little to no daily chores
- Simple pricing model on acquisition as well as expansion
- Low cost per TB while maintaining sufficient performance
- Snapshots on filesystems and volumes
- Asynchronous replication across sites
NetApp
Everyone knows NetApp - I don't know if anyone ever got fired for buying NetApp... First of all, the web user interface of the NetApp is not very impressive - it is very rudimentary and you have nearly no insight into what the appliance is doing. Which client is doing how many IOs, what are the latencies, how many writes versus reads are going to the bottom shelf, etc. etc. These are all questions you cannot answer by looking at the user interface. So that's a downer, if your job is to figure out why your storage infrastructure isn't delivering... Second, and this was probably what did it for us, was, we could simply not get decent NFS performance from it when doing IO on huge numbers of small files (typical Maildir setup). We were testing in a vendor lab with a vendor provided engineer and he ended up concluding that the performance was "just fine". We could see that if this was "just fine", then "just fine" would not cut it for us. So basically we decided against it because of manageability and performance.
IBM Storwize Unified
The new kid at the block (at that time). The Storwize is a block storage system (and probably a rather good one at that) that does iSCSI (and other block protocols if you need them), coupled with a clustered pair of GPFS serving front ends for your NFS (and CIFS) needs. Now, using GPFS in a cluster is (at least in theory) brilliant, because this is a properly clustered file system quite unlike both NetApp WAFL and Oracle ZFS. What this means, is, the clustered heads run active-active, and if one fails, you do not have downtime at all. There is no fail-over delay in an active-active setup. The UI was modern - I think it is the UI team from the XIV which got to do the UI for the Storwize too - pretty and functional. It had many more data points to inspect than the NetApp (but it doesn't come near the Oracle). Performance on the small file NFS workload was much better than the NetApp - it was, as I remember it, acceptable. In the end, it was a closer race between the Storwize and the Oracle - we needed site replication of data, and the Storwize could not do that at the time. This was one of the major factors in us choosing Oracle over Storwize. Everyone on the team would have their own angle on these systems of course, but a couple of downsides on the Storwize that I noted were:
- The pricing model - complicated set of licenses
- Upgrades - extra hardware and extra licenses required
- An architectural kludge; isolated block storage solution plus NFS heads hanging on top, managed by a UI that attempts to span both worlds - when you delete a file on the GPFS, that does not necessarily free space on the block volume, for example
- One of the few systems at the time to allow tiering between flash, fast disk and slow disk
- Proper cluster file system (GPFS) on NFS heads for zero-downtime node failure
- Vendor provided lab time, equipment and personnel for testing - very nice!
Oracle ZFS Appliance
So what happened when we tested the Oracle appliance? One was provided to us for testing and we ran the same workload on it as we did the others. A few things stood out when testing this system:
- File-systems and volumes provision instantly - there is no waiting for volume provisioning and formatting. A LUN or share of any size is instantly available when created.
- One large pool of storage is shared among all file systems and LUNs - this allows all-out thin provisioning. You can limit the maximum size of a file system by setting a quota (and you should). So basically, instead of re-sizing file systems the old fashioned way (which is often time consuming for the admin and resource intensive on the storage system, causing slowdowns for clients), you simply change the quota on the file system. Again, the change is instant with no impact on performance.
- Analytics. From the web user interface one can drill down into incredible detail on the system and workload. From starting points like Network IO, Disk IO, NFS operations etc. you can drill down "by client", "by latency", "by type of operation" and so forth - and you can drill down again and again. So displaying the latency of read operations to a particular drive for example, is a straight forward operation. I do not doubt that experts can do that using a command line or a third party tool on some other systems too, but on this system it is straight forward and intuitive.
- The magic numbers are gone. There are no practical limits. You can have any number of files of any size in any number of directories with any number of snapshots in any number of file systems. Most competing systems seem to have reachable limits on snapshot counts, file system sizes, LUN sizes and sometimes file sizes.