Earlier this month Dell EMC announced XtremIO X2, the next generation hardware platform of their once flagship all-flash storage offering.  With the new systems, storage capacity per node has been increased from 25 to 72 drives.  The numbers quoted for initial deployments and upgrades seems to counter the original design of data protection on XtremIO, otherwise known as XDP.  Has Dell EMC therefore abandoned this technology, or is it simply a rewrite?

XDP

Perhaps an acronym for XtremIO Data Protection, XDP is a modified diagonal parity RAID-6 variant that in the original XtremIO X1 models delivered only an 8% overhead on capacity, with low write amplification on flash of 1.22x the logical I/O requests.  This efficiency was achieved through a 23+2 RAID-6 protection scheme, that uses 28 rows of data per  “group” (I deliberately quote this loosely defined term).  If you want to find more about XDP, there’s a specific data protection white paper from EMC – XtremIO Data Protection (XDP) that goes into detail on the protection scheme.  Although I don’t know for definite, my assumption is that the protection scheme uses 28 rows (rather than k-1, where k is the number of data drives) because this leaves plenty of space for partial or full writes as the array starts to get full (plus adheres to the prime number requirements, see later).

Patent

Looking at the original patent for XDP, the design says that the RAID-6 diagonal parity requires a number of data disks (k) that are prime, with k-1 rows to build out the parity, where the number of rows plus one should be prime.  The examples used in the above white paper are based on 5 data disks with 4 rows.  In XtremIO X1, the data disk count is either 23 or 11 respectively with either a full or “starter X-brick”.  The number of rows is 28 (which is one less than 29, a prime).  The details of the patent indicate that there is a slight additional overhead of the protection scheme RDP (not yet branded XDP presumably), which is mitigated by increasing the disk count k.

X2

Now let’s look at X2.  The published announcements and the details contained in Itzik Reich’s blog (CTO for XtremIO) show a capability of up to 72 drives in a new DAE (Disk Array Enclosure).  The initial starting point is 18 drives, with SSDs added in packs of 6.  From other comments I’ve read, at greater than 36 drives a second XDP group is created, so for 72 drives this presumably means 2x 36 XDP disk groups at a full expansion.  Based on our discussion above, the new drive options raise questions about how XDP is now implemented.  Previously the scheme was based on a fixed 25 (or 13) drive architecture, with no spares and dual parity.  Now we see drive configurations of 18, 24, 30 and 36 drives, none of which meet the “prime directive” for RDP/XDP, even if we took off some drives as spares.

So what does this mean for XDP in the new X2?  Have Dell EMC redesigned XDP to work differently?  I can see a scenario where the new XtremIO stripe size is smaller than the number of drives present, in the initial instance, the lowest prime less than 18 and accommodating 2 parity drives would be 13, making the new protection system 13+2.   Rather than having a fixed parity stripe across all drives, is conceivable that XDP v2 could be writing 13+2 stripes across 18 drives.  Bearing in mind that XDP already had an 11+2 implementation (starter X-Brick) then this could now be the stripe size for all systems.

What happens at expansion time?  When a further 6 drives are added, is the RAID strip size expanded or left the same?  Expanding the stripe size wouldn’t make much sense as the system would then be unbalanced from a performance perspective and at risk of space fragmentation.  So, presumably the size is kept consistent irrespective of the number of drives in the system.

The Architect’s View

It’s good to see customers now have expansion options for fully exploiting their XtremIO X2 investment.  It may seem a bit picky to be questioning the way in which data protection is now implemented on XtremIO X2.  However consider that when the original platform was launched, the XDP savings were heavily promoted.  2/25 drives gives the 8% overhead; with 11+2 this results in a 15.3% overhead – a significant difference.  As a customer, I’d like to know what exactly is going on under the covers, both from a performance and capacity perspective.  Previously EMC was keen to share the benefits of XtremIO.  So, why the lack of detail now?

Further Reading

Comments are always welcome; please read our Comments Policy first.  If you have any related links of interest, please feel free to add them as a comment for consideration.  

Copyright (c) 2009-2017 – Chris M Evans, first published on https://blog.architecting.it, do not reproduce without permission.

We share because we care!Share on Facebook0Share on Google+0Tweet about this on TwitterShare on LinkedIn0Buffer this pageEmail this to someoneShare on Reddit0Share on StumbleUpon0

Written by Chris Evans

  • Itzik Reich

    Hi Chris,thanks for the write up.
    below you can see my comments..let us know if you want to have a deeper dive

    Earlier
    this month Dell EMC announced XtremIO X2, the next generation hardware platform
    of their once flagship all-flash storage offering. With the new systems,
    storage capacity per node has been increased from 25 to 72 drives. The
    numbers quoted for initial deployments and upgrades seems to counter the
    original design of data protection on XtremIO, otherwise known as XDP.
    Has Dell EMC therefore abandoned this technology, or is it simply a rewrite?

    [Itzik Reich]: The XDP mechanism was not changed at all, the
    design of XDP does not limit the number of drives in XDP, it gives a relation
    between the number of drives in a stripe (that is the protected sub-element in
    the XDP) and the number of rows in stripe, that must be lower than the number
    of drives(also no assumption was taken on the height of the row ). Not all the
    drives must exists in XDP to support its efficiency and protection.

    XDP

    Perhaps
    an acronym for XtremIO Data Protection, XDP is a modified diagonal parity
    RAID-6 variant that in the original XtremIO X1 models delivered only an 8%
    overhead on capacity, with low write amplification on flash of 1.22x the
    logical I/O requests. This efficiency was achieved through a 23+2 RAID-6
    protection scheme, that uses 28 rows of data per “group” (I deliberately
    quote this loosely defined term). If you want to find more about XDP,
    there’s a specific data protection white paper from EMC – XtremIO Data Protection (XDP) that
    goes into detail on the protection scheme. Although I don’t know for
    definite, my assumption is that the protection scheme uses 28 rows (rather than k-1,
    where k is the number of data drives) because this leaves plenty of space for partial or
    full writes as the array starts to get full (plus adheres to the prime number requirements, see later).

    [Itzik Reich]: The XDP was not modified. The 28 rows used in X1,
    just represent the P-1, of the RAID algorithm, were P is a prime larger than
    the number of drives(in X1 P=29). The requirement of P being a Prime, is
    related to the required guarantee that double failure can be recovered using
    diagonal and row parities. In X2 we use 34+2 RAID-6 variant protection with
    verging number of rows depending on compression scheme. Stripe size is fixed
    cross all compression types. XDP was conversely designed to excel when dealing
    with partial stripe writes, which becomes the normal, steady-state operating
    condition when an array has been in service for some time.

    Patent

    Looking
    at the original patent for
    XDP, the design says that the RAID-6 diagonal parity requires a number of data
    disks (k) that are prime, with k-1 rows to build out
    the parity, where the number of rows plus one should be prime. The
    examples used in the above white paper are based on 5 data disks with 4
    rows. In XtremIO X1, the data disk count is either 23 or 11 respectively
    with either a full or “starter X-brick”. The number of rows is 28 (which
    is one less than 29, a prime). The details of the patent indicate that
    there is a slight additional overhead of the protection scheme RDP (not yet
    branded XDP presumably), which is mitigated by increasing the disk count k.

    [Itzik Reich]: this still stays correct, with the adjustment to
    34+2

    X2

    Now let’s look at X2. The published announcements and the details contained
    in Itzik Reich’s blog (CTO for XtremIO) show a capability of up to 72 drives in a new DAE (Disk Array
    Enclosure). The initial starting point is 18 drives, with SSDs added in
    packs of 6. From other comments I’ve read, at greater than 36 drives a
    second XDP group is created, so for 72 drives this presumably means 2x 36 XDP
    disk groups at a full expansion. Based on our discussion above, the new
    drive options raise questions about how XDP is now implemented.
    Previously the scheme was based on a fixed 25 (or 13) drive architecture, with
    no spares and dual parity. [Itzik Reich]: XDP architecture gives full flexibility on how
    many drives can it sustain, it’s more of the question of balance between XDP
    efficiency and MTBF consideration. Easy way to think about the move 18->36
    is think of the non-existing drives 19-36 as virtual(all zero) drives, and the
    scale-up just makes them available(the blocks on those drives turn from not in
    parity to free). The scale up is also done without moving any user data, and no
    changes in XDP other than transforming these new drives virtual->physical.
    For non-compress pages we use P=41 with 40 rows.

    Now
    we see drive configurations of 18, 24, 30 and 36 drives, none of which meet the
    “prime directive” for RDP/XDP, even if we took off some drives as spares.

    [Itzik Reich]: XDP with 18 drives, is the same as XDP with 36
    drives, only drives 19-36 are non-existent, and we don’t need to access them to
    calculate parities(they are treated as all zero drives). When we scale up from
    18 drives to, say 24 drives, the only thing that we do is move corresponding
    portion of parities to the new drives, and mark in XDP MD those drives offsets
    as available for write(this is part of the XDP magic, you don’t need to write
    anything to drives to make a page available, no user data is moved during the
    scale up). As load will continue the drives will get load, until they are
    balanced with the existing drives, and from now on all writes will be evenly
    spread cross drives, ensuring no misbalancesingle component bottleneck exists.
    Same happens as we continue to scale up to 36 drives.

    So
    what does this mean for XDP in the new X2? Have Dell EMC redesigned XDP
    to work differently?

    [Itzik Reich]: No XDP was not redesigned, we modified the Prime
    to fit the new number of drives in XDP, that is now 36.

    I can see a scenario where the new XtremIO stripe size is smaller than the number
    of drives present, in the initial instance, the lowest prime less than 18 and
    accommodating 2 parity drives would be 13, making the new protection system
    13+2. Rather than having a fixed parity stripe across all drives, is
    conceivable that XDP v2 could be writing 13+2 stripes across 18 drives.
    Bearing in mind that XDP already had an 11+2 implementation (starter X-Brick)
    then this could now be the stripe size for all systems.

    [Itzik Reich]: starter brick in X1 had the same XDP and the same
    prime as fully populated X1 system(Prime was 29). The XDP layout when starting
    with 18 drives and scaling to 36 is identical to XDP having 36 drives to begin
    with. The prime of a stripe does not change during the scale-up.

    What happens at expansion time? When a further 6 drives are added, is the RAID
    strip size expanded or left the same?

    [Itzik Reich]: it was 36 drives to begin with, so the scale up
    only made those drives available for write(only MD operations and some parities
    columns movement)

    Expanding
    the stripe size wouldn’t make much sense as the system would then be unbalanced
    from a performance perspective and at risk of space fragmentation. So,
    presumably the size is kept consistent irrespective of the number of drives in
    the system.

    [Itzik Reich]:XDP was designed to give
    superior performance even on full clusters, without having any garbage
    collection.

    The Architect’s View

    It’s good to see customers now have expansion options for fully exploiting their
    XtremIO X2 investment. It may seem a bit picky to be questioning the way
    in which data protection is now implemented on XtremIO X2. However
    consider that when the original platform was launched, the XDP savings were
    heavily promoted. 2/25 drives gives the 8% overhead; with 11+2 this
    results in a 15.3% overhead – a significant difference.

    [Itzik Reich]: On fully populated XDP the Parities overhead will
    actually be improved, to 2/36 that is 5.5%, and starting with XDP overhead of
    2/18 on half populated XDP that are 11%

    As a customer, I’d like to know what exactly is going on under the covers, both
    from a performance and capacity perspective. Previously EMC was keen to
    share the benefits of XtremIO. So, why the lack of detail now?

    [Itzik Reich]: we actually further improved the efficiency as explained earlier