All posts by Heisenberg

Storage Performance fundamentals

1. Throughput

Throughput is simply, the amount of material or items passing through a system or process. Another term for Throughput is Bandwidth. You can think of it as how fast can I transfer data in and out of my storage array.

Typically Measured as:
Data volume per second (sometimes per hour for larger volumes)

2. IOPs

Input and/or Output operations per second is the number of I/O Operations currently occurring. Some systems have fixed I/O sizes which is easier to calculate from, others will report at any given time what the I/O size is anyway.

Typically Measured as :
I/O per second = Input and/or Output / second

IOPs in a storage array is typically measured as front end IOPs to or from the host

Back end IOPs is to or from the storage media.

3. Latency ( response time )

Typically Measured as :
milliseconds or microseconds to respond to a host I/O

Storage concepts

Storage Architecture Concepts.

RAID Architectures

RAID – Redundant Array of Independent Disks

Typically drives of varying sizes can be used in a RAID configuration, however they will also size to the lowest drive format / geometry.

If you want to understand RAID concepts in more details, you can check the WIKI and various other online places rather than have me repeat it all here. Modern All Flash storage arrays that are less concerned with performance will primarily go with a dual or triple parity RAID for HA / Resilience

RAID 0

Good for performance , but no fault tolerance – striping

RAID 1

Good for read performance – mirroring

RAID 2

Forget RAID 2 – I have never in 20+ years used this

RAID 3

Forget RAID 3 – I have never in 20+ years used this

RAID 4

Forget RAID 4 – I have never in 20+ years used this

RAID 5

Good for performance. Block level striping with a single parity bit distributed over all the drives in the group. A minimum of 3 drives is required, 2 data and 1 parity but typically around 5-7 drives would be used as it allows for 1 failed drive but you don’t want to build your stripe too wide. Typically used with performance based 10K HDDs.

RAID 6

Good for resilience, similar to RAID 5, but has an additional parity bit (2). Write performance is traded off for redundancy as for every write you to also write 2 data bits so read performance could be 3x better than write performance. Often used for very wide stripes such as 18+2 or even wider when you are trying to maintain performance with data resilience.

Non RAID Architectures

JBOD

Just a Bunch Of Disks.

SPAN / BIG

Similar to RAID, but just a concatenation of drives to be used in a single volume and drives can be varying sizes, which is not good from a performance stand point.

MAID

Massive Array of Idle Drives, typically used for Write Once, Read Occasionally. Not a performance tier.

ERASURE CODING

Erasure Coding and RAID can sometimes be confused. You can use Erasure Coding in a similar way as RAID, however data is distributed and encoded (and can also contain parity data). Erasure Coding will use less capacity than RAID.

Erasure Coding is also used for things like distributed applications and Object storage providing redundancy at scale.

Data Architecture concepts

There are two main Data Architecture Concepts that I generally refer to.

DATA CENTRIC

Data centric refers to an architecture where data is the primary and permanent asset, and applications come and go. In the data centric architecture, the data model precedes the implementation of any given application and will be around and valid long after it is gone

DATA DRIVEN

Datadriven. When a company employs a “datadriven” approach, it means it makes strategic decisions based on data analysis and interpretation. A datadriven approach enables companies to examine and organise their data with the goal of better serving their customers and consumers

Move to singapore



The photo is my own. taken at East Coast Park….

It is 2019 now and I have not published anything for over a year.

Granted I don’t think I have any followers but I do hope to start publish more this year.

I have been busy uprooting my family from NZ and migrating to Singapore.

Singapore is a great central hub, I have technical responsibility for Pure Storage FlashBlade high performance file and object sales in ASEAN & Greater China.

I have spent the past almost 18 months getting my head around things and also trying to make a living 🙂

So far things have worked out well, I have put solutions in for solutions for Oracle Data Warehouses, Big Data Analytics Platforms replacing legacy data lakes and AI/ML solutions integrating NVIDIA DGX-1 AI appliances.

In Pure Storage context, these AI solutions are called an AIRI, AI Ready Infrastructure .

Business Lessons From a Boat Race

Earlier this week Emirates Team New Zealand (ETNZ) took the Americas Cup from Oracle Team USA (OTUSA) and it was a great day to be a Kiwi.

There are some key lessons to be learnt from this on why ETNZ won. Although to some this was just a boat race, it can also be viewed as a modern day version of David and Goliath. ETNZ as David and OTUSA as the well funded, well provisioned giant Goliath. With two boats that for the layman look the same but are not.

ETNZ knew two things coming into this race, you can’t rely on winning if you do things the same way as everyone else or as they have always been done and you need a world class team of support and specialists around you to ensure everything goes off without a hitch.

In traditional yachting you rely on ropes and pulleys and hands on man-power to trim the sails and guide the boat. ETNZ used technology and innovation to do things smarter.

They asked themselves how could we do better? They took what looked like a regular sail boat and reinvented it. They hired a world class design team from non traditional sailing backgrounds, a surfer from the U.S. that could build a drone to show them every aspect of the boat from every possible angle in a way that had never been done before.

ETNZ changed the “engine” of the boat. In the latest Americas Cup boats they rely heavily on hydraulic power, traditionally generated by human grinders that peddle with their arms to build hydraulic pressure. ETNZ looked at this process and decided it was inefficient so they swapped traditional grinders for Olympic cyclist. Cyclists obviously use leg muscles, a much bigger muscle group to do the same job. The Cyclists could also stay lower in the boat, creating less windage. The ETNZ cyclists were specialists in their on board jobs and did not have to worry about doing other tasks.

This innovation allowed ETNZ to build hydraulic power much faster and more efficiently than normal, requiring much less energy expended.

The net result was that the ETNZ boat was more efficient because of the technology and innovation that went into it and using the most appropriate tool for the job.

OTUSA are a very smart team, they knew that the Kiwis had a technical advantage, so they tried to imitate them. They tried to copy the technology but in the end their boat was not built to take advantage of the innovation and their retrofit was a poor imitation and in the end, they took it off.

The last point I want to make is that if you capsize as ETNZ did and if your sailing hard you probably will, you want to make damn sure that you have a the team behind you that is there to support you is world class and that they can get you back on the course quickly and efficiently so that you can continue your race.

What lessons can be learned from this?

For the layman, many things that are similar look the same. The devil is in the detail, was this device built for the technology it is using or was the technology added as an afterthought because someone else has done it.

Something that is purpose built for a task is much better than something that is bolted on or retrofitted later. Pure Storage is the founder of the modern All Flash Array. The FlashArray was purpose built to take advantage of flash in the modern enterprise.

Technology is evolving to make things more efficient and give you a tactical and operational advantage, use it. Look to companies that are innovating, that want to do things better to give you a business advantage.

Build solutions around world class teams that can challenge the norm and be innovative. Pure Storage has a SatMetrix certified NPS of 83.5, that’s huge, that’s world class and it has been independently verified. Customers that use Pure, love Pure and are happy to tell you and anyone that will listen about it.

My original article
https://www.linkedin.com/pulse/business-lessons-from-boat-race-alex-johnson

April 2016 VMUG

Pure Storage sponsored Auckland VMUG April 6 2016

We had another fantastic record turnout for the Auckland VMUG last week. With ~60 people turning up to hear Craig Waters present on the Pure Storage integration points with the VMware vRealize suite.

Craig did 4 demos on the day including integration with vRealize Log Insight, Automation , Orchestration and one other that escapes me. Check out the blog by Cody Hosterman as well with some great insights into this.

Some of our friends from IBM and Nutanix decided to drop in and learn a few things, but overall the attendance was mostly VMware admins looking to learn something.

We were fortunate that a bunch of the Nutanix crew are VCDX so we had an impromptu VCDX panel that also included Nick Bowie from ViFX for the punters in the crowd with the hard questions. I think most people were a bit scared to put their hands up to be honest.

Lastly Alan Baird from VMWare got up and presented on Tractors, or Containers, not sure which – but most likely containers 🙂

AKLVMUG
Auckland VMUG

IMG_5509

All told it was a great turnout, and a great evening.

The true NDU

I remember the first time I saw colour TV. I came home from school and sat down to watch my favourite after school show. 10 minutes in and I was asking my parents what was wrong with the TV because it kept changing channel. My father laughed at me.

On closer inspection, I noticed that not only was the massive intricately carved TV box that we bought back from Singapore gone, it was replaced with a smaller simple brown box, that was displaying colour and had my father had a wireless remote in his hand! Wired remotes were still common then.

My world had changed forever. Technology had evolved to enhance my user experience and I loved it.

Over the course of the next 30 or so years I was lucky enough to have experienced several such moments. Enhancements in technology have allowed us to transform the user experience in many areas. How amazing was it when you didn’t have to tie up the telephone line any more to be online!

10 years ago when we wanted to lifecycle storage, we we doing so with storage virtualization and that was awesome, but you run into the problem of how you lifecycle the lifecycle appliance? Then came storage migration at the hypervisor layer which was great for virtualised systems, but not so good for physical.

Last week I had a wow that’s so cool moment, when during the business day we non-disruptively upgraded a customer from our legacy hardware into our new hardware.

You might be saying to yourself – so what, I do that all the time! or everyone can do that, its easy!

I beg to differ. In recent times this has become easier for folks running virtual workloads and for me as a storage guy VMware changed the game by allowing people to non-disruptively move data stores between storage devices. If you had a modern array, the array would also be considerate enough to do most of the heavy lifting for you as VMware created the VAAI APIs that allowed the storage to offload a bunch of the t10 SCSI commands, thus freeing up compute CPU cycles. Old news I know, but there are still legacy storage arrays that don’t support these things.

What I want to be clear on though was that I didn’t have to use any software tools to do this!

The architecture of the Pure Storage array is what allows me to do it. More importantly it allows me to do it without impacting host IO.

The company I was doing the upgrade for is a very sensitive company, with sensitive data and conservative customers. We started the upgrade at 10:30 am and it was complete in a few short hours.

Not only did we swap the storage controllers, but we upgraded the capacity at the same time.

The measure of success was no help desk calls and we were successful.
To be clear, this was done during business hours with a full complement of staff working and apps running.

I love my job. I work for a disruptive company changing the industry of technical storage.

Maximum HBA port queue depth in an all Flash Array

Per port queue depths is a common question we hear and how we compare to these limits other vendors publish based on array model. 

Pure does not have queue depths assigned on a per port basis and we do not modify the Qlogic HBA per port queue depth of 4096.  We also do not issue SCSI “busy” responses back to hosts (instead, we will queue all I/O in the array) there is occasionally some host queue depth adjustments needed for some environments.  The goal is to not unnecessarily limit outstanding I/O’s to the array during normal steady state, but also to avoid a  situation in cases of massive bursts of I/O from many hosts concurrently.

With legacy mechanical disk arrays, it was important to have as many outstanding I/O’s as possible due to the seeking and rotating going on.  

With all flash arrays, and Pure in particular, it is not necessary to keep the queues full and the I/O responses are returned much faster as well, meaning it’s pretty hard to keep the queues full (at least for most organic workloads).  One of the key values for Pure is the simplicity – you should not have to do the same types of computations and planning like you are used to with mechanical disk storage.  In some cases, there may be some adjustments in aggregate for all hosts connected to the same array to mitigate a IO storm (ensuring host-level fairness), in most cases it is not necessary as we remove the need to do per-port planning.

Auckland vMUG July 2015

We had a great time last night (30-07-15) at the @PureStorage sponsored  @AucklandVMUG event. Approx 65 attendees registered and we would have had at leas that number in the room.

The photo below

BlueStone Room
BlueStone Room

Lots of great guests;

  • Nick Bowie kicked us off even with getting over a cold  (@NickBowieNZ). VCDX5-DCV #202, VCP6-NV, NPP #43.
  • Craig Waters took us through Flash in the Data Centre                      (@cswaters1).  VCAP-DCA/DCD, VCP-DV 3-5, vExpert 2012-14
  • Alastair Cooke spoke initially about training then gave us a good talk about your career and how to manage it yourself  (@DemitasseNZ) vExpert

Nick Bowie
Nick Bowie

Lots of great attendance from the members of the Auckland chapter of vMUG, plus a couple of sneakie entries from other vendors wanting to check us out.

We had a major prize of a 250GB 3D NAND SSD that was won by Diego Nievas from Datacom after a draw by Stuart Blythe, Country Manager of Pure Storage in NZ

Diego Nievas wins SSD
Diego Nievas wins SSD

Kudos to Sanit, Michael, Steven and Ping for putting on a great event.