3Par StoreServ 7000 – Step by step Installation and Configuration (Part 1)

<UPDATE – 2-19-2013> – HP’s 3Par team was gracious enough to reach out and speak to me regarding some of the hiccups I experienced.  First, on the Virtual Storage processor they state that the file system error could be caused by having the wrong time on the ESX host which I can confirm is accurate.  Since we were also setting up new Blade Servers the right time had not yet been set.  The other issues are currently being worked on by the 3Par team and they will be fixed by the time most see their system.  I can’t say enough about the guys I spoke with, they are truly passionate about the product and are the utmost professionals who will not rest until they have a perfect product.  I look forward to working with them on future enhancements.

Last month HP announced the long awaited “baby” 3Par into their storage portfolio, the StoreServ 7000 series.  As predicted, 3Par’s entry into the SMB storage market has created a lot of excitement for the IT departments that wanted 3Par technology but didn’t have the budget for the existing models.  Since the majority of my clients are in the alternative investment space (Hedge Funds, etc.) they typically ask for the latest and greatest technology to stay on the cutting edge.  Naturally, being one of the first to implement a new technology product will most likely net you some of the bugs that present themselves in the first versions of products.  Since the Storeserv 7000 series uses that same underlying technology and operating system as the other mature 3Par products, I had no reservations recommending it to my client.  One nice thing about this 3Par is the ability for the customer to perform their own installation.  Previously, only authorized 3Par technicians were required for the initial install, setup, and configuring.

However, as you will see later in this article I did run into an issue with the initial setup in the new Smartstart Setup Wizards due to a bad setup script from HP, so I had to perform the “birth process” from the command line via serial port.  I actually prefer doing these type of procedures from the command line, since you learn a lot more about the technology that is usually hidden under the covers.  The rest of this post will be dedicated to walking you through the entire setup process from installing the drives, zoning the fiber to the blade servers, and creating a virtual volume to use in VMware.  Note that most of the installations steps can also be applied to the bigger P10000 systems as well since all 3Par’s share the common operating environment.

Ok, let’s get started – my client purchased a StoreServ 7200 with no extra disk shelves.  A simple 2U unit with 7TB of space was more than enough for them for now.  We removed everything from the boxes and were blown away with how awesome this array looks in person. The following are some pictures form when we first put the array together.

HP/3Par Licensing documents and software

3Par 7200 freshly out of the box

Front right

StoreServ 7200 SFF 2.5inch 450GB 10k hard drives
ready to be installed

Rear shot of the array

3Par controllers

Controllers removed from array

Controller backplane.

7200 with 16 hard drives installed

Rear shot of the 7200 – Two 8GB fiber ports on each
controller.

Close up of the power supply units

As you can see form the pictures the StoreServ 7200 comes out of the box pretty much ready to go in terms of physical assembly with the hard drives being only pieces that need to be installed.  We opted to go for the 2.5 inch small form factor SAS hard drives to maintain the 2U form factor.  Included in the box were SAS cables for future disk shelf expansion, power cables, labels, software, and licensing.  After taking a good look at the array, the design seems to be well thought out by HP Engineers – there is even a nifty rod that doubles as a pull handle\locking mechanism for the controller nodes.  It’s amazing HP was able to pack this amount of power and features into a 2U box.  On the rear of the chassis we have the two standard 110 volt power supplies that back each other up in case one experiences a heart attack.  In the middle you have your two controller nodes that house the special 3Par ASICs, processing power via a 1.8GHz chip, and your cache repository.  The base configuration we ordered came with the following I/O ports on each controller: Two 8Gb fiber ports, one RJ45 for remote copy, another RJ45 for Management, and finally three manufacturer diagnostic ports.

At this point this first thing I wanted to do is make sure that we registered the product and and make sure that we obtained all pertinent license keys.  As seen in the first picture the blue envelopes contain a printed paper from HP with an Entitlement Order Number (EON) number.  You will notice that some of the EON’s can be the same for different licenses which is normal due to the bundling.  When you have written down all of your EON’s you will need to register them on HP’s licensing portal.  The site will instantly send you all of the associated license keys.  The keys will be entered into the 3Par toward the end of the “birth process”.

Next, we have to initiate the “birth process” of the 7000.  First, we are going to setup and configure the 7000’s Service processor  HP now supports the Service Processor in a virtualized environment and makes it super easy by providing a pre-made OVF package.  The OVF resides on one of the DVD’s that come in the box.  Fire up your vSphere client and connect to one of your ESXi hosts.  Once you are connected navigate to File > Deploy OVF template.  On the source page, click browse to import the OVF from its location on the DVD, click next.  On the OVF Template Details page, verify the OVF template, then click next.  On the name and location page, enter a name for the virtual storage processor then click next.  On the disk format page, select Thin Provision, then click next.  Now click finish and check the box on the bottom to power on the VM once the deployment is complete.

Now, here is where I ran into the first of my two major issues with the array setup, which I attributed to poor HP quality control.   So, when I booted up the vSP the Linux OS ran through the standard file system check during boot it stopped and informed me that the file system wasn’t consistent and needed to repaired – oh great!!! <SEE TOP OF POST FOR UPDATE – THIS ISSUE WAS DUE TO MY ESX HOST’S TIME BEING WRONG>

Since I was probably one of the first HP resellers in the world to set up a 7000 I realized that finding someone at HP to actually acknowledge this issue and help me would be a daunting task.  At the end, it actually worked out well.  After about two hours of navigating through different storage support groups I finally found a very knowledgeable and helpful engineer in Colorado who saw this issue in HP’s internal training classes.  Even though the error gives you some guidance on the switches that you should use with the fsck command he advised to use some different ones.  He had me enter in the following command (without the quotes) to repair the file system:  “fsck –yvf /dev/sda*” I hit enter and let it run, after it ran through I typed in reboot as depicted below.

Now that the Service Processor’s file system is repaired it should now be in an operational state.  We need to connect to the SP via a web browser to start the birth process.  The SP received an IP address via DHCP which you can discover by going to the “summary” tab for the VM under the vSphere client.  Now, in your browser type the web address as follows: https://IP/sp/SpSetupWizard.html  This should get you to the Service processor setup wizard as seen in the below picture.

Click next on on the welcome screen.

I got stuck on the next screen since the documentation provided didn’t tell me how to obtain my Service processor ID.  HP ended up forwarding me a document which explained how they changed the process and that the 3Par serial number should be entered.  The article ended up getting posted on the HP site here.  As it states the S/N should be prefixed with SP000.  So, if you serial number is 1601234 the Service Processor ID will be SP0001601234.

Next, fill out the hostname and IP address information, along with the DNS support info, and advanced settings if applicable.  Hit next when complete.

The next screen pertains to the “phone home” feature of the SP.  This will allow the array to send HP information on the status of the array.  This is a nice feature since Hp will know if the array had a bad drive and automatically get the ball rolling to let you know about it and get you a replacement.  If you don’t have a proxy server for internet access you can leave these settings blank.

Simple stuff here – I recommend using an NTP server.

Set a new password for user names “setupusr” and “3Parcust”

Make sure all information is correct and click next.

Now, the “birth process” will start.

Once the Service Processor has been setup and configured you will return to the main menu.  The next step is to run the “Storage System Setup Wizard” to initialize the actual array which is part of the process that carves out all the chunklets.

Here is where I ran into some serious issues.  No matter how many times I tired and enter in the serial number the process would always kick back with the same error shown in red.  I went through all of my wiring to make sure the SP was on the same subnet and able to communicate to the array.  As I previously stated I was finally able to make contact with a 3Par engineer that was notified of this problem.  He stated that their is a known problem with the wizard driven setup process and a file containing some metadata for proper detection of the array was accidently left out.  I’m starting to wonder if it was the same guy that compiled the OVF ;).

This is where things got interesting.  The only thing we could do at this point is manually setup the array from the command line via serial port.  I originally recorded the entire process but since I don’t posses enough patience to get into video editing I will continue to use screenshots.  At the end of the entire process I actually ended up enjoying seeing the entire process in the command prompt since I saw everything that was going on under the covers.  I imagine that by the time most people read this HP would have fixed all these issues and the wizard should work flawlessly.  Anyway, the following are the steps taken to manually bring the array alive via command prompt.

First, we will need to hook up the array to a laptop via a USB-to-Serial converter. Also, the system comes with two connectors that have a serial interface on one end and a RJ45 on the other end.  Make sure to utilize the gray connector (PN 180-0055-01) which you should connect a CAT 5 cable into, the other end should connect into the MFG port on the first controller.  Use a program like “Putty” to connect to the array, make sure to use 57600 as the baud rate, 8 bits, no parity, 1 stop bit, disable XON/XOFF.

Once you are connected you should should get a prompt to enter in a username and password.  I used “console” as the username and the password for your system can be obtained by contacting HP or you can reach out to me directly as it appears to be confidential info.

You will now be presented with the 3Par Console Menu.  To start the process I selected number 6 and hit enter.  The console will not ask you to enter in “yes” to confirm that you are OK with the system to shut down and reboot.

3Par booting up…  Looks like an Intel 6 core 1.8Ghz chip is in each controller in the 7200.

Now that everything is fresh and clean we are going to select number 1 to run the “Out of the Box Procedure” then type in “Yes” to continue with the procedure.

I can confirm that we indeed do have two nodes in our 7200 by hitting “c”

I typed in 1 to select New York.  I then hit enter to confirm that the time reported was correct.

Typed in a name for my 7000 and confirmed.

Confirmed all OS versions looked good.

Verified that the OS was seeing all 16 drives.  The OS then ran a check to make sure all drives and drive cages had the latest firmware.  This setup also made sure that all connections looked good.

The system then prompted me to run a stress test on the entire array by running a heave amount of I/O through all the disks. I deferred this for later due to a time crunch but recommend that you run this immediately to weed out any potential array issues that can be lurking. The test takes about two hours and 15 minutes.  You will notice than an error message popped up stating that my array has less cages than nodes.  Since I currently didn’t have any connected add-on drives cages (or shelves as some would call it) I am considered to only have one drive cage.  The system has two controller nodes so obviously we have more nodes than cages.  For protection and performance reasons you should always have multiple drive cages.  My client was small enough to not require this.  This error will eventually clear itself after a few prompts.

I put in an available static IP with corresponding netmask, gateway, and temporary DNS.  The DNS should point to a local DNS server, such as an AD integrated box.  I made sure to not disable any encrypted ports.  Once completed I hit “y” to confirm everything looked good.

The setup process is now running – as you can see the chunklets are now being created.

Now, the system wants us  to dictate on how we want the spare chunklets aggregated.  I selected “D” for default which puts us at 13%.

Looking at the shot above we can see that more license keys need to be applied since I have more than 8 disks and other features that were ordered.  We will add this later.

I selected “No” for the CIM API’s.

Storage Array setup is now complete.

I then ran option 10 which will go through all the components in the array to make sure all components are healthy and ready to go.

Now, we have to mate the Service processor up to the storage array.  We can start this process by browsing to the SP to access the Service Processor Onsite Customer Care (SPOCC) via a web browser.  Simply type in the IP address of the SP that we configured earlier using https://.

Once you are in SPOCC, on the left click SPMaint.

Click on “Add New INserv”.  Once there add the array’s IP we created in the command line setup.  Use account 3Paradm and the password you setup earlier.  We now have the array and SP mated up.

In part two of this article we will go through the remaining tasks to setup the array.  This will include configuring the zoning between the array and the HP blade servers.  For this task we will be configuring this on our Brocade fiber blade switches.  I will also take you through applying the license keys via the command line, setting up your hosts in the InForm management console and carving out a LUN to our VMware hosts.  This will require us to setup our CPG’s and VV’s which we will export to our hosts.

As I said earlier, I wasn’t concerned much on some of the hurdles I faced, since most of it adhered to components that are not mission critical to the underlying operation of the array.  The Smartstart wizards are a nice approach to make an advanced array approachable to a normal system admin.  I’m sure HP will fix these issues in the next week or so.

Thanks for reading, see you in Part 2.

-Justin Vashisht (3cVguy)