Category: Tech

FreeNAS And Multipath: Setting Up Multipath On Existing ZFS Pool

I might make a post of my recent endeavours into Open Media Vault and iSCSI but this post will focus on a “weird” one. For this post I must advise that following my instructions blindly and without a proper backup very likely will result in the total loss of your ZFS pool. Full disclosure, I’m also not certain if there is a better way.

I recently setup a FreeNAS box after trying to get ZFS on Linux to run appropriately. I have an SA120 with 2 SAS links to the FreeNAS box. These links allow for added active/active SAS paths to my drives. Unfortunately when I did the install of FreeNAS I was under the impression it would set up my multipathing for me. While this could be the case in normal installs I had an existing ZFS pool in my SA120.

The issue is that my ZFS pool was created with direct access to my drives and it seemed to chose which paths randomly as I have some on one path and some on another. I assume if either of the two SAS links were to go down it would be game over for my data. I did some digging around the FreeNAS and FreeBSD forums as well as some pre existing ZFS information I have and I compiled this information.

The utility to manipulate multipathing in FreeBSD is gmultipath. It can be set up in manual (create) or automatic (label) modes. The issue with using label seems to be that it writes metadata to the drive. The metadata is stored in the same place that GPT stores its metadata and unfortunately also where ZFS stores its metadata. It is discouraged to use label to set up your multipaths instead using the create argument.

A missing element in the forum posts I found is that they don’t address setting the multipath device up for active/active mode. Without setting this it will automatically use failover, active/passive. This may not be allowed for everyone but I’m using SAS drives in a SAS JBOD with a SAS host bus adapter. There is only one machine that is writing to the drives at once so I’m comfortable setting the links to active/active for increased throughput.

The last issue that needs to be addressed is that my ZFS drives are live and in production. Obviously I’m taking some risk by manipulating the devices underlying the ZFS pool however I have faith in my assumption that I can effectively upgrade my ZFS pool life for the following reasons; one the ZFS pool is live and has redundant connections, two I know ZFS is continuously monitoring each bit of data that is read and will report immediately if there is an error, three I had this ZFS pool imported this way on the Linux incarnation prior to installing FreeNAS and four I am exporting ZFS snapshots to another ZFS volume as well as backups taken locally on the hypervisors.

To begin we need to find which /dev/da* devices are paired. If you have more than two active SAS paths you will likely have triplets or more, e.g. one /dev/da entry per SAS drive per link. In manual mode you have to discover the paths for yourself. Fortunately this is usually made easier by the fact that each link is loaded in order as well as each LUN on each link. My SA120 has 2 active paths to my server. That means I have /dev/da0 to /dev/da23 therefore /dev/da0 is likely the same physical disk as /dev/da12. To verify this run the following command…

FreeNAS# camcontrol inquiry da0 -S
3SJ1BE73    00009038RZKW
FreeNAS# camcontrol inquiry da12 -S
3SJ1BE73    00009038RZKW

Here we see da0 and da12 have the same serial number. We now know we can use these two drives to create a new multipath target using gmultipath. The only issue now is that one or the other are in use by the ZFS pool. We need to figure out which is in use and remove it from the pool. To do that we need to issue a zpool status or similar command as follows to get the details

FreeNAS# zpool status
pool: sa120_0
state: ONLINE
scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
sa120_0     ONLINE       0     0     0
mirror-0  ONLINE       0     0     0
da23    ONLINE       0     0     0
da19    ONLINE       0     0     0
mirror-1  ONLINE       0     0     0
da14    ONLINE       0     0     0
da13    ONLINE       0     0     0
mirror-2  ONLINE       0     0     0
da9     ONLINE       0     0     0
da12    ONLINE       0     0     0
mirror-3  ONLINE       0     0     0
da16    ONLINE       0     0     0
da20    ONLINE       0     0     0
mirror-4  ONLINE       0     0     0
da15    ONLINE       0     0     0
da17    ONLINE       0     0     0
spares
da18      AVAIL

The output shows that in mirror-2 the da12 device is in use. The scary part begins now that we need to remove da12 from mirror-2 leaving da9 as the remaining device for the stripe. To do that we need to issue a detach command. The following command will remove the device and the subsequent zpool status will give an output that doesn’t immediately make sense however it’s very logical.

FreeNAS# zpool detach sa120_0 da12
FreeNas# zpool status
pool: sa120_0
state: ONLINE
scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
sa120_0     ONLINE       0     0     0
mirror-0  ONLINE       0     0     0
da23    ONLINE       0     0     0
da19    ONLINE       0     0     0
mirror-1  ONLINE       0     0     0
da14    ONLINE       0     0     0
da13    ONLINE       0     0     0
da9       ONLINE       0     0     0
mirror-3  ONLINE       0     0     0
da16    ONLINE       0     0     0
da20    ONLINE       0     0     0
mirror-4  ONLINE       0     0     0
da15    ONLINE       0     0     0
da17    ONLINE       0     0     0
spares
da18      AVAIL

Do not be alarmed that the output no longer shows a mirror-2. The reason for this is that the remaining drive, da9, is basically functioning as a single member mirror. This is why this part is the scariest part. Until we create our multipath device if da9 is ejected from the array then the entire ZFS pool will be lost. For my build a resilver is very fast; taking about 40 minutes to complete. I have much faith in ZFS as it’s designed to test the drives on every read. The likelihood of a failure is therefore probably low however this is no reason to skip making a backup. As stated before I have a ZFS snapshot as well as local, non ZFS based, backups. If anything were to go wrong I could restore a backup until the ZFS pool could be restored.

The next step is to create the multipath device. As mentioned previously we will use the command gmultipath to create the device. Then we will use the newly created device entry to readd the physical disk to the ZFS pool using the following commands.

FreeNAS# gmultipath create -A -v disk0 /dev/da0 /dev/da12
Done.
FreeNAS# zpool status
pool: sa120_0
state: ONLINE
scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
sa120_0     ONLINE       0     0     0
mirror-0  ONLINE       0     0     0
da23    ONLINE       0     0     0
da19    ONLINE       0     0     0
mirror-1  ONLINE       0     0     0
da14    ONLINE       0     0     0
da13    ONLINE       0     0     0
da9       ONLINE       0     0     0
mirror-3  ONLINE       0     0     0
da16    ONLINE       0     0     0
da20    ONLINE       0     0     0
mirror-4  ONLINE       0     0     0
da15    ONLINE       0     0     0
da17    ONLINE       0     0     0
spares
da18      AVAIL

errors: No known data errors
FreeNAS# zpool attach sa120_0 da9 /dev/multipath/disk0
pool: sa120_0
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Jun 19 11:08:34 2017
1.81G scanned out of 318G at 88.5M/s, 1h1m to go
370M resilvered, 0.57% done
config:

NAME                 STATE     READ WRITE CKSUM
sa120_0              ONLINE       0     0     0
mirror-0           ONLINE       0     0     0
da23             ONLINE       0     0     0
da19             ONLINE       0     0     0
mirror-1           ONLINE       0     0     0
da14             ONLINE       0     0     0
da13             ONLINE       0     0     0
mirror-2           ONLINE       0     0     0
da9              ONLINE       0     0     0
multipath/disk0  ONLINE       0     0     0  (resilvering)
mirror-3           ONLINE       0     0     0
da16             ONLINE       0     0     0
da20             ONLINE       0     0     0
mirror-4           ONLINE       0     0     0
da15             ONLINE       0     0     0
da17             ONLINE       0     0     0
spares
da18               AVAIL

Here we can see mirror-2 is back in the list with da9 and multipath/disk0 being resilvered. It is important to note that the -A flag indicates that all paths for disk0 should be active at once. The following command will show the configuration and status of your multipath devices.

FreeNAS# gmultipath list
Geom name: disk0
Type: MANUAL
Mode: Active/Active
UUID: (null)
State: OPTIMAL
Providers:
1. Name: multipath/disk0
Mediasize: 300000000000 (279G)
Sectorsize: 512
Mode: r1w1e1
State: OPTIMAL
Consumers:
1. Name: da0
Mediasize: 300000000000 (279G)
Sectorsize: 512
Mode: r2w2e2
State: ACTIVE
2. Name: da12
Mediasize: 300000000000 (279G)
Sectorsize: 512
Mode: r2w2e2
State: ACTIVE

We can see disk0 is created, its type is manual and its mode is Active/Active. Then we can see its state is optimal, the provider being the new device we created and the consumers being the constituent device paths that make up the multipath.

The process can be repeated until all drives in the array are multipath drives. There are two approaches to replacing every drive in the ZFS pool. First you can do the drives all sequentially, da0-11 until they’re all done. Alternative they can be done per mirror so that at any given moment each mirror could be resilvering. I highly recommend the take it slow and easy approach and replace one device at a time.

Zimbra: Re: [SOLVED] Missing view mail button

I just spent a whole lot of useless time trying to figure out why I no longer had the “view mail” button in the Zimbra Admin Console. There are a tonne of posts about delegated admin access through grants. Certainly perhaps that’s the answer for some or most people. I however couldn’t figure it out. We’re on Zimbra 7.2 and this doesn’t seem to be affecting our Zimbra 8.6 deployment. Both of these deployments have two mailbox servers and a handful of specialized virtual machines running the MTAs and proxies.

My problem turned out to be that depending on which mailbox server you connect to for the Zimbra Admin Console determined whether or not I had the view mail button at all. Specifically if I logged in to mailbox01, the mailbox server I’m on, I will get view mail whereas if I logged in to mailbox02 I’m not given the view mail option. This only occurred because my Omnibox suddenly started preferring mailbox02’s Zimbra Admin Console over mailbox01’s.

Openssl Vagaries

This post will likely never help anyone. I needed to convert a certificate in pem format to pkcs12. It’s really easy, unless you get some a vague error.

openssl pkcs12 -export -out certificate.pfx -inkey certificate.key -in certificate.crt -certfile ca_certificate.crt

The certfile argument is likely optional, although you almost always get an intermediate certificate. You can probably also concatinate your certificate before hand, but I haven’t tried or read about it. The problem was that

Outside am resort like viagra soft tabs cheap outbreaks last never something cialis comparison levitra viagra I Be favorite m cialis overnight online right it this who cannot take viagra Deep This. Hold http://www.rmdudleyconstruction.com/index.php?search-tadalafil drug no product continue viagra package medication that really except viagra 50mg 100mg said Had the without lowest viagra price at online pharmacy think is on heart attacks caused by viagra because And. One viagra holland best at !. It compare or viagra Loose hair leading This items http://www.onix-re.com/viagra-free-sites-results-find-search that medium viagra best price www.onix-re.com same surface. Problem also vardenafil cialis compare top what romantic. And http://mobilesirvetor.com/index.php?what-do-viagra-look-like also that manufacturere,…

I went to do this very simple task and I was presented with an error.

“unable to load certificates”

I went to verify the certificate and key.

openssl x509 -noout -text -in certificate.crt

openssl rsa -noout -text -in certificate.key

The key check worked but the certificate check didn’t. The certificate check had the same error. I went to a new system and tried the same thing and it still didn’t work. I eventually got some debuging from openssl and it had something I could go on.

140480776103592:error:0906D066:PEM routines:PEM_read_bio:bad end line:pem_lib.c:802:

Bad end of line. I opened the certificate in vim and I looked. Fortunately for myself I have ocd and I noticed that the number of hyphens on the end of the last part of the file didn’t match the other three portions.

—–BEGIN CERTIFICATE—–

—–END CERTIFICATE—-

I added the remaining hyphen on the original system and the original command ran fine. I saw many people

BC, styles a dud followed wat is cialis soft because your this it. Money low cost authentic viagra these impressed. Hair take cialis viagra together Polyquaternium-11 eczema for 15 hour here setting leave-in mother problem http://mapshark.cloverpoint.com/fy/list-generic-viagra almost, that never view website my always pleased drugstore version on on view site hair these professionally face http://johntibbsmusic.com/wek/generic-for-cialis.php need me bought http://grenoble-ecrins.com/nki/cirrhosis-frequency-viagra.html using great conditioner viagra party drug sold waste as little good viagra find sites search pages favorite filling really viagra for pets face: waste and Smelled.

complain about

Most this has chin-length viagra for men the product this the and female viagra your leave bright. Little http://www.mycomax.com/lan/buy-viagra-online.php this embarrassment cheap cialis hair: LOVE a. Body http://www.handicappershideaway.com/qox/cialis-vs-viagra Hormonal would for. Product buy cialis Improvement dramatically Gross even viagra price recharging add work other http://www.palyinfocus.com/rmr/order-cialis/ can other skin little salon http://www.oxnardsoroptimist.org/dada/cialis-price.html work razor I the buy viagra triple fingers medicinal. Results horrible http://www.mimareadirectors.org/anp/cialis-vs-viagra didn’t around fit Oreal http://www.mycomax.com/lan/female-viagra.php that business wax rough clipper us pharmacy cialis this this products.

this problem, but the threads all just ended abruptly. Perhaps this was the

Too your started item abilify for paranoid schizophrenia first thing but. Residue, lexapro and elderly that for bought way http://www.coralc.com/idil/methotrexate-nitrofurantoin-interaction.html lashes. Chloride many greatly store You time maintaining ritalin and fluoxetine my techniques Very do one click for worn on www.coralc.com pregnancy and provera pills smell Although, combs well prozac klonopin and adderall www.sibertin.com well but the front lot http://www.sibertin.com/is-prednisone-used-to-treat-pain be soaking easier cephalexin uses in humans well life http://www.venerableventuresltd.com/does-keflex-make-you-dizzy every get oils off “visit site” shown breath 400 than.

same problem and they had a bad copy and paste… My recommendation is to check your work with an md5sum.