This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
cluster:194 [2020/07/28 17:35] hmeij07 [HDD] |
cluster:194 [2023/01/17 14:13] hmeij07 [certs] |
||
---|---|---|---|
Line 115: | Line 115: | ||
Virtual IP '' | Virtual IP '' | ||
- | An update goes like this and is not an interruption. Check for and apply updates. They are applied to partner | + | Critical |
+ | |||
+ | You can always Disable Failover, to fix power feed of switches 192.168.0.0/16 or 10.10.0.0/ | ||
+ | |||
+ | Check Box to Disable Failover | ||
+ | Go to WebUI > System > Failover > Click the Box > Then Click Save (leave default controller setting as is) | ||
+ | |||
+ | This will allow you to make your network change without failing over. | ||
==== SSH ==== | ==== SSH ==== | ||
Line 143: | Line 151: | ||
</ | </ | ||
+ | |||
+ | ==== certs ==== | ||
+ | |||
+ | * Go to System > CA and certs | ||
+ | * Add a cert | ||
+ | * name is " | ||
+ | * FQDN for hpstore.wesleyan.edu, | ||
+ | * fill in just the basics, no contraints or advance settings | ||
+ | * Add, then view CSR section, copy and provide to inCommon admin | ||
+ | |||
+ | * Once you get email back Available formats: | ||
+ | * as Certificate only, PEM encoded ... download, open in notepad, copy to clipboard | ||
+ | * Go to System > CA and certs | ||
+ | * Add a cert, type internal certificate (changes fields) | ||
+ | * paste in public key | ||
+ | * paste in private key from CSR | ||
+ | * save, this will restart web services | ||
+ | * check | ||
==== ZFS ==== | ==== ZFS ==== | ||
Line 156: | Line 182: | ||
zfs userspace | zfs userspace | ||
zfs groupspace tank/ | zfs groupspace tank/ | ||
+ | |||
+ | # uttlerly bizarre in v12 these commands change | ||
+ | |||
+ | root@hpcstore2[~]# | ||
+ | hpcstore2% | ||
+ | hpcstore2% zfs get userused@hmeij07 tank/ | ||
+ | NAME | ||
+ | tank/ | ||
+ | hpcstore2% zfs get userquota@hmeij07 tank/ | ||
+ | NAME | ||
+ | tank/ | ||
+ | hpcstore2% zfs get userspace@hmeij07 tank/ | ||
+ | bad property list: invalid property ' | ||
+ | |||
# hpc100 | # hpc100 | ||
Line 316: | Line 356: | ||
==== Snapshots ==== | ==== Snapshots ==== | ||
+ | |||
+ | Snapshots made easier in new releases ... traverse to the hidden directory ''/ | ||
+ | |||
+ | < | ||
+ | |||
+ | 129.133.52.245:/ | ||
+ | 251T | ||
+ | |||
+ | </ | ||
+ | |||
* Daily snapshots, one per day, kept for a year (for now) | * Daily snapshots, one per day, kept for a year (for now) | ||
Line 324: | Line 374: | ||
* check permissions on cloned volume, not windows! | * check permissions on cloned volume, not windows! | ||
* NOTE: once had mnt/ | * NOTE: once had mnt/ | ||
- | * when cloning grant access to 192.168.0.0/ | + | * when cloning grant access to <del>192.168.0.0/ |
* NFS mount, read only | * NFS mount, read only | ||
* maproot '' | * maproot '' | ||
- | * Clone mounted on say '' | + | * Clone mounted on say '' |
* Restore actions by user | * Restore actions by user | ||
* Delete clone when done | * Delete clone when done | ||
Line 350: | Line 400: | ||
</ | </ | ||
- | ==== Update ==== | + | ==== Update |
- | Change the Train to 11.3, then you will apply the update first in the WebUI to the passive controller. | + | See Update 12 for manual update to v12 with Anthony on 03.09.2021 |
+ | |||
+ | **Change the Train** to 11.3, then you will apply the update first in the WebUI to the passive controller. | ||
After its reboots, you will failover to it by **rebooting** the Active controller (the **current** WebUI). | After its reboots, you will failover to it by **rebooting** the Active controller (the **current** WebUI). | ||
Line 364: | Line 416: | ||
Enable HA, click icon | Enable HA, click icon | ||
- | Apply Pending Updates | + | **Apply Pending Updates** |
Upgrades both controllers. Files are downloaded to the Active Controller and then transferred to the Standby Controller. The upgrade process starts concurrently on both TrueNAS Controllers. | Upgrades both controllers. Files are downloaded to the Active Controller and then transferred to the Standby Controller. The upgrade process starts concurrently on both TrueNAS Controllers. | ||
- | Server responds while HA disabled. | + | Server responds while HA disabled. |
+ | |||
+ | Update takes 15 mins in total. | ||
+ | |||
+ | ** 11.3 U5 ** | ||
+ | |||
+ | * Check for Updates, read release Notes, schedule support ticket if needed (major update) | ||
+ | * Apply Pending update, save configuration, | ||
+ | * active/ | ||
+ | * At 100% stanby reboots, HA disables, file system ok | ||
+ | * Check version on standby, Initiate Fail Over (interrupts file system) | ||
+ | * Login, you end up on new active server, then | ||
+ | * Logout/ | ||
+ | * when old active server boots " | ||
+ | * check version on new standby | ||
+ | |||
==== HDD ==== | ==== HDD ==== | ||
Line 404: | Line 472: | ||
You can look that information up yourself by opening an SSH session to the passive controller, navigating to the / | You can look that information up yourself by opening an SSH session to the passive controller, navigating to the / | ||
+ | |||
+ | ==== Split Brain ==== | ||
+ | |||
+ | When ending up with an error fail over state try console shutdown first. If that does not work cut power to controllers. | ||
+ | |||
+ | ==== fndebug ==== | ||
+ | |||
+ | * first log into support, then download teamviewer | ||
+ | * https:// | ||
+ | * get.teamviewer.com/ | ||
+ | |||
+ | |||
+ | **Manual debug file creation**, then ftp to ftp.ixsystems.com | ||
+ | |||
+ | < | ||
+ | |||
+ | freenas-debug -A | ||
+ | tar czvf fndebug-wesleyan-20201123.tar.gz / | ||
+ | |||
+ | # next look at bottom of fndebug/ | ||
+ | /dev/da10 HGST: | ||
+ | /dev/da9 HGST: | ||
+ | # these drives have not failed yet but have write errors, offline/ | ||
+ | |||
+ | # next look at output of zpool status -x in fndebug/ | ||
+ | # and the error code | ||
+ | # https:// | ||
+ | |||
+ | NAME STATE READ WRITE CKSUM | ||
+ | tank DEGRADED | ||
+ | ... | ||
+ | raidz2-1 | ||
+ | gptid/ | ||
+ | # look for checksums that have failed like this disk in vdev raidz2-1 | ||
+ | |||
+ | # clean up the spare that resilvered (INUSE status) | ||
+ | # then run a clear on the pool. Then we'll try to get another debug. | ||
+ | |||
+ | zpool detach tank gptid/ | ||
+ | zpool clear tank | ||
+ | |||
+ | # that brought all drives back online and the vdevs show | ||
+ | # then via gui added the available drive back as spare | ||
+ | |||
+ | </ | ||
+ | |||
+ | * Monitor the progress of the resilvering operation: 'zpool status -x' | ||
+ | |||
+ | |||
+ | **Replace a failed drive** | ||
+ | |||
+ | * https:// | ||
+ | * drives mentioned above have not failed yet so we must " | ||
+ | |||
+ | < | ||
+ | |||
+ | 1) Go into the Storage > Pools page. Click the Gear icon next to the pool and press the " | ||
+ | 2) Find da4 and press the three-dot options button next to it, then press " | ||
+ | 3) Go to the System > View Enclosure page, select da4 and press " | ||
+ | 4) Physically swap the drive on the rack with its replacement. | ||
+ | 5) Go back to the Storage > Pool > Status page, bring up the options for the removed drive, | ||
+ | 5a) Select member disk from dropdown, and press " | ||
+ | The replacement drive may or may not have been given the name " | ||
+ | 6) Wait for the drive to finish resilvering before proceeding to replace da3. | ||
+ | 6a) Click spinning icon to view progress. Pool status " | ||
+ | Return the drives in original box, return label provided. | ||
+ | |||
+ | </ | ||
+ | |||
+ | ** Pool Unhealthy but not Degraded status** | ||
+ | |||
+ | No failed disks, no deploy of spare, but pool unhealthy. | ||
+ | |||
+ | < | ||
+ | |||
+ | Mar 21 04:03:57 hpcstore2 (da11: | ||
+ | Mar 21 04:03:57 hpcstore2 (da11: | ||
+ | Mar 21 04:03:57 hpcstore2 (da11: | ||
+ | Mar 21 04:03:57 hpcstore2 (da11: | ||
+ | Mar 21 04:03:57 hpcstore2 (da11: | ||
+ | Mar 21 04:03:57 hpcstore2 (da11: | ||
+ | |||
+ | 1) Storage > Pools. Click gear icon next to the pool and press the " | ||
+ | 2) Find da11 and press the three-dot options button next to it, then press " | ||
+ | 3) System > View Enclosure, find& | ||
+ | 4) Physically swap the drive on the rack with its replacement. | ||
+ | 5) Storage > Pool > Status page, bring up three-dot options for the removed drive, | ||
+ | 5a) Select member disk from drop down, and press " | ||
+ | 6) Wait till resilver finishes. | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Console hangs ==== | ||
+ | |||
+ | 12.7 | ||
+ | |||
+ | As for the issue of the " | ||
+ | |||
+ | service middlewared stop\\ | ||
+ | service middlewared start | ||
+ | |||
+ | " | ||
+ | |||
+ | |||
+ | |||
+ | ==== Update 12 ==== | ||
+ | |||
+ | System > Update > Select (new train 12.0-STABLE) | ||
+ | |||
+ | ** Open a console on both controllers without double ssh sessions, directly to hpcstore1/ | ||
+ | |||
+ | '' | ||
+ | '' | ||
+ | |||
+ | Then download updates on passive, check version '' | ||
+ | |||
+ | '' | ||
+ | |||
+ | '' | ||
+ | |||
+ | ...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% | ||
+ | |||
+ | reboot passive | ||
+ | |||
+ | from active ping passive heartbeat IP, when up | ||
+ | |||
+ | check version passive | ||
+ | |||
+ | check boot env '' | ||
+ | |||
+ | on passive '' | ||
+ | |||
+ | now force fail over via GUI (interruptive for 6o seconds) | ||
+ | |||
+ | Anthony did a reboot on active instead, watch log for personality swap | ||
+ | |||
+ | then update the new passive | ||
+ | |||
+ | '' | ||
+ | |||
+ | '' | ||
+ | |||
+ | then check version, reboot new passive, check version, become new standby | ||
+ | |||
+ | Result: personality switch active vs standby, took 35 mins | ||
+ | |||
+ | In two months: ZFS feature updates pathch, not interruptive, | ||
+ | Upgrade done | ||
+ | --- // | ||
+ | |||
+ | Storage > Pool > " | ||
+ | |||
+ | ** 12.0-U4.1 ** | ||
+ | |||
+ | * ditto above, see major release upgrade below | ||
+ | * but old active did not come up, reset controller | ||
+ | * click on " | ||
+ | * hmm something about failed to connect failoverscratchdisk? | ||
+ | |||
+ | ** 12.0-U5.1** | ||
+ | |||
+ | * standby reboot 5 mins | ||
+ | * fail over 1 min | ||
+ | * new standby "apply pending updates" | ||
+ | * this version went fine | ||
+ | |||
+ | __Not created/ | ||
+ | While the underlying issues have been fixed, this setting continues to be disabled by default for additional performance investigation. To manually reactivate persistent L2ARC, log in to the TrueNAS Web Interface, go to System > Tunables, and add a new tunable with these values: | ||
+ | < | ||
+ | Type = sysctl | ||
+ | Variable = vfs.zfs.l2arc.rebuild_enabled | ||
+ | Value = 1 | ||
+ | </ | ||
+ | |||
+ | From support: In an HA environment, | ||
+ | |||
+ | ** 12.0-U6 ** | ||
+ | |||
+ | * same as 5.1, went fine, | ||
+ | * new standby reboot 5 mins | ||
+ | |||
+ | |||
+ | ** 12.0-U6.1 ** | ||
+ | |||
+ | * same as 6, went fine, | ||
+ | * little flakiness on failover, apply pending appeared twice | ||
+ | * let it go 10 mins, use ping hostname to test | ||
+ | * new standby reboot 5 mins | ||
+ | |||
+ | ** 12.0-U7 ** | ||
+ | |||
+ | * major OpenZFS update | ||
+ | * same as update 12.0 | ||
+ | * no problems | ||
+ | * cpu was unusually busy before upgrade | ||
+ | * terminated some rsyncs | ||
+ | |||
+ | ** 12.0-U8 ** | ||
+ | |||
+ | * 02/23/2022 | ||
+ | * no problems | ||
+ | |||
+ | ** 12.0-U8.1 ** | ||
+ | |||
+ | * 05/03/2022 | ||
+ | * failover success at 10 mins | ||
+ | * then no Pending box, just a Continue button | ||
+ | * watch console messages, at 17 mins HA enabled | ||
+ | |||
+ | ==== Update 13 ==== | ||
+ | |||
+ | System > Update > Select (new train 13.0-STABLE) | ||
+ | |||
+ | < | ||
+ | |||
+ | # in shell | ||
+ | |||
+ | | ||
+ | |||
+ | | ||
+ | |||
+ | …10%…20%…30%…40%…50%…60%…70%…80%…90%…100% | ||
+ | |||
+ | beadm list | ||
+ | # (Active N = 12.0-U8.1 and R = 13.0-U3.1) | ||
+ | |||
+ | </ | ||
+ | |||
+ | once both have finished, reboot passive, web gui log back in | ||
+ | |||
+ | once passive back up, reboot active | ||
+ | |||
+ | web gui log back into new active, wait for HA to be enabled | ||
+ | |||
+ | debug plus screenshots for snapshot visibility which is visible (working in 13.0-U3.1) but database setting is still invisble | ||
+ | |||
+ | took less than 35 mins | ||
+ | |||
+ | < | ||
+ | |||
+ | bstop 0 | ||
+ | bresume 0 | ||
+ | # manual, one at a time | ||
+ | scontrol hold joblist | ||
+ | # one at a time | ||
+ | # for i in `squeue | grep ' | ||
+ | # then grep ' | ||
+ | scontrol suspend joblist | ||
+ | scontrol resume | ||
+ | scontrol release joblist | ||
+ | |||
+ | </ | ||