hrms-sfis-upgrade:interim_portfolio_steps_for_go_live
Steps to initiate the Interim Portfolio for the go-live weekend
- Backup PSPROD 81 database on Fuji (~45 minutes)
- Stop the listener on Fuji; rename the listener log to start fresh; restart the listener (while the backup is happening)
- Alter the sqlnet.ora, tnsnames.ora on the interim portfolio machines, locally redirecting the tns entries PSPROD and SFIS to PS81TRNG on Bigsky.
- interim portfolio machines are portfolio3,4 and 6 and are currently in the wesep round robin
- change the oracle client configuration on the shared /opt/oracle9 so that all machines that use that client are directed to PS81TRNG, which includes clusterw1 and 2 and other machines that may use this client that we may have missed
- overlay the PS81TRNG database on Bigsky with the fresh backup of PSPROD 81 from Fuji.
- Stop the listener on Bigsky; rename the listener log to start fresh; restart the listener (while the backup is happening)
- Alter the sqlnet.ora, tnsnames.ora on Bigsky, locally redirecting the tns entries PSPROD and SFIS to PS81TRNG on Bigsky.
- remove the IP addresses of all portfolio machines from the PSPROD 81 database on Fuji in order to prevent any mistaken portfolio contact.
- remove all IP addresses of all portfolio machines from the PS81TRNG database on Bigsky *except* for the interim portfolio machine IP addresses, in order to prevent any mistaken rogue portfolio contact.
- login to the CURL.OPS$NOTESR5 database schema, and create database link PSPROD to redirect to PS81TRNG and *not* PSPROD database.
Techy Timeline
- webmail available all the time
Thursday, July 12, 10:00 AM:
- not needed, no drops. disable snapshot controller onecard & blackboards
- DONE marked with '#PSUPG#' disable portfolio5 cron jobs, includes bulkemail
- DONE marked with '#PSUPG#' disable htt process on: clusterw3 - nobody cron (1 htt_scanner, 1 htt_starter), goofy - www cron (2 htt_starter, 2 htt_scanner), portfolio5 - notesr5 cron (6 htt_converter)
- DONE create a local oracle client for portfolio5
- DONE configure local tnsnames.ora & portfolio5 (ports 80/443 also know as portfolio5b) to function as final test portfolio for saturday/sunday testing
Thursday, July 12, 12:00 PM:
- DONE bring eportfolio down (3/4/6 running against PSPROD/fuji + CURL) with alternate landing page
- DONE bring blackboard down with alternate landing page
- DONE bring portfolio5 test vhosts down
- ??? bring IAS down (pub/int/ext)
- DONE summerreg down
Thursday, July 12, 3:00 PM:
- DONE start of final backup of databases by SW CURL/PSPROD pre-upgrade
- DONE bring CURL up as soon as possible, notify Henk, BB becomes available
- DONE followed by database overlay on PS81TRNG/bigsky
- DONE bring PS81TRNG/bigsky up
- DONE IAS repoint to PS81TRNG
avoid oranames … change tnsname.ora on _shared_ oracle client, point SFIS → PS81TRNG- DONE bring interim eportfolio up (3/4/6 running RO mode) … 07/11/07 links not available in original CURL
- blackboard is up, wesmaps is up …
Friday, July 13, 5:00 PM:
- DONE wessms: on portfolio5 has been hardwired to make connections to real CURL and PS81TRNG so that it can remain running when the test server is brought up and CURL refers to CURLTEST
- DONE turn bulkemail back on and point to real CURL
- DONE have portfolio5b test server up and running
- DONE redirect portfoliotest test server to informational page with reference to portfolio5b
Saturday, July 14, 1:00 PM:
- interim eportfolio (CURL/PS81TRNG) remains up
- original CURL remains up
- blackboard is up, wesmaps is up …
- all links will be available in CURLTEST
- NO ??? firewall for VLAN 20 access only
- TD's group first tests the portal
- TD's group next tests applications via portfolio5b …
Saturday, July 14, 5:00 PM:
- testing stops, no action items
Sunday, July 15, 11:00 AM:
- portfolio5b and portfoliotest, no action items (links are already open in CURLTEST)
- –> this is the campus wide testing period which is schedule to last till ~5:00 PM (can be extended)
- NO ??? firewall to only allow access for wesleyan.edu (requires off-site VPN usage)
- at 5PM <hi orange> point of no return </hi> all clear invokes final database overlay by SW erasing all test data inserted in SFIS89PROD …. can be done anytime AFTER MIDNIGHT WHEN SUMMER REG CLOSES
- interim portfolio remains up
- blackboard remains up, wesmaps remains up
Sunday, July 15th, midnight:
- summereg closes
Monday, July 16, 06:00 AM:
- prep to go-live
- eportfolio down, blackboard down, wesmaps down for final CURL backup before upgrade …
- SW conversion of original CURL to
newCURL (done in place) for 08:00 AM release - SW last overlay of SFIS89PROD for 08:00 AM release
- IAS repoint to SFIS89PROD
- DONE wessms: undo connections to real CURL and PS81TRNG and point back to CURL/SFIS in wessms_utils.pl
- DONE bulkemail: undo connections to real CURL and PS81TRNG
Monday, July 16, 8:00 AM:
- blackboard up, wesdmaps up …
- DONE switch back to original config on shared oracle client (use oranames)
- DONE interim portfolio restarts and switches from oldCURL/PS81TRNG to newCURL/SFIS89PROD.
- sometime monday PM, portfolio5b rejoins the eportfolio servers in normal operations
Monday, July 16 onwards:
- payroll must be available by 10:00 AM
- find & fix
- DONE start cronjobs, includes bulkemail
- no needed, no drops. enable snapshot controller onecard & blackboards
- DONE start htt process on condor's goofy
hrms-sfis-upgrade/interim_portfolio_steps_for_go_live.txt · Last modified: by 127.0.0.1
