User Tools

Site Tools


hrms-sfis-upgrade:interim_portfolio_steps_for_go_live

Steps to initiate the Interim Portfolio for the go-live weekend

  1. Backup PSPROD 81 database on Fuji (~45 minutes)
  2. Stop the listener on Fuji; rename the listener log to start fresh; restart the listener (while the backup is happening)
  3. Alter the sqlnet.ora, tnsnames.ora on the interim portfolio machines, locally redirecting the tns entries PSPROD and SFIS to PS81TRNG on Bigsky.
    • interim portfolio machines are portfolio3,4 and 6 and are currently in the wesep round robin
    • change the oracle client configuration on the shared /opt/oracle9 so that all machines that use that client are directed to PS81TRNG, which includes clusterw1 and 2 and other machines that may use this client that we may have missed
  4. overlay the PS81TRNG database on Bigsky with the fresh backup of PSPROD 81 from Fuji.
  5. Stop the listener on Bigsky; rename the listener log to start fresh; restart the listener (while the backup is happening)
  6. Alter the sqlnet.ora, tnsnames.ora on Bigsky, locally redirecting the tns entries PSPROD and SFIS to PS81TRNG on Bigsky.
  7. remove the IP addresses of all portfolio machines from the PSPROD 81 database on Fuji in order to prevent any mistaken portfolio contact.
  8. remove all IP addresses of all portfolio machines from the PS81TRNG database on Bigsky *except* for the interim portfolio machine IP addresses, in order to prevent any mistaken rogue portfolio contact.
  9. login to the CURL.OPS$NOTESR5 database schema, and create database link PSPROD to redirect to PS81TRNG and *not* PSPROD database.

Techy Timeline

  • webmail available all the time

Thursday, July 12, 10:00 AM:

  • not needed, no drops. disable snapshot controller onecard & blackboards
  • DONE marked with '#PSUPG#' disable portfolio5 cron jobs, includes bulkemail
  • DONE marked with '#PSUPG#' disable htt process on: clusterw3 - nobody cron (1 htt_scanner, 1 htt_starter), goofy - www cron (2 htt_starter, 2 htt_scanner), portfolio5 - notesr5 cron (6 htt_converter)
  • DONE create a local oracle client for portfolio5
  • DONE configure local tnsnames.ora & portfolio5 (ports 80/443 also know as portfolio5b) to function as final test portfolio for saturday/sunday testing

Thursday, July 12, 12:00 PM:

  • DONE bring eportfolio down (3/4/6 running against PSPROD/fuji + CURL) with alternate landing page
  • DONE bring blackboard down with alternate landing page
  • DONE bring portfolio5 test vhosts down
  • ??? bring IAS down (pub/int/ext)
  • DONE summerreg down

Thursday, July 12, 3:00 PM:

  • DONE start of final backup of databases by SW CURL/PSPROD pre-upgrade
  • DONE bring CURL up as soon as possible, notify Henk, BB becomes available
  • DONE followed by database overlay on PS81TRNG/bigsky
  • DONE bring PS81TRNG/bigsky up
  • DONE IAS repoint to PS81TRNG
  • avoid oranames … change tnsname.ora on _shared_ oracle client, point SFIS → PS81TRNG
  • DONE bring interim eportfolio up (3/4/6 running RO mode) … 07/11/07 links not available in original CURL
  • blackboard is up, wesmaps is up …

Friday, July 13, 5:00 PM:

  • DONE wessms: on portfolio5 has been hardwired to make connections to real CURL and PS81TRNG so that it can remain running when the test server is brought up and CURL refers to CURLTEST
  • DONE turn bulkemail back on and point to real CURL
  • DONE have portfolio5b test server up and running
  • DONE redirect portfoliotest test server to informational page with reference to portfolio5b

Saturday, July 14, 1:00 PM:

  • interim eportfolio (CURL/PS81TRNG) remains up
  • original CURL remains up
  • blackboard is up, wesmaps is up …
  • all links will be available in CURLTEST
  • NO ??? firewall for VLAN 20 access only
  • TD's group first tests the portal
  • TD's group next tests applications via portfolio5b …

Saturday, July 14, 5:00 PM:

  • testing stops, no action items

Sunday, July 15, 11:00 AM:

  • portfolio5b and portfoliotest, no action items (links are already open in CURLTEST)
  • –> this is the campus wide testing period which is schedule to last till ~5:00 PM (can be extended)
  • NO ??? firewall to only allow access for wesleyan.edu (requires off-site VPN usage)
  • at 5PM point of no return all clear invokes final database overlay by SW erasing all test data inserted in SFIS89PROD …. can be done anytime AFTER MIDNIGHT WHEN SUMMER REG CLOSES
  • interim portfolio remains up
  • blackboard remains up, wesmaps remains up

Sunday, July 15th, midnight:

  • summereg closes

Monday, July 16, 06:00 AM:

  • prep to go-live
  • eportfolio down, blackboard down, wesmaps down for final CURL backup before upgrade …
  • SW conversion of original CURL to new CURL (done in place) for 08:00 AM release
  • SW last overlay of SFIS89PROD for 08:00 AM release
  • IAS repoint to SFIS89PROD
  • DONE wessms: undo connections to real CURL and PS81TRNG and point back to CURL/SFIS in wessms_utils.pl
  • DONE bulkemail: undo connections to real CURL and PS81TRNG

Monday, July 16, 8:00 AM:

  • blackboard up, wesdmaps up …
  • DONE switch back to original config on shared oracle client (use oranames)
  • DONE interim portfolio restarts and switches from oldCURL/PS81TRNG to newCURL/SFIS89PROD.
  • sometime monday PM, portfolio5b rejoins the eportfolio servers in normal operations

Monday, July 16 onwards:

  • payroll must be available by 10:00 AM
  • find & fix
  • DONE start cronjobs, includes bulkemail
  • no needed, no drops. enable snapshot controller onecard & blackboards
  • DONE start htt process on condor's goofy
hrms-sfis-upgrade/interim_portfolio_steps_for_go_live.txt · Last modified: 2007/07/16 10:01 (external edit)