| 1 |
/home/production/cvs/JSOC/doc/whattodo_start_stop_lev1_0_sums.txt 28Jun2012 |
| 2 |
/home/production/cvs/JSOC/doc/whattodo_start_stop_lev1_0_sums.txt 25Oct2013 |
| 3 |
|
| 4 |
|
| 5 |
Do as user production or jsocprod as indicated. Do in this order: |
| 6 |
|
| 7 |
NOTE: Now the drms/sums restart feature is working, there is no need |
| 8 |
to do the "Stop lev1" and "Start lev0" sections below. |
| 9 |
|
| 10 |
STOP: |
| 11 |
===== |
| 12 |
|
| 13 |
On cl1n003: Stop lev1 processing for AIA as jsocprod |
| 14 |
---------------------------------------------------- |
| 15 |
> touch /usr/local/logs/lev1/build_mgr_stop_aia |
| 16 |
|
| 17 |
Wait until you see on the screen running build_lev1_mgr on cl1n003: |
| 18 |
|
| 19 |
Stop file /usr/local/logs/lev1/build_mgr_stop_aia seen. |
| 20 |
Wait until all children are done and exit... |
| 21 |
|
| 22 |
Of course, if you not local with the screen, use ps. |
| 23 |
|
| 24 |
(note: ignore any core dump) |
| 25 |
|
| 26 |
|
| 27 |
On cl1n002: Stop lev1 processing for HMI as jsocprod |
| 28 |
---------------------------------------------------- |
| 29 |
> touch /usr/local/logs/lev1/build_mgr_stop_hmi |
| 30 |
|
| 31 |
Wait until you see on the screen running build_lev1_mgr on cl1n002: |
| 32 |
|
| 33 |
Stop file /usr/local/logs/lev1/build_mgr_stop_hmi seen. |
| 34 |
Wait until all children are done and exit... |
| 35 |
|
| 36 |
Of course, if you not local with the screen, use ps. |
| 37 |
|
| 38 |
(note: ignore any core dump) |
| 39 |
|
| 40 |
|
| 41 |
On cl1n001: Stop lev0 processing as jsocprod |
| 42 |
-------------------------------------------- |
| 43 |
|
| 44 |
> cd /home/jsocprod |
| 45 |
> stop_lev0_HMI.pl |
| 46 |
|
| 47 |
This may take a few minutes while the processing of ingest_lev0 for |
| 48 |
VC02 and VC05 completes to a commit point. |
| 49 |
If the ingest_lev0 process is hung, it may not see the stop file signal. |
| 50 |
Then do a kill -9 of the ingest_lev0 processes. |
| 51 |
Make sure ingest_lev0 is not running before you try to restart it. |
| 52 |
|
| 53 |
Likewise for AIA VC01 and VC04: |
| 54 |
|
| 55 |
> stop_lev0_AIA.pl |
| 56 |
|
| 57 |
|
| 58 |
On j1: Stop SUMS as production |
| 59 |
------------------------------------- |
| 60 |
|
| 61 |
> cd /home/production/cvs/JSOC/base/sums/scripts |
| 62 |
> sum_stop_j1.MULTI -f (note: may need the -f(orce) flag) |
| 63 |
|
| 64 |
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
| 65 |
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
| 66 |
|
| 67 |
START: |
| 68 |
===== |
| 69 |
|
| 70 |
On j1: Start SUMS as production |
| 71 |
------------------------------------- |
| 72 |
|
| 73 |
!!!NOTE: sums will be moved to k1. Use these: |
| 74 |
sum_start_k1.MULTI sum_stop_k1.MULTI |
| 75 |
|
| 76 |
> cd /home/production/cvs/JSOC/base/sums/scripts |
| 77 |
> sum_start_j1.MULTI |
| 78 |
|
| 79 |
|
| 80 |
On d02: Restart t950view as production |
| 81 |
-------------------------------------- |
| 82 |
This is optional if you want to view the tape activity. |
| 83 |
Restore the minimized t950view window and hit the Exit button (upper right), |
| 84 |
or use ps and kill. |
| 85 |
> /home/production/cvs/JSOC/base/sums/scripts/t950view -i jsoc_sums |
| 86 |
(password: hmi4sdo) |
| 87 |
|
| 88 |
|
| 89 |
On cl1n001: Start lev0 processing (if needed) as jsocprod |
| 90 |
--------------------------------------------------------- |
| 91 |
This should be done through 'screen' running on cl1n001. |
| 92 |
For "How to Use 'screen'" see: |
| 93 |
http://linux.about.com/od/Bash_Scripting_Solutions/a/How-To-Use-The-Gnu-Linux-Screen-Terminal-Window-Manager.htm |
| 94 |
This will allow the lev1 processing to continue if your w/s (xim) crashes, |
| 95 |
and the screens to be re-attached to, when xim reboots, or from another w/s. |
| 96 |
|
| 97 |
xim:/home/jim> ssh jsocprod@j1 |
| 98 |
Password: |
| 99 |
Last login: Fri Apr 27 12:14:41 2012 from xim.stanford.edu |
| 100 |
Fri Apr 27 12:20:49 PDT 2012 |
| 101 |
j1:/home/jsocprod> ssh cl1n001 |
| 102 |
cl1n001:/home/jsocprod> screen |
| 103 |
Directory: /home/jsocprod |
| 104 |
csh> cd |
| 105 |
cl1n001:/home/jsocprod> Ctrl-a c (to create anothe screen) |
| 106 |
To toggle between screens do: Ctrl-a space |
| 107 |
Or to go to screen 2: Ctrl-a 2 |
| 108 |
To disconnect from 'screen' do: Ctrl-a d |
| 109 |
(You will still see a process running called SCREEN). |
| 110 |
To reconnect to a previous screen (after disconnect or machine restart or |
| 111 |
from another workstation from a cl1n001 window): |
| 112 |
> screen -r |
| 113 |
To kill a screen window: Ctrl-a K (on some machine small 'k') |
| 114 |
To see all your screens created and which on you're on now: Ctrl-a w |
| 115 |
To see all screen key bindings: Ctrl-a ? |
| 116 |
|
| 117 |
In screen 0 do: |
| 118 |
|
| 119 |
!!Make sure run Phil's script for watchlev0 in the background on cl1n001: |
| 120 |
/home/production/cvs/JSOC/base/sums/scripts/get_dcs_times.csh |
| 121 |
|
| 122 |
> cd /home/jsocprod |
| 123 |
> doingestlev0_HMI.pl & |
| 124 |
> doingestlev0_AIA.pl & |
| 125 |
|
| 126 |
On cl1n002: Start lev1 processing for HMI (if needed) as jsocprod |
| 127 |
----------------------------------------------------------------- |
| 128 |
Run a 'screen' on cl1n002: |
| 129 |
xim:/home/jim> ssh jsocprod@j1 |
| 130 |
Password: |
| 131 |
Last login: Thu Apr 26 14:54:53 2012 from xim.stanford.edu |
| 132 |
Directory: /home/jsocprod |
| 133 |
j1:/home/jsocprod> ssh cl1n002 |
| 134 |
> screen |
| 135 |
> cd /home/jsocprod |
| 136 |
> build_lev1_mgr mode=recnum |
| 137 |
instru=hmi dsin=hmi.lev0a dsout=hmi.lev1_nrt brec=0 erec=0 |
| 138 |
|
| 139 |
On cl1n003: Start lev1 processing for AIA (if needed) as jsocprod |
| 140 |
----------------------------------------------------------------- |
| 141 |
Run a 'screen' on cl1n003 |
| 142 |
xim:/home/jim> ssh jsocprod@j1 |
| 143 |
Password: |
| 144 |
Last login: Thu Apr 26 14:54:53 2012 from xim.stanford.edu |
| 145 |
Directory: /home/jsocprod |
| 146 |
j1:/home/jsocprod> ssh cl1n003 |
| 147 |
> screen |
| 148 |
> cd /home/jsocprod |
| 149 |
> build_lev1_mgr mode=recnum |
| 150 |
instru=aia dsin=aia.lev0 dsout=aia.lev1_nrt2 brec=0 erec=0 |
| 151 |
|
| 152 |
=========================================================================== |
| 153 |
For data capture machines see: |
| 154 |
|
| 155 |
> cd /home/production/cvs/JSOC/doc |
| 156 |
|
| 157 |
whattodo_dcs.txt |
| 158 |
(dsc0_just_rebooted.txt may be needed if you want to do Pg warmstart) |
| 159 |
|
| 160 |
If it's dcs1 that you need, then you have to transcribe a bit. |
| 161 |
Remember that dcs0 and dcs1 databases are in warm standby on dcs2. |
| 162 |
If you want to ignore the instructions for this for now, I'll take care |
| 163 |
of it when I'm back. Just let me know that you did so. |
| 164 |
|
| 165 |
Info on log files and cron jobs is in logs_cron_etc_admin.txt |
| 166 |
|
