CM/V1.0/drms_module.txt

1. ============= DRMS Module overview ==============

DRMS modules share a common main function found in
jsoc/src/base/drms/jsoc_main.c. The module executable is run like a
normal Unix command line program (see below) and its main function
performs the following steps:

1. Parses the command line options and stores them in a 
   global data structure.
2. Connects to a DRMS session master process.
3. Calls the module's main function "DoIt()".
4. If DoIt() returns a non-zero status code, indicating an error,
   it sends an abort request to the DRMS session master.
   If DoIt() returns a zero status code (success) it sends a message 
   to the DRMS session master asking it to commit the data generated
   or modified by the module when the session ends.
5. Disconnects from the DRMS session master.

The top-level file for implementing a module should
look something like this:


------------------ module.c ------------------------
#include "drms.h"
#include "jsoc_main.h"

/* List of default parameter values. */
DefaultParams_t default_params[] = { 
  {"parm_name1", "default value1"},
  {"parm_name2", "default value2"},
  {"mandatory_parm3", NULL },
/*  ... more default named parameter values ... */
  {NULL, NULL} /* List must end in {NULL, NULL}. */
};

/* Module main function. */
int DoIt(void)
{
  int status;

  /* Do work...
     ... 
     Set status == 0 to indicate success.
     Set status != 0 to indicate failure. */

  return status;
}
----------------------------------------------------


2. =========== DRMS command line parsing ==============

The functions associated with command line parsing are found in
jsoc/src/util/util/cmdparams.{c,h}.
            
The DRMS main program parses the module command line and stores
the information in a global data structure

CmdParams_t cmdparams;

that can be used to access the parameters from anywhere within the
module code, including library subroutines. The command line consists 
of four types of tokens

 * named parameters given in one of the forms "variable= value", 
   "variable=value"  or "--variable value"
 * single letter flags "-a -b -c" which can also be written in 
   concatenated form "-abc". Flags are translated into named single 
   letter named parameters with the value "1".
 * unnamed argument strings of the form "value"
 * command line files of the form "@filename". Each line such a file
   is parsed as an additional command line. Command files may contain
   references to other command files. Blank lines or lines beginning 
   in "#" are treated as comment lines and ignored.
   Command line files are a convenient mechanism go circumvent the
   limitation on the number of command line arguments in most operating 
   systems.


-------- example -----------
Example: Assume that the file inputs.conf contains the three lines 

# This is a test
input1.txt
input2.txt

then the command line

  module.exe  -vf test=debug abc.txt --log logfile def.bin @inputs.conf

will get parsed to have 3 named parameters

  v    = "1"
  f    = "1"
  test = "debug"
  log  = "logfile"

and 4 unnamed arguments

  abc.txt
  def.bin
  input1.txt
  input2.txt
-------- end example ------

The values of the named parameters are read using the following 
functions:

  char *cmdparams_get_str(CmdParams_t *parms, char *name, int *status);
  int8_t cmdparams_get_int8(CmdParams_t *parms, char *name, int *status);
  int16_t cmdparams_get_int16(CmdParams_t *parms, char *name, int *status);
  int32_t cmdparams_get_int32(CmdParams_t *parms, char *name, int *status);
  int64_t cmdparams_get_int64(CmdParams_t *parms, char *name, int *status);
  float cmdparams_get_float(CmdParams_t *parms, char *name, int *status);
  double cmdparams_get_double(CmdParams_t *parms, char *name, int *status);
  double cmdparams_get_time(CmdParams_t *parms, char *name, int *status);

If the named parameter is was not given on the command line
the functions above try to obtain its value from the environment
using the getenv function. Therefore the commands

  module.exe blah="Hello"

and

  setenv blah Hello
  module.exe

should have the same outcome.

The function

  int cmdparams_exists(CmdParams_t *parms, char *name);

returns 1 if a named parameter matching the string in "name"
was given on the command line, and 0 if no such parameters was
given.

The (string) values of the unnamed arguments are read using the 
following functions:

  char *cmdparams_getarg(CmdParams_t *parms, int num);
  int cmdparams_numargs(CmdParams_t *parms);


cmdparams_getarg(cmdparms, 0);

returns the name of the running program (argv[0]).

Default values for parameters can be given in the global struct 
default_params that must be present in the module. The struct 
takes the following form:

DefaultParams_t
 default_params[] = { 
  {"parm_name1", "default value1"},
  {"parm_name2", "default value2"},
  {"mandatory_parm3", NULL },
/*  ... more default named parameter values ... */
  {NULL, NULL} /* List must end in {NULL, NULL}. */
};

If the value field in the struct for a given parameter is
NULL it means that the parameter is mandatory and must be
present on the command line. If not, an error message will
be printed out and the module terminated immediately after
command line parsing.


3. =========== DRMS data functions ==============

The module read and writes data using the functions described in
jsoc/CM/*/drms_api.txt.


4. =========== Running a DRMS module ===========

Running one or more DRMS modules involves three main steps 

a) starting a DRMS session, 
b) runnning the module(s) and 
c) closing the session.

The final step will either commit all the data generated by 
modules in the session or discard it if an error occured.

The script /jsoc/scripts/drms/drms_run automates the three steps
detailed below, and allows modules (or scripts containing multiple 
module commands) to be run with a single command.
The command

  host:~> drms_run <command> [options...]

will start a new DRMS server, run <command> and depending on the exit
status of <command> will either commit or discard changes to the
database and stop the DRMS server. drms_run will use the drms_server
executable pointed to by the environment variable DRMS_SERVER_EXE. If
DRMS_SERVER_EXE is not set drms_run will assume that an executable
"drms_server" is in your path. The output from the DRMS server is
piped to the file pointed to by the environment variable
DRMS_LOGFILE. If DRMS_SERVER_EXE is not set drms_run will create a log
file in /tmp/DRMS.<pid>, where <pid> is the PID of the drms_run
script interpreter.

The three steps are carried out as follows: 

  a) Before you run modules you must have a DRMS server running to 
     act as a session master. This can be done by running the command

     host:~>  jsoc/bin/<target>/drms_server -f

     The server will print out what interface it is listening
     for connections on. For example:

      akhenaten:~/jsoc> bin/custom.akhenaten/drms_server -f
      DRMS_HOST = akhenaten.Stanford.EDU
      DRMS_PORT = 33137
      DRMS_PID = 20955
      DRMS_SESSIONID = 38
      DRMS server started with pid=20955, noshare=0, noroe=0
      ...

     The "-f" flag makes the server run in the foreground. Without
     "-f" the drms_server command spawn a server in a background 
     process, prints the connection info to stdout (as above) 
     and exits. 

     The server will print log messages to stdout and
     stderr (TBD: Clean up error handling and logging.), and these
     should be piped to a file if you intend to keep them.   

    
  b) Now you can run the module(s). The modules do not need to run on
     the same host as the server. They can run on any host as long as
     they are able to open a TCP socket connection to the server
     process.

     When running a module, the named parameter DRMSSESSION must be set 
     to indicate the host and port where the DRMS server is listening 
     for connection attempts. It is perhaps most convenient to do this 
     by setting the environment variable DRMSSESSION. In the example above 
     this would mean executing the command: 
   
     akhenaten:~/jsoc> setenv DRMSSESSION akhenaten:33137

     Each module that connects causes the server to spawn a new thread
     to service the new client.  The server can service multiple
     clients simultaneously, but database operations are serialized
     within the server and executed sequentially using a shared
     connection to the DRMS database.


  3. When all modules have finished successfully you can either
     
       a) tell the DRMS server stop and commit all data generated or 
          modified by the modules to the DRMS database by sending a 
          SIGUSR1 signal to it. In the example above that would mean 
          issuing the command

          akhenaten:~/jsoc> kill -s USR1 20955
     
     or if an error occurs you can
  
      b) tell the DRMS server to abort and discard all data generated
         by the modules by sending it a SIGTERM, SIGQUIT or SIGINT.
         In the example above that could be done by pressing CTRL-C
         in the terminal where the server is running or by issuing
         the command 

         akhenaten:~/jsoc> kill -s INT 20955    
     
         It should be safe to kill the server with SIGKILL (kill -9).
         It will have the same effect as a regular abort except that it 
         leaves a stale entry in DRMS's active session table.


Revision:	1.1.1.1 (vendor branch)
Committed:	Tue Oct 2 00:12:21 2007 UTC (15 years, 11 months ago) by arta
Content type:	text/plain
Branch:	MAIN, Vtag
CVS Tags:	Ver_6-0, Ver_6-1, Ver_6-2, Ver_6-3, Ver_6-4, Ver_4-3, Ver_4-0, Ver_4-1, NetDRMS_Ver_8-8, NewTree01_cp03_JSOC, Ver_4-4, Ver_8-5, Ver_4-7, NewTree01_cp05_JSOC, Ver_5-14, Ver_5-13, Ver_5-12, Ver_5-11, Ver_5-10, Ver_LATEST, NetDRMS_Ver_LATEST, Ver_4-6, NewTree01_cp04_JSOC, NetDRMS_Ver_8-12, NetDRMS_Ver_8-10, NetDRMS_Ver_8-11, NetDRMS_Ver_9-1, NetDRMS_Ver_9-0, NetDRMS_Ver_9-3, NetDRMS_Ver_9-2, NetDRMS_Ver_9-5, NetDRMS_Ver_9-4, Ver_7-0, Ver_5-6, Ver_4-5, NewTree01_cp07_JSOC, NewTree01_cp08_JSOC, NewTree01_cp01_JSOC, Ver_4-2, NetDRMS_Ver_9-41, Ver_9-41, NewTree01_cp02_JSOC, NetDRMS_Ver_8-4, NetDRMS_Ver_8-5, Ver_5-8, NetDRMS_Ver_8-6, Ver_5-7, Ver_8-8, NetDRMS_Ver_8-7, NewTree01_cp06_JSOC, Ver_5-9, Ver_8-2, Ver_9-3, Ver_8-0, Ver_8-1, Ver_8-6, Ver_8-7, Ver_8-4, Ver_8-11, Ver_5-3, Ver_5-2, Ver_5-1, Ver_5-0, Ver_7-1, Ver_9-1, Ver_5-5, Ver_8-3, NewTree01_cp09_JSOC, Ver_9-5, Ver_9-4, Ver_8-10, Ver_9-2, Ver_8-12, Ver_9-0, HEAD
Changes since 1.1:	+0 -0 lines
Log Message:	First new, reorganized JSOC tree
#	Content
1	1. ============= DRMS Module overview ==============
2
3	DRMS modules share a common main function found in
4	jsoc/src/base/drms/jsoc_main.c. The module executable is run like a
5	normal Unix command line program (see below) and its main function
6	performs the following steps:
7
8	1. Parses the command line options and stores them in a
9	global data structure.
10	2. Connects to a DRMS session master process.
11	3. Calls the module's main function "DoIt()".
12	4. If DoIt() returns a non-zero status code, indicating an error,
13	it sends an abort request to the DRMS session master.
14	If DoIt() returns a zero status code (success) it sends a message
15	to the DRMS session master asking it to commit the data generated
16	or modified by the module when the session ends.
17	5. Disconnects from the DRMS session master.
18
19	The top-level file for implementing a module should
20	look something like this:
21
22
23	------------------ module.c ------------------------
24	#include "drms.h"
25	#include "jsoc_main.h"
26
27	/* List of default parameter values. */
28	DefaultParams_t default_params[] = {
29	{"parm_name1", "default value1"},
30	{"parm_name2", "default value2"},
31	{"mandatory_parm3", NULL },
32	/* ... more default named parameter values ... */
33	{NULL, NULL} /* List must end in {NULL, NULL}. */
34	};
35
36	/* Module main function. */
37	int DoIt(void)
38	{
39	int status;
40
41	/* Do work...
42	...
43	Set status == 0 to indicate success.
44	Set status != 0 to indicate failure. */
45
46	return status;
47	}
48	----------------------------------------------------
49
50
51	2. =========== DRMS command line parsing ==============
52
53	The functions associated with command line parsing are found in
54	jsoc/src/util/util/cmdparams.{c,h}.
55
56	The DRMS main program parses the module command line and stores
57	the information in a global data structure
58
59	CmdParams_t cmdparams;
60
61	that can be used to access the parameters from anywhere within the
62	module code, including library subroutines. The command line consists
63	of four types of tokens
64
65	* named parameters given in one of the forms "variable= value",
66	"variable=value" or "--variable value"
67	* single letter flags "-a -b -c" which can also be written in
68	concatenated form "-abc". Flags are translated into named single
69	letter named parameters with the value "1".
70	* unnamed argument strings of the form "value"
71	* command line files of the form "@filename". Each line such a file
72	is parsed as an additional command line. Command files may contain
73	references to other command files. Blank lines or lines beginning
74	in "#" are treated as comment lines and ignored.
75	Command line files are a convenient mechanism go circumvent the
76	limitation on the number of command line arguments in most operating
77	systems.
78
79
80	-------- example -----------
81	Example: Assume that the file inputs.conf contains the three lines
82
83	# This is a test
84	input1.txt
85	input2.txt
86
87	then the command line
88
89	module.exe -vf test=debug abc.txt --log logfile def.bin @inputs.conf
90
91	will get parsed to have 3 named parameters
92
93	v = "1"
94	f = "1"
95	test = "debug"
96	log = "logfile"
97
98	and 4 unnamed arguments
99
100	abc.txt
101	def.bin
102	input1.txt
103	input2.txt
104	-------- end example ------
105
106	The values of the named parameters are read using the following
107	functions:
108
109	char cmdparams_get_str(CmdParams_t parms, char name, int status);
110	int8_t cmdparams_get_int8(CmdParams_t parms, char name, int *status);
111	int16_t cmdparams_get_int16(CmdParams_t parms, char name, int *status);
112	int32_t cmdparams_get_int32(CmdParams_t parms, char name, int *status);
113	int64_t cmdparams_get_int64(CmdParams_t parms, char name, int *status);
114	float cmdparams_get_float(CmdParams_t parms, char name, int *status);
115	double cmdparams_get_double(CmdParams_t parms, char name, int *status);
116	double cmdparams_get_time(CmdParams_t parms, char name, int *status);
117
118	If the named parameter is was not given on the command line
119	the functions above try to obtain its value from the environment
120	using the getenv function. Therefore the commands
121
122	module.exe blah="Hello"
123
124	and
125
126	setenv blah Hello
127	module.exe
128
129	should have the same outcome.
130
131	The function
132
133	int cmdparams_exists(CmdParams_t parms, char name);
134
135	returns 1 if a named parameter matching the string in "name"
136	was given on the command line, and 0 if no such parameters was
137	given.
138
139	The (string) values of the unnamed arguments are read using the
140	following functions:
141
142	char cmdparams_getarg(CmdParams_t parms, int num);
143	int cmdparams_numargs(CmdParams_t *parms);
144
145
146	cmdparams_getarg(cmdparms, 0);
147
148	returns the name of the running program (argv[0]).
149
150	Default values for parameters can be given in the global struct
151	default_params that must be present in the module. The struct
152	takes the following form:
153
154	DefaultParams_t
155	default_params[] = {
156	{"parm_name1", "default value1"},
157	{"parm_name2", "default value2"},
158	{"mandatory_parm3", NULL },
159	/* ... more default named parameter values ... */
160	{NULL, NULL} /* List must end in {NULL, NULL}. */
161	};
162
163	If the value field in the struct for a given parameter is
164	NULL it means that the parameter is mandatory and must be
165	present on the command line. If not, an error message will
166	be printed out and the module terminated immediately after
167	command line parsing.
168
169
170	3. =========== DRMS data functions ==============
171
172	The module read and writes data using the functions described in
173	jsoc/CM/*/drms_api.txt.
174
175
176	4. =========== Running a DRMS module ===========
177
178	Running one or more DRMS modules involves three main steps
179
180	a) starting a DRMS session,
181	b) runnning the module(s) and
182	c) closing the session.
183
184	The final step will either commit all the data generated by
185	modules in the session or discard it if an error occured.
186
187	The script /jsoc/scripts/drms/drms_run automates the three steps
188	detailed below, and allows modules (or scripts containing multiple
189	module commands) to be run with a single command.
190	The command
191
192	host:~> drms_run <command> [options...]
193
194	will start a new DRMS server, run <command> and depending on the exit
195	status of <command> will either commit or discard changes to the
196	database and stop the DRMS server. drms_run will use the drms_server
197	executable pointed to by the environment variable DRMS_SERVER_EXE. If
198	DRMS_SERVER_EXE is not set drms_run will assume that an executable
199	"drms_server" is in your path. The output from the DRMS server is
200	piped to the file pointed to by the environment variable
201	DRMS_LOGFILE. If DRMS_SERVER_EXE is not set drms_run will create a log
202	file in /tmp/DRMS.<pid>, where <pid> is the PID of the drms_run
203	script interpreter.
204
205	The three steps are carried out as follows:
206
207	a) Before you run modules you must have a DRMS server running to
208	act as a session master. This can be done by running the command
209
210	host:~> jsoc/bin/<target>/drms_server -f
211
212	The server will print out what interface it is listening
213	for connections on. For example:
214
215	akhenaten:~/jsoc> bin/custom.akhenaten/drms_server -f
216	DRMS_HOST = akhenaten.Stanford.EDU
217	DRMS_PORT = 33137
218	DRMS_PID = 20955
219	DRMS_SESSIONID = 38
220	DRMS server started with pid=20955, noshare=0, noroe=0
221	...
222
223	The "-f" flag makes the server run in the foreground. Without
224	"-f" the drms_server command spawn a server in a background
225	process, prints the connection info to stdout (as above)
226	and exits.
227
228	The server will print log messages to stdout and
229	stderr (TBD: Clean up error handling and logging.), and these
230	should be piped to a file if you intend to keep them.
231
232
233	b) Now you can run the module(s). The modules do not need to run on
234	the same host as the server. They can run on any host as long as
235	they are able to open a TCP socket connection to the server
236	process.
237
238	When running a module, the named parameter DRMSSESSION must be set
239	to indicate the host and port where the DRMS server is listening
240	for connection attempts. It is perhaps most convenient to do this
241	by setting the environment variable DRMSSESSION. In the example above
242	this would mean executing the command:
243
244	akhenaten:~/jsoc> setenv DRMSSESSION akhenaten:33137
245
246	Each module that connects causes the server to spawn a new thread
247	to service the new client. The server can service multiple
248	clients simultaneously, but database operations are serialized
249	within the server and executed sequentially using a shared
250	connection to the DRMS database.
251
252
253	3. When all modules have finished successfully you can either
254
255	a) tell the DRMS server stop and commit all data generated or
256	modified by the modules to the DRMS database by sending a
257	SIGUSR1 signal to it. In the example above that would mean
258	issuing the command
259
260	akhenaten:~/jsoc> kill -s USR1 20955
261
262	or if an error occurs you can
263
264	b) tell the DRMS server to abort and discard all data generated
265	by the modules by sending it a SIGTERM, SIGQUIT or SIGINT.
266	In the example above that could be done by pressing CTRL-C
267	in the terminal where the server is running or by issuing
268	the command
269
270	akhenaten:~/jsoc> kill -s INT 20955
271
272	It should be safe to kill the server with SIGKILL (kill -9).
273	It will have the same effect as a regular abort except that it
274	leaves a stale entry in DRMS's active session table.
275
276