ViewVC Help
View File | Revision Log | Show Annotations | Root Listing
Revision: 1.11
Committed: Fri Nov 4 00:57:19 2011 UTC (11 years, 10 months ago) by rick
Branch: MAIN
CVS Tags: NetDRMS_Ver_6-4, NetDRMS_Ver_8-0, NetDRMS_Ver_6-2, NetDRMS_Ver_6-3, NetDRMS_Ver_6-0, NetDRMS_Ver_6-1, Ver_6-0, Ver_6-1, Ver_6-2, Ver_6-3, Ver_6-4, NetDRMS_Ver_8-8, Ver_8-5, NetDRMS_Ver_7-0, NetDRMS_Ver_8-1, Ver_7-0, Ver_LATEST, NetDRMS_Ver_LATEST, NetDRMS_Ver_8-12, NetDRMS_Ver_8-10, NetDRMS_Ver_8-11, NetDRMS_Ver_9-1, NetDRMS_Ver_9-0, NetDRMS_Ver_9-3, NetDRMS_Ver_9-2, NetDRMS_Ver_9-5, NetDRMS_Ver_9-4, NetDRMS_Ver_8-2, NetDRMS_Ver_8-3, NetDRMS_Ver_9-41, Ver_9-41, Ver_DRMSLATEST, NetDRMS_Ver_8-4, NetDRMS_Ver_8-5, NetDRMS_Ver_8-6, Ver_8-8, NetDRMS_Ver_8-7, Ver_8-2, Ver_9-3, Ver_8-0, Ver_8-1, Ver_8-6, Ver_8-7, Ver_8-4, Ver_8-11, Ver_7-1, Ver_9-1, Ver_8-3, NetDRMS_Ver_7-1, Ver_9-5, Ver_9-4, Ver_8-10, Ver_9-2, Ver_8-12, Ver_9-0, HEAD
Changes since 1.10: +11 -0 lines
Log Message:
added instructions for building debug versions of modules

File Contents

# Content
1 This directory contains a "cookbook" collection of sample DRMS modules,
2 along with a simplified make system that can be used to build comparable
3 applications modules outside of the NetDRMS/JSOC distribution trees.
6 Table of Contents
8 Directories
9 CVS/ CVS repository information
10 Makevars/ files containing system/architecture-specific make variable
11 definitions for use in a generic Makefile
13 Regular Program
14 smpl_00.c does (almost) nothing at all - like hello_world, but exhibits
15 use of the command-line parsing used by JSOC module drivers
17 DRMS Modules
18 smpl_01.c same as smpl_00, but written as a module
19 smpl_02.c echoes its arguments, of a wide variety of types
20 smpl_03.c prints a list of the data series known to DRMS - a simple
21 version of show_series
22 smpl_04.c prints the number of unique records in a selected data series
23 and the number of data segments per record - a simple version of
24 show_info
25 smpl_05.c inserts data records from noaa_ar.dat into series created with
26 noaa_ar.jsd
27 smpl_06.c updates data records in series built with smpl_05 with new
28 keyword values
29 smpl_07.c inserts image data in record segments in a data series created
30 with images.jsd
31 smpl_08.c calculates and reports primary statistics of record segments
32 such as those generated with smpl_07
34 Other
35 Makefile The generic Makefile
36 ReadMe (this file)
37 images.jsd A JSD file that can be used to build a (keywordless) DRMS data
38 series for storing data segments
39 noaa_ar.dat A summary listing of data from daily NOAA Active Region reports
40 for regions with spots from 1996 onward; can be used as input for a
41 DRMS data series
42 noaa_ar.jsd A JSD file that can be used to build a (segmentless) DRMS data
43 series for the data from noaa_ar.dat
45 ********************************************************************************
47 Recipe 0: How to write and build a program in this directory, using features
48 of the DRMS library
50 The program smpl_00.c is technically not a DRMS module, just a standard C
51 program similar to the familiar "Hello, world" one. The only real difference
52 is that it is linked against the DRMS API to use its command-line argument
53 parsing features. The message that the program prints when executed can be
54 changed from the default, "done", by running the command:
56 smpl_00 print=something
57 or
58 smpl_00 print= "Hello, world!"
60 (White space after the '=' sign is optional; quotes are necessary if the
61 argument string has embedded white space.)
63 The program can be built, like all of the modules in this directory as well,
64 by typing
66 make smpl_00
68 (alternatively, you can simply type make and all the modules in the target
69 list MODS in the Makefile will be compiled.) The Makefile is generic: it is
70 designed to work on different hardware and operating system platforms by
71 accessing an appropriate set of make variables definitions from the various
72 files in the subdirectory makevars. To add a new module that does not require
73 linking with any additional libraries beyond those required for DRMS, all you
74 need to do (besides writing the module!) is to append its name (without the
75 filename extension) to the list of MODS in the Makefile.
77 Note that the Makefile requires the external definition of the environment
78 variable JPLAT to the appropriate name for your build platform. If, when you
79 run the make, you get a message like:
81 Error: no appropriate Makevars; is $JPLAT defined? won't compile...
83 Either you have not defined the system-appropriate environment variable,
84 or there is no file Makevars_$ in the Makevars directory. It should
85 be straightforward to create an appropriate one by replicating one of the
86 existing examples and substituting appropriate command paths. Likewise,
87 if you get a message like:
89 Makefile:1: Makevars/ No such file or directory
90 make: *** No rule to make target `Makevars/'. Stop.
92 it means that you have defined the environment variable JPLAT (as XXX), but
93 there is no corresponding in the directory. Again, you can
94 create the appropriate one by replicating one of the existing examples and
95 substituting appropriate command paths.
97 (Alternatively, if you will only be building and running on one platform
98 ever, you could just add a platform definition line like:
100 JPLAT = linux_x86_64
102 or whatever before the first line in the Makefile.)
104 The architecture-dependent Makevars_*.mk files define absolute paths to
105 essential commands, especially compilers. If you get messages like:
107 make: /usr/local/bin/icc: Command not found
109 You must suitably redefine the relevant Make variables (e.g. CC or ICC in
110 this example) in the file being included. If you have to change from icc
111 to gcc, be sure to change the appropriate CFLAGS and LDFLAGS variables as
112 well, removing the -xW option.
114 The Makefile also requires external definition of the environment variable
115 DRMS to be the root of your NetDRMS distribution as described in the
116 installation instructions. If you get an error message like:
118 error: could not open source file "cmdparams.h"
120 it is probably because you have not correctly set the DRMS environment variable.
121 (Again, if you wish you could define DRMS in the Makefile rather than as an
122 environment variable.)
124 The Makefile assumes that all third-party libraries, in particular those for
125 cfitsio and postgres that are essential for DRMS, reside in one of the default
126 search locations, /usr/lib or /usr/local/lib. If they are elsewehere, then
127 you will need to add the base path to the LIBD definition line in the Makefile.
129 In order to build a debug version of any of the modules in the project (with
130 symbols and without optimization), you can set the environment variable JPLAT
131 to debug. The debug "platform" variables are identical to those in the
132 linux_x86_64 file, except for the C compiler options. Since it is very likely
133 in debugging that you may need symbols in the DRMS library as well, you should
134 build a debug platform version of that library by running (in the DRMS root
135 directory):
137 setenv JSOC_DEBUG 1
138 make MACH=debug
140 ********************************************************************************
142 Recipe 1: How to write and build a minimal DRMS module
144 The program smpl_01.c is a true DRMS module; it exhibits the minimum structure
145 necessary for a C program that will be able to interact with the DRMS API to
146 fetch and store date in the DRMS and SUMS. This module in fact does neither,
147 as it is identical in functionality to smpl_00.c; but it nevertheless opens
148 the required connection to the DRMS, and will fail if that is not possible for
149 some reason, such as the database server being unavailable or the user lacking
150 proper authorization to access it. It can therefore serve as a useful test for
151 minimal connectivity to DRMS.
153 Note that this program does not have a main() entry, it is instead called
154 DoIt(). All DRMS modules are called DoIt() and link against another main()
155 program in the DRMS library that opens a connection to DRMS and SUMS for
156 them, calls DoIt(), and then closes the connection, saving or discarding
157 the results according to the return status of the module.
159 In addition to the "print" argument, this module also features two "flag"
160 arguments. If you run
161 smpl_01 -v
162 it will print out a little more information about what it is up to (this is
163 a common flag), and if you run
164 smpl_01 -a
165 it will force an abort, that is, returns a non-zero value to the main program.
166 You can also run
167 smpl_01 -av
168 or smpl_01 -a -v
170 You can get a list of the arguments, and default values (if any), by typing
171 smpl_01 -H
172 This is a feature of all DRMS modules, as it is handled by the (hidden) main
173 program. Likewise, the command
174 smpl_01 -V
175 results in some extra verbose information from the main program itself before
176 and after the module is called.
178 ********************************************************************************
180 Recipe 2: Command-line Argument Parsing
182 Module smpl_02 also has no interaction with the DRMS database. It shows the
183 full range of command-line argument types and how they are to be processed
184 in the module. Type
185 smpl_02 -H
186 to see the full argument list and then fool around with various values for the
187 arguments. Most of the argument types should be self explanatory. ARG_INTS and
188 ARG_FLOATS can be used to specify arrays of arbitrary length (including zero)
189 of integers or floats, comma separated and enclosed (if there are more than one)
190 within matching pairs of either brackets, braces, or parentheses. ARG_NUME is
191 used for an enumerated list of possible strings, returning the order of the
192 selected string in the enumerated list, similar to an enum declarator in C.
194 ********************************************************************************
196 Recipe 3: Communicating with the DRMS Database - show_series
198 Module smpl_03 shows how direct communication with the database can be done
199 through a module. It is basically a simple version of the show_series
200 application. The module still bypasses most of the API. Normally you should
201 not need to communicate with the database at such a low level in a module, but
202 this shows how to do it if you are familiar with SQL and need more power and
203 flexibility than is provided by the record management API. You can use this
204 module, like show_series, to see what data series are already in your DRMS
205 database.
207 ********************************************************************************
209 Recipe 4: Beginning to Use the API - show_info
211 Module smpl_04 is the first one to actually use the DRMS record management
212 API. It is a (very) simplified version of the show_info application. You can
213 use this module to begin to explore the various data series in your DRMS.
215 ********************************************************************************
217 Recipe 5: Adding Records to a Data Series
219 Up to this point, all of the cookbook modules operated on data series that
220 already existed. At some point, you need to be able to create and populate
221 data series yourself. Module smpl_05 is an example of a module that can be
222 used to add (or update) records in a data series, in this case one containing
223 records correspnding to the daily NOAA reports of individual solar active
224 regions with spots. The data from those reports has been assembled in a
225 simple ASCII table text form in the file noaa_ar.dat. The module smpl_05
226 reads data from that table and inserts the appropriate records in a data
227 data series. In order to exercise it, you will first need to create a data
228 series with the appropriate structure that you can write to. To do so, run
229 create_series noaa_ar.jsd
230 You will probably have to change the Seriesname from drms.NOAA_ar to
231 something in a different namespace from drms; each user should have a
232 personal namespace. Then you can run the module with the appropriate value
233 for the "ds" argument.
235 Any time you want to remove this series (to try out different series
236 modifications in the JSD for example), you can delete it by typing
237 delete_series drms.NOAA_ar
238 (or whatever its series name is) and answering yes to the safeguard prompts.
239 If you do not change the series name in the JSD, you can also force its
240 deletion and recreation by typing
241 create_series -f noaa_ar.jsd
242 If you give it a new series name of course it will not remove the old one.
244 Once you have created the data series, you can run
245 smpl_05
246 The default value for the ds parameter is drms.noaa_ar, so you will have to
247 run the module with a different series name for the "ds" argument the series
248 you created has a diffeent name. The default value for the "data" argument
249 is noaa_ar.dat, a file included in the cookbook.
251 ********************************************************************************
253 Recipe 6: Modifying Records in a Data Series
255 If you inspect the last few records in the data series just created, for
256 example with
257 show_info -a ds= drms.noaa_ar n= -5
258 you will discover that the penultimate record, that for the observation of
259 AR 11024 on 2009.07.10, has a very strange and probably erroneous value
260 for its latitude, -9 deg, when the latitude on the preceding and following
261 days was -25 deg. This is the result of what was presumably a typographical
262 error in the NOAA/USAF report for that day (SRS Number 191 Issued at 0030Z
263 on 10 Jul 2009). Module smpl_06 is an example of one that could be used to
264 correct or update the values of selected records in a data series. It shows
265 how runtime parameters (and internally computed variables) can be used in
266 place of record set specifiers to restrict the recordset, and how key values
267 in selected records can be "changed". In order to fix the relevant record,
268 you could run either
269 smpl_06 date= 2009.07.10 ar= 11024 key= lat value= -25
270 or
271 smpl_06 ds= "drms.noaa_ar[2009.07.10][11024]" key= lat value= -25
272 and then reinspect the last few records. The record in question should have
273 been updated with a new value for its latitude. In reality, the DRMS API
274 does not allow for actual value changes to records in the database, which is
275 why the function drms_clone_record() is used in this update module. If you
276 type
277 show_info -ar ds= "drms.noaa_ar[\!Region=11024\!]"
278 you will see that there are really two entries in the database with the same
279 values for the two prime keys Date=2009.07.10 and Region=11024. The one with
280 the higher recnum is the only one that will show up in ordinary queries.
282 ********************************************************************************
284 Recipe 7: Creating Records with Data Segments
286 The data series worked with in the previous two examples had no associated
287 data segments, repositories for bulk data such as images or other data arrays.
288 The JSD file images.jsd describes a series with the opposite structure. It
289 contains no keywords at all in the database (except for the hidden keywords
290 like recnum, so that all records are unique). However, it does allow for a
291 segment, which can be any 2-dimensional array of short ints (which can
292 represent scaled floats), stored internally as Rice-compressed FITS binary
293 tables. As in the preparations for Recipe 5, run
294 create_series images.jsd
296 You can then run smpl_07 to create records (one at a time) in this series.
297 Each will have a data segment of specified size and shape, with the values
298 filled randomly using a somewhat Byzantine procedure to create a semblance
299 of large-scale structure. You might want to fool around with different values
300 of the parameters dist and seed. In order to view the data segments, run for
301 example
302 show_info -pq ds= drms.images"[]"
303 and then run a viewer, e.g. ds9, on the fits files in the displayed SUMS
304 directories.
306 Note a few interesting features of the JSD file you used to create this data
307 series. First the Archive flag is a negative number. Normally the Archive flag
308 is set to 1 if the data segments are to be archived to tape before they age
309 off the SUMS disks (in 10 days in this case), 0 otherwise. However, if the
310 data segments are allowed to disappear without having been archived to tape,
311 the data records in the DRMS will still remain. Sometimes that is useful, but
312 in this case, with a series that contains no ancillary data at all, that would
313 be pointless. The value of -1 will force the DRMS to actually remove any records
314 when their corresponding data segments age off the disk.
316 Also, note that the module automatically created FITS files named v.fits
317 with binary table row compression. That is because of the parameters specified
318 for the series Segment "Data" in the JSD file. You might want to experiment
319 with different values for the various fields in the Segment descriptor, "name",
320 scope, "type", "naxis" (which would cause the module to fail if changed),
321 "axis_n" (which must be specified if scope is variable rather than vardim and
322 might cause the module to fail if inappropriate values of parameters are
323 chosen, "protocol", the compression and scaling parameters. In order to
324 conduct such experiments you must either create a new series with a different
325 series name by specifying a different value for Seriesname in the JSD, or, if
326 you wish to keep the series name, recreate the series afresh, since its
327 essential structure is being changed. To do the latter, run:
328 delete_series drms.images
329 (or whatever name you gave the series) and then
330 create_series images.jsd
331 Alternatively, you can just run:
332 create_series -f images.jsd
333 which will force deletion of the existing series if it has the same Seriesname.
335 ********************************************************************************
337 Recipe 8: Analyzing Data in Record Data Segments
339 Now that you have created a data series with records containing "real" data,
340 you can begin to explore the functions that fetch data from SUMS segments into
341 memory for processing.
343 smpl_08.c is a simple program that calculates and prints the primary statistics
344 (count, mean, standard deviation) of each segment (or a selected segment) of
345 a selected data set. It takes as arguments the input data set specification and
346 optionally a segment name as well. The default record set specified is the
347 last record inserted into the data series by smpl_07. (If you gave that series
348 a different name from drms.images you will either have to change the default
349 value for the "ds" argument in smpl_08.c or provide the correct series name
350 as the value of the argument ds when you run smpl_08.