There are actually two parts to the archiver, the controller GUI which the operator uses and the process which does the actual archiving to disk. This section first lists the running set of archiver requirements, then highlights a few use cases to provide context for some of the requirements. Finally, an overview of the design is laid out.
Here follows the current basic list of requirements for the new radar archiver:
Reliability and recoverability, obviously, and capable of throughput which exceeds the radar data rate.
The archiver must have a simple GUI controller for the operator to use, and the GUI should be available on the operator display even if the archiver itself is running on another host.
Support redundant archiving to files on (removable) disks. Currently tape is being used, but only one tape is written at a time during an ELDORA flight. Instead, migrate to redundant, hot-swappable disk drives and support multiple simultaneous (parallel) archiving processes, on one host or multiple hosts.
In writing to files on disk, automatically segment the archive data stream into a sequence of files with conventional names. Eventually the file name pattern and segment size could be configurable, and segments could be triggered interactively by the operator.
All of the archiving components need to provide feedback and status info to the operator, such as error conditions or the need to swap a disk or tape.
Produce a standard archive format which can be shared among radars as much as possible and which promotes implementation sharing between downstream radar processing tools. For the moment this will remain the DORADE format as currently archived by ELDORA. Note that this requirement need not be a function of the archiver so much as a characteristic of the opaque stream of bytes sent to the archiver.
There is also a collection of ideas for future features:
Index streams: parallel to the archive data stream, keep a reduced, online index. For example, take snapshots of the data at regular intervals, possilby even allowing the scientist to flag certain times. The index could be browsed more quickly than the full archive stream after a flight or IOP, and it could provide rudimentary 'back-looks' during a flight.
The Operator is the person filling the role of operator for the radar. The operator typically is responsible for routine supervision and control of the radar, including watching for the correct flow of data from generation through to archive recording. Maintenance tasks might include loading, labelling, and replacing archive media, and troubleshooting or restarting various components in the system if problems arise. The operator needs an intuitive and thorough display of the current status of all of the components as well as easy access to the control interfaces for those components.
It is time to start an archiving session. The operator verifies that the archiving service is running, otherwise starts it up. The operator tells the archiver to prepare to start a new archive session, perhaps including the loading of new media. The archiver may have information it needs to start a new session, like the operator name and an IOP or flight number to which the session belongs. Some or all of the configuration information may be initialized from a cache of information from previous sessions, from defaults, or from some master configuration database. The archiver should present its configuration information to the operator, and the operator should verify it. Then the operator can trigger the start of archiving.
The archiver needs to be told when to start archiving, where to archive to, perhaps how long to archive, perhaps exactly what of the data stream to archive, and where to report errors. Since there may be multiple data sources on the radar network, the archiver needs to know which of the sources to archive. It also needs basic project documentation like the project name, the name of the operator, a flight number if applicable, other contextual information. The operator needs to supply that information, and it needs to be stored somewhere from which it can be retrieved by the archiver.
This suggests a list of services which need to be available on the network:
Project Information Service
Error Reporting Service (or more likely a basic Log Service)
Archive Configuration Service
During normal operations, the archiver continuously listens on the network for data, assembles packets into logical blocks and performs some sort of basic verification of completeness, then writes the block to a file on the removable disk. Periodically, the archiver updates its status, where the status includes the number of bytes written, the rate at which bytes have received and written to disk since the last update, any network errors such as retransmission requests, and finally the amount of space left on the disk. The updated status must be published or somehow pushed to the archiver's controllers and views so they can update their display.
The archiver controller should keep a history of archiver status and log messages, so the operator can refer back to them easily when trying to diagnose the timing of a problem. Better yet, the Log Service would keep a record of log messages from all components and allow the operator to view the unified history log interactively.
If the archiver encounters an error, it must send notification of that error to the operator display, along with information on possible causes and remedies. If the problem affects the archived data stream, the archiver should record a notice of the problem in the archive file and continue archiving from there if possible.
When an archiving session has completed, the operator selects the stop button on the archiver control window. The archiver receives the stop message, stops archiving to disk, and closes the archive file. However, it does not exit but instead remains active waiting for another restart.
The archiver is a typical radar data consumer, in fact it is not even very specific to radar data. There are a few logical operations which it must support, such as starting and stopping, and it must provide a notification of its status and especially any error conditions. Thus the archiving service can be distinguished from the GUI controller, and the GUI controller conceptually is just one possible recipient of archiver status notifications. Figure 3 contains the class diagram for this design approach.
The class diagram does not imply that all of the classes will be part of the same process. Instead, it is possible and even desirable for the controller process to run separately from the archiver, such as on the operator host. Thus this design also includes a NameService, a single global service by which references to other services can be looked up by their name.
The ArchiverService interface encapsulates the logical methods of an archiver, such as stopping and starting. The ArchiverConfig object encapsulates the configuration information which the archiver needs to operate. When an archiver service boots up, it binds itself to a NameService with a known name. Given this name, an ArchiveController can lookup() the corresponding archiver service and with the returned reference begin configuring and controlling the archvier.
The ArchiverService interface and the NameService effectively decouple much of the archiver service implementation from any potential clients. In fact, there may be multiple implementations of the ArchiverService interface, such as one which records to tape in addition to the one which records to disk. The controller could control either the disk or the tape archiver without needing any changes, not even recompilation. This decoupling enables recovery from a crashed controller. The archiver service has no notion of who has references to it or who is controlling it, so restarting a crash controller is transparent to the actual archiver, which is obviously the more critical service.
Also, part of controller transparency is due to the publishing of archiver status messages to an event channel. The event channel effectively decouples the archiver from any specifics about who or how many are receiving its status messages, if anyone. A monitor process can sit on the net and listen to lots of event streams and log them all in one place, or it can pop-up warnings to the operator if any of the monitored event channels show errors.
Figure 4 depicts the likely deployment diagram for the ELDORA archiver components.
One of the intended advantages to the distributed archiver design is that it can be easily mapped onto existing middleware implementations, especially CORBA. The CORBA standard specifies not only the mechanics and protocols for a distributed object-oriented system, it also specifies common object services (COS) like event channels and a name service. So most CORBA implementations also contain implementations for those services. This includes the TAO ORB (The ACE ORB), based upon the ACE framework. (See the section called “ACE”.)
The archiver objects and interfaces can defined with the CORBA Interface Definition Language (IDL), and from those definitions an IDL compiler generates C++ stubs and skeletons. Documentation for the IDL defintions can be found here, generated with Doxygen: Archiver IDL documentation.
A prototype for the controller GUI has been created with the help of the Qt designer application, an interactive GUI editor. Below are examples of the controller window (Figure 5) and the configuration dialog (Figure 6).
