FLATLAND96: Site ASTER Messages, 10 Entries..

  Krypton #1133 was calibrated in the Thunder before
being shipped to Flatland. Below are the calibration 
coefficients for the sensor. The pathlength was measured
to be 1.458 cm.

Vapor range	 V0	  Kw
  (g/m^3)	(mv)	(m^3/gcm)
2 - 19		2854	-0.142
2 - 9		2952	-0.145
8-19		2573	-0.137

The values for the 2-19 vapor range have been entered
into $ASTER/cal_files/krypton1133

        Adam <--> FreeWave connections
                    
50-cond. ribbon     Serial A    FW
from matrix         9 pin       DB9
                    on adam 


RDin    5               1       2
TDout   3               2       3
GND     13              4       5
RTSout  7               3       7
CTSin   9               5       8

Both freewaves should be configured for 38400 baud, pt to pt.
The ASTER freewave is to be connected to serial port B.

  Cosmos is now running via FreeWave link.


Also there are 3 eggs in the nest at the base
of the PAM tripod.

Installed new software on the Matrix card. Writes to the ingest output
socket in the sync task are now non-blocking, so the adam does not crash if
the network connection is jambed.

Samples can be lost when the network is jambed, however.  To see the total
count of lost samples, do

	sshow cosmos 0

You'll see output like so:


c:000 cosmos mem:131572 nent:153 nlost:0 nlate:0 pS:0 pD:0 dT:5410 msec, blocked:0, lost:0, writerr:0

The last three parameters (blocked, lost, writeerr) result from network jambs.
They are totals since the last adam reboot.

If the network is jambed for more than two minutes, then ingest will
timeout and shutdown, in which case the adam will need to be rebooted.

The latest code was being too verbose with errors messages from the adam
to the console.  The newest sync will only log an error every 100
lost samples, not every 100 blocked writes, as before.

The size of the output sample buffer was increased from 8192 to 16384 bytes.
The write threshold is 4096 bytes, meaning writes of 4096 bytes
are attempted when 4096 bytes are available (or 10 seconds have
passed since the last write).
Samples are not discarded unless the buffer would overflow, at 16384 bytes.

Two new data variables are now available from the adam:
	blocked.cosmos		# of blocked socket writes/second
	lost.cosmos		# of lost samples/second

These can be displayed by cockpit and xstrip.  They are also
averaged in the covar files.

From the attached plot, you can see that network jambs are occuring every 
5 minutes. It is not exactly 5 minutes however. They occur in groups of 6,
with five spikes spaced 4 min 40 seconds apart, and then a longer gap of
about 6 minutes 30 seconds, so that the entire group takes about exactly 
30 minutes.

I made this plot from Splus:
	> fun.plot.prep(c("blocked.cosmos","lost.cosmos"),1996,191)

(Day 191 is Jul 9).

Since they are not spaced exactly 5 minutes apart, it does not apear
to be timed with the pam polling process.  To further exclude pam
as a suspect, John suggested that I shut down eve_rf on cocklebur.
I shut it down from 00:21 to 00:34 on Jul 10.  The blockages
still occured, so pam is off the hook.

Could it be a profiler?

A useful diagnostic is to display "blocked.cosmos" with xstrip, and set
the options->chartwidth to 3000, which results in a grid line every 10
minutes.

Here is a window dump of an xstrip plot of blocked.cosmos and lost.cosmos

Press "Grapics Viewer" to see it.

We forgot to note that we rebooted aster itself yesterday (and thus
also cosmos) because the serial port was "locked" and wouldn't let
the dp process connect to the outside world over the modem.
(We had tried restarting dp, but even kermit couldn't connect to the
port.)  Rebooting fixed the problem.

Cosmos has been up and down several times today.  Latest outage was at 21:30.
Had to go out and reset the ADAM.  Other outages rebooted automatically.

On Monday, August 5th these changes were made to the aster system:

ingest: increased the no-activity timeout from 2 minutes to
	 5 minutes	
sync code on matrix: increased the sample buffer from 16*4096 to
			24 * 4096 bytes.

Ingest was rebuilt, installed and restarted.  The matrix code
was rebuilt.  Since the adam was conveniently crashing every hour or so
I just let it load the new code and spawn a new ingest on its next reboot,
which happened at 21:36 on aug 5th.

It has been up since then, so perhaps these changes helped.