FLATLAND96: ADAM/NETWORK Messages: 32 Entries..

        Adam <--> FreeWave connections
                    
50-cond. ribbon     Serial A    FW
from matrix         9 pin       DB9
                    on adam 


RDin    5               1       2
TDout   3               2       3
GND     13              4       5
RTSout  7               3       7
CTSin   9               5       8

Both freewaves should be configured for 38400 baud, pt to pt.
The ASTER freewave is to be connected to serial port B.

 The Boulder connection went down sometime this morning.
ISS was also down so I would assume it had something to
do with the local netrwork link. System was back up at
13:55 GMT.

  Cosmos is now running via FreeWave link.


Also there are 3 eggs in the nest at the base
of the PAM tripod.

  The krypton is running on cosmos.

  Cosmos was re-booted to correct a channel_config error.

  Notice cosmos has gone down. I tryed an mxreset from base.
This was unsuccessful so I am heading to the corn fields.

  A reset on cosmos got it going again. At this time the reason for
the crash is unknown. Cosmos backup around  20:15 GMT.

  Cosmos went down after Gordon loaded some new code.
Gordon is looking at the code and will modify.

  Cosmos went down for a short period starting at 16:28 GMT.
Based on the console log it was done by Gordon to download
new software code. It was backup at 16:33 GMT.

The latest code was being too verbose with errors messages from the adam
to the console.  The newest sync will only log an error every 100
lost samples, not every 100 blocked writes, as before.

The size of the output sample buffer was increased from 8192 to 16384 bytes.
The write threshold is 4096 bytes, meaning writes of 4096 bytes
are attempted when 4096 bytes are available (or 10 seconds have
passed since the last write).
Samples are not discarded unless the buffer would overflow, at 16384 bytes.

Two new data variables are now available from the adam:
	blocked.cosmos		# of blocked socket writes/second
	lost.cosmos		# of lost samples/second

These can be displayed by cockpit and xstrip.  They are also
averaged in the covar files.

From the attached plot, you can see that network jambs are occuring every 
5 minutes. It is not exactly 5 minutes however. They occur in groups of 6,
with five spikes spaced 4 min 40 seconds apart, and then a longer gap of
about 6 minutes 30 seconds, so that the entire group takes about exactly 
30 minutes.

I made this plot from Splus:
	> fun.plot.prep(c("blocked.cosmos","lost.cosmos"),1996,191)

(Day 191 is Jul 9).

Since they are not spaced exactly 5 minutes apart, it does not apear
to be timed with the pam polling process.  To further exclude pam
as a suspect, John suggested that I shut down eve_rf on cocklebur.
I shut it down from 00:21 to 00:34 on Jul 10.  The blockages
still occured, so pam is off the hook.

Could it be a profiler?

A useful diagnostic is to display "blocked.cosmos" with xstrip, and set
the options->chartwidth to 3000, which results in a grid line every 10
minutes.

Here is a window dump of an xstrip plot of blocked.cosmos and lost.cosmos

Press "Grapics Viewer" to see it.

  Cosmos will be down for awhile due to a large
mechanical rodent crewing up the power cable; ie,
a farmer decided to cut the hay to the north of the
site not knowing about the power cable. The cable
was cut in two places.
  Pam should stay alive since the battery is fully
charged. A backup battery will be put on charge in
case repairs take awhile.

  Cosmos is down for the night. It appears to have a
backplane problem. Gordon has been informed of the problem.
We should have a spare set of ADAM boards sent out. Will
try things in the morning.

  The matrix card had to be replaced. Cosmos is
looking good now.

  The Datel voltage source is attached to analog channel 0 on
cosmos. It is being used to check out the analog channels for the
ozone system.

  Cosmos went down at 14:34 GMT. mxreset would not
get it going again and cosmos does not respond to
ping. Will do a hard reset.

We also reset EVE to try to reset the Tc spiking.

  Cosmos was shutdown at 17:55 GMT to see if the
FreeWave communication is the source of profiler
interference.

  Cosmos is back up. During the bootup period there was no
noise seen on the profiler. Will monitor the profiler.

Cosmos died about 0745 last night with "ingest socket write to aster.8000"
errors.  In the confusion of introducing Matt, Greg arriving, and Lou calling,
I neglected to check_aster or look at the log file to isolate the problem.
We rebooted it about 40 minutes ago and it seems to be running again.

We came in this morning to find that cosmos had crashed again, with network 
error messages in the "asterlog" file which were similar to those from 
previous crashes.  I rebooted (with the key, since it didn't ping), and
it crashed again in about 2 hours.  We have taken the following steps:
1. "niced" the pam ppf2netcdf process, which may give the network handling
higher relative priority
2. removed a lot of the error messages in "sync.cc" to avoid the network
being "message bombed" when a crash starts.  We have noticed that once
cosmos gets one sample that doesn't get through that a crash is imminent.
3. We looked at the statistics on the freewave in the trailer which said
that only 57% of messages were received.  However, we don't trust this
number, since Matt got 9% when he repeated it, and 0% on the pam modem.

After discussions with Steve S., there are many more options - some harder
than others - depending on what the source of these crashes is.  At this
point, we don't know if the problem is aster being overloaded, the RF link
not being reliable (or being interfered with) or a power glitch (our only
guess as to why it is worse now that Greg is here).

P.S. it crashed again while I was typing this!

I see that cosmos ingest stopped again; cosmos did not produce
the EPIPE errors, and did not "self mxreset". I have done an
mxreset to try to bring it back.

The cosmos ingestor ran happily all night; about 14 hours so
far. Who knows what is causing it to intermittantly timeout
on the polling?

This time, cosmos self-recovered.  Wonder why it didn't last night?

I noticed, by grepping on
EOF in asterlog, that the cosmos
problems are mostly between 15Z and 0Z.

It has successfully reset itself several
times this afternoon., so it looks like
the auto mxreset is working okay
right now.

cosmos ran flawlessly last night, and then crashed this morning (while I
was at the site).  Apparently, the antenna switch yesterday didn't solve
the problem.  It rebooted by itself as usual.

...but it lost a few samples just as we walked in now.

We've just added Tony's ozone instrument (conc.O3.secl.3m) and turned on
an additional channel for the CU instrument if it ever comes.  I just
rebooted cosmos to make the change.  (mxreset killed it, so the key was
necessary).

Note that the tower was down for about 2 hours (5pm to 7pm local) while
we did that.  This was mostly due to prop cabling problems - see next
comment.

Cosmos has been up and down several times today.  Latest outage was at 21:30.
Had to go out and reset the ADAM.  Other outages rebooted automatically.

On Monday, August 5th these changes were made to the aster system:

ingest: increased the no-activity timeout from 2 minutes to
	 5 minutes	
sync code on matrix: increased the sample buffer from 16*4096 to
			24 * 4096 bytes.

Ingest was rebuilt, installed and restarted.  The matrix code
was rebuilt.  Since the adam was conveniently crashing every hour or so
I just let it load the new code and spawn a new ingest on its next reboot,
which happened at 21:36 on aug 5th.

It has been up since then, so perhaps these changes helped.

cosmos was shut down at aprox. 14:25, freewaves are going down iminately,
everything is close to shutdown. pam 1 will continue to xmit via goes until
station teardown.