CACHE94: ADAMS/NETWORK Messages: 17 Entries..

marigold crashed around 12:49 last night! 
 

COSMOS CRASH! 
 
  All serial channels on cosmos have come 
to a halt. This may have something to do 
with the higher data rates on the props. 
I will restart cosmos and see if it happens 
again. 
 

COSMOS died at ca. 2:20 Aug 12 
 
recarnation at 04:21 was succesfull 
 

Cosmos and Marigold crash at 1600gmt 
 
Cosmos and Marigold crash at 1600gmt while Gordon 
logged in. Tasks dataget and recDatel were suspended 
in both adams. 
Adams restarted with mxreset. 
 
 

COSMOS crash! 
 
 Cosmos morning crash occurred at 15:05 GMT. 
 

MARIGOLD crashed again! 
 
 Marigold went down at 17:10 GMT. It appears 
Peter did something with the FSSP at that time. 
Marigold restarted with a data_on command. 
 

COSMOS decided to take a nap @02:05,226 
 
Awoke after a strong mxreset. 
archive_cosmos and covar_cosmos followed  
immediatly afterwards. 
 
 
 

COSMOS DOWN! 
 
 Cosmos died. Tasks suspended in matrix, 
"dataGet","recDatel". 
 

marigold crash 
 
I accidentally crashed marigold by turning off the 
switch on the Peltier cooler (which I didn't know kills 
ALL adam power).  I did this since it appears that the 
cooler is not functioning on marigold.  The external 
fan is always on, the cooler is not very cold (though 
it does appear to be slightly cooler than outside), 
and the internal airflow is quite low.  We've just 
brought marigold back up.  Sorry. 
 
P.S. I've plotted the ADAM internal temperatures for the 
entire experiment.  cosmos ranged from 22-25, while  
marigold ranged from 27-31.  This is a bit warm, but 
(obviously) not too bad. 
 
 

marigold hung at 0156 when I ran a second  
cockpit display for it (on cocklebur). Restarted 
at 0208.  
 
Later discovered note from Steve O. warning 
not to run dual cockpits on marigold, for 
this exact reason. 
 
 

System Down 1537 - 1800 JD239 
 
All adams went down around 1537. Andy 
reported a power glitch at the site 
at that time. 
 
It took a while to get every thing 
running again; we had to cycle the 
power on a couple of prop vanes, 
fix a broken calibration file (it 
needed to have a trailing CR added), 
and fiddle with cosmos serial i/o. 
 
We finally got cosmos going; suspect 
that the serial card does not come 
up properly if you do siostatus 
before it is completely running. 
 

ADAM reboots and siostatus command 
 
It appears that one should not 
send the siostatus command to 
a serial card until the adam 
is running and completely  
operational. (a couple of  
minutes?) 
 

marigold died at jday 240, 02:20 
 
mxreset did not do the job. 
key turn did it at 2:38. 
restarted archiving and covaring for marigold. 
 
 

cosmos just went down; may have been related
to scsi transport problems associated with
a flakey exabyte drive

Cosmos crashed last night and was restarted after the second re-power up `.

Sequence:

turn off adam power for 1 min.
type siostatus goldenrod- check if #records increased since last check
type archive_cosmos
type covar_cosmos

check_aster is useful to see if ingestors all running (4 I think)


The shortcut alternative, which did NOT work last night was
instead of power down type
 mxreset cosmos

followed by the checking and restarting the archive and covar routines
as in power donw option.

we did not have to restart the sonics.

when the crash occurred only the serila instruments failed to send data
  flux tower sonics and AA hygrometer. PVMs were being acquired apparently.

At 3:30 GMT, noticed that cosmos cockpit indicated no data.
check_aster, and siostatus goldenrod indicated that file sizes
were not growing.

Used the key on the ADAM (cosmos) to turn it off, and then back on.
Restarted archive and ingest programs at 3:35 or so.

The SPARC 10 crashed and rebooted at 00:46Z, jd 256.
I was trying to read a floppy disk in ASTER's floppy drive when it crashed.
 /var/adm/messages shows an I/O error on the floppy, and then an immediate
"panic on cpu 0: pc_rele: nrefs & frefs count" message.

I'll use cocklebur's floppy drive in the future.

The whole aster data system had to then be restarted.  I used the opportunity
to install a new EPROM into arnica, the Ironics 3234 board on marigold, and
installed a new vxWorks kernal on $ASTER/vx/config/iv3234.  The EPROM 
and kernal are built from the new BSP from Ironics.