SOCRATES chat log RF01

[2018/01/15 21:57:35 UTC] <Pei-gv> Set up Mission coordinator display.
[2018/01/15 21:57:50 UTC] <Pei-gv> Limit the remote HawkEye display to one for this flight
[2018/01/15 21:58:05 UTC] <Pei-gv> cycle power on CMU
[2018/01/15 21:58:51 UTC] <Pei-gv> CMU communication up
[2018/01/15 21:59:23 UTC] <Pei-gv> APS pressure look good. 1415 PSI, 25.3 psi
[2018/01/15 21:59:35 UTC] <Pei-gv> 28vps 16.2c
[2018/01/15 21:59:49 UTC] <Pei-gv> NSCAL
[2018/01/15 22:29:28 UTC] <ScottEllis-Boulder> And are you going to put in the pointing correction numbers I sent Saturday, or keep them the same?
[2018/01/15 22:30:13 UTC] <Pei-gv> We meant to have that discussion
[2018/01/15 22:30:28 UTC] <Pei-gv> Let me see if I can msg ChrisBurghart-phone
[2018/01/15 22:30:47 UTC] <ScottEllis-Boulder> Ok
[2018/01/15 22:31:10 UTC] <Pei-gv> ScottEllis-Boulder, all staff are off the plane at this point
[2018/01/15 22:31:27 UTC] <Pei-gv> FIL ON
[2018/01/15 22:31:41 UTC] <Pei-gv> bench test
[2018/01/15 22:33:30 UTC] <Pei-gv> ChrisBurghart-phone, notice the Rsync is syncing data from 04hr now. Is this normal?
[2018/01/15 22:34:19 UTC] <Pei-gv> ChrisBurghart-phone, is it simple to change the BeamCorr numbers?
[2018/01/15 22:38:13 UTC] <Pei-gv> ChrisBurghart-phone, do we still need to make sure the images land in catalog at this point? Or this is established?
[2018/01/15 22:38:56 UTC] <Pei-gv> ScottEllis-Boulder, being by myself on board, I am not entirely comfortable to change the numbers..
[2018/01/15 22:39:07 UTC] <ScottEllis-Boulder> The latest images I see on the catalog are from 1/8
[2018/01/15 22:39:09 UTC] <Pei-gv> I would vote for using these numbers for now.
[2018/01/15 22:39:21 UTC] <Pei-gv> ScottEllis-Boulder, Thanks for checking!
[2018/01/15 22:39:30 UTC] <Pei-gv> door closed
[2018/01/15 22:39:32 UTC] <ScottEllis-Boulder> Pei-gv, No problem. Let's keep them the same, for this flight
[2018/01/15 22:39:46 UTC] <Pei-gv> ScottEllis-Boulder, copy that
[2018/01/15 22:40:01 UTC] <ChrisBurghart-phone> Pei-ghost, I restored the sending of catalog images, so they should just go. I'll check.
[2018/01/15 22:40:42 UTC] <Pei-gv> ChrisBurghart-phone, thanks! no hurry on this check
[2018/01/15 22:41:46 UTC] <Pei-gv> FireFly Warning went away
[2018/01/15 22:46:51 UTC] <Pei-gv> taxiing
[2018/01/15 22:47:03 UTC] <Pei-gv> HV ON
[2018/01/15 22:47:09 UTC] <Pei-gv> tx-v
[2018/01/15 22:47:21 UTC] <Pei-gv> switch off
[2018/01/15 22:47:32 UTC] <Pei-gv> seeing clouds. all look good!
[2018/01/15 22:54:22 UTC] <Pei-gv> take off
[2018/01/15 22:56:23 UTC] <Pei-gv> nadir pointing
[2018/01/15 22:58:15 UTC] <ScottEllis-Boulder> How does the surface Vr look?
[2018/01/15 23:03:21 UTC] <Pei-gv> ScottEllis-Boulder, slightly positive
[2018/01/15 23:03:45 UTC] <Pei-gv> In case you don't know yet. We will be doing the up/down wind legs on our return close to Hobart.
[2018/01/15 23:04:28 UTC] <ScottEllis-Boulder> Thanks, that should be good for calibrating the pointing
[2018/01/15 23:08:56 UTC] <ScottEllis-Boulder> ChrisBurghart-phone, I am not seeing images on the catalog yet. Are you seeing them locally?
[2018/01/15 23:09:02 UTC] <Pei-gv> Spectracom alarm
[2018/01/15 23:09:19 UTC] <Pei-gv> check gaps around 23:08 time period
[2018/01/15 23:09:35 UTC] <Pei-gv> don't think it's directly related to Spectracom issue
[2018/01/15 23:09:57 UTC] <Pei-gv> Notice the system is a bit sluggish. typing on Chat has a bit delay from time to time
[2018/01/15 23:10:34 UTC] <Pei-gv> 23:10:00 none transmission for ~20 second
[2018/01/15 23:11:10 UTC] <Pei-gv> ScottEllis-Boulder, velociy is consistently slightly positive. The sea looks real calm
[2018/01/15 23:12:02 UTC] <Pei-gv> Wrong wind direction
[2018/01/15 23:12:23 UTC] <Pei-gv> another none transmission 23:10:30
[2018/01/15 23:13:23 UTC] <Pei-gv> All clear in radar range.
[2018/01/15 23:14:26 UTC] <Pei-gv> Spectracom Alarm-major 1pps not in spec
[2018/01/15 23:22:13 UTC] <Pei-gv> 23:21:30 transmission gap
[2018/01/15 23:23:43 UTC] <Pei-gv> 23:23:05 5 second gap from no transmission
[2018/01/15 23:26:46 UTC] <Pei-gv> ChrisBurghart-phone, Heading_deg on our Moments-10Hz data: difference with IWG1 is 360 deg.
[2018/01/15 23:27:12 UTC] <Pei-gv> Shouldn't it be showing more or less the same numbers? I understand they are essentially the same numbers...
[2018/01/15 23:27:26 UTC] <Pei-gv> 23:26:35 transmission gap
[2018/01/15 23:28:16 UTC] <Pei-gv> Track_deg on Moments-10Hz values 0.000 when IWG1 showes numbers.
[2018/01/15 23:31:58 UTC] <Pei-gv> ScottEllis-Boulder, Looking from the AcGeorefCompare App, I am seeing velocity consistently around 0.5 m/s
[2018/01/15 23:32:21 UTC] <Pei-gv> ChrisBurghart-phone, we may have a problem on acserver hsrl images.
[2018/01/15 23:32:31 UTC] <Pei-gv> I think the hcr images were copied on there.
[2018/01/15 23:33:09 UTC] <ScottEllis-Boulder> Pei-gv, that is consistent with the ferry flight for the same drift values.
[2018/01/15 23:36:27 UTC] <Pei-gv> clouds at ~3kft heights about 5 minutes ago. Shows up on HSRL but not HCR. Dont think we have the sensitivity to see it at that range.
[2018/01/15 23:36:45 UTC] <Pei-gv> currently at ~21kft altitude
[2018/01/15 23:38:51 UTC] <ScottEllis-Boulder> Pei-gv, I think you are right, those clouds look super thin on HSRL and the images from the GV camera show only tiny clouds.
[2018/01/15 23:44:53 UTC] <Pei-gv> spectracom Synchonise error!
[2018/01/15 23:45:14 UTC] <Pei-gv> Now I am getting 125 MHz PLO lock error
[2018/01/15 23:46:18 UTC] <Pei-gv> Profiling legs start at 59S. I want to see if I can deal with this Spectracom problem.
[2018/01/15 23:46:23 UTC] <Pei-gv> Cycle power now
[2018/01/15 23:47:19 UTC] <Pei-gv> ScottEllis-Boulder, there is barely any return. I am hoping to see the 'pilar' signature from vel or Z...
[2018/01/15 23:47:41 UTC] <Pei-gv> 23:47:10 transmission gap ~5 sec
[2018/01/15 23:48:33 UTC] <ScottEllis-Boulder> Pei-gv, What do you mean barely any return? I can't see any catalog images. Is the surface not showing up?
[2018/01/15 23:50:03 UTC] <Pei-gv> ScottEllis-Boulder, sorry for the confusion! I meant not cloud echo. I see ground alright.
[2018/01/15 23:50:37 UTC] <Pei-gv> TFOM Current/Max 15/5 what does this mean
[2018/01/15 23:50:50 UTC] <ScottEllis-Boulder> Oh, that is good. those clouds should be extremely low DBZ
[2018/01/15 23:51:47 UTC] <Pei-gv> super wimpy clouds at 3kft again. We don't see them.
[2018/01/15 23:53:07 UTC] <Pei-gv> 23:52:30 data gap plus "smear" data
[2018/01/15 23:54:11 UTC] <ScottEllis-Boulder> Pei-gv, I am not surprised you don't see those clouds
[2018/01/15 23:55:06 UTC] <ScottEllis-Boulder> Still not seeing HCR catalog images, but HSRL are there
[2018/01/15 23:56:08 UTC] <Pei-gv> Approaching 49S. Clouds below. Will perform surface CAL when it turns clear
[2018/01/15 23:56:26 UTC] <Pei-gv> ScottEllis-Boulder, now we are seeing more of the wimpy clouds.
[2018/01/15 23:57:28 UTC] <Pei-gv> ChrisBurghart-ops, I assume you are reading the log. We need to figure out a way to copy those HSRL images to acserver
[2018/01/15 23:57:43 UTC] <Pei-gv> It appears that HCR data is copied there.
[2018/01/15 23:59:08 UTC] <Pei-gv> Spectrcom still remains unsyncronized and major alarm

[2018/01/16 00:00:18 UTC] <Pei-gv> Turning to weigh point

[2018/01/16 00:01:15 UTC] <Pei-gv> Seeing more white wakes on the surface of the ocean. assume the sea surface is rough?
[2018/01/16 00:01:27 UTC] <Pei-gv> trying surface scan now
[2018/01/16 00:02:15 UTC] <ChrisBurghart-ops> Pei-gv, getting some system stuff set up here at the ops center so I can log into the plane
[2018/01/16 00:02:32 UTC] <Pei-gv> fast scan start
[2018/01/16 00:02:54 UTC] <Pei-gv> ChrisBurghart-ops, copy that. So you will fix it from there.
[2018/01/16 00:02:59 UTC] <Pei-gv> let me know what it's done
[2018/01/16 00:03:20 UTC] <Pei-gv> 00:02:44 transmission gap
[2018/01/16 00:03:37 UTC] <Pei-gv> ChrisBurghart-ops, thanks! Or if I can do it from here. Fire the command over
[2018/01/16 00:03:52 UTC] <Pei-gv> I am happy to try
[2018/01/16 00:04:20 UTC] <Pei-gv> clouds below
[2018/01/16 00:04:41 UTC] <Pei-gv> around 3kf
[2018/01/16 00:04:45 UTC] <ScottEllis-Boulder> Pei-gv, May not be the best time for surface cal
[2018/01/16 00:05:14 UTC] <Pei-gv> ScottEllis-Boulder, copy that. We don't have AVAPS today.
[2018/01/16 00:06:10 UTC] <Pei-gv> ndir pointin
[2018/01/16 00:08:29 UTC] <Pei-gv> Seeing those clouds around 3~4kft.
[2018/01/16 00:08:50 UTC] <ScottEllis-Boulder> Yeah, they look a little thicker on the camera.
[2018/01/16 00:09:55 UTC] <Pei-gv> ScottEllis-Boulder, they are pretty far away from the radar though.
[2018/01/16 00:11:00 UTC] <Pei-gv> Chris-ops, is there any chance that Spectracom is ever going to lock?
[2018/01/16 00:11:31 UTC] <Pei-gv> Was power cycling it a good thing to do? I don't like to run the radar with free running PLOs....
[2018/01/16 00:12:44 UTC] <Pei-gv> Getting into lots of clouds.
[2018/01/16 00:13:10 UTC] <Pei-gv> 00:12:15 transmision gap
[2018/01/16 00:13:22 UTC] <Pei-gv> 00:12:43 data gap
[2018/01/16 00:13:53 UTC] <Pei-gv> We should run with TOSHIBA drives and see if those gaps go away for RF02
[2018/01/16 00:15:46 UTC] <Pei-gv> Operator should utilize the ACSERVER/map overlaid with Himawari-8 imagry to figure out where to drop the radar cal sonde and perform the surface cal.
[2018/01/16 00:16:02 UTC] <Pei-gv> I was waiting until it reaches 49S. Not a good approach. Missed the opportunity.
[2018/01/16 00:20:23 UTC] <Pei-gv> ScottEllis-Boulder, 00:19 has beautiful cloud structure. This data gaps are starting to annoy me.
[2018/01/16 00:20:28 UTC] <Pei-gv> We are missing great data!
[2018/01/16 00:20:51 UTC] <Pei-gv> We may need to losen the criteria on the "no transmission gap" too
[2018/01/16 00:23:15 UTC] <ScottEllis-Boulder> Sorry to hear about the data gaps. How long do they last?
[2018/01/16 00:23:42 UTC] <ChrisBurghart-ops> Pei-gv, the Spectracom frequently loses 1 PPS validity for long periods of time, but seems to recover it (mostly) often enough to keep the unit in holdover.
[2018/01/16 00:23:55 UTC] <Pei-gv> ScottEllis-Boulder, please see the 00:19 images. I don't see apparent pilar structures...
[2018/01/16 00:24:13 UTC] <Pei-gv> 00:23:45 transmission gap
[2018/01/16 00:24:57 UTC] <ScottEllis-Boulder> Pei-gv, I still don't see catalog images of data. Am glad to hear there is no pillar evidence
[2018/01/16 00:25:17 UTC] <Pei-gv> ChrisBurghart-ops, I belive it passed the holdover. I am getting the 125MHZ ALM
[2018/01/16 00:25:31 UTC] <Pei-gv> cycled power. Never came back
[2018/01/16 00:25:53 UTC] <Pei-gv> ScottEllis-Boulder, not apparent to me, doesn't mean it's not there.
[2018/01/16 00:26:49 UTC] <Pei-gv> ChrisBurghart-ops, I wonder if the GPS signal is less/weaker so far down south. And this is the issue you don't see on FF
[2018/01/16 00:27:07 UTC] <ChrisBurghart-ops> Pei-gv, we'll have to consider increasing the holdover time. The web interface to the Spectracom should allow you to do that, if you're so bold.
[2018/01/16 00:27:27 UTC] <Pei-gv> ChrisBurghart-ops, now I am NOT transmitting
[2018/01/16 00:27:30 UTC] <Pei-gv> not sure why
[2018/01/16 00:28:13 UTC] <ChrisBurghart-ops> Pei-gv, Is it because 125 MHz isn't locked?
[2018/01/16 00:28:20 UTC] <Pei-gv> EMS switch faults
[2018/01/16 00:28:41 UTC] <Pei-gv> clear faults
[2018/01/16 00:29:46 UTC] <Pei-gv> ChrisBurghart-ops, it's still pretty sluggish. I just noticed the Chat locked up for a sec. and see data gap on HawkEye at the same time
[2018/01/16 00:30:07 UTC] <Pei-gv> Chat locked up again
[2018/01/16 00:30:36 UTC] <Pei-gv> I also noticed that whenever Jeff moves the HawkEye display, I got a delay
[2018/01/16 00:32:57 UTC] <ChrisBurghart-ops> Pei-gv, I have to run back to the hotel and get my laptop which has working VPN. It's taking too long to get VPN working on the HCR workstation here.
[2018/01/16 00:32:58 UTC] <Pei-gv> ScottEllis-Boulder, 00:30 good data image, do'nt see the pilar structure. Maybe because my display is set to 90 sec?
[2018/01/16 00:33:33 UTC] <Pei-gv> Anything I can try on my end to get the HSRL images to the directory?
[2018/01/16 00:37:49 UTC] <ScottEllis-Boulder> Pei-gv, Ok, I'll have a look when the images go to the catalog.
[2018/01/16 00:41:25 UTC] <Pei-gv> Eyeballing: crosswind legs around 00:10
[2018/01/16 00:42:11 UTC] <Pei-gv> ~00:25 INS manuever for Jorgen
[2018/01/16 00:43:42 UTC] <Pei-gv> TFOM current/Max 10/5
[2018/01/16 00:44:02 UTC] <Pei-gv> Maybe loosen the TFOM criteria
[2018/01/16 00:58:21 UTC] <Pei-gv> cycle power on Spectracom again
[2018/01/16 01:00:59 UTC] <Pei-gv> ChrisBurghart-ops, where is the HcrMonitor.ops.log? can't seem to find it
[2018/01/16 01:05:26 UTC] <Pei-gv> Spectracom: Frequency not in spec error went away.
[2018/01/16 01:05:47 UTC] <Pei-gv> Spectracom: Still holds the 1pps not in spec and not in sync faults.
[2018/01/16 01:06:00 UTC] <Pei-gv> I don't think I can solve this issue here.
[2018/01/16 01:09:22 UTC] <Pei-gv> HawkEye Display frozen
[2018/01/16 01:09:32 UTC] <Pei-gv> alrigth I have to do something
[2018/01/16 01:10:12 UTC] <Pei-gv> Rsync hasn't run since 23:59:48
[2018/01/16 01:10:26 UTC] <Pei-gv> I am unplugging one drive
[2018/01/16 01:10:43 UTC] <Pei-gv> unmount
[2018/01/16 01:10:51 UTC] <Pei-gv> unmount HCR104
[2018/01/16 01:11:20 UTC] <Pei-gv> start closing windows
[2018/01/16 01:11:43 UTC] <Pei-gv> hcrdrx.ops.log window hasn't printed since 23:59:58
[2018/01/16 01:12:58 UTC] <Pei-gv> ok. USB Rsync process revived.
[2018/01/16 01:13:23 UTC] <Pei-gv> 01:12:34 writing data from 00:38:27
[2018/01/16 01:13:43 UTC] <ScottEllis-Boulder> I have catalog images now.
[2018/01/16 01:15:19 UTC] <Pei-gv> Restart AcGeorefCompare. Values updating.
[2018/01/16 01:17:29 UTC] <Pei-gv> Writing data 00:45Z onto HCR103
[2018/01/16 01:19:24 UTC] <Pei-gv> TFOM 9/5
[2018/01/16 01:22:00 UTC] <Pei-gv> AcGeorefCompare frozen again
[2018/01/16 01:25:45 UTC] <Pei-gv> Notice the Moments-10Hz Heading is now correct with IWG1
[2018/01/16 01:27:18 UTC] <Pei-gv> Notice Spatracom Status frozen at 01:26:50
[2018/01/16 01:28:37 UTC] <Pei-gv> ScottEllis-Boulder, can you see images after 01:10?
[2018/01/16 01:29:11 UTC] <ScottEllis-Boulder> Yep, latest is 0120 to 0125
[2018/01/16 01:29:36 UTC] <Pei-gv> do you see any black gaps since 01:10?
[2018/01/16 01:29:47 UTC] <Pei-gv> I don't think I see any so far.
[2018/01/16 01:30:06 UTC] <jemmett-ops> Pei-gv, we need to do something about this Sprectracom
[2018/01/16 01:30:28 UTC] <Pei-gv> jemmett-ops, agree. Thinking about flying FireFly.
[2018/01/16 01:30:48 UTC] <Pei-gv> This is too much of a cost than resource to us
[2018/01/16 01:30:49 UTC] <ScottEllis-Boulder> 01:23, 01:16:30, 01:13:30 and 01:10:30
[2018/01/16 01:32:09 UTC] <ScottEllis-Boulder> Those are apparent drop-outs
[2018/01/16 01:32:11 UTC] <Pei-gv> ScottEllis-Boulder, are they gaps for no transmission or no data? I am looking for 'no data' gaps
[2018/01/16 01:33:06 UTC] <Pei-gv> jemmett-ops, we should at least see how FireFly performs.
[2018/01/16 01:33:20 UTC] <ScottEllis-Boulder> How do I tell the difference? They are black gaps in the surface echo and bang
[2018/01/16 01:34:50 UTC] <Pei-gv> If you still see the range-corrected reflectivity, then there is no transmission. If you do'nt even see the range-corrected (colored data), it will be a complete black gap , as if there is no data.
[2018/01/16 01:35:18 UTC] <Pei-gv> 01:34:40 transmission gap
[2018/01/16 01:36:52 UTC] <ScottEllis-Boulder> There is very little range-corrected noise on the catalog images. But I think they are complete gaps. jemmett-ops and ChrisBurghart-ops can you have a look to?
[2018/01/16 01:38:03 UTC] <Pei-gv> ScottEllis-Boulder, I also realized that you are seeing the 5-minute plots as oppose to my 90 sec...
[2018/01/16 01:40:42 UTC] <Pei-gv> jemmett-ops and ScottEllis-Boulder, I am getting a little concerned about HCR not seeing the wimpy clouds.
[2018/01/16 01:40:59 UTC] <Pei-gv> They look nice and puffy at times.
[2018/01/16 01:42:34 UTC] <ScottEllis-Boulder> They look like liquid clouds at low levels (puffy). They can have quite low Z levels. Ice clouds would be higher Z.
[2018/01/16 01:44:01 UTC] <Pei-gv> ScottEllis-Boulder, Thanks. That's helpful. We are about to descend for profiling legs.
[2018/01/16 01:44:06 UTC] <Pei-gv> turning now
[2018/01/16 01:46:48 UTC] <Pei-gv> 01:46:10 no transmission. WHY????
[2018/01/16 01:47:08 UTC] <ChrisBurghart-ops> Pei-gv, any log following windows (e.g., for hcrdrx or rsync to USB) need to be restarted after we cross into a new day. The ones you have running right now are showing the log files for 2018/01/15.
[2018/01/16 01:47:59 UTC] <Pei-gv> ChrisBurghart-ops, I got the USB log back by stopping and restarting it.
[2018/01/16 01:48:13 UTC] <Pei-gv> But I have also seen it freeze
[2018/01/16 01:48:33 UTC] <Pei-gv> 01:47:50 data gap
[2018/01/16 01:49:03 UTC] <ChrisBurghart-ops> Pei-gv, Yep, and you can kill the hcrdrx log following and start it up again to see today's file for hcrdrx.
[2018/01/16 01:49:49 UTC] <Pei-gv> ChrisBurghart-ops, we are going to start our profiling leg soon
[2018/01/16 01:50:08 UTC] <Pei-gv> We gotta look into this stupid 5 second no transmission issue
[2018/01/16 01:50:18 UTC] <Pei-gv> 01:49:40 no transmission
[2018/01/16 01:50:18 UTC] <ScottEllis-Boulder> I am seeing cloud echoes with a  value of about -25 dBZ at a range of about 4.3 km
[2018/01/16 01:50:32 UTC] <ChrisBurghart-ops> I got HCR image transfer to the catalog going again while I was at the hotel, and also got HSRL images moving to the acserver.
[2018/01/16 01:53:01 UTC] <Pei-gv> Turn around to head west, for 10 minute above cloud leg
[2018/01/16 01:53:25 UTC] <Pei-gv> when turn back to east, do sawtooth
[2018/01/16 01:53:51 UTC] <Pei-gv> Pretty above-cloud leg
[2018/01/16 01:53:51 UTC] <ChrisBurghart-ops> Pei-gv, The transmit dropouts are almost certainly related to system sluggishness. For safety, keeping HV on requires a heartbeat signal to be sent frequently. If the system gets bogged down, the heartbeats end up going out late and HV gets turned off at least until they return.
[2018/01/16 01:55:12 UTC] <Pei-gv> ChrisBurghart-ops, so they are all related to the system issue then. Let's try the TOSHIBA drives RF02 and see how it does.
[2018/01/16 01:55:27 UTC] <Pei-gv> Now I am running with only one USB drive
[2018/01/16 01:55:52 UTC] <Pei-gv> it seems to reduce the frequency of the no transmission events
[2018/01/16 01:56:25 UTC] <Pei-gv> ChrisBurghart-ops, I am so tempted to take out both drives. We are missing quite a few good data sets.
[2018/01/16 01:56:53 UTC] <Pei-gv> I am not taking HCR103 out at the moment. But very tempted
[2018/01/16 01:57:09 UTC] <Pei-gv> above-cloud leg
[2018/01/16 01:59:09 UTC] <ChrisBurghart-ops> Pei-gv, If you pull the disk, we can copy data off the archiver during maintenance day tommorow.
[2018/01/16 01:59:29 UTC] <Pei-gv> ChrisBurghart-ops, ok. I will let you know if I do that
[2018/01/16 02:00:09 UTC] <Pei-gv> ChrisBurghart-ops, the HCR103 is copying 01:39 datasets
[2018/01/16 02:00:17 UTC] <Pei-gv> ~20 minute lag
[2018/01/16 02:02:06 UTC] <Pei-gv> Man. We are right at the cloud tops
[2018/01/16 02:02:59 UTC] <Pei-gv> ChrisBurghart-ops, I am going to pull the drive now.
[2018/01/16 02:03:09 UTC] <Pei-gv> I can plug it back in later
[2018/01/16 02:06:16 UTC] <ChrisBurghart-ops> Pei-gv, 20 minutes behind is a big lag, especially this early in the flight. I agree the drive should be removed.
[2018/01/16 02:06:45 UTC] <Pei-gv> ChrisBurghart-ops, just don't want to see those gaps and miss this profiling leg
[2018/01/16 02:07:08 UTC] <Pei-gv> turning.
[2018/01/16 02:07:58 UTC] <Pei-gv> ChrisBurghart-ops, still see the no HV event
[2018/01/16 02:08:02 UTC] <ChrisBurghart-ops> I would not try plugging it in later. It isn't keeping up now, and I can't think of a reason it would work better later.
[2018/01/16 02:09:58 UTC] <ChrisBurghart-ops> Pei-gv, can you see anything that looks unusual if you run 'top' on rds?
[2018/01/16 02:11:17 UTC] <Pei-gv> ChrisBurghart-ops, not really.
[2018/01/16 02:11:27 UTC] <Pei-gv> hcrdrx, Cmigits HcrMonitor...
[2018/01/16 02:11:40 UTC] <Pei-gv> hcrdrx is constantly at ~40%
[2018/01/16 02:13:07 UTC] <ChrisBurghart-ops> How about on archiver?
[2018/01/16 02:15:13 UTC] <ChrisBurghart-ops> On there, I'm curious if any swap memory is being used. That's shown in about the 5th line which starts with "KiB Swap", and I'm interested in the "xx used".
[2018/01/16 02:15:52 UTC] <Pei-gv> ChrisBurghart-ops, sorry. was working on the instrument. please hold on the archiver stuff
[2018/01/16 02:18:12 UTC] <Pei-gv> turning
[2018/01/16 02:20:28 UTC] <Pei-gv> nadir pointin
[2018/01/16 02:23:23 UTC] <Pei-gv> It's a bit nerve wracking to try to flip both as fast as you can
[2018/01/16 02:23:36 UTC] <Pei-gv> and it's a 2 minute sawtooth
[2018/01/16 02:25:27 UTC] <Pei-gv> zenith pointing
[2018/01/16 02:29:09 UTC] <Pei-gv> 7500ft, 10 minutes leg in cloud next
[2018/01/16 02:30:24 UTC] <Pei-gv> 02:29:30 event? lost heart beat?
[2018/01/16 02:31:32 UTC] <Pei-gv> 400 ft above water
[2018/01/16 02:33:28 UTC] <Pei-gv> 02:32:45 no transmission event
[2018/01/16 02:42:07 UTC] <Pei-gv> nadir poinitng
[2018/01/16 02:48:49 UTC] <ChrisBurghart-ops> Pei-gv, any improvement in system sluggishness since removing the drive?
[2018/01/16 02:50:00 UTC] <Pei-gv> ChrisBurghart-ops, getting back to you on the 'top' process on Archiver
[2018/01/16 02:50:09 UTC] <Pei-gv> I see kiB Swap 0 total, 0 free and 0 used.
[2018/01/16 02:50:28 UTC] <Pei-gv> 28915896 avail Mem
[2018/01/16 02:51:50 UTC] <Pei-gv> ScottEllis-Boulder, and ChrisBurghart-ops, we may not be able to get the legs in today. May be running low on fuel
[2018/01/16 02:53:19 UTC] <ChrisBurghart-ops> Hmm, we should probably have some swap space on that machine... With that much available memory, it shouldn't be an issue, though.
[2018/01/16 02:54:34 UTC] <Pei-gv> I am not sure what it is. It is definitely still showing that heart beat gap from time to time. Even though there is NO drives to write
[2018/01/16 02:55:02 UTC] <Pei-gv> This doesn't make sense to me
[2018/01/16 02:55:47 UTC] <ScottEllis-Boulder> Pei-gv, For the pointing legs, if they can just go downwind for a few minutes on the way back, it would be great.
[2018/01/16 02:56:47 UTC] <Pei-gv> ScottEllis-Boulder, I am asking for 7 minutes. How short can you live with?
[2018/01/16 02:57:30 UTC] <ScottEllis-Boulder> 7 min sounds fine. 5 would be sort of OK.
[2018/01/16 03:14:29 UTC] <Pei-gv> Now I waited for ~10 minutes of not touching anything. I can't get the heart beat thing to fault
[2018/01/16 03:15:33 UTC] <Pei-gv> ocean surface CAL-fast scan
[2018/01/16 03:15:42 UTC] <Pei-gv> fairly clear down
[2018/01/16 03:16:24 UTC] <Pei-gv> ETA 5pm ish
[2018/01/16 03:19:25 UTC] <Pei-gv> slow sc172an 168-
[2018/01/16 03:19:32 UTC] <Pei-gv> slow scan 168-172
[2018/01/16 03:20:25 UTC] <Pei-gv> slowscan 2 03:20:10
[2018/01/16 03:21:05 UTC] <Pei-gv> nadir pointing
[2018/01/16 03:21:14 UTC] <Pei-gv> NSCAL
[2018/01/16 03:22:37 UTC] <Pei-gv> ChrisBurghart-ops, now that we are on return flight, I can't seem to see the heartbeat error.
[2018/01/16 03:23:07 UTC] <Pei-gv> Maybe it's related to all the activities going on on the net also?
[2018/01/16 03:23:52 UTC] <ChrisBurghart-ops> What net activities?
[2018/01/16 03:28:15 UTC] <Pei-gv> 03:28:00 fast scan
[2018/01/16 03:28:52 UTC] <Pei-gv> network traffic? ethernet activity? hsrl data coming from ethernet that is delayed?
[2018/01/16 03:29:41 UTC] <Pei-gv> performing fast scan. Extremely clear below. PI approved this activity
[2018/01/16 03:31:11 UTC] <Pei-gv> 03:30:30 heartbeat error
[2018/01/16 03:32:12 UTC] <Pei-gv> 03:32:00 nadir pointin
[2018/01/16 03:32:34 UTC] <Pei-gv> 03:32:10 heartbeat error
[2018/01/16 03:33:22 UTC] <Pei-gv> 03:33:08 slow scan -1
[2018/01/16 03:35:25 UTC] <Pei-gv> 03:33:45 end.
[2018/01/16 03:35:54 UTC] <Pei-gv> 03:34:10~03:34:50 slow scan 2
[2018/01/16 03:36:33 UTC] <Pei-gv> may have a roll action from airplane platform. Pilot were going to turn for Jorgen's maneuver
[2018/01/16 03:36:58 UTC] <Pei-gv> 03:36:38 heartbeat error
[2018/01/16 03:38:43 UTC] <Pei-gv> INS maneuver around 03:37Z
[2018/01/16 03:39:27 UTC] <Pei-gv> 03:39:10 heartbeat error
[2018/01/16 03:43:22 UTC] <Pei-gv> HCR upwind leg.
[2018/01/16 03:43:30 UTC] <Pei-gv> Request for min.5 minutes
[2018/01/16 03:44:12 UTC] <Pei-gv> request into the wind & kill drift, minimum 5 minutes.
[2018/01/16 03:45:33 UTC] <Pei-gv> ScottEllis-Boulder, you are seeing HCR images, correct?
[2018/01/16 03:46:09 UTC] <ScottEllis-Boulder> Yes
[2018/01/16 03:46:27 UTC] <Pei-gv> 03:45:40 heartbeat error
[2018/01/16 03:47:26 UTC] <Pei-gv> 03:47:10 heartbeat
[2018/01/16 03:47:40 UTC] <ChrisBurghart-ops> Pei-gv, can you get copies of the cfradial 10Hz files for this calibration leg? I'd like to get those to Scott, and we won't have them on a removable drive at the end of the flight.
[2018/01/16 03:48:42 UTC] <Pei-gv> end of the leg. Turning for down wind. [03:47:51]
[2018/01/16 03:49:10 UTC] <Pei-gv> I can. I will copy them to my drive.
[2018/01/16 03:49:28 UTC] <Pei-gv> 03:48:45 heartbeat
[2018/01/16 03:51:52 UTC] <Pei-gv> in downwind leg.
[2018/01/16 03:52:32 UTC] <Pei-gv> ChrisBurghart-ops, can I plug the HCR001 drive onto Archiver? will that make the system go crazy/confused?
[2018/01/16 03:54:50 UTC] <ChrisBurghart-ops> Pei-gv, the system will start writing the drive at the next 5 minute mark and will do what it can to catch up. If you're going to put a drive on, put on the one you removed last, since it has less catching up to do.
[2018/01/16 03:55:18 UTC] <Pei-gv> ok. Will do
[2018/01/16 03:55:20 UTC] <ChrisBurghart-ops> Did you notice any change in system usability after you removed the second drive?
[2018/01/16 03:55:25 UTC] <Pei-gv> end of downwind leg
[2018/01/16 03:55:40 UTC] <Pei-gv> ChrisBurghart-ops, unfortunately, I do.
[2018/01/16 03:56:09 UTC] <Pei-gv> I think I see less of the "black gap + smear data" feature but still seeing the heartbeat related transmission dropouts.
[2018/01/16 03:56:47 UTC] <Pei-gv> that puzzles me. We took the drives out. And the system is still having issue with heartbeats.
[2018/01/16 03:57:14 UTC] <ChrisBurghart-ops> If you put the drive back, keep an eye on the system, and remove the drive again if there are significant problems.
[2018/01/16 03:59:40 UTC] <Pei-gv> ChrisBurghart-ops, it's only about ~25 MB. I will put it on my memory stick
[2018/01/16 03:59:53 UTC] <Pei-gv> this should avoid the potential issue?
[2018/01/16 04:02:24 UTC] <ChrisBurghart-ops> Pei-gv, definitely copy the CFRadial files for the cal legs onto a USB stick.
[2018/01/16 04:03:04 UTC] <Pei-gv> done!
[2018/01/16 04:06:53 UTC] <ChrisBurghart-ops> Here's my thought about putting a 2.5" drive back. It would be great to stop_all on rds 20-30 minutes before you land, then plug the drive (or both) back in. The next update will get all of the CFRadial files and the latest logs very quickly, and then bog down doing the time series. However, just getting all of the CFRadial files and logs will be quite helpful here.
[2018/01/16 04:10:22 UTC] <Pei-gv> ChrisBurghart-ops, copy that. Will do that as long as we have super boring weather.
[2018/01/16 04:10:46 UTC] <Pei-gv> ChrisBurghart-ops, will evaluate about an hour from now.
[2018/01/16 04:11:27 UTC] <Pei-gv> ScottEllis-Boulder, it's interesting. I really don't see those pilar feature in the data today. please let me know what you see at some point...
[2018/01/16 04:27:37 UTC] <Pei-gv> I can see some pilar like feature now. It's also more turbulent.
[2018/01/16 04:37:03 UTC] <ScottEllis-Boulder> I don't see the obvious 15 to 30 sec period pillar like oscillations. I do see some signs of turbulence at much higher frequency. I see up to 0430.
[2018/01/16 04:39:07 UTC] <Pei-gv> ScottEllis-Boulder, copy that. Isn't that strange?!
[2018/01/16 04:41:31 UTC] <ScottEllis-Boulder> I don't think so, we see impacts from turbulence from time to time. ChrisBurghart-ops can verify that the 100 Hz INS data are being used for the correction.
[2018/01/16 04:42:14 UTC] <ScottEllis-Boulder> That made a difference when we compared to using the 10 Hz for INS corrections.
[2018/01/16 04:45:03 UTC] <ChrisBurghart-ops> Pei-gv, Based on the 04:10 vis imagery, you'll lose most of the clouds soon, and that will continue to Hobart.
[2018/01/16 04:45:48 UTC] <Pei-gv> ChrisBurghart-ops, copy that. Will probably shut down soon.
[2018/01/16 04:49:23 UTC] <Pei-gv> ChrisBurghart-ops, got the instruction to shut hcr ~30 minutes prior.
[2018/01/16 04:51:36 UTC] <ChrisBurghart-ops> Pei-gv, Great! Please plug in both drives after you stop_all on rds. That will mean less to copy tomorrow.
[2018/01/16 04:52:28 UTC] <Pei-gv> ChrisBurghart-ops, copy that. We are approximately 1:20 to home.
[2018/01/16 04:53:35 UTC] <ChrisBurghart-ops> Pei-gv, Also, please pull just one drive from the plane. I'll try to get files from it to Boulder tonight.
[2018/01/16 05:23:25 UTC] <ChrisBurghart-ops> Pei-gv, How's she behaving? Shutting down soon?
[2018/01/16 05:24:21 UTC] <ChrisBurghart-ops> (Looks like you're still more than a half hour out, so probably not shutting down yet.)
[2018/01/16 05:24:27 UTC] <Pei-gv> ChrisBurghart-ops, no error so far.
[2018/01/16 05:24:31 UTC] <Pei-gv> do you have an ETA?
[2018/01/16 05:24:40 UTC] <Pei-gv> I think we are still close to an hour out...
[2018/01/16 05:25:32 UTC] <ChrisBurghart-ops> Cory just emailed that the ETA is 1710 local.
[2018/01/16 05:25:59 UTC] <Pei-gv> ChrisBurghart-ops, ok. shutting down now
[2018/01/16 05:26:22 UTC] <Pei-gv> stopped hcrdrx
[2018/01/16 05:27:20 UTC] <Pei-gv> ChrisBurghart-ops, plug in HCR103
[2018/01/16 05:27:50 UTC] <ChrisBurghart-ops> Please keep running, system!
[2018/01/16 05:28:14 UTC] <Pei-gv> ChrisBurghart-ops, waiting for Rsync to start...
[2018/01/16 05:30:40 UTC] <Pei-gv> ChrisBurghart-ops, Rsync just started
[2018/01/16 05:33:53 UTC] <Pei-gv> Looks like about an hour worth of data to catch up on HCR103
[2018/01/16 05:34:10 UTC] <ChrisBurghart-ops> That's all? I'm surprised.
[2018/01/16 05:35:47 UTC] <Pei-gv> ChrisBurghart-ops, I hope I am reading this right...
[2018/01/16 05:37:17 UTC] <ChrisBurghart-ops> I guess there's a very small chance it will catch up on all of the data, but I'm not holding my breath. Anything that gets on there now will help tomorrow.
[2018/01/16 05:43:51 UTC] <Pei-gv> ChrisBurghart-ops, what I don't understand is I see a process of copying data from 0525hr.
[2018/01/16 05:44:30 UTC] <ChrisBurghart-ops> Where do you see this process?
[2018/01/16 05:44:39 UTC] <Pei-gv> and the next process is copying data from 0156 hour
[2018/01/16 05:44:48 UTC] <Pei-gv> both 1/16.
[2018/01/16 05:45:02 UTC] <Pei-gv> And comment/insight on this?
[2018/01/16 05:45:33 UTC] <ChrisBurghart-ops> Is it copying time series files from 0525? Or something else?
[2018/01/16 05:45:55 UTC] <Pei-gv> ChrisBurghart-ops, it's interesting thing on Top process: only 1.3GB of memory is free at the moment
[2018/01/16 05:46:20 UTC] <ChrisBurghart-ops> Is HSRL still running?
[2018/01/16 05:46:33 UTC] <Pei-gv> 65GB total, 47.1GB used, 18.3 GB buffer/cache
[2018/01/16 05:46:53 UTC] <Pei-gv> and 267MB free at the moment.
[2018/01/16 05:47:07 UTC] <Pei-gv> this sounds like a problem to me?
[2018/01/16 05:47:22 UTC] <Pei-gv> I shut down HSRL. Only hsrl-archiver is still on
[2018/01/16 05:48:17 UTC] <Pei-gv> ChrisBurghart-ops, if it is agreeable, I am going to do stop_all on RDS
[2018/01/16 05:48:47 UTC] <ChrisBurghart-ops> I thought you did the RDS stop_all some time ago...
[2018/01/16 05:49:18 UTC] <Pei-gv> did not yet. Just hcrdrx
[2018/01/16 05:50:02 UTC] <ChrisBurghart-ops> OK, that's fine.
[2018/01/16 05:51:41 UTC] <ChrisBurghart-ops> As for memory, it's not unusual to see no "free" memory. The buffer/cache memory is effectively free, as the OS is just using it while it isn't needed by any processes.
[2018/01/16 05:58:54 UTC] <Pei-gv> ChrisBurghart-ops, alright. We are three hours worth of data behind.
[2018/01/16 05:59:03 UTC] <Pei-gv> just checked the drive
[2018/01/16 06:02:10 UTC] <ChrisBurghart-ops> Pei-gv, 3 hours behind sounds more like what I was expecting.
[2018/01/16 06:05:08 UTC] <Pei-gv> ChrisBurghart-ops, Just landed
[2018/01/16 06:05:25 UTC] <Pei-gv> Will stop everything once we stop and door opened