SOCRATES chat log RF03

[2018/01/22 19:39:48 UTC] <Pei-gv> NSCAL
[2018/01/22 20:35:12 UTC] <Pei-gv> NSCAL again
[2018/01/22 20:44:43 UTC] <Pei-gv> engine start
[2018/01/22 20:45:37 UTC] <Pei-gv> ChrisBurghart-phone, no missing pulses so far!
[2018/01/22 20:50:17 UTC] <Pei-gv> bench test 20:49:49
[2018/01/22 20:50:32 UTC] <Pei-gv> FIL ready to go.
[2018/01/22 20:50:51 UTC] <Pei-gv> door closed
[2018/01/22 20:53:35 UTC] <Pei-gv> transmit
[2018/01/22 20:55:08 UTC] <Pei-gv> All green lights, team!
[2018/01/22 20:55:45 UTC] <ScottEllis-FL> We like green!
[2018/01/22 21:04:23 UTC] <Pei-gv> took off
[2018/01/22 21:07:46 UTC] <Pei-gv> nadir pointing
[2018/01/22 21:09:42 UTC] <Pei-gv> ScottEllis-FL, velocity is at ~-0.21 m/s, according to the AcGeorefCompare App
[2018/01/22 21:10:05 UTC] <ScottEllis-FL> Pei-gv, Hmm, that is high. Are you still climbing?
[2018/01/22 21:10:08 UTC] <Pei-gv> Spectracom Error.
[2018/01/22 21:10:13 UTC] <Pei-gv> ScottEllis-FL, we are
[2018/01/22 21:10:30 UTC] <ScottEllis-FL> Pei-gv, Let me know how it looks when you are flat and level.
[2018/01/22 21:11:07 UTC] <Pei-gv> ScottEllis-FL, that was a level leg. Maybe I should have waited a little longer. Will report later
[2018/01/22 21:13:41 UTC] <Pei-gv> vivek-ops, do we need absolutely clear condition for the surface cal?
[2018/01/22 21:14:07 UTC] <Pei-gv> I am not sure if I can get completely clear condition.
[2018/01/22 21:14:38 UTC] <Pei-gv> Judging from the satellite image overlaid on acserver map.
[2018/01/22 21:17:03 UTC] <vivek-ops> Pei, Scattered thin  liquid clouds are okay. Cirrus and ice clouds may not be an issue.
[2018/01/22 21:18:58 UTC] <ScottEllis-FL> Not sure why the surface Vr seems worse than last flight.
[2018/01/22 21:19:52 UTC] <Pei-gv> ScottEllis-FL, I am not sure. I will remind the for a s turn
[2018/01/22 21:20:27 UTC] <Pei-gv> ScottEllis-FL, they are busy with changing waypoint. Will do the surface scan and then request for S turn
[2018/01/22 21:20:42 UTC] <ScottEllis-FL> Thanks. I don't think that should be an issue this early, but they are good to do.
[2018/01/22 21:20:57 UTC] <ScottEllis-FL> I see a dropout at 21:12
[2018/01/22 21:21:00 UTC] <Pei-gv> vivek-ops, it's hard to see what kind of clouds are ahead.
[2018/01/22 21:21:52 UTC] <ScottEllis-FL> vivek-ops, On the GV camera, they look like low liquid clouds, but it is hard to tell
[2018/01/22 21:22:24 UTC] <vivek-ops> Pei, Agree, moving platform is difficult. Camera is a good suggestion.
[2018/01/22 21:24:18 UTC] <Pei-gv> vivek-ops, and ScottEllis-FL, Thanks. I am not sure if we can get a surface scan in before 49S
[2018/01/22 21:26:09 UTC] <Pei-gv> vivek-ops, & ScottEllis-FL, Just talked to Greg. If we can't drop before 49S. We will try on our return
[2018/01/22 21:27:24 UTC] <Hayman_mobile> hsrl show low liquid clouds
[2018/01/22 21:29:35 UTC] <vivek-ops> Pei, According to the PI, the low clouds are everywhere.
[2018/01/22 21:29:54 UTC] <ScottEllis-FL> ChrisBurghart-phone, Is it possible the pointing correction changes from last time aren't being used now?
[2018/01/22 21:30:33 UTC] <ScottEllis-FL> Hayman_mobile, Thanks! vivek-ops, that is in agreement with the satellite and GV camera.
[2018/01/22 21:31:17 UTC] <Pei-gv> Lots of clouds in the radar range.
[2018/01/22 21:32:16 UTC] <Pei-gv> Spectracom continues to have problems
[2018/01/22 21:32:53 UTC] <vivek-ops> Pei, As of now the optimal window of the surface is cal is during the return leg;  i.e. many hours away
[2018/01/22 21:34:54 UTC] <Pei-gv> vivek-ops, we will try right after this pack of clouds..
[2018/01/22 21:38:13 UTC] <Pei-gv> issue sonde drop command
[2018/01/22 21:38:22 UTC] <Pei-gv> Waiting for this patch to clear
[2018/01/22 21:39:12 UTC] <Pei-gv> Sonde is dropped
[2018/01/22 21:39:24 UTC] <Pei-gv> We have pretty thick clouds though
[2018/01/22 21:39:52 UTC] <Pei-gv> start scaning
[2018/01/22 21:40:33 UTC] <AndroUser2> ScottEllis-FL, Nothing changed in the pointing correction. It should be the same as last flight.
[2018/01/22 21:42:56 UTC] <vivek-ops> This surface cal will be helpful for evaluating the impact of cloud cover.
[2018/01/22 21:44:17 UTC] <Pei-gv> vivek-ops, thanks. We gave it a try.. it looked clear on the fore camera.
[2018/01/22 21:45:11 UTC] <Pei-gv> 21:43:35 slow scan 1
[2018/01/22 21:45:31 UTC] <Pei-gv> 21:44:45 Slow scan 2
[2018/01/22 21:45:45 UTC] <Pei-gv> nadir pointing
[2018/01/22 21:46:53 UTC] <Pei-gv> Spectracom turned yellow shortly and then back to major alarm
[2018/01/22 21:54:15 UTC] <vivek-ops> Pei, Good. During the Cal scattered clouds; not bad.
[2018/01/22 21:59:04 UTC] <Pei-gv> notice latency on both INS1 & INS2 data
[2018/01/22 22:06:07 UTC] <Pei-gv> Max power latency at 21:51:58, 21:48:27 and 21:50:22.
[2018/01/22 22:06:57 UTC] <Pei-gv> 5 second gap seems to be associated with the "Dropping 4 TsPrintMaxPower" msg, instead of the Max power latency
[2018/01/22 22:08:49 UTC] <Pei-gv> Correction/addition: 21:48 and 21:51 events have heartbeat dropouts.
[2018/01/22 22:09:07 UTC] <Pei-gv> Spectracom turn green
[2018/01/22 22:11:19 UTC] <Pei-gv> 22:00:25, 22:01:56 heartbeat timeouts
[2018/01/22 22:11:39 UTC] <Pei-gv> I guess we have passed the HCR 90 minutes
[2018/01/22 22:15:18 UTC] <ChrisBurghart-ops> Pei-gv, I forgot to check. Are both USB drives plugged in? I meant to run with just one.
[2018/01/22 22:16:43 UTC] <Pei-gv> ChrisBurghart-ops, there is only one running. I unplugged the second one. We are writing to HCR41 today
[2018/01/22 22:17:00 UTC] <ChrisBurghart-ops> Ok, that's good.
[2018/01/22 22:18:00 UTC] <ChrisBurghart-ops> Surface Vr looks pretty good in the most recent images, hovering around zero with some positives and negatives.
[2018/01/22 22:19:03 UTC] <Pei-gv> ChrisBurghart-ops, agreed. What puzzled me is I see slightly difference between real time display vs. images in acserver
[2018/01/22 22:19:18 UTC] <ChrisBurghart-ops> Does the ocean look rough right now? That seemed to be associated with higher variability in the surface Vr last flight, and that's what I'm seeing now.
[2018/01/22 22:19:47 UTC] <ChrisBurghart-ops> Pei-gv, What sorts of differences are you seeing between them?
[2018/01/22 22:19:54 UTC] <Pei-gv> ChrisBurghart-ops, can't see the ocean. pretty cloudy now
[2018/01/22 22:20:46 UTC] <Pei-gv> ChrisBurghart-ops, Correction. I think they look better now. Scratched my previous comment
[2018/01/22 22:22:35 UTC] <ChrisBurghart-ops> surface Vr notes were just comments
[2018/01/22 22:23:11 UTC] <ChrisBurghart-ops> the question after was in reference to your note that real-time display and catalog images look different
[2018/01/22 22:23:31 UTC] <Pei-gv> ChrisBurghart-ops, can you look at the source of the heartbeat timeouts? Now I am dying to know the cause of it...
[2018/01/22 22:25:40 UTC] <Pei-gv> ChrisBurghart-ops, regarding the difference between images and Hawkeye. I took another look at them. I guess they are not THAT different.  A bit noisier on HawkEye than images.
[2018/01/22 22:25:56 UTC] <Pei-gv> Spectracom Turn red
[2018/01/22 22:26:09 UTC] <Pei-gv> Major: 1PPS not is spec
[2018/01/22 22:28:39 UTC] <ChrisBurghart-ops> Pei-gv, OK, I won't worry about the display then. And it looks like business as usual for the Spectracom.
[2018/01/22 22:28:44 UTC] <Pei-gv> S manuever for INS
[2018/01/22 22:33:10 UTC] <Pei-gv> Now all clear. perfect for the surface CAL....
[2018/01/22 22:33:58 UTC] <ChrisBurghart-ops> Pei-gv, get them to drop a sonde, and you're all set!
[2018/01/22 22:34:31 UTC] <Pei-gv> ChrisBurghart-ops, Ha. I wish.
[2018/01/22 22:37:22 UTC] <ScottEllis-FL> I just got caught up on the Velocity discussion. I was very puzzled by the positive surface Vr that seems better now. The crosswind and thus drift, seem about the same sort of ruling out rotation error! I wonder if the INS is drifting again?
[2018/01/22 22:38:06 UTC] <Pei-gv> fast scan cal
[2018/01/22 22:38:46 UTC] <Pei-gv> ScottEllis-FL, did you see that we just did a INS S turns
[2018/01/22 22:39:36 UTC] <ScottEllis-FL> Oh, now I see, it I'll have a look before and after, thanks Pei-gv
[2018/01/22 22:41:16 UTC] <Pei-gv> nadir point
[2018/01/22 22:41:19 UTC] <ScottEllis-FL> Pei-gv, Looks ok from here. What are you seeing?
[2018/01/22 22:42:03 UTC] <Pei-gv> ScottEllis-FL, it looks good from here too. I did not think our INS drifted much, at least not from naked eyes..
[2018/01/22 22:42:35 UTC] <ScottEllis-FL> Ok, thanks, I am still puzzled by the larger errors at the very beginning of the flight
[2018/01/22 22:42:45 UTC] <Pei-gv> The surface velocity does look less "noisy" after the turns
[2018/01/22 22:42:49 UTC] <ScottEllis-FL> But glad it looks better now
[2018/01/22 22:44:00 UTC] <eric_loew-FL> ChrisBurghart-ops: Do you not have a spare Spectracom there? I have one I could ship to you. Phase noise spec is not quite as good, but better than what you have now!
[2018/01/22 22:44:18 UTC] <Pei-gv> ChrisBurghart-ops, I assume we don't see HSRL's rsync process in the "archive_to_USB" window?
[2018/01/22 22:44:43 UTC] <Pei-gv> eric_loew-FL, Not to worry. We are running the system with FireFly now
[2018/01/22 22:45:25 UTC] <Pei-gv> eric_loew-FL, The notes were to provide some logs for the unit.
[2018/01/22 22:45:37 UTC] <ScottEllis-FL> I was just asked this: Hi Scott- are you aware of any software running on the GV that might be accessing Jeppesen Weather Controller? There's a lot of bandwidth being used on satcom accessing a Jeppesen service.
[2018/01/22 22:47:31 UTC] <ScottEllis-FL> Erikj-eol sent me the message with the question.
[2018/01/22 22:47:35 UTC] <Pei-gv> jemmett-OPS, You may have a good system for tomorrow...
[2018/01/22 22:49:49 UTC] <Pei-gv> Archive_to_USB seems to finish consistently by minute 2:50
[2018/01/22 22:50:17 UTC] <Pei-gv> So Rsyncing HSRL at minute 3 & 8 may be a good choice
[2018/01/22 22:53:09 UTC] <Pei-gv> turning
[2018/01/22 22:54:29 UTC] <Pei-gv> ETA to Investigator 4 minute
[2018/01/22 22:54:40 UTC] <Pei-gv> pass to the west of the ship
[2018/01/22 22:55:10 UTC] <Pei-gv> headed NW after passing the ship
[2018/01/22 22:56:58 UTC] <Pei-gv> 7 miles to ship
[2018/01/22 22:57:34 UTC] <ScottEllis-FL> ChrisBurghart-ops, what should I tell erik about accessing the Jeppesen Wx cntrler?
[2018/01/22 22:58:21 UTC] <Pei-gv> right over the boat
[2018/01/22 22:58:29 UTC] <Pei-gv> sonde release. Turn NW
[2018/01/22 22:59:02 UTC] <Pei-gv> drop down to 18kft
[2018/01/22 23:00:34 UTC] <ChrisBurghart-ops> ScottEllis-FL, I don't know a thing about a Jeppesen Wx system... Seems like that would be in the cockpit.
[2018/01/22 23:01:34 UTC] <Pei-gv> 22:54:40 5-second gap
[2018/01/22 23:02:23 UTC] <Pei-gv> 10 minute leg above clouds
[2018/01/22 23:11:32 UTC] <ChrisBurghart-ops> Pei-gv, disappointing that there's that much lag. I wonder if atop would show anything of interest?
[2018/01/22 23:12:15 UTC] <Pei-gv> ChrisBurghart-ops, and ScottEllis-FL, Notice the ground velocity is slowly drifting to green.
[2018/01/22 23:12:51 UTC] <ScottEllis-FL> ChrisBurghart-ops, thanks for the Jeppesen answer. I passed it along
[2018/01/22 23:13:57 UTC] <ChrisBurghart-ops> Pei-gv, That's disappointing, too. There was turning for the Investigator not that long ago.
[2018/01/22 23:14:07 UTC] <ScottEllis-FL> Pei-gv, I noticed. Not sure what is happening.
[2018/01/22 23:14:23 UTC] <ChrisBurghart-ops> ScottEllis-FL, No problem. I'm trying to contact Erik directly as well.
[2018/01/22 23:14:24 UTC] <Pei-gv> ChrisBurghart-ops, what am I seeing? The lists of processes or the summary section on top?
[2018/01/22 23:14:39 UTC] <Pei-gv> I usually just monitor the individual processes..
[2018/01/22 23:16:39 UTC] <ChrisBurghart-ops> Pei-gv, I think mostly the summary section. If nothing is flagged with red color, then it's probably not worth your time to try to decode stuff it's showing.
[2018/01/22 23:17:43 UTC] <Pei-gv> ChrisBurghart-ops, just got a red from sdc writing/reading... I assume this is normal
[2018/01/22 23:18:20 UTC] <ChrisBurghart-ops> Yes, I would expect that during parts of the rsync.
[2018/01/22 23:22:14 UTC] <Pei-gv> drop down to 13kft
[2018/01/22 23:27:03 UTC] <Pei-gv> Huge drop out 23:26:20, more than 5seconds
[2018/01/22 23:31:40 UTC] <Pei-gv> ChrisBurghart-ops, if the system stays like this after the profiling legs, do you mind me venture out and try the mission coordinator?
[2018/01/22 23:32:49 UTC] <ChrisBurghart-ops> Pei-gv, Feel free at your discretion. You can always stop the display if it becomes an issue.
[2018/01/22 23:33:24 UTC] <Pei-gv> To my eyes, I think it runs much better today.
[2018/01/22 23:35:11 UTC] <Pei-gv> Continue until 134E and then return. Likely do sawtooth after the waypoint
[2018/01/22 23:41:11 UTC] <Pei-gv> 19:45:25 start hcrdrx with 2017 calibration file_HCR1
[2018/01/22 23:42:00 UTC] <Pei-gv> ChrisBurghart-ops, is sda the system drive? or the RAID?
[2018/01/22 23:45:01 UTC] <Pei-gv> waypoint
[2018/01/22 23:45:11 UTC] <Pei-gv> going down to 6kft
[2018/01/22 23:45:21 UTC] <Pei-gv> turnng first
[2018/01/22 23:46:30 UTC] <ChrisBurghart-ops> Pei-gv, try "df | grep sda" and see what shows up.
[2018/01/22 23:47:35 UTC] <Pei-gv> ChrisBurghart-ops, alright. That's the data drive. I see it being written on ATOP
[2018/01/22 23:53:03 UTC] <Pei-gv> start profiling
[2018/01/23 00:07:19 UTC] <Pei-gv> 00:06:25 gap
[2018/01/23 00:13:18 UTC] <Pei-gv> 00:13:10 gap
[2018/01/23 00:17:27 UTC] <Pei-gv> 10 minute leg 11.5kft
[2018/01/23 00:26:29 UTC] <Pei-gv> rim at the tip of nosecone
[2018/01/23 00:29:15 UTC] <Pei-gv> ChrisBurghart-ops, can we adjust the images to plot the maximum alititude at 25kft?
[2018/01/23 00:29:32 UTC] <Pei-gv> It just seems silly to me that we are plotting upto 50kft.
[2018/01/23 00:31:25 UTC] <ChrisBurghart-ops> Pei-gv, we can change the altitudes on the plots, but then we should probably also regenerate the images from the first two flights with the new scaling
[2018/01/23 00:33:59 UTC] <Pei-gv> 00:33:40 big gap
[2018/01/23 00:52:13 UTC] <Pei-gv> end of 10 minutes
[2018/01/23 01:03:40 UTC] <Pei-gv> top of boat
[2018/01/23 01:06:20 UTC] <Pei-gv> time on station remaining = 20~40 min
[2018/01/23 01:14:01 UTC] <Pei-gv> 01:13:20 big gap
[2018/01/23 01:19:37 UTC] <Pei-gv> 10 minutes below clouds
[2018/01/23 01:39:17 UTC] <Pei-gv> 01:38:55 gap
[2018/01/23 01:51:35 UTC] <Pei-gv> 1:50:58 gap, ~7 second?
[2018/01/23 01:59:55 UTC] <Pei-gv>   1:59:33 gap
[2018/01/23 02:02:20 UTC] <Pei-gv> sawtooth now
[2018/01/23 02:03:15 UTC] <Pei-gv> I have been watching. I do'nt think I can see any correlation between the two disks write and the gaps..
[2018/01/23 02:04:01 UTC] <Pei-gv> I saw a couple of times of em1 (network) spiked at 92% (red)
[2018/01/23 02:04:27 UTC] <ChrisBurghart-ops> Pei-gv, Maybe I missed something. Are you running with both USB disks now, or "two disks write" is something else?
[2018/01/23 02:05:17 UTC] <Pei-gv> ChrisBurghart-ops, Sorry for the confusion. I am monitoring writing actions to USB and RAID. We run only one USB disk for this flight
[2018/01/23 02:06:22 UTC] <Pei-gv> 02:06:05 gap
[2018/01/23 02:07:26 UTC] <ChrisBurghart-ops> Did you ever start a mission coordinator display? If so, was the em1 spike after that?
[2018/01/23 02:08:06 UTC] <Pei-gv> ChrisBurghart-ops, haven't gotten to that yet. plan to do that on the way home. Still profiling
[2018/01/23 02:11:43 UTC] <ChrisBurghart-ops> Pei-gv, display testing on the return ferry sounds like a good idea.
[2018/01/23 02:13:00 UTC] <Pei-gv> Getting another unresponsive period
[2018/01/23 02:13:18 UTC] <ChrisBurghart-ops> Keep an eye on atop if you can
[2018/01/23 02:13:30 UTC] <Pei-gv> Saw disk writing to USB spiked up to 101% and stuck there for a while
[2018/01/23 02:14:23 UTC] <Pei-gv> I could not get any response from Archiver on any click. Had to use 'tab" through the windows to type on chat.
[2018/01/23 02:14:51 UTC] <Pei-gv> That incident did not trigger any gap though.
[2018/01/23 02:15:52 UTC] <ChrisBurghart-ops> Now that you mention it, I've seen that behavior a couple of times, too. I wrote it off as another type of keyboard problem (and it may be)
[2018/01/23 02:18:09 UTC] <Pei-gv> ChrisBurghart-ops, I would think it's an Archiver problem. When the Archiver is responding, the mouse shows an arrow when you point.
[2018/01/23 02:18:30 UTC] <Pei-gv> 02:18:10 gap
[2018/01/23 02:19:05 UTC] <Pei-gv> another em1 85%
[2018/01/23 02:19:30 UTC] <Pei-gv> I get HawkEye display delay for a few seconds
[2018/01/23 02:20:57 UTC] <Pei-gv> When the Archiver was not responding, I got a "bar" looking icon when I use the mouse
[2018/01/23 02:21:49 UTC] <ChrisBurghart-ops> I see an abrupt change in Vr at 2:10:50 from showing high frequency pillars to a very smooth display.
[2018/01/23 02:22:42 UTC] <ChrisBurghart-ops> The high frequency pillars started when you changed from nadir to zenith pointing
[2018/01/23 02:23:51 UTC] <Pei-gv> ChrisBurghart-ops, I noticed that. It was pretty bumpy during that time. I had to hold on to things to walk to HSRL
[2018/01/23 02:25:17 UTC] <ChrisBurghart-ops> OK, it does look like the switch occurred about where you climbed out of the boundary layer.
[2018/01/23 02:25:54 UTC] <ChrisBurghart-ops> The pillars might be there in the nadir pointing data just before, but not apparent because of the color scale.
[2018/01/23 02:27:13 UTC] <Pei-gv> 2:26:40 gap
[2018/01/23 02:27:40 UTC] <Pei-gv> The disk writing spiked at 101% when the gap happened. May have correlation?
[2018/01/23 02:27:59 UTC] <Pei-gv> now display stalled for a few second
[2018/01/23 02:28:29 UTC] <Pei-gv> the disk writing stayed at 101% until HawkEye start responding again
[2018/01/23 02:28:37 UTC] <Pei-gv> em184%
[2018/01/23 02:29:37 UTC] <ScottEllis-FL> ChrisBurghart-ops, I think the pillars are evidenced in the surface vr in the nadir pointing data
[2018/01/23 02:29:42 UTC] <Pei-gv> ChrisBurghart-ops, whatever this problem is, it defintely worsen as the flight time goes on
[2018/01/23 02:30:37 UTC] <Pei-gv> 02:30:15 gap
[2018/01/23 02:30:59 UTC] <Pei-gv> busy writing sda (raid)
[2018/01/23 02:32:10 UTC] <ChrisBurghart-ops> Pei-gv, Do you mean the problem of the system getting really sluggish in responding to you?
[2018/01/23 02:32:18 UTC] <Pei-gv> ChrisBurghart-ops, can the ATOP activity be logged?
[2018/01/23 02:33:43 UTC] <Pei-gv> ChrisBurghart-ops, no. I meant the occurance of the gaps are getting more frequent as time goes
[2018/01/23 02:34:37 UTC] <ChrisBurghart-ops> And I'll look into atop logging now.
[2018/01/23 02:48:57 UTC] <ChrisBurghart-ops> Pei-gv, Here's a command to log atop results every 10 seconds:
[2018/01/23 02:49:26 UTC] <Pei-gv> ChrisBurghart-ops, you want me to try? or this is for the next flight?
[2018/01/23 02:49:44 UTC] <ChrisBurghart-ops> atop -w /home/hcr/projDir/logs/status/today/atop.log 10
[2018/01/23 02:49:45 UTC] <Pei-gv> Fire over if you want me to do it
[2018/01/23 02:50:05 UTC] <ChrisBurghart-ops> Yes, please start it now.
[2018/01/23 02:50:49 UTC] <ChrisBurghart-ops> Since it will log to the same place as other archiver log files, the log will be written to the USB disk as well.
[2018/01/23 02:51:05 UTC] <Pei-gv> Running it now
[2018/01/23 02:53:50 UTC] <Pei-gv> 101% writing definitely contribute to HawkEye display delay
[2018/01/23 02:57:43 UTC] <ChrisBurghart-ops> Pei-gv, In the future, probably better to just throw the atop into the background with '&' at the end. I assume the one you started is just sitting there quietly and not giving back your terminal...
[2018/01/23 03:22:57 UTC] <Pei-gv> 03:22:25 gap
[2018/01/23 03:23:14 UTC] <Pei-gv> Estimated time to end= 45 minutes!
[2018/01/23 03:25:45 UTC] <Pei-gv> NSCAL
[2018/01/23 03:32:59 UTC] <Pei-gv> NSCAL endded at 03:31:00\