How to recover ARGOS

Procedure to recover from BCU crash

1. Set LAS and LAN in simulation (Arbitrator GUI, Tab: Expert->Commissioning)
2. Open LGSW engineering GUI and go to Tab 'Rack I/O' (Press beside LGSW)
3. Reset BCU by 'BCU Reset' 'ON' and then 'OFF' after a view seconds
4. Go through power up sequence until LGSW is in 'STARTED UP"
5. Put LAS and LAN back in operation (Arbitrator GUI Tab: Expert-> Commissioning)
6. Press 'Prepare'

Troubleshoot:

• Item 6 fails with not allowed to change state: LAN, LAS have not been in 'STARTED UP' or 'READY_TO_OBSERVE'
  • Start power up sequence from scratch with LAN, LAS in operation, maybe you have to subvert the arbitrator state to UNKNOWN before.
• If connection with AOS, FLAO or AdSec is not working well:
  • Restart Arbitrator repeat procedure (remember to restart the arbitrator GUI if you restart the arbitrator process)

Procedure to recover from DEAD_DEVICE_SERVER

• Restart the dead server (there might be some fixing necessary, because normally they do not die without reason)
• Clear the Error with 'Clear error'
• Open LAS controller GUI, Tab 'Commissioning', skip to the previous state (at night time normally 'READY_TO_OBSERVE'), close GUI
• Open LAN controller GUI, Tab 'Commissioning', skip to the previous state (at night time normally 'READY_TO_OBSERVE'), close GUI
• Set LAS and LAN in simulation (Tab Commissioning)
• Go through power up sequence until subsystems are synchronized again (normally Prepare)
• Put LAS and LAN back in operation (Tab Commissioning)

Procedure to recover from Adsec_stop

In principal after an Adsec_stop/Adsec_start everything reconnects and looks fine. However experience showed, that it is not the full truth.We experienced often, that after such intervention with the next preset FLAO fails and after some debugging we normally ended in restarting FLAO.
Then, with the next preset often ARGOS had no communication. Therefore, we recommend to restart FLAO and ARGOS arbitrator immediately
after such intervention.

• Adsec software back up again.
• Stop FLAO software (w_stop)
• Stop ARGOS arbitrator (monit interface)
• Start FLAO software (w_start)
• Start ARGOS arbitrator (monit interface)
• Put LAS and LAN in simulation
• Go through power up sequence until subsystems are synchronized again (normally Prepare)
• Put LAS and LAN back in operation (Tab Commissioning)

Procedure to restart ARGOS arbitrator

• Restart ARGOS arbitrator (monit interface)
• Put LAS and LAN in simulation
• Go through power up sequence until subsystems are synchronized again (normally Prepare)
• Put LAS and LAN back in operation (Tab Commissioning)

Troubleshoot LAT

LAT can fail or not align for following reasons:

• Do not reach patrol field with LAT alignment
  • Nominal positions can be wrong
  • Rarer case: LM1 is not working properly
  • Rarer case: Calibration is wrong
• Alignment algorithm fails
  • Lasers are to faint

    • Currently the threshold for calculation is 600. This can be changed in the LAN config file or the exposure time is increased in the LAN lab GUI (currently (201802) set to 400000 previously 200000). It looks like there is no default setting during setup of this exposure time.

ARGOS Is not connected to the AOS

• Is the AOS enabled for ARGOS? This is done automaticualy when the telescope is authorizedfor LUCI-ARGOS.
• Was there other AO work done during the day? e.g. LBTI or LN? If so you must restart the AOS.

-- LorenzoBusoni - 23 Feb 2018