kpfcc.py

Module for preparing all data specific to from Keck Observatory’s custom made Observing Block (OB) database. This is specific to the KPF-CC program and the observatory’s infrastructure as way to power the prep kpfcc command. New observatories should write their own module to connect to a new “prep <your observatory>” command.

class astroq.queue.kpfcc.Access_KPFCC(semester_start_date, semester_length, n_nights_in_semester, today_starting_night, current_day, all_dates_dict, all_dates_array, slot_size, slots_needed_for_exposure_dict, custom_file, allocation_file, past_history, output_directory, run_weather_loss, run_band3, observatory_string, request_frame)[source]

Bases: Access

Keck Observatory-specific Access class that inherits from the base Access class. Overrides compute_altaz() and compute_clear() methods with Keck-specific implementations.

compute_allocated(): Compute boolean mask of is_allocated for all targets according to the allocated times.

compute_altaz(tel_min)[source]

Compute boolean mask of is_altaz for targets according to a minimum elevation. Specific to Keck Observatory’s K1/K2 pointing limits. This method overrides the base class method to include Keck-specific nasmyth deck constraints.

Parameters:: tel_min (float) – the minimum elevation for the telescope (ignored, uses self.tel_min instead)
Returns:: boolean mask of is_altaz for targets
Return type:: is_altaz (array)

compute_clear()[source]: Compute boolean mask of is_clear for all targets according to the clear times.

compute_custom(): Compute boolean mask of is_custom for all targets according to the custom times.

compute_future()

Compute boolean mask of is_future for all targets according to today’s current_day.

Args: :returns: boolean mask of is_altaz for targets :rtype: is_altaz (array)

compute_inter(): Compute boolean mask of is_inter for all targets according to the internight cadence.

compute_moon(): Compute boolean mask of is_moon for all targets according to the moon’s position.

get_loss_stats(weather_loss_file): Gather the loss probabilities for each night in the semester from the saved historical weather data.

observability(requests_frame, access=None)

Extract a dictionary of the available indices from the record array returned by produce_ultimate_map

Parameters:

requests_frame – DataFrame containing request information
access – Optional record array from produce_ultimate_map (if None, this function will compute it)

Returns:

Dictionary where keys are target names and values are lists of available slots per night

Return type:

df (dict)

produce_ultimate_map(running_backup_stars=False): Compute boolean mask of is_observable for all targets according to the ultimate map.

simulate_weather_losses(covariance=0.14)

Simulate nights totally lost to weather using historical data

Parameters:: covariance (float) – the added percent chance that tomorrow will be lost if today is lost
Returns:: Trues represent clear nights, Falses represent weathered nights
Return type:: is_clear (array)

astroq.queue.kpfcc._validate_datetime_format(datetime_str)[source]

Validate that datetime string follows YYYY-MM-DDTHH:MM or YYYY-MM-DDTHH:MM:SS format.

Parameters:: datetime_str (str) – The datetime string to validate
Returns:: True if format is valid, False otherwise
Return type:: bool

astroq.queue.kpfcc.analyze_bad_obs(trimmed_good, bad_OBs_values, bad_OBs_hasFields, awarded_programs, required_fields=['_id', 'metadata.semid', 'target.target_name', 'target.ra', 'target.dec', 'observation.exposure_time', 'observation.num_exposures', 'schedule.num_nights_per_semester', 'schedule.num_internight_cadence', 'schedule.desired_num_visits_per_night', 'schedule.minimum_num_visits_per_night', 'schedule.num_intranight_cadence', 'schedule.minimum_elevation', 'schedule.minimum_moon_separation', 'schedule.weather_band_1', 'schedule.weather_band_2', 'schedule.weather_band_3', 'target.gaia_id', 'target.t_eff', 'target.j_mag', 'target.g_mag', 'target.pm_ra', 'target.pm_dec', 'target.epoch', 'observation.exp_meter_threshold', 'metadata.ob_inactive'])[source]

Analyze the bad OBs and produce a count of bad OBs by semester and a histogram of bad OBs by field.

Parameters:

trimmed_good (pandas DataFrame) – the good OBs
bad_OBs_values (pandas DataFrame) – the values of the fields in the bad OBs
bad_OBs_hasFields (pandas DataFrame) – the existence of the fields in the bad OBs
awarded_programs (list) – a list of the awarded programs
required_fields (list) – a list of the required fields

Returns:

dict {metadata.semid: count of bad OBs} - bad_field_histogram: dict {field: count of times field was missing in a bad OB}

Return type:

bad_obs_count_by_semid

astroq.queue.kpfcc.apply_safety_valves(value_df, presence_df)[source]

Apply safety valve defaults to fill in missing or empty values for certain fields. This function modifies value_df and presence_df in place.

Parameters:

value_df (pandas DataFrame) – DataFrame with OB values
presence_df (pandas DataFrame) – DataFrame indicating field presence

Returns:

Modified DataFrame with safety valve defaults applied presence_df (pandas DataFrame): Modified DataFrame with presence updated

Return type:

value_df (pandas DataFrame)

astroq.queue.kpfcc.cast_columns(df)[source]

Cast columns to their appropriate data types based on the column_definitions dictionary.

Parameters:: df (pandas DataFrame) – the DataFrame to cast the columns of
Returns:: the DataFrame with the columns cast to the appropriate data types
Return type:: df (pandas DataFrame)

astroq.queue.kpfcc.compute_exposure_info(equations, gmag, thresh)[source]

astroq.queue.kpfcc.create_checks_dataframes(OBs, required_fields)[source]

Create the dataframes to determine the good and bad OBs.

Parameters:

OBs (json) – the OB information in json format
required_fields (list) – a list of the required fields that must be present

Returns:

a DataFrame with the values of the OBs presence_df (pandas DataFrame): a DataFrame with the indication of fields existing or not for the OBs all_true_mask (pandas Series): a mask indicating which OBs in the list are good.

Return type:

value_df (pandas DataFrame)

astroq.queue.kpfcc.filter_request_csv(request_df, weather_band_num)[source]

Filter request.csv file to only keep rows where weather_band_X = True

Parameters:

request_file_path (str) – Path to the request.csv file
weather_band_num (int) – Weather band number to filter by

Returns:

True if filtering was successful, False otherwise

Return type:

bool

astroq.queue.kpfcc.flatten(d, parent_key='', sep='.')[source]

Flatten a dictionary into a single level.

Parameters:

d (dict) – the nested dictionary to flatten
parent_key (str) – the parent key
sep (str) – the separator between the parent key and the child key

Returns:

a dictionary with the flattened keys and values

Return type:

items (dict)

astroq.queue.kpfcc.format_custom_csv(OBs)[source]

Format the custom.csv file from the OBs.

Parameters:: OBs (json) – the OB information in json format
Returns:: a DataFrame with the custom information, equivalent to the custom.csv file.
Return type:: custom_frame (pandas DataFrame)

astroq.queue.kpfcc.format_keck_allocation_info(allocation_file)[source]

An alternate way to produce the allocation.csv file. Read in a Keck operations schedule file.

Parameters:: allocation_file (str) – the path and filename to the downloaded csv
Returns:: a DataFrame with the allocation information, equivalent to the allocation.csv file. hours_by_program (dict): a dictionary mapping the program code to the total hours allocated to that program nights_by_program (dict): a dictionary mapping the program code to the total nights allocated to that program
Return type:: allocation_frame (pandas DataFrame)

astroq.queue.kpfcc.format_kpf_row(row, obs_time, first_available, last_available, current_day, filler_flag=False, extra=False)[source]

Format request data in the specific way needed for the script (relates to the Keck “Magiq” software’s data ingestion requirements).

Parameters:

row (dataframe) – a single row from the requests sheet dataframe
obs_time (str) – the timestamp of the night to begin the exposure according to the TTP. In format HH:MM in HST timezone
first_available (str) – the timestamp of the night where the star is first accessible. In format HH:MM in HST timezone.
last_available (str) – the timestamp of the night where the star is last accessible. In format HH:MM in HST timezone.
filler_flag (boolean) – True of the target was added in the bonus round
extra (boolean) – is this an “extra” target

Returns:

the properly formatted string to be included in the script file

Return type:

line (str)

astroq.queue.kpfcc.get_request_sheet(OBs, awarded_programs, savepath)[source]

Produce the request.csv file from the json OBs.

Parameters:

OBs (json) – the OB information in json format
awarded_programs (list) – a list of the awarded programs
savepath (str) – the path and filename where to save the request sheet

Returns:

a DataFrame with the OBs that pass the checks bad_obs_values (pandas DataFrame): a DataFrame with the values of the bad OBs fields bad_obs_hasFields (pandas DataFrame): a DataFrame with the indication of fields existing or not for thebad OBs bad_obs_count_by_semid (pandas DataFrame): a DataFrame with the count of bad OBs by semester, for admin plotting purposes bad_field_histogram (pandas DataFrame): a DataFrame with the histogram of bad OBs by field, for admin plotting purposes

Return type:

good_obs (pandas DataFrame)

astroq.queue.kpfcc.inspect_row(df_exists, df_values, row_num, required_fields=['_id', 'metadata.semid', 'target.target_name', 'target.ra', 'target.dec', 'observation.exposure_time', 'observation.num_exposures', 'schedule.num_nights_per_semester', 'schedule.num_internight_cadence', 'schedule.desired_num_visits_per_night', 'schedule.minimum_num_visits_per_night', 'schedule.num_intranight_cadence', 'schedule.minimum_elevation', 'schedule.minimum_moon_separation', 'schedule.weather_band_1', 'schedule.weather_band_2', 'schedule.weather_band_3', 'target.gaia_id', 'target.t_eff', 'target.j_mag', 'target.g_mag', 'target.pm_ra', 'target.pm_dec', 'target.epoch', 'observation.exp_meter_threshold', 'metadata.ob_inactive'])[source]

Inspect and print a summary of a specific row’s key existence and requirement status.

Parameters:

df_exists (pandas DataFrame) – the existence of the fields in the OBs
df_values (pandas DataFrame) – the values of the fields in the OBs
row_num (int) – the row number to inspect
required_fields (list) – a list of the required fields

Returns:

the email body for the inspection

Return type:

email_body (str)

astroq.queue.kpfcc.load_fit_results_h5(filename)[source]: Load fit_results and equations dict from an HDF5 file saved by save_fit_results_h5. Returns (fit_results, equations).

astroq.queue.kpfcc.plot_bad_obs_histograms(bad_obs_count_by_semid, bad_field_histogram)[source]

Plots histograms for bad_obs_count_by_semid and bad_field_histogram. X: keys, Y: values.

Parameters:

bad_obs_count_by_semid (dict) – a dictionary mapping the program code to the count of bad OBs
bad_field_histogram (dict) – a dictionary mapping the field to the count of times field was missing in a bad OB

Returns:

None

astroq.queue.kpfcc.pm_correcter(ra, dec, pmra, pmdec, current_day, equinox='2000')[source]

Update a star’s coordinates due to proper motion.

Parameters:

ra (float) – RA in degrees
dec (float) – Dec in degrees
pmra (float) – proper motion in RA (mas/yr), including cos(Dec)
pmdec (float) – proper motion in Dec (mas/yr)
equinox (str) – original epoch (e.g. ‘2000.0’)
current_day (str) – date to which to propagate (e.g. ‘2025-04-30’)

Returns:

updated coordinates as strings

Return type:

formatted_ra (str), formatted_dec (str)

astroq.queue.kpfcc.pull_OB_histories(semester)[source]

Pull the latest database OBs down to local.

Parameters:

semester (str)
histories (bool)

Returns:

data (json) - the OB information in json format

astroq.queue.kpfcc.pull_OBs(semester)[source]

Pull the latest info from Keck Observatory’s KPF-CC database OBs down to local machine. Note you must set environment variables KECK_OB_DATABASE_API_USERNAME and KECK_OB_DATABASE_API_PASSWORD to your credentials.

Parameters:: semester (str)
Returns:: data (json) - the OB information in json format

astroq.queue.kpfcc.pull_allocation_info(start_date, numdays, instrument, conversion_ratio=12.0)[source]

Pull the allocation information directly from the Keck Observatory’s operations schedule via the API.

Parameters:

start_date (str) – the start date of the allocation (day one of the semester)
numdays (int) – the number of days beyond the start_date to pull allocation information (usually ~180)
instrument (str) – the instrument to pull allocation data, here it is “KPF-CC”
conversion_ratio (float) – factor to convert nights to hours (e.g. hours per night). Default 12.0.

Returns:

a DataFrame with the allocation information, equivalent to the allocation.csv file. hours_by_program (dict): a dictionary mapping the program code to the total hours allocated to that program nights_by_program (dict): a dictionary mapping the program code to the total nights allocated to that program

Return type:

allocation_frame (pandas DataFrame)

astroq.queue.kpfcc.recompute_exposure_times(request_frame, slowdown_factor)[source]

Recompute the exposure times for the request frame based on the band number slowdown factor.

Parameters:

request_frame (pandas DataFrame) – the request.csv in dataframe format
slowdown_factor (float) – the slowdown factor to apply to the exposure times

Returns:

a list of the new exposure times based on slowdown.

Return type:

new_exptimes (list)

astroq.queue.kpfcc.sort_good_bad(OBs, awarded_programs)[source]

Sort the OBs into good and bad buckets.

Parameters:

OBs (json) – the OB information in json format
awarded_programs (list) – a list of the awarded programs

Returns:

a DataFrame with the good OBs bad_OBs_values (pandas DataFrame): a DataFrame with the values of the bad OBs fields bad_OBs_hasFields (pandas DataFrame): a DataFrame with the indication of fields existing or not for the bad OBs

Return type:

trimmed_good (pandas DataFrame)

astroq.queue.kpfcc.update_allocation_file(allocation_df, current_date)[source]

Update allocation.csv file with today’s 12-degree twilight times

Parameters:

allocation_file_path (str) – Path to the allocation.csv file
current_date (str) – Current date in YYYY-MM-DD format

Returns:

True if update was successful, False otherwise

Return type:

bool

astroq.queue.kpfcc.validate_and_convert_coordinates(df)[source]

Validate and convert RA/Dec coordinates from string format (hourangle/deg) to degrees. Removes rows with invalid coordinates and prints warnings for removed targets.

Parameters:: df (pandas DataFrame) – DataFrame with ‘ra’ and ‘dec’ columns as strings in hourangle/deg format
Returns:: DataFrame with valid coordinates converted to degrees, invalid rows removed
Return type:: df (pandas DataFrame)

astroq.queue.kpfcc.write_starlist(frame, solution_frame, night_start_time, extras, filler_stars, current_day, outputdir, version='nominal')[source]

Generate the nightly script in the format required by the Keck “Magiq” software. Backwards compatable to pre-KPF-CC observing.

Parameters:

frame (dataframe) – the request_frame of just the targets that were selected to be observed tonight
solution_frame (dataframe) – the solution attribute from the TTP model.plotly object
night_start_time (astropy time object) – Beginning of observing interval
extras (array) – starnames of “extra” stars (those not fit into the script)
filler_stars (array) – star names of the stars added in the bonus round
current_day (str) – today’s date in format YYYY-MM-DD
outputdir (str) – the directory to save the script file
version (str) – a tag for thescript (e.g. nominal, slowdown, backups, etc)

Returns:

the script file as a string

Return type:

lines (str)