Meersolar Pipeline Module

This is main module of the pipeline handling all calibration and imaging jobs.

meersolar.pipeline.basic_cal module

meersolar.pipeline.basic_cal.main()

meersolar.pipeline.basic_cal.run_applycal(msname='', field='', scan='', applymode='', flagbackup=True, gaintable=[], gainfield=[], interp=[], calwt=[], parang=False, n_threads=-1, dry_run=False)

Perform apply calibration

meersolar.pipeline.basic_cal.run_bandpass(msname='', field='', scan='', uvrange='', refant='', solint='inf', solnorm=False, combine='', gaintable=[], gainfield=[], interp=[], n_threads=-1, dry_run=False)

Perform bandpass calibration

meersolar.pipeline.basic_cal.run_basic_cal_rounds(msname, workdir, refant='', uvrange='', keep_backup=False, do_delaycal=False, perform_polcal=False, cpu_frac=0.8, mem_frac=0.8)

Perform basic calibration rounds

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Warking directory

• refant (str, optional) – Reference antenna

• uvrange (str, optional) – UV-range

• do_delaycal (bool, optional) – Do delay calibration or not

• perform_polcal (bool, optional) – Perform polarization calibration for fullpolar data

• keep_backup (bool, optional) – Keep backup of ms after each calibration rounds

• cpu_frac (float, options) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

• int – Success message

• list – Caltables

meersolar.pipeline.basic_cal.run_delaycal(msname='', field='', scan='', refant='', refantmode='flex', solint='inf', combine='', gaintable=[], gainfield=[], interp=[], n_threads=-1, dry_run=False)

Perform delay calibration

meersolar.pipeline.basic_cal.run_gaincal(msname='', field='', scan='', uvrange='', refant='', gaintype='G', solint='inf', calmode='ap', refantmode='flex', solmode='', smodel=[], rmsthresh=[], combine='', append=False, gaintable=[], gainfield=[], interp=[], n_threads=-1, dry_run=False)

Perform gain calibration

meersolar.pipeline.basic_cal.run_leakagecal(msname='', field='', scan='', uvrange='', refant='', combine='', gaintable=[], gainfield=[], interp=[], n_threads=-1, dry_run=False)

Perform relative leakage calibration (pol-conversion calibration)

meersolar.pipeline.basic_cal.run_polcal(msname='', field='', scan='', uvrange='', refant='', combine='', gaintable=[], gainfield=[], interp=[], n_threads=-1, dry_run=False)

Perform cross-hand phase calibration

meersolar.pipeline.basic_cal.run_postcal_flag(msname='', datacolumn='corrected', uvrange='', mode='rflag', n_threads=-1, memory_limit=-1, dry_run=False)

Perform apply calibration

meersolar.pipeline.basic_cal.single_round_cal_and_flag(msname, workdir, cal_round, refant, uvrange, fluxcal_scans, fluxcal_fields, phasecal_scans, phasecal_fields, phasecal_fluxes, polcal_scans=[], polcal_fields=[], do_delaycal=False, do_phasecal=False, do_leakagecal=False, do_polcal=False, do_postcal_flag=True, cpu_frac=0.8, mem_frac=0.8)

Single round calibration and post-calibration flagging

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Work directory

• cal_round (int) – Calibration round number

• refant (str) – Reference antenna

• uvrange (str) – UV-range

• fluxcal_scans (dict) – Fluxcal scans for fluxcal fields

• fluxcal_fields (list) – Fluxcal field names

• phasecal_scans (dict) – Phasecal scans for phasecal fields

• phasecal_fields (list) – Phasecal field names

• phasecal_fluxes (dict) – Phasecal fluxes

• polcal_scans (list, optional) – Polarized calibrator scans

• polcal_fields (list, optional) – Polarized calibrator fields

• do_delaycal (bool, optional) – Perform delay calibration or not

• do_phasecal (bool, optional) – Perform calibration on phasecal fields

• do_leakagecal (bool, optional) – Perform leakage calibration or nor

• do_phasecal – Perform calibration using polcal fields (Note: leakage calibration is always done using unpolarized fluxcal. This is for crossphase and polarization angle calibration)

• do_postcal_flag (bool, optional) – Peform post-calibration flagging

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

• int – Success message

• str – Caltables

meersolar.pipeline.basic_func module

class meersolar.pipeline.basic_func.LogTailHandler(logfile, logger)

Bases: FileSystemEventHandler

Continuous logging

on_modified(event)

Called when a file or directory is modified.

Parameters:

event (DirModifiedEvent or FileModifiedEvent) – Event representing file/directory modification.

class meersolar.pipeline.basic_func.RemoteLogger(job_id='default', log_id='run_default', remote_link='', password='')

Bases: Handler

Remote logging handler for posting log messages to a web endpoint.

emit(record)

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

class meersolar.pipeline.basic_func.SmartDefaultsHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)

Bases: ArgumentDefaultsHelpFormatter

meersolar.pipeline.basic_func.angular_separation_equatorial(ra1, dec1, ra2, dec2)

Calculate angular seperation between two equatorial coordinates

Parameters:

• ra1 (float) – RA of the first coordinate in degree

• dec1 (float) – DEC of the first coordinate in degree

• ra2 (float) – RA of the second coordinate in degree

• dec2 (float) – DEC of the second coordinate in degree

Returns:

Angular distance in degree

Return type:

float

meersolar.pipeline.basic_func.average_timestamp(timestamps)

Compute the average timestamp using astropy from a list of ISO 8601 strings.

Parameters:

timestamps (list) – timestamps (list of str): List of timestamp strings in ‘YYYY-MM-DDTHH:MM:SS’ format.

Returns:

Average timestamp in ‘YYYY-MM-DDTHH:MM:SS’ format.

Return type:

str

meersolar.pipeline.basic_func.baseline_names(msname)

Get baseline names

Parameters:

msname (str) – Measurement set name

Returns:

Baseline names list

Return type:

list

meersolar.pipeline.basic_func.calc_bw_smearing_freqwidth(msname, full_FoV=False, FWHM=True)

Function to calculate spectral width to procude bandwidth smearing

Parameters:

• msname (str) – Name of the measurement set

• full_FoV (bool, optional) – Consider smearing within solar disc or full FoV

• FWHM (bool, optional) – If using full FoV, consider upto FWHM or first null

Returns:

Spectral width in MHz

Return type:

float

meersolar.pipeline.basic_func.calc_cellsize(msname, num_pixel_in_psf)

Calculate pixel size in arcsec

Parameters:

• msname (str) – Name of the measurement set

• num_pixel_in_psf (float) – Number of pixels in one PSF

Returns:

Pixel size in arcsec

Return type:

int

meersolar.pipeline.basic_func.calc_dyn_range(imagename, modelname, residualname, fits_mask='')

Calculate dynamic ranges.

Parameters:

• imagename (list or str) – Image FITS file(s)

• modelname (list or str) – Model FITS file(s)

• residualname (list ot str) – Residual FITS file(s)

• fits_mask (str, optional) – FITS file mask

Returns:

• model_flux (float) – Total model flux.

• dyn_range_rms (float) – Max/RMS dynamic range.

• rms (float) – RMS of the image

meersolar.pipeline.basic_func.calc_field_of_view(msname, FWHM=True)

Calculate optimum field of view in arcsec.

Parameters:

• msname (str) – Measurement set name

• FWHM (bool, optional) – Upto FWHM, otherwise upto first null

Returns:

Field of view in arcsec

Return type:

float

meersolar.pipeline.basic_func.calc_flag_fraction(msname='', field='', scan='', n_threads=-1, dry_run=False)

Function to calculate the fraction of total data flagged.

Parameters:

• msname (str) – Name of the measurement set

• field (str, optional) – Field names

• scan (str, optional) – Scan names

• n_threads (int, optional) – Number of OpenMP threads

Returns:

Fraction of the total data flagged

Return type:

float

meersolar.pipeline.basic_func.calc_fractional_bandwidth(msname)

Calculate fractional bandwidh

Parameters:

msname (str) – Name of measurement set

Returns:

Fraction bandwidth in percentage

Return type:

float

meersolar.pipeline.basic_func.calc_maxuv(msname, chan_number=-1)

Calculate maximum UV

Parameters:

• msname (str) – Name of the measurement set

• chan_number (int, optional) – Channel number

Returns:

• float – Maximum UV in meter

• float – Maximum UV in wavelength

meersolar.pipeline.basic_func.calc_minuv(msname, chan_number=-1)

Calculate minimum UV

Parameters:

• msname (str) – Name of the measurement set

• chan_number (int, optional) – Channel number

Returns:

• float – Minimum UV in meter

• float – Minimum UV in wavelength

meersolar.pipeline.basic_func.calc_multiscale_scales(msname, num_pixel_in_psf, chan_number=-1, max_scale=16)

Calculate multiscale scales

Parameters:

• msname (str) – Name of the measurement set

• num_pixel_in_psf (float) – Number of pixels in one PSF

• max_scale (float, optional) – Maximum scale in arcmin

Returns:

Multiscale scales in pixel units

Return type:

list

meersolar.pipeline.basic_func.calc_npix_in_psf(weight, robust=0.0)

Calculate number of pixels in a PSF (could be in fraction)

Parameters:

• weight (str) – Image weighting scheme

• robust (float, optional) – Briggs weighting robust parameter (-1,1)

Returns:

Number of pixels in a PSF

Return type:

float

meersolar.pipeline.basic_func.calc_psf(msname, chan_number=-1)

Function to calculate PSF size in arcsec

Parameters:

• msname (str) – Name of the measurement set

• chan_number (int, optional) – Channel number

Returns:

PSF size in arcsec

Return type:

float

meersolar.pipeline.basic_func.calc_solar_image_stat(imagename, disc_size=18)

Calculate solar image dynamic range

Parameters:

• imagename (str) – Fits image name

• disc_size (float, optional) – Solar disc size in arcmin (default : 18)

Returns:

• float – Maximum value

• float – Minimum value

• float – RMS values

• float – Total value

• float – Mean value

• float – Median value

• float – RMS dynamic range

• float – Min-max dynamic range

meersolar.pipeline.basic_func.calc_time_smearing_timewidth(msname, full_FoV=False, FWHM=True)

Calculate maximum time averaging to avoid time smearing over full FoV.

Parameters:

• msname (str) – Measurement set name

• full_FoV (bool, optional) – Consider smearing within solar disc or full FoV

• FWHM (bool, optional) – If using full FoV, consider upto FWHM or first null

Returns:

delta_t_max – Maximum allowable time averaging in seconds.

Return type:

float

meersolar.pipeline.basic_func.ceil_to_multiple(n, base)

Round up to the next multiple

Parameters:

• n (float) – The number

• base (float) – Whose multiple will be

Returns:

The modified number

Return type:

float

meersolar.pipeline.basic_func.check_datacolumn_valid(msname, datacolumn='DATA')

Check whether a data column exists and valid

Parameters:

• msname (str) – Measurement set

• datacolumn (str, optional) – Data column string in table (e.g.,DATA, CORRECTED_DATA’, MODEL_DATA, FLAG, WEIGHT, WEIGHT_SPECTRUM, SIGMA, SIGMA_SPECTRUM)

Returns:

Whether valid data column is present or not

Return type:

bool

meersolar.pipeline.basic_func.check_scan_in_caltable(caltable, scan)

Check scan number available in caltable or not

Parameters:

• caltable (str) – Name of the caltable

• scan (int) – Scan number

Returns:

Whether scan is present in the caltable or not

Return type:

bool

meersolar.pipeline.basic_func.check_udocker_container(name)

Check whether a docker container is present or not

Parameters:

name (str) – Container name

Returns:

Whether present or not

Return type:

bool

meersolar.pipeline.basic_func.clean_shutdown(observer)

meersolar.pipeline.basic_func.correct_missing_col_subms(msname)

Correct for missing colurmns in sub-MSs

Parameters:

msname (str) – Name of the measurement set

meersolar.pipeline.basic_func.correct_solar_sidereal_motion(msname='', verbose=False, dry_run=False)

Correct sodereal motion of the Sun

Parameters:

msname (str) – Name of the measurement set

Returns:

Success message

Return type:

int

meersolar.pipeline.basic_func.create_batch_script_nonhpc(cmd, workdir, basename, write_logfile=True)

Function to make a batch script not non-HPC environment

Parameters:

• cmd (str) – Command to run

• workdir (str) – Work directory of the measurement set

• basename (str) – Base name of the batch files

• write_logfile (bool, optional) – Write log file or not

Returns:

• str – Batch file name

• str – Log file name

meersolar.pipeline.basic_func.create_circular_mask(msname, cellsize, imsize, mask_radius=20)

Create fits solar mask

Parameters:

• msname (str) – Name of the measurement set

• cellsize (float) – Cell size in arcsec

• imsize (int) – Imsize in number of pixels

• mask_radius (float) – Mask radius in arcmin

Returns:

Fits mask file name

Return type:

str

meersolar.pipeline.basic_func.create_circular_mask_array(data, radius)

Creating circular mask of a Numpy array

Parameters:

• data (numpy.array) – 2D numpy array

• radius (int) – Radius in pixels

Returns:

Mask array

Return type:

numpy.array

meersolar.pipeline.basic_func.create_logger(logname, logfile, verbose=False)

Create logger.

Parameters:

• logname (str) – Name of the log

• workdir (str, optional) – Name of the working directory

• verbose (bool, optional) – Verbose output or not

• logfile (str, optional) – Log file name

Returns:

• logger – Python logging object

• str – Log file name

meersolar.pipeline.basic_func.cutout_image(fits_file, output_file, x_deg=2)

Cutout central part of the image

Parameters:

• fits_file (str) – Input fits file

• output_file (str) – Output fits file name (If same as input, input image will be overwritten)

• x_deg (float, optional) – Size of the output image in degree

Returns:

Output image name

Return type:

str

meersolar.pipeline.basic_func.delaycal(msname='', caltable='', refant='', solint='inf', dry_run=False)

General delay calibration using CASA, not assuming any point source

Parameters:

• msname (str, optional) – Measurement set

• caltable (str, optional) – Caltable name

• refant (str, optional) – Reference antenna

• solint (str, optional) – Solution interval

Returns:

Caltable name

Return type:

str

meersolar.pipeline.basic_func.determine_noise_diode_cal_scan(msname, scan)

Determine whether a calibrator scan is a noise-diode cal scan or not

Parameters:

• msname (str) – Name of the measurement set

• scan (int) – Scan number

Returns:

Whether it is noise-diode cal scan or not

Return type:

bool

meersolar.pipeline.basic_func.do_flag_backup(msname, flagtype='flagdata')

Take a flag backup

Parameters:

• msname (str) – Measurement set name

• flagtype (str, optional) – Flag type

meersolar.pipeline.basic_func.drop_cache(path, verbose=False)

Drop file cache for a file or all files under a directory.

Parameters:

path (str) – File or directory path

meersolar.pipeline.basic_func.drop_file_cache(filepath, verbose=False)

Advise the OS to drop the given file from the page cache. Safe, per-file, no sudo required.

meersolar.pipeline.basic_func.enhance_offlimb(sunpy_map, do_sharpen=True)

Enhance off-disk emission

Parameters:

• sunpy_map (sunpy.map) – Sunpy map

• do_sharpen (bool, optional) – Sharpen images

Returns:

Off-disk enhanced emission

Return type:

sunpy.map

meersolar.pipeline.basic_func.flag_outside_uvrange(vis, uvrange, n_threads=-1, flagbackup=True)

Flag outside the given uv range

Parameters:

• vis (str) – Measurement set name

• uvrange (str) – UV-range

• n_threads (int, optional) – Number of OpenMP threads to use

• flagbackup (bool, optional) – Flag backup

meersolar.pipeline.basic_func.generate_password(length=6)

Generate secure 6-character password with letters, digits, and symbols

meersolar.pipeline.basic_func.generate_tb_map(imagename, outfile='')

Function to generate brightness temperature map

Parameters:

• imagename (str) – Name of the flux calibrated image

• outfile (str, optional) – Output brightess temperature image name

Returns:

Output image name

Return type:

str

meersolar.pipeline.basic_func.get_bad_ants(msname='', fieldnames=[], n_threads=-1, dry_run=False)

Get bad antennas

Parameters:

• msname (str) – Name of the ms

• fieldnames (list, optional) – Fluxcal field names

• n_threads (int, optional) – Number of OpenMP threads

Returns:

• list – Bad antenna list

• str – Bad antenna string

meersolar.pipeline.basic_func.get_bad_chans(msname)

Get bad channels to flag

Parameters:

msname (str) – Name of the ms

Returns:

SPW string of bad channels

Return type:

str

meersolar.pipeline.basic_func.get_band_name(msname)

Get band name

Parameters:

msname (str) – Name of the ms

Returns:

Band name (‘U’,’L’,’S’)

Return type:

str

meersolar.pipeline.basic_func.get_cal_target_scans(msname)

Get calibrator and target scans

Parameters:

msname (str) – Name of the measurement set

Returns:

• list – Target scan numbers

• list – Calibrator scan numbers

meersolar.pipeline.basic_func.get_caltable_fields(caltable)

Get caltable field names

Parameters:

caltable (str) – Caltable name

Returns:

Field names

Return type:

list

meersolar.pipeline.basic_func.get_chans_flag(msname='', field='', n_threads=-1, dry_run=False)

Get flag/unflag channel list

Parameters:

• msname (str) – Measurement set name

• field (str, optional) – Field name or ID

• n_threads (int, optional) – Number of OpenMP threads

Returns:

• list – Unflag channel list

• list – Flag channel list

meersolar.pipeline.basic_func.get_chunk_size(msname, memory_limit=-1, only_autocorr=False)

Get time chunk size for a memory limit

Parameters:

• msname (str) – Measurement set

• memory_limit (int, optional) – Memory limit

• only_autocorr (bool, optional) – Only aut-correlation

Returns:

Number of chunks

Return type:

int

meersolar.pipeline.basic_func.get_column_size(msname, only_autocorr=False)

Get time chunk size for a memory limit

Parameters:

• msname (str) – Measurement set

• only_autocorr (bool, optional) – Only auto-correlations

Returns:

A single datacolumn data size in GB

Return type:

float

meersolar.pipeline.basic_func.get_common_spw(spw1, spw2)

Return common spectral windows in merged CASA string format.

Parameters:

• spw1 (str) – First spectral window

• spw2 (str) – Second spectral window

Returns:

Merged spectral window

Return type:

str

meersolar.pipeline.basic_func.get_dask_client(n_jobs, dask_dir, cpu_frac=0.8, mem_frac=0.8, spill_frac=0.6, min_mem_per_job=-1, min_cpu_per_job=1, only_cal=False)

Create a Dask client optimized for one-task-per-worker execution, where each worker is a separate process that can use multiple threads internally.

Parameters:

• n_jobs (int) – Number of MS tasks (ideally = number of MS files)

• dask_dir (str) – Dask temporary directory

• cpu_frac (float) – Fraction of total CPUs to use

• mem_frac (float) – Fraction of total memory to use

• spill_frac (float, optional) – Spill to disk at this fraction

• min_mem_per_job (float, optional) – Minimum memory per job

• min_cpu_per_job (int, optional) – Minimum CPU threads per job

• only_cal (bool, optional) – Only calculate number of workers

Returns:

• client (dask.distributed.Client) – Dask clinet

• cluster (dask.distributed.LocalCluster) – Dask cluster

• n_workers (int) – Number of workers

• threads_per_worker (int) – Threads per worker to use

meersolar.pipeline.basic_func.get_datadir()

Get package data directory

meersolar.pipeline.basic_func.get_emails()

meersolar.pipeline.basic_func.get_fluxcals(msname)

Get fluxcal field names and scans

Parameters:

msname (str) – Name of the ms

Returns:

• list – Fluxcal field names

• dict – Fluxcal scans

meersolar.pipeline.basic_func.get_good_chans(msname)

Get good channel range to perform gaincal

Parameters:

msname (str) – Name of the ms

Returns:

SPW string

Return type:

str

meersolar.pipeline.basic_func.get_jobid()

Get MeerSOLAR Job ID with millisecond-level uniqueness.

Returns:

Job ID in the format YYYYMMDDHHMMSSmmm (milliseconds)

Return type:

int

meersolar.pipeline.basic_func.get_logid(logfile)

Get log id for remote logger from logfile name

meersolar.pipeline.basic_func.get_meermap(fits_image, band='', do_sharpen=False)

Make MeerKAT sunpy map

Parameters:

• fits_image (str) – MeerKAT fits image

• band (str, optional) – Band name

• do_sharpen (bool, optional) – Sharpen the image

Returns:

Sunpy map

Return type:

sunpy.map

meersolar.pipeline.basic_func.get_meersolar_cachedir()

meersolar.pipeline.basic_func.get_ms_scan_size(msname, scan, only_autocorr=False)

Get measurement set scan size

Parameters:

• msname (str) – Measurement set

• scan (int) – Scan number

• only_autocorr (bool, optional) – Only for auto-correlations

Returns:

Size in GB

Return type:

float

meersolar.pipeline.basic_func.get_ms_scans(msname)

Get scans of the measurement set

Parameters:

msname (str) – Measurement set

Returns:

Scan list

Return type:

list

meersolar.pipeline.basic_func.get_ms_size(msname, only_autocorr=False)

Get measurement set total size

Parameters:

• msname (str) – Measurement set name

• only_autocorr (bool, optional) – Only auto-correlation

Returns:

Size in GB

Return type:

float

meersolar.pipeline.basic_func.get_multiscale_bias(freq, bias_min=0.6, bias_max=0.9)

Get frequency dependent multiscale bias

Parameters:

• freq (float) – Frequency in MHz

• bias_min (float, optional) – Minimum bias at minimum L-band frequency

• bias_max (float, optional) – Maximum bias at maximum L-band frequency

Returns:

Multiscale bias patrameter

Return type:

float

meersolar.pipeline.basic_func.get_nprocess_meersolar(jobid)

Get numbers of MeerSOLAR processes currently running

Parameters:

• workdir (str) – Work directory name

• jobid (int) – MeerSOLAR Job ID

Returns:

Number of running processes

Return type:

int

meersolar.pipeline.basic_func.get_optimal_image_interval(msname, temporal_tol_factor=0.1, spectral_tol_factor=0.1, chan_range='', timestamp_range='', max_nchan=-1, max_ntime=-1)

Get optimal image spectral temporal interval such that total flux max-median in each chunk is within tolerance limit

Parameters:

• msname (str) – Name of the measurement set

• temporal_tol_factor (float, optional) – Tolerance factor for temporal variation (default : 0.1, 10%)

• spectral_tol_factor (float, optional) – Tolerance factor for spectral variation (default : 0.1, 10%)

• chan_range (str, optional) – Channel range

• timestamp_range (str, optional) – Timestamp range

• max_nchan (int, optional) – Maxmium number of spectral chunk

• max_ntime (int, optional) – Maximum number of temporal chunk

Returns:

• int – Number of time intervals to average

• int – Number of channels to averages

meersolar.pipeline.basic_func.get_phasecals(msname)

Get phasecal field names and scans

Parameters:

msname (str) – Name of the ms

Returns:

• list – Phasecal field names

• dict – Phasecal scans

• dict – Phasecal flux

meersolar.pipeline.basic_func.get_phasecenter(msname, field)

Get phasecenter of the measurement set

Parameters:

msname (str) – Name of the measurement set

Returns:

• float – RA in degree

• float – DEC in degree

meersolar.pipeline.basic_func.get_polcals(msname)

Get polarization calibrator field names and scans

Parameters:

msname (str) – Name of the ms

Returns:

• list – Polcal field names

• dict – Polcal scans

meersolar.pipeline.basic_func.get_refant(msname='', n_threads=-1, dry_run=False)

Get reference antenna

Parameters:

• msname (str) – Name of the measurement set

• n_threads (int, optional) – Number of OpenMP threads

Returns:

Reference antenna

Return type:

str

meersolar.pipeline.basic_func.get_remote_logger_link()

meersolar.pipeline.basic_func.get_solar_elevation_MeerKAT(date_time='')

Get solar elevation at MeerKAT at a time

Parameters:

date_time (str) – Date and time in ‘yyyy-mm-ddTHH:MM:SS’ format (default: current time)

Returns:

Solar elevation in degree

Return type:

float

meersolar.pipeline.basic_func.get_submsname_scans(msname)

Get sub-MS names for each scans of an multi-MS

Parameters:

msname (str) – Name of the measurement set

Returns:

• list – msname list

• list – scan list

meersolar.pipeline.basic_func.get_suvi_map(obs_date, obs_time, workdir, wavelength=195)

Get GOES SUVI map

Parameters:

• obs_date (str) – Observation date in yyyy-mm-dd format

• obs_time (str) – Observation time in hh:mm format

• workdir (str) – Work directory

• wavelength (float, optional) – Wavelength, options: 94, 131, 171, 195, 284, 304 Å

Returns:

Sunpy SUVIMap

Return type:

sunpy.map

meersolar.pipeline.basic_func.get_target_fields(msname)

Get target fields

Parameters:

msname (str) – Name of the measurement set

Returns:

• list – Target field names

• dict – Target field scans

meersolar.pipeline.basic_func.get_timeranges_for_scan(msname, scan, time_interval, time_window, quack_timestamps=-1)

Get time ranges for a scan with certain time intervals

Parameters:

• msname (str) – Name of the measurement set

• scan (int) – Scan number

• time_interval (float) – Time interval in seconds

• time_window (float) – Time window in seconds

• quack_timestamps (int, optional) – Number of timestamps ignored at the start and end of each scan

Returns:

List of time ranges

Return type:

list

meersolar.pipeline.basic_func.get_unflagged_antennas(msname='', scan='', n_threads=-1, dry_run=False)

Get unflagged antennas of a scan

Parameters:

• msname (str) – Name of the measurement set

• scan (str) – Scans

• n_threads (int, optional) – Number of OpenMP threads

Returns:

• numpy.array – Unflagged antenna names

• numpy.array – Flag fraction list

meersolar.pipeline.basic_func.get_valid_scans(msname, field='', min_scan_time=1, n_threads=-1)

Get valid list of scans

Parameters:

• msname (str) – Measurement set name

• min_scan_time (float) – Minimum valid scan time in minute

Returns:

Valid scan list

Return type:

list

meersolar.pipeline.basic_func.has_space(path, required_gb)

meersolar.pipeline.basic_func.init_logger(logname, logfile, jobname='', password='')

Initialize a local + optional remote logger with watchdog-based tailing.

Parameters:

• logname (str) – Logger name.

• logfile (str) – Path to the local logfile to also monitor.

• jobname (str, optional) – Remote logger job ID.

• password (str) – Password used for remote authentication.

Returns:

logger – Configured logger instance.

Return type:

logging.Logger

meersolar.pipeline.basic_func.init_udocker()

meersolar.pipeline.basic_func.initialize_wsclean_container(name='meerwsclean')

Initialize WSClean container

Parameters:

name (str) – Name of the container

Returns:

Whether initialized successfully or not

Return type:

bool

meersolar.pipeline.basic_func.limit_threads(n_threads=-1)

Limit number of threads usuage

Parameters:

n_threads (int, optional) – Number of threads

meersolar.pipeline.basic_func.make_ds_plot(dsfiles, plot_file=None, showgui=False)

Make dynamic spectrum plot

Parameters:

• dsfile (list) – DS files list

• plot_file (str, optional) – Plot file name to save the plot

• showgui (bool, optional) – Show GUI

Returns:

Plot name

Return type:

str

meersolar.pipeline.basic_func.make_freqavg_image(wsclean_images, outfile_name, keep_wsclean_images=True)

Convert WSClean images into a frequency averaged image

Parameters:

• wsclean_images (list) – List of WSClean images.

• outfile_name (str) – Name of the output file.

• keep_wsclean_images (bool, optional) – Whether to retain the original WSClean images (default: True).

Returns:

Output image name.

Return type:

str

meersolar.pipeline.basic_func.make_meer_overlay(meerkat_image, suvi_wavelength=195, plot_file_prefix=None, plot_meer_colormap=True, enhance_offdisk=True, contour_levels=[0.05, 0.1, 0.2, 0.4, 0.6, 0.8], do_sharpen_suvi=True, xlim=[-1600, 1600], ylim=[-1600, 1600], extensions=['png'], outdirs=[], showgui=False)

Make overlay of MeerKAT image on GOES SUVI image

Parameters:

• meerkat_image (str) – MeerKAT image

• suvi_wavelength (float, optional) – GOES SUVI wavelength, options: 94, 131, 171, 195, 284, 304 Å

• plot_file_prefix (str, optional) – Plot file prefix name

• plot_meer_colormap (bool, optional) – Plot MeerKAT map colormap

• enhance_offdisk (bool, optional) – Enhance off-disk emission

• contour_levels (list, optional) – Contour levels in fraction of peak

• do_sharpen_suvi (bool, optional) – Do sharpen SUVI images

• xlim (list, optional) – X-axis limit in arcsec

• tlim (list, optional) – Y-axis limit in arcsec

• extensions (list, optional) – Image file extensions

• outdirs (list, optional) – Output directories for each extensions

• showgui (bool, optional) – Show GUI

Returns:

Plot file names

Return type:

list

meersolar.pipeline.basic_func.make_solar_DS(msname, workdir, ds_file_name='', extension='png', target_scans=[], scans=[], merge_scan=False, showgui=False, cpu_frac=0.8, mem_frac=0.8)

Make solar dynamic spectrum and plots

Parameters:

• msname (str) – Measurement set name’

• workdir (str) – Work directory

• ds_file_name (str, optional) – DS file name prefix

• extension (str, optional) – Image file extension

• target_scans (list, optional) – Target scans

• scans (list, optional) – Scan list

• merge_scan (bool, optional) – Merge scans in one plot or not

• showgui (bool, optional) – Show GUI

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

meersolar.pipeline.basic_func.make_stokes_wsclean_imagecube(wsclean_images, outfile_name, keep_wsclean_images=True)

Convert WSClean images into a Stokes cube image.

Parameters:

• wsclean_images (list) – List of WSClean images.

• outfile_name (str) – Name of the output file.

• keep_wsclean_images (bool, optional) – Whether to retain the original WSClean images (default: True).

Returns:

Output image name.

Return type:

str

meersolar.pipeline.basic_func.make_timeavg_image(wsclean_images, outfile_name, keep_wsclean_images=True)

Convert WSClean images into a time averaged image

Parameters:

• wsclean_images (list) – List of WSClean images.

• outfile_name (str) – Name of the output file.

• keep_wsclean_images (bool, optional) – Whether to retain the original WSClean images (default: True).

Returns:

Output image name.

Return type:

str

meersolar.pipeline.basic_func.max_time_solar_smearing(msname)

Max allowable time averaging to avoid solar motion smearing.

Parameters:

msname (str) – Measurement set name

Returns:

t_max – Maximum time averaging in seconds.

Return type:

float

meersolar.pipeline.basic_func.merge_caltables(caltables, merged_caltable, append=False, keepcopy=False)

Merge multiple same type of caltables

Parameters:

• caltables (list) – Caltable list

• merged_caltable (str) – Merged caltable name

• append (bool, optional) – Append with exisiting caltable

• keepcopy (bool, opitonal) – Keep input caltables or not

Returns:

Merged caltable

Return type:

str

meersolar.pipeline.basic_func.mjdsec_to_timestamp(mjdsec, str_format=0)

Convert CASA MJD seceonds to CASA timestamp

Parameters:

• mjdsec (float) – CASA MJD seconds

• str_format (int) – Time stamp format (0: yyyy-mm-ddTHH:MM:SS.ff, 1: yyyy/mm/dd/HH:MM:SS.ff, 2: yyyy-mm-dd HH:MM:SS)

Returns:

CASA time stamp in UTC at ISOT format

Return type:

str

meersolar.pipeline.basic_func.move_to_sun(msname, only_uvw=False)

Move the phasecenter of the measurement set at the center of the Sun (Assuming ms has one scan)

Parameters:

• msname (str) – Name of the measurement set

• only_uvw (bool, optional) – Note: This is required when visibilities are properly phase rotated in correlator to track the Sun, but while creating the MS, UVW values are estimated using a wrong phase center at the start of solar center at the start.

Returns:

Success message

Return type:

int

meersolar.pipeline.basic_func.ping_logger(jobid, remote_jobid, stop_event, remote_link='')

Ping a job-specific keep-alive endpoint periodically until stop_event is set.

meersolar.pipeline.basic_func.plot_goes_full_timeseries(msname, workdir, plot_file_prefix=None, extension='png', showgui=False)

Plot GOES full time series on the day of observation

Parameters:

• msname (str) – Measurement set

• workdir (str) – Work directory

• plot_file_prefix (str, optional) – Plot file name prefix

• extension (str, optional) – Save file extension

• showgui (bool, optional) – Show GUI

Returns:

Plot file name

Return type:

str

meersolar.pipeline.basic_func.plot_in_hpc(fits_image, draw_limb=False, extensions=['png'], outdirs=[], plot_range=[], power=0.5, xlim=[-1600, 1600], ylim=[-1600, 1600], contour_levels=[], band='', showgui=False)

Function to convert MeerKAT image into Helioprojective co-ordinate

Parameters:

• fits_image (str) – Name of the fits image

• draw_limb (bool, optional) – Draw solar limb or not

• extensions (list, optional) – Output file extensions

• outdirs (list, optional) – Output directories for each extensions

• plot_range (list, optional) – Plot range

• power (float, optional) – Power stretch

• xlim (list) – X axis limit in arcsecond

• ylim (list) – Y axis limit in arcsecond

• contour_levels (list, optional) – Contour levels in fraction of peak, both positive and negative values allowed

• band (str, optional) – Band name

• showgui (bool, optional) – Show GUI

Returns:

• outfiles – Saved plot file names

• sunpy.Map – MeerKAT image in helioprojective co-ordinate

meersolar.pipeline.basic_func.radec_sun(msname)

RA DEC of the Sun at the start of the scan

Parameters:

msname (str) – Name of the measurement set

Returns:

• str – RA DEC of the Sun in J2000

• str – RA string

• str – DEC string

• float – RA in degree

• float – DEC in degree

meersolar.pipeline.basic_func.reset_weights_and_flags(msname='', restore_flag=True, n_threads=-1, force_reset=False, dry_run=False)

Reset weights and flags for the ms

Parameters:

• msname (str) – Measurement set

• restore_flag (bool, optional) – Restore flags or not

• n_threads (int, optional) – Number of OpenMP threads

• force_reset (bool, optional) – Force reset

meersolar.pipeline.basic_func.run_chgcenter(msname, ra, dec, only_uvw=False, container_name='meerwsclean', verbose=False, dry_run=False)

Run chgcenter inside a udocker container (no root permission required).

Parameters:

• msname (str) – Name of the measurement set

• ra (str) – RA can either be 00h00m00.0s or 00:00:00.0

• dec (str) – Dec can either be 00d00m00.0s or 00.00.00.0

• only_uvw (bool, optional) – Update only UVW values Note: This is required when visibilities are properly phase rotated in correlator, but while creating the MS, UVW values are estimated using a wrong phase center.

• container_name (str, optional) – Container name

• verbose (bool, optional) – Verbose output

Returns:

Success message

Return type:

int

meersolar.pipeline.basic_func.run_limited_memory_task(task, dask_dir='/tmp', timeout=30)

Run a task for a limited time, then kill and return memory usage.

Parameters:

• task (dask.delayed) – Dask delayed task object

• timeout (int) – Time in seconds to let the task run

Returns:

Memory used by task (in GB)

Return type:

float

meersolar.pipeline.basic_func.run_solar_sidereal_cor(msname='', only_uvw=False, container_name='meerwsclean', verbose=False, dry_run=False)

Run chgcenter inside a udocker container to correct solar sidereal motion (no root permission required).

Parameters:

• msname (str) – Name of the measurement set

• only_uvw (bool, optional) – Update only UVW values Note: This is required when visibilities are properly phase rotated in correlator to track the Sun, but while creating the MS, UVW values are estimated using the first phasecenter of the Sun.

• container_name (str, optional) – Container name

• verbose (bool, optional) – Verbose output or not

Returns:

Success message

Return type:

int

meersolar.pipeline.basic_func.run_wsclean(wsclean_cmd, container_name='meerwsclean', check_container=False, verbose=False, dry_run=False)

Run WSClean inside a udocker container (no root permission required).

Parameters:

• wsclean_cmd (str) – Full WSClean command as a string.

• container_name (str, optional) – Container name

• check_container (bool, optional) – Check container presence or not

• verbose (bool, optional) – Verbose output or not

Returns:

Success message

Return type:

int

meersolar.pipeline.basic_func.save_in_hpc(fits_image, outdir='', xlim=[-1600, 1600], ylim=[-1600, 1600])

Save solar image in helioprojective coordinates

Parameters:

• fits_image (str) – FITS image name

• outdir (str, optional) – Output directory

• xlim (list) – X axis limit in arcsecond

• ylim (list) – Y axis limit in arcsecond

Returns:

FITS image in helioprojective coordinate

Return type:

str

meersolar.pipeline.basic_func.save_main_process_info(pid, jobid, msname, workdir, outdir, cpu_frac, mem_frac)

Save MeerSOLAR main processes info

Parameters:

• pid (int) – Main job process id

• jobid (int) – MeerSOLAR Job ID

• msname (str) – Main measurement set

• workdir (str) – Work directory

• outdir (str) – Output directory

• cpu_frac (float) – CPU fraction of the job

• mem_frac (float) – Mempry fraction of the job

Returns:

Job info file name

Return type:

str

meersolar.pipeline.basic_func.save_pid(pid, pid_file)

Save PID

Parameters:

• pid (int) – Process ID

• pid_file (str) – File to save

meersolar.pipeline.basic_func.scans_in_timerange(msname='', timerange='', dry_run=False)

Get scans in the given timerange

Parameters:

• msname (str) – Measurement set

• timerange (str) – Time range with date and time

Returns:

Scan dict for timerange

Return type:

dict

meersolar.pipeline.basic_func.shm_or_tmp(required_gb, workdir, prefix='meersolar_', verbose=False)

Create a temporary working directory: 1. Try /dev/shm if it has required space 2. Else TMPDIR if set and has space 4. Else work directory Temporarily sets TMPDIR to the selected path during the context. Cleans up after use.

Parameters:

• required_gb (float) – Required disk space in GB

• workdir (str) – Fall back work directory

• prefix (str, optional) – Temp directory prefix

• verbose (bool, optional) – Verbose

meersolar.pipeline.basic_func.split_into_chunks(lst, target_chunk_size)

Split a list into equal number of elements

Parameters:

• lst (list) – List of numbers

• target_chunk_size (int) – Number of elements per chunk

Returns:

Chunked list

Return type:

list

meersolar.pipeline.basic_func.split_noise_diode_scans(msname='', noise_on_ms='', noise_off_ms='', field='', scan='', datacolumn='data', n_threads=-1, dry_run=True)

Split noise diode on and off timestamps into two seperate measurement sets

Parameters:

• msname (str) – Measurement set

• noise_on_ms (str, optional) – Noise diode on ms

• noise_off_ms (str, optional) – Noise diode off ms

• field (str, optional) – Field name or id

• scan (str, optional) – Scan number

• datacolumn (str, optional) – Data column to split

• n_threads (int, optional) – Number of OpenMP threads

Returns:

splited ms names

Return type:

tuple

meersolar.pipeline.basic_func.suppress_casa_output()

meersolar.pipeline.basic_func.timestamp_to_mjdsec(timestamp, date_format=0)

Convert timestamp to mjd second.

Parameters:

• timestamp (str) – Time stamp to convert

• date_format (int, optional) –

  Datetime string format

  0: ‘YYYY/MM/DD/hh:mm:ss’

  1: ‘YYYY-MM-DDThh:mm:ss’

  2: ‘YYYY-MM-DD hh:mm:ss’

  3: ‘YYYY_MM_DD_hh_mm_ss’

Returns:

Return correspondong MJD second of the day

Return type:

float

meersolar.pipeline.basic_func.tmp_with_cache_rel(required_gb, workdir, prefix='meersolar_', verbose=False)

Combined context manager: - Uses shm_or_tmp() for workspace - Drops kernel page cache for all files in that directory on exit

Parameters:

• required_gb (float) – Required disk space in GB

• workdir (str) – Fall back work directory

• prefix (str, optional) – Temp directory prefix

• verbose (bool, optional) – Verbose

meersolar.pipeline.do_apply_basiccal module

meersolar.pipeline.do_apply_basiccal.applysol(msname='', gaintable=[], gainfield=[], interp=[], parang=False, applymode='calflag', overwrite_datacolumn=False, n_threads=-1, memory_limit=-1, force_apply=False, soltype='basic', do_post_flag=False, dry_run=False)

Apply flux calibrated and attenuation calibrated solutions

Parameters:

• msname (str) – Measurement set

• gaintable (list, optional) – Caltable list

• gainfield (list, optional) – Gain field list

• interp (list, optional) – Gain interpolation

• parang (bool, optional) – Parallactic angle apply or not

• applymode (str, optional) – Apply mode

• overwrite_datacolumn (bool, optional) – Overwrite data column with corrected solutions

• n_threads (int, optional) – Number of OpenMP threads

• memory_limit (float, optional) – Memory limit in GB

• force_apply (bool, optional) – Force to apply solutions if it is already applied

• soltype (str, optional) – Solution type

• do_post_flag (bool, optional) – Do post calibration flagging

Returns:

Success message

Return type:

int

meersolar.pipeline.do_apply_basiccal.filter_outliers(data, threshold=5, max_iter=3)

Filter outliers and perform cubic spline fitting

Parameters:

• y (numpy.array) – Y values

• threshold (float) – Threshold of filtering

• max_iter (int) – Maximum number of iterations

Returns:

Clean Y-values

Return type:

numpy.array

meersolar.pipeline.do_apply_basiccal.interpolate_nans(data)

Linearly interpolate NaNs in 1D array.

meersolar.pipeline.do_apply_basiccal.main()

meersolar.pipeline.do_apply_basiccal.run_all_applysol(mslist, workdir, caldir, use_only_bandpass=False, overwrite_datacolumn=False, applymode='calflag', force_apply=False, do_post_flag=False, cpu_frac=0.8, mem_frac=0.8)

Apply self-calibrator solutions on all target scans

Parameters:

• mslist (str) – Measurement set list

• workdir (str) – Working directory

• caldir (str) – Calibration directory

• use_only_bandpass (bool, optional) – Use only bandpass solutions

• overwrite_datacolumn (bool, optional) – Overwrite data column or not

• applymode (str, optional) – Apply mode

• force_apply (bool, optional) – Force to apply solutions even already applied

• do_post_flag (bool, optional) – Do post calibration flagging

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Calibrated target scans

Return type:

list

meersolar.pipeline.do_apply_basiccal.scale_bandpass(bandpass_table, att_table, freqavg=10)

Scale a bandpass calibration table using attenuation data.

Parameters:

• bandpass_table (str) – Input bandpass calibration table.

• att_table (str) – NumPy .npy file containing attenuation frequency and values.

• freqavg (float, optional) – Frequency average in MHz for polynomial fitting (default is 10 MHz). Final table has same number of channels as input.

Returns:

Name of the output table.

Return type:

str

meersolar.pipeline.do_apply_selfcal module

meersolar.pipeline.do_apply_selfcal.main()

meersolar.pipeline.do_apply_selfcal.run_all_applysol(mslist, workdir, caldir, overwrite_datacolumn=False, applymode='calonly', force_apply=False, cpu_frac=0.8, mem_frac=0.8)

Apply self-calibrator solutions on all target scans

Parameters:

• mslist (str) – Measurement set list

• workdir (str) – Working directory

• caldir (str) – Calibration directory

• overwrite_datacolumn (bool, optional) – Overwrite data column or not

• applymode (str, optional) – Apply mode

• force_apply (bool, optional) – Force to apply solutions even already applied

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Calibrated target scans

Return type:

list

meersolar.pipeline.do_fluxcal module

meersolar.pipeline.do_fluxcal.estimate_att(msname, workdir, noise_on_caltable, noise_off_caltable, noise_diode_flux_scan, valid_target_scans, time_window=900, cpu_frac=0.8, mem_frac=0.8)

Estimate attenaution scaling

Parameters:

• msname (str) – Measurement set name

• workdir (str) – Working directory

• noise_on_caltable (int) – Caltable with noise diode on

• noise_off_caltable (str) – Caltable with noise diode off

• noise_diode_flux_scan (float) – Fluxcal scan with noise diode

• valid_target_scans (list) – Valid list of target scans

• time_window (float, optional) – Time window in second to use

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

• int – Success message

• dict – Dictionary with attenuation spectra for each solar scan

• list – Attenuation spectra file list

meersolar.pipeline.do_fluxcal.get_att_per_ant(cal_msname, source_msname, scale_factor, ant_list=[])

Get per antenna attenuatioin array

Parameters:

• cal_msname (str) – Fluxcal scan with noise diode

• source_msname (str) – Source scan with noise diode

• scale_factor (numpy.array) – Scaling factor for on-off gain offset in fluxcal scan

• ant_list (list, optional) – Antenna list, default: all antennas

Returns:

Attenuation per antenna array. Shape : npol, nchan, nantenna

Return type:

numpy.array

meersolar.pipeline.do_fluxcal.get_on_off_power(msname='', scale_factor='', ant_list=[], dry_run=False)

Get noise diode on and off power averaged over antennas

Parameters:

• msname (str) – Measurement set name

• ant_list (list) – Antenna id list

Returns:

Spectra of power difference

Return type:

numpy.array

meersolar.pipeline.do_fluxcal.get_power_diff(cal_msname='', source_msname='', on_cal='', off_cal='', n_threads=-1, memory_limit=-1, dry_run=False)

Estimate power level difference between alternative correlator dumps.

Parameters:

• cal_msname (str) – Fluxcal measurement set

• source_msname (str) – Source measurement set

• on_cal (str) – Noise diode on caltable

• off_cal (str) – Noise diode off caltable

• n_threads (int, optional) – Number of OpenMP threads

• memory_limit (float, optional) – Memory limit in GB

Returns:

• numpy.array – Attenuation spectra for both polarizations avergaed over all antennas

• numpy.array – Attenuation spectra for both polarizations for all antennas

meersolar.pipeline.do_fluxcal.main()

meersolar.pipeline.do_fluxcal.run_noise_cal(msname, workdir, keep_backup=False, cpu_frac=0.8, mem_frac=0.8)

Perform flux calibration using noise diode

Parameters:

• msname (str) – Measurement set

• workdir (str) – Working directory

• keep_backup (bool, optional) – Keep backup

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

• int – Success message

• dict – Attenuation values for different scans

• list – File list saved attenuation values for different scans

meersolar.pipeline.do_fluxcal.split_autocorr(msname, workdir, scan_list, time_window=-1, cpu_frac=0.8, mem_frac=0.8)

Split auto-correlations

Parameters:

• msname (str) – Measurement set

• workdir (str) – Working directory

• scan_list (list) – Scan list

• time_window (float, optional) – Time window in seconds from start time of the scan

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Splited ms list

Return type:

list

meersolar.pipeline.do_fluxcal.split_casatask(msname='', outputvis='', scan='', time_range='', n_threads=-1, dry_run=False)

meersolar.pipeline.do_imaging module

meersolar.pipeline.do_imaging.main()

meersolar.pipeline.do_imaging.perform_imaging(msname='', workdir='', datacolumn='CORRECTED_DATA', freqrange='', timerange='', imagedir='', imsize=1024, cellsize=2, nchan=1, ntime=1, pol='I', weight='briggs', robust=0.0, minuv=0, threshold=1.0, use_multiscale=True, use_solar_mask=True, mask_radius=25, savemodel=True, saveres=True, ncpu=-1, mem=-1, band='', cutout_rsun=2.5, make_overlay=True, make_plots=True, logfile='imaging.log', dry_run=False)

Perform spectropolarimetric snapshot imaging of a ms

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Work directory name

• datacolumn (str, optional) – Data column

• freqrange (str, optional) – Frequency range to image

• imagedir (str, optional) – Image directory name (default: workdir). Images, models, residuals will be saved in directories named images. models, residuals inside imagedir

• imsize (int, optional) – Image size in pixels

• cellsize (float, optional) – Cell size in arcseconds

• nchan (int, optional) – Number of spectral channels

• ntime (int, optional) – Number of temporal slices

• pol (str, optional) – Stokes parameters to image

• weight (str, optional) – Image weighting scheme

• robust (float, optional) – Briggs weighting robustness parameter

• minuv (float, optional) – Minimum UV-lambda to be used in imaging

• threshold (float, optional) – CLEAN threshold

• use_multiscale (bool, optional) – Use multiscale or not

• use_solar_mask (bool, optional) – Use solar mask

• mask_radius (float, optional) – Mask radius in arcminute

• savemodel (bool, optional) – Save model images or not

• saveres (bool, optional) – Save residual images or not

• band (str, optional) – Band name

• cutout_rsun (float, optional) – Cutout image size in solar radii from center (default: 2.5 solar radii)

• make_overlay (bool, optional) – Make SUVI MeerKAT overlay

• make_plots (bool, optional) – Make radio map helioprojective plots

• logfile (str, optional) – Log file name

• ncpu (int, optional) – Number of CPU threads to use

• mem (float, optional) – Memory in GB to use

Returns:

• int – Success message

• list – List of images [[images],[models],[residuals]]

meersolar.pipeline.do_imaging.rename_image(imagename, imagedir='', pol='', band='', cutout_rsun=2.5, make_overlay=True, make_plots=True)

Rename and move image to image directory

Parameters:

• imagename (str) – Image name

• imagedir (str, optional) – Image directory (default given image directory)

• pol (str, optional) – Stokes parameters

• band (str, optional) – Observing band

• cutout_rsun (float, optional) – Cutout in solar radii from center (default: 2.5 solar radii)

• make_overlay (bool, optional) – Make overlay on SUVI

• make_plots (bool, optional) – Make radio map plot in helioprojective coordinates

Returns:

New imagename with full path

Return type:

str

meersolar.pipeline.do_imaging.run_all_imaging(mslist=[], mainlogger=None, workdir='', outdir='', freqrange='', timerange='', datacolumn='CORRECTED_DATA', freqres=-1, timeres=-1, weight='briggs', robust=0.0, minuv=0, pol='I', threshold=1.0, use_multiscale=True, use_solar_mask=True, imaging_params={}, savemodel=False, saveres=False, band='', cutout_rsun=2.5, make_overlay=True, make_plots=True, cpu_frac=0.8, mem_frac=0.8, logfile='imaging.log')

Run spectropolarimetric snapshot imaging on a list of measurement sets

Parameters:

• mslist (list) – Measurement set list

• mainlogger (str) – Python logger

• workdir (str) – Work directory

• outdir (str) – Output directory

• freqrange (str, optional) – Frequency range to image

• timerange (str, optional) – Time range

• datacolumn (str, optional) – Data column

• freqres (float, optional) – Frequency resolution of spectral chunk in MHz

• timeres (float, optional) – Time resolution of temporal chunk in seconds

• weight (str, optional) – Image weighting

• robust (float, optional) – Briggs weighting robust parameter

• minuv (float, optional) – Minimum UV-lambda to use in imaging

• pol (str, optional) – Stokes parameters to image

• threshold (float, optional) – CLEAN threshold

• use_multiscale (bool, optional) – Use multiscale or not

• use_solar_mask (bool, optional) – Use solar mask

• savemodel (bool, optional) – Save model images or not

• saveres (bool, optional) – Save residual images or not

• band (str, optional) – Band name

• cutout_rsun (float, optional) – Cutout image size from center in solar radii (default : 2.5 solar radii)

• make_overlay (bool, optional) – Make SUVI MeerKAT overlay

• make_plots (bool, optional) – Make radio image helioprojective plots

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.do_partition module

meersolar.pipeline.do_partition.main()

meersolar.pipeline.do_partition.partion_ms(msname, outputms, workdir, fields='', scans='', width=1, timebin='', fullpol=False, datacolumn='DATA', cpu_frac=0.8, mem_frac=0.8)

Perform mstransform of a single scan

Parameters:

• msname (str) – Name of the measurement set

• outputms (str) – Output ms name

• workdir (str) – Work directory

• field (str, optional) – Fields to be splited

• scans (str, optional) – Scans to split

• width (int, optional) – Number of channels to average

• timebin (str, optional) – Time to average

• fullpol (bool, optional) – Split full polar

• datacolumn (str, optional) – Data column to split

• ncpu (int, optional) – Number of CPU threads to use

Returns:

Output multi-measurement set name

Return type:

str

meersolar.pipeline.do_partition.single_mstransform(msname='', outputms='', field='', scan='', width=1, timebin='', corr='', datacolumn='DATA', n_threads=-1, dry_run=False)

Perform mstransform of a single scan

Parameters:

• msname (str) – Name of the measurement set

• outputms (str) – Output ms name

• field (str, optional) – Field name

• scan (str, optional) – Scans to split

• width (int, optional) – Number of channels to average

• timebin (str, optional) – Time to average

• corr (str, optional) – Correlation to split

• datacolumn (str, optional) – Data column to split

• n_threads (int, optional) – Number of CPU threads

Returns:

Output measurement set name

Return type:

str

meersolar.pipeline.do_selfcal module

meersolar.pipeline.do_selfcal.do_selfcal(msname='', workdir='', selfcaldir='', start_threshold=5, end_threshold=3, max_iter=100, max_DR=1000, min_iter=2, DR_convegerence_frac=0.3, uvrange='', minuv=0, solint='60s', weight='briggs', robust=0.0, do_apcal=True, min_tol_factor=1.0, applymode='calonly', solar_selfcal=True, ncpu=-1, mem=-1, dry_run=False, logfile='selfcal.log')

Do selfcal iterations and use convergence rules to stop

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Work directory

• selfcaldir (str) – Working directory

• start_threshold (int, optional) – Start CLEAN threhold

• end_threshold (int, optional) – End CLEAN threshold

• max_iter (int, optional) – Maximum numbers of selfcal iterations

• max_DR (float, optional) – Maximum dynamic range

• min_iter (int, optional) – Minimum numbers of seflcal iterations at different stages

• DR_convegerence_frac (float, optional) – Dynamic range fractional change to consider as converged

• uvrange (str, optional) – UV-range for calibration

• minuv (float, optionial) – Minimum UV-lambda to use in imaging

• solint (str, optional) – Solutions interval

• weight (str, optional) – Imaging weighting

• robust (float, optional) – Briggs weighting robust parameter (-1 to 1)

• do_apcal (bool, optional) – Perform ap-selfcal or not

• min_tol_factor (float, optional) – Minimum tolerable variation in temporal direction in percentage

• applymode (str, optional) – Solution apply mode

• solar_selfcal (bool, optional) – Whether is is solar selfcal or not

• ncpu (int, optional) – Number of CPU threads to use

• mem (float, optional) – Memory in GB to use

• logfile (str, optional) – Log file name

Returns:

• int – Success message

• str – Final caltable

meersolar.pipeline.do_selfcal.main()

meersolar.pipeline.do_selfcal.single_selfcal_iteration(msname, logger, selfcaldir, cellsize, imsize, round_number=0, uvrange='', minuv=0, calmode='ap', solint='60s', refant='1', solmode='', gaintype='T', applymode='calonly', threshold=3, weight='briggs', robust=0.0, use_previous_model=False, use_solar_mask=True, mask_radius=20, min_tol_factor=-1, ncpu=-1, mem=-1)

A single self-calibration round

Parameters:

• msname (str) – Name of the measurement set

• logger (logger) – Python logger

• selfcaldir (str) – Self-calibration directory

• cellsize (float) – Cellsize in arcsec

• imsize (int) – Image pixel size

• round_number (int, optional) – Selfcal iteration number

• uvrange (float, optional) – UV range for calibration

• calmode (str, optional) – Calibration mode (‘p’ or ‘ap’)

• solint (str, optional) – Solution intervals

• refant (str, optional) – Reference antenna

• applymode (str, optional) – Solution apply mode (calonly or calflag)

• threshold (float, optional) – Imaging and auto-masking threshold

• weight (str, optional) – Image weighting

• robust (float, optional) – Robust parameter for briggs weighting

• use_previous_model (bool, optional) – Use previous model

• use_solar_mask (bool, optional) – Use solar disk mask or not

• mask_radius (float, optional) – Mask radius in arcminute

• min_tol_factor (float, optional) – Minimum tolerance factor

• ncpu (int, optional) – Number of CPUs to use in WSClean

• mem (float, optional) – Memory usage limit in WSClean

Returns:

• int – Success message

• str – Caltable name

• float – RMS based dynamic range

• float – RMS of the image

• str – Image name

• str – Model image name

• str – Residual image name

meersolar.pipeline.do_sidereal_cor module

meersolar.pipeline.do_sidereal_cor.cor_sidereal_motion(mslist, workdir, cpu_frac=0.8, mem_frac=0.8, max_cpu_frac=0.8, max_mem_frac=0.8)

Perform sidereal motion correction

Parameters:

• mslist (list) – Measurement set list

• workdir (str) – Work directory

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

• max_cpu_frac (float, optional) – Maximum CPU fraction to use

• max_mem_frac (float, optional) – Maximum memory fraction to use

Returns:

• int – Success message

• list – List of sidereal motion corrected measurement sets

meersolar.pipeline.do_sidereal_cor.main()

meersolar.pipeline.do_target_split module

meersolar.pipeline.do_target_split.chanlist_to_str(lst)

meersolar.pipeline.do_target_split.main()

meersolar.pipeline.do_target_split.split_scan(msname='', outputvis='', scan='', width='', timebin='', datacolumn='', spw='', corr='', timerange='', n_threads=-1, dry_run=False)

Split a single target scan

Parameters:

• msname (str) – Measurement set

• outputvis (str) – Output measurement set

• scan (int) – Scan number

• width (int) – Channel width

• timebin (str) – Timebin width

• datacolumn (str) – Datacolumn to split

• spw (str, optional) – Spectral window to split

• corr (str, optional) – Correlation to split

• timerange (str, optional) – Time range to split

• n_threads (int, optional) – Number of OpenMP threads

Returns:

Splited measurement set

Return type:

str

meersolar.pipeline.do_target_split.split_target_scans(msname, workdir, timeres, freqres, datacolumn, spw='', spectral_chunk=-1, n_spectral_chunk=-1, scans=[], prefix='targets', fullpol=False, time_interval=-1, time_window=-1, quack_timestamps=-1, merge_spws=False, cpu_frac=0.8, mem_frac=0.8, max_cpu_frac=0.8, max_mem_frac=0.8)

Split target scans

Parameters:

• msname (str) – Measurement set

• workdir (str) – Work directory

• timeres (float) – Time resolution in seconds

• freqres (float) – Frequency resolution in MHz

• datacolumn (str) – Data column to split

• spw (str, optional) – Spectral window

• spectral_chunk (float, optional) – Spectral chunk in MHz

• n_spectral_chunk (int, optional) – Number of spectral chunks to split from the beginning

• scans (list) – Scan list to split

• prefix (str, optional) – Splited ms prefix

• fullpol (bool, optional) – Full polar split

• time_interval (float) – Time interval in seconds

• time_window (float) – Time window in seconds

• quack_timestamps (int, optional) – Number of timestamps ignored at the start and end of each scan

• merge_spws (bool, optional) – Merge spectral window ranges

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

• max_cpu_frac (float, optional) – Maximum CPU fraction to use

• max_mem_frac (float, optional) – Maximum memory fraction to use

Returns:

Splited ms list

Return type:

list

meersolar.pipeline.flagging module

meersolar.pipeline.flagging.do_flagging(msname, datacolumn='data', flag_bad_ants=True, flag_bad_spw=True, use_tfcrop=True, use_rflag=False, flagdimension='freqtime', flag_autocorr=True, flag_backup=True, cpu_frac=0.8, mem_frac=0.8)

Function to perform initial flagging

Parameters:

• msname (str) – Name of the ms

• datacolumn (str, optional) – Data column

• flag_bad_ants (bool, optional) – Flag bad antennas

• flag_bad_spw (bool, optional) – Flag bad channels

• use_tfcrop (bool, optional) – Use tfcrop or not

• use_rflag (bool, optional) – Use rflag or not

• flagdimension (str, optional) – Flag dimension (only for tfcrop)

• flag_autocorr (bool,optional) – Flag auto-correlations

• flag_backup (bool, optional) – Flag backup

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.flagging.main()

meersolar.pipeline.flagging.single_ms_flag(msname='', badspw='', bad_ants_str='', datacolumn='data', use_tfcrop=True, use_rflag=False, flagdimension='freqtime', flag_autocorr=True, n_threads=-1, memory_limit=-1, dry_run=False)

Flag on a single ms

Parameters:

• msname (str) – Measurement set name

• badspw (str, optional) – Bad spectral window

• bad_ants_str (str, optional) – Bad antenna string

• datacolumn (str, optional) – Data column

• use_tfcrop (str, optional) – Use tfcrop or not

• use_rflag (str, optional) – Use rflag or not

• flagdimension (str, optional) – Flag dimension (only applicable for tfcrop)

• flag_autocorr (bool, optional) – Flag autocorrelations or not

• n_threads (int, optional) – Number of OpenMP threads

• memory_limit (float, optional) – Memory limit in GB

• dry_run (bool, optional) – Return the amount of pre-occupied memory in GB

meersolar.pipeline.import_model module

meersolar.pipeline.import_model.get_polmodel_coeff(model_file)

Get polarization model coefficients

Parameters:

model_file (str) – Model ascii file name

Returns:

• float – Reference frequency in GHz

• float – Stokes I flux density in Jy at reference frequency

• list – Stokes I spectrum log-polynomial coefficients

• list – List of linear polarization fraction polynomial coefficients

• list – List of linear polarization angle (in radians) coefficients

meersolar.pipeline.import_model.import_all_models(msname, workdir, cpu_frac=0.8, mem_frac=0.8)

Import all calibrator models

Parameters:

• msname (str) – Measurement set name

• workdir (str) – Work directory

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.import_model.import_fluxcal_models(mslist, fluxcal_fields, fluxcal_scans, ncpus=1, mem_frac=0.8)

Import model visibilities using crystalball

Parameters:

• mslist (list) – List of sub-MS

• fluxcal_fields (list) – Fluxcal fields

• fluxcal_scans (dict) – Fluxcal scans dict

• ncpus (int) – Number of CPU threads to use

• mem_frac (float) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.import_model.import_phasecal_models(mslist, phasecal_fields, phasecal_scans, workdir, cpu_frac=0.8, mem_frac=0.8)

Import model visibilities for phasecal

Parameters:

• mslist (list) – List of sub-MS

• phasecal_fields (list) – Phasecal fields

• phasecal_scans (list) – Phasecal scans

• workdir (str) – Work directory

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.import_model.import_polcal_model(mslist, polcal_fields, polcal_scans, workdir, cpu_frac=0.8, mem_frac=0.8)

Import model for polarization calibrators (3C286 or 3C138)

Parameters:

• mslist (list) – List of sub-MS

• polcal_fields (list) – Polcal fields

• polcal_scans (list) – Polcal scans

• workdir (str) – Work directory

• n_threads (int, optional) – Number of OpenMP threads

Returns:

Success message

Return type:

int

meersolar.pipeline.import_model.main()

meersolar.pipeline.import_model.phasecal_setjy(msname='', field='', ismms=False, n_threads=-1, dry_run=False)

Setjy phasecal fields

Parameters:

• msname (str) – Measurement set

• field (str, optional) – Phasecal fields

• ismms (bool, optional) – Is a multi-ms ot not

• n_threads (int, optional) – Number of OpenMP threads

meersolar.pipeline.import_model.polcal_setjy(msname='', field_name='', n_threads=-1, ismms=True, dry_run=False)

Setjy polcal fields (3C286 or 3C138)

Parameters:

• msname (str) – Name of measurement set

• field (str, optional) – Field name

• n_threads (int, optional) – Number of OpenMP threads

meersolar.pipeline.init_data module

meersolar.pipeline.init_data.download_with_parfive(record_id, update=False, output_dir='zenodo_download')

meersolar.pipeline.init_data.get_zenodo_file_urls(record_id)

meersolar.pipeline.init_data.init_meersolar_data(update=False, remote_link=None, emails=None)

Initiate MeerSOLAR data

Parameters:

• update (bool, optional) – Update data, if already exists

• remote_link (str, optional) – Remote logger link to save in database

• emails (str, optional) – Email addresses to send remote logger JobID and password

meersolar.pipeline.init_data.main()

meersolar.pipeline.master_controller module

meersolar.pipeline.master_controller.check_status(workdir, basename)

Check job status

Parameters:

• workdir (str) – Work directory

• basename (str) – Basename

Returns:

Finished or not

Return type:

bool

meersolar.pipeline.master_controller.exit_job(start_time, mspath='', workdir='')

meersolar.pipeline.master_controller.main()

meersolar.pipeline.master_controller.master_control(msname, workdir, outdir, solar_data=True, do_forcereset_weightflag=False, do_cal_partition=True, do_cal_flag=True, do_import_model=True, do_basic_cal=True, do_noise_cal=True, do_applycal=True, do_target_split=True, target_scans=[], freqrange='', timerange='', uvrange='', do_polcal=False, do_selfcal=True, do_selfcal_split=True, do_apply_selfcal=True, do_ap_selfcal=True, solar_selfcal=True, solint='5min', do_sidereal_cor=False, make_ds=True, do_imaging=True, do_pbcor=True, weight='briggs', robust=0.0, minuv=0, image_freqres=-1, image_timeres=-1, pol='IQUV', apply_parang=True, clean_threshold=1.0, use_multiscale=True, use_solar_mask=True, cutout_rsun=2.5, make_overlay=True, cpu_frac=0.8, mem_frac=0.8, n_nodes=0, keep_backup=False, remote_logger=False, remote_logger_waittime=0)

Master controller of the entire pipeline

Parameters:

• msname (str) – Measurement set name

• workdir (str) – Work directory path

• outdir (str) – Output directory

• solar_data (bool, optional) – Whether it is solar data or not

• do_forcereset_weightflag (bool, optional) – Reset weights and flags of the input ms

• do_cal_partition (bool, optional) – Make calibrator multi-MS

• do_cal_flag (bool, optional) – Perform flagging on calibrator

• do_import_model (bool, optional) – Import model visibilities of flux and polarization calibrators

• do_basic_cal (bool, optional) – Perform basic calibration

• do_noise_cal (bool, optional) – Peform calibration of solar attenuators using noise diode (only used if solar_data=True)

• do_applycal (bool, optional) – Apply basic calibration on target scans

• do_target_split (bool, optional) – Split target scans into chunks

• target_scans (list, optional) – Target scans to self-cal and image

• freqrange (str, optional) – Frequency range to image in MHz (xx1~xx2,xx3~xx4,)

• timerange (str, optional) – Time range to image in YYYY/MM/DD/hh:mm:ss format (tt1~tt2,tt3~tt4,…)

• uvrange (str, optional) – UV-range for calibration

• do_polcal (bool, optional) – Perform full-polarization calibration and imaging

• do_selfcal (bool, optional) – Perform self-calibration

• do_selfcal_split (bool, optional) – Split data after each round of self-calibration

• do_apply_selfcal (bool, optional) – Apply self-calibration solutions

• do_ap_selfcal (bool, optional) – Perform amplitude-phase self-cal or not

• solar_selfcal (bool, optional) – Whether self-calibration is performing on solar observation or not

• solint (str, optional) – Solution intervals in self-cal

• do_sidereal_cor (bool, optional) – Perform solar sidereal motion correction or not

• make_ds (bool, optional) – Make dynamic spectra

• do_imaging (bool, optional) – Perform final imaging

• do_pbcor (bool, optional) – Perform primary beam correction

• weight (str, optional) – Image weighting

• robust (float, optional) – Robust parameter for briggs weighting (-1 to 1)

• minuv (float, optional) – Minimum UV-lambda for final imaging

• image_freqres (float, optional) – Image frequency resolution in MHz (-1 means full bandwidth)

• image_timeres (float, optional) – Image temporal resolution in seconds (-1 means full scan duration)

• pol (str, optional) – Stokes parameters of final imaging

• apply_parang (bool, optional) – Apply parallactic angle correction

• clean_threshold (float, optional) – CLEAN threshold of final imaging

• use_multiscale (bool, optional) – Use multiscale scales or not

• use_solar_mask (bool, optional) – Use solar mask

• cutout_rsun (float, optional) – Cutout image size from center in solar radii (default : 2.5 solar radii)

• make_overlay (bool, optional) – Make SUVI MeerKAT overlay

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

• n_nodes (int, optional) – Number of nodes to use (Only for cluster architecture)

• keep_backup (bool, optional) – Keep backup of self-cal rounds and final models and residual images

• remote_logger (bool, optional) – Enable remote logging of the pipeline status

• remote_logger_waittime (int, optional) – Wait time (in seconds) before connecting to remote logger

Returns:

Success message

Return type:

int

meersolar.pipeline.master_controller.run_apply_basiccal_sol(target_mslist, workdir, caldir, use_only_bandpass=False, overwrite_datacolumn=True, applymode='calflag', jobid=0, cpu_frac=0.8, mem_frac=0.8)

Apply basic calibration solutions on splited target scans

Parameters:

• target_mslist (list) – Target measurement set list

• workdir (str) – Working directory

• caldir (str) – Caltable directory

• use_only_bandpass (bool) – Use only bandpass solutions

• applymode (str, optional) – Applycal mode

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

• overwrite_datacolumn (bool) – Overwrite data column or not

Returns:

Success message for applying calibration solutions and spliting target scans

Return type:

int

meersolar.pipeline.master_controller.run_apply_pbcor(imagedir, workdir, apply_parang=True, jobid=0, cpu_frac=0.8, mem_frac=0.8)

Apply primary beam corrections on all images

Parameters:

• imagedir (str) – Image directory name

• workdir (str) – Work directory

• apply_parang (bool, optional) – Apply parallactic angle correction

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message for applying primary beam correction on all images

Return type:

int

meersolar.pipeline.master_controller.run_apply_selfcal_sol(target_mslist, workdir, caldir, overwrite_datacolumn=True, applymode='calflag', jobid=0, cpu_frac=0.8, mem_frac=0.8)

Apply self-calibration solutions on splited target scans

Parameters:

• target_mslist (list) – Target measurement set list

• workdir (str) – Working directory

• caldir (str) – Caltable directory

• use_only_bandpass (bool) – Use only bandpass solutions

• applymode (str, optional) – Applycal mode

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

• overwrite_datacolumn (bool) – Overwrite data column or not

Returns:

Success message for applying calibration solutions and spliting target scans

Return type:

int

meersolar.pipeline.master_controller.run_basic_cal_jobs(msname, workdir, caldir, perform_polcal=False, jobid=0, cpu_frac=0.8, mem_frac=0.8, keep_backup=False)

Perform basic calibration

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Working directory

• caldir (str) – Caltable directory

• perform_polcal (bool, optional) – Perform full polarization calibration

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

• keep_backup (bool, optional) – Keep backups

Returns:

Success message for basic calibration

Return type:

int

meersolar.pipeline.master_controller.run_ds_jobs(msname, workdir, outdir, target_scans=[], jobid=0, cpu_frac=0.8, mem_frac=0.8)

Make dynamic spectra of the target scans

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Name of the work directory

• outdir (str) – Output directory

• target_scans (list, optional) – Target scans

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.master_controller.run_flag(msname, workdir, flag_calibrators=True, jobid=0, cpu_frac=0.8, mem_frac=0.8)

Run flagging jobs

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Working directory

• flag_calibrators (bool, optional) – Flag calibrator fields

• jobid (int, optional) – Job ID

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.master_controller.run_imaging_jobs(mslist, workdir, outdir, freqrange='', timerange='', minuv=-1, weight='briggs', robust=0.0, pol='IQUV', freqres=-1, timeres=-1, band='', threshold=1.0, use_multiscale=True, use_solar_mask=True, cutout_rsun=2.5, make_overlay=True, savemodel=False, saveres=False, jobid=0, cpu_frac=0.8, mem_frac=0.8)

Self-calibration on target scans

Parameters:

• mslist (list) – Target measurement set list

• workdir (str) – Working directory

• outdir (str) – Output image directory

• freqrange (str, optional) – Frequency range to image in MHz

• timerange (str, optional) – Time range to image (YYYY/MM/DD/hh:mm:ss~YYYY/MM/DD/hh:mm:ss)

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

• minuv (float, optionial) – Minimum UV-lambda to use in imaging

• weight (str, optional) – Imaging weighting

• robust (float, optional) – Briggs weighting robust parameter (-1 to 1)

• pol (str, optional) – Stokes parameters to image

• freqres (float, optional) – Frequency resolution of spectral chunk in MHz (default : -1, no spectral chunking)

• timeres (float, optional) – Time resolution of temporal chunks in MHz (default : -1, no temporal chunking)

• band (str, optional) – Band name

• threshold (float, optional) – CLEAN threshold

• use_multiscale (bool, optional) – Use multiscale or not

• use_solar_mask (bool, optional) – Use solar mask or not

• cutout_rsun (float, optional) – Cutout image size from center in solar radii (default : 2.5 solar radii)

• make_overlay (bool, optional) – Make SUVI MeerKAT overlay

• savemodel (bool, optional) – Save model images or not

• saveres (bool, optional) – Save residual images or not

Returns:

Success message for imaging

Return type:

int

meersolar.pipeline.master_controller.run_import_model(msname, workdir, jobid=0, cpu_frac=0.8, mem_frac=0.8)

Importing calibrator models

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Working directory

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.master_controller.run_noise_diode_cal(msname, workdir, caldir, keep_backup=False, jobid=0, cpu_frac=0.8, mem_frac=0.8)

Perform noise diode based flux calibration

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Working directory

• caldir (str) – Caltable directory

• keep_backup (bool, optional) – Keep backup

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message for noise diode based flux calibration

Return type:

int

meersolar.pipeline.master_controller.run_partion(msname, workdir, split_fullpol=False, jobid=0, cpu_frac=0.8, mem_frac=0.8)

Perform basic calibration

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Working directory

• split_fullpol (bool, optional) – Perform full polar split

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.master_controller.run_selfcal_jobs(mslist, workdir, caldir, start_thresh=5.0, stop_thresh=3.0, max_iter=100, max_DR=1000, min_iter=2, conv_frac=0.3, solint='60s', do_apcal=True, solar_selfcal=True, keep_backup=False, uvrange='', minuv=0, weight='briggs', robust=0.0, applymode='calonly', min_tol_factor=10.0, jobid=0, cpu_frac=0.8, mem_frac=0.8)

Self-calibration on target scans

Parameters:

• mslist (list) – Target measurement set list

• workdir (str) – Working directory

• caldir (str) – Caltable directory

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

• start_threshold (int, optional) – Start CLEAN threhold

• end_threshold (int, optional) – End CLEAN threshold

• max_iter (int, optional) – Maximum numbers of selfcal iterations

• max_DR (float, optional) – Maximum dynamic range

• min_iter (int, optional) – Minimum numbers of seflcal iterations at different stages

• conv_frac (float, optional) – Dynamic range fractional change to consider as converged

• uvrange (str, optional) – UV-range for calibration

• minuv (float, optionial) – Minimum UV-lambda to use in imaging

• weight (str, optional) – Image weighitng scheme

• robust (float, optional) – Robustness parameter for briggs weighting

• solint (str, optional) – Solutions interval

• do_apcal (bool, optional) – Perform ap-selfcal or not

• min_tol_factor (float, optional) – Minimum tolerance in temporal variation in imaging

• applymode (str, optional) – Solution apply mode

• solar_selfcal (bool, optional) – Whether is is solar selfcal or not

Returns:

Success message for self-calibration

Return type:

int

meersolar.pipeline.master_controller.run_solar_siderealcor_jobs(mslist, workdir, prefix='targets', jobid=0, cpu_frac=0.8, mem_frac=0.8, max_cpu_frac=0.8, max_mem_frac=0.8)

Apply sidereal motion correction of the Sun

Parameters:

• mslist (str) – List of the measurement sets

• workdir (str) – Work directory

• prefix (str, optional) – Measurement set prefix

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

• max_cpu_frac (float, optional) – Maximum CPU fraction to use

• max_mem_frac (float, optional) – Maximum memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.master_controller.run_target_split_jobs(msname, workdir, datacolumn='data', spw='', timeres=-1, freqres=-1, target_freq_chunk=-1, n_spectral_chunk=-1, target_scans=[], prefix='targets', split_fullpol=False, merge_spws=False, time_window=-1, time_interval=-1, jobid=0, cpu_frac=0.8, mem_frac=0.8, max_cpu_frac=0.8, max_mem_frac=0.8)

Split target scans

Parameters:

• msname (str) – Name of the measurement set

• workdir (str) – Working directory

• datacolumn (str, optional) – Data column

• spw (str, optional) – Spectral window to split

• timeres (float, optional) – Time bin to average in seconds

• freqres (float, optional) – Frequency averaging in MHz

• target_freq_chunk (float, optional) – Target frequency chunk in MHz

• n_spectral_chunk (int, optional) – Number of spectral chunks to split

• target_scans (list, optional) – Target scans

• prefix (str, optional) – Prefix of splited targets

• split_fullpol (bool, optional) – Split full polar data or not

• merge_spws (bool, optional) – Merge spectral windows

• time_window (float, optional) – Time window in seconds

• time_interval (float, optional) – Time interval in seconds

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message for spliting target scans

Return type:

int

meersolar.pipeline.master_controller.start_ping_logger(jobid, remote_jobid, waittime, remote_link='')

Run ping remote logger in background

Parameters:

• jobid (int) – Job ID

• remote_jobid (str) – Remote log job ID

• waittime (float) – Wait time before process ends in hour

• remote_link (str, optional) – Remote logger link

Returns:

PID of the background job

Return type:

int

meersolar.pipeline.meer_pbcor module

meersolar.pipeline.meer_pbcor.get_fits_freq(image_file)

meersolar.pipeline.meer_pbcor.main()

meersolar.pipeline.meer_pbcor.pbcor_all_images(imagedir, make_TB=True, make_plots=True, apply_parang=True, jobid=0, cpu_frac=0.8, mem_frac=0.8)

Correct primary beam of MeerKAT for images in a directory

Parameters:

• imagedir (str) – Name of the image directory

• make_TB (bool, optional) – Make brightness temperature map

• make_plots (bool, optional) – Make plots

• apply_parang (bool, optional) – Apply parallactic angle correction

• jobid (int, optional) – Job ID

• cpu_frac (float, optional) – CPU fraction to use

• mem_frac (float, optional) – Memory fraction to use

Returns:

Success message

Return type:

int

meersolar.pipeline.meer_pbcor.run_pbcor(imagename, pbdir, pbcor_dir, apply_parang, jobid=0, ncpu=8)

meersolar.pipeline.single_image_meerpbcor module

meersolar.pipeline.single_image_meerpbcor.apply_parallactic_rotation(jones, p_angle)

Apply left-side parallactic rotation: J’ = J.R(p_angle) as needed in the RIME context (sky-frame transformation).

Parameters:

• jones (ndarray) – Jones matrix, shape (4, H, W), with components: [0] = J_00, [1] = J_01, [2] = J_10, [3] = J_11

• chi (float) – Parallactic angle in degree

Returns:

jones_rot – Rotated Jones matrix, shape (4, H, W)

Return type:

ndarray

meersolar.pipeline.single_image_meerpbcor.get_IQUV(filename)

Get IQUV from a fits

Parameters:

filename (str) – Fits image name

Returns:

Stokes

Return type:

dict

meersolar.pipeline.single_image_meerpbcor.get_beam_interpolator(jones, coords)

Get beam interpolator

Parameters:

• jones (numpy.array) – Jones array (shape, npol, l_npix, m_npix)

• coords (numpy.array) – l,m coordinates

Returns:

Interpolation functions

Return type:

interpolator

meersolar.pipeline.single_image_meerpbcor.get_brightness(stokes)

Returns brightness matrix from stokes dictionary (X and Y are in opposite convention of IAU in MeerKAT)

meersolar.pipeline.single_image_meerpbcor.get_image_beam(image_file, pbdir, save_beam=True, band='', apply_parang=True, n_cpu=8, verbose=False)

Get image beam

Parameters:

• image_file (str) – Image file name

• pbdir (str) – Primary beam directory

• save_beam (bool, optional) – Save beam of the image

• band (str, optional) – Band name

• apply_parang (bool, optional) – Apply parallactic angle correction

• n_cpu (int, optinal) – Number of CPU threads to use

• verbose (bool, optional) – Verbose output

Returns:

Jones array

Return type:

numpy.array

meersolar.pipeline.single_image_meerpbcor.get_parallactic_angle(obs_time, ra_deg, dec_deg, LAT=-30.7133, LON=21.4429, ALT=1086.6)

Get parallactic angle

Parameters:

• obs_time (str) – Observation time in YYY-MM-DDThh:mm:ss format

• ra (float) – RA in degree

• dec (float) – DEC in degree

Returns:

Parallactic angle in degree

Return type:

float

meersolar.pipeline.single_image_meerpbcor.get_pbcor_image(image_file, pbdir, pbcor_dir, save_beam=True, band='', apply_parang=True, n_cpu=8, verbose=False)

Get primary beam corrected image

Parameters:

• image_file (str) – Image file name

• pbdir (str) – Primary beam directory

• pbcor_dir (str) – Primary beam corrected image directory

• save_beam (bool, optional) – Save the beam for the image

• band (str, optional) – Band name

• apply_parang (bool, optional) – Apply parallactic correction

• n_cpu (int, optional) – Number of CPU threads to use

• verbose (bool, optional) – Verbose output

Returns:

Primary beam corrected image

Return type:

str

meersolar.pipeline.single_image_meerpbcor.get_pointingcenter_radec(image_file)

Get image pointing center RA DEC

Parameters:

image_file (str) – Image file name

Returns:

• float – RA in degree

• float – DEC in degree

meersolar.pipeline.single_image_meerpbcor.get_radec_grid(image_file)

Get RA and Dec arrays for all pixels in an image.

Parameters:

image_file (str) – FITS image file name

Returns:

• ra (2D numpy.ndarray) – RA values in degrees for each pixel

• dec (2D numpy.ndarray) – Dec values in degrees for each pixel

meersolar.pipeline.single_image_meerpbcor.load_beam(image_file, band='')

Load MeerKAT beam

Parameters:

• image_file (str) – Image name (Assuming single spectral image)

• band (str, optional) – Band name (If not provided, check from header or frequency)

Returns:

• numpy.array – l,m coordinates

• numpy.array – Full Jones complex beam

meersolar.pipeline.single_image_meerpbcor.main()

meersolar.pipeline.single_image_meerpbcor.make_stokes(b)

Makes stokes images from brightness matrix

meersolar.pipeline.single_image_meerpbcor.put_IQUV(filename, stokes, header)

Put IQUV into a fits

Parameters:

• filename (str) – Fits image name

• stokes (dict) – Stokes

• header (dict) – Image header

Returns:

Stokes

Return type:

dict

meersolar.pipeline.single_image_meerpbcor.radec_to_lm(ra_deg, dec_deg, ra0_deg, dec0_deg)

Convert RA/Dec to l,m direction cosines relative to a phase center.

Parameters:

• ra_deg (2D arrays) – RA and Dec in degrees

• dec_deg (2D arrays) – RA and Dec in degrees

• ra0_deg (float) – Phase center RA and Dec in degrees

• dec0_deg (float) – Phase center RA and Dec in degrees

Returns:

l, m – Direction cosines (dimensionless)

Return type:

2D arrays

meersolar.pipeline.viewer module

class meersolar.pipeline.viewer.LogViewer(max_lines=10000)

Bases: QWidget

append_log_line(text)

calc_list_width()

closeEvent(self, a0: QCloseEvent | None)

load_log_content(item)

refresh_logs()

setup_ui()

class meersolar.pipeline.viewer.SmartDefaultsHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)

Bases: ArgumentDefaultsHelpFormatter

class meersolar.pipeline.viewer.TailWatcher(file_path)

Bases: FileSystemEventHandler, QObject

new_line

int = …, arguments: Sequence = …) -> PYQT_SIGNAL

types is normally a sequence of individual types. Each type is either a type object or a string that is the name of a C++ type. Alternatively each type could itself be a sequence of types each describing a different overloaded signal. name is the optional C++ name of the signal. If it is not specified then the name of the class attribute that is bound to the signal is used. revision is the optional revision of the signal that is exported to QML. If it is not specified then 0 is used. arguments is the optional sequence of the names of the signal’s arguments.

Type:

pyqtSignal(*types, name

Type:

str = …, revision

on_modified(event)

Called when a file or directory is modified.

Parameters:

event (DirModifiedEvent or FileModifiedEvent) – Event representing file/directory modification.

start()

stop()

meersolar.pipeline.viewer.drop_cache(path, verbose=False)

Drop file cache for a file or all files under a directory.

Parameters:

path (str) – File or directory path

meersolar.pipeline.viewer.drop_file_cache(filepath, verbose=False)

Advise the OS to drop the given file from the page cache. Safe, per-file, no sudo required.

meersolar.pipeline.viewer.get_logid(logfile)

meersolar.pipeline.viewer.get_meersolar_cachedir()

meersolar.pipeline.viewer.main()

Module contents