Meersolar Utils Module

This module contains all basic functionalities required for the pipeline.

Note

All functions in submodules are also available from meersolar.utils namespace as well.

meersolar.utils.basic_utils module

meersolar.utils.basic_utils.angular_separation_equatorial(ra1, dec1, ra2, dec2)

Calculate angular seperation between two equatorial coordinates

Parameters:

• ra1 (float) – RA of the first coordinate in degree

• dec1 (float) – DEC of the first coordinate in degree

• ra2 (float) – RA of the second coordinate in degree

• dec2 (float) – DEC of the second coordinate in degree

Returns:

Angular distance in degree

Return type:

float

meersolar.utils.basic_utils.average_timestamp(timestamps)

Compute the average timestamp using astropy from a list of ISO 8601 strings.

Parameters:

timestamps (list) – timestamps (list of str): List of timestamp strings in ‘YYYY-MM-DDTHH:MM:SS’ format.

Returns:

Average timestamp in ‘YYYY-MM-DDTHH:MM:SS’ format.

Return type:

str

meersolar.utils.basic_utils.ceil_to_multiple(n, base)

Round up to the next multiple

Parameters:

• n (float) – The number

• base (float) – Whose multiple will be

Returns:

The modified number

Return type:

float

meersolar.utils.basic_utils.create_datadir(datadir='')

Create data directory

Parameters:

datadir (str, optional) – User provided custom data directory

meersolar.utils.basic_utils.get_cachedir()

Get cache directory

meersolar.utils.basic_utils.get_datadir()

Get package data directory

Returns:

Data directory

Return type:

str

meersolar.utils.basic_utils.mjdsec_to_timestamp(mjdsec, str_format=0)

Convert CASA MJD seceonds to CASA timestamp

Parameters:

• mjdsec (float) – CASA MJD seconds

• str_format (int) – Time stamp format (0: yyyy-mm-ddTHH:MM:SS.ff, 1: yyyy/mm/dd/HH:MM:SS.ff, 2: yyyy-mm-dd HH:MM:SS)

Returns:

CASA time stamp in UTC at ISOT format

Return type:

str

meersolar.utils.basic_utils.ra_dec_to_deg(ra_hms, dec_dms)

Convert RA and Dec from hms and dms format to degrees

Parameters:

• ra_hms (str) – Right Ascension in ‘hms’ format

• dec_dms (str) – Declination in ‘dms’ format

Returns:

RA and Dec in degrees

Return type:

tuple

meersolar.utils.basic_utils.ra_dec_to_hms_dms(ra_deg, dec_deg)

Convert RA and Dec in degrees to hms and dms format

Parameters:

• ra_deg (float) – Right Ascension in degrees.

• dec_deg (float) – Declination in degrees.

Returns:

RA in h:m:s format, Dec in d:m:s format (e.g., ‘1h5m0s’, ‘1d5m0s’).

Return type:

tuple

meersolar.utils.basic_utils.split_into_chunks(lst, target_chunk_size)

Split a list into equal number of elements

Parameters:

• lst (list) – List of numbers

• target_chunk_size (int) – Number of elements per chunk

Returns:

Chunked list

Return type:

list

meersolar.utils.basic_utils.suppress_output()

Supress CASA terminal output

meersolar.utils.basic_utils.timestamp_to_mjdsec(timestamp, date_format=0)

Convert timestamp to mjd second.

Parameters:

• timestamp (str) – Time stamp to convert

• date_format (int, optional) –

  Datetime string format

  0: ‘YYYY/MM/DD/hh:mm:ss’

  1: ‘YYYY-MM-DDThh:mm:ss’

  2: ‘YYYY-MM-DD hh:mm:ss’

  3: ‘YYYY_MM_DD_hh_mm_ss’

Returns:

Return correspondong MJD second of the day

Return type:

float

meersolar.utils.calibration module

meersolar.utils.calibration.calc_bw_smearing_freqwidth(msname, full_FoV=False, FWHM=True)

Function to calculate spectral width to procude bandwidth smearing

Parameters:

• msname (str) – Name of the measurement set

• full_FoV (bool, optional) – Consider smearing within solar disc or full FoV

• FWHM (bool, optional) – If using full FoV, consider upto FWHM or first null

Returns:

Spectral width in MHz

Return type:

float

meersolar.utils.calibration.calc_time_smearing_timewidth(msname, full_FoV=False, FWHM=True)

Calculate maximum time averaging to avoid time smearing over full FoV.

Parameters:

• msname (str) – Measurement set name

• full_FoV (bool, optional) – Consider smearing within solar disc or full FoV

• FWHM (bool, optional) – If using full FoV, consider upto FWHM or first null

Returns:

delta_t_max – Maximum allowable time averaging in seconds.

Return type:

float

meersolar.utils.calibration.delaycal(vis='', caltable='', spw='', field='', scan='', uvrange='', refant='', refantmode='flex', solint='inf', combine='scan', gaintable=[], gainfield=[], interp=[])

General delay calibration using CASA, not assuming any point source

Parameters:

• vis (str) – Measurement set

• caltable (str) – Caltable name

• spw (str, optional) – Spectral window

• field (str, optional) – Field name

• scan (str, optional) – Scan number

• uvrange (str, optional) – UV-range

• refant (str, optional) – Reference antenna (require one reference antenna, can not keep black)

• refantmode (str, optional) – Refant mode

• solint (str, optional) – Solution interval

• combine (str, optional) – Combine data

• gaintable (list, optional) – Previous gaintables

• gainfield (list, optional) – Gain fields

• interp (list, optional) – Interpolation solutions

Returns:

Caltable name

Return type:

str

meersolar.utils.calibration.get_psf_size(msname, chan_number=-1)

Function to calculate PSF size in arcsec

Parameters:

• msname (str) – Name of the measurement set

• chan_number (int, optional) – Channel number

Returns:

PSF size in arcsec

Return type:

float

meersolar.utils.calibration.max_time_solar_smearing(msname)

Max allowable time averaging to avoid solar motion smearing.

Parameters:

msname (str) – Measurement set name

Returns:

t_max – Maximum time averaging in seconds.

Return type:

float

meersolar.utils.calibration.merge_caltables(caltables, merged_caltable, append=False, keepcopy=False)

Merge multiple same type of caltables

Parameters:

• caltables (list) – Caltable list

• merged_caltable (str) – Merged caltable name

• append (bool, optional) – Append with exisiting caltable

• keepcopy (bool, opitonal) – Keep input caltables or not

Returns:

Merged caltable

Return type:

str

meersolar.utils.casatasks module

meersolar.utils.casatasks.check_scan_in_caltable(caltable, scan)

Check scan number available in caltable or not

Parameters:

• caltable (str) – Name of the caltable

• scan (int) – Scan number

Returns:

Whether scan is present in the caltable or not

Return type:

bool

meersolar.utils.casatasks.correct_missing_col_subms(msname)

Correct for missing colurmns in sub-MSs

Parameters:

msname (str) – Name of the measurement set

meersolar.utils.casatasks.reset_weights_and_flags(msname='', restore_flag=True, force_reset=False, n_threads=-1)

Reset weights and flags for the ms

Parameters:

• msname (str) – Measurement set

• restore_flag (bool, optional) – Restore flags or not

• force_reset (bool, optional) – Force reset

meersolar.utils.casatasks.single_mstransform(msname='', outputms='', field='', scan='', width=1, timebin='', datacolumn='DATA', spw='', corr='', timerange='', numsubms='auto', n_threads=-1)

Perform mstransform of a single scan

Parameters:

• msname (str) – Name of the measurement set

• outputms (str) – Output ms name

• scan (int) – Scan to split (a single scan)

• field (str, optional) – Field name

• width (int, optional) – Number of channels to average

• timebin (str, optional) – Time to average

• datacolumn (str, optional) – Data column to split

• spw (str, optional) – Spectral window

• corr (str, optional) – Correlation to split

• timerange (str, optional) – Time range

• numsubms (str, optional) – Number of subms

• n_threads (int, optional) – Number of CPU threads

Returns:

Output measurement set name

Return type:

str

meersolar.utils.flagging module

meersolar.utils.flagging.calc_flag_fraction(msname='', field='', scan='', n_threads=-1)

Function to calculate the fraction of total data flagged.

Parameters:

• msname (str) – Name of the measurement set

• field (str, optional) – Field names

• scan (str, optional) – Scan names

Returns:

Fraction of the total data flagged

Return type:

float

meersolar.utils.flagging.do_flag_backup(msname, flagtype='flagdata')

Take a flag backup

Parameters:

• msname (str) – Measurement set name

• flagtype (str, optional) – Flag type

meersolar.utils.flagging.flag_outside_uvrange(vis, uvrange, n_threads=-1, flagbackup=True)

Flag outside the given uv range

Parameters:

• vis (str) – Measurement set name

• uvrange (str) – UV-range

• flagbackup (bool, optional) – Flag backup

meersolar.utils.flagging.flagsummary(msname, summary_file)

Save flag summary

Parameters:

• msname (str) – Measurement set name

• summary_file (str) – Summary file name

Returns:

Summary file

Return type:

str

meersolar.utils.flagging.get_chans_flag(msname='', field='', n_threads=-1)

Get flag/unflag channel list

Parameters:

• msname (str) – Measurement set name

• field (str, optional) – Field name or ID

Returns:

• list – Unflag channel list

• list – Flag channel list

meersolar.utils.flagging.get_unflagged_antennas(msname='', scan='', n_threads=-1)

Get unflagged antennas of a scan

Parameters:

• msname (str) – Name of the measurement set

• scan (str) – Scans

Returns:

• numpy.array – Unflagged antenna names

• numpy.array – Flag fraction list

meersolar.utils.image_utils module

meersolar.utils.image_utils.calc_dyn_range(imagename, modelname, residualname, fits_mask='')

Calculate dynamic ranges.

Parameters:

• imagename (list or str) – Image FITS file(s)

• modelname (list or str) – Model FITS file(s)

• residualname (list ot str) – Residual FITS file(s)

• fits_mask (str, optional) – FITS file mask

Returns:

• model_flux (float) – Total model flux.

• dyn_range_rms (float) – Max/RMS dynamic range.

• rms (float) – RMS of the image

meersolar.utils.image_utils.calc_solar_image_stat(imagename, disc_size=18)

Calculate solar image dynamic range

Parameters:

• imagename (str) – Fits image name

• disc_size (float, optional) – Solar disc size in arcmin (default : 18)

Returns:

• float – Maximum value

• float – Minimum value

• float – RMS values

• float – Total value

• float – Mean value

• float – Median value

• float – RMS dynamic range

• float – Min-max dynamic range

meersolar.utils.image_utils.create_circular_mask(msname, cellsize, imsize, mask_radius=20)

Create fits solar mask

Parameters:

• msname (str) – Name of the measurement set

• cellsize (float) – Cell size in arcsec

• imsize (int) – Imsize in number of pixels

• mask_radius (float) – Mask radius in arcmin

Returns:

Fits mask file name

Return type:

str

meersolar.utils.image_utils.create_circular_mask_array(data, radius)

Creating circular mask of a Numpy array

Parameters:

• data (numpy.array) – 2D numpy array

• radius (int) – Radius in pixels

Returns:

Mask array

Return type:

numpy.array

meersolar.utils.image_utils.cutout_image(fits_file, output_file, x_deg=2)

Cutout central part of the image

Parameters:

• fits_file (str) – Input fits file

• output_file (str) – Output fits file name (If same as input, input image will be overwritten)

• x_deg (float, optional) – Size of the output image in degree

Returns:

Output image name

Return type:

str

meersolar.utils.image_utils.generate_tb_map(imagename, outfile='')

Function to generate brightness temperature map

Parameters:

• imagename (str) – Name of the flux calibrated image

• outfile (str, optional) – Output brightess temperature image name

Returns:

Output image name

Return type:

str

meersolar.utils.image_utils.make_freqavg_image(wsclean_images, outfile_name, keep_wsclean_images=True)

Convert WSClean images into a frequency averaged image

Parameters:

• wsclean_images (list) – List of WSClean images.

• outfile_name (str) – Name of the output file.

• keep_wsclean_images (bool, optional) – Whether to retain the original WSClean images (default: True).

Returns:

Output image name.

Return type:

str

meersolar.utils.image_utils.make_stokes_wsclean_imagecube(wsclean_images, outfile_name, keep_wsclean_images=True)

Convert WSClean images into a Stokes cube image.

Parameters:

• wsclean_images (list) – List of WSClean images.

• outfile_name (str) – Name of the output file.

• keep_wsclean_images (bool, optional) – Whether to retain the original WSClean images (default: True).

Returns:

Output image name.

Return type:

str

meersolar.utils.image_utils.make_timeavg_image(wsclean_images, outfile_name, keep_wsclean_images=True)

Convert WSClean images into a time averaged image

Parameters:

• wsclean_images (list) – List of WSClean images.

• outfile_name (str) – Name of the output file.

• keep_wsclean_images (bool, optional) – Whether to retain the original WSClean images (default: True).

Returns:

Output image name.

Return type:

str

meersolar.utils.imaging module

meersolar.utils.imaging.calc_cellsize(msname, num_pixel_in_psf)

Calculate pixel size in arcsec

Parameters:

• msname (str) – Name of the measurement set

• num_pixel_in_psf (float) – Number of pixels in one PSF

Returns:

Pixel size in arcsec

Return type:

int

meersolar.utils.imaging.calc_field_of_view(msname, FWHM=True)

Calculate optimum field of view in arcsec.

Parameters:

• msname (str) – Measurement set name

• FWHM (bool, optional) – Upto FWHM, otherwise upto first null

Returns:

Field of view in arcsec

Return type:

float

meersolar.utils.imaging.calc_maxuv(msname, chan_number=-1)

Calculate maximum UV

Parameters:

• msname (str) – Name of the measurement set

• chan_number (int, optional) – Channel number

Returns:

• float – Maximum UV in meter

• float – Maximum UV in wavelength

meersolar.utils.imaging.calc_minuv(msname, chan_number=-1)

Calculate minimum UV

Parameters:

• msname (str) – Name of the measurement set

• chan_number (int, optional) – Channel number

Returns:

• float – Minimum UV in meter

• float – Minimum UV in wavelength

meersolar.utils.imaging.calc_multiscale_scales(msname, num_pixel_in_psf, chan_number=-1, max_scale=16)

Calculate multiscale scales

Parameters:

• msname (str) – Name of the measurement set

• num_pixel_in_psf (float) – Number of pixels in one PSF

• max_scale (float, optional) – Maximum scale in arcmin

Returns:

Multiscale scales in pixel units

Return type:

list

meersolar.utils.imaging.calc_npix_in_psf(weight, robust=0.0)

Calculate number of pixels in a PSF (could be fractional)

Parameters:

• weight (str) – Image weighting scheme

• robust (float, optional) – Briggs weighting robust parameter (-1 to +1)

Returns:

Number of pixels in a PSF

Return type:

float

meersolar.utils.imaging.calc_psf(msname, chan_number=-1)

Function to calculate PSF size in arcsec

Parameters:

• msname (str) – Name of the measurement set

• chan_number (int, optional) – Channel number

Returns:

PSF size in arcsec

Return type:

float

meersolar.utils.imaging.calc_sun_dia(freqMHz)

Function to calculate the diameter of the Sun at a given frequency (White 2016)

Parameters:

freq (float) – Frequency in MHz

Returns:

Diameter of the Sun in arcmin

Return type:

float

meersolar.utils.imaging.get_multiscale_bias(freq, bias_min=0.6, bias_max=0.9, minfreq=1015, maxfreq=1670)

Get frequency dependent multiscale bias

Parameters:

• freq (float) – Frequency in MHz

• bias_min (float, optional) – Minimum bias at minimum L-band frequency

• bias_max (float, optional) – Maximum bias at maximum L-band frequency

• minfreq (float, optional) – Minimum frequency range in MHz

• maxfreq (float, optional) – Maximum frequency range in MHz

Returns:

Multiscale bias patrameter

Return type:

float

meersolar.utils.imaging.get_optimal_image_interval(msname, temporal_tol_factor=0.1, spectral_tol_factor=0.1, chan_range='', timestamp_range='', max_nchan=-1, max_ntime=-1)

Get optimal image spectral temporal interval such that total flux max-median in each chunk is within tolerance limit

Parameters:

• msname (str) – Name of the measurement set

• temporal_tol_factor (float, optional) – Tolerance factor for temporal variation (default : 0.1, 10%)

• spectral_tol_factor (float, optional) – Tolerance factor for spectral variation (default : 0.1, 10%)

• chan_range (str, optional) – Channel range

• timestamp_range (str, optional) – Timestamp range

• max_nchan (int, optional) – Maxmium number of spectral chunk

• max_ntime (int, optional) – Maximum number of temporal chunk

Returns:

• int – Number of time intervals to average

• int – Number of channels to averages

meersolar.utils.logger_utils module

class meersolar.utils.logger_utils.LogTailHandler(logfile, logger)

Bases: FileSystemEventHandler

Continuous logging

on_modified(event)

Called when a file or directory is modified.

Parameters:

event (DirModifiedEvent or FileModifiedEvent) – Event representing file/directory modification.

class meersolar.utils.logger_utils.RemoteLogger(job_id='default', log_id='run_default', remote_link='', password='')

Bases: Handler

Remote logging handler for posting log messages to a web endpoint.

emit(record)

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

class meersolar.utils.logger_utils.SmartDefaultsHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)

Bases: ArgumentDefaultsHelpFormatter

class meersolar.utils.logger_utils.StreamToLogger(logger, log_level=20)

Bases: object

flush()

write(message)

meersolar.utils.logger_utils.clean_shutdown(observer)

meersolar.utils.logger_utils.create_logger(logname, logfile, get_print=False, verbose=False)

Create logger.

Parameters:

• logname (str) – Name of the log

• logfile (str, optional) – Log file name

• get_print (bool, optional) – Get print output to log

• verbose (bool, optional) – Verbose output or not

Returns:

• logger – Python logging object

• str – Log file name

meersolar.utils.logger_utils.generate_password(length=6)

Generate secure 6-character password with letters, digits, and symbols

meersolar.utils.logger_utils.get_emails()

meersolar.utils.logger_utils.get_logid(logfile)

Get log id for remote logger from logfile name

meersolar.utils.logger_utils.get_remote_logger_link()

meersolar.utils.logger_utils.init_logger(logname, logfile, jobname='', password='')

Initialize a remote logger with watchdog-based tailing.

Parameters:

• logname (str) – Logger name.

• logfile (str) – Path to the local logfile to also monitor.

• jobname (str, optional) – Remote logger job ID.

• password (str) – Password used for remote authentication.

Returns:

Observer object

Return type:

observer

meersolar.utils.logger_utils.ping_logger(jobid, remote_jobid, stop_event, remote_link='')

Ping a job-specific keep-alive endpoint periodically until stop_event is set.

meersolar.utils.meer_ploting_utils module

meersolar.utils.meer_ploting_utils.enhance_offlimb(sunpy_map, do_sharpen=True)

Enhance off-disk emission

Parameters:

• sunpy_map (sunpy.map) – Sunpy map

• do_sharpen (bool, optional) – Sharpen images

Returns:

Off-disk enhanced emission

Return type:

sunpy.map

meersolar.utils.meer_ploting_utils.get_aia_map(obs_date, obs_time, workdir, wavelength=193, keep_aia_fits=False)

Get SDO AIA map

Parameters:

• obs_date (str) – Observation date in yyyy-mm-dd format

• obs_time (str) – Observation time in hh:mm format

• workdir (str) – Work directory

• wavelength (float, optional) – Wavelength, options: 94, 131, 171, 193, 211, 304, 335 Å

• keep_aia_fits (bool, optional) – Keep AIA fits file or not

Returns:

Sunpy AIAMap

Return type:

sunpy.map

meersolar.utils.meer_ploting_utils.get_meermap(fits_image, band='', do_sharpen=False)

Make MeerKAT sunpy map

Parameters:

• fits_image (str) – MeerKAT fits image

• band (str, optional) – Band name

• do_sharpen (bool, optional) – Sharpen the image

Returns:

Sunpy map

Return type:

sunpy.map

meersolar.utils.meer_ploting_utils.get_suvi_map(obs_date, obs_time, workdir, wavelength=195, keep_suvi_fits=False)

Get GOES SUVI map

Parameters:

• obs_date (str) – Observation date in yyyy-mm-dd format

• obs_time (str) – Observation time in hh:mm format

• workdir (str) – Work directory

• wavelength (float, optional) – Wavelength, options: 94, 131, 171, 195, 284, 304 Å

• keep_suvi_fits (bool, optional) – Keep SUVI fits file or not

Returns:

Sunpy SUVIMap

Return type:

sunpy.map

meersolar.utils.meer_ploting_utils.make_ds_file_per_scan(msname, save_file, scan, datacolumn)

Extract dynamic spectrum from measurement set

Parameters:

• msname (str) – Measurement set name

• save_file (str) – File name to save dynamic spectrum

• scan (int) – Scan number

• datacolumn (str) – Data column name

Returns:

Dynamic spectrum file

Return type:

str

meersolar.utils.meer_ploting_utils.make_ds_plot(dsfiles, plot_file=None, showgui=False)

Make dynamic spectrum plot

Parameters:

• dsfile (list) – DS files list

• plot_file (str, optional) – Plot file name to save the plot

• showgui (bool, optional) – Show GUI

Returns:

Plot name

Return type:

str

meersolar.utils.meer_ploting_utils.make_meer_overlay(meerkat_image, suvi_wavelength=195, plot_file_prefix=None, plot_meer_colormap=True, enhance_offdisk=True, contour_levels=[0.05, 0.1, 0.2, 0.4, 0.6, 0.8], do_sharpen_suvi=True, xlim=[-1600, 1600], ylim=[-1600, 1600], extensions=['png'], outdirs=[], ncpu=-1, showgui=False, verbose=False)

Make overlay of MeerKAT image on GOES SUVI image

Parameters:

• meerkat_image (str) – MeerKAT image

• suvi_wavelength (float, optional) – GOES SUVI wavelength, options: 94, 131, 171, 195, 284, 304 Å

• plot_file_prefix (str, optional) – Plot file prefix name

• plot_meer_colormap (bool, optional) – Plot MeerKAT map colormap

• enhance_offdisk (bool, optional) – Enhance off-disk emission

• contour_levels (list, optional) – Contour levels in fraction of peak

• do_sharpen_suvi (bool, optional) – Do sharpen SUVI images

• xlim (list, optional) – X-axis limit in arcsec

• tlim (list, optional) – Y-axis limit in arcsec

• extensions (list, optional) – Image file extensions

• outdirs (list, optional) – Output directories for each extensions

• ncpu (int, optional) – Number of CPUs to use

• showgui (bool, optional) – Show GUI

• verbose (bool, optinal) – Verbose output

Returns:

Plot file names

Return type:

list

meersolar.utils.meer_ploting_utils.plot_caltable_diagnostics(caltable, outdir='')

Plot diagonistic plot of a caltable

Parameters:

• caltable (str) – Caltable name

• outdir (str, optional) – Output directory

Returns:

• int – Success messsage

• str – Output file

meersolar.utils.meer_ploting_utils.plot_goes_full_timeseries(msname, workdir, plot_file_prefix=None, extension='png', showgui=False)

Plot GOES full time series on the day of observation

Parameters:

• msname (str) – Measurement set

• workdir (str) – Work directory

• plot_file_prefix (str, optional) – Plot file name prefix

• extension (str, optional) – Save file extension

• showgui (bool, optional) – Show GUI

Returns:

Plot file name

Return type:

str

meersolar.utils.meer_ploting_utils.plot_in_hpc(fits_image, draw_limb=False, extensions=['png'], outdirs=[], plot_range=[], power=0.5, xlim=[-1600, 1600], ylim=[-1600, 1600], contour_levels=[], band='', showgui=False)

Function to convert MeerKAT image into Helioprojective co-ordinate

Parameters:

• fits_image (str) – Name of the fits image

• draw_limb (bool, optional) – Draw solar limb or not

• extensions (list, optional) – Output file extensions

• outdirs (list, optional) – Output directories for each extensions

• plot_range (list, optional) – Plot range

• power (float, optional) – Power stretch

• xlim (list) – X axis limit in arcsecond

• ylim (list) – Y axis limit in arcsecond

• contour_levels (list, optional) – Contour levels in fraction of peak, both positive and negative values allowed

• band (str, optional) – Band name

• showgui (bool, optional) – Show GUI

Returns:

• outfiles – Saved plot file names

• sunpy.Map – MeerKAT image in helioprojective co-ordinate

meersolar.utils.meer_ploting_utils.plot_ms_diagnostics(msname, outdir='', dask_client=None, cpu_frac=0.8, mem_frac=0.8)

Plot diagonistics plots for measurement set

Parameters:

• msname (str) – Measurement set

• outdir (str, optional) – Output directory

• dask_client (dask.client) – Dask client

• cpu_frac (float, optional) – CPU fraction

• mem_frac (float, optional) – Memory fraction

Returns:

• int – Success message

• list – Output plot file list

meersolar.utils.meer_ploting_utils.rename_meersolar_image(imagename, imagedir='', pol='', band='', attcal='NOINFO', cutout_rsun=2.5, make_overlay=True, make_plots=True)

Rename and move image to image directory

Parameters:

• imagename (str) – Image name

• imagedir (str, optional) – Image directory (default given image directory)

• pol (str, optional) – Stokes parameters

• band (str, optional) – Observing band

• attcal (str, optional) – Solar attenuation calibrated or not

• cutout_rsun (float, optional) – Cutout in solar radii from center (default: 2.5 solar radii)

• make_overlay (bool, optional) – Make overlay on SUVI

• make_plots (bool, optional) – Make radio map plot in helioprojective coordinates

Returns:

New imagename with full path

Return type:

str

meersolar.utils.meer_ploting_utils.save_in_hpc(fits_image, outdir='', xlim=[-1600, 1600], ylim=[-1600, 1600])

Save solar image in helioprojective coordinates

Parameters:

• fits_image (str) – FITS image name

• outdir (str, optional) – Output directory

• xlim (list) – X axis limit in arcsecond

• ylim (list) – Y axis limit in arcsecond

Returns:

FITS image in helioprojective coordinate

Return type:

str

meersolar.utils.meer_utils module

meersolar.utils.meer_utils.determine_noise_diode_cal_scan(msname, scan)

Determine whether a calibrator scan is a noise-diode cal scan or not

Parameters:

• msname (str) – Name of the measurement set

• scan (int) – Scan number

Returns:

Whether it is noise-diode cal scan or not

Return type:

bool

meersolar.utils.meer_utils.get_bad_chans(msname)

Get bad channels to flag

Parameters:

msname (str) – Name of the ms

Returns:

SPW string of bad channels

Return type:

str

meersolar.utils.meer_utils.get_band_name(msname)

Get band name

Parameters:

msname (str) – Name of the ms

Returns:

Band name (‘U’,’L’,’S’)

Return type:

str

meersolar.utils.meer_utils.get_cal_target_scans(msname)

Get calibrator and target scans

Parameters:

msname (str) – Name of the measurement set

Returns:

• list – Target scan numbers

• list – Calibrator scan numbers

• list – Fluxcal scans

• list – Phasecal scans

• list – Polcal scans

meersolar.utils.meer_utils.get_caltable_fields(caltable)

Get caltable field names

Parameters:

caltable (str) – Caltable name

Returns:

Field names

Return type:

list

meersolar.utils.meer_utils.get_fluxcals(msname)

Get fluxcal field names and scans (all scans, valids and invalids

Parameters:

msname (str) – Name of the ms

Returns:

• list – Fluxcal field names

• dict – Fluxcal scans

meersolar.utils.meer_utils.get_good_chans(msname)

Get good channel range to perform gaincal

Parameters:

msname (str) – Name of the ms

Returns:

SPW string

Return type:

str

meersolar.utils.meer_utils.get_phasecals(msname)

Get phasecal field names and scans (all scans, valids and invalids)

Parameters:

msname (str) – Name of the ms

Returns:

• list – Phasecal field names

• dict – Phasecal scans

• dict – Phasecal flux

meersolar.utils.meer_utils.get_polcals(msname)

Get polarization calibrator field names and scans (all scans, valids and invalids

Parameters:

msname (str) – Name of the ms

Returns:

• list – Polcal field names

• dict – Polcal scans

meersolar.utils.meer_utils.get_target_fields(msname)

Get target fields

Parameters:

msname (str) – Name of the measurement set

Returns:

• list – Target field names

• dict – Target field scans

meersolar.utils.meer_utils.get_valid_scans(msname, field='', min_scan_time=1, n_threads=-1)

Get valid list of scans

Parameters:

• msname (str) – Measurement set name

• field (str) – Field names (comma seperated)

• min_scan_time (float) – Minimum valid scan time in minute

Returns:

Valid scan list

Return type:

list

meersolar.utils.meer_utils.split_noise_diode_scans(msname='', noise_on_ms='', noise_off_ms='', field='', scan='', datacolumn='data', n_threads=-1)

Split noise diode on and off timestamps into two seperate measurement sets

Parameters:

• msname (str) – Measurement set

• noise_on_ms (str, optional) – Noise diode on ms

• noise_off_ms (str, optional) – Noise diode off ms

• field (str, optional) – Field name or id

• scan (str, optional) – Scan number

• datacolumn (str, optional) – Data column to split

Returns:

splited ms names

Return type:

tuple

meersolar.utils.ms_metadata module

meersolar.utils.ms_metadata.baseline_names(msname)

Get baseline names

Parameters:

msname (str) – Measurement set name

Returns:

Baseline names list

Return type:

list

meersolar.utils.ms_metadata.calc_fractional_bandwidth(msname)

Calculate fractional bandwidh

Parameters:

msname (str) – Name of measurement set

Returns:

Fraction bandwidth in percentage

Return type:

float

meersolar.utils.ms_metadata.check_datacolumn_valid(msname, datacolumn='DATA')

Check whether a data column exists and valid

Parameters:

• msname (str) – Measurement set

• datacolumn (str, optional) – Data column string in table (e.g.,DATA, CORRECTED_DATA’, MODEL_DATA, FLAG, WEIGHT, WEIGHT_SPECTRUM, SIGMA, SIGMA_SPECTRUM)

Returns:

Whether valid data column is present or not

Return type:

bool

meersolar.utils.ms_metadata.get_bad_ants(msname='', fieldnames=[], n_threads=-1)

Get bad antennas

Parameters:

• msname (str) – Name of the ms

• fieldnames (list, optional) – Fluxcal field names

Returns:

• list – Bad antenna list

• str – Bad antenna string

meersolar.utils.ms_metadata.get_chunk_size(msname, memory_limit=-1, only_autocorr=False)

Get time chunk size for a memory limit

Parameters:

• msname (str) – Measurement set

• memory_limit (int, optional) – Memory limit

• only_autocorr (bool, optional) – Only aut-correlation

Returns:

Number of chunks

Return type:

int

meersolar.utils.ms_metadata.get_column_size(msname, only_autocorr=False)

Get datacolumn size (Note: this is true datasize in memory)

Parameters:

• msname (str) – Measurement set

• only_autocorr (bool, optional) – Only auto-correlations

Returns:

A single datacolumn data size in GB

Return type:

float

meersolar.utils.ms_metadata.get_common_spw(spw1, spw2)

Return common spectral windows in merged CASA string format.

Parameters:

• spw1 (str) – First spectral window (0:xx~yy)

• spw2 (str) – Second spectral window (0:xx1~yy1)

Returns:

Merged spectral window

Return type:

str

meersolar.utils.ms_metadata.get_ms_scan_size(msname, scan, only_autocorr=False)

Get measurement set scan size

Parameters:

• msname (str) – Measurement set

• scan (int) – Scan number

• only_autocorr (bool, optional) – Only for auto-correlations

Returns:

Size in GB

Return type:

float

meersolar.utils.ms_metadata.get_ms_scans(msname)

Get scans of the measurement set

Parameters:

msname (str) – Measurement set

Returns:

Scan list

Return type:

list

meersolar.utils.ms_metadata.get_ms_size(msname, only_autocorr=False)

Get measurement set total size on-disk (Note: it could be smaller than actual data size, because of data compression)

Parameters:

• msname (str) – Measurement set name

• only_autocorr (bool, optional) – Only auto-correlation

Returns:

Size in GB

Return type:

float

meersolar.utils.ms_metadata.get_observatory_coord(msname)

Get observatory coordinate

Parameters:

msname (str) – Measurement set

Returns:

• float – Latitude in degrees

• float – Longitude in degrees

• float – Height in meters

meersolar.utils.ms_metadata.get_observatory_name(msname)

Get observatory name

Parameters:

msname (str) – Measurement set

Returns:

Observatory name in all upper case

Return type:

str

meersolar.utils.ms_metadata.get_phasecenter(msname, fieldID=0)

Get phasecenter of the measurement set

Parameters:

• msname (str) – Name of the measurement set

• fieldID (int, optional) – Zero based field ID

Returns:

• float – RA in degree

• float – DEC in degree

meersolar.utils.ms_metadata.get_pol_names(msname, fullpol=True)

Get correlation names

Parameters:

• msname (str) – Measurement set

• fullpol (bool, optional) – Full polarization products or not

Returns:

List of cross correlation product names

Return type:

list

meersolar.utils.ms_metadata.get_refant(msname='', field='', n_threads=-1)

Get reference antenna

Parameters:

• msname (str) – Name of the measurement set

• field (str, optional) – Field name

Returns:

Reference antenna

Return type:

str

meersolar.utils.ms_metadata.get_submsname_scans(msname)

Get sub-MS names for each scans of an multi-MS

Parameters:

msname (str) – Name of the measurement set

Returns:

• list – msname list

• list – scan list

meersolar.utils.ms_metadata.get_timeranges_for_scan(msname, scan, time_interval, time_window, quack_timestamps=-1)

Get time ranges for a scan with certain time intervals

Parameters:

• msname (str) – Name of the measurement set

• scan (int) – Scan number

• time_interval (float) – Time interval in seconds

• time_window (float) – Time window in seconds

• quack_timestamps (int, optional) – Number of timestamps ignored at the start and end of each scan

Returns:

List of time ranges

Return type:

list

meersolar.utils.ms_metadata.get_uvrange_exclude(uvrange)

Get uv-range(s) excluding the given uv-range

Parameters:

uvrange (str) – UV-range in CASA format

Returns:

List of uvranges excluding the given uv-range

Return type:

list

meersolar.utils.ms_metadata.scans_in_timerange(msname='', timerange='')

Get scans in the given timerange

Parameters:

• msname (str) – Measurement set

• timerange (str) – Time range with date and time

Returns:

Scan dict for timerange

Return type:

dict

meersolar.utils.prefect_logger_utils module

meersolar.utils.prefect_setup_utils module

meersolar.utils.proc_manage_utils module

meersolar.utils.proc_manage_utils.generate_activate_env(outfile='activate_env.sh')

Generate a shell script that activates the current Python environment.

This works for both Conda and virtualenv environments and is safe for use in non-interactive shells (e.g., Slurm batch jobs) by explicitly sourcing conda.sh.

If conda is not found in $PATH, it will try loading either anaconda or anaconda3 module.

Parameters:

outfile (str) – Path to the shell script to write (default: ./activate_env.sh).

Returns:

Output file name

Return type:

str

meersolar.utils.proc_manage_utils.get_jobid()

Get Job ID with millisecond-level uniqueness.

Returns:

Job ID in the format YYYYMMDDHHMMSSmmm (milliseconds)

Return type:

int

meersolar.utils.proc_manage_utils.get_local_dask_cluster(njobs, dask_dir, cpu_frac=0.8, mem_frac=0.8, ncpu=-1, mem=-1, spill_frac=0.7, verbose=True)

Create a local Dask cluster

Parameters:

• njobs (int) – Number of MS tasks (ideally = number of MS files)

• dask_dir (str) – Dask temporary directory

• cpu_frac (float, optional) – Fraction of total CPUs to use

• mem_frac (float, optional) – Fraction of total memory to use

• ncpu (int, optional) – Number of CPUs to use (if specified, cpu_frac will be ignored)

• mem (float, optional) – Memory in GB to use (if specified, mem_frac will be ignored)

• spill_frac (float, optional) – Spill to disk at this fraction

• verbose (bool, optional) – Verbose (details of cluster)

Returns:

• client (dask.distributed.Client) – Dask client

• cluster (dask.distributed.LocalCluster) – Dask cluster

• str – Dask directory

meersolar.utils.proc_manage_utils.get_nprocess_solarpipe(jobid)

Get numbers of processes currently running

Parameters:

• workdir (str) – Work directory name

• jobid (int) – Job ID

Returns:

Number of running processes

Return type:

int

meersolar.utils.proc_manage_utils.get_scheduler_name()

Get job scheduler available

Returns:

Scheduler name (local, pbs, slurm)

Return type:

str

meersolar.utils.proc_manage_utils.get_slurm_dask_cluster(njobs, config_yaml, dask_dir, cpu_frac=0.8, mem_frac=0.8, ncpu=-1, mem=-1, spill_frac=0.7, verbose=True)

Launch a SLURMCluster using a YAML configuration and return a connected Dask client.

Parameters:

• njobs (int) – Number of expected tasks (used for worker scaling)

• config_yaml (str) – Path to Dask SLURMCluster YAML configuration

• dask_dir (str) – Dask working directory (for temporary files)

• cpu_frac (float) – Fraction of total CPUs to use (ignored if ncpu > 0)

• mem_frac (float) – Fraction of total RAM to use (ignored if mem > 0)

• ncpu (int) – Total CPUs to use (overrides cpu_frac)

• mem (float) – Total memory (in GB) to use (overrides mem_frac)

• spill_frac (float) – Fraction of memory to spill to disk

• verbose (bool) – Print Dask dashboard URL and diagnostics

Returns:

• client (dask.distributed.Client) – Connected Dask client

• cluster (dask_jobqueue.SLURMCluster) – SLURM Dask cluster

• str – Dask directory used

meersolar.utils.proc_manage_utils.get_total_worker(cluster)

Get total workers in the cluster

Parameters:

cluster (dask.cluster) – Dask cluster

Returns:

Number of workers

Return type:

int

meersolar.utils.proc_manage_utils.save_main_process_info(pid, jobid, msname, workdir, outdir, cpu_frac, mem_frac)

Save main processes info

Parameters:

• pid (int) – Main job process id

• jobid (int) – Job ID

• msname (str) – Main measurement set

• workdir (str) – Work directory

• outdir (str) – Output directory

• cpu_frac (float) – CPU fraction of the job

• mem_frac (float) – Memory fraction of the job

Returns:

Job info file name

Return type:

str

meersolar.utils.proc_manage_utils.save_pid(pid, pid_file)

Save PID

Parameters:

• pid (int) – Process ID

• pid_file (str) – File to save

meersolar.utils.proc_manage_utils.scale_worker_and_wait(dask_cluster, nworker, timeout=60, poll_interval=1)

Scale worker and wait until it is done

Parameters:

• dask_cluster (dask.cluster) – Dask cluster

• nworker (int) – Number of worker

• timeout (float, optional) – Timeout, show a warning and move

• poll_interval (float, optional) – Check interval in seconds

meersolar.utils.proc_manage_utils.wait_for_dask_workers(client, min_worker=1, timeout=60)

Wait until the Dask cluster has a minimum number of total and/or new workers.

Parameters:

• client (dask.distributed.Client) – Dask client

• min_worker (int, optional) – Minimum new connected workers (default: 1)

• timeout (float, optional) – Maximum time to wait in seconds (default: 60)

Raises:

TimeoutError – If the required number of workers do not connect in time.

meersolar.utils.resource_utils module

meersolar.utils.resource_utils.drop_cache(path, verbose=False)

Drop file cache for a file or all files under a directory.

Parameters:

path (str) – File or directory path

meersolar.utils.resource_utils.drop_file_cache(filepath, verbose=False)

Advise the OS to drop the given file from the page cache. Safe, per-file, no sudo required.

meersolar.utils.resource_utils.has_space(path, required_gb)

meersolar.utils.resource_utils.limit_threads(n_threads=-1)

Limit number of threads usuage

Parameters:

n_threads (int, optional) – Number of threads

meersolar.utils.resource_utils.shm_or_tmp(required_gb, workdir, prefix='solar_', verbose=False)

Create a temporary working directory: 1. Try /dev/shm if it has required space 2. Else TMPDIR if set and has space 4. Else work directory Temporarily sets TMPDIR to the selected path during the context. Cleans up after use.

Parameters:

• required_gb (float) – Required disk space in GB

• workdir (str) – Fall back work directory

• prefix (str, optional) – Temp directory prefix

• verbose (bool, optional) – Verbose

meersolar.utils.resource_utils.tmp_with_cache_rel(required_gb, workdir, prefix='solar_', verbose=False)

Combined context manager: - Uses shm_or_tmp() for workspace - Drops kernel page cache for all files in that directory on exit

Parameters:

• required_gb (float) – Required disk space in GB

• workdir (str) – Fall back work directory

• prefix (str, optional) – Temp directory prefix

• verbose (bool, optional) – Verbose

meersolar.utils.selfcal_utils module

meersolar.utils.selfcal_utils.intensity_selfcal(msname, logger, selfcaldir, cellsize, imsize, round_number=0, uvrange='', minuv=0, calmode='ap', solint='60s', refant='1', solmode='', gaintype='T', applymode='calonly', threshold=3, weight='briggs', robust=0.0, use_previous_model=False, use_solar_mask=True, mask_radius=20, min_tol_factor=-1, ncpu=-1, mem=-1)

A single self-calibration round

Parameters:

• msname (str) – Name of the measurement set

• logger (logger) – Python logger

• selfcaldir (str) – Self-calibration directory

• cellsize (float) – Cellsize in arcsec

• imsize (int) – Image pixel size

• round_number (int, optional) – Selfcal iteration number

• uvrange (float, optional) – UV range for calibration

• calmode (str, optional) – Calibration mode (‘p’ or ‘ap’)

• solint (str, optional) – Solution intervals

• refant (str, optional) – Reference antenna

• applymode (str, optional) – Solution apply mode (calonly or calflag)

• threshold (float, optional) – Imaging and auto-masking threshold

• weight (str, optional) – Image weighting

• robust (float, optional) – Robust parameter for briggs weighting

• use_previous_model (bool, optional) – Use previous model

• use_solar_mask (bool, optional) – Use solar disk mask or not

• mask_radius (float, optional) – Mask radius in arcminute

• min_tol_factor (float, optional) – Minimum tolerance factor

• ncpu (int, optional) – Number of CPUs to use in WSClean

• mem (float, optional) – Memory usage limit in WSClean

Returns:

• int – Success message

• str – Caltable name

• float – RMS based dynamic range

• float – RMS of the image

• str – Image name

• str – Model image name

• str – Residual image name

meersolar.utils.sunpos_utils module

meersolar.utils.udocker_utils module

meersolar.utils.udocker_utils.check_udocker_container(name)

Check whether a docker container is present or not

Parameters:

name (str) – Container name

Returns:

Whether present or not

Return type:

bool

meersolar.utils.udocker_utils.init_udocker()

meersolar.utils.udocker_utils.initialize_shadems_container(name='solarshadems', update=False)

Initialize shadems container

Parameters:

• name (str, optional) – Name of the container

• update (bool, optional) – Update container

Returns:

Whether initialized successfully or not

Return type:

bool

meersolar.utils.udocker_utils.initialize_wsclean_container(name='solarwsclean', update=False)

Initialize WSClean container

Parameters:

• name (str, optional) – Name of the container

• update (bool, optional) – Update container

Returns:

Whether initialized successfully or not

Return type:

bool

meersolar.utils.udocker_utils.run_chgcenter(msname, ra, dec, only_uvw=False, container_name='solarwsclean', check_container=False, verbose=False)

Run chgcenter inside a udocker container (no root permission required).

Parameters:

• msname (str) – Name of the measurement set

• ra (str) – RA can either be 00h00m00.0s or 00:00:00.0

• dec (str) – Dec can either be 00d00m00.0s or 00.00.00.0

• only_uvw (bool, optional) – Update only UVW values Note: This is required when visibilities are properly phase rotated in correlator, but while creating the MS, UVW values are estimated using a wrong phase center.

• check_container (bool, optional) – Check container

• container_name (str, optional) – Container name

• verbose (bool, optional) – Verbose output

Returns:

Success message

Return type:

int

meersolar.utils.udocker_utils.run_shadems(cmd, container_name='solarshadems', check_container=False, verbose=False)

Run shadems inside a udocker container (no root permission required).

Parameters:

• cmd (str) – Shadems command

• container_name (str, optional) – Container name

• check_container (bool, optional) – Check container

• verbose (bool, optional) – Verbose output

Returns:

Success message

Return type:

int

meersolar.utils.udocker_utils.run_solar_sidereal_cor(msname='', only_uvw=False, container_name='solarwsclean', check_container=False, verbose=False)

Run chgcenter inside a udocker container to correct solar sidereal motion (no root permission required).

Parameters:

• msname (str) – Name of the measurement set

• only_uvw (bool, optional) – Update only UVW values Note: This is required when visibilities are properly phase rotated in correlator to track the Sun, but while creating the MS, UVW values are estimated using the first phasecenter of the Sun.

• check_container (bool, optional) – Check container

• container_name (str, optional) – Container name

• verbose (bool, optional) – Verbose output or not

Returns:

Success message

Return type:

int

meersolar.utils.udocker_utils.run_wsclean(wsclean_cmd, container_name='solarwsclean', check_container=False, verbose=False)

Run WSClean inside a udocker container (no root permission required).

Parameters:

• wsclean_cmd (str) – Full WSClean command as a string.

• container_name (str, optional) – Container name

• check_container (bool, optional) – Check container presence or not

• verbose (bool, optional) – Verbose output or not

Returns:

Success message

Return type:

int

meersolar.utils.udocker_utils.set_udocker_env()