JSOC Help and Examples

---

Examples of JSOC RecordSet Specifiers

Basics

For all users, it is highly recommended that you understand the instructions at JSOC Data Access.

Once you are familiar with the basic structure of JSOC Datasets, including the JSOC term "PrimeKeyword" please review JSOC data series in the JSOC Wiki.

How to Get Data FAQ provides an accessible description of how to get data.

Simple Time-Series Examples

Many JSOC dataseries are a time-ordered sequence of records, such as hmi.M_45s - the set of HMI magnetograms observed once each 45 seconds. For such series the first PrimeKey is the time of the record. Its name will ordinarily be "T_REC" for HMI and MDI data but may be some other time keyword such as "T_OBS". The name of a PrimeKey need not be specified if you know the order in which they are defined. In a dataseries that is effectively a time series, the standard is that the first PrimeKey will be the the ordering time keyword.

Example 1 - Export the single most recent HMI 45s magnetogram

• Go to http://jsoc.stanford.edu/ajax/exportdata.html
• In the RecordSet box, enter the data series and time, hmi.M_45s[$], where the $ inside the brackets is the symbol for the final record in the series.
• Hit the Recount button to make sure your query produced something (it should report a Record Count of 1)
• The default is to provide a stripped down fits file, but since you probably want fits files with all the Keywords, under Method select either url or ftp.
  When you do this a number of options with reasonable defaults will pop up. You can ignore them, except to enter your name in the Requestor box and contact email (or "NO") in the Notify slot.
• Click Check Params for Export to verify everything is OK.
• Select the green Submit Export Request button to request fits file. It will report your RequestID and an estimate of the size of the request.
• After a minute click the Submit Status Request button and it will provide a link to your fits file.

Example 2 - Export All of the 45s Magnetograms for Some Time Range

Suppose you want all of the 45s magnetograms for July 30, 2010 between 13:30 and 14:00 TAI.
• Following the instructions in Example 1, enter the RecordSet:
  • hmi.M_45s[2010.07.25_13:30_TAI-2010.07.25_14:00_TAI]
  Or, almost same set, but a more compact specification
  • hmi.M_45s[2010.07.25_13:30_TAI/30m]

The difference between the two samples above is that the first form, a range of times, includes any records that exist for the full range including both end times. In the second case, where the "/30m" can be read as "over 30 minutes", the span of records matching will stop just before the given time plus the span duration. This makes sense if you want all the images for May 1, 2010 then you would write: "hmi.M_45s[2010.05.01/1d]" and the result would include only magnetograms for May 1, not the first one for May 2. Of course you could specify "hmi.M_45s[2010.05.01_00:00:00-2010.05.01_23:59:15]" but the compact form is both easier to type and easier to read. The first sample above returns 41 records and the second returns 40.

Example 3: Export a Selection of AIA Data at Two Wavelengths

Suppose you want 12s AIA EUV data for July 30, 2010 between 13:30 and 14:00 UTC for only the 19.3 nm line.
Using exportdata as described in examples one and 2 above:
• aia.lev1_euv_12s[2010.07.25_13:30/30m][? WAVELNTH = 193 ?]

If you want data for the same time, but also for 33.5nm

Using exportdata as described in examples one and 2 above:
• aia_lev1_euv_12s[2010.07.25_13:30_TAI/30m][193,335]

Example 4 - Export a 720s Magnetogram Every 96 Minutes for One Day

Suppose you want to compare the 720s hmi magnetograms with the 96-minute mdi magnetograms on May 1, 2010.
Following the examples above, run exportdata twice, entering the RecordSets:
• hmi.M_720s[2010.05.01/1d@96m] and
• mdi.fd_M_96m_lev182[2010.05.01/1d]

The first request will return the hmi magnetograms beginning 2010.05.01 for the 1 day (using the "/" operator), but only one magnetogram every 96 minutes (using the @ operator).

The second request will return all of the level 1.8.2 96-minute mdi magnetograms for the same day.

If you just want selected records for a known time range, say Bp,Br,and Bt for SHARP 3563 in equal area projection then:

• hmi.sharp_cea_720s[3563][2014.01.06_23:45-2014.01.07_0:14]{Bp,Br,Bt}

Using the lookdata Tool

The lookdata utility can be used for searching, examining, and exporting data. There is documentation at Lookdata Wiki Page and at the PDF Tutorial. Lookdata helps you search for a data series, gives information about the series you select, allows to you build up a RecordSet query, reports Keyword values for the records you select, allows you to graph the values of Keyword Series, and transfers the results of your search to the exportdata utility described above.

Example 5: Use lookdata to Find Data and Examine Keywords Before Exporting

Suppose you want to see if the mean field of the HMI 720s magnetograms varies with the velocity of the SDO spacecraft during the day.

This example is a little longer because it introduces another useful utility, the lookdata program.

• Go to http://jsoc.stanford.edu/ajax/lookdata.html
• Under 1. Find a list of dataseries enter part of the data series name you want to look for, "M_720s". Then click Fetch seriesname list to search for jsoc data series containing "M_720s".
• Panel 2 now contains a list of data series. Click the name of the series you want, hmi.M_720s. This will take you to the RecordSet Select panel.
  The RecordSet Select panel gives a lot of summary information about the data series, including the times of the first and last record. You can check the SeriesContent panel for a description of all the keywords.
• In Panel 3 you can enter the RecordSet information as described above, or you can check the box to be assisted by the QueryBuilder. Check the box.
  • Query Builder for hmi.M_720s

             T_REC	[    ]	(time-slotted, 720.0)	 expects TAI
             CAMERA	[    ]	(int)	
             where clause	[?  ?]	Uses prime key logic
             where clause	[!  !]	use disables prime key logic
      

  This example shows what you see with the checkbox checked and the hmi.M_720s series selected. It shows that hmi.M_720s has two PrimeKeys, "T_REC" and "CAMERA". HMI has two cameras, one called 1:Side and the other 2:Front (referring to the location of the electronics box on the HMI instrument). Since each basic observable such as magnetic field or Doppler velocity can be computed for data taken by either camera, we indicate the particular camera with the CAMERA PrimeKey. At least up through July 2010, all 720s-second cadence Magnetograms have been made using only the Front camera so we may safely omit the CAMERA keyword in a data request.
  • In the T_REC box enter "2010.07.05/1d". Check the GetRecordCount box and you will see that there were the nominal 120 magnetograms for the day.
• In Panel 4 appears a list where you can Select Keywords. Scroll down and click on QUALITY.
• In Panel 5 appears a list where you can Select Segments. Click on NONE.
• At the bottom of the page click on Fetch Keyword Values for RecordSet. This will take you to the Values Display panel. A row for each record shows the values of the keywords. You might have to make the column wider to see the value of the QUALITY keyword. Scroll down and you will notice a non-zero QUALITY value at 2010.07.05_06:00:00_TAI where one or more of the expected filtergrams was missing (See QUALITY for more info).
• Go back to the RecordSet Select page by clicking on the appropriate tab near the top of the page. You'll see everything as you left it before you fetched the keyword values.
• Now in Section 3 in the where clause [? ?] section enter "QUALITY=0", then click GetRecordCount. You'll see there are just 114 "perfect" magnetograms for the day.
• To export these 114, go to the Export Data tab by selecting the appropriate label at the top of the page. Once there, click Export to be taken to the exportdata utility described in the previous examples.
• To further examine the keywords lookdata can create a graph.
  • Go to the RecordSet Select tab and in Panel 4 Select Keywords click on **ALL**. Then once again click on Fetch Keyword Values for RecordSet. This will take you to the Values Display tab and show the values of all of the Keywords in the series for the selected record set. Explore as you like.
  • Next, go to the Graph tab by selecting the appropriate tab at the top of the page. Select for the X Axis T_REC and select keywords DATAMEAN and OBS_VR by <ctrl-click>ing.
  • Finally, click on Create Graph button to see if the Mean Field varies during the day as the spacecraft velocity changes during the day.

Example 6: Use lookdata to Find The SHARP Data Associated with NOAA AR 11158

The SHARP data series contains space weather information and magnetic field data for HMI Active Region Patches (HARPs). The data are organized by prime keywords HARPNUM (the HARP number) and time T_REC. Suppose you know the time interval and NOAA active region number you are interested in, but not the HARP (HMI Active Region Patch) number. Just to make it a little more interesting, a HARP can be quite large and encompass more than one NOAA active region. The first-appearing NOAA number associated with a HARP will generally be recorded in the keyword NOAA_AR, but a list of all of the NOAA numbers appears only as a comma separated list in the string keyword NOAA_ARS. Suppose you are interested in the vector magnetic field data for AR 11158 in February, 2011.
• Following the example above, use lookdata to find the SHARP data series hmi.sharp_720s.
• In Section 3 of the Record Select tab, check the QueryBuilder box.
• In the T_REC section, enter the time range for 2011 February as 2011.02.01/28d.
• Just below in the Where [? ?] section enter NOAA_ARS ~ "11158" to match the characters 11158 in the keyword NOAA_ARS. The quotation marks around 11158 are important.
• Click on GetRecordCount to check the number of records
• Finally, after selecting the Keywords, Segments, and Links to retrive, click on Fetch Keyword Record Values Warning: Lookdata may fail if the number of keywords is too large, select a subset if you get the error message Something went wrong with series data request.

  Once you have identified the HARPNUM for the region of interest (in this case 377), you can specify the HARPNUM directly in future requests; it will be much quicker than the Where [? ?] query.

Note About Notation for Time

The JSOC DRMS time parsing code can recognize several formats for expressing time. In particular it understands a subset of ISO-8601 formats such as 2010-07-25T13:30:00 and the SDO/HMI (and SOHO/MDI) format of 2010.07.25_13:30_00. Trailing time or date parts that are omitted default to the start of the respecitve interval. E.g. 2010.07.25_13 corresponds to 13:00 UTC on 25 July 2010 as one would expect. If the ISO format is used, at least the hour part must be present so the programs can tell the difference between an ISO time and a range of times. See JSOC-TN 07-001 for the full specification.

Time zones are a special topic. HMI data it time tagged in TAI time, "International Atomic Time", since they are observed on SDO using TAI time. There are no leap-seconds in TAI time as there are in UTC so for long time series such as the 14 year SOHO/MDI set of heliosiesmology data one does not need to worry about the 5 leap-seconds that were inserted during that time. All SDO data are intrinsically observed using TAI. For the convenience of people the times can also be expressed in UTC. HMI data has "nicer" time stamps when expressed in TAI. By choice, AIA uses UTC for most user-level time stamps. Either may be used for any JSOC data requests. Since the offset, now 34 seconds, is more than half the basic HMI obseving cadence of 45 seconds, you will often not get the record you expect if you use UTC to ask for HMI data. The default for an omitted time zone specifier is UTC. Thus if you ask for hmi.M_45s[2010.07.25] you will get the magnetogram labelled hmi.M_45s[2010.07.25_00:00:45_TAI].

The rule of thumb is: Use TAI when specifing HMI or MDI data but use UTC for AIA.

More About Prime Keys

A JSOC Dataseries may have multiple Prime Keys, between 0 and 15. Usually 1 or a few. When multiple Prime Keys are used, a record is fully specified by the complete set of keyword values for the PrimeKeys. The PrimeKeys can be thought of as a filter acting on the full set of records in the given dataseries. I.e. only records matching all of the given PrimeKey values will be selected. If some PrimeKeys are omitted then all records matching the given PrimeKeys but having any values for the omitted PrimeKeys will be returned.

The full set of PrimeKeys for each series are listed in lookdata both on the "Series Content" tab and at the top of the "RecordSet Select" tab. If you check the checkbox labelled "Check box to show the QueryBuilder" located just under the title "Select Records and Get Record Count" on the "RecordSet Select" tab, lookdata will provide you with an expanded template for building a recordset query using all of the available PrimeKeys.

Generally, each PrimeKey will be in some sense orthogonal to the others. They specify different quantities which are all needed for a unique record specification. This is not required. In fact, AIA and HMI each use both T_OBS and FSN as PrimeKeys for the "Level-1" dataseries. T_OBS is always the time of observation of a quantity. For the low-level HMI and AIA data, "FSN" is the "Frame Sequence Number" and is a unique number provided by the onboard software when each image is created. Since no two images can be made at exactly the same time, FSN is redundant with T_OBS. However, rounded to the nearest second and in testing multiple images can be obtained with the same T_OBS rounded value. But the reason we chose to use both for this low-level data is that the data is generated based on FSN but used based on T_OBS. You should use "T_OBS" for aia.lev1 data unless you really know what you are after.

While the built-in PrimeKey formats are convenient with notation like "2010.07.25/6h" sometimes the user wants to either build a query from other not Prime Keywords or to make a query that is noyt easy to specify using the PrimeKey special logic. In those cases we provide the ability to specify one or more very geenral SQL "Where" clauses. An SQL where clause contents may be provided in one or more "[? ... ?]" or "[! ... !]" clauses. The word "where" is not needed, or allowed.