Author Topic: To process a new version of the Studies spreadsheet  (Read 1983 times)

PeterC

  • Administrator
  • Member
  • *****
  • Posts: 483
  • Passionate about my one-place study
    • View Profile
    • Holywell-cum-Needingworth History
To process a new version of the Studies spreadsheet
« on: 9 April 2018, 03:57:51 »
The SOPS' Studies spreadsheet lists all the One-Place Studies registered with the society and relevant data about them. M4OPS requires a json version of the latest of these, to provide information about studies known to M4OPS such as Continent/Country.

Whenever a new study is added to M4OPS that is not in the most recent studies.json file we need to create a new version of this file.  The process for doing this is in five parts:
  • Obtain a copy of the latest spreadsheet on your PC
  • Prepare this copy of the spreadsheet so that it is suitable for M4OPS
  • Transfer a copy of this prepared spreadsheet to the M4OPS server
  • Compile the spreadsheet into a json format
  • Put the json file into the appropriate places and test

To obtain a copy of the latest spreadsheet:
  • Obtain the latest "StudiesYYYYMMDD.xlsx" file from the Society and download to your PC (Peter uses the Mapping\Software\M4OPS folder)
  • Note that people's names are not needed, and not usually supplied due to privacy concerns
  • (If not available, or as a temporary expedient, you could just add new studies to the current xlsx file)

To prepare this copy of the spreadsheet so that it is suitable for M4OPS:
  • Open the downloaded file in Excel, and enable editing
  • File/Save As Text (Tab delimited) (*.txt)
  • If an option, click on OK for the Active Sheet only
  • Exit Excel
  • Rename the file as "StudiesYYYYMMDD.xlsx - Active.tsv" (tsv is our name for Tab Separated Values)
  • Open this file in an editor (Peter uses Notepad++) and
    • (it may help to show all white space characters, particularly tabs)
    • delete all the empty (or tab-only) rows at the end of the file
    • delete any empty (or tab-only) rows in the middle of the file
    • check there are no semi-colons in the file (and if there are change them)
    • change all tabs to semi-colons
    • check for any double quotes, which may indicate a split line, and remove them
    • delete excess semi-colons at the end of lines (including the last line), except for any representing blank website values
    • Encoding - click on Convert to UTF-8, save and reload to check it is still encoded as UTF-8 (as json needs this)

To transfer a copy of this prepared spreadsheet to the M4OPS server:
  • FTP the .tsv to /ShowMapsDev/
  • On /ShowMapsDev/ rename the old Studies.csv as StudiesOLD.csv (having deleted any previous OLD versions)
  • On /ShowMapsDev/ rename the .tsv file to Studies.csv

To compile the spreadsheet into a json format:
  • Open mapping4ops.org/ShowMapsDev/CompileStudies.php?BRs&Test to precompile the Studies.json file
  • Make sure that the only OPS listed as "Location not set from Studies.csv" are those we know of (eg BAL, BRI, POR, USG, EDI, LGIWG), or for which we have not yet compiled the OPS.json file
  • (If there are problems later you may want to check the resulting Studies.json file)

To put the json file into the appropriate places and test:
  • On your local PC (Peter uses the Mapping\Software\M4OPS folder) rename the old Studies.json as StudiesOLD.json (having deleted any previous OLD versions)
  • FTP Studies.json from /ShowMapsDev/ to your local PC (Peter uses the Mapping\Software\M4OPS folder)
  • Test the development M4OPS, and when happy all is OK, FTP the Studies.json file from your local PC to /ShowMaps/ (ie the production M4OPS system)
  • (Note that there is no need to repeat the precompile process on the production system, as all that M4OPS needs is this Studies.json file)
  • Test the production M4OPS

« Last Edit: 25 January 2021, 15:39:28 by PeterC »