Mapping for One-Place Studies - a forum

To create a Modifiable Feature Layer (MFL) go to your OPS in M4OPS and then:

• under Advanced Options click on the MFL tab
• click on New - this will create a new blank Modifiable Feature Layer (MFL), and open some more buttons
• click in the MFL name box and enter a name by which this Modifiable Feature Layer (MFL) will be known
• in most cases leave the Fields dropdown as "Minimal Fields" (explained later)
• now for each feature you want to create:
  • depending on the feature you want to create, click on the Point, Line or Polygon button (the only difference between a Line and Polygon is that the first and last points of a Polygon are joined up
  • (note you can change everything later, so it is much better to start with just an approximate Point, or a simple Line or Polygon, in approximately the right area)
  • on the map, click where you want your first point
  • for a Line or Polygon, you now need to:
    • click on further places on the map to create vertices (points on the Line or Polygon which are automatically joined up)
    • double click to create the last vertex and indicate you have finished
  • having finished the "geometry" of the feature, you will be shown a little form where you can enter its "short text" and "description" (these are  the Minimal Fields mentioned above) - use this to name and describe the feature
  • click on Save to save these fields, then Close to close the little form
• click where you want your next feature etc, again enter its  "short text", Save and Close the little form
• repeat as often as you want (but bearing in mind that this is a prototype, do not spend too long entering large amounts of data as it might all disappear)
• at any point click on the main Save button to save this Modifiable Feature Layer (MFL) in M4OPS (you can do this as many times as you want)
• later, if you wish to move the positions of any of the points (or vertices on a Line or Polygon) then click on Move, and then click and drag the points to exactly where you want them
• if you click and drag any point on a Line or Polygon this creates a new vertex - and they can therefore be made as complex as you wish
• to delete any point (or vertex on a Line or Polygon) then click on Move, and double-click on the point
• if you wish to enter, or change, the properties (fields) of any points, or to change the type of geometry (point, line, polygon) of any of the features, then click on Alter and fill in the form - remembering to Save the form before you Close it
• when you have finished with this MFL click on the Close button, and all the extra buttons will disappear
• to look at, add to, or modify your MFL simply click on the MFL tab and select your saved MFL from the dropdown - and then continue as above, making sure you save the MFL when you have down any changes

When you are happy with the details of your features (particularly their "geometry" ie where they are and their shapes) discuss with Peter what to do with it.

OpenAddresses.io is "The free and open global address collection." Anyone can Parse & import into a database, put on a map, or use for geocoding.

Some countries (eg France) are much better covered than others (eg UK).

Morphology is "an exploratory cartographic tool based on the Mapzen platform that eschews common communicative elements like color and symbols. Instead, it seeks to show the world, and all its constituent parts, as a series of carefully chosen lines."

The “True Size” Maps Shows You the Real Size of Every Country (and Will Change Your Mental Picture of the World).

The True Size is "a website that lets you compare the size of any nation or US state to other land masses, by allowing you to move them around to anywhere else on the map."

(From curbed New York) "There are more than 1 million buildings in New York City, and many of those are clustered on the island of Manhattan, spanning architectural styles, hundreds of years, and all manner of types. And now, a nifty new tool, called All the Buildings in Manhattan, takes information about those myriad structures and puts it in one colorful data visualization."

The tool was created by Taylor Baldwin, a software engineer at BuzzFeed (“One thing I learned is that there is an unbelievable amount of data the city makes available.”)

(Note that I had difficulty getting it to work.)

Viz Palette is a colour picker tool to help with selecting a palette for data visualizations, including maps.

An example of one of Peter Watson's Valuation transcripts for Herne Bay in Word document format is attached to this post.  It comprises separate tables, one for each property in the 1910 Valuation, interspersed by some images to help remind about the property. For convenience each document contains a limited number of properties (eg 100), and this example shows just a few.

Each table has the following data for each property:

• ValuationID (when preceded by V this is used as featureid)
• Situation
• Description (which for technical reasons internally we call Designation)
• Gross Value (£ and optionally shillings and pence)
• Occupier
• Former Sale - the date the property was last sold, and hopefully the amount of sale, + additional cost (eg if it was bought as land, then a house built in it)
  • [Note this is displayed/stored after Owner in M4OPS]
• Owner
• Particulars

Broadly speaking the stages of the process are:

• Manipulate the Word Documents to prepare a single table of data
• Convert this table into a list of features for M4OPS
• Add the All Features layer
• Upload and compile
• Georeference the Features

Manipulate the Word Documents to prepare a single table of data:

• In Word, create a new landscape document
• Empty the document, headers and footers of all content
• For each transcript document: Select all, copy and paste onto the end of this landscape document (selecting the default - destination theme)
• To remove header lines: Select all of style Title and delete
• Use Replace All various times:
  • Images: ^g to null
  • Manual line breaks: ^l to space
  • Tabs: ^t to space
  • Semicolons: ; to #semicolon#
  • (Matching Case:) Description or description to #Dscn# or #dscn# (technical problem!)
  • Other special characters/words may need converting:
    • (These are all defined in the CSV2HTMLConvsArray in M4OPS.json)
    • non-breaking space (^s) to #nbsp#
    • less than or greater than to #lt# or #gt#
    • ampersand to #amp#
    • single or double quotation mark to #apos# or #quot#
    • cent, pound, yen, euro to #cent#, #pound#, #yen#, #euro#
    • copyright (©) or registered trademark (®) to #copy# or #reg#
  • [(unnecessary?)Extra paras: ^p^p to ^p – repeat until 0 (any left delete manually)]
  • Single paras: ^p to null
• Select all, copy and paste to another blank landscape document as text only
• Select all and insert table (Convert: paragraphs, 5 columns, Autofit contents)
• Check for changes in pattern, and for each: Undo, adjust, redo
  • [eg in the attached document there is an issue after id 63]
• Delete the blank column (must be narrow if it is empty)
• Convert to text (separate text with tabs)
• Select all and insert table (Convert: tabs, 8 columns, Autofit contents) – if not 8 then undo, resolve issue, and redo
• Save the document

Convert this table into a list of features for M4OPS:

• Insert 4 (blank) columns to the right of the table (these fields will be filled by geometries)
• (At this point, if relevant, transfer any geometry cells from an existing features.csv file)
• Check there are no semicolons in the document
• Convert to text (separate text with semicolons)
• Replace ;space by ;
• Replace ^p by ^pV (thereby preceding each id by the letter V)
  • ensure the first line has a V at the start
  • Replace V^p by null (thereby removing most spurious lines)
  • remove any (last?) line(s) with just a V on them
• If there is no existing Features.csv text file then:
  • Select all, copy and paste into a blank text file
  • Copy the following string and paste it as a new line at the beginning of this file
    • featureid;Situation;Designation;GrossValue;Occupier;FormerSale;Owner;Particulars;Lon(X);Lat(Y);GeomType;GeomCoords
  • Under this header line enter a blank line and a line starting Comment, with the date etc
  • Save the file as Features.csv
• If there is an existing Features.csv text file then:
  • Open this existing Features.csv text file and make sure that:
    • it has been updated with all georeferencing (by downloading from M4OPS)
    • it's fields are all in the same order as the list above
  • In the file you have been processing, select all, copy and paste (append) them to the existing Features.csv text file

See Compiling Features for the next stage.

To process a new version of the SOPS' Studies spreadsheet:

• Go to https://drive.google.com/drive/shared-with-me and right-click/download the latest StudiesYYYYMMDD.xlsx file to the Mapping\Software\M4OPS folder
• Open the downloaded file in Excel, and enable editing
• File/Save As Text (Tab delimited) (*.txt) for the Active Sheet only
• Exit Excel
• Rename the file as StudiesYYYYMMDD.xlsx - Active.tsv (tsv is our name for Tab Separated Values)
• Open this file in an editor and
  • delete all the empty rows at the end of the file
  • check there are no semi-colons in the file (and if there are change them)
  • change all tabs to semi-colons
  • delete all the excess semi-colons at the end of lines (including the last line)
  • Encoding - click on Convert to UTF-8, save and reload to check it is still encoded as UTF-8 (as json needs this)
• FTP the .tsv to /ShowMapsDev/
• On /ShowMapsDev/ rename the old Studies.csv as StudiesOLD.csv (having deleted any previous OLD versions)
• On /ShowMapsDev/ rename the .tsv file to Studies.csv

To process the Studies.csv (because it is new, or there is a new study in the OPS folder structure):

• Open mapping4ops.org/ShowMapsDev/CompileStudies.php?BRs&Test to precompile the Studies.json file
• Make sure that the only OPS listed as "Location not set from Studies.csv" are those we know of (eg BAL, BRI, POR, USG, EDI, LGIWG), or for which we have not yet compiled the OPS.json file
• (If there are problems later you may want to check the resulting Studies.json file)
• On our local M4OPS rename the old Studies.json as StudiesOLD.json (having deleted any previous OLD versions)
• FTP Studies.json from /ShowMapsDev/ to our local M4OPS
• Test M4OPS, and when happy all is OK, FTP Studies.json from our local M4OPS to /ShowMaps/ (ie the production M4OPS system)
• (Note that there is no need to repeat the precompile process on the production system, as all that M4OPS needs is the Studies.json file)
• Test the production M4OPS