PDF to JPG Conversion with Python (for Windows)

I recently had a torrid time trying to research and implement a Python script that could batch convert from PDF to JPG. There are numerous entries online that aim to help (and did so in parts) but I struggled to find one with a concise workflow from start to finish that satisfied my criteria and involved setting up what’s required to implement such. The below could be slated for not being the most ‘Pythonic’ way to get it done but it certainly made my life easier. I was struggling with Wand and ImageMagick as per most posts until I luckily stumbled across an entry on StackOverflow where floqqi, my new hero, answered my prayers. I felt that if I struggled with this that there must be others out there with the same predicament and I hope that the title of this post will help it come to the forefront of searches and aid fellow Python snippet researchers in finding some salvation.
Note: I am using Python 2.7 32-bit on Windows 7 Professional

1. Install ImageMagick
floqqi recommends downloading the latest version, which at the time of writing this is 7.0.4-3. I had already installed an earlier version while trying to get the Wand module to work. My version is 6.9.7-3. If you hover over the links you should be able to see the full link name http://www.imagemagick.org/download/binaries/ImageMagick-6.9.7-3-Q8-x86-dll.exe, or just click that link to download the same version I did.
Run the installer, accept the license agreement, and click Next on the Information window. In the Select Additional Tasks make sure that Install development headers and libraries for C and C++ is selected.

imagemagick

Click Next and then Install.

2. Install GhostScript
I
nstall the 32-bit Ghostscript AGPL Release

3. Set Environment Variables
Create a new System Variable (Advanced System Settings > Environment Variables) called MAGICK_HOME and insert the Image Magick installation path as the value. This will be similar to C:\Program Files (x86)\ImageMagick-6.9.7-Q8

MAGICK_HOME

Click OK and and make sure that the same value (C:\Program Files (x86)\ImageMagick-6.9.7-Q8) is at the start of the Path variable. After this entry in the Path variable insert the entry for GhostScript which will be similar to C:\Program Files (x86)\gs\gs9.20\bin
Note: make sure that the entries are separated by a semi-colon (;)

4. Check if steps 1-3 have been correctly configured
Open the Command Prompt and enter…

convert file1.pdf file2.jpg

where file.pdf and file2.jpg are fully qualified paths for an input PDF and and output JPG (or the current directory contains the file).

convert cmd

If no errors are presented and the JPG has been created you can move on to the next step. Otherwise step into some troubleshooting.

5. Install PythonMagick
I downloaded the Python 2.7 32-bit whl file PythonMagick‑0.9.10‑cp27‑none‑win32.whl and then used pip to install from the command prompt.

pip install C:\Users\glen.bambrick\Downloads\pip install PythonMagick‑0.9.10‑cp27‑none‑win32.whl

Open up a Python IDE and test to see if you can import PythonMagick

import PythonMagick

We now have everything set up and can begin to write a script that will convert multiple (single page) PDFs to JPGs. 

Import the necessary modules.

import os, PythonMagick
from PythonMagick import Image
from datetime import datetime

Ok so datetime isn’t necessary but I like to time my scripts and see if it can be improved upon. Set the start time for the script

start_time = datetime.now()

A couple of global variables, one for the directory that holds the PDFs, and another to hold a hexidecimal value for the background colour ‘white’. After trial and error I noticed that some JPGs were being exported with a black background instead of white and this will be used to force a white background. I found a useful link on StackOverflow to help overcome this.

pdf_dir = r"C:\MyPDFs"
bg_colour = "#ffffff"

We loop through each PDF in the folder

for pdf in [pdf_file for pdf_file in os.listdir(pdf_dir) if pdf_file.endswith(".pdf")]:

Set and read in each PDF. density is the resolution.

    input_pdf = pdf_dir + "\\" + pdf
    img = Image()
    img.density('300')
    img.read(input_pdf)

Get the dimensions of the image.

    size = "%sx%s" % (img.columns(), img.rows())

Build the JPG for output. This part must be the Magick in PythonMagic because for a small portion of it I am mystified. See that last link to StackOverflow for the origin of the code here. The PythonMagick documentation is tough to digest and in various threads read the laments about how poor it is.

    output_img = Image(size, bg_colour)
    output_img.type = img.type
    output_img.composite(img, 0, 0, PythonMagick.CompositeOperator.SrcOverCompositeOp)
    output_img.resize(str(img.rows()))
    output_img.magick('JPG')
    output_img.quality(75)

And lastly we write out our JPG

    output_jpg = input_pdf.replace(".pdf", ".jpg")
    output_img.write(output_jpg)

And see how long it took the script to run.

print datetime.now() - start_time

This places the output JPGs in the same folder as the PDFs. Based on the resolution (density) and quality settings the process can be a bit lengthy. Using the settings above it took 9 minutes to do 20 PDF to JPG Conversions. You will need to figure out the optimum resolution and quality for your purpose. Low res took 46 seconds for all 20.

As always I feel a sense of achievement when I get a Python script to work and hope that this post will spur on some comments to make the above process more efficient. Feel free to post links to any resources, maybe comment to help myself and other readers, or if this helped you in anyway let me know and I’ll pass the thanks on to floqqi and the rest of the crew. This script is the limit of my knowledge with PythonMagick and this is thanks to those that have endeavoured before me and referenced in the links throughout this post. Thanks guys.

Complete script…

import os, PythonMagick
from PythonMagick import Image
from datetime import datetime

start_time = datetime.now()

pdf_dir = r"C:\MyPDFs"
bg_colour = "#ffffff"

for pdf in [pdf_file for pdf_file in os.listdir(pdf_dir) if pdf_file.endswith(".pdf")]:

    input_pdf = pdf_dir + "\\" + pdf
    img = Image()
    img.density('300')
    img.read(input_pdf)

    size = "%sx%s" % (img.columns(), img.rows())

    output_img = Image(size, bg_colour)
    output_img.type = img.type
    output_img.composite(img, 0, 0, PythonMagick.CompositeOperator.SrcOverCompositeOp)
    output_img.resize(str(img.rows()))
    output_img.magick('JPG')
    output_img.quality(75)


    output_jpg = input_pdf.replace(".pdf", ".jpg")
    output_img.write(output_jpg)

print datetime.now() - start_time

Table or Feature Class Attributes to CSV with ArcPy (Python)

Here’s a little function for exporting an attribute table from ArcGIS to a CSV file. The function takes two arguments, these are a file-path to the input feature class or table and a file-path for the output CSV file (see example down further).

First import the necessary modules.

import arcpy, csv

Inside the function we use ArcPy to get a list of the field names.

def tableToCSV(input_tbl, csv_filepath):
    fld_list = arcpy.ListFields(input_tbl)
    fld_names = [fld.name for fld in fld_list]

We then open a CSV file to write the data to.

    with open(csv_filepath, 'wb') as csv_file:
        writer = csv.writer(csv_file)

The first row of the output CSV file contains the header which is the list of field names.

        writer.writerow(fld_names)

We then use the ArcPy SearchCursor to access the attributes in the table for each row and write each row to the output CSV file.

        with arcpy.da.SearchCursor(input_tbl, fld_names) as cursor:
            for row in cursor:
                writer.writerow(row)

And close the CSV file.

    csv_file.close()

Full script example…

import arcpy, csv

def tableToCSV(input_tbl, csv_filepath):
    fld_list = arcpy.ListFields(input_tbl)
    fld_names = [fld.name for fld in fld_list]
    with open(csv_filepath, 'wb') as csv_file:
        writer = csv.writer(csv_file)
        writer.writerow(fld_names)
        with arcpy.da.SearchCursor(input_tbl, fld_names) as cursor:
            for row in cursor:
                writer.writerow(row)
        print csv_filepath + " CREATED"
    csv_file.close()

fc = r"C:\Users\******\Documents\ArcGIS\Default.gdb\my_fc"
out_csv = r"C:\Users\******\Documents\output_file.csv"

tableToCSV(fc, out_csv)

Feel free to ask questions, comment, or help build upon this example.

My First Encounter with arcpy.da.UpdateCursor

I have been using arcpy intermittently over the past year and a half mainly for automating and chaining batch processing to save myself countless hours of repetition. This week, however, I had to implement a facet of arcpy that I had not yet had the opportunity to utilise – the data access module.

Data Cursor

The Scenario
A file geodatabase with 75 feature classes each containing hundreds to thousands of features. These feature classes were the product of a CAD (Bentley Microstation) to GIS conversions via FME with data coming from 50+ CAD files. As a result of the conversion each feature class could contain features with various attributes from one or multiple CAD files but each feature class consisted of the same schema which was helpful.

cad2gis

The main issue was that the version number for a chunk of the CAD files had not been corrected. Two things needed to be fixed: i) the ‘REV_NUM’ attribute for all feature classes needed to be ‘Ver2’, there would be a mix of ‘Ver1’ and ‘Ver2’,  and ii) in the ‘MODEL_SUMMARY’ if ‘Ver1’ was found anywhere in the text it needed to be replaced with ‘Ver2’. There was one other issue and this stemmed from creating new features and not attributing them, this would have left a ‘NULL’ value in the ‘MODEL’ field (and the other fields). All features had to have standardised attributes. The script would not fix these but merely highlight the feature classes.

OK so a quick recap…
1. Set the ‘REV_NUM’ for every feature to ‘Ver2’
2. Find and replace ‘Ver1’ with ‘Ver2’ in the text string of ‘MODEL_SUMMARY’ for all features.
3. Find all feature classes that have ‘NULL’ in the ‘MODEL’ field.

The Script
Let’s take a look at the thirteen lines of code required to complete the mission.

import arcpy

arcpy.env.workspace = r"C:\Users\*****\Documents\CleanedUp\Feature_Classes.gdb"
fc_list = arcpy.ListFeatureClasses()
fields = ["MODEL", "MODEL_SUMMARY", "REV_NUM"]

for fc in fc_list:
 with arcpy.da.UpdateCursor(fc, fields) as cursor:
  for row in cursor:
   if row[0] == None or row[0] == "":
    print fc + ": Null value found for MODEL"
    break
   if row[1] != None:
    row[1] = row[1].replace("Ver1", "Ver2")
   row[2] = "Ver2"
   cursor.updateRow(row)

The Breakdown
Import the arcpy library (you need ArcGIS installed and a valid license to use)

import arcpy

Set the workspace path to the relevant file geodatabase

arcpy.env.workspace = r"C:\Users\*****\Documents\CleanedUp\Feature_Classes.gdb"

Create a list of all the feature classes within the file geodatabase.

fc_list = arcpy.ListFeatureClasses()

We know the names of the fields we wish to access so we add these to a list.

fields = ["MODEL", "MODEL_SUMMARY", "REV_NUM"]

For each feature class in the geodatabase we want to access the attributes of each feature for the relevant fields.

for fc in fc_list:
 with arcpy.da.UpdateCursor(fc, fields) as cursor:
  for row in cursor:

If the ‘MODEL’ attribute has a None (NULL) or empty string value then print the feature class name to the screen. Once one is found we can break out and move onto the next feature class.

   if row[0] == None or row[0] == "":
    print fc + ": Null value found for MODEL"
    break

We know have a list of feature classes that we can fix the attributes manually.

Next we find any instance of ‘Ver1’ in ‘MODEL_SUMMARY’ text strings and replace it with ‘Ver2’….

   if row[1] != None:
    row[1] = row[1].replace("Ver1", "Ver2")

…and update all ‘REV_NUM’ attributes to ‘Ver2’ regardless of what is already attributed. This is like using the Field Calculator to update.

   row[2] = "Ver2"

Perform and commit the above updates for each feature.

   cursor.updateRow(row)

Very handy to update the data you need and this script can certainly be extended to handle more complex operations using the arcpy.da.UpdateCursor module.

Check out the documentation for arcpy.da.UpdateCursor

Extract PDF Pages and Rename Based on Text in Each Page (Python)

I was recently tasked with traversing through a directory and subsequent sub-directories to find PDFs and split any multi-page files into single-page files. The end goal was to name each extracted page, that was now an individual PDF, with a document number present on each page. There was possibly over 100 PDF files in the directory and each PDF could have one to more than ten pages. At the extreme I could have been looking at around one-thousand pages to extract and rename – a task that would have been very time consuming and mind numbing to do manually.

The PDFs contained map books produced using data driven pages in ArcGIS, it was conceivable that I could also re-open the original MXDs and re-export the map book as individual pages and naming appropriately based on the document name in the attribute table. Since I was not the creator of any of these PDFs and they all came from different teams, hunting down the correct MXDs and exporting would be cumbersome and also very time consuming. There had to be a more interesting and time efficient way…

…A quick research via Google on some Python modules and I had what I needed to complete my task in a more automated and time efficient manner. I needed three modules;
(1) os – for traversing through the directories and files and for renaming the files
(2) PyPDF2 – to read/write PDF files and also to extract text from pages
(3) re – the regular expression module to find the text needed to rename the file.
The next step was write down some pseudocode to map out what needed to be achieved and then to get coding…

Let’s begin by importing the modules at the top of the script.

import os, PyPDF2, re

Define a function to extract the pages. This function will take two parameters; the path to the root directory and the path to a folder to extract the pages to. The ‘extract_to_folder’ needs to be on the same level or above the root directory. Use your operating system to create the folder named ‘extracted’ and also create a second folder called ‘renamed’.

def split_pdf_pages(root_directory, extract_to_folder):

Next we use the os module to search from the root directory down to find any PDF files and store the full filepath as a variable, one at a time.

for root, dirs, files in os.walk(root_directory):
 for filename in files:
  basename, extension = os.path.splitext(filename)
   if extension == ".pdf":
    fullpath = root + "\\" + basename + extension

We then open that PDF in read mode.

    opened_pdf = PyPDF2.PdfFileReader(open(fullpath,"rb"))

For each page in the PDF the page is extracted and saved as a new PDF file in the ‘extracted’ folder. The below snippet was sourced from stackoverflow.

    for i in range(opened_pdf.numPages):
     output = PyPDF2.PdfFileWriter()
     output.addPage(opened_pdf.getPage(i))
     with open(extract_to_folder + "\\" + basename + "-%s.pdf" % i, "wb") as output_pdf:
      output.write(output_pdf)

That completes our function to strip out individual pages from PDF files in a root directory and down through all corresponding sub-directories. This function might be all you need as you can rename the extracted pages as you save each file. The next task for me, however, was to rename the PDFs based on text contained in each individual file.

Define a function called ‘rename_pdfs’ that takes two arguments; the path to the folder where the extracted pages reside and the renamed folder. Loop through each PDF and create a filepath to each one.

def rename_pdfs(extraced_pdf_folder, rename_folder):
 for root, dirs, files in os.walk(extraced_pdf_folder):
  for filename in files:
   basename, extension = os.path.splitext(filename)
   if extension == ".pdf":
    fullpath = root + "\\" + basename + extension

Open each PDF in read mode…

    pdf_file_obj = open(fullpath, "rb")
    pdf_reader = PyPDF2.PdfFileReader(pdf_file_obj)

…and create a page object.

    page_obj = pdf_reader.getPage(0)

Now we extract the text from the page.

    pdf_text = page_obj.extractText()

My task was made quite easy because each page had a unique document number with a certain amount of characters prefixed the exact same for each. This meant that I could use regular expression, the re module, to find the prefix and then obtain the rest of the document number.

The code below finds the document number prefix in the text extracted from the page and appends the next 14 characters to the prefix to give the full document number.

    for index in re.finditer("THE-DOC-PREFIX-", pdf_text):
     doc_ext = pdf_text[index.end():index.end() + 14]
     doc_num = "THE-DOC-PREFIX-" + doc_ext
     pdf_file_obj.close()

The last thing to do is to use the document number to rename the PDF

    os.rename(fullpath, rename_folder + "\\" + doc_num + ".pdf")

That completes the two functions required to complete the task.

Set up the variables required for the function parameters…

root_dir = r"C:\Users\******\Documents\original"
extract_to = r"C:\Users\******\Documents\extracted"
rename_to = r"C:\Users\******\Documents\renamed"

…and then call each function.

split_pdf_pages(root_dir, extract_to)
rename_pdfs(extract_to,rename_to)

Run the script. The original files will remain and the renamed extracted pages will be in the renamed folder. Any PDF page that failed to be renamed will still be in the extracted folder and you can rename these manually. This failure to rename every PDF is because of the make-up of the PDF i.e. the way it was exported from a piece of software or how it was created. In a test run, out of 206 pages, 10 pages failed to be renamed. When I opened the pages the select tool was unable to highlight text and everything was embedded as an image, hence why the script couldn’t read any text to rename the document.

I hope someone out there will find this useful. I am always happy that my code works but appreciate if you have any constructive comments or hints and tips to make the code more efficient.

Here’s the full script…

# import the neccessary modules
import os, PyPDF2, re

# function to extract the individual pages from each pdf found
def split_pdf_pages(root_directory, extract_to_folder):
 # traverse down through the root directory to sub-directories
 for root, dirs, files in os.walk(root_directory):
  for filename in files:
   basename, extension = os.path.splitext(filename)
   # if a file is a pdf
   if extension == ".pdf":
    # create a reference to the full filename path
    fullpath = root + "\\" + basename + extension

    # open the pdf in read mode
    opened_pdf = PyPDF2.PdfFileReader(open(fullpath,"rb"))

    # for each page in the pdf
    for i in range(opened_pdf.numPages):
    # write the page to a new pdf
     output = PyPDF2.PdfFileWriter()
     output.addPage(opened_pdf.getPage(i))
     with open(extract_to_folder + "\\" + basename + "-%s.pdf" % i, "wb") as output_pdf:
      output.write(output_pdf)

# function for renaming the single page pdfs based on text in the pdf
def rename_pdfs(extraced_pdf_folder, rename_folder):
 # traverse down through the root directory to sub-directories
 for root, dirs, files in os.walk(extraced_pdf_folder):
  for filename in files:
   basename, extension = os.path.splitext(filename)
   # if a file is a pdf
   if extension == ".pdf":
    # create a reference to the full filename path
    fullpath = root + "\\" + basename + extension

    # open the individual pdf
    pdf_file_obj = open(fullpath, "rb")
    pdf_reader = PyPDF2.PdfFileReader(pdf_file_obj)

    # access the individual page
    page_obj = pdf_reader.getPage(0)
    # extract the the text
    pdf_text = page_obj.extractText()

    # use regex to find information
    for index in re.finditer("THE-DOC-PREFIX-", pdf_text):
     doc_ext = pdf_text[index.end():index.end() + 14]
     doc_num = "THE-DOC-PREFIX-" + doc_ext
     pdf_file_obj.close()
     # rename the pdf based on the information in the pdf
     os.rename(fullpath, rename_folder + "\\" + doc_num + ".pdf")

# parameter variables
root_dir = r"C:\Users\******\Documents\rename_pdf"
extract_to = r"C:\Users\******\Documents\extracted"
rename_to = r"C:\Users\******\Documents\renamed"

# use the two functions
split_pdf_pages(root_dir, extract_to)
rename_pdfs(extract_to,rename_to)

Resources:
PyPDF2
Automate the Boring Stuff with Python

GIS Career Advice Suggestions; my experiences & opinions.

Warning: this is a lengthy read that may borderline a rant, you have been warned!

dos_donts

Career advice is everywhere these days from generic all-round advice via do’s and do not’s encompassing any career to must do’s and must not’s for a more focused delineated career path. There are a few great articles out there but most, however, seem to lack the personal experience of the author, and while I wouldn’t necessarily disagree with some or any of the advice, I feel that it often does not reflect reality, well my reality anyway. With that in mind I decided to put in words some ‘suggestions’ based on my experiences to date. Notice that I have labelled these ‘suggestions’ because I don’t think it’s possible to give someone career advice without knowing their individual personal situation. If the generic career advice was golden we would be all going into interviews and employment like robots acting the same as each other. The suggestions below are not numbered and I’m not initiating that you follow them in any particular systematic way or at all in fact if you don’t feel they’re for you. You might find some of these to be enlightening with a fresh approach or on the other-hand you might completely disagree with some that go completely against the grain, you might even think that one or two are complete tripe and a terrible suggestion, but at the end of the day these are my experiences, my opinions, and what has worked for me and what I have learned along the way. So here we go…

Be Educated

be_educated

While experience stands out on a resume it is highly unlikely that your experience would be on your resume without an educational background. If you want a potential employer to take you serious for an entry level or graduate role then having the necessary education is a huge step in the right direction. Many university courses now offer GIS modules as part of a wider discipline such as environmental sciences or engineering for examples. While these modules are a good way to get acquainted with GIS software they will more than likely lack in many aspects. In particular the theory behind GIS and GIScience, the theory behind how the analysis methods and techniques work through their underlying algorithms and these are often fundamental to problem solving in the workplace. One very simple example that I have encountered numerous times is ‘why are my points appearing in the middle of the ocean?’ Because most of use say ‘lat-long’ we assume that latitude is the x-axis and longitude the y when it is the opposite. At the end of the day anyone can make a map, it’s not really the hard part of GIS, but making a good map after performing several geoprocessing tasks and performing geostatistical analysis on data and putting forward competent results is! If you are serious about a career in GIS I suggest that you attend a course that purely focuses on GIS, how and why the software does what it does and how you can use the software to solve questions, problems, and aid in better decision making. I have completed two full-time GIS focused postgraduate courses, the GIS and GISci landscape is vast and ever growing and no course can completely cover every aspect, but they are the foundation for future endeavours both educationally and professionally.

Don’t Stop Learning

dont_stop_learning

I made this mistake and I know a lot of others out there that this will ring true with. I finished a postgraduate in GIS at the end of 2007 and set about applying my education in the workplace to earn some money. At the time I was delighted to just be finished with studying, mainly because I hated it. I didn’t put too much effort into obtaining a degree and a postgraduate, I was just going through the motions. This was because I really didn’t have a clue about what I wanted to be when I grow up and GIS kind of just happened for me, it wasn’t something I was aiming for, in fact I hadn’t even heard of it until I applied for the postgraduate course. My first job was a summer role as work experience as part of the postgraduate, I was merely digitizing and I wasn’t even using any of the major software, but it paid well for someone that hadn’t even graduated yet. I worked in Canada for a year and then in Australia for 4 years. GIS was great, it got me a job wherever I went, or was it my own personality that got me these jobs and the way I applied for them, perhaps a combination of the above? (see next section), but in those 5 years I had lost track of the GIS landscape. I remember looking at GIS roles just to see what’s out there and I began to panic. Every GIS Technician role now wanted some or all of; Python programming and ArcPy, JavaScript and various libraries, spatial databases and spatial SQL, graphic design, and many more facets that were well beyond my capabilities. I had none of these in my arsenal. Sure, I had experience behind me but experience in what exactly? I had a title of GIS Manager but really I was a glorified GIS Technician that could make a great map and perform the basic analysis as needed with data management skills in the land of the archaic folder based system. Although I picked up many other skills along the way would the 5 years’ experience I had stack up against a recent graduate who ticked more boxes than me? I severely doubted it. They could do more and get paid less for it. GIS had become my bread and butter and I wanted to excel so I made the decision to return to education to study IT with a focus on programming and databases, followed by a Masters in Geocomputation to merge the IT and GIS nicely. Now I was back up to speed with the industry and aim to keep it that way. Although it may suit some of you to do so, I suggest that you never have to follow my route and return to full-time study at the age of 30, simply put some time each week towards constant skills development and keep an eye on the job market so you can keep the development focused. With an ever growing library of printed books, eBooks, online courses both free and at a cost, blogs and video tutorial, there really is no excuse.

Don’t Hide Behind a Keyboard (if possible)!

keyboard

I have always found it relatively easy to find employment wherever I have been, Ireland, Canada, Australia and more recently the UK. I get sick and tired of hearing up and coming GIS graduates, and even some more seasoned GIS professionals, complaining about being unemployed and that they are applying for everything out there but getting nowhere. When I question their techniques it’s the same story for the majority, they are only focusing on online job postings or connecting with recruiters on LinkedIn. This has to be the laziest type of job hunting possible and you are pitting yourself against a lot of competition each time you hit that submit button. All you are with the online job posting approach is an electronic document, you might not even make it to being a printed document. You have no personality and your Calibri 11pt font on your PDF resume is not going to make you stand out from the Times New Roman 12pt font from the other applicant. I get it, this is the way the world has gone, but you can use this to swing in your favour. Twice in my career I have found employment by using the following methods. First of all, do not limit yourself to online job postings. Not all the GIS jobs out there are advertised and some places may be just starting to consider expanding or even using GIS for the first time. Many places already know who they want to hire but have to go through the advertisement process anyway. Research all the companies in your area that you know will either have or might have a GIS department, including those that are advertising, draft your cover letter and resume, print them off, place them in a nice A4 envelope for each company, research the company and who is more than likely the person to talk to regarding an advertised role or any future roles, physically cut the umbilical cord between you and the keyboard and prepare yourself for some human interaction outside the dungeon of your four walls. Take yourself to their office and ask to speak to someone, that person who you found on LinkedIn or the company website who you feel is the best person to talk to, or even just ask for the person responsible for the GIS team. You are playing the percentage game here, more than likely you will struggle to get past reception but be patient and persistent without being forceful, if you get to talk to one out of ten well that’s an achievement and you have gone further than the button bashing has got you. The more you do this the easier it get’s to get through to the person you are after and your percentage of hits and connecting in real life rises. If you get the talk to someone you are already being interviewed informally so be prepared. And you are now a human in the process and not digital document. I happened to stumble upon a company who were contemplating expanding, I was interviewed on the spot on a Thursday and started working on the Monday.

I understand that this approach only works if there are companies located locally or an acceptable commute away. If you are within an hour away from a plethora of potential employers and you’re hiding in room, in your pyjamas, applying to them through the personality of your keyboard, well I suggest that you get your head examined.

GIS Related Certifications Are Not That Important

gis_certs

Controversial I know, and I can hear a crowd of you shouting at me, but I’m not saying don’t get them, I just feel that they carry very little weight in a GIS career. If you have a solid GIS educational background why do you need to have certifications on top of that? Surely in those fine educational establishments you were exposed to a variety of GIS software to perform basic to more advanced tasks? and if not then perhaps certifications are actually a good choice for you. Certifications generally mean that you know your way around a certain piece of software and that you meet some competency level with the implementation of that software, but most GIS software can perform the exact same or similar tasks, especially the basics, so converting from using one to the other is not major and there is a fantastic new search engine, I’m not sure if you’ve heard of it, called Google, to aid with the transition if you need some help. In the past and even more recently I have considered gaining some certifications, ESRI certs and Oracle SQL exams for examples and these have dropped significantly down the pecking order in my list of priorities. I think I was just using it as motivation to keep learning and even worse I think I was just buying into the whole game that I actually needed them. How many of you have learned something, not applied it for years and now it really shouldn’t appear on your skills set? I attended courses in AutoCAD for example about 4 years ago. At the time I was quite proficient for the 3 to 6 months I was using it, but recently I opened AutoCAD and I was lost. I’d be fooling a potential employer if I left AutoCAD on my resume. My point is that certifications can go stale. You might get ESRI certs and then land a role that uses MapInfo primarily and not return to Arc for years. Many companies these days will put you through a couple of tests during or after an interview to show that you can back up what is on your resume. These might be using ArcGIS to perform some tasks and generate some output or manipulate some data in Excel and import into a GIS. Whether or not you have a certification if you fail the test you are not getting considered for the job. Did having the certification on your resume get you the interview in the first place? I don’t believe that having a GIS related certification on your resume stands out more than simply listing that you have that skill backed up by your educational background (or work experiences if any), and a portfolio either printed or as a website. I agree that studying towards certifications is a great way to keep learning in a focused manner with something to work towards but they really aren’t that impressive (in my opinion). I may return one day on a certification quest but only if it enhances my knowledge of something in particular. I suggest that if you feel that certifications make you stand out from the crowd then go for it, it may give you more confidence, but don’t stress that you don’t have any, I don’t think an employer is going to think you’re lying about your knowledge of ArcGIS just because you don’t have a certification listed on your resume to somewhat prove it when you have a solid GIS degree (or postgraduate), and/or experience behind you.

Don’t Work For Free! (unless volunteering)

get_paid

Tom Cruise (Jerry Maguire – if you didn’t quite get that one)

Whoa! What!? But (unpaid) internships are an integral part of the career path process, they give you valuable experience to help you develop skills and expose you to industry standards. Absolute nonsense (for the majority in GIS), you are being taken advantage of. You cannot put a price on your time left on this planet and yet here you are working for free, more than likely doing the boring mundane crap that no one else wants to do, putting in ridiculous hours beyond the expectations of a full-time employee because you want to look good, lining the pockets of someone else, and most likely with no chance of a permanent role after that because they’ll replace you with another intern, there’s plenty of them in the pipeline. I’m sure there are some great internship stories out there but they will be few and far between. It’s rare that anyone starts their career in GIS not doing the boring mundane crap like digitization and data entry. You can train a monkey to do these things, but you’re an educated monkey and deserve a little more respect. Now I’m not saying don’t pursue a role where you have to do these things, we’ve all done them and even today I have to do some digitization and data entry, it comes part and parcel with the territory, but at least get paid for it. If you are going down the internship route at least attempt to negotiate a stipend so you don’t feel like your’e selling your soul. You’ve just spent a lot of money educating yourself, it’s payback time. The downside to my argument is that it might actually look good on your resume that you interned somewhere, but this evaluation depends on where you are located in the world (the society you live in), the company, and the actual experience gained from the internship. If you have no formal education in GIS an internship paid or not is probably a good choice if making the transition. If you are going to work for free volunteer your services to a charity or non-profit organisation where you have more prospects to be involved in both the mundane to the more exciting tasks and potentially at a higher level. Volunteering will also look just as good on your resume and your soul and self-respect will be intact. Some of you are probably thinking ‘but if I don’t take an internship to get on the experience ladder I’ll struggle to get elsewhere’, don’t sell yourself short, an internship could actually be a waste of time if you are learning absolutely nothing new, you can focus your energy elsewhere in the meantime. While not working make sure that you put plenty of hours into professional development each day, similar to the ‘don’t stop learning section’, there are so many free and cheap avenues to take to continue your GIS knowledge growth. I suggest getting your hands on the ESRI Guide to GIS Analysis Volumes 1 to 3, these will open your eyes beyond simple mapping and the material will be great to reference in an interview to show your interest in GIS and that you really know what you’re talking about.

Don’t Work for the Sake of Working

work

This suggestion only really relates to those that are in a position to do so. If your GIS career goal is making simple maps for the rest of your days, not really having to extend your brain power too far, then this does not apply to you, you’re more than likely already living your dream. Before Christmas I was working for a few months and was really only doing so for the small bit of income. I was learning nothing from the role and the monotony from it all was affecting my mood. I decided over Christmas that I wasn’t going to apply for a role or accept any roles that I really had very little interest in. I was going to devote my time to up-skilling and putting some blog posts together to attract interest. This has been my best choice to date. I made a list of the books, online courses, and other learning material that I wanted to go through and put aside around 6 hours a day, almost as if it was my full time job. I could do this because I was in a position to do so. If I had a mortgage, kids, or other responsibilities I would not of had the comfort of going down this route without some support. I have talked to so many people that are ‘just happy to be working’ even though they are miserable and have no progression prospects in those GIS jobs. Does 5 years experience doing the most simplistic GIS tasks look good on your resume? In my opinion, no! It looks like you are happy to stay at that level with little to no ambition. If you’re the type of person that can leave a depressing job in the evening and go enhance your skills at night well then I tip my hat to you. But for the majority of us the energy zapping 9 -5 – I want to shoot myself in the face – job means we’re straight home to the couch to complain about how shit our job is to whoever will listen. I once heard that ‘a job of no interest does more damage to your mental health than being unemployed’, and I support this statement. I went on unemployment benefits, learned a ridiculous amount of material in two months that I was genuinely interested in, and I am now applying some of these up-skilled learnings to a new role getting paid more than decent. Maybe I just got lucky or maybe it was motivation. You will be at different levels at different stages in your career, assess what you want and make the necessary changes to achieve your goals, even if a means to this is through unemployment.

Teach Others When You Can

knowledge

Be confident in your abilities as soon as you graduate. When you do eventually land your first role it is highly likely that you will know some things that others in the company do not because your course was tailored differently with modern techniques or many of the GIS users in there are basic users. Always take the time out to help a fellow colleague with troubleshooting or suggesting how you would go about doing things if they are stuck and keep that mentality throughout your career. I once worked for a company where people kept their talents to themselves, they wanted to be one step ahead of each other in a race for promotion, the only problem was that there was no room for promotion and the whole situation made for a sour work environment. One guy even spouted on about his ‘dominance in Arc’ as if he ruled over the rest of us. It’s frustrating sitting around being stuck when someone can help you but won’t. Don’t get caught up in getting ahead. Helping others opens the door for others to help and teach you when you need it and creates a pleasant knowledge driven environment.

Never Burn Bridges

bridges

I have walked out of a job in the past, the company was sold to me as a tight knit family community that helped each other out. This couldn’t have been further from the truth and it wore me down to the point where I just had enough and walked. I’m proud I stuck up for myself (I won’t go into details) but if something similar was to happen now I would handle it in a different way and be more professional about it all. My walkout never had any adverse affects on my GIS career but it’s not something I recommend doing as on a couple of occasions I had to explain why I lasted 3 months in a company and the nature of why I left and you really have to be careful how you choose your words in those situations, especially in an interview environment. On the other-hand I have worked for the same employer three times and it’s a great feeling when someone wants you back and are confident in your abilities. I suggest that you attempt to keep in touch with past employers and fellow colleagues. LinkedIn has made this quite easy and it is something you should be using to your advantage to keep connected with your professional network.

My Thoughts on Interviews

People are people, they are a fellow human beings who one day were in the position you are now. They should not be feared and you should not see an interview as a daunting experience. Society has painted the interview experience the way its is and it makes people nervous. Not getting the job is not the end of the world and you get better at interviews the more frequent you do them. I always thought I interviewed well. That was until I went for an IT role about 6 months ago and felt completely unnerved. This was simply my mind telling me this isn’t really what I wanted, my answers to questions seemed to always relate back to my GIS career and I knew quite quick that rejection was looming and I was quite happy with that. Shortly after I had an interview for a GIS role and I was back in flying form. I knew what I was talking about, what I wanted, what I could do, what I could bring to the team, my limitations, my aspirations, my goals, my salary expectations (which eventually put me out of the race because our evaluations were way off). People spend hours and even days preparing for an interview, building up the stress levels before they have even sat down with the prospective employer. You do not need to research the ins and outs of a company, you just have a look at their general activities, what industry or industries they operate in. GIS is GIS, pretty much the same or similar wherever you work or whatever industry you work in. I have worked in agriculture, oil and gas, mineral exploration, and engineering and had no clue about these industries before I started. Your GIS skills are often what’s wanted, not the industry that is looking for them.

Have some interview questions ready to ask. A couple of my favourites are; With the GIS landscape ever changing, how do you make sure that staff are kept abreast of best practices and have access to learning new methods and techniques as the industry evolves?, this indicates that you want somewhere where you can evolve and learn as you progress; You have painted this role in a glorified manner, but tell me one negative thing about working at this office location?, this will show confidence and that you are not afraid to ask difficult questions.

Be confident even if you have to fake it. It’s a horrible feeling leaving an interview knowing that nerves have cost you. Sit up straight, make eye contact, speak slow and clear as this will make you seem more relaxed and eventually you will be. The interviewer is not there to catch you out, they have your resume and can see what you have to offer, you wouldn’t have made it this far if they didn’t consider you a potential candidate for the position. Smile, smile, smile, always be smiling, you will stand out a mile from other candidates that may have interviewed better, a constant smile will work wonders.

And remember, you’re interviewing them too, you need to find out if you really want to work there so ask the right questions to get the information you need.

Make Yourself Indispensable

When you are employed find something niche that adds to your stock. Something that if you were to disappear tomorrow they would be completely lost without you. It’s actually easier to do than you think but involves going way beyond simple GIS techniques and requires a lot of motivation. You may need to get to grips with specialist software such as FME, programming and scripting such as Python and JavaScript, learning geostatistical methods and chaining several intricate processing tasks with automation for examples but certainly not limited to these. I suggest that you find and work towards finding something that makes them fear losing you.

Suggestions and Opinions Rant Over

theend

If you are looking for GIS career advice make sure that you talk to GIS professionals that have been through what you are beginning to embark on and don’t rely on the generic career advice posts that paint everything in black and white. You need to tailor your do’s and do not’s to suit you as an individual and the path that you want to take. The more you talk to the more likely you will come across someone with similar experiences to you and can point you in the right direction. Avoid taking career advice from academics who like to talk about working in the ‘real world’ but have never done so.

You can use or dismiss some the suggestions above because at the end of the day they are only suggestions and as said at the beginning these are my personal experiences and my opinions.

Book Review: The ESRI Guide to GIS Analysis Vol. 2: Spatial Measurements & Statistics

Title: The ESRI Guide to GIS Analysis Vol. 2: Spatial Measurements & Statistics
Author: Andy Mitchell
Publisher: ESRI Press
Year: 2005
Aimed at: GIS/Analysts/Map Designers – intermediate
Purchased from: www.wordery.com

ESRI GA V2

This textbook acts as companion text for GIS Tutorial 2: Spatial Analysis Workbook (for ArcGIS 10.3.x) where you can match up the chapters in each book. Although not a necessity, I would recommend using both texts in tandem to apply the theory and methods discussed with practical tutorials and walkthroughs using ArcGIS. This is the second book of the series and follows on from The ESRI Guide to GIS Analysis Volume 1: Geographic Patterns & Relationships.

The first chapter is, inevitably, an introduction to spatial measurements and statistics. You perform analysis to answer questions and to answer these questions you not only need data but you also need to understand the data. Are you using nominal, ordinal, interval or ratio values, or a combination of these? The type of value(s) will shape the analysis techniques and methods used to calculate the statistics. You will need to interpret the statistics, test their significance and question the results. These elements are briefly visited with the premise of getting more in-depth as the book progresses. The chapter ends with a section on ‘Understanding data distributions’ which is essentially a brief introduction to data exploratory techniques such as describing frequency distributions, spatial distributions, and the presence of outliers and how they can affect analysis.

Chapter 2 discusses measuring geographic distributions with the bulk of the chapter focused on finding the center (mean, meridian, central feature), and measuring compactness (standard distance), orientation and direction of distributions (spatial trends). These are discussed for points, lines, and areal features and also using weighted factors based on attributes. These are useful for adding statistical confidence to patterns derived from a map. Formulas and equations begin to surface and although not necessary to learn them off by heart, because the GIS does all the heavy lifting for you, it gives insight into what goes on under the hood, and knowing the underlying theory and formulas can often aid in troubleshooting and producing accurate analysis. The last section of this chapter is fundamental to the rest of the text, testing statistical significance. This allows you to measure a confidence level for your analysis using the null hypothesis, p-value, and z-score. This can be a difficult topic to comprehend and may require further reading.

The third chapter, a lengthy one, is based around using statistical analysis to identify patterns, to enhance and backup the visual analysis of the map with confidence or to find patterns not may not have been immediately obvious. The human eye will often see patterns that do not really exist, so alternatively, statistical analysis might indicate what you thought was a strong pattern was actually quite weak. The statistical analysis methods are beginning to heat up and here we are introduced to; the Kolmorogov-Smirnov test and Chi Square test for quadrat analysis in identifying patterns in areas of equal size; the nearest neighbour index for calculating the average distance between features and identifying clustering or dispersion; and the K-function as an alternative to the nearest neighbour index, each used to measure the pattern of feature locations. These are followed by measuring the spatial pattern of feature values using; the join count statistic for areas with categories; Geary’s c and Moran’s I for measuring the similarity of nearby features, and the General-G statistic for measuring the concentration of high and low values for features having continuous values. The formulas for each are presented along with testing the significance of and interpreting the results. The final section of this chapter discusses defining spatial neighbourhoods and weights when analysing patterns. There are a few things to consider such as local or regional influences, thresholds of influence, interaction between adjacent features, and the rate of regional decline of influence.

Chapter 4 is titled ‘Identifying Clusters’ with a main focus on hotspot analysis. First, we are introduced to nearest neighbour hierarchical clustering which is heavily used in crime analysis. While Chapter 3 discussed global methods for identifying patterns and returns a single statistic, this chapter focuses on local statistics to show where these patterns exist within the global setting. Geary’s c and Moran’s I both have local versions and their definition, implementation, and factors influencing the results are discussed and critiqued along with Art Getis’ and Keith Ord’s Gi* method for identifying hot and cold spots.While the methods in Chapter 3 enforced that there are patterns in the data (or not), the methods in Chapter 4 highlight where these clustered patterns are. The last section of Chapter 4 discusses using statistics with geographic data; how the very nature of geographic data affects your analysis, how geographic data is represented in a GIS affects your data analysis, the influence of the study area boundary, and GIS data and errors.

“To the extent you’re confident in the quality of your GIS data, you can be confident in the quality of your analysis results.”

The last chapter ventures away from identifying patterns and clusters and focuses on analysing geographic relationships and using statistics to analyse such. Geographic relationships and processes are used to predict where something is likely to occur and examining why things occur where they do. Chapter 5 looks at statistical methods for identifying geographical relationships with a Pearson’s correlation coefficient and Spearman’s correlation coefficient discussed and assessed. Linear regression (ordinary least squares), and geographically weighted regression are presented as methods for analysing geographic processes. These methods warrant a full text in their own right and there is a list of further reading available at the end of the chapter.

Overall Verdict: I feel that I will be referring back to this text a lot. Having recently completed a MSc in Geocomputation I wish that this had crossed my path during the course of my studies and I would highly recommend this book to anyone venturing into spatial analysis where statistics can aid and back up the analysis. Although they are littered throughout the chapters, you really do not need to get bogged down with the formulas behind the statistical analysis techniques, the most important points is that you understand what the methods are performing, their limitations, and how to assess the results and this book really is a fantastic reference for doing just that. Knowing the theory is a huge step to being able to apply the analysis techniques confidently and derive accurate reporting of your data.

Book Review: Learning ArcGIS Geodatabases [eBook]

Title: Learning ArcGIS Geodatabases
Author: Hussein Nasser
Publisher: Packt Publishing
Year: 2014
Aimed at: ArcGIS – beginner to advanced
Purchased from: www.packtpub.com

Learning ArcGIS Geodatabases

After using MapInfo for four years my familiarity with ArcGIS severely declined. The last time I utilised ArcGIS in employment shapefiles were predominantly used but I knew geodatabases were the way forward. If they were going to play a big part in future employment it made sense to get more intimate with them and learn their inner secrets. This compact eBook seemed like a good place to start…

The first chapter is short and sweet and delivered at a beginner’s level with nice point to point walkthroughs and screenshots to make sure you are following correctly. You are briefed on how to design, author, and edit a geodatabase. The design process involves designing the schema and specifying the field names, data types, and the geometry types for the feature class you wish to create. This logical design is then implemented as a physical schema within the file geodatabase. Finally, we add data to the geodatabase through the use of editing tools in ArcGIS and assign attribute data for each feature created. Very simple stuff so far that provides a foundation for getting set-up for the rest of the book.

The second chapter is a lot bulkier and builds upon the first. The initial task in Chapter 2 is to add new attributes to the feature classes followed by altering field properties to suit requirements. You are introduced to domains, designed to help you reduce errors while creating features and preserve data integrity, and subtypes. We are shown how to create a relationship class so we can link one feature in a spatial dataset to multiple records in a non-spatial table stored in the geodatabase as an object table. The next venture in this chapter takes a quick look at converting labels to an annotation class before ending with importing other datasets such as shapefiles, CAD files, and coverage classes and integrating them into the geodatabase as a single point of spatial reference for a project.

Chapter 3 looks at improving the rough and ready design of the geodatabase through entity-relationship modelling, which is a logical diagram of the geodatabase that shows relationships in the data. It is used to reduce the cost of future maintenance. Most of the steps from the first two chapters are revisited as we are taken through creating a geodatabase based on the new entity relationship model. The new model reduces the number of feature classes and improves efficiency through domains, subtypes and relationship classes. Besides a new train of thought on modelling a geodatabase for simplicity the only new technical feature presented in the chapter is enabling attachments in the feature class. It is important to test the design of the geodatabases through ArcGIS, testing includes adding a feature, making use of the domains and subtypes, and test the attachment capabilities to make sure that your set-up works as it should.

Chapter 4 begins with the premise of optimizing geodatabases through tuning tools. Three key optimizing features are discussed; indexing, compressing, and compacting. The simplicity of the first three chapters dwindles and we enter a more intermediate realm. For indexing, how to enable attribute indexing and spatial indexing in ArcGIS is discussed along with using indexes effectively. Many of you may have heard about database indexing before, but the concept of compression and compacting in a database may be foreign. These concepts are explored and their effective implementation explained.

The first part of the fifth chapter steps away from the GUI of ArcGIS for Desktop and ArcCatalog and switches to Python programming for geodatabase tasks. Although laden with simplicity, if you have absolutely no experience with programming or knowledge of the general concepts well then this chapter may be beyond your comprehension, but I would suggest performing the walkthroughs as it might give you an appetite for future programming endeavours. We are shown how to programmatically create a file geodatabase, add fields, delete fields, and make a copy of a feature class to another feature class. All this is achieved through Python using the arcpy module. Although aimed at highlighting the integration of programming with geodatabase creation and maintenance the author also highlights how programming and automation improves efficiency.

The second part of the chapter provides an alternative to using programming for geoprocessing automation in the form of the Model Builder. The walkthrough shows us how to use the Model Builder to build a simple model to create a file geodatabase and add a feature class to it.

The final chapter steps up a level from file geodatabases to enterprise geodatabases.

“An enterprise geodatabase is a geodatabase that is built and configured on top of a powerful relational database management system. These geodatabases are designed for multiple users operating simultaneously over a network.”

The author walks us through installing Microsoft SQL Server Express and lists some of the benefits of employing an enterprise geodatabase system. Once the installation is complete the next step is to connect to the database from a local and remote machine. Once connections are established and tested an enterprise geodatabase can be created to and its functionality utilised. You can also migrate a file geodatabase to and enterprise geodatabase. The last part of Chapter 6 shows how privileges can be used to grant users access to data that you have created or deny them access. Security is an integral part of database management.

Overall Verdict: for such a compact eBook (158 pages) it packs a decent amount of information that provides good value for money, and it also introduces other learning ventures that come part and parcel with databases in general and therefore geodatabases. Many of the sections could be expanded based on their material but the pagination would then increase into many hundreds (and more) and beyond the scope of this book. The author, Hussein Nasser, does a great job with limiting the focus to the workings of geodatabases and not veering off on any unnecessary tangents. I would recommend using complimentary material to bolster your knowledge with regards to many of the aspects such as entity-relationship diagrams, indexing (both spatial and non-spatial), Python programming, the Model Builder, enterprise geodatabases and anything else you found interesting that was only briefly touched on. Overall the text is a foundation for easing your way into geodatabase life, especially if shapefiles are still the centre of you GIS data universe.