PostGIS

I’ve been tinkering with a couple of things and toying with another couple of ideas that when all combined form what I would call a data pipeline. I quite like the idea of building this data pipeline so I’ve decided that I shall do just that.

The things I’ve been working on already are a web scraper and a database. The other ideas I’ve been keen to explore are a data portal and an orchestrator. In reverse order the specific technologies are:

• Apache Airflow
• CKAN
• PostgreSQL and PostGIS
• Python

For the web scraper I’m going to revisit and rewrite the thoroughly unstructured and messy python scripts I wrote for my UK Freight Transport System map data. For the database I’m going to continue working on my geolab personal PostGIS server. The CKAN data portal is presently the least explored area, but I like what I see, and look forward to learning more about it. Lastly is Apache Airflow, which will literally tie all the other components together, in theory. As in, it could run the scraper on a schedule, transform its results and stick them in PostGIS, then something something prepare an export suitable for sharing via CKAN? We shall see.

Stuff from the last week:

I’ve been running through my ankiboxes and considering the purposes of my different IW queues and associated ankiboxes a little more closely.

I redesigned part of my homepage. The “recent feed” part now has two columns – one for blog posts and another for weeknotes. Both of these things live under the /blog/ section of my site but I figured out a way to list them separately as well as altogether, as demonstrated by my new homepage and my new index pages: blog.html and weeknotes.html and archives.html

I started a new lab page – my first since 2019 apparently – for “geolab” my personal PostGIS server. I’ll be adding to this incrementally as I refine my slightly messier note I’ve been working on for the last few weeks.

I managed to get caught in torrential rain twice in the last week. The second time I was slightly more prepared and had an umbrella, but honestly I didn’t end up that much better off ha!

In Lesson 4 of GEOG 868 I completed a site selection using PostGIS.

A few things I got up to this week:

In Lesson 3 of GEOG 868 I played around with PostgreSQL and PostGIS for the first time. For the project at the end you are given a class_roster.txt file that contains a list of students and their postal codes. The task was to join that with data provided earlier in the lesson and then plot the students’ locations on a map in QGIS. Here’s how I went about doing that.

So. Hello again.