python

I’ve been tinkering with a couple of things and toying with another couple of ideas that when all combined form what I would call a data pipeline. I quite like the idea of building this data pipeline so I’ve decided that I shall do just that.

The things I’ve been working on already are a web scraper and a database. The other ideas I’ve been keen to explore are a data portal and an orchestrator. In reverse order the specific technologies are:

• Apache Airflow
• CKAN
• PostgreSQL and PostGIS
• Python

For the web scraper I’m going to revisit and rewrite the thoroughly unstructured and messy python scripts I wrote for my UK Freight Transport System map data. For the database I’m going to continue working on my geolab personal PostGIS server. The CKAN data portal is presently the least explored area, but I like what I see, and look forward to learning more about it. Lastly is Apache Airflow, which will literally tie all the other components together, in theory. As in, it could run the scraper on a schedule, transform its results and stick them in PostGIS, then something something prepare an export suitable for sharing via CKAN? We shall see.

I’ve been using ArcGIS for the first time in the last couple of weeks and I’ve come across the Swipe tool. My immediate thought was “oh that’s awesome” which was swiftly followed by “I wonder if QGIS has this as well?”

Oops, my first late weeknote! I intended to get it done at the weekend rather than on Wednesday like usual, but that has ended up slipping over to today, Monday. Anyway. What GIS things did I get up to since my last post?

I am exploring the world of Geographic Information Systems (GIS).

I have been reading the Gentle Introduction to GIS from the QGIS documentation and also working through the QGIS Training Manual. In the last couple of weeks I completed the first assignment.

“A man is not dead while his name is still spoken.”

One day I discovered the existence of Looking Glasses. It was so long ago now that I don’t remember exactly what led to it, but I know I found http://www.bgplookingglass.com and became quite intrigued by it.

A Looking Glass is a system that network operators might use to find out Internet routing and BGP-related information. They provide insight into how a particular router connects the Autonomous Systems that make up the internet.

But this post isn’t about Looking Glasses, it’s about something else I found. So back to my story…

TL;DR #

I made a RandomNote for Joplin button for macOS. You can find the bash, python and AppleScript concoction here.

Apologies to non-macOS Joplin users, I have not made any other verions of this.

But if you like sticking things together, the bash and python elements could be of use to you…

What’s RandomNote? #

I discovered the concept of “RandomNote” via Tiago Forte’s PARA series. To describe it at surface level: he created a button that opens a random note from his Evernote notes.

The purpose of such a button is to aid in serendipitous rediscovery of your old notes. That’s not going much deeper, to be honest. To see what inspired me into action I encourage you to read Tiago’s post about it. He has a way with words which I do not.

The rest of this post is an extension of the “API / Automation Friendly” criteria from my OneNote vs Joplin blog post.