Managing large data base

Hi !

First of all, thank you for this amazing plugin. I am using it on a daily basis and it has been very helpful for my professional research and projects.

I was trying to work with bigger amount of data with elk, through a first step of simplification of the ism file through QGIS.

However I won't be able to export the file as OSM but only as a vector file, thus, won't use it with elk.

My goal is to study, as an example, the road network for the whole london. Meaning, I don't need the data for the buildings, leisure etc. For now, when using elk it first compute the whole data set through the "location" component. Right?

Then I am able to select which one to preview etc. But since it has first to calculate for the whole data base, I can't study those bigger area because they contain more information.

Would it be possible to somehow sort this issue with elk or qgis or any sort of data management?

Thank you. :)

Replies to This Discussion

Permalink Reply by Timothy Logan on August 22, 2017 at 7:45am

The OSM file contains a few different groups of information, but I'll focus on the Node and Way data. I was recently working with a fairly large OSM file (184MB) for Washington DC that contained 762,367 nodes and 95,265 ways. These nodes represent the lat/lon coordinates of any data within the file, and the ways represent a specific feature, which in addition to data about that feature will also have a list of node references that point it back to one or more of the nodes.

The Location component goes through and first processes the nodes, converting the lat/lon coordinates to a Rhino Point and then combines that point with the node's ID and any other information that may be stored within it and outputs that list of converted nodes through the OSM output of the Location component. For my Washington DC file the Location component took 13.3 seconds to calculate and convert all of the nodes.

The big time sink comes in the next part, the OSM Data component that will parse through the ways for ones of that match the selected feature type and subtypes, and then lookup the referenced points against the list of incoming OSM Points from the Location node. This node took 12.9 minutes to calculate and is what really becomes problematic when dealing with large datasets. It's probably also worth noting that I'm running these on a relatively powerful desktop.

I did run a test to see how simplifying the dataset would affect the overall times, and as with everything there are tradeoffs. The big one is the time to parse through the file to find the relevant information and save it to a new OSM file is much slower than I would hope. I didn't spend much time on it and it's probably far from optimized, but I wrote a C# script to read through an OSM file and extract only the relevant nodes and ways that correspond to an input feature type, then output a new OSM file with those extracted nodes and ways. The result was a significant reduction in file size, down to 26MB, but it took 22.8 minutes to process. It's a big text file, but I have to think I'm doing something wrong if it took that long.

Once that process was complete and I input the new OSM file into Elk it did run much quicker. The time for the node processing (Location component) was down to 1.3 seconds and the time for extracting the ways (OSM Data component) was down to 1.1 minutes. This probably doesn't go far enough, as it's not pulling in relationship data, but that could be added if necessary. It looks like it's missing some information since there's a difference of 321 branches of data between the stripped down version of the file and the full file, but most of it is there.