extracting three numbers from a text file

I have a text file which is separated by spaces. I need to extract out the 3rd, 4th and 5th column to use as x,y,z coordinates. I have tried the text split using a single space as the delimiter and then using the cull command to remove the resulting empty string. However due to the size of the file, it takes sometime (>5 minutes) to process the file. Is there a faster way to do this?

Many thanks

William

Attachments:

datafile.JPG, 87 KB

Replies to This Discussion

Permalink Reply by Armin Seltz on July 27, 2016 at 1:36am

I am doing the same thing a lot in a patch I have been working on for a long time. I have something like 30 columns and usually around 6000 lines in it. It takes no time at all to read and split those. Could you attach one of the files to check it?

Could it be that you are reading the text file from a file server? The native GH File Read component has a bug where if the file is on a file server it will take a few minutes before reading it properly. Here is the link to the discussion and David's custom component (readlines.gh) works around that: http://www.grasshopper3d.com/forum/topics/reading-a-csv-file-from-a...

Hope that helps.

Permalink Reply by WT4 on July 27, 2016 at 6:36am

My problem is not so much the reading but the splitting because of the spaces used as delimiters.

As shown in the jpg file, it results in having to create a 'tree' structure with about 3.5 million entries, the majority of which are <empty> strings.

Attachments:

split.JPG, 126 KB

Permalink Reply by WT4 on July 27, 2016 at 6:58am

Here is a sample file. Many thanks

William

Attachments:

1601-1b.txt, 5 KB

Permalink Reply by ng5 Alex on July 27, 2016 at 7:36am

Permalink Reply by WT4 on July 27, 2016 at 8:42am

Many thanks for this. This just goes to show how powerful the python language is. Just one more favour though. In the original file there was a header line which I had edited out. Can you just change the python script to allow me to specify how many header lines to ignore before working on the rest of the file to split. I have never written any python script before.

Attachments:

1601b_withheader-1.txt, 2 KB

Permalink Reply by WT4 on July 27, 2016 at 9:44am

Thank you for your help. I will now persevere to learn python within grasshopper.

William

Permalink Reply by WT4 on July 27, 2016 at 11:48pm

Hi Marcus

The 'For line in file:' reads all the remaining lines in the file. If for examples there are 10000 lines but I only want to process the first 5000 lines how do I go about it. I have been able to specify the start within the file by incorporating a for loop

for I in range (0,10):

next(file)

For line in file:

Permalink Reply by WT4 on July 28, 2016 at 6:05am

Thank you so much for your help

Permalink Reply by ng5 Alex on July 27, 2016 at 6:51am

Hello WT4,

what data are present in txt file, other than xyz coordinates. Do you have options to modify the way the txt file is saved?

there is volvox plugin that can load point clouds in .txt format really fast. I have worked with 1-2 gb txt files and they load pretty quickly. the text file needs to have in each line data for each point. According to these data you provide a text mask to volvox like (x y z r g b u v w) describing coordinates, color and vectors. You can skip colors and vectors and load only xyz coordinates.

If you cannot alter the way the txt file is saved you can use Editpad Pro or similar software, that work with huge txt files, to clean it up from empty entries etc, delete whole rows.

best

alex

Permalink Reply by WT4 on July 27, 2016 at 7:04am

I cannot modify the data structure because field service engineers are going to use the script. They want to be able to feed the data as shown in the images into a script that I have written without the need for pre-editing of the data. Many thanks William

Permalink Reply by Armin Seltz on July 27, 2016 at 7:04am

Youcchhh. Ok, that is a lot of data. The only thing I can think of to reduce it would be to open the file in something like Excel using the .csv import, delete all columns you dont need, save as .csv again (or save as Excel file and use one of the many excel readers in GH, although I doubt they will be much faster) and then use that data. Maybe even save each column in a separate .txt file, so it doesn't have to do the splitting, which is very calculation intensive.

Otherwise you will just have to bite the bullet and wait. Remember that it will only calculate the splitting once, since any changes will only be calculated down-stream.

Things are also really slow, because a text file, after is being split, will still be strings and has to be converted to numbers. Dont do that with all the <empty> strings still in the data, because it will try to interpret them as an equation first (see: http://www.grasshopper3d.com/forum/topics/text-to-integer-very-slow).

Permalink Reply by WT4 on July 27, 2016 at 2:11pm

Thank you for your reply. Marcus has provided a solution in Python. Never realised how powerful Python is.