speed questions

Hi all,

I have a python node that does some heavy computation that iterates across arrays of between 100 and 60000 points. The node spits out the solution for all of the points. The next node in line takes the solution in as a list of solutions, looks for a solution in the based on some input for the index value, and then uses the solution to do some more computation.

When I get above about 1000 points the second node gets unusably sluggish to respond. I'm contemplating having the first node write the solutions to a file on the harddrive, then have the second node look for a specific line in the file and parse it back out to use to compute a solution in the second node.

Two questions:

1) Do you have thoughts on what will be faster - writing to/reading from the harddrive vs. storing and accessing all of these solutions from memory?

2) When you have data coming out of an output in one python node, is that data repeated/copied in memory when it is put into an input in another python node?

Thanks so much.

Replies to This Discussion

Permalink Reply by David Rutten on March 7, 2013 at 8:25pm

Memory access is way faster than disk access.

Data is copied all the time in Grasshopper, since downstream components aren't allowed to change data shared with other components.

Rule #1 in performance optimization: Figure out where the bottleneck is. Don't assume it's due to memory access or data copying, measure it. Unfortunately you can only measure your own code, but I suggest you keep track of how much time is spend in certain loops so you get a better idea of why it slows down so fast. You can use the System.Diagnostics.Stopwatch class to keep accurate time.

David Rutten

david@mcneel.com

Poprad, Slovakia

Permalink Reply by person on March 8, 2013 at 4:33pm

That's all great to know, David, thank you for the post. I will look into the stopwatch class and see what I can learn. Is there also a way to keep track how much memory your using per definition or per array?

I did run into an unusual instance recently when I sent data in the form of a list from one python component to another, and then used list.del and list.pop to delete items from that list. The data in the previous node was changed. I ran a print len(list) in the python node earlier in the chain before and after using list.del and list.pop in the next python node in the chain. I noticed that the length of the list in the python node earlier in the chain shrank to match the effect of the list.del / list.pop that was run in the next python node. I will try to replicate and post code.

Permalink Reply by Mostapha Sadeghipour Roudsari on March 8, 2013 at 4:42pm

Kendra,

What you get is normal since in python both lists are referring to the same place somewhere in the memory (they are the same list!). You should make a new copy of the list if you want the changes not to effect the first list. Something like:

list2 = []; [list2.append(item) for item in list1]

and then change the items in list2.

As a side note print is so expensive computationally and can slow down the process. Use it carefully or it can slow down the process itself.

-Mostapha

Permalink Reply by person on March 8, 2013 at 5:07pm

Thanks Mostapha, part of a generation of home schooled coders here.

So, let's say I have a list of numbers called numberList, and I then pipe numberList through a definition that adds the value 2 to each number in numberList. From what you are saying, it would be more memory efficient to code: numberList = addTwo(numberList) rather then numberListPlus2 = addTwo(numberList) because you would not be creating an additional list. Is this accurate?

def addTwo (list) :

temp = []

for i in range(len(list)) :

a = list[i] + 2

temp.append(a)

return temp

numberList = [1,2,3,4,5,6,6,7,8 etc. ]

#numberListPlus2 = addTwo(numberList) ## This makes an additional list

numberList = addTwo(numberList) ## This writes over the existing numberList

Permalink Reply by Mostapha Sadeghipour Roudsari on March 8, 2013 at 5:20pm

Kendra,

What I was trying to say is how lists work in Python. My understanding from your description was that you see the changes in the original list that you don't want to happen. Is that right? What I mentioned above is why this is happening and how you can solve it. Here is another example:

list1 = [1,2,3,4]
list2 = list1

list2[0] = 0

However I changed the first item in list2 if you print list1 it will return [0,2,3,4]! as I said they are the same list. If you want the changes in list2 doesn't effect list1 you should do something like this:

list1 = [1,2,3,4]
list2 = []
[list2.append(i) for i in list1]
list2[0] = 0

print list1 will return [1,2,3,4]

I think it will be helpful if you can post your script. Hope it helps.

-Mostapha

Permalink Reply by person on March 8, 2013 at 5:24pm

Got it. I should have mentioned that my most recent post was a more general observation of list behavior in python.

Permalink Reply by Anders Holden Deleuran on March 9, 2013 at 3:45pm

In continuation of that, I'm not really sure if any memory benefits of modifying lists in place outweighs the potential problems this may cause (i.e. losing track of what's what ID-wise). It would be interesting to examine this more in-depth actually. Either way, a few more Pythonic nuggets related to your examples above. Apologies if these are obvious!

The first case (adding to a list) can be written like so:

numberList1 = [1,2,3,4,5,6,6,7,8]
numberList2 = [i+2 for i in numberList1]
print numberList2

The second case (copying a list) can be performed using the list() function:

list1 = [1,2,3,4]
list2 = list(list1)
list2[0] = 0
print list1

Or by using the slice notation:

list1 = [1,2,3,4]
list2 = list1[:]
list2[0] = 0
print list1

Permalink Reply by Mostapha Sadeghipour Roudsari on March 10, 2013 at 4:40pm

Interesting! I didn't know that list() makes a copy of the list. I was only using it to convert tuple to list! Thanks Anders for posting.

PS: Do you have a Python cheat sheet somewhere on your blog? You should post one if you haven't done it already! ;)

Permalink Reply by Anders Holden Deleuran on March 11, 2013 at 3:50am

There's also the general Python copy functions. However I believe that those methods (particularly deepcopy) may be slower than what we've already covered, at least for lists.

:D

Permalink Reply by Anders Holden Deleuran on March 8, 2013 at 12:48pm

The following does not directly answer your questions, but maybe be helpful if you're experiencing slow execution speeds in a Python component in general:

If you are using the Rhinoscriptsyntax library this may substantially slow things down as compared to straight up RhinoCommon (see this thread).

Beyond that it sounds like you might have some bottlenecks and/or "logical errors" in your scripts which are hard to guess without any code. As David suggests you probably want to profile the code to find these (I put up an example file on how to do this in the other thread).

Hope that helps..

Permalink Reply by person on March 8, 2013 at 4:24pm

Thanks Anders. I've been wondering how RhinoCommon compared to rhinoscriptsyntax. I typically use a mix of both but it sounds like RhinoCommon might be the way to go.

RSS