Python default encoding - how to set UTF-8?

Hello
I'm trying to save cyrillic UTF-8 string to shapefile with python plugin "pyshp".

Error is

Runtime error (EncoderFallbackException): 'ascii' codec can't decode byte 0 in position 0: ordinal not in range 


Traceback:   

line 891, in _Writer__dbfRecords, "C:\Program Files\Rhinoceros 5 (64-bit)\Plug-ins\IronPython\Lib\shapefile.py"   

line 1005, in saveDbf, "C:\Program Files\Rhinoceros 5 (64-bit)\Plug-ins\IronPython\Lib\shapefile.py"   

line 1033, in save, "C:\Program Files\Rhinoceros 5 (64-bit)\Plug-ins\IronPython\Lib\shapefile.py"

In Python 2.7 when "coding: UTF-8" was set everything works fine.
So, how to set default encoding in IronPython?

Pyshp plugin file in which error was http://pyshp.googlecode.com/svn/trunk/shapefile.py

Replies to This Discussion

Permalink Reply by Giulio Piacentino on January 8, 2014 at 2:16am

Does calling sys.setdefaultencoding('utf-8') as stated in:
https://mail.python.org/pipermail/ironpython-users/2005-December/00...

work? More background is I think here. Rhino currently uses IronPython based on Python 2.7.x.

Thanks,

Giulio
--
Giulio Piacentino
for Robert McNeel & Associates
giulio@mcneel.com

Permalink Reply by Giulio Piacentino on January 8, 2014 at 7:56am

I really do not know what might be the cause of this. Maybe you might try to change the encoding in the open() call that will probably be somewhere in the library. But maybe the best idea is to ask the people who wrote the library originally. Do you have their contact at your disposal?

Thanks,

Giulio
--
Giulio Piacentino
for Robert McNeel & Associates
giulio@mcneel.com

Permalink Reply by Benjamin Golder on January 10, 2014 at 2:22pm

Hi Alexey,

when you pass the cyrillic text data to the shapefile plug, is it already encoded as utf-8 or is it decoded as unicode?

If you look carefully at the shapefile.py source code, you can see that it assumes you are writing str data to a file, which in in python versions before python 3, is ascii. Unless you rewrite portions of shapefile.py, you can only pass it text data that has already been encoded in utf-8.

So depending on how you are inputting the cyrillic text, you may need to convert it to unicode, decode it to unicode, or encode it to utf-8. But when it goes to shapefile.save(), it should be encoded as utf-8.

Permalink Reply by Benjamin Golder on January 10, 2014 at 2:31pm

These commands might be relevant

# here we get unicode type object, 
# assuming our str has been encoded as utf-8
unicode_data = data.decode("utf-8")

# here we convert the unicode to a str, encoding it using utf-8 
encoded_string_data = unicode_data.encode("utf-8")

RSS