Changing character set to UTF-8!

Post by **Merri** » 2005-09-17 15:39

I'm changing the character set for every language to UTF-8. I have to edit a lot of files and convert each of them from their own character encoding to UTF-8, but I guess it will be worth it. After the change we should have no problems with characters not displaying correctly.

Will be quite a lot of work, I had to find a program to do the job for me and it isn't very convenient to use for conversion. I hope I won't make any mistakes

Due to the change some or many of the old posts will display incorrectly! - I can't help it all that much, the only way would be to rewrite the messages using correct character set :/

Post by **Merri** » 2005-09-17 16:36

The change is now done! Tell me if you find any weird text etc, I can't read most of the languages of the board so I made the changes being almost blind.

I ended up coding my own program to do the character coding changes

Then the bad news: all the posts that used ISO-8859-1 display incorrectly. This means that posts in Finnish, French, German, Portuguese and Spanish display incorrectly. However, most Chinese and Japanese posts display correctly as they were UTF-8 already. I have fixed a two Chinese and Japanese posts to UTF-8. As for the other posts, I hope you are able to fix the posts by yourself if the need be

There are too many for me to fix.

Sora · Post by **Sora** » 2005-09-17 23:18

[quote="Merri"]Then the bad news: all the posts that used ISO-8859-1 display incorrectly. This means that posts in Finnish, French, German, Portuguese and Spanish display incorrectly. However, most Chinese and Japanese posts display correctly as they were UTF-8 already. I have fixed a two Chinese and Japanese posts to UTF-8. As for the other posts, I hope you are able to fix the posts by yourself if the need be :) There are too many for me to fix.[/quote]
I feel quite angry right now. Because all the posts of the French and Spanish section have encoding problems. And it will ask a lot of time from the people from each section to edit them all.

I would have truly appreciated to know you were going to change the character set SOME DAYS BEFORE. This way, I would have saved all the French posts on my computer and they would have been easier to edit after ; now I will have to read each of them and try to guess what was the word used because not only the special characters are replaced by strange Chinese signs, but also the letters around those special characters, which makes the work even harder.

Edit : this place was meant to be international right from the beginning ; why not choosing the UTF-8 encoding from the start ? Was it hard to set up ?

chibiNekomancer · Post by **chibiNekomancer** » 2005-09-17 23:30

The change was so sudden!! @_@

Um.... i guess that from now on, the posts will be ok, but about editing the other posts... i think that is.... well not practical. I mean, its more than 100 posts for the spanish section alone! :x

I am not sure how the encoding works, but cant it be only for the sections that need it? >.<

Post by **Merri** » 2005-09-17 23:43

[quote="Sora"]Edit : this place was meant to be international right from the beginning ; why not choosing the UTF-8 encoding from the start ? Was it hard to set up ?[/quote]

Yes. When I started the forum, I didn't know of any easy way to change text character sets from one to another. However, earlier in the summer I happened to code a bit of a code that made it possible to change the text encoding. I used this code to convert the text files for me. Also, I basically had no option but to put the files up right away: otherwise it would just be forgotten etc. - and as you could guess, phpBB doesn't have UTF-8 text files available anywhere.

You can still read all the old messages by changing the browser's character encoding; that is what can be easily used to copy a message, then going editing and pasting. Character encoding goes automatically back to UTF-8 by loading a page, but clipboard stores everything in Unicode.

chibiNekomancer: I can't limit it to certain language or section (well, I could always recode the whole board, but then that wouldn't be practical). It is better everyone uses the same character encoding and not just their own. It will be much better in the future.

If it helps at all, I can start massediting messages to UTF-8.

Edit I think I figured out a solution! I could rip off the whole text table from the database and then massfix the characters.

Edit As off-topic, I tend to have this "I make a good thing and start planning on doing something and then get kicked in the head" syndrome.

Sora · Post by **Sora** » 2005-09-18 00:57

[quote="Merri"]You can still read all the old messages by changing the browser's character encoding; that is what can be easily used to copy a message, then going editing and pasting. Character encoding goes automatically back to UTF-8 by loading a page, but clipboard stores everything in Unicode.[/quote]
That's what I am doing, opening a window with UTF encoding and another with the right encoding for French language and editing in the UTF window looking at the posts in the other window ; I can't copy and past that easily, because there is some BBcodes and smileys in the posts.

I wonder how many hours this will take me... =_=

Post by **Merri** » 2005-09-18 01:05

There are always the hard ways and the easy ways... I started converting the language files the hard way, but soon ought I could do some programming. It would have taken hours to manually edit the files, coding a program took only about an hour.

The problem would be that I'd have to close the forum while I do the conversion; I probably do it some other time, I'm now all too tired.

Post by **Merri** » 2005-09-18 01:13

Just noticed some people might have problems to login, ie. Jerome. I fixed their nicks and sent an e-mail with a new password (old one wouldn't have worked because it is encoded using the nickname). Sorry about that and hope it isn't too big of a problem.

There were only three persons with a problematic nickname... luckily this board is still a very small one.

Sora · Post by **Sora** » 2005-09-18 01:41

I finished editing the French posts. It's 3:40 am. Good night~

(I had noticed the problem with Jérôme's nick and I was going to ask you to fix it)[/size]

Post by **Merri** » 2005-09-18 01:53

Sora, sometimes I wonder if you're mad or just a self-tormentor

Yuki-chan · Post by **Yuki-chan** » 2005-09-18 02:42

*cough cough*
Well...i dont know if this is the right place to discuss that, but i want to give my opinions about what you did Merri...
I know you didnt mean to harm anyone, but did you think it was a good idea to change everything just because of one person?! And it was about a chinese person!! Who understands Chinese here rise your hand _o_
You made one person pleased, but the majority(French and Spanish people) will have to work for hours and fix every post...
THEY ARE THE PRIORITY!! If they get mad and leave this forum, you'll be paying for no one...

And you created a lot of strange sections XD Theres only 2 portuguese speakers here...and both of them understand english, at least...if we had like....10 portuguese speakers, that would be understandable XD

Well, in fact i was going just to say to you to ask the French and Spanish moderators before doing such huges modifications
Its so unfair...now they'll have to work for hours for something you did without discussin about it...

EDITED!

[quote="Merri"]Sora, sometimes I wonder if you're mad or just a self-tormentor

[/quote]

Its one of your rules "do not curse others too much"...but isnt that what you just did?

Sora · Post by **Sora** » 2005-09-18 03:06

[quote="Merri"]Sora, sometimes I wonder if you're mad or just a self-tormentor :) [/quote]
Do you think I truly enjoy editing posts for more than 3 hours instead of working on personal things ?!?

Sora · Post by **Sora** » 2005-09-18 03:46

Really, what's wrong about giving a bit of yourself to that place..?!?

This forum is important for me, I've never worked so much on any project before ; I really want this forum to become an international place for all fans from any country to come and have fun together.

This was our original idea. I don't want to leave it behind and I will do everything I can to make people feel fine here and stay.

Tenson · Post by **Tenson** » 2005-09-18 08:58

What Merri did was intended as a help, but in the short term it is annoying. :?
Regarding at long term, with UTF all new messages from now on are going to be viewed right forever, without need of change browser encoding.

I'm going to take a look to the Spanish forum, and fix wrong characters.

Post by **Merri** » 2005-09-18 14:38

[quote="Sora"]Really, what's wrong about giving a bit of yourself to that place..?!?[/quote]

Well, I could've fixed it by myself in an easier fashion, though it is now getting more and more pointless as you've seen a lot of trouble. I didn't choose the right place to be funny, but it seems I'm getting misunderstood a lot. It probably didn't come out in the msg, but I did and do respect the trouble you saw, even after I told I figured out a way where I could fix the mess.

Yuki: it wasn't only one person and the amount would have just increased over the time. Besides everyone who visited didn't see everything correctly which probably lessened the amount of people who might have joined: "they can't even display the characters correctly! Bad place! Not interested! Will go away!". I guess now it is possible to register a nickname using Far East characters (it wasn't really possible earlier, because the characters would've messed up).

As I told before, now that I thought it more, I could've done the changes all by myself. Just that it came to my mind too late. I still think the advantage in doing this now, immediately, was better than waiting for a few weeks and doing then (if still remembering to do it). There are many things to consider in this... for example, I don't have the interest and will to be here doing things every day. So when I get into doing something, I better do it immediately and not push it away. It will just make the things not to happen. Atleast this was done, because it really had to be done sooner or later. We don't need to do this a second time nor do we need to edit other people's messages to display correctly in the future (which I did do for a few messages).

I created more forums for one reason: there is one for each supported language (minus Finnish, because I know Maaya fanbase isn't strong enough in here). Portuguese is used in Brazil and Portugal, which is a pretty large land area covering a lot of people. The smallest language area would be German or Korean for the forums we now have. I have no idea if those interest enough for new people to join. Atleast they are there now.