Changing character set to UTF-8!

Support & site news.
User avatar
Merri
Posting Freak
Posts: 819
Joined: 2004-04-01 03:16
Location: Finland
Contact:

Changing character set to UTF-8!

Post by Merri »

I'm changing the character set for every language to UTF-8. I have to edit a lot of files and convert each of them from their own character encoding to UTF-8, but I guess it will be worth it. After the change we should have no problems with characters not displaying correctly.

Will be quite a lot of work, I had to find a program to do the job for me and it isn't very convenient to use for conversion. I hope I won't make any mistakes :)


Due to the change some or many of the old posts will display incorrectly! - I can't help it all that much, the only way would be to rewrite the messages using correct character set :/
Now stuck in programming JavaScript
User avatar
Merri
Posting Freak
Posts: 819
Joined: 2004-04-01 03:16
Location: Finland
Contact:

Post by Merri »

The change is now done! Tell me if you find any weird text etc, I can't read most of the languages of the board so I made the changes being almost blind.

I ended up coding my own program to do the character coding changes :D


Then the bad news: all the posts that used ISO-8859-1 display incorrectly. This means that posts in Finnish, French, German, Portuguese and Spanish display incorrectly. However, most Chinese and Japanese posts display correctly as they were UTF-8 already. I have fixed a two Chinese and Japanese posts to UTF-8. As for the other posts, I hope you are able to fix the posts by yourself if the need be :) There are too many for me to fix.
Now stuck in programming JavaScript
User avatar
Sora
Senior Member
Posts: 444
Joined: 2004-04-03 11:05
Contact:

Post by Sora »

[quote="Merri"]Then the bad news: all the posts that used ISO-8859-1 display incorrectly. This means that posts in Finnish, French, German, Portuguese and Spanish display incorrectly. However, most Chinese and Japanese posts display correctly as they were UTF-8 already. I have fixed a two Chinese and Japanese posts to UTF-8. As for the other posts, I hope you are able to fix the posts by yourself if the need be :) There are too many for me to fix.[/quote]
I feel quite angry right now. Because all the posts of the French and Spanish section have encoding problems. And it will ask a lot of time from the people from each section to edit them all.

I would have truly appreciated to know you were going to change the character set SOME DAYS BEFORE. This way, I would have saved all the French posts on my computer and they would have been easier to edit after ; now I will have to read each of them and try to guess what was the word used because not only the special characters are replaced by strange Chinese signs, but also the letters around those special characters, which makes the work even harder.

Edit : this place was meant to be international right from the beginning ; why not choosing the UTF-8 encoding from the start ? Was it hard to set up ?
☆NIKOPACHI☆ Love and... Peace !! \(^O^)/
Image
User avatar
chibiNekomancer
Senior Member
Posts: 351
Joined: 2004-04-02 03:33
Location: México

Post by chibiNekomancer »

The change was so sudden!! @_@

Um.... i guess that from now on, the posts will be ok, but about editing the other posts... i think that is.... well not practical. I mean, its more than 100 posts for the spanish section alone! :x

I am not sure how the encoding works, but cant it be only for the sections that need it? >.<
Neko ga Suki na Anata to
Inu ga Suki na Watashi
-- Neko to Inu, Maaya Sakamoto
User avatar
Merri
Posting Freak
Posts: 819
Joined: 2004-04-01 03:16
Location: Finland
Contact:

Post by Merri »

[quote="Sora"]Edit : this place was meant to be international right from the beginning ; why not choosing the UTF-8 encoding from the start ? Was it hard to set up ?[/quote]

Yes. When I started the forum, I didn't know of any easy way to change text character sets from one to another. However, earlier in the summer I happened to code a bit of a code that made it possible to change the text encoding. I used this code to convert the text files for me. Also, I basically had no option but to put the files up right away: otherwise it would just be forgotten etc. - and as you could guess, phpBB doesn't have UTF-8 text files available anywhere.

You can still read all the old messages by changing the browser's character encoding; that is what can be easily used to copy a message, then going editing and pasting. Character encoding goes automatically back to UTF-8 by loading a page, but clipboard stores everything in Unicode.


chibiNekomancer: I can't limit it to certain language or section (well, I could always recode the whole board, but then that wouldn't be practical). It is better everyone uses the same character encoding and not just their own. It will be much better in the future.


If it helps at all, I can start massediting messages to UTF-8.


Edit I think I figured out a solution! I could rip off the whole text table from the database and then massfix the characters.


Edit As off-topic, I tend to have this "I make a good thing and start planning on doing something and then get kicked in the head" syndrome.
Now stuck in programming JavaScript
User avatar
Sora
Senior Member
Posts: 444
Joined: 2004-04-03 11:05
Contact:

Post by Sora »

[quote="Merri"]You can still read all the old messages by changing the browser's character encoding; that is what can be easily used to copy a message, then going editing and pasting. Character encoding goes automatically back to UTF-8 by loading a page, but clipboard stores everything in Unicode.[/quote]
That's what I am doing, opening a window with UTF encoding and another with the right encoding for French language and editing in the UTF window looking at the posts in the other window ; I can't copy and past that easily, because there is some BBcodes and smileys in the posts.

I wonder how many hours this will take me... =_=
☆NIKOPACHI☆ Love and... Peace !! \(^O^)/
Image
User avatar
Merri
Posting Freak
Posts: 819
Joined: 2004-04-01 03:16
Location: Finland
Contact:

Post by Merri »

There are always the hard ways and the easy ways... I started converting the language files the hard way, but soon ought I could do some programming. It would have taken hours to manually edit the files, coding a program took only about an hour.

The problem would be that I'd have to close the forum while I do the conversion; I probably do it some other time, I'm now all too tired.
Now stuck in programming JavaScript
User avatar
Merri
Posting Freak
Posts: 819
Joined: 2004-04-01 03:16
Location: Finland
Contact:

Post by Merri »

Just noticed some people might have problems to login, ie. Jerome. I fixed their nicks and sent an e-mail with a new password (old one wouldn't have worked because it is encoded using the nickname). Sorry about that and hope it isn't too big of a problem.

There were only three persons with a problematic nickname... luckily this board is still a very small one.
Now stuck in programming JavaScript
User avatar
Sora
Senior Member
Posts: 444
Joined: 2004-04-03 11:05
Contact:

Post by Sora »

I finished editing the French posts. It's 3:40 am. Good night~

(I had noticed the problem with Jérôme's nick and I was going to ask you to fix it)[/size]
☆NIKOPACHI☆ Love and... Peace !! \(^O^)/
Image
User avatar
Merri
Posting Freak
Posts: 819
Joined: 2004-04-01 03:16
Location: Finland
Contact:

Post by Merri »

Sora, sometimes I wonder if you're mad or just a self-tormentor :)
Now stuck in programming JavaScript
User avatar
Yuki-chan
Senior Member
Posts: 325
Joined: 2005-02-15 00:53

Post by Yuki-chan »

*cough cough*
Well...i dont know if this is the right place to discuss that, but i want to give my opinions about what you did Merri...
I know you didnt mean to harm anyone, but did you think it was a good idea to change everything just because of one person?! And it was about a chinese person!! Who understands Chinese here rise your hand _o_
You made one person pleased, but the majority(French and Spanish people) will have to work for hours and fix every post...
THEY ARE THE PRIORITY!! If they get mad and leave this forum, you'll be paying for no one...

And you created a lot of strange sections XD Theres only 2 portuguese speakers here...and both of them understand english, at least...if we had like....10 portuguese speakers, that would be understandable XD

Well, in fact i was going just to say to you to ask the French and Spanish moderators before doing such huges modifications
Its so unfair...now they'll have to work for hours for something you did without discussin about it...

EDITED!

[quote="Merri"]Sora, sometimes I wonder if you're mad or just a self-tormentor :) [/quote]

Its one of your rules "do not curse others too much"...but isnt that what you just did?
"Missing someone doesnt mean you are apart but that one day you were together"
User avatar
Sora
Senior Member
Posts: 444
Joined: 2004-04-03 11:05
Contact:

Post by Sora »

[quote="Merri"]Sora, sometimes I wonder if you're mad or just a self-tormentor :) [/quote]
Do you think I truly enjoy editing posts for more than 3 hours instead of working on personal things ?!?
☆NIKOPACHI☆ Love and... Peace !! \(^O^)/
Image
User avatar
Sora
Senior Member
Posts: 444
Joined: 2004-04-03 11:05
Contact:

Post by Sora »

Really, what's wrong about giving a bit of yourself to that place..?!?

This forum is important for me, I've never worked so much on any project before ; I really want this forum to become an international place for all fans from any country to come and have fun together.

This was our original idea. I don't want to leave it behind and I will do everything I can to make people feel fine here and stay.
☆NIKOPACHI☆ Love and... Peace !! \(^O^)/
Image
User avatar
Tenson
Senior Member
Posts: 387
Joined: 2004-04-01 18:01
Location: Seville, Spain

Post by Tenson »

What Merri did was intended as a help, but in the short term it is annoying. :?
Regarding at long term, with UTF all new messages from now on are going to be viewed right forever, without need of change browser encoding.

I'm going to take a look to the Spanish forum, and fix wrong characters.
User avatar
Merri
Posting Freak
Posts: 819
Joined: 2004-04-01 03:16
Location: Finland
Contact:

Post by Merri »

[quote="Sora"]Really, what's wrong about giving a bit of yourself to that place..?!?[/quote]

Well, I could've fixed it by myself in an easier fashion, though it is now getting more and more pointless as you've seen a lot of trouble. I didn't choose the right place to be funny, but it seems I'm getting misunderstood a lot. It probably didn't come out in the msg, but I did and do respect the trouble you saw, even after I told I figured out a way where I could fix the mess.


Yuki: it wasn't only one person and the amount would have just increased over the time. Besides everyone who visited didn't see everything correctly which probably lessened the amount of people who might have joined: "they can't even display the characters correctly! Bad place! Not interested! Will go away!". I guess now it is possible to register a nickname using Far East characters (it wasn't really possible earlier, because the characters would've messed up).

As I told before, now that I thought it more, I could've done the changes all by myself. Just that it came to my mind too late. I still think the advantage in doing this now, immediately, was better than waiting for a few weeks and doing then (if still remembering to do it). There are many things to consider in this... for example, I don't have the interest and will to be here doing things every day. So when I get into doing something, I better do it immediately and not push it away. It will just make the things not to happen. Atleast this was done, because it really had to be done sooner or later. We don't need to do this a second time nor do we need to edit other people's messages to display correctly in the future (which I did do for a few messages).


I created more forums for one reason: there is one for each supported language (minus Finnish, because I know Maaya fanbase isn't strong enough in here). Portuguese is used in Brazil and Portugal, which is a pretty large land area covering a lot of people. The smallest language area would be German or Korean for the forums we now have. I have no idea if those interest enough for new people to join. Atleast they are there now.
Now stuck in programming JavaScript
Post Reply