Anyone got a python module for Fujitsu ATLAS?

Re: Anyone got a python module for Fujitsu ATLAS?

gandalfrockman said:

I'm writing a quick and dirty (probably an understatement) python tool to do rapid machine translations of rpg maker patches made using Habisain's AWESOME RPGTrans tool.

Does anyone have a python module to tie into fujitsu ATLAS?

I'm using google translate right now, but I'm already running into their bot detection just testing the damned thing, and frankly a properly configured ATLAS does a much better translation with the right dictionaries plugged into it.

I can probably write something to tie python into atlas myself if I have to, but it will take me MUCH longer and it will be ugly. Anyone got a python module they can toss me?

Click to expand...

Uhhhh, you do know there is a software called visual novel reader that uses python ans does texthooking/machine translation with just about anything right?

I use it on rpg maker games vn flash, just about anything. The software is downright amazing. I even lets you choose betwen online translations and gives options for offline (atlus being ne of them).

annot.me/wiki/en/VNR

For some reason out of the numerous hentai related communities I encounter, I never see people advocating this beauty.

Re: Anyone got a python module for Fujitsu ATLAS?

Uh... Well, out of interest, I checked out VNR's source code ( )... but, it's not a Python program. Not in the slightest. It's all in C++, and therefore not really useful to Python programmers.

Re: Anyone got a python module for Fujitsu ATLAS?

habisain said:

Hmm. You sure on that? Number 1) I've had the x...y...z problem happen with Atlas clipboard translate and Number 2) TA basically calls the same functions as my module, although it does do a substantial amount of preformatting. In particular, I notice it replaces all \t, \r, \n characters with a space, and converts all Shift-JIS punctuation to ASCII punctuation.

Now, TA does load a couple more DLLs, but I can't see any uses of them. I'd like to believe that means they're not important, but I guess they might be... Anyhow, the extra libraries are awdict and awuenv, should be easy enough to add loads for these.

Click to expand...

I'm not 100%, certain but I'm fairly confident since the same sentences aren't getting treated identically by the 3 applications (TA,py3atlastranslator,and atlas) Ill know for sure soon enough.


Might not be able to get much done on this over the weekend I might be going out of town.

Re: Anyone got a python module for Fujitsu ATLAS?

meh65 said:

Uhhhh, you do know there is a software called visual novel reader that uses python ans does texthooking/machine translation with just about anything right?

I use it on rpg maker games vn flash, just about anything. The software is downright amazing. I even lets you choose betwen online translations and gives options for offline (atlus being ne of them).

annot.me/wiki/en/VNR

For some reason out of the numerous hentai related communities I encounter, I never see people advocating this beauty.

Click to expand...

I'm aware of it, but the end goal here is a translation patch, in other words the text will be REINSERTED into the game in english in such a way that it appears in place of the japanese text.

Re: Anyone got a python module for Fujitsu ATLAS?

Personally, my suspicion is that the ATLAS engine cannot handle certain characters or character combinations and relies on the frontend to remove or replace them appropriately. TA/QuickAtlas apply differemt levels of pre-formatting, while the Python bindings are just raw bindings onto the engine and don't have preformatting.

I might have a bit of time to play around with this over the weekend though. Could you post some of the strings which get different results in the 3 programs?

Re: Anyone got a python module for Fujitsu ATLAS?

habisain said:

Personally, my suspicion is that the ATLAS engine cannot handle certain characters or character combinations and relies on the frontend to remove or replace them appropriately. TA/QuickAtlas apply differemt levels of pre-formatting, while the Python bindings are just raw bindings onto the engine and don't have preformatting.

I might have a bit of time to play around with this over the weekend though. Could you post some of the strings which get different results in the 3 programs?

Click to expand...

Its certainly possible, if that's the case the .dll in question would just be responsible for filtering and reinserting punctuation and special characters not "advanced sentence magic" it's plausible and it certainly wouldn't be the first time a vendors support team had exaggerated the complexity of their product.

I'll be working on my translation rules and custom dictionary merge this weekend, I can't take my pre/post processing any further until after I have that sorted out a bit more; then I can setup a preprocess and a postprocess cfg file so that people can fine tune things easily for a specific project without having to futz with the atlas dictionary too much.

Here's map015 from saitama, it has some of the trouble sentences.

Re: Anyone got a python module for Fujitsu ATLAS?

Well, I'll probably have a look at that tonight. I've had a cursory glance through and I have a strong suspicion that the floating dots and possibly smilies are responsible.

Re: Anyone got a python module for Fujitsu ATLAS?

Just a quick update: Custom dictionary merge is almost finished, but I have some bugs to squish in regards to my post-processing; basically its complete shit, and I don't how I ever thought my original plan was a good idea so I'm re-writing it.

Lesson learned: DO NOT WRITE COMPLEX STRING MANIPULATION LOGIC WHILE BINGE CODING AT 3:00 AM!

Re: Anyone got a python module for Fujitsu ATLAS?

Well, I've done my testing, and it seems that I was right: ATLAS core cannot handle Shift-JIS punctuation, and a frontend must strip these out. To wit:

遥かな「さいたま！」\|を夢見て多くの者達がこの塔の秘密に挑んでいった・・・。
Much..provide..Saitama..dream..a lot..person..this..tower..secret..challenge.

遥かな"さいたま!"を夢見て多くの者達がこの塔の秘密に挑んでいった....
"Saitama" was dreamt and a lot of people challenged about this tower much ..it was.. behind the curtain.

This isn't quite the same as TA's or QuickATLAS's output though, because both of these break the string around the「」marks and concatenates the result of translating the beginning, inside quotes and end separately. QuickATLAS also breaks around the \|, which TA doesn't; by accident, this makes TA's output read better than QuickATLAS's, because ATLAS core translates | to a yen sign, meaning that を will translate correctly; in QuickATLAS it just gives を as [wo].

Anyhow, I'll probably make a preprocessor for py3atlastranslate soonish. It seems that it is very much necessary.

Re: Anyone got a python module for Fujitsu ATLAS?

I'm thinking the way I'll handle the pre-processing is with a config file that lists pairs. By default all the problem characters will be there, but people can add their own string pairs as well.

That way it isn't hard-coded and people can edit it for custom content for games. That should get it to handle names and lore specific stuff a little better without having to do a dictionary merge. I'm starting to realize most people don't want to mess with their atlas environment anymore than necessary and I can't say I blame them; its a fussy, poorly documented application and Fujitsu's official 300+ page manual doesn't say much this user community is likely to find helpful.

I was just wrapping up my dictionary when some malevolent soul sent me a 24,000 word japanese to english porn dictionary merge file for atlas.....
I don't know where the hell this thing came from and I'm not sure I want to... its very .... 'comprehensive' ... it even has an entry for what I think is a "rusty venture" (the sex act, not the fictional character) I'm not sure whether to be impressed or disturbed.

They asked me not to redistribute it, but suggested that it might serve as a useful resource.

So I'm taking another look at how I'm building the dictionary file, and how I'm handling nouns vs verbs in order to better accommodate all the new ones I'll be adding as well as some adjustments for my understanding of which part of speech is appropriate for which sex acts; I'd never given it any real thought before but they have some really bizarre grammatical exceptions. ie "Deep-throat" is a verb, but "Blowjob" is an associative noun, usually tied to a verb. So grammatical deconstruction of sex-acts and their respective parts of speech; that's probably going to be a pretty surreal way to spend an evening or two, but I've enlisted a female friend to help out on that part, so at least it wont be boring.

Dayjob is getting gnarly so I won't be able to get much done until this weekend.

Re: Anyone got a python module for Fujitsu ATLAS?

Status update!

-Cleanup pass and coding for some goofy exceptions.

-I HATE "\>\n", the fix I came up for it works, but its HIDEOUS, and it has to go outside my preprocessor, besides which replacing it can break the games that use it anyway.

-cleanup of my line break logic, and some checks to fix weird sentences that generate repeated Line breaks in postprocessing.

-Good lord, this dictionary is going to take a while... even with help, its going to take me a while to get it put together,the end product will be a combination of one of the older (pre duplication problem) FHC dictionaries, the mystery porno dictionary, and another one I found on hongfire that had been updated more recently, but I have to be selective, and 2 of those 3 dictionaries have crap loads of game specific proper nouns that I need to cull out since they break stuff all over the place when your ACTUAL proper nouns are represented by \n[033] and the like....

-I'm going to be putting together some screenshots and html of the proper atlas config. This is REALLY important; you can get some shockingly good translation if it's done right, and you HAVE to configure the bracket usage in such a way that it will not conflict with the way brackets are handled in RPGMaker. (This actually fixes some of those sentence structure problems we ran into)

WHAT IN GODS NAME MADE ME THINK THIS WAS A GOOD PROJECT TO USE AS A "LEARN PYTHON IN A WEEK" EXERCISE?!?!

Seriously, this is WAY more complicated than I anticipated, to the extent that doing it "Quick and Dirty" isn't even feasible.... evidenced by the fact I've needed to do a dedicated cleanup pass, and start working out a project plan for it....

Re: Anyone got a python module for Fujitsu ATLAS?

Two words: Feature creep

Seriously, getting an OK-ish machine translated version quickly with a Python script would be pretty easy. Getting a good machine translated veresion quickly is hard.

Re: Anyone got a python module for Fujitsu ATLAS?

habisain said:

Two words: Feature creep

Seriously, getting an OK-ish machine translated version quickly with a Python script would be pretty easy. Getting a good machine translated veresion quickly is hard.

Click to expand...

I had something 'ok-ish' over a week ago, it inserted ok and it was generally playable... but it wasn't 'ok-ish' enough, it wasn't superior to google translate results, dropped loads of text, inserted random control characters and line breaks, and frequently crashed either your patcher or the patched game until the problem translation was identified and fixed manually.

Good news is I have decided NOT to worry about how long it takes to translate a script initially, the ballooning dictionary continually increases the amount of time it takes to translate. It's still a helluva lot faster than an online translation, but it isn't 'fast'.

Re: Anyone got a python module for Fujitsu ATLAS?

Wait, you crashed RPGMaker Trans? That's odd. I thought I'd gotten all the problems with crashing RPGMaker Trans with input files down to soft "can't insert text" errors. Any chance you can remember what was crashing it?

Re: Anyone got a python module for Fujitsu ATLAS?

habisain said:

Wait, you crashed RPGMaker Trans? That's odd. I thought I'd gotten all the problems with crashing RPGMaker Trans with input files down to soft "can't insert text" errors. Any chance you can remember what was crashing it?

Click to expand...

Ill take a look and see I think I remember which test game and which build of mine caused the issue, it was something like:

"CRLF\>CRLF\> (random shiftjis special character stupidity) CRLFCRLFCRLF\>CRLF"

where crlf is an in-line carriage return and line feed; something that should never exist in your input files, the way I'm doing the postprocessing now it should never happen. I'll try to pin it down more precisely for you.

Re: Anyone got a python module for Fujitsu ATLAS?

Carriage returns shouldn't be a problem, actually. The first line in the parser is "string = string.replace('\r', '')", i.e. delete all carriage returns. Ah well. If you find the problem string, let me know, and any other crashes I'd also appreciate if you let me know.

Re: Anyone got a python module for Fujitsu ATLAS?

habisain said:

Carriage returns shouldn't be a problem, actually. The first line in the parser is "string = string.replace('\r', '')", i.e. delete all carriage returns. Ah well. If you find the problem string, let me know, and any other crashes I'd also appreciate if you let me know.

Click to expand...

Will do, I would have reported it initially (I know someone from my team sent you a bug report once before), but figured it was just me feeding something incredibly stupid into the parser.

Re: Anyone got a python module for Fujitsu ATLAS?

Oh, not saying you didn't feed in a stupid character. But it's still my fault, because the patcher should be more robust than that.

Re: Anyone got a python module for Fujitsu ATLAS?

Still alive and still working on this when I can; surprise business trip may be imminent though. If I have to go out of town Ill try and post something testable before I leave.

Re: Anyone got a python module for Fujitsu ATLAS?

Decided that if I've gone this far, I might as well go the rest of the way.
I'm adding support for some other extraction formats. Still going through and looking at different engines, but I'm targeting KiriKiri, RpgMaker, and plaintext for my first release.

I've already talked with several translators, and I have a good idea about some other engines to look at next, but if anyone has anything specific they want me to support go ahead and post it here. I will probably move this to a blog at some point soon.