How to create engram for Polish?

[AKmatiAK]

Is it hard to get engram layout for language like polish? I created my layout based on engram, but maybe it can be further optimized.

[AKmatiAK]

I updated my layout, here it is:

altgr layer remains the same

[binarybottle]

Please take a look at the optimized layout for Spanish (https://github.com/binarybottle/engram-es) that we created.

If there is very accurate and representative 1-gram and bigram frequency data for the Polish language (including symbols), then we could apply a modified version of the code to generate an Engram layout optimized for Polish.

[AKmatiAK]

Hi. I contacted polish corpus creators and I got data up to 5-grams. It's available here: n-grams pl data. Do I need to further process it or it's enough?

also, here is list of avaible resources that might be useful: link

[binarybottle]

This corpus looks pretty official! I like that it has a broad variety of book and news sources. Too bad it doesn't include spoken transcripts or social media sources. Anyway, I would be happy to help with this but it will be a couple of months before I can get to it -- buried with projects right now.

[AKmatiAK]

Ok, thank u for help ;)

[binarybottle]

@iandoug -- Given your experience helping to clean up the Spanish corpus, do you have any concerns about the proposed Polish corpus?: http://zil.ipipan.waw.pl/NKJPNGrams

[iandoug]

Hi Arno

For keyboard layout use, I prefer to strip texts not normally typed on computer keyboard (like spoken transcripts or tweets) because that will mess up the character frequencies and n-grams.

"Each unigram is maximum continuous chunk of non-whitespace lower-case characters."

That is the normal way of doing it. Ian of course is not normal and does it Case Sensitive ... :-) Because typing Th is different to th.

It looks like they only have "1-million-word subcorpus" available to download.

Is this typical Polish text?

Masz ty duszę? Powiedz!
Tak jest. Od ręki.
To chyba dobra formuła.

Do lines starting with a dash indicate dialogue? There seems to be a lot of that, which is going to bump up the dash frequency. Should maybe strip out leading dashes.

I have not seen any HTML etc, but since this is "manually annotated", I guess it is "clean" in that regard. There are some ALL CAPS sentences.

Will see if I can extract the text and do some analysis over the weekend. We are currently having rolling blackouts so that messes up plans.

[iandoug]

For future reference: Leipzig

https://wortschatz.uni-leipzig.de/en/download/Polish#pol_newscrawl_2011

[iandoug]

What's the difference in quotes? Should both be on keyboard?

która znalazła się w zestawieniu "Billboard Magazine".

” 2012, a zespół otrzymał nominację do nagród „Songlines Music Awards” 2012 w kategorii „Best Group”.
” (2012) oraz realizator dźwięku przy filmie „

[AKmatiAK]

Do lines starting with a dash indicate dialogue? There seems to be a lot of that, which is going to bump up the dash frequency. Should maybe strip out leading dashes.

yes, they're for dialogues, yet not often used for things other than books.

What's the difference in quotes? Should both be on keyboard?

bottom quotes are rarely used, shouldn't be on keyboard (they are now superseeded by both upper quotes and meaning is the same)

Is this typical Polish text?

Masz ty duszę? Powiedz!
Tak jest. Od ręki.
To chyba dobra formuła.

2,3 typical, 1 is correct but rather from books

[binarybottle]

Thank you for taking a look, @iandoug! I don't know Polish, so I will defer to @AKmatiAK and other Polish speakers/typists. A corpus of only 1 million words is pretty small, but I hope it represents what people type.

[iandoug]

I took a look at the linked corpus, not wild about it, seems to contain a lot of dialogue. Will try cleaning up some of it as next step after this.

Instead, I grabbed all the1M files from the Leipzig Polish corpus. After looking at those, decided to only use the "news" files, the rest is going to be a mess to clean. So that supplies 9 million sentences.

After tweaking my Spanish cleanup program, now have a 688 MB text file to play with. I grabbed some Polish books from Gutenbreg ... only a few, most seem to be poetry or dialogue-heavy novels. Will try my usual "extract some text" approach with those to add to the Leipzig file.

Current char distribution looks like this.
Provisional list, may change ...

char-dist-1.txt

polishfreq1.txt

[iandoug]

Gonna have to do this on ISO not ANSI ... need that extra key. Even then, going to be challenging. :-)

112 characters?

[binarybottle]

@iandoug -- Using news files sounds reasonable, but I wouldn't throw out dialogues -- they are far closer to how people type emails than books are.