Scribusoid

Author Topic: Syntax Highlighting - German Umlaute problem  (Read 356 times)

Michihalo

  • Newbie
  • *
  • Posts: 6
  • Karma: 0
  • Scribus User
Syntax Highlighting - German Umlaute problem
« on: March 28, 2019, 10:02:52 am »
I am trying to implement a syntax highlighting script (since i couldn't find one)
The basics work already but i encountered a Problem when using german Umlaute.
When there is an Umlaut in a sentence, the find() function of python suddenly returns a wrong position. The function itself does work correctly when i try the same setup in python command line, so i guess it has something to do with the scribus.getAllText function? Do i have to convert to ASCII or anything?

In the attatchment you can see the basic while loop and 2 Tests, one with no Umlaut and one with one Umlaut. You can see the difference in the result

a.l.e

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1818
  • Karma: 26
    • the graphics lab
Re: Syntax Highlighting - German Umlaute problem
« Reply #1 on: March 28, 2019, 12:49:58 pm »
interesting!

the error is probably due to the fact that python2 cannot correctly manage utf-8 strings.

you're welcome to continue your effort, but i fear that it will be harder than you think and you will need further help to correctly implement a basic syntax highlighting (among other things, you will probably need some sort of parser).

my last words for now:

- i think that it would be worth to do that in c++ rather than in python (the scripter API is too hard to use for such a complex task vs. c++ is harder to program than python)
- there is a way to get text in scribus to have the syntax highlighed, but -- iirc --it involves an export to html from an editor that can do that and then going through libre office... (i should have some notes somewhere)

: - )

(and avoid umlauts in your code...)

Michihalo

  • Newbie
  • *
  • Posts: 6
  • Karma: 0
  • Scribus User
Re: Syntax Highlighting - German Umlaute problem
« Reply #2 on: March 29, 2019, 11:24:42 am »
Thanks for your response.
I feared someone was writing something like this :/

Problem is, we have to write a documentation in german, so Umlaute are unavoidable.
I pobalby have to keep the frame of the source code unlinked from the one with the rest of the documentation, then this could work.

I read about the workaround with HTML export and stuff, but this seems a little to complicated. In Indesign you can apply character styles via grep, which is a lot easier. So it seems I am stuck with Indesign for now.

Thanks for the input though. I will try a little more, but since im neither a python nor a c++ expert i doubt than something more than a dubious hack will come out of this ;)


a.l.e

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1818
  • Karma: 26
    • the graphics lab
Re: Syntax Highlighting - German Umlaute problem
« Reply #3 on: March 29, 2019, 01:10:40 pm »
no! don't do that! : -)

anyway, i've opened a ticket asking for a "simple" solution that could enable some sort of syntax highlighting in scribus:

https://bugs.scribus.net/view.php?id=15623

amair

  • Newbie
  • *
  • Posts: 2
  • Karma: 0
  • Scribus User
Re: Syntax Highlighting - German Umlaute problem
« Reply #4 on: March 30, 2019, 08:53:39 am »
Hi Michihalo,

I did a script lately where - I guess - had a similar problem. I manged to finish the script using calls "decode('utf-8')".

Maybe it'll help if you try this (untested!):
Code: [Select]
searchText = scribus.getAllText(textbox).decode('utf-8')
textLength = len(searchText)
...
# use decode('utf-8') also for "keyword"

I'm no Scribus scripting expert, but maybe this shows you a direction to go...

Regards,
Andreas

Michihalo

  • Newbie
  • *
  • Posts: 6
  • Karma: 0
  • Scribus User
Re: Syntax Highlighting - German Umlaute problem
« Reply #5 on: March 31, 2019, 10:57:47 pm »
Thanks amair,

That did the trick!!
Using the stable version 1.4.8 was not possible, because there where some functions I wanted to have. But using version 1.5.4 it all works now.
Possible scenarios where this doesn't work, I immagine, is, when someone uses double and single quotes right after another. But this is something i can handle manually.

The script also creates Character Styles for the keywords it finds (if they are not already created) so you can edit them again later.

Run SyntaxHighlighter in Scribus. The second python file only contains two arrays, one with the keywords and one with start and end characters for comments. So it should be possible for everyone, to just replace these two with other keywords, to use it with whichever language should be syntax highlighted.

Feel free to use, hope this helps someone
cheers
-Mike


a.l.e

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1818
  • Karma: 26
    • the graphics lab
Re: Syntax Highlighting - German Umlaute problem
« Reply #6 on: April 01, 2019, 10:32:11 am »
nice!

now, i'm really tempted to convert this in a script that is really able to parse  "any" code... but i fear that it would be a huge thing...

a.l.e

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1818
  • Karma: 26
    • the graphics lab
Re: Syntax Highlighting - German Umlaute problem
« Reply #7 on: April 03, 2019, 06:14:33 pm »
... i'm giving pygments a try... the first bits seem to be promising, even if pygments has not been built for that...

a.l.e

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1818
  • Karma: 26
    • the graphics lab
Re: Syntax Highlighting - German Umlaute problem
« Reply #8 on: April 04, 2019, 02:34:16 pm »
wow!

i have a first working implementation:

https://github.com/aoloe/scribus-script-repository/tree/master/syntax-highlight

there is still lot to be done, but i could successfully highlight the two following python snippets:

Code: [Select]
print("Hello World")
and

Code: [Select]
for i in range(3):
    print('Hello World')

using pygments allows to correctly parse the code and brings the support for many languages.

there is still lot to be done and the scribus scripter API lacks some of the features needed to make a good script. but the patches to be done are probably mostly trivial.

If somebody wants to help, there are several things that people can contribute, even for people with very basic coding skills (like the mapping of the token types with scribus styles: there are dozens of token types, and we don't want to have that many styles being created!).

if you want to try the script, you might also want to download the .sla file attached: it has styles defined for the token types in the code that is in the file itself.

also, if you want the script to create the styles, you will need a very recent build (from last night, at the time of writing...), since there were (and still are) a few bugs in createCharStyle().

Michihalo

  • Newbie
  • *
  • Posts: 6
  • Karma: 0
  • Scribus User
Re: Syntax Highlighting - German Umlaute problem
« Reply #9 on: April 05, 2019, 11:52:06 am »
I now used my script twice for the last documentations and it worked perfectly for my needs (I put a sample page of my documentation in the attachment, just to be sure...)

This might be a stupid question, but what would be the extra gain using pygments?



a.l.e

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1818
  • Karma: 26
    • the graphics lab
Re: Syntax Highlighting - German Umlaute problem
« Reply #10 on: April 05, 2019, 01:04:11 pm »
first: you already did a very good job, the output you're showing looks good and you don't need to do otherwise if the output meets your needs!

why doing it with pygments?

on the one side, pygments is parsing your code and detecting tokens, not just doing a pattern matching.

this should

- better support edge cases (like using keywords in a context where they are not keywords (strings, comments; you script seems to get that right for pascal, but it's not sure it will work the same for every language)
- correctly support the whole syntax of a given language.

on the other side, pygments already has lexers for dozens of languages, so we can use the script for any language. we don't have to recreate the syntax files.


generally speaking, as far as i know, pattern matching is a good idea for a simple solution, but if one wants a more general -- long term -- script, correctly parsing the text is mostly a better solution.

personally, i will now stop working on "my" syntax-highlighter: i have no need for it right now.
i consider it a good proof concept and i will continue the work as soon as i have to put code in a scribus file or somebody asks for features and provides samples to be parsed.
(this cards will be a good candidate: https://github.com/CoderDojoZH/resources/blob/master/cards-python/mu-pgzero/check-collision/check-collision-deCH-a6.pdf : - )

Michihalo

  • Newbie
  • *
  • Posts: 6
  • Karma: 0
  • Scribus User
Re: Syntax Highlighting - German Umlaute problem
« Reply #11 on: April 05, 2019, 04:05:10 pm »
Okay, i see....

Right now, the simple script is sufficient for me. We usually do not have more than a few hundred lines of code, so it is not that hard to keep track of things at the moment.
I see why one would want to use pygments though. (I actually just found such a falsely highlighted edge case as you were describing in one of my documentations.)

I don't know pygments, and I am only learning how to code at the moment, but i would love to help creating a general syntax highlighting Script, since I really think, its a nice feature to have in scribus. (especially since we are starting to learn other programming languages as well next semester) ;)

Long story short, I would love to help ;)

a.l.e

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1818
  • Karma: 26
    • the graphics lab
Re: Syntax Highlighting - German Umlaute problem
« Reply #12 on: April 05, 2019, 07:02:36 pm »
you're very welcome!

i can think of three tasks that can be done next:

- matching the token types with styles in scribus (there are dozen of token types and we don't want to create that many of styles)
- eventually, finding a good way for the "user" to provide custom matchings (a json file? called with the same name as the python script, with a json ending and placed next to the .sla file?
- applying to the scribus style, the bold and underline defined in the token type style.

the first task is a rather boring one, while the other two should not be too difficult and still be a bit more interesting.

imo, the best way to collaborate is to work with git... if you don't know yet about it, it is worth to learn... if you're currently learning programming.
if you need help there, just aks!

i guess that the first steps will be:

- getting the github repository cloned to your computer
- reading the code and i've written and understand how it works...
  most of it is rather simple python code, but there are a few tricky things, that you might not know about yet. as an example:
  - `l = [i * 2 for i in range(0, 6, 2)]` is called list comprehension and creates a list l with the elements [0, 4, 12]
    (https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions)
  - the operators * and ** expand list and dictionaries to single elements that can be passed as arguments to functions.
    (https://docs.python.org/3/tutorial/controlflow.html#unpacking-argument-lists)

if you want, pick a task and i will be glad to help you. if you  need help, of course!

a.l.e

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1818
  • Karma: 26
    • the graphics lab
Re: Syntax Highlighting - German Umlaute problem
« Reply #13 on: April 09, 2019, 03:58:53 pm »
a short update: i will do a presentation about creating teaching material on mai 10.

until then i will need to get the script to the point where it correctly highlights my pygame zero samples...

 

palmate