Scribus in a workflow? (text issues)

[FairMiles]

Hi.

I layout a scientific publication with not-so-many articles per year. For context, scientific articles are structured texts written/edited by external authors in their word-processors (usually .doc or .docx), some figures and tables with their captions and lists of formatted references/bibliography. Articles fit within a fixed set of few sections with a rather spartan design. Another article: iterate. Three things are crucial:

• to keep a consistent design (master pages, styles)
• to layout ilustrations among text easily (frames), and
• to fully preserve (e.g., italics, subscripts, symbols) and layout (hyphenation, columns) text.

I've been hoping to move my complete workflow to open source since years. I've been working with Ubuntu Linux, Open/LibreOffice and Gimp since 2011, but I am still stuck building everything with PageMaker 6.5 (can you believe it? ) running in a WindowsXP guest (in a virtual machine). Now a small window of opportunity opened and I decided to finally make the move to Scribus!

However... 

After some weeks of reading and trying (forum, email list, Mantis tracker, articles, manuals, wiki) I still cannot find a way to get text into frames and styles without losing all the work that have gone into building it in the first place, i.e., italics, sub/superscripts or small caps! I cannot afford to not only apply styles and manually check hyphenation but also to recover every subscript or italic that went there. It is a bit dangerous too. Nor I can move every author to use some particular markup codes instead of the "normal way" (like it or not) which would facilitate some import script.

I tried getting text from .doc/.odt and also .html, which seems to better preserve markup, but it all dissapear anyway when a Scribus (paragraph) style is applied. I found so many bugs reported through time (e.g., #477 in 2004, #9989 in 2011, #10661 in 2012, #12800 in 2014, #14916 in 2017...) and even reasonable solutions to the problem (several subordinated character styles for a paragraph style created when importing, I think in the email list from 2004!!!) that I cannot get how it was not fixed yet, as if it was some kind of optional feature. [rant] I know: "proper" italics vs "false" italics, "semantics" vs "appeareance", whatever... But how can Scribus be a "mature DTP" if it is (still) unable to properly import (formatted) text, which is the backbone of desktop publishing? That should have been the first task to accomplish, particularly because the (standard) recommendation is to do the heavy work on text (correcting, formating) previously in a word processor! [/rant] Sorry: I think I built so much hope for Scribus in all this time that now I feel really dissapointed...

Back to the point. Anyone can provide any solution/idea? Have anyone been able to put Scribus into a desktop publishing workflow that has (previously formatted) text as main input?

Thank you (and sorry, again...)

[a.l.e]

the problem i see, is that people like you who have a real workflow

a/ do not get enough weight when setting the priorities for the scribus development
b/ do not put their money / energy into scribus to improve

there is not simple solution to this, but i agree that it's a shame that there is no documented / working way to get basic formatting from an editor into scribus.

personally, because of the type of documents i'm creating, i don't import formatted text into scribus and i have not much experience with it.
but for my old memories with pagemaker (have you already heard of it? : - )  a good way to solve the issue was to apply the paragraph and character styles in the editor before importing the text into the layout program.

i'm not 100% sure that will work well with scribus, but if you can try that out and check the multiple options available when importing, to see if it works or not then:

- you might find a workflow that works
- if none works, define one that should work.

as far as i can tell, it would be much easier to fix the importer (html or odt) to have a sane behavior rather than changing the way applying the styles in scribus currently works.

and yes, in my eyes the preferred way to import formatted text into scribus is odt, followed by html. we should take some effort that at least one of both works "correctly".

it would be nice if you could prepare a few simple odt documents with styles and document how they import into scribus if you want to match the odt styles with the .sla ones.
you could post the files and the results in here and i would for sure do my tests with them, too!

deal?

ciao
a.l.e

[FairMiles]

Thanks for your reply. And for the support.

Deal! I will. Not able to dedicate full-time, but I will search for those moments to advance neatly.

Things I noticed/read that made me guess that a bigger change in Scribus would be required (than just working with the import filter):
* the lack of "false" italics/bold (or buttons/shortcuts) to apply them once in Scribus [but sub/superscripts and small caps are available through the Properties window  ]
* (single) Character style defined with/for each Paragraph style [i.e., "we don't want "false fonts" because they generate problems later, but we do not allow associations of "true fonts" so they can work as font families"]
* no recognition of Character styles applied in .odt/.doc source [but they are recognized/replicated when importing .html  ]

I imagine Scribus may have implemented an "italics" button [and Ctrl+I shortcut] that instead of creating a "false font" would apply a subordinate character style within the same paragraph style, changing only the font variant [or refusing to do it if no proper variant is available/declared] in a similar way that dependent (paragraph) styles change when the parent ("based on") is modified. Same for bold. THEN the import filter can be fitted to recognize the source and apply the same procedure automagically. Same as Scribus should do when applying some paragraph style to text already in a Scribus frame (detect previous formatting => apply new style without "erasing" everything, plus an instruction/button to "erase all (previous) format" to apply when desired).

Anyway, I'll do my homework.

Chau 

[FairMiles]

OK, this is my report of explorations for a workflow to layout a scientific journal starting from (incorrectly) formatted word processor documents by external authors. Tested with Scribus 1.4.6 in Ubuntu Linux 16.04 (+ LibreOffice 5.1.6.2)

1 - Text document received as .doc or .docx with no (or uncontrolled) paragraph styles, a general font (Times New Roman) and direct formatting applied in the word processor (italics, superscripts, bold, small caps)

Workflow A1: Practical, Scribus-intensive (in MSWord+PageMaker style)

2 - Document opened/imported in LibreOffice Writer and saved as .odt. [example: MStest.odt]

3 - Template in Scribus with master pages, paragraph/character styles and text frames [example: Hornero artículo.sla]

4 - Text inserted into Scribus text frame (Get Text...; no overwrite or merge styles; styles imported with odt name appended)
   (a) Importer: OpenDocument Text Document
      -> All direct formatting is recognized, including italics and bold (i.e., using family fonts)
   (b) Importer: OpenOffice.org Writer Document
      -> All direct formatting is lost

5 - Scribus Paragraph styles applied
   (a) in main window (editing frame contents + options in Properties|Text)
      -> All direct formatting is lost: Character styles must be reapplied to italics and bold texts + Superscripts and small caps must be reapplied as direct formatting  [+ Character styles must be defined completely for each paragraph style (no partial/relative changes or hierarchy)]
   (b) in Story Editor (options in top bar)
      -> Direct formatting such as superscript and small caps is preserved BUT some characteristics of Paragraph Style (e.g., font) are not applied to text in italics or bold (?!) + Character styles cannot be applied directly nor searching and replacing.

Workflow A2: Scribus-intensive based on .html (in MSWord+PageMaker style)

2 - Document opened/imported in LibreOffice Writer and saved as .html [example: MStest.html]

3 - Template in Scribus with master pages, paragraph/character styles and text frames [example: Hornero artículo.sla]

4 - Text inserted into Scribus text frame (Get Text...)
   (a) Importer: HTML Document | Need to define replace fonts for html fonts, such as "bold" or "bold italic" (editable in File |Preferences|Fonts|Font substitutions or/and File|Document Setup|Fonts|Font substitutions)
      -> All direct formatting but Small Caps is recognized, including italics and bold (i.e., using substitution fonts as defined in Preferences)

5 - Scribus Paragraph styles applied
   (a) in main window (editing frame contents + options in Properties|Text)
      -> All direct formatting is lost [see above]
   (b) Paragraph styles applied in Story Editor (options in top bar)
      -> Direct formatting such as superscript is preserved, text in italics or bold remain as substituted when imported + Character styles cannot be applied directly nor searching and replacing.


[to be continued: Workflow B: Text styles applied in LibreOffice Writer]

[attachment deleted by admin]

[mnawij]

Hey a.l.e.

Just a quick aside, is there a way for Scribus to import an .xml file that saves the formatting since .sla file are saved in xml?  I know both Word and LibreOffice can write xml files... would it be possible to code the MS Word importer from LO and then write out an xml file from LO export?  I have not taken a coding class since 1993 so I might be a bit rusty on the logic possibilities.

FairMiles, I am 100% in agreement with you on this issue.  As someone who has been trying to move to complete Open Source apps for my business for a couple years now, this is one area that really needs to be updated.  This is one of a handful of issues that needs to be implemented in Scribus before I can switch away from Adobe apps for my print shop.  I get at least a dozen or more Word files to work with each month to convert to InDesign currently.

Regards,

mnawij

[FairMiles]

Part 2.

Workflow B1: LibreOffice Paragraph Styles + Scribus

2 - Document opened/imported in LibreOffice Writer, Paragraph Styles applied [keeping direct formatting for italics/bold] and saved as .odt. [example: MStest_B.odt]

3 - Template in Scribus with master pages, paragraph/character styles and text frames [example: Hornero artículo.sla]

4 - Text inserted into Scribus text frame (Get Text...; overwrite/merge styles; styles imported with odt name appended)
      -> Direct formatting is recognized, but some Paragraph Styles differ from those in .odt version (e.g., imported as "P1" and "P2" instead of "Título" and "Texto"); some characteristics are lost (Caps in "Título" when imported as "P1"); styles names do not import correctly with special characters ("EpÃgrafe" instead of "Epígrafe"; "Filiación" instead of "Filiación")
   
5 - If Scribus Paragraph styles are applied to correct Writer Styles not/badly imported, direct formatting is lost if done in main window (through Properties). It has to be done in Story editor. Some characteristics may not be applied to texts with direct formatting [see Workflow A]

Workflow B2: LibreOffice Paragraph Styles + Scribus, based on .html

2 - Document opened/imported in LibreOffice Writer, Paragraph Styles applied [keeping direct formatting for italics/bold] and saved as .html. [example: MStest_B.html]

3 - Template in Scribus with master pages, paragraph/character styles and text frames [example: Hornero artículo.sla]

4 - Text inserted into Scribus text frame (Get Text...)
      -> Direct formatting is recognized except Caps and Small Caps (some spaces were lost in text originally in SmallCaps), including italics and bold (i.e., using substitution fonts as defined in Preferences). However, everything imports as "Default Paragraph Style" (only "html_p" style is added to the list of Scribus styles)

5 - If Scribus Paragraph styles are applied to correct Writer Styles not/badly imported, direct formatting is lost if done in main window (through Properties). It has to be done in Story editor. Some characteristics may not be applied to texts with direct formatting [see Workflow A]

[to be continued: Workflow C: LibreOffice Paragraph & Character Styles + Scribus (no direct formatting)]

[attachment deleted by admin]

[FairMiles]

Part 3.

Workflow C1: LibreOffice Paragraph & Character Styles (no direct formatting) + Scribus

2 - Document opened/imported in LibreOffice Writer, direct formatting replaced by Character Styles with Find+Replace [e.g. (a) search all instances of Format=italic, (b) remove direct formatting for selection, (c) apply "Emphasis" character style], remove direct formatting for all text [does not remove applied character styles], Paragraph Styles applied, saved as .odt. [example: MStest_C.odt]

3 - Template in Scribus with master pages, paragraph/character styles and text frames [example: Hornero artículo.sla]

4 - Text inserted into Scribus text frame (Get Text...; styles imported with odt name appended)
      -> Character styles are lost + some Paragraph Styles differ from those in .odt version (e.g., imported as "P1" instead of "Título"); some characteristics lost (Caps in "Título" when imported as "P1"); styles names do not import correctly with special characters ("EpÃgrafe" instead of "Epígrafe"; "Filiación" instead of "Filiación")

5 -  Scribus Paragraph styles must be applied in Story editor [see Workflow A]

Workflow C2: LibreOffice Paragraph & Character Styles (no direct formatting) + Scribus, based on .html

2 - Document opened/imported in LibreOffice Writer, direct formatting replaced by Character Styles with Find+Replace [e.g. (a) search all instances of Format=italic, (b) remove direct formatting for selection, (c) apply "Emphasis" character style], remove direct formatting for all text [does not remove applied character styles], Paragraph Styles applied, saved as .html [example: MStest_C.html]

3 - Template in Scribus with master pages, paragraph/character styles and text frames [example: Hornero artículo.sla]

4 - Text inserted into Scribus text frame (Get Text...)
      -> All Paragraph and Character Styles from Writer are lost (only "HTML_p" style is added to the list of Scribus styles) but italics, bold and superscripts preserved (Caps and SmallCaps are lost, and some spaces were lost in text originally in SmallCaps).

5 -  Scribus Paragraph styles must be applied in Story editor [see Workflow A]

[to be continued: Workflow D: LibreOffice Template: do I need Scribus?]

- - -

[attachment deleted by admin]

[a.l.e]

hi fairmiles,

i'm following your write ups and -- in parallel -- i'm creating a set of files for reproducing the single behaviors:

- with detailed instructions on how to reproduce
- a description of the current results and
- a list of improvements that are necessary or nice to have

i'm starting with the html import, since it's a bit easier to create the demo files. but i plan to do the same files for odt, too.

i'll make the repository public as soon as i consider that the structure is definitive and i will then welcome contributions!

and concerning @mnavij concern about importing xml files: i would first get the html import straight and then see if there is a real need for some solution for custom xml...
what exactly do you want to do with xml wthat cannot be done with some python scripting (python can read xml and there is a scribus API for python) or with html?

ciao
a.l.e

[FairMiles]

Part 4 (last)

Workflow D: LibreOffice Template & Styles: do I need Scribus?

2 - [same as Workflow C] Document opened/imported in LibreOffice Writer, styles imported from Writer template (see below), direct formatting replaced by Character Styles with Find+Replace [e.g. (a) search all instances of Format=italic, (b) remove direct formatting for selection, (c) apply "Emphasis" character style], remove direct formatting for all text [does not remove applied character styles], Paragraph Styles applied, saved as .odt. [example: MStest_C.odt]

3 - Template in LibreOffice Writer with default page style + 2-column section for main text, frame styles to position typical illustrations (single-column width, full width, etc), paragraph/character styles, header and formatted dummy text [example: Hornero_Artículo.ott]

4 - Text copied+pasted to a new document based on template [New|Templates]. Imported images, tables into frames, referable semi-automatic captions, "relative" character styles for ad-hoc formatting (e.g. "Emphasis" to apply [only] italics to ANY paragraph styles), spell check and automatic (configurable) hyphenation [example: MStest_D.odt]. PDF export.
      -> Frames are not so easy to handle in a 2-column layout; their positioning and sizing is not as confortable (trustable?) as in a "true DTP" (afaik). Frames and sometimes colums and paragraphs need some fiddling with custom (not-styled) parameters. Still, everything seems to work.

5 - No Scribus (maybe for the cover?)

[attachment deleted by admin]

[FairMiles]

Some conclusions.

* If (paragraph+character) styles must/should be applied to text in a word processor (such as LO Writer) before importing it, then Scribus may become redundant, or even a complication (compare Workflows C and D in previous posts). In any case, if that is the proper workflow path according to Scribus, then it should at least be able to import correctly the formatted text. That is not the case, particularly for character styles.

* Though LO Writer can handle the design of publications (see http://designingwithlibreoffice.com/wp-content/uploads/2016/03/DWL_fullbook.pdf  and https://wiki.documentfoundation.org/images/0/0d/WG54-WriterGuideLO.pdf) I think it is (still?) not a proper DTP because of its limitations to handle (more complex) layouts: a full document made of independent+linked frames (à la Scribus) is probably a bad long-term idea. But neither is Scribus a proper DTP if it cannot easily handle (formatted) text or apply very relevant features for adequate typography (hyphenation, multilingual dictionaries, captions, text tables...). I guess the relative weight in a project between graphics+layout vs. text may heavily influence the pros/cons balance of the different approaches by these tools. [I guess Scribus may also tradeoff pros&cons against drawing tools such as Inkscape or LO Draw if the project is too graphics-heavy, but that discussion is for another thread]

* I'm no programmer nor I know the structure of Scribus, but I guess that in general it may benefit from trying to better integrate/complement with other (existing, popular, open source) tools and projects. From what I have explored, recognizing and applying subordinate/relative character styles (as apparently handled, though not clearly documented, by LO writer) and facilitating the application of paragraph styles may be first important steps to solve some of its current limitations.

I think the current status of Scribus makes it no good tool to include it within a workflow that depends on (partially) formatted imported text. Even more, some important details in the way styles are defined and applied within Scribus make it not particularly well fitted for works that heavily depend on text even when not already formatted.

I really hope Scribus can someday become the mature tool I expected when I started this exploration.

[dragonfly]

I read this thread with some interest since I have returned, after some time,  to explore Scribus fitting into a workflow/toolchain for my own purposes.  My workflow covers document and web content but the latter is optional.  I am on Ubuntu desktop.

The problem set out in workflows above is that formatting is lost when pre-authored documents are imported into Scribus.

It is encouraging to see that there is now the option of developing python scripts to extend Scribus functionality through the Scribus API.

Might I suggest another workflow to consider:

(a) Install pandoc.  https://pandoc.org/

(b) Suggest to multiple authors that they write their content in markdown (the lowest common denominator).

(c) Where there are already legacy pre-written documents in different formats (*.docx, *.odt etc), consider converting them back to markdown so that they can be imported into Scribus (see idea on markdown script below).

(d) Develop a Scribus script which imports from markdown into Scribus frames allowing markdown styles to be faithfully reproduced in Scribus frames (including styling).

(e) This workflow ensures that standards can be set for importing documents into Scribus.

I did some more searching and found this workflow.  This mentions pandoc and Scribus.

https://gitlab.com/DigitalPublishingToolkit/Hybrid-Publishing-Resources/wikis/home

My next step will be to concentrate on a Scribus script which can import markdown into Scribus frames.

Then I will explore another script to convert Scribus export SVG into HTML (for web publishing option)

[FairMiles]

Hi dragonfly,

Might I suggest another workflow to consider:

It's great that you are considering the inclusion of Scribus within a (semi-automatic) workflow. Maybe the addition of filters (to import, to transform) it's the proper solution for Scribus, I don't know. As a general user, however, I see your proposed workflow a bit overkill. One first problem is:

(b) Suggest to multiple authors that they write their content in markdown (the lowest common denominator).

That can be a (very) difficult one depending on the (average) computer/digital literacy of your community of authors. Just as an example, I can't get some/many authors to provide adecuately formatted bitmap figures even after (re)sending detailed instructions... And then

(c) Where there are already legacy pre-written documents in different formats (*.docx, *.odt etc), consider converting them back to markdown so that they can be imported into Scribus (see idea on markdown script below).

That is the main (time-consuming) problem for an efficient workflow. Unless that is automatic and non-dependent on differently configured documents, it becomes very similar to my Workflows C (with Scribus) and D (without) above.

From the Scribus point of view (and that is why I started this thread on this forum) the problem then is that once you get a (very) automated procedure, Scribus becomes innecesary, superfluous or even a complication. If you can't use it to apply uniform format/styles (because the heavy load you do it automatically or in a different program, or because you lose too much previous work when you do it) then it is not very adequate for a text-based workflow based on pre-authored (non-uniform) documents.

It doesn't seem like an impossible task: you can see Writer providing something very similar. I think it's more a problem of development attention than programming dificulty, as a.l.e. suggested. Just as a reference, consider that I do apply paragraph styles to formatted imported text (as .rtf) from a toolbar or a right-click menu without losing character properties (italics, bold, etc.) in an Adobe PageMaker version from 1996...