Proposal for new SLA file structure

[tim_occ]

Hello everyone,

I know that this is a sensitive topic. Nevertheless, I believe that the SLA file needs some improvements. Among other things, it still contains German attribute names for historical reasons. I also noticed that the file structure is "too flat". When the file is read in, all lines are read individually and checked for corresponding tags. This means that an unnecessary number of comparisons have to be carried out, which is particularly slow for large files. In addition, the efficiency advantages of the XML format are not used, namely that information can be read quickly via blocks and processed further in a targeted manner. In newer implementations, these blocks already exist, which is good.

I therefore suggest that most of the old tags be grouped into new parent tags.

An example using color tags. The same applies to styles and many other tags.
Previously:

Code Select Expand

<COLOR NAME="Black" SPACE="CMYK" C="0" M="0" Y="0" K="100"/>


Suggestion:

Code Select Expand

<Colors>
  <COLOR NAME="Black" SPACE="CMYK" C="0" M="0" Y="0" K="100"/>
</Colors>

Grouping tags has the advantage that you don't have to go through the entire document line by line to extract all the colors. This is a big advantage when importing data from one document into another document.

As far as I can tell, the new file loader only needs to be able to read the new format, as the old formats will continue to work via the old file loader plugins. In addition, Craig has already built in a function that allows users to save a new document in an older format. So at this point, I don't see any major problems changing the file format for newer versions of Scribus.

See attached file for a proposal structure. It is not intended to be complete.

PS: It is a good chance to replace the German names too and unify the code style, like "COLOR", "MultiLine" or "useColor".

[a.l.e]

basically: a very good idea.

some input from my side:

• if we create a new file format, i would love to have it specified in a DTD (there are enough people who want to do "crazy" things with scribus files and are already parsing and modifying them).
• would it be possible to have default values for some (many) attributes? when i have a look at a .sla file, i see lot (but lot!) of values that have not been been set specifically for that document....
• the elements should have a bit less arguments (your idea of embedding tags into others seems good to me)
• in a second step, we might want to allow some of the content to be external (text, styles, colors, ...).
• the external files might be packaged inside of a zip or really be located on a different part of the disk (or the net)
• and if we're already at that, we might want to think that in the future it should be possible to allow several people to work at the same time on a specific file (locking of resources)
• ... and allow the file to be accessed on a server from different computers (even not at the same time).

we don't have to implement more feature that there are now in scribus. but if we do a change, i would love if we could make sure that the new .sla definition could make some future progress easier (or at least not being in the way).
if possible

[tim_occ]

Hi ale,

very good input.

You're right, I think that before we consider just "minor" changes, a new file format should first and foremost be somewhat future-proof.

The idea of collaborative work is fascinating. From my experience, the biggest problem is when 2 users edit the same file. We have the same problem in the development of Scribus and merge conflicts can occur.

I think that in order for collaborative work to be possible at all, we need to split the current file into many small files that can then be changed independently of each other.

As an example, a Sketch app file is structured similarly to what you suggested. It is a zip file with separate files. See here: https://developer.sketch.com/file-format/

We could take a similar approach. As a principle for a new file format, the structure should be split into blocks that are exclusively referenced.
These blocks could then either be stored in folders (wherever they are located) or as a "compact version", these blocks are stored in the SLA file, as is the case now.

Here is an illustration of how I imagine the structure.

Code Select Expand

<?xml version="1.0" encoding="utf-8"?>
<scribusUTF8New Version="2.0">
<index>
    <item type="colors" location="" />
    <item type="gradients" location="" />
    <item type="patterns" location="" />
    <item type="opticalMarginSets" location="" />
    <item type="hyphens" location="" />
    <item type="characterStyle" location="" />
    <item type="paragraphStyle" location="" />
    <item type="lineStyle" location="" />
    <item type="noteStyle" location="" />
    <item type="toc" location="" />
    <item type="indexes" location="" />
    <item type="sections" location="" />
    <item type="layers" location="" />
    <item type="masterPages" location="" />
    <item type="pages" location="" />
    <item type="docItemAttributes" location="" />
    <item type="pdf" location="" />
    <item type="printers" location="" />
    <item type="images" location="" />
    <item type="text" location="" />
</index>
<document>
    ...
</document>
</scribusUTF8New>

The index contains a list of the locations of the various assets. If an item is missing, Scribus would assume the information is within the file itself. For stored assets, relative or absolute paths would then be specified here. Storing files also allows you to work with a version control such as git or diff.

[tim_occ]

It looks like InDesign Page Layout Template Files (indt) take a similar approach. It is a binary file format, but most likely an archive file, as I have found references to embedded XML and xpacket files.

[MrB]

There's lots to work through if we want a comprehensive new file format. Not opposed to it.
However, 1.7.0 needs releasing, and I don't think 1.7.x should include such a new format. I'd rather work to deprecate 1.6.x, and stabilise 1.7.x to 1.8.0 (or whatever we call it, in reference to Ale's comments on versioning elsewhere). This helps us drop Qt 5. 1.7.x can have 1-2 more versions to ensure all the UI changes so far are fine. I want to release 1.6.3 and 1.7.0 this year still.

The file format needs to be comprehensive and contain everything to support transition between computers, operating systems and versions of Scribus. Therefore, I believe omitting default values is not possible as they may change between versions of the code.

It can be developed in parallel while other work continues, as long as the active load/save plugin is not this one. Easy to achieve and part of why I broke the old loaders into separate plugins many years ago.

Collaborative working is of interest. This was a plan about 20 years ago too, but the then developer didn't continue with it. He had written a server stub for Scribus that would enable clients to track changes (although it never got to a working stage). Unfortunately, there are parts of Scribus that would work well with this, eg sharing styles between running clients, but there would be a ton of work to get some of the most basic functions with some collaborative function. I think there's better things to focus on, eg tables.

Anyway, happy to discuss and observe...

[AdmFubar]

The file format needs to be comprehensive and contain everything to support transition between computers, operating systems and versions of Scribus. Therefore, I believe omitting default values is not possible as they may change between versions of the code.

Was about to mention the version differences. glad it was noted.

[tim_occ]

Craig, I agree with your arguments, including that Scribus needs improvements to basic features first, such as tables, vector path editing, master pages, and more.

The collaborative work aspect naturally adds more complexity to the topic, and a new file format can be given lower priority. It was just an idea I had in mind.

[MrB]

All the ideas are great. They just need ordering so they fit with releases etc.

I think they can be worked on simultaneously too.