PDFs greater than 4Gb 'need repair' in Adobe Reader, and crash Foxit outright

[kiwi_steve]

Hi all, I have just spent a frustrating week back and forth with my printers for a book I have written in Scribus 1.5.8 over the last 4 years (it's worth noting I tried upgrading to Scribus 1.6 a while back, but it crashes every time I try opening the 1.5.8 file... so I have stuck with the older version).

The entire book is over 400 pages and has nearly 900 images, some very big.  With 'Compress Text and Vector Graphics' turned off during PDF creation, it comes out at 8.7Gb in size. 

This file will (unhappily) open in Acrobat Reader - I get a small window popup, titled 'Rebuild', and with the message 'This file is damaged but is being repaired'. After 20 seconds or so, the file opens, and looks fine.  The printers can't work with it, though.  This was terrifying for me for a while there, as I thought all my work was going to have to be redone in some expensive software that I don't know how to use, and don't want to have to buy or rent...

I eventually sent it in 4 smaller sections, which their printing system was able to reassemble, and so it looks like they can actually print it (I'll know more next week)... but it took some figuring out on their and my part.

Since then, I have made some trial exports to try and work out what is happening (each build takes quite a while to run) - if I set 'Compress Text and Vector Graphics' on, with Image Quality compression to None, and Maximum Image Resolution to 220DPI, the file size comes in at 3.7Gb, and the PDF will open just fine.  If I set the max DPI to 240DPI, the file size in 4.3Gb, and it throws the Rebuild warning.

My takeaway from this (at the moment, I am still experimenting) is that 4Gb, or thereabouts, is a limit.  I'm old enough to remember FAT32 (and possibly plain old FAT before that, but I'm not prepared to confirm that  ) and the 4Gb file size limit associated with it... is this possibly something related?

Anyway, I would love to know if there is something about large PDFs created in Scribus that is causing this problem... I've tried searching for that Rebuild error message relating to Acrobat, but it gets drowned out by the more common failed to open, corrupted PDF error and the one link I did find, really didn't tell me anything useful.  I was also unable to find anything related to large PDFs in Scribus, other than a lot of people complaining about the PDF file size that Scribus creates (I don't really care about the size, I just want the printer to be able to take the PDF and turn it into a nice shiny book).

Has anyone struck this, or better - does anyone know why this is happening - or does anyone have a sensible workaround other than to limit the DPI during export (it's essentially a large-format picture book, so I don't want to reduce the quality or DPI of the images if I can avoid it).

Cheers, Steve

[AdmFubar]

are you using  thee 32 bit? or 64 bit version if scribus?

[kiwi_steve]

are you using  thee 32 bit? or 64 bit version if scribus?

Hi, it's the 64-bit version:

Scribus 1.5.8
23 Jan 2022
Build ID C--T-*-C1.16.0-Windows-64bit

Cheers, Steve

[AdmFubar]

thought if you were using a 32 bit version there might be an odd bug with 4gb files.

Next is to check the images. make sure they are error free.
I'd also run a test on your system's ram, memtest86+ will give it a thorough exercising.

[kiwi_steve]

Yep, I saw where you were going with that... and TBH I wasn't 100% sure which I was using until I checked... I normally go with the 64-bit option when it's available though.

I'll try the RAM test out of interest, but I've compiled the PDF on my home development PC, and my work laptop with exactly the same results. 

Re: the images - there are nearly 900 of them, so I don't even know where to start with that.

But if I split the file into 3 smaller PDFs (with no compression / DPI limits) less than 4Gb each, then each opens fine... so it seems to be something to do with the creation of a PDF crossing that 4Gb boundary rather than something inside the file itself (unless it's a combination of the two).

It's a very interesting problem... I may need to file a bug report.

Cheers, Steve

[AdmFubar]

next .. what are your machine specs? desktop, laptop? beefy enough power supply?
May need to break up the book into chapters. and check for this issue.

[kiwi_steve]

As I said, I tried this on my dev PC (i5 gen 10, 16Gb RAM, 650W PSU etc... grunty enough) and on my work laptop (Ryzen 7 6800, 16Gb RAM)... both with identical results.  I don't think it's a hardware issue.

I've broken the file up, and it's fine, as long as the resultant PDF is below 4Gb.  Something about crossing that threshold is causing an issue.

I might try making some large images and then making a new 'book' for them, and try to replicate the problem outside of this project.  I'll also try Scribus 1.6 (which I can't use for this one because I started in 1.5.8, and if I try to open it in 1.6 it crashes every time.  I might try a build of 1.7 maybe...

For now though, fortunately, the printers were able to assemble the book from the smaller files I sent through, and they are printing it right now... so I should get to see 4 years of work in print in the next few days, which is pretty exciting.

And on that note, I want to be clear that I am definitely not complaining about Scribus - I love it... it's been great for this work, even with its quirks (I am not a fan of the spell checking, and hyphenation never worked properly).  I would love to know what is causing this problem, but it is definitely not a deal-breaker for me... I am using it to write volume 2 of the book I've just finished.

Cheers, Steve

[AdmFubar]

I might try making some large images and then making a new 'book' for them, and try to replicate the problem outside of this project.  I'll also try Scribus 1.6 (which I can't use for this one because I started in 1.5.8, and if I try to open it in 1.6 it crashes every time.  I might try a build of 1.7 maybe...

something is broken in that file, it should load into 1.6 without a hitch.
have you tried a print preview in scribus?

[kiwi_steve]

I have tried looking through the print preview - I can view the book cover-to-cover with no problems in the viewer.

Definitely crashes in 1.6, although I will give it another go just for fun.

The printers managed to print an approval copy for me after I split the file down into smaller sections... and that has been signed off, and it's now in the final print run, which is exciting.

Cheers, Steve