As far as I know there are a few things affecting the file size to a large extent:
1. Embedding fonts. Complete fonts are big.
This page containts some good explanations.
https://www.evermap.com/PDFFileSize.aspAlso rememeber that Scribus will embed all fonts that are defined in a style in the document, even if there is no text in the final PDF using that font. So if you have defined many styles, with different fonts, but do not use them. They will still contribute to the file size.
A very common issue is that the default font for text frames is embedded, even though it is not used by any text.
2. Scribus' goal is good print quality, which means it places text with high precision. While another application may "place the cursor and then output a full line of text", Scribus more or less places the cursor for each word or even each glyph (to make sure the glyphs end up exactly where they are supposed to). This means that the commands to move the cursor will take up a a lot of space, sometimes even more space than the actual text.
While large images are often a problem with PDFs created with other programs, for Scribus created documents the images are often not the cause of large file size.