libctf, serialize: preparatory steps

The new serializer is quite a lot more customizable than the old, because it
can write out BTF as well as CTF: you can ask to write out BTF or fail,
write out CTF if required to avoid information loss, otherwise BTF, or
always write out CTF.

Callers often need to find out whether a dict could be written out as BTF
before deciding how to write it out (because a dict can never be written out
as BTF if it is compressed, a caller might well want to ask if there is
anything else that prevents BTF writeout -- say, slices, conflicting types,
or CTF_K_BIG -- before deciding whether to compress it).  GNU ld will do
this whenever it is passed only BTF sections on the input.

Figuring out whether a dict can be written out as BTF is quite expensive: we
have to traverse all the types and check them, including every member of
every struct.  So we'd rather do that work only once.  This means making a
lot of state once private to ctf_preserialize public enough that another
function can initialize it; and since the whole API is available after
calling this function and before serializing, we should probably arrange
that if we do things we know will invalidate the results of all this
checking, we are forced to do it again.

This commit does that, moving all the existing serialization state into a
new ctf_serialize_t and adding to it.  Several functions grow force_ctf
arguments that allow the caller to force CTF emission even if the type
section looks BTFish: the writeout code and archive creation use this to
force CTF emission if we are compressing, and archive creation uses it
to force CTF emission if a CTF multi-member archive is in use, because
BTF doesn't support archives at all so there's no point maintaining
BTF compatibility in that case.  The ctf_write* functions gain support for
writing out BTF headers as well as CTF, depending on whether what was
ultimately written out was actually BTF or not.

Even more than most commits in this series, there is no way this is
going to compile right now: we're in the middle of a major transition,
completed in the next few commits.

libctf/ctf-impl.h

@@ -356,6 +356,19 @@ typedef struct ctf_dedup
   ctf_dict_t *cd_output;
 } ctf_dedup_t;
 
+/* Serializer state.
+
+   This connects the various stages of serialization (ctf_link_output_is_btf,
+   ctf_preserialize, ctf_serialize).  */
+
+typedef struct ctf_serialize
+{
+  int cs_initialized;		/* If 0, needs reinitialization.  */
+  unsigned char *cs_buf;	/* CTF buffer in mid-serialization.  */
+  size_t cs_buf_size;		/* Length of that buffer.  */
+  int cs_is_btf;
+} ctf_serialize_t;
+
 /* The ctf_dict is the structure used to represent a CTF dictionary to library
    clients, who see it only as an opaque pointer.  Modifications can therefore
    be made freely to this structure without regard to client versioning.  The
@@ -402,9 +415,7 @@ struct ctf_dict
   unsigned char *ctf_buf;	  /* Uncompressed CTF data buffer, including
 				     CTFv4 header portion.  */
   size_t ctf_size;		  /* Size of CTF header + uncompressed data.  */
-  unsigned char *ctf_serializing_buf; /* CTF buffer in mid-serialization.  */
-  size_t ctf_serializing_buf_size; /* Length of that buffer.  */
-  size_t ctf_serializing_nvars;	  /* Number of those vars.  */
+  ctf_serialize_t ctf_serialize;  /* State internal to ctf-serialize.c.  */
   uint32_t *ctf_sxlate;		  /* Translation table for unindexed symtypetab
 				     entries.  */
   unsigned long ctf_nsyms;	  /* Number of entries in symtab xlate table.  */
@@ -782,7 +793,7 @@ extern void ctf_str_purge_refs (ctf_dict_t *fp);
 extern void ctf_str_rollback (ctf_dict_t *, ctf_snapshot_id_t);
 extern const ctf_strs_writable_t *ctf_str_write_strtab (ctf_dict_t *);
 
-extern int ctf_preserialize (ctf_dict_t *fp);
+extern int ctf_preserialize (ctf_dict_t *fp, int force_ctf);
 extern void ctf_depreserialize (ctf_dict_t *fp);
 
 extern struct ctf_archive_internal *