3 Commits

Author	SHA1	Message	Date
Nick Alcock	b5d3790c66	libctf: consecutive ctf_id_t assignment ... This change modifies type ID assignment in CTF so that it works like BTF: rather than flipping the high bit on for types in child dicts, types ascend directly from IDs in the parent to IDs in the child, without interruption (so type 0x4 in the parent is immediately followed by 0x5 in all children). Doing this while retaining useful semantics for modification of parents is challenging. By definition, child type IDs are not known until the parent is written out, but we don't want to find ourselves constrained to adding types to the parent in one go, followed by all child types: that would make the deduplicator a nightmare and would frankly make the entire ctf_add*() interface next to useless: all existing clients that add types at all add types to both parents and children without regard for ordering, and breaking that would probably necessitate redesigning all of them. So we have to be a litle cleverer. We approach this the same way as we approach strings in the recent refs rework: if a parent has children attached (or has ever had them attached since it was created or last read in), any new types created in the parent are assigned provisional IDs starting at the very top of the type space and working down. (Their indexes in the internal libctf arrays remain unchanged, so we don't suddenly need multigigabyte indexes!). At writeout (preserialization) time, we traverse the type table (and all other table containing type IDs) and assign refs to every type ID in exactly the same way we assign refs to every string offset (just a different set of refs -- we don't want to update type IDs with string offset values!). For a parent dict with children, these refs are real entities in memory: pointers to the memory locations where type IDs are stored, tracked in the DTD of each type. As we traverse the type table, we assign real IDs to each type (by simple incrementation), storing those IDs in a new dtd_final_type field in the DTD for each type. Once the type table and all other tables containing type IDs are fully traversed, we update all the refs and overwrite the IDs currently residing in each with the final IDs for each type. That fixes up IDs in the parent dict itself (including forward references in structs and the like: that's why the ref updates only happen at the end); but what about child dicts' references, both to parent types and to their own? We add armouring to enforce that parent dicts are always serialized before their children (which ctf-link.c already does, because it's a precondition for strtab deduplication), and then arrange that when a ref is added to a type whose ID has been assigned (has a dtd_final_type), we just immediately do an update rather than storing a ref for later updating. Since the parent is already serialized, all parent type IDs have a dtd_final_type by this point, and all parent IDs in the children are properly updated. The child types can now be renumbered now we now the number of types in the parent, and their refs updated identically to what was just done with the parent. One wrinkle: before the child refs are updated, while we are working over the child's type section, the type IDs in the child start from 1 (or something like that), which might seem to overlap the parent IDs. But this is not the case: when you serialize the parent, the IDs written out to disk are changed, but the only change to the representation in memory is that we remember a dtd_final_type for each type (and use it to update all the child type refs): its ID in memory is the same as it always was, a nonoverlapping provisional ID higher than any other valid ID. We enforce all of this by asserting that when you add a ref to a type, the memory location that is modified must be in the buffer being serialized: the code will not let you accidentally modify the actual DTDs in memory. We track the number of types in the parent in a new CTFv4 (not BTF) header field (the dumper is updated): we will also use this to open CTFv3 child dicts without change by simply declaring for them that the parent dict has 2^31 types in it (or 2^15, for v2 and below): the IDs in the children then naturally come out right with no other changes needed. (Right now, opening CTFv3 child dicts requires extra compatibility code that has not been written, but that code will no longer need to worry about type ID differences.) Various things are newly forbidden: - you cannot ctf_import() a child into a parent if you already ctf_add()ed types to the child, because all its IDs would change (and since you already cannot ctf_add() types to a child that hasn't had its parent imported, this in practice means only that ctf_create() must be followed immediately by a ctf_import() if this is a new child, which all sane clients were doing anyway). - You cannot import a child into a parent which has the wrong number of (non-provisional) types, again because all its IDs would be wrong: because parents only add types in the provisional space if children are attached to it, this would break the not unknown case of opening an archive, adding types to the parent, and only then importing children into it, so we add a special case: archive members which are not children in an archive with more than one member always pretend to have at least one child, so type additions in them are always provisional even before you ctf_import anything. In practice, this does exactly what we want, since all archives so far are created by the linker and have one parent and N children of that parent. Because this introduces huge gaps between index and type ID for provisional types, some extra assertions are added to ensure that the internal ctf_type_to_index() is only ever called on types in the current dict (never a parent dict): before now, this was just taken on trust, and it was often wrong (which at best led to wrong results, as wrong array indexes were used, and at worst to a buffer overflow). When hash debugging is on (suggesting that the user doesn't mind expensive checks), every ctf_type_to_index() triggers a ctf_index_to_type() to make sure that the operations are proper inverses. Lots and lots of tests are added to verify that assignment works and that updating of every type kind works fine -- existing tests suffice for type IDs in the variable and symtypetab sections. The ld-ctf tests get a bunch of largely display-based updates: various tests refer to 0x8... type IDs, which no longer exist, and because the IDs are shorter all the spacing and alignment has changed.	2025-03-16 15:25:27 +00:00
Nick Alcock	a480362d88	libctf: string: refs rework ... This commit moves provisional (not-yet-serialized) string refs towards the scheme to be used for CTF IDs in the future. In particular - provisional string offsets now count downwards from just under the external string offset space (all bits on but the high bit). This makes it possible to detect an overflowing strtab, and also makes it trivial to determine whether any string offset (ref) updates were missed -- where before we might get a slightly corrupted or incorrect string, we now get a huge high strtab offset corresponding to no string, and an error is emitted at read time. - refs are emitted at serialization time during the pass through the types. They are strictly associated with the newly-written-out buffer: the existing opened CTF dict is not changed, though it does still get the new strtab so that new refs to the same string can just refer directly to it. The provisional strtab hash table that contains these strings is not deleted after serialization (because we might serialize again): instead, we keep track in the parent of the lowest-yet-used ("latest") provisional strtab offset, and any strtab offset above that, but not external (high-bit-on) is considered provisional. This is sort-of-enforced by moving most of the ref-addition function declarations (including ctf_str_add_ref) to a new ctf-ref.h, which is not included by ctf-create.c or ctf-open.c. - because we don't add refs when adding types, we don't need to handle the case where we add things to expanding vlens (enums, struct members) and have to realloc() them. So the entire painful movable refs system can just be deleted, along with the ability to remove refs piecemeal at all (purging all of them is still possible). Strings added during type addition are added via ctf_str_add(), which adds no refs: the strings are picked up at serialization time and refs to their final, serialized resting place added. The DTDs never have any refs in them, and their provisional strtab offsets are never updated by the ref system. This caused several bugs to fall out of the earlier work and get fixed. In particular, attempts to look up a string in a child dict now search the parent's provisional strtab too: we add some extra special casing for the null string so we don't need to worry about deduplication moving it somewhere other than offset zero. Finally, the optimization that removes an unreferenced synthetic external strtab (the record of the strings the linker has told us about, kept around internally for lookup during late serialization) is faulty: references to a strtab entry will only produce CTF-level refs if their value might change, and an external string's offset won't change, so it produces no refs: worse yet, even if we did get a ref (say, if the string was originally believed to be internal and only later were we told that the linker knew about it too), when we serialize a strtab, all its refs are dropped (since they've been updated and can no longer change); so if we serialized it a second time, its synthetic external strtab would be considered empty and dropped, even though the same external strings as before still exist, referencing it. We must keep the synthetic external strtab around as long as external strings exist that reference it, i.e. for the life of the dict. One benefit of all this: now we're emitting provisional string offsets at a really high value, it's out of the way of the consecutive, deduplicated string offsets in child dicts. So we can drop the constraint that you cannot add strings to a dict with children, which allows us to add types freely to parent dicts again. What you can't do is write that dict out again: when we serialize, we currently update the dict being serialized with the updated strtabs: when you write a dict out, its provisional strings become real strings, and suddenly the offsets would overlap once more. But opening a dict and its children, adding to it, and then writing it out again is rare indeed, and we have a workaround: anyone wanting to do this can just use ctf_link instead.	2025-02-28 15:13:24 +00:00
Nick Alcock	d7474051e8	libctf: propagate errors from parents correctly ... CTF dicts have per-dict errno values: as with other errno values these are set on error and left unchanged on success. This means that all errors *must* set the CTF errno: if a call leaves it unchanged, the caller is apt to find a previous, lingering error and misinterpret it as the real error. There are many places in libctf where we carry out operations on parent dicts as a result of carrying out other user-requested operations on child dicts (e.g. looking up information on a pointer to a type will look up the type as well: the pointer might well be in a child and the type it's a pointer to in the parent). Those operations on the parent might fail; if they do, the error must be correctly reflected on the child that the user-visible operation was carried out on. In many places this was not happening. So, audit and fix all those places. Add tests for as many of those cases as possible so they don't regress. libctf/ * ctf-create.c (ctf_add_slice): Use the original dict. * ctf-lookup.c (ctf_lookup_variable): Propagate errors. (ctf_lookup_symbol_idx): Likewise. * ctf-types.c (ctf_member_next): Likewise. (ctf_type_resolve_unsliced): Likewise. (ctf_type_aname): Likewise. (ctf_member_info): Likewise. (ctf_type_rvisit): Likewise. (ctf_func_type_info): Set the error on the right dict. (ctf_type_encoding): Use the original dict. * testsuite/libctf-writable/error-propagation.*: New test.	2023-04-08 16:07:17 +01:00