.. a major configuration cleanup
... major enhancement of automake support.
... and it contains a *major* breakthough:
Automake support for libchip and libmisc *LEAF* directories.
To implement this I have used several nasty tricks
* The basical trick is to wrap an old Makefile.in's contents into a
Makefile.am and still continue to use (i.e include) the old
*.cfg files.
* Replaced each INSTALL_IF_CHANGE and INSTALL_VARIANT with make
dependencies
* Add a gnu-make ifdef AUTOMAKE to main.cfg to avoid conflicts between
automake and RTEMS make rules
* Replaced each install:: and preinstall:: rule with make dependencies
* Replaced SUB_DIRS with SUBDIRS in all Makefile.ins (Automake
convention)
* Removed each manually added autoconf substitution which automake
performs automatically.
This is not yet full automake support, because using the temporary
installation directory, preinstallation in general and building variants
are in contradiction to automake's basic working principles ...
... the new Makefile.ams work still somewhat clumsy
... nevertheless they work (quite well).
WARNING:
At first glance this patch is small, but
* it affects the whole configuration system.
* it opens the road to introducing automake to all Makefile.ins
currently not being under automake control.
JOEL> Does this remove or add any files?
Both, all Makefile.ins below libchip and libmisc get replaced with
Makefile.ams.
This patch addresses a few configuration issues and removes some
duplicate configuration files.
Background:
* some files used in AC_INIT were not unique enough and can lead to
problems if a user plays with configure scripts.
* the Makefile templates are independent of the target/cpu and bsp, so
the bsp dependent versions (c/src/make/Templates) are removed.
To apply:
patch -p1 < rtems-rc-19990820-3.diff
rm -f c/src/make/Templates
./autogen
ATTENTION: This patch removes several files and one directory :)
This patch is an addition to "The big-patch"
CHANGES:
* FIX: c/Makefile.am: bogus comment which changed the behavior of
c/Makefile.am removed
* FIX: make/custom/ts_i386ex.cfg did not set HAS_NETWORKING correctly
(Me thinks it might have been me who added this bogus setting :-).
* NEW: removing make targets get, protos, debug_install, profile_install
* NEW: replacing clobber with distclean
* NEW: Reimplement distclean and clean as reverse depth first make
targets (adaptation to automake's behavior)
* NEW: removing RCS_CLEAN from make distclean (tools/build/rcs_clean is
still in - remove it?)
* NEW: "$(RM) Makefile" added to make distclean (adaptation to
automake's behavior)
* NEW: "$(RM) config.cache config.log" to CLOBBER_ADDITIONS in
[lib|exec|tests]/Makefile.in (adaptation to automake's behavior)
* NEW: "$(CLEAN_PROTOS)" removed (Not used anywhere)
* NEW: binpatch.c moved from i386 bsp tools to tools/build (AFAIS,
binpatch is not specific to the pc386 BSP at all)
* NEW: AC_EXEEXT added to all configure scripts which contain AC_PROG_CC
(Cygwin support)
* NEW/Experimental: An experimental implementation of temporary
installation tree support in libbsp/i386/pc386/tools/Makefile.am, based
on dependency tracking with make, instead of applying INSTALL_CHANGE.
REMARK:
* This patch is small in size, but changes the behavior of "make
clean|distclean|clobber" basically.
* This patch does not alter building/compiling RTEMS, ie. there should
be no need to rerun all "make all" building tests.
KNOWN BUGS:
* make RTEMS_BSP="..." distclean in c/ runs "make distclean" in BSPs
subdirectories passed through RTEMS_BSP and in "c/." only, but does not
descend into other BSP subdirectories previously configured with
different settings of make RTEMS_BSP="...".
=> Workaround: always use the same setting of RTEMS_BSP when working
inside the build-tree.
* "make [distclean|clean]" do not clean subdirectories, which have been
configured at configuration time, but which are not used due to
make-time configuration (e.g. macros/networking/rdgb subdirectories).
This will problem will vanish by itself when migrating from make-time to
configuration-time configuration
APPLYING THE PATCH
mv c/src/lib/libbsp/i386/pc386/tools/binpatch.c tools/build
patch -p1 < rtems-rc-19990709-2.diff
autogen
to move RTEMS more to automake/autoconf and GNU compliance.
Finally, here they are: the "big-patch" patches - merged into one big
patch (~1.5MB).
Sorry for the delay, but testing took much more time than I had expected
- esp. reworking the acpolish script triggered many more tiny issues
than I had expected (cf. below).
At least, now you've got something to spend your weekend with :-.
WARNINGS:
* I've gone a little (??) further than I had announced before.
* Several directories have been moved.
* Several files have been added and removed
* I have tested it with many BSPs/CPUs and a variety of permutiations of
configuration flags, but not with all.
* Most parts of the patch are automatically generated, however there are
many tiny manual modifications.
APPLYING THE PATCH:
./autogen -c
mkdir tools
mv c/src/exec/score/tools tools/cpu
mv c/build-tools tools/build
mv c/update-tools tools/update
patch -p1 -E < rtems-rc-19990709-0.diff
./autogen
If the patch doesn't apply to rtems-cvs, I would suggest that you should
try to apply it brute-force and then to run tools/update/rtems-polish.sh
-ac -am afterwards. A recursive diff between rtems-19990709 + patch and
rtems-cvs + patch then should report only a few dozen significant
changes to configuration files which need to be merged manually (IIRC, I
did not change any source files).
*** Attention: There are files to be removed, moved, copied and added
in/to CVS!
NEWS/CHANGES:
1. Configuration takes place in 3 stages: 1. per host (toplevel
configure script), 2. per target (c/configure), 3. per bsp
c/src/configure automatically triggered from ./configure and
c/Makefile.am.
2. Building of subdirectory c/ takes place in c/$(target_alias) for
cross-targets in c/ for native targets
3. Building of subdirectory c/src takes place in c/${target_alias}/<bsp>
for cross-targets, c/<bsp> for native targets
4. c/build-tools moved to tools/build
5. c/src/exec/score/cpu/tools moved to tools/cpu (=cpu-tools split out)
6. c/update-tools moved to tools/update
7. New subdirectory c/src/make, handles files from make/ on a per BSP
basis
8. Maintainer mode support: Ie. if configuring with
--enable-maintainer-mode disabled (the default), then tracking of many
dependencies will be disabled in Makefiles. Esp. many dependencies for
auto* generated files will be switched off in Makefiles. Ie. if not
using "--enable-maintainer-mode" many auto* generated files will not be
updated automatically, i.e. normal users should not be required to have
auto* tools anymore (untested).
9. Independent configuration scripts for / (toplevel), tools/build,
tools/cpu, tools/update, c/, c/src/, c/src/exec, c/src/lib, c/src/tests,
c/src/make
10. Automake support for all directories above and besides c/src
11. "preinstall" now is implemented as depth-first recursive make target
12. host compiled tools (exception bsp-tools) are accessed in location
in the build tree instead of inside the build-tree when building RTEMS.
13. RTEMS_ROOT and PROJECT_ROOT now point to directories inside the
build-tree - many tiny changes as consequence from this.
14. --with-cross-host support removed (offically announced obsolete by
cygnus)
15. Changing the order of building libraries below c/src/lib/
16. Former toplevel configure script broken into aclocal/*.m4 macros
17. Newlib now detected by configure macros, RTEMS_HAS_NEWLIB removed
from *cfg.
18. sptables.h now generated by autoconf
19. Rules for "mkinstalldirs temporary installation tree" moved from
c/Makefile to subdirectories.
20. Cpu-tools do not get installed.
21. FIX: Use ACLOCAL_AMFLAGS instead of ACLOCAL = -I ... in Makefile.ams
which are in directories with own configure scripts.
22. Hardcoding BSP names into libbsp/.../tools to avoid RTEMS_BSP get
overridden from the environment.
22. FIX: Handling of MP_PIECES in various Makefiles
23. FIX: Removing "::" rules from some Makefile.ins
24. FIX: File permission chaos: (-m 444 and -m 555 vs. -m 644 and -m
755) - Now all include files use -m 644.
25. Removed many gnumake-conditionals in Makefile.ins - Partially
replaced with automake-conditional, partially replaced with
conditionalized Makefile variables (... _yes_V)
26. Massively reworked acpolish: acpolish now parses Makefile.ins and
interprets parts of the Makefile.ins.
27. FIX: Some $(wildcard $(srcdir)/*.h) macros removed / replaced with
explicit lists of files in Makefile.ins.
28. FIX: Replacing MKLIB with RANLIB in Makefile.ins
29. HACK: Add preinstallation for pc386 specific
$(PROJECT_RELEASE)/BootImgs directory
... many more details, I can't recall
KNOWN BUGS:
1. make [debug|profile]_install do not do what they are promissing.
"make [debug|profile] install" does what "make [debug|profile]_install"
has been doing. Proposal: remove [debug|profile]_install
2. Dependencies between temporary installation tree and source tree are
not yet handled correctly.
3. Dependencies between temporary installation tree and source tree are
handled ineffencently (Using INSTALL_CHANGE instead of make
dependencies)
4. RTEMS_ROOT, PROJECT_ROOT, top_builddir, RTEMS_TOPdir now are
redundant.
5. The new configure scripts still are in their infancy. They contain
redundant checks and might still contain bugs, too.
6. RTEMS autoconf Makefile.ins use a mixture of configuration
information gathered in c/$(target_alias)/<bsp>/make and of information
collected from their configure scripts.
7. make dist is not fully functional
8. Subdirectory host-/build-/target- configure options (--target,
--host, --build) do not conform to Cygnus/GNU conventions.
9. Some RTEMS autoconf Makefile.in's makefile targets are not supported
in automake Makefile.ams/ins (e.g. get, clobber).
10. Some automake standard targets are not propagated from toplevel and
c/Makefile.am to autoconf subdirectories (eg. make dist).
11. rpcgen generated files are not part of the source-tree (Automake
conventions favor supplying generated files inside the source-tree,
however there is no support for rpcgen generated files in automake, cf.
yacc/lex support in automake).
12. RTEMS_HAS_RDBG handling is flaky. make/*.cfg use RTEMS_HAS_RDBG per
CPU, while librdb's sources can only be built per BSP. Raises the more
general question whether librdbg located correctly in the source-tree.
13. All make/*cfg files are configured per cpu, currently there is no
location to store per-bsp configuration information --> bsp.cfg, per
aconfig.h?
14. "make install" without having run "make all" beforehand does not
work.
15. handling of --enable-multiprocessing seems to be broken in
make/custom/*
16. Makefile.ins still exploit many gmake features.
17. File permisson chaos on libraries (no explict -m for
libraries/rels/etc).
18. mcp750 Makefiles are broken (Note: I *do* mean buggy - I am not
talking about "not-conforming to conventions", here :-).
19. Dependencies between configure scripts are not handled, eg. aborting
"make RTEMS_BSP=<bsp>" can leave the build-tree in an unusable state.
20. "make clean" does not delete <build-tree>/<bsp>. This is intentional
for now, because rerunning "make" after "make clean" requires an
explicit "make preinstall" afterwards now. This should be done
automatically, but doesn't work in this case for now. To work around
this problem <build-tree>/<bsp> is kept during "make clean" for now
(HACK).
TODO:
1. split out host-compiled bsp-tools
2. Use Cygnus/GNU standards for cross-compiling target-subdir
(CC=CC_FOR_TARGET .. configure --host=${target_alias}
--build=`config.guess'}), to be added to toplevel configure script after
splitting out bsp-tools.
3. Exploit per cpu support directory (c/src/<cpu>)- Splitting out
per-cpu libraries - Are there any?
4. Further automake support
5. Converting subdirectories into standalone / self-contained
subdirectories (Esp. moving their headers to the same common root as
their sources, eg. mv lib/include/rtems++
lib/librtems++/include/rtems++) - This is the main obstacle which
prevents moving further towards automake.
6. Propagating values from *.cfg into Makefiles instead of propagating
them at make time via Makefile-fragments (i.e. try to avoid using
*.cfg).
7. Testing on cygwin host (I *do* expect cygwin specific problems).
8. The ARCH in o-$(ARCH)-$(VARIANT) build-subdirectories is not needed
anymore.
GENERAL ISSUES:
1. Temporary installation tree -- Ian and I seem to disagree basically.
Though I think that I understand his argumentation, I do not share it.
IMO, his way of using the buildtree is mis-using the build-tree, relying
on an inofficial feature of RTEMS's current implementation, which
doesn't even work correctly in the current build-tree, though it
attempts hard to do so. From my very POV, it unnecessarily complicates
the structures of the source- and build-trees. It is not supported by
automake (No automatic generation for the necessary rules) and
complicates the transition to automake significantly (Generating the
rules with an enhanced version of acpolish could be possible).
As Ian correctly pointed out, here a management decision is needed -
though I don't see the need to draw this decision in short terms.
2. preinstallation generally is a sure means to spoil the structure of
the source tree, IMHO (No ranting intended, I am completly serious about
this one). eg. through tree dependencies. The worst problem related to
this I have found in the meantime is bsp_specs. bsp_specs is part of
libbsp, ie. there is *no* way to build *any* part of the source tree
*without* having a BSP *preinstalled*.
Note: This issue is related to issue 1., but is not identical - The
difference is the change of the order make rules have to be triggered.
While preinstallation triggers rules spread all over the source tree
before a "make all" can be run, a temporary installation tree could also
be installed by post "make all" hooks (all-local:, to be run after make
all in a directory is completed) if the directories' dependencies would
be a tree,
3. Stuctural dependencies between subdirectories.
4. Depth of the source tree (Prevents multilibbing and introduces many
unnecessary configure scripts).
5. per cpu vs. per bsp configuration (There are no real per-cpu parts
yets :-).
6. automake does not support $makefiles in AC_OUTPUT. Unlike before, we
now should try to avoid RTEMS_CHECK_MAKEFILE and to hard-code as much
paths to Makefiles as possible.
7. General redesign of the source tree
8. Main installation point - Changing it to ${prefix}/${target_alias}. ?
Besides item 8. (which is a must, IMHO), as far as I see most of them
can not be solved soon and will remain issues in the mid- to long-term
:-.
REMARKS:
* You (as the maintainer) should always use --enable-maintainer-mode
when building RTEMS to ensure that maintainer mode generated files (esp.
those in c/src/make) will be updated when make/* files have changed.
* Use @RTEMS_BSP@ in Makefile.ins and Makefile.ams below c/src/,
$(RTEMS_BSP) or ${RTEMS_BSP} will be overridden from environment
variables when using make RTEMS_BSP="....".
* c/src/make is a temporary cludge until configuration issues are
solved. At the moment it is configured per bsp, but contains
per-target/cpu info only. Its main purpose now is to circumvent
modifying make/*.cfg files, because I consider make/* to be frozen for
backward compatibilty.
* This patch should only affect configuration files. At least I do not
remember having touched any source files.
* To build the bare bsp you now need to mention it in --enable-rtemsbsp.
Example: building gensh1 and sh1/bare simultaneously:
../rtems-rc-19990709-1/configure --target=sh-rtems \
--enable-rtemsbsp="bare gensh1" \
--prefix=/tmp/rtems \
--enable-bare-cpu-cflags='-DMHZ=20 -m1
-DCPU_CONSOLE_DEVNAME=\"/dev/null\"' \
--enable-bare-cpu-model=sh7032 \
--enable-maintainer-mode \
--enable-cxx
make
make install
* The next steps in development would be to split out bsp-tools and then
to change to Cygnus/GNU canonicalization conventions for building the c/
subdirectory afterwards (i.e. many standard AC_*.m4 macros could be used
instead of customized versions)
FINAL REMARK:
The issues mentioned in the lists above sound much worser than the
situation actually is. Most of them are not specific to this patch, but
are also valid for the snapshot. I just wrote down what I came across
when working on the patch over the last few weeks.
I wouldn't be too surprised if you don't like the patch at the current
point in development. I am willing to discuss details and problems, I
also have no problem if you would post-pone applying this patch to times
after 4.1, but rejecting it as a whole for all times would be a false
management decision, IMHO.
Therefore I would suggest that you, if your time constaints allow it,
should at least play a little while with this patch to understand what
is going on and before drawing a decision on how to handle this
proposal. I know this patch is neither perfect nor complete, but I
consider it to be a major breakthrough. Don't be anxious because of the
size of the patch, the core of the patch is rather small, the size is
mainly the side effect of some systematic cleanups inside the Makefiles
(result of acpolish).
Feel free to ask if you encounter problems, if you don't understand
something or if you meet bugs - I am far from being perfect and am
prepared to answer them.
Ralf.
--
Ralf Corsepius
Forschungsinstitut fuer Anwendungsorientierte Wissensverarbeitung (FAW)
Helmholtzstr. 16, 89081 Ulm, Germany Tel: +49/731/501-8690
mailto:corsepiu@faw.uni-ulm.de FAX: +49/731/501-999
http://www.faw.uni-ulm.de
FYI: I am not talking about using "make -C <dir>", which probably
is much faster on M$ hosts than RTEMS's implementation, but about
removing --enable-gmake-print support and to apply a variant of
automake's subdirectory.
Automake's subdirectory rule seems to be a little bit faster, but I
wouldn't bet on this.
Attached to this mail is my proposal.
After applying the patch, please run
cvs rm aclocal/enable-gmake-print.m4
./autogen
Yet some more modifications, I would recommend to be considered before
releasing a snapshot:
1. Cleanup to aclocal/
cvs rm -f aclocal/cygwin.m4
cvs rm -f aclocal/exeext.m4
They are neither used nor needed anymore, however they also don't
disturb (we use autoconf-2.13's AC_EXEEXT instead, now)
----------
2. rtems-rc-19990328-0.diff
Some (minor) bug-fixes:
* make/Templates/Makefile.inc.in: use the new installation directory
($(prefix)/ instead of $(prefix)/rtems/)
* c/src/exec/score/tools/generic/Makefile.am: added line to include local.am
* c/src/exec/score/tools/*/configure.in: added CVS Id header
----------
3. rtems-rc-19990328-1.diff
Enhancements and cleanups to autogen, rtems-polish.sh, configure.in etc.
* autogen: Use the file "VERSION" to detect RTEMS toplevel directory,
extended usage-message, use "find -print"
* c/update-tools/cipolish: New script to beautify configure.in scripts
* c/update-tools/rtems-polish.sh: Use the file "VERSION" to detect RTEMS
toplevel directory, extended usage-message, added variable for perl
scripts' subdirectory, use "find -print", cipolish support, new options
-ac -am -ci.
* aclocal/*.m4, configure.in: moved some AC_SUBST lines to aclocal/*.m4
(reduces size of configure.in
scripts, eases splitting configure.in scripts).
----------
> 5) rtems-rc-19990202-1.diff/reorg-install.sh
>
> reorg-install.sh fixes a Makefile variable name clash of RTEMS
> configuration files and automake/autoconf standards.
> Until now, RTEMS used $(INSTALL) for install-if-change. Automake and
> autoconf use $(INSTALL) for a bsd-compatible install. As
> install-if-change and bsd-install are not compatible, I renamed all
> references to install-if-changed to $(INSTALL_CHANGED) and used
> $(INSTALL) for bsd-install (==automake/autoconf standard). When
> automake will be introduced install-if-change will probably be replaced
> by $(INSTALL) and therefore will slowly vanish. For the moment, this
> patch fixes a very nasty problem which prevents adding any automake file
> until now (There are still more).
Here is the result of my nightly work to get RTEMS_ROOT=$srcdir working
with different shells and relative/absolute paths.
What I did is relatively simple in principle:
Instead of setting RTEMS_ROOT in configure.in and then let configure
substitute @RTEMS_ROOT@ inside the Makefiles, I now let each Makefile
set RTEMS_ROOT from each Makefile's @top_srcdir@ value.
The difference is subtile, but with enormous side effects:
- If RTEMS_ROOT is set in configure, then the same single value will be
propagated to all Makefiles. This breaks using relative paths, as the
relative path to the root of the source tree is used inside of all
subdirectory Makefiles.
- Now each Makefile.in sets RTEMS_ROOT = @top_srcdir@. top_srcdir is
computed individually by configure for each single Makefile.in, hereby
receiving the correct value, no matter if relative or absolute paths are
used.
To get this working, I needed to remove setting RTEMS_ROOT from
target.cfg.in, because this overrides the value of RTEMS_ROOT from each
individual Makefile.
Furthermore, I removed RTEMS_CUSTOM from the Makefiles and replaced all
"include $(RTEMS_CUSTOM)" directives with"include
$(RTEMS_ROOT)/make/custom/$(RTEMS_BSP)". Perhaps you don't like this,
but I think, to have one variable less is clearer and easier to
understand than having several variables refering to the next one.
I enclose a small patch to this mail, which
- fixes the config.h problem (to finally clearify misunderstands)
- removes assignment/subsitution of RTEMS_ROOT from configure.in
- contains a workaround for the application Makefile's RTEMS_ROOT
problem (reported by Eric)
- removes some unused lines from the toplevel Makefile.in
- removes assignment of RTEMS_ROOT from make/target.cfg.in
