diff options
author | 2017-09-14 23:14:39 +0200 | |
---|---|---|
committer | 2017-10-09 12:08:51 +0200 | |
commit | c6fe2071a2e83be2203196ad7f9459941821a034 (patch) | |
tree | d81e1d9898c05917e05203af9803b581dff0d915 /glep-0033.rst | |
parent | glep-0045: Mark Final since GLEP 1 now uses ISO 8601 dates (diff) | |
download | glep-c6fe2071a2e83be2203196ad7f9459941821a034.tar.gz glep-c6fe2071a2e83be2203196ad7f9459941821a034.tar.bz2 glep-c6fe2071a2e83be2203196ad7f9459941821a034.zip |
Rename all GLEPs to .rst
Diffstat (limited to 'glep-0033.rst')
-rw-r--r-- | glep-0033.rst | 481 |
1 files changed, 481 insertions, 0 deletions
diff --git a/glep-0033.rst b/glep-0033.rst new file mode 100644 index 0000000..3b62182 --- /dev/null +++ b/glep-0033.rst @@ -0,0 +1,481 @@ +GLEP: 33 +Title: Eclass Restructure/Redesign +Version: $Revision$ +Last-Modified: $Date$ +Author: Brian Harring <ferringb@gentoo.org>, John Mylchreest <johnm@gentoo.org> +Status: Deferred +Type: Standards Track +Content-Type: text/x-rst +Created: 29-Jan-2005 +Post-History: 29-Jan-2005 6-Mar-2005 15-Sep-2005 5-Sep-2006 + +Status +====== + +Approved by the Gentoo Council on 15 September 2005. As of Sept. 2006 +this GLEP is on hold, pending future revisions. + +Abstract +======== + +For any design, the transition from theoretical to applied exposes inadequacies +in the original design. This document is intended to document, and propose a +revision of the current eclass setup to address current eclass inadequacies. + +This document proposes several things- the creation of ebuild libraries, 'elibs', +a narrowing of the focus of eclasses, a move of eclasses w/in the tree, the +addition of changelogs, and a way to allow for simple eclass gpg signing. +In general, a large scale restructuring of what eclasses are and how they're +implemented. Essentially version two of the eclass setup. + + +Terminology +=========== + +From this point on, the proposed eclass setup will be called 'new eclasses', the +existing crop (as of this writing) will be referenced as 'old eclasses'. The +distinction is elaborated on within this document. + + +Motivation and Rationale +======================== + +Eclasses within the tree currently are a bit of a mess- they're forced to +maintain backwards compatibility w/ all previous functionality. In effect, +their api is constant, and can only be added to- never changing the existing +functionality. This obviously is quite limiting, and leads to cruft accruing in +eclasses as a eclasses design is refined. This needs to be dealt with prior to +eclass code reaching a critical mass where they become unmanageable/fragile +(recent pushes for eclass versioning could be interpreted as proof of this). + +Beyond that, eclasses were originally intended as a method to allow for ebuilds +to use a pre-existing block of code, rather then having to duplicate the code in +each ebuild. This is a good thing, but there are ill effects that result from +the current design. Eclasses inherit other eclasses to get a single function- in +doing so, modifying the the exported 'template' (default src_compile, default +src_unpack, various vars, etc). All the eclass designer was after was reusing a +function, not making their eclass sensitive to changes in the template of the +eclass it's inheriting. The eclass designer -should- be aware of changes in the +function they're using, but shouldn't have to worry about their default src_* +and pkg_* functions being overwritten, let alone the env changes. + +Addressing up front why a collection of eclass refinements are being rolled into +a single set of changes, parts of this proposal -could- be split into multiple +phases. Why do it though? It's simpler for developers to know that the first +eclass specification was this, and that the second specification is that, +rather then requiring them to be aware of what phase of eclass changes is in +progress. + +By rolling all changes into one large change, a line is intentionally drawn in +the sand. Old eclasses allowed for this, behaved this way. New eclasses allow +for that, and behave this way. This should reduce misconceptions about what is +allowed/possible with eclasses, thus reducing bugs that result from said +misconceptions. + +A few words on elibs- think of them as a clear definition between behavioral +functionality of an eclass, and the library functionality. Eclass's modify +template data, and are the basis for other ebuilds- elibs, however are *just* +common bash functionality. + +Consider the majority of the portage bin/* scripts- these all are candidates for +being added to the tree as elibs, as is the bulk of eutils. + + +Specification +============= + +The various parts of this proposal are broken down into a set of changes and +elaborations on why a proposed change is preferable. It's advisable to the +reader that this be read serially, rather then jumping around. + + +Ebuild Libraries (elibs for short) +---------------------------------- + +As briefly touched upon in Motivation and Rationale, the original eclass design +allowed for the eclass to modify the metadata of an ebuild, metadata being the +DEPENDS, RDEPENDS, SRC_URI, IUSE, etc, vars that are required to be constant, +and used by portage for dep resolution, fetching, etc. Using the earlier +example, if you're after a single function from an eclass (say epatch from +eutils), you -don't- want the metadata modifications the eclass you're +inheriting might do. You want to treat the eclass you're pulling from as a +library, pure and simple. + +A new directory named elib should be added to the top level of the tree to serve +as a repository of ebuild function libraries. Rather then relying on using the +source command, an 'elib' function should be added to portage to import that +libraries functionality. The reason for the indirection via the function is +mostly related to portage internals, but it does serve as an abstraction such +that (for example) zsh compatibility hacks could be hidden in the elib function. + +Elib's will be collections of bash functions- they're not allowed to do anything +in the global scope aside from function definition, and any -minimal- +initialization of the library that is absolutely needed. Additionally, they +cannot modify any ebuild template functions- src_compile, src_unpack. Since they are +required to not modify the metadata keys, nor in any way affect the ebuild aside +from providing functionality, they can be conditionally pulled in. They also +are allowed to pull in other elibs, but strictly just elibs- no eclasses, just +other elibs. A real world example would be the eutils eclass. + +Portage, since the elib's don't modify metadata, isn't required to track elibs +as it tracks eclasses. Thus a change in an elib doesn't result in half the tree +forced to be regenerated/marked stale when changed (this is more of an infra +benefit, although regen's that take too long due to eclass changes have been +known to cause rsync issues due to missing timestamps). + +Elibs will not be available in the global scope of an eclass, or ebuild- nor during the +depends phase (basically a phase that sources the ebuild, to get its metadata). Elib +calls in the global scope will be tracked, but the elib will not be loaded till just before +the setup phase (pkg_setup). There are two reasons for this- first, it ensures elibs are +completely incapable of modifying metadata. There is no room for confusion, late loading +of elibs gives you the functionality for all phases, except for depends- depends being the +only phase that is capable of specifying metadata. Second, as an added bonus, late +loading reduces the amount of bash sourced for a regen- faster regens. This however is minor, +and is an ancillary benefit of the first reason. + +There are a few further restrictions with elibs--mainly, elibs to load can only be specified +in either global scope, or in the setup, unpack, compile, test, and install phases. You can +not load elibs in prerm, postrm, preinst, and postinst. The reason being, for \*rm phases, +installed pkgs will have to look to the tree for the elib, which allows for api drift to cause +breakage. For \*inst phases, same thing, except the culprit is binpkgs. + +There is a final restriction--elibs cannot change their exported api dependent on the api +(as some eclass do for example). The reason mainly being that elibs are loaded once--not +multiple times, as eclasses are. + +To clarify, for example this is invalid. +:: + + if [[ -n ${SOME_VAR} ]]; then + func x() { echo "I'm accessible only via tweaking some var";} + else + func x() { echo "this is invalid, do not do it."; } + fi + + +Regarding maintainability of elibs, it should be a less of a load then old +eclasses. One of the major issues with old eclasses is that their functions are +quite incestuous- they're bound tightly to the env they're defined in. This +makes eclass functions a bit fragile- the restrictions on what can, and cannot +be done in elibs will address this, making functionality less fragile (thus a +bit more maintainable). + +There is no need for backwards compatibility with elibs- they just must work +against the current tree. Thus elibs can be removed when the tree no longer +needs them. The reasons for this are explained below. + +Structuring of the elibs directory will be exactly the same as that of the new +eclass directory (detailed below), sans a different extension. + +As to why there are so many restrictions, the answer is simple- the definition of +what elibs are, what they are capable of, and how to use them is nailed down as much as +possible to avoid *any* ambiguity related to them. The intention is to make it clear, +such that no misconceptions occur, resulting in bugs. + +The reduced role of Eclasses, and a clarification of existing Eclass requirements +--------------------------------------------------------------------------------- + +Since elibs are now intended on holding common bash functionality, the focus of +eclasses should be in defining an appropriate template for ebuilds. For example, +defining common DEPENDS, RDEPENDS, src_compile functions, src_unpack, etc. +Additionally, eclasses should pull in any elibs they need for functionality. + +Eclass functionality that isn't directly related to the metadata, or src_* and +pkg_* funcs should be shifted into elibs to allow for maximal code reuse. This +however isn't a hard requirement, merely a strongly worded suggestion. + +Previously, it was 'strongly' suggested by developers to avoid having any code +executed in the global scope that wasn't required. This suggestion is now a +requirement. Execute only what must be executed in the global scope. Any code +executed in the global scope that is related to configuring/building the package +must be placed in pkg_setup. Metadata keys (already a rule, but now stated as +an absolute requirement to clarify it) *must* be constant. The results of +metadata keys exported from an ebuild on system A, must be *exactly* the same as +the keys exported on system B. + +If an eclass (or ebuild for that matter) violates this constant requirement, it +leads to portage doing the wrong thing for rsync users- for example, wrong deps +pulled in, leading to compilation failure, or dud deps. + +If the existing metadata isn't flexible enough for what is required for a +package, the parsing of the metadata is changed to address that. Cases where +the constant requirement is violated are known, and a select few are allowed- +these are exceptions to the rule that are required due to inadequacies in +portage. Any case where it's determined the constant requirement may need to be +violated the dev must make it aware to the majority of devs, along with the portage +devs. This should be done prior to committing. + +It's quite likely there is a way to allow what you're attempting- if you just go +and do it, the rsync users (our user base) suffer the results of compilation +failures and unneeded deps being pulled in. + +After that stern reminder, back to new eclasses. Defining INHERITED and ECLASS +within the eclass is no longer required. Portage already handles those vars if +they aren't defined. + +As with elibs, it's no longer required that backwards compatibility be maintained +indefinitely- compatibility must be maintained against the current tree, but +just that. As such new eclasses (the true distinction of new vs old is +elaborated in the next section) can be removed from the tree once they're no +longer in use. + + +The end of backwards compatibility... +------------------------------------- + +With current eclasses, once the eclass is in use, its api can no longer be +changed, nor can the eclass ever be removed from the tree. This is why we still +have *ancient* eclasses that are completely unused sitting in the tree, for +example inherit.eclass. The reason for this, not surprisingly, is a portage +deficiency: on unmerging an installed ebuild, portage used the eclass from the +current tree. + +For a real world example of this, if you merged a glibc 2 years back, whatever +eclasses it used must still be compatible, or you may not be able to unmerge the +older glibc version during an upgrade to a newer version. So either the glibc +maintainer is left with the option of leaving people using ancient versions out +in the rain, or maintaining an ever increasing load of backwards compatibility +cruft in any used eclasses. + +Binpkgs suffer a similar fate. Merging of a binpkg pulls needed eclasses from +the tree, so you may not be able to even merge a binpkg if the eclasses api has +changed. If the eclass was removed, you can't even merge the binpkg, period. + +The next major release of portage will address this- the environment that the +ebuild was built in already contains the eclasses functions, as such the env can +be re-used rather then relying on the eclass. In other words, binpkgs and +installed ebuilds will no longer go and pull needed eclasses from the tree, +they'll use the 'saved' version of the eclass they were built/merged with. + +So the backwards compatibility requirement for users of the next major portage +version (and beyond) isn't required. All the cruft can be dropped. + +The problem is that there will be users using older versions of portage that don't +support this functionality- these older installations *cannot* use the +new eclasses, due to the fact that their portage version is incapable of +properly relying on the env- in other words, the varying api of the eclass will +result in user-visible failures during unmerging. + +So we're able to do a clean break of all old eclasses, and api cruft, but we need +a means to basically disallow access to the new eclasses for all portage versions +incapable of properly handling the env requirements. + +Unfortunately, we cannot just rely on a different grouping/naming convention within +the old eclass directory. The new eclasses must be inaccessible, and portage throws +a snag into this- the existing inherit function that is used to handle existing +eclasses. Basically, whatever it's passed (inherit kernel or inherit +kernel/kernel) it will pull in (kernel.eclass, and kernel/kernel.eclass +respectively). So even if the new eclasses were implemented within a +subdirectory of the eclass dir in the tree, all current portage versions would +still be able to access them. + +In other words, these new eclasses would in effect, be old eclasses since older +portage versions could still access them. + + +Tree restructuring +------------------ + +There are only two way to block the existing (as of this writing) inherit +functionality from accessing the new eclasses- either change the extension of +eclasses to something other then 'eclass', or to have them stored in a separate +subdirectory of the tree then eclass. + +The latter is preferable, and the proposed solution. Reasons are- the current +eclass directory is already overgrown. Structuring of the new eclass dir +(clarified below) will allow for easier signing, ChangeLogs, and grouping of +eclasses. New eclasses allow for something akin to a clean break and have new +capabilities/requirements, thus it's advisable to start with a clean directory, +devoid of all cruft from the old eclass implementation. + +If it's unclear as to why the old inherit function *cannot* access the new +eclasses, please reread the previous section. It's unfortunately a requirement +to take advantage of all that the next major portage release will allow. + +The proposed directory structure is ${PORTDIR}/include/{eclass,elib}. +Something like ${PORTDIR}/new-eclass, or ${PORTDIR}/eclass-ng could be used +(although many would cringe at the -ng), but such a name is unwise. Consider the +possibility (likely a fact) that new eclasses someday may be found lacking, and +refined further (version three as it were). Or perhaps we want to add yet more +functionality with direct relation to sourcing new files, and we would then need +to further populate ${PORTDIR}. + +The new-eclass directory will be (at least) 2 levels deep- for example: + +:: + kernel/ + kernel/linux-info.eclass + kernel/linux-mod.eclass + kernel/kernel-2.6.eclass + kernel/kernel-2.4.eclass + kernel/ChangeLog + kernel/Manifest + +No eclasses will be allowed in the base directory- grouping of new eclasses will +be required to help keep things tidy, and for the following reasons. Grouping +of eclasses allows for the addition of ChangeLogs that are specific to that +group of eclasses, grouping of files/patches as needed, and allows for +saner/easier signing of eclasses- you can just stick a signed +Manifest file w/in that grouping, thus providing the information portage needs +to ensure no files are missing, and that nothing has been tainted. + +The elib directory will be structured in the same way, for the same reasons. + +Repoman will have to be extended to work within new eclass and elib groups, and +to handle signing and committing. This is intentional, and a good thing. This +gives repoman the possibility of doing sanity checks on elibs/new eclasses. + +Note these checks will not prevent developers from doing dumb things with eclass- +these checks would only be capable of doing basic sanity checks, such as syntax checks. +There is no way to prevent people from doing dumb things (exempting perhaps repeated +applications of a cattle prod)- these are strictly automatic checks, akin to repoman's +dependency checks. + + +The start of a different phase of backwards compatibility +--------------------------------------------------------- + +As clarified above, new eclasses will exist in a separate directory that will be +intentionally inaccessible to the inherit function. As such, users of older +portage versions *will* have to upgrade to merge any ebuild that uses elibs/new +eclasses. A depend on the next major portage version would transparently handle +this for rsync users. + +There still is the issue of users who haven't upgraded to the required portage +version. This is a minor concern frankly- portage releases include new +functionality, and bug fixes. If they won't upgrade, it's assumed they have +their reasons and are big boys, thus able to handle the complications themselves. + +The real issue is broken envs, whether in binpkgs, or for installed packages. +Two options exist- either the old eclasses are left in the tree indefinitely, or +they're left for N months, then shifted out of the tree, and into a tarball that +can be merged. + +Shifting them out of the tree is advisable for several reasons- less cruft in +the tree, but more importantly the fact that they are not signed (thus an angle +for attack). Note that the proposed method of eclass signing doesn't even try +to address them. Frankly, it's not worth the effort supporting two variations +of eclass signing, when the old eclass setup isn't designed to allow for easy +signing. + +If this approach is taken, then either the old eclasses would have to be merged +to an overlay directory's eclass directory (ugly), or to a safe location that +portage's inherit function knows to look for (less ugly). + +For users who do not upgrade within the window of N months while the old +eclasses are in the tree, as stated, it's assumed they know what they are doing. +If they specifically block the new portage version, as the ebuilds in the tree +migrate to the new eclasses, they will have less and less ebuilds available to +them. If they tried injecting the new portage version (lying to portage, +essentially), portage would bail since it cannot find the new eclass. +For ebuilds that use the new eclasses, there really isn't any way to sidestep +the portage version requirement- same as it has been for other portage features. + +What is a bit more annoying is that once the old eclasses are out of the tree, +if a user has not upgraded to a portage version supporting env processing, they +will lose the ability to unmerge any installed ebuild that used an old +eclass. Same cause, different symptom being they will lose the ability to merge +any tbz2 that uses old eclasses also. + +There is one additional case that is a rarity, but should be noted- if a user +has suffered significant corruption of their installed package database (vdb). This is +ignoring the question of whether the vdb is even usable at this point, but the possibility +exists for the saved envs to be non usable due to either A) missing, or B) corrupted. +In such a case, even with the new portage capabilities, they would need +the old eclass compat ebuild. + +Note for this to happen requires either rather... unwise uses of root, or significant +fs corruption. Regardless of the cause, it's quite likely for this to even become an +issue, the system's vdb is completely unusable. It's a moot issue at that point. +If you lose your vdb, or it gets seriously damaged, it's akin to lobotomizing portage- +it doesn't know what's installed, it doesn't know of its own files, and in general, +a rebuilding of the system is about the only sane course of action. The missing env is +truly the least of the users concern in such a case. + +Continuing with the more likely scenario, users unwilling to upgrade portage will +*not* be left out in the rain. Merging the old eclass compat ebuild will provide +the missing eclasses, thus providing that lost functionality. + +Note the intention isn't to force them to upgrade, hence the ability to restore the +lost functionality. The intention is to clean up the existing mess, and allow us +to move forward. The saying "you've got to break a few eggs to make an omelet" +is akin, exempting the fact we're providing a way to make the eggs whole again +(the king's men would've loved such an option). + + +Migrating to the new setup +-------------------------- + +As has been done in the past whenever a change in the tree results in ebuilds +requiring a specific version of portage, as ebuilds migrate to the new eclasses, +they should depend on a version of portage that supports it. From the users +viewpoint, this transparently handles the migration. + +This isn't so transparent for devs or a particular infrastructure server however. +Devs, due to them using cvs for their tree, lack the pregenerated cache rsync +users have. Devs will have to be early adopters of the new portage. Older +portage versions won't be able to access the new eclasses, thus the local cache +generation for that ebuild will fail, ergo the depends on a newer portage +version won't transparently handle it for them. + +Additionally, prior to any ebuilds in the tree using the new eclasses, the +infrastructure server that generates the cache for rsync users will have to +either be upgraded to a version of portage supporting new eclasses, or patched. +The former being much more preferable then the latter for the portage devs. + +Beyond that, an appropriate window for old eclasses to exist in the tree must be +determined, and prior to that window passing, an ebuild must be added to the tree +so users can get the old eclasses if needed. + +For eclass devs to migrate from old to new, it is possible for them to just +transfer the old eclass into an appropriate grouping in the new eclass directory, +although it's advisable they cleanse all cruft out of the eclass. You can +migrate ebuilds gradually over to the new eclass, and don't have to worry about +having to support ebuilds from X years back. + +Essentially, you have a chance to nail the design perfectly/cleanly, and have a +window in which to redesign it. It's humbly suggested eclass devs take +advantage of it. :) + + +Backwards Compatibility +======================= + +All backwards compatibility issues are addressed in line, but a recap is offered- +it's suggested that if the a particular compatibility issue is +questioned/worried over, the reader read the relevant section. There should be +a more in depth discussion of the issue, along with a more extensive explanation +of the potential solutions, and reasons for the chosen solution. + +To recap: +:: + + New eclasses and elib functionality will be tied to a specific portage + version. A DEPENDs on said portage version should address this for rsync + users who refuse to upgrade to a portage version that supports the new + eclasses/elibs and will gradually be unable to merge ebuilds that use said + functionality. It is their choice to upgrade, as such, the gradual + 'thinning' of available ebuilds should they block the portage upgrade is + their responsibility. + + Old eclasses at some point in the future should be removed from the tree, + and released in a tarball/ebuild. This will cause installed ebuilds that + rely on the old eclass to be unable to unmerge, with the same applying for + merging of binpkgs dependent on the following paragraph. + + The old eclass-compat is only required for users who do not upgrade their + portage installation, and one further exemption- if the user has somehow + corrupted/destroyed their installed pkgs database (/var/db/pkg currently), + in the process, they've lost their saved environments. The eclass-compat + ebuild would be required for ebuilds that required older eclasses in such a + case. Note, this case is rare also- as clarified above, it's mentioned + strictly to be complete, it's not much of a real world scenario as elaborated + above. + + +Copyright +========= + +This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 +Unported License. To view a copy of this license, visit +http://creativecommons.org/licenses/by-sa/3.0/. |