0004538: Symbolic links not created in new sparse-root zone

Notes
(0008241) phil (reporter) 2010-09-02 19:23	How is /opt/csw declared, in this "sparse root zone"? standalone filesystem? pkg-inherit? part of "/" ?

(0008242) gadavis (reporter) 2010-09-02 19:47	Sorry, should have been explicit about that - /opt/csw ends up as part of the zone "/" partition, but no pkg-inherit-dir for it. We tend to have different packages installed on a per-zone basis than what is installed in the global zone. Here's a typical zone configuration from one of our systems: create -b set zonepath=/zones/anfds set autoboot=true set limitpriv=default set ip-type=shared add inherit-pkg-dir set dir=/lib end add inherit-pkg-dir set dir=/platform end add inherit-pkg-dir set dir=/sbin end add inherit-pkg-dir set dir=/usr end add fs set dir=/usr/local set special=/zones/anfds/local set type=lofs end add net set address=NETADDR Censored set physical=aggr1 end add net set address=NETADDR Censored set physical=e1000g1003 end add attr set name=comment set type=string set value="ANF Directory Server" end

(0008244) phil (reporter) 2010-09-02 22:53	Thanks for the info. I'd say you're doing it "wrong", then. Either you should have /opt/csw as inherit-pkg-dir... (or possibly have it just directly mounted read-only from global zone...) or you should have it installed as a fully local package. Whats the point of having a zone-local filesystem for /opt/csw, if you're not going to have actual "local" packages installed? If you could recreate, or adjust, the zone, and let me know if this fixes your issue, then I would be happy to, on our side, update our documents with this additional information, in whichever locations you might suggest. I know you mentioned that you "tend to have different packages installed on a per-zone basis than what is installed in the global zone". but you cant "half-share" packages like this. pick one way or the other. ERROR: Sanity check failed ;-)

(0008245) gadavis (reporter) 2010-09-02 23:30	Phil, I'm confused what you mean by "or you should have it installed as a fully local package." Can you suggest what changes I would need to make to my zonecfg commands, or elaborate further? I'm not sharing any files from in the /opt/csw/ tree with the global zone. There are no loopback mounts anywhere for /opt/csw in the zone manifest that I pasted above. So therefore, a newly created sparse-root zone has it's own full copies of whatever CSW packages were installed in the global zone. The only exception to this rule is CSWclassutils, which has to put it's class action scripts in /usr. From the section on "Sparse Root Zones" in zonecfg(1M): Packages that do not deliver content into read-only loopback mount file systems are fully installed.

(0008246) phil (reporter) 2010-09-03 00:12	Part one of reply: To have a fully local package install, without changing anything else in your setup, you would have to do BOTH of - stop installing CSW packages in the global zone(or use pkgadd -G) - run pkgadd individually in each zone Note, however, that using pkgadd -G is not a good solution, since sun screwed up/orphaned the implementation. There is no pkgrm -G. So the next time you do pkgrm in the global(ie for an upgrade!), you will then pkgrm in all child zones automatically! part two of reply: the problem is that if you are not inherit-pkg-dir-ing, then updates may not get shared out to child zones properly. Things get copied okay initially, when you first create a child zone.. but then things drift, I think. Things get especially dangerous in a mixed mode, when you pkgadd/rm from global; You have class action scripts, and/or pre/postinstall scripts running, from the package in the global zone, that expect the same version of files in the package, but the actual files in the child zone may not be the same version. Note: I have not actually TESTED the behaviour in mixed-mode recently, but I believe this is the source of your problems. I think I'll go do some experimenting.

(0008247) gadavis (reporter) 2010-09-03 01:21	I can see where the class action scripts could cause problems in this mixed-mode when packages are added and removed. However, just to be clear, no further pkgadd commands were run in either the global zone or non-global zone after the zone was installed with "zoneadm -z whatever install" It's worth determining what exactly happens with the class action scripts when zoneadm install is run and the list of currently installed packages is duplicated to the zone.

(0008248) phil (reporter) 2010-09-03 01:58	Is your "alternatives" package up to date? I can imagine how the older implementation would have this problem, but not the newer one. (which is exactly why I WROTE the newer implementation! :-)

(0008249) phil (reporter) 2010-09-03 03:02	Hmm.. btw i take back what I said about pkgadd -G. If you always use it for global-zone CSW packages, it seems as though pkgrm knows the package was installed with -G, and so will not pkgrm in the child zones. I havent yet found out where it STORES this information in the global zone, which is making me a bit... irritated... but observationally, it at least appears to follow this behaviour somehow.

(0008263) gadavis (reporter) 2010-09-03 19:54	CSWalternatives is 1.0,REV=2010.05.21 which is what is reported as current.

(0008265) phil (reporter) 2010-09-03 23:43	Thanks for checking. I've done some more testing, and I have replicated it myself. Summary of behaviour: When creating a new zone, the "zoneadm install" command would seem to go through the package database in the global zone... NOT the filesystems... and copy every file that is both in the database, and not in a "inherit-pkg-dir" directory. Files in a filesystem, but not in the pkg database, do not get copied. The "alternatives" symlinks fit into this category. That being said... what to do about it? I could theoretically hack the "alternatives" scripts in nasty ways, to register the links with a fake package. or even the CSWalternatives package. But this does not seem clean to me. And then there remains the issue of: SHOULD I even attempt to fix this? It would validate a deployment method which in my opinion, will only lead to futhre trouble eventually. I think that if someone wants fully local package deployment to each zone, they should deploy packages locally. And if they want global installation of packages... they should use the inherit-pkg-dir functionality that is provided for that sort of thing! (specifically, for /opt/csw not /etc/opt/csw). It helps them a bunch also, avoiding needless redundant backups,etc. All zone-local configs should now be in /etc/opt/csw anyway, and any csw packages that dont follow this, should have a bug filed against them. What do you think, Geoff?

(0008267) gadavis (reporter) 2010-09-04 00:54	Well, here are my thoughts on it. Our systems have a baseline Solaris install of SUNW packages that we don't tweak on a per host basis, at least for those in the /usr tree. I think the jumpstart cluster used is SUNWcxall However, the individual zones vary significantly from one another in terms of their OpenCSW packages. Our development zones have compilers installed, automake, header files, etc - all stuff that doesn't really belong on a production system. Often, development and production Zones live on the same hosting physical system. Even among production systems, there's little reason for one of our data acquisition hosts to have Apache installed for example. In summary, the biggest difference between individual Zones in our environment is it's selection of CSW packages, not SUNW packages (Sun Studio being the only notable exception). Having a whole-root zone would result in a much larger duplication of packages.

(0008268) phil (reporter) 2010-09-04 01:03	Oh, I'm not saying individual root per zone at all!! I believe you can mix and match "sparse-ROOT", but zone-local /opt/csw. It is called sparse root, not "sparse packages" after all :) It kinda sounds like you're actually doing that already? Are you perhaps adding SOME csw packages at the global zone, but other packages at the individual zone level?

(0008269) gadavis (reporter) 2010-09-04 01:09	That is correct. We have a baseline of CSW packages installed in the global zone, things like sudo, cfengine, and their dependencies. We have been installing them without the -G option since all of our Zones need these common management tools as well. I guess we could start installing them with the -G option, but it will probably break the special case of CSWclassutils which has to have some of it's files in /usr somewhere.

(0008270) phil (reporter) 2010-09-04 02:16	Mmm, yeah CSWclassutils is unfortunately a special, somewhat ugly case. The good news is, that one doesnt have any postinstall scripts, etc. :) For everything else though, sounds like zone local is what you need, to ensure that everything works 100% as expected, when you create a new zone without cloning. OR.... you would do that special case re-running the alternatives script for new zones. sigh. Extra background: This could be "fixed" by installf getting called when symlinks are created... but installf requires a package to associate the symlinks with in the pkg database. which package would I use; itself, or the top level alternative being linked? Also, I'm trying to keep the "alternatives" script itself, portable. So I dont want to hardcode "CSWxxx" stuff in it. Hmm.. although I did have to hardcode /opt/csw in it anyway.... I MIGHT do that after all, I suppose. Another approach might be the other class-action wrappers, which in theory could be adjusted to register the symlinks when they call it, at pkgadd/rm time. Except that it would not be registered when 'alternatives' was called directly.

(0008271) phil (reporter) 2010-09-04 03:02	FYI: I updated http://wiki.opencsw.org/zone-behaviour [^] with "complicated problem" sections, listed in the table of contents for easy reference. You might want to chew over those while contemplating your zone/pkg management strategy. I welcome feedback.

(0008320) phil (reporter) 2010-10-02 18:25	Since I have proposed an official "workaround" (declare /opt/csw as inherit-pkg-dir) and since there was no further feedback, I'm closing this bug.

(0008342) gadavis (reporter) 2010-10-04 02:11	Sorry, I've been knee-deep in systems migrations since this bug was filed. I cannot accept inherit-pkg-dir as a solution, as it does not work at all in an environment where the zones on a system are minimized to only have the necessary packages installed on them. There should be some way to trigger alternatives to do a fixup of it's symlinks without having to manually iterate through a directory listing.

(0008349) philadmin (viewer) 2010-10-04 19:29	Please tell me of a trigger to use, and I'll be happy to try to use it. Unfortunately, I'm not aware of any. I might instead attempt to make things a little easier for you, by adding a "--relink-all" option to alternatives. That would trigger the "go through the directory" behaviour. The only other solution, would involve having to actually register the symlinks created, with a package. Then the symlink should be duplicated upon zone creation. but I didnt see you comment about that. There's also the question of, WHICH package to register it with? Potentially the one containing the binary being linked to, but I'm not sure.

(0008350) gadavis (reporter) 2010-10-04 20:05	I unfortunately don't know my way around the class action scripts to say where the action should run. Certainly a --relink-all option would be useful in the mean time. I would suggest that the symlink should be registered with the alternatives package, since it's not really part of the dependent packages that alternatives is switching between.

(0009067) jcraig (reporter) 2011-05-24 20:47	This problem has nothing to do with the existence/non-existence of inherit package. Tracing through the "zoneadm install" command it appears that the CAS used by alternatives does not result in a persistent file. I'll not pretend to be an expert here, as I'm merely a hack, but I believe the i.cswalternative command should use the "installf" command to register this symlink with the system. When I look in the /var/sadm/install/contents file I do not see an entry for /opt/csw/bin/sudo file. My reading of the package developer guide leads me to believe files delivered by a package must either by declared statically in the pkgmap or dynamically via installf. I suspect that the zoneadm install process is removing / disallowing the creation of this file as it isn't/wasn't properly registered. I'll continue to hack away at a solution but I thought someone more familiar with packaging might resolve this issue sooner. The net effect is that packages installed from the global zone during the zoneadm install process will not have their alternatives symlink created.

(0009068) phil (reporter) 2011-05-24 23:38	Please scroll back and reread my earlier notes and question on "installf".

(0009069) phil (reporter) 2011-05-24 23:44	Ah, I didnt notice that gadavis did answer with a suggestion on which package for installf. The one problem with that, is that when the "alternatives" package gets upgraded, the symlinks would get removed, and thus "lose" preferences, I think. To preserve preferences, you have to have something of persistance, outside of what is registered in the package database. Currently, the symlinks themselves serve as that, but only if they are not registered. I'll try to get around to implemeting the "relink-all" option soon though.

(0009070) phil (reporter) 2011-05-24 23:53	wow, its been a long time since I looked at this code. it is NOT true that when the alternatives package gets removed, that manual overrides get removed also. I'll try to think of this a bit more, when I implement relink-all

(0009074) jcraig (reporter) 2011-05-25 15:09	An elegant fix to this would be to have the "alternatives" binary use installf/removef to manage the symlink and register the link to the CSWalternatives package. This would keep a valid reference in place as the preference is changed and resolve package installation issues during zone installs. I tested this by calling installf for the sudo link and then installing a new whole root zone. Once installed, the sudo link existed. As alternatives are implemented now, and given the statement that the only supported zone implementation is a sparse CSW design, an end user gets one and only one choice between alternatives for every zone on a particular box. This level of support is too narrow, in my opinion.

(0009075) phil (reporter) 2011-05-25 20:31	Well, I've grabbed the low-hanging fruit for right now. Geoff (or jcraig, if you wish :) please try out http://buildfarm.opencsw.org/experimental/phil/alternatives [^] and see if it is useful. Note that it is the script, not a package, and that I decided to make the argument, --relink-missing I think that name demonstrates more, its actual behaviour: that it will not change existing links. Whereas my originally chosen name of--relink-all seems to imply it would reset links.

(0009078) phil (reporter) 2011-05-25 22:20	actually, I've now created a full test package. Try it if you wish. manual download from http://buildfarm.opencsw.org/experimental/phil [^] or set your url temporarily to http://buildfarm.opencsw.org/opencsw/experimental/phil [^] This has the extra command option, the grep fix, plus some README files.

(0010545) pfelecan (manager) 2013-08-25 15:35	This being too old I think that its relevancy is no more. If I'm wrong, please re-open it.

OpenCSW Bug Tracker