Discussion:
FInding locally installed packages in different environments
(too old to reply)
Urs Liska
2018-04-13 06:58:15 UTC
Permalink
Raw Message
Hi all,

there's an issue I've come across occasionally, and I'd like to clarify
this once and for all ...

I have installed texlive-full on Debian, actually I've dragged this
along through a number of installations while keeping my home partition,
so maybe there's something I have done in the past to make things "work
for me".

My $TEXMFHOME is ~/texmf, and I have a number of custom packages I use
and make available through this $TEXMFHOME.

However, since I maintain my stuff with Git I prefer having the actual
code sitting below ~/git/latex/<packages>, and I have
~/texmf/tex/latex/latex-git as a symlink to ~/git/latex.

Through this LaTeX finds any resources in and below ~/git/latex, which
is what I want - so it "works for me". It also has always worked when I
started using a fresh distribution (but always Debian-based distros if
that matters).

The problem is that collaborators don't always seem to be able to follow
instructions like:

* look for or create and cd into ~/texmf/tex/latex
* create a symlink with
ln -s /path/to/our/project/latex-resources
* try compiling the document

I read about having to use texhash and/or mktexlsr - once or after every
changes to ~/texmf - but I don't seem to have to do that on my computer.
Is the behaviour of finding packages in the "home" tree depending on the
LInux distro or the way how TeXLive is installed (manual/package)? Do
the differen LaTeX engines behave differently in this regard?

Currently I have a contributor using Gentoo Linux with texlive-full
installed from the distro's repositories. He has added the symlink to
our project's latex directory but when compiling (with lualatex
filename) it chokes upon not finding the document class (in that directory).

I would really like to know what steps have to be taken and if they
differ in any way depending on the environment. It's not only that I
need to get this contributor up and running, but I need to write a CG
that will free me from unnecessary hassles with future contributors.

Thanks for any clarification
Urs
Zdenek Wagner
2018-04-13 08:48:41 UTC
Permalink
Raw Message
Hi,

it depends on your configuration whether you need mktexlsr or not. The
paths are configured in texmf.cnf. If the path begins with two exclamation
marks, only the ls-R file is used for locating the file. This means that
mktexlsr is needed. If the path does not have exclamation marks which is
the case of $TEXMFHOME, ls-R files are not used. It is described in detail
in the documentation of kpathsea, I do not remember the exact details. The
location of $TEXMFHOME can be found by kpsewhich. So if you wish to provide
the files for non-experts, it would be better to provide a script which
will first locate itself, then locate $TEXMFHOME, create the directories
(if they do not exist) and create the symlink. The command you need is:

kpsewhich -var-value TEXMFHOME

Variables TEXMF and TEXMFDBS contain the paths with exclamation marks. If
the path is here with the exlamation marks, you must call mktexlsr after
creation of the symlink.


Zdeněk Wagner
http://ttsm.icpf.cas.cz/team/wagner.shtml
http://icebearsoft.euweb.cz
Post by Urs Liska
Hi all,
there's an issue I've come across occasionally, and I'd like to clarify
this once and for all ...
I have installed texlive-full on Debian, actually I've dragged this along
through a number of installations while keeping my home partition, so maybe
there's something I have done in the past to make things "work for me".
My $TEXMFHOME is ~/texmf, and I have a number of custom packages I use and
make available through this $TEXMFHOME.
However, since I maintain my stuff with Git I prefer having the actual
code sitting below ~/git/latex/<packages>, and I have
~/texmf/tex/latex/latex-git as a symlink to ~/git/latex.
Through this LaTeX finds any resources in and below ~/git/latex, which is
what I want - so it "works for me". It also has always worked when I
started using a fresh distribution (but always Debian-based distros if that
matters).
The problem is that collaborators don't always seem to be able to follow
- look for or create and cd into ~/texmf/tex/latex
- create a symlink with
ln -s /path/to/our/project/latex-resources
- try compiling the document
I read about having to use texhash and/or mktexlsr - once or after every
changes to ~/texmf - but I don't seem to have to do that on my computer. Is
the behaviour of finding packages in the "home" tree depending on the LInux
distro or the way how TeXLive is installed (manual/package)? Do the
differen LaTeX engines behave differently in this regard?
Currently I have a contributor using Gentoo Linux with texlive-full
installed from the distro's repositories. He has added the symlink to our
project's latex directory but when compiling (with lualatex filename) it
chokes upon not finding the document class (in that directory).
I would really like to know what steps have to be taken and if they differ
in any way depending on the environment. It's not only that I need to get
this contributor up and running, but I need to write a CG that will free me
from unnecessary hassles with future contributors.
Thanks for any clarification
Urs
Norbert Preining
2018-04-13 10:30:14 UTC
Permalink
Raw Message
Hi Urs,
Post by Urs Liska
* look for or create and cd into ~/texmf/tex/latex
* create a symlink with
ln -s /path/to/our/project/latex-resources
* try compiling the document
Sounds fine. I would expect that the same thing works in all other
variants of TeX Live, too. TEXMFHOME should NOT be prefixed with !!, so
searching should work.

You can ask your co-authors to run
kpsewhich --debug=32 <one-of-the-files-there>
and see what is logged, it might give hints why the dirs are not
searched. --debug=-1 which logs everything, but can be quite
overwhelming.
Post by Urs Liska
I read about having to use texhash and/or mktexlsr - once or after every
changes to ~/texmf - but I don't seem to have to do that on my computer. Is
If there is an ls-R file, please remove it!!! That is 100 times safer.

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Urs Liska
2018-04-19 14:57:34 UTC
Permalink
Raw Message
Post by Norbert Preining
Hi Urs,
Post by Urs Liska
* look for or create and cd into ~/texmf/tex/latex
* create a symlink with
ln -s /path/to/our/project/latex-resources
* try compiling the document
Sounds fine. I would expect that the same thing works in all other
variants of TeX Live, too. TEXMFHOME should NOT be prefixed with !!, so
searching should work.
You can ask your co-authors to run
kpsewhich --debug=32 <one-of-the-files-there>
and see what is logged, it might give hints why the dirs are not
searched. --debug=-1 which logs everything, but can be quite
overwhelming.
Post by Urs Liska
I read about having to use texhash and/or mktexlsr - once or after every
changes to ~/texmf - but I don't seem to have to do that on my computer. Is
If there is an ls-R file, please remove it!!! That is 100 times safer.
Norbert
It turned out there was another issue at play (after the actual errors
with wrong paths were sorted out).
As this was a new installation ~/texmf hadn't existed before, and the
contributor basically created ~/texmf/tex/latex, which contained two
symlinks to our LaTeX resources.
I ordered him to create a hidden dummy directory with a hidden dummy
file in it - and voilà, the classes and packages were found.

I think I didn't explicitly state that we use LuaLaTeX, but I asked if
the different engines behave differently in this respect.
And I recall from earlier questions that having at least ony actual
directory available was necessary for LuaLaTeX to follow symlinks to
find fonts. It seems this also goes for regular files.

To what extent is this known behaviour? And is this *acceptable*
behaviour? Or did I even misunderstood anything and it now works by
accident?

Thanks
Urs
Norbert Preining
2018-04-19 15:04:08 UTC
Permalink
Raw Message
I think I didn't explicitly state that we use LuaLaTeX, but I asked if the
different engines behave differently in this respect.
And I recall from earlier questions that having at least ony actual
directory available was necessary for LuaLaTeX to follow symlinks to find
fonts. It seems this also goes for regular files.
To what extent is this known behaviour? And is this *acceptable* behaviour?
Or did I even misunderstood anything and it now works by accident?
I never heard about this, but IMNSHO this is absolutely unacceptable
behaviour.

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Urs Liska
2018-04-19 15:12:13 UTC
Permalink
Raw Message
Post by Norbert Preining
I think I didn't explicitly state that we use LuaLaTeX, but I asked if the
different engines behave differently in this respect.
And I recall from earlier questions that having at least ony actual
directory available was necessary for LuaLaTeX to follow symlinks to find
fonts. It seems this also goes for regular files.
To what extent is this known behaviour? And is this *acceptable* behaviour?
Or did I even misunderstood anything and it now works by accident?
I never heard about this, but IMNSHO this is absolutely unacceptable
behaviour.
Hm, I should have searched my tex.stackexchange account before ...
https://tex.stackexchange.com/questions/132908/fontspec-search-path-in-texmfhome

From Ulrike's answer it should be clear that the issue is not limited
to fonts but general.
Now I also see that ~/texmf/tex/latex in *my* installation contains an
empty hidden directory ...

I still find it extremely strange that this should be "irremediable" ...

Urs
Post by Norbert Preining
Norbert
--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Karl Berry
2018-04-19 22:08:01 UTC
Permalink
Raw Message
I still find it extremely strange that this should be "irremediable" ...

It is not technically "irremediable". As I commented in
kpathsea/elt-dirs.c more than 25 years ago, it is perfectly easy to
avoid the optimization of a directory having 2 links => leaf => don't
check symlinks. Non-Unixy systems don't have that optimization
possibility, for instance.

However, the downside of turning off the optimization is that every
entry of every (heretofore leaf) directory will be stat(2)d. This is an
expensive operation, even on today's computers, because all that data
won't be in the disk/cpu caches, and will get promptly thrown away.

Having symlinks to outside dirs seems relatively rare to me. I don't
like the idea of making everyone pay a price to support a rare
occurrence. Creating an empty directory in a directory with symlinks
that should be followed seems like a reasonable workaround to me.

However ... I guess the counterargument is that this whole optimization
only happens in non-!! directories like TEXMFHOME, and those are most
likely small, so then it doesn't matter how much extra work the
searching does.

So, I don't know. -k


-----------------------------------------------------------------------------
kpathsea/elt-dirs.c
..
/* Should we recurse? To see if the subdirectory is a
leaf, check if it has two links (one for . and one for
..). This means that symbolic links to directories do
not affect the leaf-ness. This is arguably wrong, but
the only alternative I know of is to stat every entry
in the directory, and that is unacceptably slow.

The #ifdef here makes all this configurable at
compile-time, so that if we're using VMS directories or
some such, we can still find subdirectories, even if it
is much slower. */
#ifdef ST_NLINK_TRICK
/* With SAS/C++ 6.55 on the Amiga, stat sets the st_nlink
field to -1 for a file, or to 1 for a directory.
Cygwin 1.7 also leaves st_nlink as 1:
http://cygwin.com/ml/cygwin-developers/2008-04/msg00110.html
*/
if (links != 2)
#endif /* ST_NLINK_TRICK */
/* All criteria are met; find subdirectories. */
do_subdir (kpse, str_list_ptr, FN_STRING (name),
potential_len, post);
..
Reinhard Kotucha
2018-04-20 21:25:23 UTC
Permalink
Raw Message
Post by Urs Liska
I still find it extremely strange that this should be "irremediable" ...
It is not technically "irremediable". As I commented in
kpathsea/elt-dirs.c more than 25 years ago, it is perfectly easy to
avoid the optimization of a directory having 2 links => leaf => don't
check symlinks. Non-Unixy systems don't have that optimization
possibility, for instance.
However, the downside of turning off the optimization is that every
entry of every (heretofore leaf) directory will be stat(2)d. This is an
expensive operation, even on today's computers, because all that data
won't be in the disk/cpu caches, and will get promptly thrown away.
Having symlinks to outside dirs seems relatively rare to me. I don't
like the idea of making everyone pay a price to support a rare
occurrence. Creating an empty directory in a directory with symlinks
that should be followed seems like a reasonable workaround to me.
However ... I guess the counterargument is that this whole optimization
only happens in non-!! directories like TEXMFHOME, and those are most
likely small, so then it doesn't matter how much extra work the
searching does.
So, I don't know. -k
Hi Karl,
maybe it's better to drop the optimization because only TEXMFHOME is
concerned which can be easily added to TEXMFDBS by users if
performance matters.

In the past there had always been an ls-R file in TEXMFHOME and users
knew that they had to run texhash after adding a new file. You
removed !! from TEXMFHOME when Don Knuth reported that whenever he
invoked tex, the format file was created again and again. Removing !!
from TEXMFHOME was a quick workaround which made Don Knuth happy. But
the actual problem was that fmtutil didn't update the ls-R file.

One of your arguments at this time was that the TEXMFHOME tree is
usually quite small and the absence of an ls-R file is acceptable.
Computers are much faster now and I'm convinced that your assumption
is still true nowadays even if optimization is turned off.

My suggestion is to turn optimization off. Everything then still
works as before with a small loss of performance. But if performance
really matters, users can still activate ls-R database lookup for
TEXMFHOME at any time.

Regards,
Reinhard
--
------------------------------------------------------------------
Reinhard Kotucha Phone: +49-511-3373112
Marschnerstr. 25
D-30167 Hannover mailto:***@web.de
------------------------------------------------------------------
Arthur Reutenauer
2018-04-21 10:02:06 UTC
Permalink
Raw Message
Post by Reinhard Kotucha
My suggestion is to turn optimization off. Everything then still
works as before with a small loss of performance. But if performance
really matters, users can still activate ls-R database lookup for
TEXMFHOME at any time.
Your conclusion may be warranted, but that’s a really bad argument.
Reversing a program’s behaviour after 25 years because “users can
always” change their configuration is just gratuitous and almost never a
good idea.

Best,

Arthur
Reinhard Kotucha
2018-04-22 00:51:03 UTC
Permalink
Raw Message
Post by Arthur Reutenauer
Post by Reinhard Kotucha
My suggestion is to turn optimization off. Everything then still
works as before with a small loss of performance. But if performance
really matters, users can still activate ls-R database lookup for
TEXMFHOME at any time.
Your conclusion may be warranted, but that’s a really bad
argument. Reversing a program’s behaviour after 25 years because
“users can always” change their configuration is just gratuitous
and almost never a good idea.
I didn't suggest to change the bavior in any way. Omitting
optimization will degrade performance a little bit, almost no user
will notice it but confusion is avoided in certain situations.

Activating ls-R database lookup for TEXMFHOME is't new at all. We had
TEXMFDBS = $TEXMF (whereas $TEXMF contains TEXMFHOME) until TeX Live
2007 already.

Karl's optimization increases performance when no ls-R file is present
and the TEXMFHOME tree has to be searched. On the other hand, if an
ls-R file exists, it will be used and performance is increased
significantly, much more than whatever can be achieved by the
filesytem search optimization code in kpathsea.

Let me explain the behavior prior to TeX Live 2007. There was no ls-R
file by default because the installer isn't aware of users and thus
cannot create files in HOME. As Karl already mentioned, TEXMFHOME is
not a !!-directory which means that if a file is not in ls-R, a
directory search is performed. When a user runs texhash the very
first time, an ls-R file is created and will be used by kpathsea. If
a user adds a file but doesn't run texhash, kpathsea will scan the
filesystem. Hence existing files are always found, but faster with an
up-to-date ls-R file.

You might wonder why I wrote most of the previous paragraph in present
tense. The reason is that everything I described is still present in
kpathsea and texmf.cnf, the only change made in TL-2007 was to remove
TEXMFHOME from TEXMFDBS. I'm not convinced that this was the best
choice but no user complained about the degradation of performance so
far.

We are not discussing behavior but only performance issues and the
question whether a decrease of performance is acceptable when the
optimization code is omitted. The main problem is that the current
code confuses users and there is no way to provide a meaningful error
message.

Karl is quite conservative in respect of changes for many very good
reasons. When I asked for replacing .zip by .tar.lzma (now .tar.xz) I
did some tests and could provide numbers. These numbers are
predictable. The decrease of file size is always the same and the
amount of time needed to de-compress files does not depend on anything
else but CPU speed. Filesystem performance is negligible in this
case.

Maesuring performance of the kpathsea optimization code is almost
impossible because the filesystem is involved. Performance depends on
the physical media and thus everybody would report different numbers,
depending on whether the file system is on a rotating disk, an SD
card, a USB-stick, or even a RAM disk. Everything is possible
nowadays and thus nothing is predictable.

The only thing which is predictable is that an ls-R database lookup is
always much more efficient than everything else. In my previous mail
I only suggested to disable the optimization code in kpathsea.

But the more I think about it, I'm convinced that the pre-TL-2007
setup should be reinstated. The change was made because a single user
reported a problem which nobody else could reproduce. Nobody made the
slightest attempt to determine what actually caused the problem.

The old setup was perfect and it's a pity that it was changed for no
good reason.
Post by Arthur Reutenauer
I don't like the idea of making everyone pay a price to support a
rare occurrence.
Don't we pay a much higher price if database lookups are disabled by
default (TEXMFHOME removed from TEXMFDBS)?

Regards,
Reinhard
--
------------------------------------------------------------------
Reinhard Kotucha Phone: +49-511-3373112
Marschnerstr. 25
D-30167 Hannover mailto:***@web.de
------------------------------------------------------------------
Loading...