Discussion:
Tutorial for setting up own TeX Live package repositories seems outdated
Add Reply
M.Eng. René Schwarz
2018-02-11 18:12:29 UTC
Reply
Permalink
Raw Message
Hi,


I am currently trying to set up an own TeX Live package repository for
the distribution of some packages not available in the standard
repository. For this purpose, I have read the tutorial available in the
TeX Live Subversion repository at

</trunk/Master/tlpkg/doc/repository-setup.txt>

in r22216. I noticed that the information in section 4 seems outdated
w.r.t. the options for `tl-update-tlpdb' presented there. The script
living at

</trunk/Master/tlpkg/bin/tl-update-tlpdb>

in most recent r45618 does not contain the optional parameters
`-tlpdb-options-from-tlpsrc', `-with-win-pattern-warning', and
`-no-revision-check' anymore. As far as I can see,
`-with-win-pattern-warning' has been renamed to
`-with-w32-pattern-warning' and `-no-revision-check' to
`-no-reverse-revision-check', both with r15651. The
`-tlpdb-options-from-tlpsrc' option was introduced with r18296 and has
been removed completely with r45618.

In addition to that, I am facing problems with understanding section 6
of the document. I prepared a directory (``ROOT'') with the following
layout:

[ROOT]/
archive
texmf-dist
tlpkg
bin
TeXLive
tlpsrc
00texlive.autopatterns.tlpsrc
00texlive.config.tlpsrc
00texlive.installation.tlpsrc
[PACKAGE_NAME].tlpsrc
texlive.tlpdb

The folder `texmf-dist' contains files for an example package in TDS
layout. After running `perl [ROOT]/tlpkg/bin/tl-update-tlpdb
--from-files', the `[ROOT]/tlpkg/texlive.tlpdb' file is being created.
However, I receive the following message during this run:

Cannot determine type of tlpdb from [ROOT]!

A subsequent run of `tl-update-containers' always leads to the following
error:

Cannot determine type of tlpdb from [ROOT]!
cannot find tlpdb in [ROOT] at [ROOT]/tlpkg/bin/tl-update-containers
line 75.

I am stuck here. Certainly I am doing something wrong, but I was not
able to figure out why the type of the tlpdb could not be determined.
Any help is appreciated. Thank you very much in advance!

P.S.: I am not a subscriber of the list; replies with my e-mail address
in CC would be nice.
--
Sincerely yours,


M.Eng. *René Schwarz*
https://www.rene-schwarz.com
Norbert Preining
2018-02-11 23:55:10 UTC
Reply
Permalink
Raw Message
Hi René
Post by M.Eng. René Schwarz
in r22216. I noticed that the information in section 4 seems outdated
That could easily be, I haven't updated it in ages.

I will update it to a better layout, since I am using it anyway in so
many projects.
Post by M.Eng. René Schwarz
Cannot determine type of tlpdb from [ROOT]!
mkdir texmf-dist/web2c/
touch texmf-dist/web2c/.dont-delete-this-dir (or anything else)

The current code determines the type of archive (.tar.xz, or files)
automatically and just bumps out because it cannot find the
texmf-dist/web2c.

Here is parts of my deployment script I use for some repos:

TLCHECKOUT=${TLCHECKOUT:-/home/norbert/Development/TeX/texlive.git}
TLNETDEST=${TLNETDEST:-/home/norbert/Domains/server/texlive.info/contrib/2017}
TLCATALOGUE=${TLCATALOGUE:-/home/norbert/Development/TeX/texcatalogue-svn}
# update tlpdb
$TLCHECKOUT/Master/tlpkg/bin/tl-update-tlpdb \
-with-w32-pattern-warning -from-git \
--catalogue=$TLCATALOGUE \
--master=`pwd`
# update containers
$TLCHECKOUT/Master/tlpkg/bin/tl-update-containers \
-master `pwd` \
-location $TLNETDEST \
-gpgcmd `pwd`/tl-sign-file \
-all # sometimes we need -recreate

Note that I am running from git - git mode is supported since some time.
The private signing file is a simple adaption of the one in the repo
using my private signing key. I need to factor out the arguments there:

if test $# -ne 1; then
echo "$0: Exactly one argument must be given, the file to sign." >&2
exit 1
fi

# remove previous signature
rm -f "$1.asc"

prg=gpg
if ! [ "x$TL_GNUPG" = "x" ] ; then
# use the environment variable TL_GNUPG
prg=$TL_GNUPG
fi

# sign
$prg --armor \
--detach-sign \
--local-user 0xEC00B8DAD32266AA \
"$1"


All the best

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
M.Eng. René Schwarz
2018-02-13 21:09:13 UTC
Reply
Permalink
Raw Message
Hi Norbert,


thank you so much for your fast answer! Creating the `texmf-dist/web2c'
directory indeed solved the issue for me.

However, there was another issue coming up: The script

</trunk/Master/tlpkg/installer/config.guess>

uses the Windows line ending style (`\r\n'), which causes the
`tl-update-containers` script run to fail in my environment. Since it is
a shell script, I would propose to convert the file to Unix line endings.

After changing that, I was able to create the repository as desired.
Thank you once again for your fast and competent support.


Kind regards,
René
--
Sincerely yours,


M.Eng. *René Schwarz*
https://www.rene-schwarz.com
Reinhard Kotucha
2018-02-13 23:13:43 UTC
Reply
Permalink
Raw Message
Post by M.Eng. René Schwarz
However, there was another issue coming up: The script
</trunk/Master/tlpkg/installer/config.guess>
uses the Windows line ending style (`\r\n'), which causes the
`tl-update-containers` script run to fail in my environment. Since
it is a shell script, I would propose to convert the file to Unix
line endings.
Hi René,
here the file /trunk/Master/tlpkg/installer/config.guess has Unix line
endings. Please check the files

/etc/subversion/config

and

~/subversion/config

on your system. Look for "svn:eol-style".

Karl wrote an SVN config file especially for TeX Live. It can be
downloaded from tug.org but I'm wondering whether it makes sense to
simply add it to the TeX Live SVN repository. It's easier to keep it
up-to-date then and SVN users don't have to fiddle with clumsy web
browsers. Users could either copy it to ~/subversion or just create a
symlink.

I don't know whether subversion supports distinct config files for
particular projects but that would be a great thing.

Regards,
Reinhard
--
------------------------------------------------------------------
Reinhard Kotucha Phone: +49-511-3373112
Marschnerstr. 25
D-30167 Hannover mailto:***@web.de
------------------------------------------------------------------
M.Eng. René Schwarz
2018-02-18 09:59:43 UTC
Reply
Permalink
Raw Message
Hi Reinhard,


thank you for your answer!
Post by Reinhard Kotucha
on your system. Look for "svn:eol-style".
The `svn:eol-style` property is set to `native` for this file. As I am
working with a Windows SVN client, I get Windows line endings for this file.
Post by Reinhard Kotucha
Karl wrote an SVN config file especially for TeX Live. It can be
downloaded from tug.org but I'm wondering whether it makes sense to
simply add it to the TeX Live SVN repository. It's easier to keep it
up-to-date then and SVN users don't have to fiddle with clumsy web
browsers. Users could either copy it to ~/subversion or just create a
symlink.
To be honest, I don't want to fiddle around with my SVN client
configuration file since changes there apply to all repositories I work
with, not only for the TeX Live one.
Post by Reinhard Kotucha
I don't know whether subversion supports distinct config files for
particular projects but that would be a great thing.
With Subversion v1.8, the `svn:auto-props` property has been introduced,
which --- according to my understanding --- allows to set all config
options in Karl's SVN config file on the root of the TeX Live SVN
repository. This way, the config options only apply to this particular
repository and the user does not have to cope with manipulating the
global SVN config file manually, which potentially introduces negative
side effects for other projects.

I would recommend considering to add this `svn:auto-props` property to
the TeX Live SVN repository root and to mark the solution with the
custom SVN config file as obsolete.


Kind regards,
René
--
Sincerely yours,


M.Eng. *René Schwarz*
***@rene-schwarz.com
https://www.rene-schwarz.com
Norbert Preining
2018-02-13 23:35:50 UTC
Reply
Permalink
Raw Message
Hi René
Post by M.Eng. René Schwarz
thank you so much for your fast answer! Creating the `texmf-dist/web2c'
directory indeed solved the issue for me.
Good to hear. I did some rework and simplification over the last two
days, please take a look at the current status of the document, too.
Thanks.
Post by M.Eng. René Schwarz
uses the Windows line ending style (`\r\n'), which causes the
As Reinhard said, this is a setup error on your side. Please see
http://tug.org/texlive/svn/setup.html Step 5

Best

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
M.Eng. René Schwarz
2018-02-18 10:13:20 UTC
Reply
Permalink
Raw Message
Hi Norbert,
Post by Norbert Preining
Good to hear. I did some rework and simplification over the last two
days, please take a look at the current status of the document, too.
Thanks.
thank you for updating the tutorial document. I think it is much clearer
now and also reflects the most recent changes in the script infrastructure.
Post by Norbert Preining
As Reinhard said, this is a setup error on your side. Please see
http://tug.org/texlive/svn/setup.html Step 5
Please see my answer to Reinhard's post.

Again, thank you so much!


Kind regards,
René
--
Sincerely yours,


M.Eng. *René Schwarz*
***@rene-schwarz.com
https://www.rene-schwarz.com
Karl Berry
2018-02-12 00:05:45 UTC
Reply
Permalink
Raw Message
if ! [ "x$TL_GNUPG" = "x" ] ; then

test ! ... is more portable than ! [ ...
(on two counts). Not that it's likely to matter. -k

$prg --armor \

"$prg"
(ditto)
Karl Berry
2018-02-13 23:28:12 UTC
Reply
Permalink
Raw Message
</trunk/Master/tlpkg/installer/config.guess>

If you check it out on a Windows system, then you get Windows line
endings (for other files too), due to svn:eol-style=native, but shells
on Windows should be able to deal with that. It can be changed, but it
doesn't seem like it should be necessary. What is your situation? Sorry
if you already said. -k
Zdenek Wagner
2018-02-13 23:37:32 UTC
Reply
Permalink
Raw Message
Hi all,

the config file is not needed for normal users. The properties are stored
with the files in the repository and used during checkout. The config file
is used only when adding a new file to the working copy, it then gets the
properties set in the config.

Zdeněk Wagner
http://ttsm.icpf.cas.cz/team/wagner.shtml
http://icebearsoft.euweb.cz
Post by M.Eng. René Schwarz
</trunk/Master/tlpkg/installer/config.guess>
If you check it out on a Windows system, then you get Windows line
endings (for other files too), due to svn:eol-style=native, but shells
on Windows should be able to deal with that. It can be changed, but it
doesn't seem like it should be necessary. What is your situation? Sorry
if you already said. -k
M.Eng. René Schwarz
2018-02-18 10:05:59 UTC
Reply
Permalink
Raw Message
Hi Karl,
Post by Karl Berry
If you check it out on a Windows system, then you get Windows line
endings (for other files too), due to svn:eol-style=native, but shells
on Windows should be able to deal with that. It can be changed, but it
doesn't seem like it should be necessary. What is your situation? Sorry
if you already said. -k
I am working with TortoiseSVN and Cygwin on Windows. Since the
`config.guess` file is a shell script --- at least to my knowledge ---
it cannot be executed natively as Windows does not provide a Bourne-like
shell. At least the shell of Cygwin cannot cope with the Windows line
endings, and also it seems strange for me that other shell
implementations for Windows would.

It would make more sense to me to set `svn:eol-style=LF` for this file.


Kind regards,
René
--
Sincerely yours,


M.Eng. *René Schwarz*
***@rene-schwarz.com
https://www.rene-schwarz.com
Karl Berry
2018-02-18 23:35:56 UTC
Reply
Permalink
Raw Message
Ok, I did svn propset svn:eol-style LF
on tlpkg/installer/config.guess (among others).

I don't understand why no Cygwin (etc.) users had this problem
before. But whatever. --thanks, karl.
Mojca Miklavec
2018-02-19 10:11:57 UTC
Reply
Permalink
Raw Message
Post by Karl Berry
Ok, I did svn propset svn:eol-style LF
on tlpkg/installer/config.guess (among others).
I don't understand why no Cygwin (etc.) users had this problem
before. But whatever. --thanks, karl.
I don't think that we have that many users setting up their own TL
package repository on a server running cygwin.

(It could also be that cygwin's own subversion sets EOL differently
than when the user does a checkout with a "native" subversion
installation or that users have different defaults set, but that's
just a super blind guess.)

Mojca
David Carlisle
2018-02-19 11:08:28 UTC
Reply
Permalink
Raw Message
On 19 February 2018 at 10:11, Mojca Miklavec
Post by Mojca Miklavec
Post by Karl Berry
Ok, I did svn propset svn:eol-style LF
on tlpkg/installer/config.guess (among others).
I don't understand why no Cygwin (etc.) users had this problem
before. But whatever. --thanks, karl.
I don't think that we have that many users setting up their own TL
package repository on a server running cygwin.
(It could also be that cygwin's own subversion sets EOL differently
than when the user does a checkout with a "native" subversion
installation or that users have different defaults set, but that's
just a super blind guess.)
yes if you use cygwin's svn then it uses the line endings specified in
the cygwin setup (which is almost always
a single #10) and cygwin's bash will understand the same line endings.

David
Post by Mojca Miklavec
Mojca
M.Eng. René Schwarz
2018-02-19 12:03:29 UTC
Reply
Permalink
Raw Message
Dear Mojca,
Post by Mojca Miklavec
I don't think that we have that many users setting up their own TL
package repository on a server running cygwin.
just to clarify: I am not running a server using Cygwin. I am just
preparing the package repository on a Windows machine, which is working
perfectly fine. The repository contents are then distributed via a Linux
server.

I also think that this is not the point. The question should be if it
makes sense to set the `svn:eol-style` to native if all
Bourne-compatible shells expect Unix line endings.


Kind regards,
René
--
Sincerely yours,


M.Eng. *René Schwarz*
***@rene-schwarz.com
https://www.rene-schwarz.com
Mojca Miklavec
2018-02-19 13:21:33 UTC
Reply
Permalink
Raw Message
Post by M.Eng. René Schwarz
Dear Mojca,
Post by Mojca Miklavec
I don't think that we have that many users setting up their own TL
package repository on a server running cygwin.
just to clarify: I am not running a server using Cygwin. I am just
preparing the package repository on a Windows machine, which is working
perfectly fine. The repository contents are then distributed via a Linux
server.
OK, but my modified statement still stands. I still don't think we
have that many users running scripts to set up their own TL package
repository on cygwin :) :) :)
Post by M.Eng. René Schwarz
I also think that this is not the point. The question should be if it
makes sense to set the `svn:eol-style` to native if all
Bourne-compatible shells expect Unix line endings.
Just for "completeness". I did a subversion checkout in msys2 about a
month ago. The subversion installed by msys2 (/usr/bin/svn) apparently
resulted in linux-style line endings despite running subversion on a
Windows machine. The config-guess script returns x86_64-pc-mingw64
(not that it would be relevant in any way).

But I tried running unix2dos and the script keeps working even after that:


$ file ./source/build-aux/config.guess
./source/build-aux/config.guess: POSIX shell script, ASCII text executable

$ ./source/build-aux/config.guess
x86_64-pc-mingw64

$ unix2dos.exe ./source/build-aux/config.guess
unix2dos: converting file ./source/build-aux/config.guess to DOS format...

$ ./source/build-aux/config.guess
x86_64-pc-mingw64

$ file ./source/build-aux/config.guess
./source/build-aux/config.guess: POSIX shell script, ASCII text
executable, with CRLF line terminators


Karl was just wondering why you would be the first one running into
that problem.
I agree that it makes no sense to set 'svn:eol-style' for shell
scripts, but I would also tend to agree that one should file a bug in
cygwin's bug tracker if it's not capable of dealing with windows-style
line endings.

On msys2:

$ sh --version
GNU bash, version 4.4.19(1)-release (x86_64-pc-msys)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>


A note to you though: if you happen to prepare the repository on
windows machine with the same settings as you are using now, linux
users will probably get other/regular files from subversion
repositories with windows line endings. Whether or not that's ok /
desirable is up to you of course. (Super off-topic: I had lots of
troubles with line endings after converting a git repository to
mercurial and VisualStudio helpfully insisting in auto-converting line
endings in all edited files. It's so super annoying if some tools
ignore line endings and others insist in using exactly what they
consider the correct form. Not that Windows, HG or VS would be my
favourite choice of tools, but sometimes one needs compromises.)

Mojca

PS: I haven't used or checked or touched cygwin since at least a
decade or longer, but msys2 is awesome and I have a feeling that it
plays more nicely with the system (no X11 though).

Loading...