Discussion:
TL2017's tlpkg/installer/wget/wget.x86_64-darwin: HTTPS support not compiled in.
(too old to reply)
Munehiro Yamamoto
2018-01-29 09:55:25 UTC
Permalink
Raw Message
Hi,

I am a member of Japanese TeX Development Community.

I just found tthat the TL2017's tlpkg/installer/wget/wget.x86_64-darwin binary
does not support HTTPS protocol.

Indeed, the wget binary which I built just works with https.

$ wget https://texlive.texjp.org/current/tlnet/tlpkg/texlive.tlpdb
--2018-01-29 18:29:12--
https://texlive.texjp.org/current/tlnet/tlpkg/texlive.tlpdb
Resolving texlive.texjp.org (texlive.texjp.org)... 133.130.98.34
Connecting to texlive.texjp.org
(texlive.texjp.org)|133.130.98.34|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13218739 (13M) [application/octet-stream]
Saving to: 'texlive.tlpdb'

But, the wget binary bundled in TL2017 does not support HTTPS protocol.

$ /opt/texlive/2017/tlpkg/installer/wget/wget.x86_64-darwin
https://texlive.texjp.org/current/tlnet/tlpkg/texlive.tlpdb
https://texlive.texjp.org/current/tlnet/tlpkg/texlive.tlpdb: HTTPS
support not compiled in.

I would like to rebuild it with https support by 12mar (TL2017
frozen), if possible.

Best regards,
Munehiro

山本 宗宏 Munehiro "munepi" Yamamoto
Founder, President & CEO, Green Cherry Ltd. <***@greencherry.jp>
http://greencherry.jp/
Public Relations, Project Vine <***@vinelinux.org> http://vinelinux.org/
GPG Key Fingerprint: 61EC 85A8 5F34 5E35 91E8 8AD0 1D28 D5DE C24B 55FD
Norbert Preining
2018-01-29 11:26:28 UTC
Permalink
Raw Message
Dear Munehiro,
Post by Munehiro Yamamoto
I would like to rebuild it with https support by 12mar (TL2017
frozen), if possible.
Please rebuild and either commit a fixed version or send it to us.

Please make sure that it runs on all supported older systems, too.

All the best

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Karl Berry
2018-01-30 18:00:59 UTC
Permalink
Raw Message
I just found tthat the TL2017's
tlpkg/installer/wget/wget.x86_64-darwin binary does not support
HTTPS protocol.

As Mojca pointed out separately, this is most likely because
I actually suggested/recommended building wget --without-ssl for TL:
http://tug.org/texlive/build.html#xz

This is because, as far as I know, it has become impossible to build a
wget binary supporting ssl which either (a) runs on wider variety of
systems than just the build system and close relatives (e.g., across
Linux-based distros), or (b) is statically linked. Both of these used to
be fairly practical, but now, due to "improvements" in the computing
world, I think they are not.

Maybe I am wrong, but I'd like to see the proof before changing or
committing anything. I suspect that if you want a full-fledged wget, it
has to be provided elsewhere. You can set TL_DOWNLOAD_PROGRAM and
TL_DOWNLOAD_ARGS to override using the TL-shipped wget.

(Maybe we should change the download logic to prefer a wget found in
PATH to our shipped one, for this reason. That has its own problems,
though.) --best, karl.
Reinhard Kotucha
2018-01-31 00:16:08 UTC
Permalink
Raw Message
Post by Karl Berry
(Maybe we should change the download logic to prefer a wget found
in PATH to our shipped one, for this reason. That has its own
problems, though.) --best, karl.
If wget is searched in PATH, I recommend to search for curl if wget
isn't found. I suppose that many non-Linux Unix systems provide curl
instead of wget by default.

getnonfreefonts, for instance, is using curl whenever wget isn't
available. It's not difficult to support both programs.

Regards,
Reinhard
--
------------------------------------------------------------------
Reinhard Kotucha Phone: +49-511-3373112
Marschnerstr. 25
D-30167 Hannover mailto:***@web.de
------------------------------------------------------------------
Norbert Preining
2018-01-31 00:35:38 UTC
Permalink
Raw Message
Post by Reinhard Kotucha
available. It's not difficult to support both programs.
Patch please ...

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
George N. White III
2018-01-31 12:43:32 UTC
Permalink
Raw Message
Post by Munehiro Yamamoto
I just found tthat the TL2017's
tlpkg/installer/wget/wget.x86_64-darwin binary does not support
HTTPS protocol.
As Mojca pointed out separately, this is most likely because
http://tug.org/texlive/build.html#xz
This is because, as far as I know, it has become impossible to build a
wget binary supporting ssl which either (a) runs on wider variety of
systems than just the build system and close relatives (e.g., across
Linux-based distros), or (b) is statically linked. Both of these used to
be fairly practical, but now, due to "improvements" in the computing
world, I think they are not.
Whatever you ship would be outdated in less than a year. Users
have been known to use TL's wget for other purposes, so it falls
into the category of an "attractive nuisance". It is best to
encourage users to rely on supported crypto with regular updates.

wget without ssl support will work with fewer and fewer servers in the
future. Meanwhile, the options for having a current downloader (wget,
curl, gnurl) have improved, so it is not unreasonable to simple state
that a suitable downloader must be provided before attempting to
run the installer. Effort would be better spent supporting
curl/gnurl as many users will already have one of these. Curl, for
example, is provided by Anaconda Python (available on linux,
macOS, and Windows).
Post by Munehiro Yamamoto
Maybe I am wrong, but I'd like to see the proof before changing or
committing anything. I suspect that if you want a full-fledged wget, it
has to be provided elsewhere. You can set TL_DOWNLOAD_PROGRAM and
TL_DOWNLOAD_ARGS to override using the TL-shipped wget.
(Maybe we should change the download logic to prefer a wget found in
PATH to our shipped one, for this reason. That has its own problems,
though.) --best, karl.
The installer should check for wget, curl, and gnurl in the PATH. If none
is found,
it can point to a document where the requirements are detailed and
suitable sources of the programs for various platforms are listed.
--
George N. White III
Philip Taylor (RHUoL)
2018-01-31 13:17:12 UTC
Permalink
Raw Message
[...]  it is not unreasonable to simple state
that a suitable downloader must be provided before attempting to
run the installer.
I am sorry, George, I completely disagree.  Someone seeking to install
TeX Live can reasonably be assumed to know a little about computer
typesetting and a little about using the operating system of his/her
choice; there is no /a priori/ evidence that he or she knows anything
more.  Therefore, to expect such a user to locate, download and install
"a suitable downloader" is (IMHO) completelu unreasonable.  A suitable
downloader should form an integral part of the TeX Live installer, and
automatically come therewith.

Philip Taylor
George N. White III
2018-01-31 15:01:49 UTC
Permalink
Raw Message
[...] it is not unreasonable to simple state
that a suitable downloader must be provided before attempting to
run the installer.
I am sorry, George, I completely disagree. Someone seeking to install TeX
Live can reasonably be assumed to know a little about computer typesetting
and a little about using the operating system of his/her choice; there is
no *a priori* evidence that he or she knows anything more. Therefore, to
expect such a user to locate, download and install "a suitable downloader"
is (IMHO) completelu unreasonable. A suitable downloader should form an
integral part of the TeX Live installer, and automatically come therewith.
Philip Taylor
The need to support HTTPS is what started this thread. Downloaders that
support HTTPS brings in many new complications such as handling HTTP
redirects to HTTPS servers, cipher suites, and certificates.
Most users rely on vendors to make this work, and regular updates to keep
it going as the standards evolve. Supporting HTTPS with TeX Live's wget
may not be practical, and in any case is a large task.


I work in a government research institute, so there are many TeX
installations. I avoid Windows, but it is the "corporate standard" so I
can't completely avoid it. I use 2 applications (ESA SNAP, QGIS. both with
normal Windows installers and regular upgrades) that provide a curl.exe in
a "private" location within the application's folders. TeX Live users
often continue with their original version for years. There have been
problems where a user needed wget so copied the version from TeX Live to a
more convenient location. Then the US gov. got rid of their HTTP servers
and users ran into (difficult to diagnose) problems with wget and HTTPS
URL's. My experience says there are far too many downloaders floating
around on Windows systems already. As file servers move to https with
newer ciphers, downloaders that have not been updated stopped working.
Windows applications protect themselves by including a private downloader
(usually curl) that gets newer versions thru the application's own
updater.

Many installers based on more normal Windows standards include downloaders
and are easier for users than the TeX Live installer. If wget is not
included, and not alread installed by some other packages, there does need
to be support in the form of a list of Windows packages with good
installers that provide a downloader. I don't have anaconda python on
Windows because another work application (ArcGIS) provides a python
system. If the TeX Live installer uses an external downloader it may be
useful to provide a tool that generates a list (with versions) of already
installed downloaders and ask the user to choose the one they wish to use.
On Unix the priorities should have already been set using the PATH
variable, so the choice would be between wget, curl, and gnurl. On
Windows a search becomes more complex, so if a downloader is not found via
the PATH and maybe a few additional common locations, then the user may
have to enter the location for the downloader they wish to use.
--
George N. White III
Zdenek Wagner
2018-01-31 21:21:49 UTC
Permalink
Raw Message
Hi,

I cannot agree. I avoid Windows whenever possible but I develop SW that
must run under Windows so I have o test it. Some multiplatform libraries
state that they rely upon curl which is not available in Windows and must
be installed separately. I have just searched through the whole disk. I
found two versions of curl.exe as a part of git but my users are not
supposed to have git. So without my SW with its own curl and without git
there would be no curl.exe on my Windows 7. And I do not have wget either
(I do not have TL on it). Thus it is too risky to assume that there is
curl.exe somewhere.


Zdeněk Wagner
http://ttsm.icpf.cas.cz/team/wagner.shtml
http://icebearsoft.euweb.cz
Post by George N. White III
[...] it is not unreasonable to simple state
that a suitable downloader must be provided before attempting to
run the installer.
I am sorry, George, I completely disagree. Someone seeking to install
TeX Live can reasonably be assumed to know a little about computer
typesetting and a little about using the operating system of his/her
choice; there is no *a priori* evidence that he or she knows anything
more. Therefore, to expect such a user to locate, download and install "a
suitable downloader" is (IMHO) completelu unreasonable. A suitable
downloader should form an integral part of the TeX Live installer, and
automatically come therewith.
Philip Taylor
The need to support HTTPS is what started this thread. Downloaders that
support HTTPS brings in many new complications such as handling HTTP
redirects to HTTPS servers, cipher suites, and certificates.
Most users rely on vendors to make this work, and regular updates to keep
it going as the standards evolve. Supporting HTTPS with TeX Live's wget
may not be practical, and in any case is a large task.
I work in a government research institute, so there are many TeX
installations. I avoid Windows, but it is the "corporate standard" so I
can't completely avoid it. I use 2 applications (ESA SNAP, QGIS. both with
normal Windows installers and regular upgrades) that provide a curl.exe in
a "private" location within the application's folders. TeX Live users
often continue with their original version for years. There have been
problems where a user needed wget so copied the version from TeX Live to a
more convenient location. Then the US gov. got rid of their HTTP servers
and users ran into (difficult to diagnose) problems with wget and HTTPS
URL's. My experience says there are far too many downloaders floating
around on Windows systems already. As file servers move to https with
newer ciphers, downloaders that have not been updated stopped working.
Windows applications protect themselves by including a private downloader
(usually curl) that gets newer versions thru the application's own
updater.
Many installers based on more normal Windows standards include downloaders
and are easier for users than the TeX Live installer. If wget is not
included, and not alread installed by some other packages, there does need
to be support in the form of a list of Windows packages with good
installers that provide a downloader. I don't have anaconda python on
Windows because another work application (ArcGIS) provides a python
system. If the TeX Live installer uses an external downloader it may be
useful to provide a tool that generates a list (with versions) of already
installed downloaders and ask the user to choose the one they wish to use.
On Unix the priorities should have already been set using the PATH
variable, so the choice would be between wget, curl, and gnurl. On
Windows a search becomes more complex, so if a downloader is not found via
the PATH and maybe a few additional common locations, then the user may
have to enter the location for the downloader they wish to use.
--
George N. White III
Akira Kakuto
2018-01-31 22:40:29 UTC
Permalink
Raw Message
Hi,
I cannot agree. I avoid Windows whenever possible but I develop SW that must run under Windows so I have o test it.
Some multiplatform libraries state that they rely upon curl which is not available in Windows and must be installed
separately. I have just searched through the whole disk. I found two versions of curl.exe as a part of git but my users are
not supposed to have git. So without my SW with its own curl and without git there would be no curl.exe on my Windows
7. And I do not have wget either (I do not have TL on it). Thus it is too risky to assume that there is curl.exe somewhere.
In
CTAN/systems/win32/w32tex
and
CTAN/systems/win32/w32tex/win64
you find context.tar.xz and context-w64.tar.xz
where curl.exe (32bit and 64bit) built with Visual Studio 2015
is provided (libs are statically linked):

curl 7.58.0 (i386-pc-win32) libcurl/7.58.0 OpenSSL/1.0.2n zlib/1.2.11 libssh2/1.8.0
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz HTTPS-proxy

Best,
Akira
Philip Taylor (RHUoL)
2018-01-31 22:50:59 UTC
Permalink
Raw Message
Post by Akira Kakuto
In
CTAN/systems/win32/w32tex
and
CTAN/systems/win32/w32tex/win64
you find context.tar.xz and context-w64.tar.xz
                                  _   _ ____  _
  Project                     ___| | | |  _ \| |
                             / __| | | | |_) | |
                            | (__| |_| |  _ <| |___
                             \___|\___/|_| \_\_____|
NAME
       curl - transfer a URL
SYNOPSIS
       curl [options] [URL...]
DESCRIPTION
       curl  is  a tool to transfer data from or to a server, using
one of the
       supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP,
HTTPS,  IMAP,
       IMAPS,  LDAP,  LDAPS,  POP3,  POP3S,  RTMP, RTSP, SCP, SFTP,
SMB, SMBS,
       SMTP, SMTPS, TELNET and TFTP). The command is designed to work 
without
       user interaction.
       curl offers a busload of useful tricks like proxy support, user
authen-
       tication, FTP upload, HTTP post, SSL connections, cookies,
file  trans-
       fer  resume,  Metalink,  and more. As you will see below, the
number of
       features will make your head spin!
       curl is powered by  libcurl  for  all  transfer-related
features.  See
       libcurl(3) for details.
** Phil.
Reinhard Kotucha
2018-01-31 22:30:03 UTC
Permalink
Raw Message
[...]  it is not unreasonable to simple state
that a suitable downloader must be provided before attempting to
run the installer.  
I am sorry, George, I completely disagree.  Someone seeking to
install TeX Live can reasonably be assumed to know a little about
computer typesetting and a little about using the operating system
of his/her choice; there is no a priori evidence that he or she
knows anything more.  Therefore, to expect such a user to locate,
download and install "a suitable downloader" is (IMHO) completelu
unreasonable.  A suitable downloader should form an integral part
of the TeX Live installer, and automatically come therewith.
TeX Live was supposed to always work out-of-the-box from the very
beginning and there are no intentions to change this behaviour,
at least not deliberately.

On Unix I don't expect any problems. If a particular program isn't
found, a user can deduce from the error message which package is
missing and use the package manager in order to download and install
the package within a few seconds.

On Windows the situation is worse. The error message says which
program is missing. The user then has to determine the location of
the package in the internet, download it, install it, ...
This is absolutely inacceptable, of course.

But I have one very important question:

Does getnonfreefonts still work on Windows?

A few years ago a concerned user asked me to make downloading the
remote part of the script more secure. I then replaced http://tug.org
by https://tug.org in order to activate SSL encryption.

On Windows, getnonfreefonts entirely depends on the wget binary
shipped with TeX Live. Nobody complained so far, hence I assume that
tlpkg/installer/wget/wget.exe supports HTTPS already.

If somebody can confirm that getnonfreefonts works on Windows at all,
we can leave the Windows part of TeX Live as it is and save a lot of
time.

Download

https://www.tug.org/fonts/getnonfreefonts/install-getnonfreefonts

to an arbitrary directory, move into this directory, run

texlua install-getnonfreefonts

and then execute

getnonfreefonts --sys --lsfonts

Does getnonfreefonts work on Windows?

Regards,
Reinhard
--
------------------------------------------------------------------
Reinhard Kotucha Phone: +49-511-3373112
Marschnerstr. 25
D-30167 Hannover mailto:***@web.de
------------------------------------------------------------------
Philip Taylor (RHUoL)
2018-01-31 22:33:07 UTC
Permalink
Raw Message
Post by Reinhard Kotucha
Does getnonfreefonts work on Windows?
P:\>cd gnff
P:\GNFF>texlua install-getnonfreefonts
Detected System: win32
Detected Installation: D:/TeX/Live/2017
mkdir D:/TeX/Live/2017/texmf-dist/scripts/getnonfreefonts
...            [done]
Installing texmf-dist/scripts/getnonfreefonts/getnonfreefonts.pl
...     [done]
Installing texmf-dist/doc/man/man1/getnonfreefonts.1
...                 [done]
Installing texmf-dist/doc/man/man1/getnonfreefonts.man1.pdf
...          [done]
md5sum: a9e772165e8fdb620bcf9c75c17facda getnonfreefonts.pl
...            [ok]
md5sum: 49be4444054d85b6037d237552a7cea1 getnonfreefonts.1
...             [ok]
md5sum: f825d523d686dbecdc787535b40f09d0 getnonfreefonts.man1.pdf
...      [ok]
Creating wrapper in 'bin/win32' ...
 file(s) copied.
  [done]
mktexlsr: Updating D:/TeX/Live/2017/texmf-dist/ls-R...
mktexlsr: Updated D:/TeX/Live/2017/texmf-dist/ls-R.
mktexlsr: Done.
P:\GNFF>getnonfreefonts --sys --lsfonts
--2018-01-31 22:32:18--
https://www.tug.org/~kotucha/getnonfreefonts/getfont.p
Resolving www.tug.org... 91.121.174.77
Connecting to www.tug.org|91.121.174.77|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32681 (32K)
Saving to: 'getfont.pl'
getfont.pl          100%[===================>]  31.92K --.-KB/s    in
0.03s
2018-01-31 22:32:18 (1.01 MB/s) - 'getfont.pl' saved [32681/32681]
------------------------------------------------
Installation directory: D:/TeX/Live/TeX-MF/Local
------------------------------------------------
arial-urw      Arial (URW) [not installed]
classico       Classico (URW) [not installed]
dayroman       Day Roman (Apostrophiclabs) [not installed]
eurofont       Euro Symbols (Adobe) [not installed]
gandhi         Gandhi (Librerias Gandhi S.A. de C.V.) [not installed]
garamond       GaramondNo8 (URW) [not installed]
garamondx      GaramondNo8 Expert (URW & Michael Sharpe)        [not
installed]
lettergothic   LetterGothic (URW) [not installed]
literaturnaya  Literatunaya (Paragraph) [not installed]
luximono       Luxi Mono (Bigelow & Holmes)                     [not
installed]
vntex-nonfree  VnTeX nonfree (URW & Han The Thanh)              [not
installed]
webomints      Webomints (Galapagos Design Group) [not installed]
P:\GNFF>
Reinhard Kotucha
2018-01-31 23:34:53 UTC
Permalink
Raw Message
Post by Reinhard Kotucha
Does getnonfreefonts work on Windows?
[stuff omitted]
Thank you very much, Phil. I deduce that we don't have to touch the
Windows specific code in TeX Live right now because HTTPS is supported
already. Good to know. It saves us a lot of time.

Regards,
Reinhard

-
------------------------------------------------------------------
Reinhard Kotucha Phone: +49-511-3373112
Marschnerstr. 25
D-30167 Hannover mailto:***@web.de
------------------------------------------------------------------
Akira Kakuto
2018-01-31 22:49:46 UTC
Permalink
Raw Message
Hi,
Post by Reinhard Kotucha
On Windows, getnonfreefonts entirely depends on the wget binary
shipped with TeX Live. Nobody complained so far, hence I assume that
tlpkg/installer/wget/wget.exe supports HTTPS already.
I built wget for windows and wget --version shows

-cares +digest -gpgme +https +ipv6 -iri +large-file -metalink -nls
-ntlm +opie -psl +ssl/gnutls

So maybe it supports HTTPS.

Best,
Akira
Nelson H. F. Beebe
2018-01-31 00:45:13 UTC
Permalink
Raw Message
Reinhard Kotucha writes today
Post by Reinhard Kotucha
If wget is searched in PATH, I recommend to search for curl if wget
isn't found. I suppose that many non-Linux Unix systems provide curl
instead of wget by default.
I suggest that if you look for curl, then also look for gnurl, a new
(December 2017) GNU project that is a fork of curl that removes
support for older protocols, and adds support for more cryptographic
backends. See

ftp://ftp.gnu.org/gnu/gnunet/

gnurl is not yet in a many GNU/Linux (or other UNIX flavor)
distributions, but I expect it will be added over the next few months.

You can use it like curl:

gnurl remote-URL > local-file


-------------------------------------------------------------------------------
- Nelson H. F. Beebe Tel: +1 801 581 5254 -
- University of Utah FAX: +1 801 581 4148 -
- Department of Mathematics, 110 LCB Internet e-mail: ***@math.utah.edu -
- 155 S 1400 E RM 233 ***@acm.org ***@computer.org -
- Salt Lake City, UT 84112-0090, USA URL: http://www.math.utah.edu/~beebe/ -
-------------------------------------------------------------------------------
Loading...