Discussion:
betterbib
(too old to reply)
Nico Schlömer
2017-10-11 22:07:57 UTC
Permalink
Raw Message
Hi everyone,

betterbib [1], a (Python) command line tool of mine which helps improving
and maintaining your BibTeX files, has picked up some popularity recently.
Perhaps it'd make a good addition to TeXLive as well. If you're interested,
let me know if any modification would have to be made.

Cheers,
Nico

[1] https://github.com/nschloe/betterbib
Karl Berry
2017-10-12 22:56:18 UTC
Permalink
Raw Message
Hi Nico - thanks for writing.

https://github.com/nschloe/betterbib

First, it needs to be available through CTAN. (ctan.org/upload)

Also, it needs to be self-locating, though I'm not sure if you need to
do anything special for this with Python.

That is, the installation on TL would be

/arbitrary/tlroot/bin/PLATFORM/betterbib
-> ../../texmf-dist/scripts/betterbib/betterbib.py

with the subsidiary .py files installed in that same
../texmf-dist/scripts/betterbib/ dir. So the main .py file needs to
find them there, at runtime.

In particular, it must not be required to run Python's usual setup.py
(which I see in your repo) at any point, since the paths are not known.

For an existing example of multi-file python script in TL, see the
pythontex package. (I hope it works.)

Best,
Karl
Nico Schlömer
2017-10-13 09:36:12 UTC
Permalink
Raw Message
Thanks Karl for the info.
Post by Karl Berry
First, it needs to be available through CTAN. (ctan.org/upload)
Is it possible to update the package via an API or would I have to go to
the website manually each time I release a new version?

Cheers,
Nico
Post by Karl Berry
Hi Nico - thanks for writing.
https://github.com/nschloe/betterbib
First, it needs to be available through CTAN. (ctan.org/upload)
Also, it needs to be self-locating, though I'm not sure if you need to
do anything special for this with Python.
That is, the installation on TL would be
/arbitrary/tlroot/bin/PLATFORM/betterbib
-> ../../texmf-dist/scripts/betterbib/betterbib.py
with the subsidiary .py files installed in that same
../texmf-dist/scripts/betterbib/ dir. So the main .py file needs to
find them there, at runtime.
In particular, it must not be required to run Python's usual setup.py
(which I see in your repo) at any point, since the paths are not known.
For an existing example of multi-file python script in TL, see the
pythontex package. (I hope it works.)
Best,
Karl
Norbert Preining
2017-10-13 09:47:28 UTC
Permalink
Raw Message
Hi Nico,
Post by Nico Schlömer
Is it possible to update the package via an API or would I have to go to
the website manually each time I release a new version?
You mean an API to get betterbib into our subversion server? No, that is
not possible.

And yes, every update you want to get into TeX Live needs to go through CTAN.
This is how everyone is doing it.

OTOH, if you meant that the API is used to update the actual installation on
the *user* computer, then well, in principle you *could* try to do it, but
in many cases the user running the script will not have *write* permissions
to the location where betterbib is installed. (Unix root/user, Win admin perm...)
So I think this is not a good idea.

All the best

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Nico Schlömer
2017-10-13 10:03:54 UTC
Permalink
Raw Message
You mean an API to get betterbib into our subversion server? No, that is not
possible.

That's what I meant.

Whenever I publish a new version of betterbib, I use a simple Makefile to
push the data upstream. This works

* for releases on GitHub [1] (via `git tag v1.2.3 && git push --tags`) as
well as
* for releases on pypi [2] (via `python setup.py bdist_wheel --universal
&& gpg --detach-sign -a dist/* && twine upload dist/*`).

This makes releasing new versions really easy for me. Having to manually
open a browser, navigate to a website, copy-and-pasting some information
from the package into input fields, and clicking a submit button would be a
step back, and I can already see myself forget doing it.

Perhaps this is a feature you'd like to add sometime in the future.

Cheers,
Nico

[1] https://github.com/nschloe/betterbib/releases
[2] https://pypi.python.org/pypi/betterbib
Hi Nico,
Post by Nico Schlömer
Is it possible to update the package via an API or would I have to go to
the website manually each time I release a new version?
You mean an API to get betterbib into our subversion server? No, that is
not possible.
And yes, every update you want to get into TeX Live needs to go through CTAN.
This is how everyone is doing it.
OTOH, if you meant that the API is used to update the actual installation on
the *user* computer, then well, in principle you *could* try to do it, but
in many cases the user running the script will not have *write* permissions
to the location where betterbib is installed. (Unix root/user, Win admin perm...)
So I think this is not a good idea.
All the best
Norbert
--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Norbert Preining
2017-10-13 11:43:31 UTC
Permalink
Raw Message
Hi Nico,
Post by Nico Schlömer
Whenever I publish a new version of betterbib, I use a simple Makefile to
push the data upstream. This works
I know, I know. I do the same for several of my projects...
Post by Nico Schlömer
from the package into input fields, and clicking a submit button would be a
step back, and I can already see myself forget doing it.
Perhaps this is a feature you'd like to add sometime in the future.
Nothing I can influence. I am not CTAN maintainer ... we are doing the TeX Live
stuff, and pull from CTAN. But CTAN maintainers is a different group.

I think they might be open to suggestions, but since all are volunteers (not like
github etc staff) implementing things takes time, or patches from contributors ;-)

All the best

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Petra Ruebe-Pugliese
2017-10-13 11:59:03 UTC
Permalink
Raw Message
On Friday 13 October 2017 at 20:43:31 +0900 Norbert Preining <***@preining.info> wrote:

[WRT automated uploads to CTAN:]
Post by Norbert Preining
I think they might be open to suggestions, but since all are volunteers (not like
github etc staff) implementing things takes time, or patches from contributors ;-)
AFAIK our webmaster has been thinking about such an interface
for a _long_ time, but it seems that this is more complicated
than expected, and then the backend doesn't seem to be
100% stable either ...

In any case: given that all the uploads are checked by hand
(although with the help of some tools) and then installed one
by one, I would be a little scared of uploads arriving with
a github-like frequency ... ;-)

All the best
Petra (from CTAN)
Norbert Preining
2017-10-13 12:34:30 UTC
Permalink
Raw Message
Hi Petra,
Post by Petra Ruebe-Pugliese
AFAIK our webmaster has been thinking about such an interface
for a _long_ time, but it seems that this is more complicated
than expected, and then the backend doesn't seem to be
I agree completely. I wasn't sure, but my guess was that this is
overly complicated.
Post by Petra Ruebe-Pugliese
by one, I would be a little scared of uploads arriving with
a github-like frequency ... ;-)
haha, indeed ;-) TeX dev pace is much more relaxed.

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Paulo Roberto Massa Cereda
2017-10-13 13:19:03 UTC
Permalink
Raw Message
Post by Norbert Preining
haha, indeed ;-) TeX dev pace is much more relaxed.
Emphasis on *relaxed*. :) arara 4.0 is on the development branch for two
years, maybe I will do something next year for the upcoming TUG in
Brazil (or not, I want to save energy too). :)
Nico Schlömer
2017-10-13 15:31:01 UTC
Permalink
Raw Message
Post by Norbert Preining
TeX dev pace is much more relaxed.
Well then, betterbib is probably not for CTAN. Thanks everyone for the
input!

Cheers,
Nico
Post by Norbert Preining
Hi Petra,
Post by Petra Ruebe-Pugliese
AFAIK our webmaster has been thinking about such an interface
for a _long_ time, but it seems that this is more complicated
than expected, and then the backend doesn't seem to be
I agree completely. I wasn't sure, but my guess was that this is
overly complicated.
Post by Petra Ruebe-Pugliese
by one, I would be a little scared of uploads arriving with
a github-like frequency ... ;-)
haha, indeed ;-) TeX dev pace is much more relaxed.
Norbert
--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Norbert Preining
2017-10-13 15:47:04 UTC
Permalink
Raw Message
Well, it is up to you whether you want your software in TeX Live, we cannot follow the wishes and desires of each of the few thousands of contributors.

Good luck with your endeavors

Norbert


--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Zdenek Wagner
2017-10-13 16:16:53 UTC
Permalink
Raw Message
Post by Norbert Preining
Well, it is up to you whether you want your software in TeX Live, we
cannot follow the wishes and desires of each of the few thousands of
contributors.
Good luck with your endeavors
You should also take into account that TeX Live is intended for users, not
for developers. If I see that a toll has too many releases within a short
time, I became afraid that it is released in a buggy state and hence I
cannot rely that it works. Or it may just evolve so rapidly which in turn
means that there is a high risk that after an update my work chains will
stop to work. I think that developers should first think what is really
necessary to release to public and what is just intended for developers
needing the bleeding edge and for beta testers..
Post by Norbert Preining
Norbert
Zdeněk Wagner
http://ttsm.icpf.cas.cz/team/wagner.shtml
http://icebearsoft.euweb.cz
--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Uwe Ziegenhagen
2017-10-13 17:26:31 UTC
Permalink
Raw Message
I am the guy who has been packaging KOMA-Script for a while now. I have
automated so much of the upload process it takes me way less than 5 minutes
to get it done. And KOMA-Script consists of 1000s of files. Why don't you
create a make target which creates the necessary files for CTAN and upload
each major release. Your work seems to be pretty cool, it could be helpful
for a lot of users.

The TeX community has given you TeX/LaTeX, by providing your work to the
community you give something back. :-)

Uwe
Post by Zdenek Wagner
Post by Norbert Preining
Well, it is up to you whether you want your software in TeX Live, we
cannot follow the wishes and desires of each of the few thousands of
contributors.
Good luck with your endeavors
You should also take into account that TeX Live is intended for users, not
for developers. If I see that a toll has too many releases within a short
time, I became afraid that it is released in a buggy state and hence I
cannot rely that it works. Or it may just evolve so rapidly which in turn
means that there is a high risk that after an update my work chains will
stop to work. I think that developers should first think what is really
necessary to release to public and what is just intended for developers
needing the bleeding edge and for beta testers..
Post by Norbert Preining
Norbert
Zdeněk Wagner
http://ttsm.icpf.cas.cz/team/wagner.shtml
http://icebearsoft.euweb.cz
--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
--
Dr. Uwe Ziegenhagen
<http://www.uweziegenhagen.de>
Gerd Neugebauer
2017-10-13 20:41:27 UTC
Permalink
Raw Message
Hi,
[...]
Post by Norbert Preining
Post by Petra Ruebe-Pugliese
AFAIK our webmaster has been thinking about such an interface
for a _long_ time, but it seems that this is more complicated
than expected, and then the backend doesn't seem to be
I agree completely. I wasn't sure, but my guess was that this is
overly complicated.
not really. I was just distracted by too many other things which came in along. Discussions and questions help me to raise the priority. Thus things should become slightly better wrt the upload automation now;-)

[...]

Ciao
Gerd
Karl Berry
2017-10-13 21:48:27 UTC
Permalink
Raw Message
Is it possible to update the package via an API or would I have to go to
the website manually each time I release a new version?

CTAN not providing a dedicated API does not have to be a barrier. Just
write a simple script to POST to the form at ctan.org/upload. I surmise
that is simpler than what any API could provide, anyway. --best, karl.
Gerd Neugebauer
2017-10-13 21:59:15 UTC
Permalink
Raw Message
Post by Nico Schlömer
Is it possible to update the package via an API or would I have to go to
the website manually each time I release a new version?
CTAN not providing a dedicated API does not have to be a barrier. Just
write a simple script to POST to the form at ctan.org/upload. I surmise
that is simpler than what any API could provide, anyway. --best, karl.
We already have such a script on CTAN. Unfortunately it assumes an old version of the upload form. Thus it is disfuctional now (and the author is unreachable).
An API has the advantage that CTAN will provide a documented and reliable interface which can be used for this purpose -- not relying on a volatile web form.

The API itself is already in place. I am working on the documentation and testing and on a script for the client side to simplify the upload: edit a configuration file and invoke a command on the command line (which can be integrated in any build and deployment tool).

Ciao
Gerd
Reinhard Kotucha
2017-10-14 21:23:27 UTC
Permalink
Raw Message
Post by Nico Schlömer
Hi everyone,
betterbib [1], a (Python) command line tool of mine which helps
improving and maintaining your BibTeX files, has picked up some
popularity recently. Perhaps it'd make a good addition to TeXLive
as well. If you're interested, let me know if any modification
would have to be made.
Cheers,
Nico
[1] https://github.com/nschloe/betterbib
Hi Nico,
how comprehensive is the Crossref database used by betterbib?
What does it support?

Regards,
Reinhard
--
------------------------------------------------------------------
Reinhard Kotucha Phone: +49-511-3373112
Marschnerstr. 25
D-30167 Hannover mailto:***@web.de
------------------------------------------------------------------
Nico Schlömer
2017-10-15 11:40:38 UTC
Permalink
Raw Message
Post by Reinhard Kotucha
how comprehensive is the Crossref database used by betterbib?
What does it support?
Crossref database contains about 100 million entries for science
publications. It's fed from the publishers themselves. On average, I would
estimate it knows about 80% of the publications cited in any given paper.

Cheers,
Nico
Post by Reinhard Kotucha
Post by Nico Schlömer
Hi everyone,
betterbib [1], a (Python) command line tool of mine which helps
improving and maintaining your BibTeX files, has picked up some
popularity recently. Perhaps it'd make a good addition to TeXLive
as well. If you're interested, let me know if any modification
would have to be made.
Cheers,
Nico
[1] https://github.com/nschloe/betterbib
Hi Nico,
how comprehensive is the Crossref database used by betterbib?
What does it support?
Regards,
Reinhard
--
------------------------------------------------------------------
Reinhard Kotucha Phon
<https://maps.google.com/?q=Phon&entry=gmail&source=g>e: +49-511-3373112
<0511%203373112>
Marschnerstr. 25
------------------------------------------------------------------
Nico Schlömer
2017-10-15 11:42:54 UTC
Permalink
Raw Message
Post by Gerd Neugebauer
The API itself is already in place. I am working on the documentation and
testing and on a script for the client side to simplify the upload:

Sounds just like what I'd need; I'll keep an eye on the development. For
now, I'll just continue publishing betterbib on PyPi.

Cheers,
Nico
Post by Gerd Neugebauer
Post by Reinhard Kotucha
how comprehensive is the Crossref database used by betterbib?
What does it support?
Crossref database contains about 100 million entries for science
publications. It's fed from the publishers themselves. On average, I would
estimate it knows about 80% of the publications cited in any given paper.
Cheers,
Nico
Post by Reinhard Kotucha
Post by Nico Schlömer
Hi everyone,
betterbib [1], a (Python) command line tool of mine which helps
improving and maintaining your BibTeX files, has picked up some
popularity recently. Perhaps it'd make a good addition to TeXLive
as well. If you're interested, let me know if any modification
would have to be made.
Cheers,
Nico
[1] https://github.com/nschloe/betterbib
Hi Nico,
how comprehensive is the Crossref database used by betterbib?
What does it support?
Regards,
Reinhard
--
------------------------------------------------------------------
Reinhard Kotucha Phon
<https://maps.google.com/?q=Phon&entry=gmail&source=g>e: +49-511-3373112
<0511%203373112>
Marschnerstr. 25
------------------------------------------------------------------
Stephan Lukasczyk
2017-10-15 13:02:09 UTC
Permalink
Raw Message
Dear Nico,
Post by Nico Schlömer
Post by Reinhard Kotucha
how comprehensive is the Crossref database used by betterbib?
What does it support?
Crossref database contains about 100 million entries for science
publications. It's fed from the publishers themselves. On average, I would
estimate it knows about 80% of the publications cited in any given paper.
Thank you for your effort. I just too a quick look into Crossref, and
was feeling a bit disappointed of the provided data. I'm in the field
of Software Engineering/Formal Verification and the examples I checked
were not of a very high quality.

If it doesn't need too much work, I would suggest to query DBLP [1]
especially for publications in computer science. They have more than
3.3 million publications only from computer science in their index, and
edit the entry's data to increase the level of comprehensiveness. The
same might hold for other disciplines and sources.

Regards,
Stephan

[1] http://dblp.uni-trier.de
--
Stephan Lukasczyk
GPG-Key: 0x06E03C26B90F403D
Nico Schlömer
2017-10-15 19:36:51 UTC
Permalink
Raw Message
Thanks Stephan for the good suggestion. I've filed an issue for DBLP [1] to
which you can subscribe if you want to keep track. It's a bit weird that
they don't provide DOIs, but it looks like they provide a nice API so it
shouldn't be too difficult to do.

Cheers,
Nico

[1] https://github.com/nschloe/betterbib/issues/65
Post by Stephan Lukasczyk
Dear Nico,
Post by Nico Schlömer
Post by Reinhard Kotucha
how comprehensive is the Crossref database used by betterbib?
What does it support?
Crossref database contains about 100 million entries for science
publications. It's fed from the publishers themselves. On average, I would
estimate it knows about 80% of the publications cited in any given paper.
Thank you for your effort. I just too a quick look into Crossref, and
was feeling a bit disappointed of the provided data. I'm in the field
of Software Engineering/Formal Verification and the examples I checked
were not of a very high quality.
If it doesn't need too much work, I would suggest to query DBLP [1]
especially for publications in computer science. They have more than
3.3 million publications only from computer science in their index, and
edit the entry's data to increase the level of comprehensiveness. The
same might hold for other disciplines and sources.
Regards,
Stephan
[1] http://dblp.uni-trier.de
--
Stephan Lukasczyk
GPG-Key: 0x06E03C26B90F403D
Martin Sievers
2017-10-16 08:39:40 UTC
Permalink
Raw Message
Post by Nico Schlömer
Thanks Stephan for the good suggestion. I've filed an issue for DBLP [1]
to which you can subscribe if you want to keep track. It's a bit weird
that they don't provide DOIs, but it looks like they provide a nice API
so it shouldn't be too difficult to do.
Dear Nico,

there is a DOI search and if I export an entry into BibTeX I get a DOI
as well.

Best regards
Martin
Nico Schlömer
2017-10-16 09:17:23 UTC
Permalink
Raw Message
Post by Martin Sievers
there is a DOI search and if I export an entry into BibTeX I get a DOI
as well.
I've seen this, but the API doesn't expose this information. Probably an
unfortunate oversight.

Cheers,
Nico
Post by Martin Sievers
Post by Nico Schlömer
Thanks Stephan for the good suggestion. I've filed an issue for DBLP [1]
to which you can subscribe if you want to keep track. It's a bit weird
that they don't provide DOIs, but it looks like they provide a nice API
so it shouldn't be too difficult to do.
Dear Nico,
there is a DOI search and if I export an entry into BibTeX I get a DOI
as well.
Best regards
Martin
Nico Schlömer
2017-10-27 10:38:54 UTC
Permalink
Raw Message
Post by Stephan Lukasczyk
If it doesn't need too much work, I would suggest to query DBLP [1]
I've added DBLP to betterbib now. Use with
```
betterbib --source dblp in.bib out.bib
```

Cheers,
Nico
Post by Stephan Lukasczyk
Post by Martin Sievers
there is a DOI search and if I export an entry into BibTeX I get a DOI
as well.
I've seen this, but the API doesn't expose this information. Probably an
unfortunate oversight.
Cheers,
Nico
Post by Martin Sievers
Post by Nico Schlömer
Thanks Stephan for the good suggestion. I've filed an issue for DBLP [1]
to which you can subscribe if you want to keep track. It's a bit weird
that they don't provide DOIs, but it looks like they provide a nice API
so it shouldn't be too difficult to do.
Dear Nico,
there is a DOI search and if I export an entry into BibTeX I get a DOI
as well.
Best regards
Martin
Loading...