Discussion:
<texmf>/luatex is coming
(too old to reply)
Taco Hoekwater
2009-01-04 14:10:09 UTC
Permalink
Hi,

In one of the next releases I want to start distributing some
lua files under <texmf>/luatex/lua/

The intentions is that lua modules that are part of the luatex
lua distribution will go there, nothing else. These are files
that can be reached via the Lua require() function using the
default compile-time package.path. Lua scripts have their place
under <texmf>/scripts/lua already, and macro package specific
lua files should remain below <texmf>/tex/, to be found via
kpse.find_file().

The immediate candidates for inclusion are:

from lpeg, currently not distributed at all

luatex/lua/re.lua

from luasocket, currently compiled into the executable:

luatex/lua/socket.lua
luatex/lua/ltn12.lua
luatex/lua/mime.lua
luatex/lua/socket/http.lua
luatex/lua/socket/url.lua
luatex/lua/socket/tp.lua
luatex/lua/socket/ftp.lua
luatex/lua/socket/smtp.lua

Some others may soon arise form the luatex core itself, for example
for the mapping of node (sub)type numbers to strings. These ids are
static for any particular version of the executable, but they are
never actually used inside the engine, so it makes sense to split
them off into a lua module that can be loaded on demand.

One warning and one question.

First the warning, for Hans in particular: make sure you start using
require() for all of the external lua modules that are embedded in
luatex. As soon as this new subsystem is in place, you can no longer
use functions like socket.bind() without require-ing the module first!

Second the question, and that is mostly for Martin: when luatex starts
shipping more stuff than just a single executable, it begins to make
sense to have an installation method. It could be nice to be able

$ ./configure; make; make install

at toplevel, instead of

$ ./build.sh ; cp <bunch of whatever to wherever> ; ...

To be able to that, the current build scripts have to be rewritten
into a unified Makefile.in. The new, to-be-created configure script can
be rather stupid and may not even have to be autoconf-generated.

The aim is to get a predictable user interface, not automating the
whole big build process. For example, if "make clean" would delete the
build tree (if there was one), then the "--make" commandline switch
would no longer be needed. Targets like "make cross" and "make native"
could replace ./build.sh.linux and ./build.sh.cross, etc.

Would you (or anyone else) be willing to work on that?

Best wishes,
Taco
Hans Hagen
2009-01-04 15:24:05 UTC
Permalink
Post by Taco Hoekwater
First the warning, for Hans in particular: make sure you start using
require() for all of the external lua modules that are embedded in
luatex. As soon as this new subsystem is in place, you can no longer
use functions like socket.bind() without require-ing the module first!
ok, no problem for the scripts, but i have to figure out a way to embed
them in the format

Hans

-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------
Jonathan Sauer
2009-01-07 10:40:42 UTC
Permalink
Hello,
Post by Hans Hagen
Post by Taco Hoekwater
First the warning, for Hans in particular: make sure you start using
require() for all of the external lua modules that are embedded in
luatex. As soon as this new subsystem is in place, you can no longer
use functions like socket.bind() without require-ing the
module first!
ok, no problem for the scripts, but i have to figure out a
way to embed them in the format
I use a custom package loader and bytecode registers. A toks register
is used to map Lua module names to the number of the bytecode register
the module is stored in. When require()ing a module, this toks register
is checked:

- Module exists: The bytecode register is executed.
- Modules does not exist: A new bytecode register is reserved, the
module loaded using loadfile(), executed and stored in the bytecode
register. Finally, the mapping is added to the toks register.

That does it. I also have a macro to do

<global module var> = require("<module name>")

and add code to \everyjob to do this again at the start of the TeX run
if called in IniTeX.


Maybe this helps,
Jonathan

P.S: My loader is (there is a toks register \LT:***@bytecode@files which
stores the mapping from module name to bytecode register, and a count
register \LT:***@next@bytecode which stores the number of the next unused
bytecode register; ":" and "@" are letters):

local escapeChars = {
['^'] = '%^',
['$'] = '%$',
['('] = '%(',
[')'] = '%)',
['%'] = '%%',
['.'] = '%.',
['['] = '%[',
[']'] = '%]',
['*'] = '%*',
['+'] = '%+',
['-'] = '%-',
['?'] = '%?'
}

local function loadfile(file, name)
local f, e = io.open(file)
if not f then
return f, e
end
local s = f:read('*a')
f:close()
return loadstring(s, '@' .. name)
end

local function loader(moduleName)
local fn = moduleName .. '.klua'

local t = tex.toks['LT:***@bytecode@files']
local f, n = t:match('|(' .. fn:gsub('.', escapeChars) ..
')|=|([0-9]+)|')

if f then
texio.write_nl('')
texio.write('(', fn, ' [from register ', n, '])')
return lua.bytecode[n]
else
rn = kpse.find_file(fn, 'tex', true)
if rn then
f, e = loadfile(rn, moduleName)
if f then
if tex.formatname == '' then
local r = tex.count['LT:***@next@bytecode']
tex.count['LT:***@next@bytecode'] = r + 1

texio.write_nl('')
texio.write('(', rn, ' [cached in register ', r, '])')
lua.bytecode[r] = f
tex.toks['LT:***@bytecode@files'] = t .. '|' .. fn .. '|=|' .. r
.. '|'
else
texio.write_nl('')
texio.write('(', rn, ')')
end
return f
else
return 'Loader error: ' .. e
end
else
return 'Loader error: Could not find file'
end
end
end

package.loaders = { loader }
Hans Hagen
2009-01-07 11:00:39 UTC
Permalink
This post might be inappropriate. Click to display it.
Taco Hoekwater
2009-01-07 11:10:17 UTC
Permalink
Post by Jonathan Sauer
package.loaders = { loader }
Once the new /luatex is in place, you while probably want to
change that to

package.loaders[#package.loaders + 1] = loader

to avoid destroying the internal loaders

Best wishes,
Taco

Norbert Preining
2009-01-04 16:08:19 UTC
Permalink
Post by Taco Hoekwater
lua files under <texmf>/luatex/lua/
The intentions is that lua modules that are part of the luatex
lua distribution will go there, nothing else. These are files
that can be reached via the Lua require() function using the
default compile-time package.path. Lua scripts have their place
under <texmf>/scripts/lua already, and macro package specific
lua files should remain below <texmf>/tex/, to be found via
kpse.find_file().
Hmmm, what about
<texmf>/scripts/lua/luatex/
? I consider that a better candidate then creating another toplevel
hierarchy.

Is there a particular need for that? The above suggestion would serve
the same purpose, and anyone can anyway install into <texmf>/luatex/lua,
too, if one wants to. So I don't see any direct advantage.
Post by Taco Hoekwater
Second the question, and that is mostly for Martin: when luatex starts
shipping more stuff than just a single executable, it begins to make
sense to have an installation method. It could be nice to be able
$ ./configure; make; make install
at toplevel, instead of
$ ./build.sh ; cp <bunch of whatever to wherever> ; ...
OHHHH PLEASE!!!!!! YES!!!!!
Post by Taco Hoekwater
To be able to that, the current build scripts have to be rewritten
into a unified Makefile.in. The new, to-be-created configure script can
be rather stupid and may not even have to be autoconf-generated.
If you do the step please make sure that dropping it into texlive trees
makes it easy already. Maybe that is already planned/done/whatever,
wanted only to remind it ;-)

Best wishes

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining <***@logic.at> Vienna University of Technology
Debian Developer <***@debian.org> Debian TeX Group
gpg DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
STURRY (n.,vb.)
A token run. Pedestrians who have chosen to cross a road immediately
in front of an approaching vehicle generally give a little wave and
break into a sturry. This gives the impression of hurrying without
having any practical effect on their speed whatsoever.
--- Douglas Adams, The Meaning of Liff
Hans Hagen
2009-01-04 16:25:57 UTC
Permalink
Post by Norbert Preining
Hmmm, what about
<texmf>/scripts/lua/luatex/
? I consider that a better candidate then creating another toplevel
hierarchy.
it also depends on the variables set; what happens if one has files with
the same names in scripts/lua/luatex and scripts/lua/whatever

i think that the idea was to have that toplevel path for libs that
belong to luatex itself, and not to some package

Hans

-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------
Taco Hoekwater
2009-01-04 16:31:08 UTC
Permalink
Hi Norbert,
Post by Norbert Preining
Hmmm, what about
<texmf>/scripts/lua/luatex/
? I consider that a better candidate then creating another toplevel
hierarchy.
Nothing has been done yet, but I favour <texmf>/luatex, and Karl
agreed with me :-)

The files in question are not scripts. For example, take socket.lua.
This is the lua part of the socket implementation, of which the C side
is compiled into the executable (as socket.a). socket.lua is not a
standalone package, and it has to have the same version exactly as
the embedded socket.a, both are distributed by luatex itself, and
both come from upstream.

And there is a precedent: bibtex, dvips, omega, all have similar
top-level directories for runtime files that do not depend on macro
packages.

Best wishes,
Taco
Norbert Preining
2009-01-04 16:34:25 UTC
Permalink
Post by Taco Hoekwater
Nothing has been done yet, but I favour <texmf>/luatex, and Karl
agreed with me :-)
Ok, that is a good indicator ;-)
Post by Taco Hoekwater
The files in question are not scripts. For example, take socket.lua.
This is the lua part of the socket implementation, of which the C side
is compiled into the executable (as socket.a). socket.lua is not a
standalone package, and it has to have the same version exactly as
the embedded socket.a, both are distributed by luatex itself, and
both come from upstream.
Ok, I see the difference now. Fine with me.
Post by Taco Hoekwater
And there is a precedent: bibtex, dvips, omega, all have similar
top-level directories for runtime files that do not depend on macro
packages.
Agreed.

Sorry for the noise.

Best wishes

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining <***@logic.at> Vienna University of Technology
Debian Developer <***@debian.org> Debian TeX Group
gpg DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
YONDER BOGINE (n.)
The kind of restaurant advertised as 'just three minutes from this
cinema' which clearly nobody ever goes to and, even if they had ever
contemplated it, have certainly changed their mind since seeing the
advert.
--- Douglas Adams, The Meaning of Liff
Hans van der Meer
2009-01-04 16:05:48 UTC
Permalink
I have a few questions about this announcement.
Lua scripts have their place under <texmf>/scripts/lua already,
I thought the lua scripts in the cont-tmf.zip distribution are
residing in <texmf>/scripts/context/lua/. Is this a change or are they
different files?

2nd question:
In many instances the system location to place lua libraries is
somewhere like
/usr/local/lib/lua/5.1/lpeg.so etc. for dynamic libraries
/usr/local/share/lua/5.1/re.lu etc. for lua coded modules
See the discussion on page 140 of the book Programming in Lua by
Roberto Ierusalimschy.
Will these locations be put on the searchpath too?
Or will the environment variables LUA_PATH and LUA_CPATH, mentioned in
the above, be included in the search?

Finally a wish related to the build/make process, since that also
seems to undergo a change. At present each compilation run also
compiles ctangle. Often that program is already in the TeX
distribution. Is it possible to use that instead and resort to
compilation only when it cannot be found?

Hans van der Meer
Hi,
In one of the next releases I want to start distributing some
lua files under <texmf>/luatex/lua/
The intentions is that lua modules that are part of the luatex
lua distribution will go there, nothing else. These are files
that can be reached via the Lua require() function using the
default compile-time package.path. Lua scripts have their place
under <texmf>/scripts/lua already, and macro package specific
lua files should remain below <texmf>/tex/, to be found via
kpse.find_file().
from lpeg, currently not distributed at all
luatex/lua/re.lua
luatex/lua/socket.lua
luatex/lua/ltn12.lua
luatex/lua/mime.lua
luatex/lua/socket/http.lua
luatex/lua/socket/url.lua
luatex/lua/socket/tp.lua
luatex/lua/socket/ftp.lua
luatex/lua/socket/smtp.lua
Some others may soon arise form the luatex core itself, for example
for the mapping of node (sub)type numbers to strings. These ids are
static for any particular version of the executable, but they are
never actually used inside the engine, so it makes sense to split
them off into a lua module that can be loaded on demand.
One warning and one question.
First the warning, for Hans in particular: make sure you start using
require() for all of the external lua modules that are embedded in
luatex. As soon as this new subsystem is in place, you can no longer
use functions like socket.bind() without require-ing the module first!
Second the question, and that is mostly for Martin: when luatex starts
shipping more stuff than just a single executable, it begins to make
sense to have an installation method. It could be nice to be able
$ ./configure; make; make install
at toplevel, instead of
$ ./build.sh ; cp <bunch of whatever to wherever> ; ...
To be able to that, the current build scripts have to be rewritten
into a unified Makefile.in. The new, to-be-created configure script can
be rather stupid and may not even have to be autoconf-generated.
The aim is to get a predictable user interface, not automating the
whole big build process. For example, if "make clean" would delete the
build tree (if there was one), then the "--make" commandline switch
would no longer be needed. Targets like "make cross" and "make native"
could replace ./build.sh.linux and ./build.sh.cross, etc.
Would you (or anyone else) be willing to work on that?
Best wishes,
Taco
_______________________________________________
dev-luatex mailing list
http://www.ntg.nl/mailman/listinfo/dev-luatex
Arthur Reutenauer
2009-01-04 19:27:08 UTC
Permalink
I thought the lua scripts in the cont-tmf.zip distribution are residing in
<texmf>/scripts/context/lua/. Is this a change or are they different files?
Different files, see Taco's message from 17:31.
Will these locations be put on the searchpath too?
Or will the environment variables LUA_PATH and LUA_CPATH, mentioned in the
above, be included in the search?
I don't know how the developers will solve that, but it's obvious that
it needs to be handled somehow (maybe through kpathsea? Yet another
variable name in texmf.cnf ...).

Arthur
Taco Hoekwater
2009-01-05 08:25:01 UTC
Permalink
Post by Arthur Reutenauer
I thought the lua scripts in the cont-tmf.zip distribution are residing in
<texmf>/scripts/context/lua/. Is this a change or are they different files?
Different files, see Taco's message from 17:31.
Will these locations be put on the searchpath too?
Or will the environment variables LUA_PATH and LUA_CPATH, mentioned in the
above, be included in the search?
I don't know how the developers will solve that, but it's obvious that
it needs to be handled somehow (maybe through kpathsea? Yet another
variable name in texmf.cnf ...).
Kpathsea has no trouble finding lua files in the tex/ path, but I
remember talking to Karl about a specific variable a while back.
That would not be a big deal, there are so many of them already
that one extra hardly matters.

The issue about <texmf>/luatex/lua is that luatex should be able
to find these files even if kpathsea initialization is off or
delayed. That is why they have to become part of the 'compiled-in'
package.path.

If|Once kpathsea is active, luatex could simply use it for finding
the lua modules, but that may never happen and I want to have a
strong guarantee about the discoverability of the files in
<texmf>/luatex/lua.

Best wishes,
Taco
Hans Hagen
2009-01-04 21:03:43 UTC
Permalink
Post by Hans van der Meer
I have a few questions about this announcement.
Lua scripts have their place under <texmf>/scripts/lua already,
I thought the lua scripts in the cont-tmf.zip distribution are residing
in <texmf>/scripts/context/lua/. Is this a change or are they different
files?
only some mtx-* scripts; they use libs that are in tex/context/base and
that will not change; macro packages should put their code under their
designated paths (for latex there is also a subpath for authors, for
context there is the /third etc subpaths for users)
Post by Hans van der Meer
In many instances the system location to place lua libraries is
somewhere like
/usr/local/lib/lua/5.1/lpeg.so etc. for dynamic libraries
/usr/local/share/lua/5.1/re.lu etc. for lua coded modules
See the discussion on page 140 of the book Programming in Lua by Roberto
Ierusalimschy.
Will these locations be put on the searchpath too?
Or will the environment variables LUA_PATH and LUA_CPATH, mentioned in
the above, be included in the search?
loading of extra libraries (the core libs will remain statically linked)
will be looked into later


Hans


-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------
Loading...