librelist archives

« back to archive

'wheel', an entry into the binary package format market

'wheel', an entry into the binary package format market

From:
Daniel Holth
Date:
2012-06-23 @ 06:14
I've learned about bento due to the discussion on python-dev, and I
think I like it.

Unaware of bento, I have been working on my own solution to some of
the same problems (really just 'my virtualenv takes too long to
populate because it recompiles everything'), which is a package format
called 'wheel' (i.e. of cheese).

A wheel is a lot like an egg, except it uses distutils2-style
.dist-info directories, and separates files into categories by putting
purelib or platlib at the root of the archive (a file WHEEL says
which), and other categories of files in
pkgname-ver.data/categoryname, for example
pyramid-1.3.2.data/scripts/pserve. So if you wanted to you could
pretty much unzip all the wheels you wanted to install into
site-packages and then "spread" the .data directory afterwards.

The fledgling implementation includes a plugin for distutils and a
patch for the non-pluggable distutils2 to generate the format,
including renaming PKG-INFO to METADATA and a few other
distutils2-style things.

But the basic idea is to categorize the files with the archive's
directory structure instead of a manifest file. How does this compare
to or fit with bento?

Thanks,

Daniel


For example,

...
     7044  2012-06-23 02:02
pyramid/scaffolds/zodb/+package+/static/pyramid-small.png
    33055  2012-06-23 02:02
pyramid/scaffolds/zodb/+package+/static/pyramid.png
       49  2012-06-23 02:02
pyramid/scaffolds/zodb/+package+/static/transparent.gif
     3378  2012-06-23 02:02
pyramid/scaffolds/zodb/+package+/templates/mytemplate.pt
     6984  2012-06-23 02:02   pyramid/fixers/fix_bfg_imports.py
       10  2012-06-23 02:02   pyramid/fixers/__init__.py
      319  2012-06-23 02:07   pyramid-1.3.2.data/scripts/ptweens
      319  2012-06-23 02:07   pyramid-1.3.2.data/scripts/proutes
      317  2012-06-23 02:07   pyramid-1.3.2.data/scripts/pshell
      321  2012-06-23 02:07   pyramid-1.3.2.data/scripts/prequest
      317  2012-06-23 02:07   pyramid-1.3.2.data/scripts/pviews
      327  2012-06-23 02:07   pyramid-1.3.2.data/scripts/bfg2pyramid
      319  2012-06-23 02:07   pyramid-1.3.2.data/scripts/pcreate
      317  2012-06-23 02:07   pyramid-1.3.2.data/scripts/pserve
        8  2012-06-23 02:07   pyramid-1.3.2.dist-info/top_level.txt
        1  2012-06-23 02:07   pyramid-1.3.2.dist-info/dependency_links.txt
      765  2012-06-23 02:07   pyramid-1.3.2.dist-info/entry_points.txt
        1  2012-06-23 02:02   pyramid-1.3.2.dist-info/not-zip-safe
    31202  2012-06-23 02:07   pyramid-1.3.2.dist-info/SOURCES.txt
    41088  2012-06-23 02:07   pyramid-1.3.2.dist-info/METADATA
       62  2012-06-23 02:07   pyramid-1.3.2.dist-info/WHEEL
    26747  2012-06-23 02:07   pyramid-1.3.2.dist-info/RECORD

Re: [bento] 'wheel', an entry into the binary package format market

From:
David Cournapeau
Date:
2012-07-01 @ 20:25
Hi Daniel,

My apologies for the late answer

On Sat, Jun 23, 2012 at 7:14 AM, Daniel Holth <dholth@gmail.com> wrote:
> I've learned about bento due to the discussion on python-dev, and I
> think I like it.
>
> Unaware of bento, I have been working on my own solution to some of
> the same problems (really just 'my virtualenv takes too long to
> populate because it recompiles everything'), which is a package format
> called 'wheel' (i.e. of cheese).
>
> A wheel is a lot like an egg, except it uses distutils2-style
> .dist-info directories, and separates files into categories by putting
> purelib or platlib at the root of the archive (a file WHEEL says
> which), and other categories of files in
> pkgname-ver.data/categoryname, for example
> pyramid-1.3.2.data/scripts/pserve. So if you wanted to you could
> pretty much unzip all the wheels you wanted to install into
> site-packages and then "spread" the .data directory afterwards.
>
> The fledgling implementation includes a plugin for distutils and a
> patch for the non-pluggable distutils2 to generate the format,
> including renaming PKG-INFO to METADATA and a few other
> distutils2-style things.
>
> But the basic idea is to categorize the files with the archive's
> directory structure instead of a manifest file. How does this compare
> to or fit with bento?

Having categories for each file is indeed similar to this build
manifest concept I have been using in bento. One of the difference is
that the build manifest encode file locations in a
distribution-neutral manner (e.g. it will record something as
$sitedir/foo/bar instead of /usr/lib/python2.7/site-packages...), so
as to allow for conversion between formats.

Does your format goes beyond "dumb binary" ? I myself am not that
interested by a dumb binary format without support for post install
scripts and uninstallation. This is where an installer comes in handy,
but I have not really worked on that yet.

David

Re: [bento] 'wheel', an entry into the binary package format market

From:
Daniel Holth
Date:
2012-07-01 @ 23:01
> Having categories for each file is indeed similar to this build
> manifest concept I have been using in bento. One of the difference is
> that the build manifest encode file locations in a
> distribution-neutral manner (e.g. it will record something as
> $sitedir/foo/bar instead of /usr/lib/python2.7/site-packages...), so
> as to allow for conversion between formats.

I think I'm doing the same thing, except it's stored as a
Name-1.0.data/sitedir/foo/bar entry in the archive.

> Does your format goes beyond "dumb binary" ? I myself am not that
> interested by a dumb binary format without support for post install
> scripts and uninstallation. This is where an installer comes in handy,
> but I have not really worked on that yet.

I have not gotten this far. You're welcome to extend the format if you
have something.

The installer is currently a patched version of pip. We are discussing
it on the virtualenv group if you want to try it.

There obviously has to be "generate the script wrappers" hooks. I
would also like to define a general way to add post-install hooks (an
entry point). For example it would be cool to write a hook that keeps
track of installations in a sql database.

Re: [bento] 'wheel', an entry into the binary package format market

From:
Dag Sverre Seljebotn
Date:
2012-07-02 @ 09:39
On 07/02/2012 01:01 AM, Daniel Holth wrote:
>> Having categories for each file is indeed similar to this build
>> manifest concept I have been using in bento. One of the difference is
>> that the build manifest encode file locations in a
>> distribution-neutral manner (e.g. it will record something as
>> $sitedir/foo/bar instead of /usr/lib/python2.7/site-packages...), so
>> as to allow for conversion between formats.
>
> I think I'm doing the same thing, except it's stored as a
> Name-1.0.data/sitedir/foo/bar entry in the archive.
>
>> Does your format goes beyond "dumb binary" ? I myself am not that
>> interested by a dumb binary format without support for post install
>> scripts and uninstallation. This is where an installer comes in handy,
>> but I have not really worked on that yet.
>
> I have not gotten this far. You're welcome to extend the format if you
> have something.
>
> The installer is currently a patched version of pip. We are discussing
> it on the virtualenv group if you want to try it.
>
> There obviously has to be "generate the script wrappers" hooks. I
> would also like to define a general way to add post-install hooks (an
> entry point). For example it would be cool to write a hook that keeps
> track of installations in a sql database.

Very interesting.

One advantage that I can see with the manifest-file approach (which 
Bento uses?) over the directory-structure-approach is that if you can 
simply slap a manifest file into the build directory and install 
directly from the build directory. So when you do a partial rebuild 
(say, only a single shared library is affected and rebuilt), you don't 
need to re-copy files (you just regenerate the manifest, which is much 
cheaper).

I.e., it means less movement/copying of files in the case that you have 
a single directory and keep rebuilding it.

What's the advantages of the directory structure approach? (can't think 
of any right now but I'm sure there are some).

Dag

Re: [bento] 'wheel', an entry into the binary package format market

From:
Daniel Holth
Date:
2012-07-02 @ 11:02
> One advantage that I can see with the manifest-file approach (which 
> Bento uses?) over the directory-structure-approach is that if you can 
> simply slap a manifest file into the build directory and install 
> directly from the build directory. So when you do a partial rebuild 
> (say, only a single shared library is affected and rebuilt), you don't 
> need to re-copy files (you just regenerate the manifest, which is much 
> cheaper).

Just like you dont use .rpm or .deb to Develop, during development you 
will simply add the source directory to your Pythonpath.

> 
> I.e., it means less movement/copying of files in the case that you have 
> a single directory and keep rebuilding it.
> 
> What's the advantages of the directory structure approach? (can't think 
> of any right now but I'm sure there are some).

The main advantage is that it can be unpacked into site-packages by a dumb
installer and then spread out later by a smarter installer.

Re: [bento] 'wheel', an entry into the binary package format market

From:
Daniel Holth
Date:
2012-07-20 @ 20:13
On Sun, Jul 1, 2012 at 4:25 PM, David Cournapeau <cournape@gmail.com> wrote:

> Does your format goes beyond "dumb binary" ? I myself am not that
> interested by a dumb binary format without support for post install
> scripts and uninstallation. This is where an installer comes in handy,
> but I have not really worked on that yet.

How would you feel about just including the bdist_wininst installer
.ini in the .dist-info/ directory? IPython specifies its post-install
script this way and adds shortcuts to the Start menu. (bdist_wininst
is an .exe extractor/installer concatenated with an .ini and a .zip)

Re: [bento] 'wheel', an entry into the binary package format market

From:
David Cournapeau
Date:
2012-07-22 @ 14:37
Hi Daniel

On Fri, Jul 20, 2012 at 9:13 PM, Daniel Holth <dholth@gmail.com> wrote:
> On Sun, Jul 1, 2012 at 4:25 PM, David Cournapeau <cournape@gmail.com> wrote:
>
>> Does your format goes beyond "dumb binary" ? I myself am not that
>> interested by a dumb binary format without support for post install
>> scripts and uninstallation. This is where an installer comes in handy,
>> but I have not really worked on that yet.
>
> How would you feel about just including the bdist_wininst installer
> .ini in the .dist-info/ directory? IPython specifies its post-install
> script this way and adds shortcuts to the Start menu. (bdist_wininst
> is an .exe extractor/installer concatenated with an .ini and a .zip)

In the case of bdist-win, it works ok because you have an included
installer, but I want something that works on every platform (as well
as on platform-specific installers which support the feature).

I am not a big fan of the whole .dist-info infrastructure: I find it
intrusive, ugly and unreliable.

David

Re: [bento] 'wheel', an entry into the binary package format market

From:
Daniel Holth
Date:
2012-07-22 @ 22:50
Even though its almost exactly the same as egg-info?
On Jul 22, 2012 10:38 AM, "David Cournapeau" <cournape@gmail.com> wrote:

> Hi Daniel
>
> On Fri, Jul 20, 2012 at 9:13 PM, Daniel Holth <dholth@gmail.com> wrote:
> > On Sun, Jul 1, 2012 at 4:25 PM, David Cournapeau <cournape@gmail.com>
> wrote:
> >
> >> Does your format goes beyond "dumb binary" ? I myself am not that
> >> interested by a dumb binary format without support for post install
> >> scripts and uninstallation. This is where an installer comes in handy,
> >> but I have not really worked on that yet.
> >
> > How would you feel about just including the bdist_wininst installer
> > .ini in the .dist-info/ directory? IPython specifies its post-install
> > script this way and adds shortcuts to the Start menu. (bdist_wininst
> > is an .exe extractor/installer concatenated with an .ini and a .zip)
>
> In the case of bdist-win, it works ok because you have an included
> installer, but I want something that works on every platform (as well
> as on platform-specific installers which support the feature).
>
> I am not a big fan of the whole .dist-info infrastructure: I find it
> intrusive, ugly and unreliable.
>
> David
>

Re: [bento] 'wheel', an entry into the binary package format market

From:
David Cournapeau
Date:
2012-07-23 @ 03:14
On Sun, Jul 22, 2012 at 11:50 PM, Daniel Holth <dholth@gmail.com> wrote:
> Even though its almost exactly the same as egg-info?

My comment certainly applies to egg-info as well. It is a baroque mix
of installation metadata, namespace-related resources. It also
conflates data with their on-file representation, and make things like
indexing and fast retrieval hard.

David

Re: [bento] 'wheel', an entry into the binary package format market

From:
Daniel Holth
Date:
2012-08-24 @ 15:26
I enjoyed the excellent Bento docs. I see the "install things anywhere
on the disk" feature uses different categories than the distutils2
effort. "Install things anywhere" is considered a misfeature by egg
(wheel has no opinion). There is some conflict between the goals of
virtualenv / buildout style package management (everything in a
per-environment directory) and system packaging.

wheel may be more neoclassical than you think. The .dist-info
directory is only a few files, just PKG-INFO (renamed to METADATA for
PEP reasons), WHEEL (package format version and packaging software
name), RECORD (a manifest with hashes), and RECORD.jws (digital
signature).

I like your generated python module with installation paths.
distutils2 handed this horribly with a second, undocumented list of
files and their install paths inside .dist-info/

In theory would bento put the package manifests / metadata in a fast
database like RPM? Does it do uninstall yet, or is that delegated to
system packaging tools? How would that interact with PYTHONPATH or
plugins?

I think bento's egg builder could leave out SOURCES.txt, I don't think
it is ever used. There may be other files that are just vestigial.

On Sun, Jul 22, 2012 at 11:14 PM, David Cournapeau <cournape@gmail.com> wrote:
> On Sun, Jul 22, 2012 at 11:50 PM, Daniel Holth <dholth@gmail.com> wrote:
>> Even though its almost exactly the same as egg-info?
>
> My comment certainly applies to egg-info as well. It is a baroque mix
> of installation metadata, namespace-related resources. It also
> conflates data with their on-file representation, and make things like
> indexing and fast retrieval hard.

Re: [bento] 'wheel', an entry into the binary package format market

From:
David Cournapeau
Date:
2012-08-29 @ 20:36
Hi Daniel,

On Fri, Aug 24, 2012 at 5:26 PM, Daniel Holth <dholth@gmail.com> wrote:
> I enjoyed the excellent Bento docs.

Thanks, although there are still many things to document

> I see the "install things anywhere
> on the disk" feature uses different categories than the distutils2
> effort. "Install things anywhere" is considered a misfeature by egg
> (wheel has no opinion). There is some conflict between the goals of
> virtualenv / buildout style package management (everything in a
> per-environment directory) and system packaging.

Yes, that's a good observation. OTOH, virtualenv/buildout style is a
strict subset of system packaging as far as I am concerned, at the
low-level at least.

And indeed, the build description file allows  to describe eggs, while
keeping the power necessary for system packaging.

> wheel may be more neoclassical than you think. The .dist-info
> directory is only a few files, just PKG-INFO (renamed to METADATA for
> PEP reasons), WHEEL (package format version and packaging software
> name), RECORD (a manifest with hashes), and RECORD.jws (digital
> signature).
>
> I like your generated python module with installation paths.
> distutils2 handed this horribly with a second, undocumented list of
> files and their install paths inside .dist-info/
>
> In theory would bento put the package manifests / metadata in a fast
> database like RPM? Does it do uninstall yet, or is that delegated to
> system packaging tools? How would that interact with PYTHONPATH or
> plugins?

So in theory, yes, that's what I want to do. Right now, bento has no
installer, though. I am relying on the distutils-like setup.py to make
bento packages pip-installable.

I don't do anything 'magic' ala pkg_resources, so PYTHONPATH is
irrelevant, as are plugins.

> I think bento's egg builder could leave out SOURCES.txt, I don't think
> it is ever used. There may be other files that are just vestigial.

True, OTOH, that's very easy to generate.

Did you try to implement wheel on top of Bento ?

David

Re: [bento] 'wheel', an entry into the binary package format market

From:
Daniel Holth
Date:
2012-08-29 @ 20:47
On Wed, Aug 29, 2012 at 4:36 PM, David Cournapeau <cournape@gmail.com> wrote:
>> In theory would bento put the package manifests / metadata in a fast
>> database like RPM? Does it do uninstall yet, or is that delegated to
>> system packaging tools? How would that interact with PYTHONPATH or
>> plugins?
>
> So in theory, yes, that's what I want to do. Right now, bento has no
> installer, though. I am relying on the distutils-like setup.py to make
> bento packages pip-installable.
>
> I don't do anything 'magic' ala pkg_resources, so PYTHONPATH is
> irrelevant, as are plugins.

One problem scanning .egg-info directories is supposed to solve is
that different packages are "installed" depending on PYTHONPATH.

> Did you try to implement wheel on top of Bento ?

I would like to do that next, and I would be happy to expand the
default list of paths wheel understands to make Bento packages happy
and to have a more versatile specification.

I'm supporting extras in this way, it has been discussed somewhat on
python-dev. It will probably need to be renamed to Metadata 1.3 to get
merged, but distribute > 0.6.28 supports this exact syntax:


https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068c0235302b0614058b7dccf30b0ce

Build requirements are Setup-Requires-Dist, but wheels are already
built and do not pay attention to it. pip never trusts bundled
PKG-INFO so it trips trying to regenerate it with missing build deps.

Re: [bento] 'wheel', an entry into the binary package format market

From:
Daniel Holth
Date:
2012-09-02 @ 02:11
>> Did you try to implement wheel on top of Bento ?

Yes. I have implemented a basic build_wheel command at
https://github.com/dholth/Bento

It is enough that "wheel sign ..." and "wheel verify ..." can work
with the package, creating a dist-info directory and generating a
RECORD with hashes of all the included files.

It doesn't copy anything (scripts, etc) into name-ver.data/scripts/
and it looks like it misses ExtraSourceFiles and probably a lot of
other categories of files. Requirements need to be expressed as
Requires-Dist: entries in the metadata, a number of things to comply
with the upcoming Metadata 1.3 need to be added, but it is a starting
point. The Bento code is easy to hack on.

Daniel