librelist archives

« back to archive

Bento as main build system - what is needed?

Bento as main build system - what is needed?

From:
Matthew Brett
Date:
2012-07-26 @ 03:06
Hi David,

I've run into another unpleasant numpy.distutils hacking episode, and that made
me think again of Bento.

That prompted me to consider again what it would take to switch to Bento as our
only build system.

Are others interested to make such a switch?

What would it take to be practical?

What help can us others offer to get us to the situation where a full switch is
practical?

Sorry if I am slow to reply, I am in Cuba at the moment, but checking my email
every couple of days.

Cheers,

Matthew

Re: [bento] Bento as main build system - what is needed?

From:
David Cournapeau
Date:
2012-07-26 @ 20:26
Hi Matthew,

On Thu, Jul 26, 2012 at 4:06 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
> Hi David,
>
> I've run into another unpleasant numpy.distutils hacking episode, and that made
> me think again of Bento.
>
> That prompted me to consider again what it would take to switch to Bento as our
> only build system.

By our, you mean Nipy, numpy, something else ? If numpy, then shall we
continue on numpy ML ?

>
> Are others interested to make such a switch?
>
> What would it take to be practical?
>
> What help can us others offer to get us to the situation where a full switch is
> practical?
>
> Sorry if I am slow to reply, I am in Cuba at the moment, but checking my email
> every couple of days.

I think there are a lot of low hanging fruits to make it happen, plus
one big issue: backward compatibility. Maybe one should just push for
bundling bento with numpy (or any other mechanism that would reach the
same goal).

Everything else, included polished windows support, should be
relatively easy to finish with little efforts,

David

Re: [bento] Bento as main build system - what is needed?

From:
Matthew Brett
Date:
2012-07-29 @ 01:10
Hi,

On Thu, Jul 26, 2012 at 4:26 PM, David Cournapeau <cournape@gmail.com> wrote:
> Hi Matthew,
>
> On Thu, Jul 26, 2012 at 4:06 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
>> Hi David,
>>
>> I've run into another unpleasant numpy.distutils hacking episode, and that made
>> me think again of Bento.
>>
>> That prompted me to consider again what it would take to switch to Bento as our
>> only build system.
>
> By our, you mean Nipy, numpy, something else ? If numpy, then shall we
> continue on numpy ML ?

Actually, I meant nipy* for now, that is, the three projects I do the
build support and releases for.  Do you think using Bento as the only
build project for numpy would fly right now?   That would be good of
course - but actually does not deal with other projects immediately
because we have to work with older versions of numpy.

>>
>> Are others interested to make such a switch?
>>
>> What would it take to be practical?
>>
>> What help can us others offer to get us to the situation where a full switch is
>> practical?
>>
>> Sorry if I am slow to reply, I am in Cuba at the moment, but checking my email
>> every couple of days.
>
> I think there are a lot of low hanging fruits to make it happen, plus
> one big issue: backward compatibility. Maybe one should just push for
> bundling bento with numpy (or any other mechanism that would reach the
> same goal).
>
> Everything else, included polished windows support, should be
> relatively easy to finish with little efforts,

I guess my question was - how do you think we can get there?

For example, as we discussed on the phone, I think I - and Fernando
when I spoke to him - were uncomfortable about putting bentomaker into
our source repos, but happy to have it in sdist archives.  I guess
that's an easy bscript away, but do you think that should be
recommended?

And, the question came up about whether to make the bundled bento
importable for pip or easy_install - so setup.py could call into a
known bento.   Does that also seem like a good way to go?

Is WAF the recommended backend now?   What do you think about bundling
waf?  Too soon?

Cheers,

Matthew

Re: [bento] Bento as main build system - what is needed?

From:
David Cournapeau
Date:
2012-08-01 @ 08:30
On Sun, Jul 29, 2012 at 2:10 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
> Hi,
>
> On Thu, Jul 26, 2012 at 4:26 PM, David Cournapeau <cournape@gmail.com> wrote:
>> Hi Matthew,
>>
>> On Thu, Jul 26, 2012 at 4:06 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
>>> Hi David,
>>>
>>> I've run into another unpleasant numpy.distutils hacking episode, and 
that made
>>> me think again of Bento.
>>>
>>> That prompted me to consider again what it would take to switch to 
Bento as our
>>> only build system.
>>
>> By our, you mean Nipy, numpy, something else ? If numpy, then shall we
>> continue on numpy ML ?
>
> Actually, I meant nipy* for now, that is, the three projects I do the
> build support and releases for.  Do you think using Bento as the only
> build project for numpy would fly right now?   That would be good of
> course - but actually does not deal with other projects immediately
> because we have to work with older versions of numpy.

Before switching, I would need to set up CI on the big three platforms
using bento instead of distutils (and fix all the ensuing issues).

>
>>>
>>> Are others interested to make such a switch?
>>>
>>> What would it take to be practical?
>>>
>>> What help can us others offer to get us to the situation where a full 
switch is
>>> practical?
>>>
>>> Sorry if I am slow to reply, I am in Cuba at the moment, but checking my email
>>> every couple of days.
>>
>> I think there are a lot of low hanging fruits to make it happen, plus
>> one big issue: backward compatibility. Maybe one should just push for
>> bundling bento with numpy (or any other mechanism that would reach the
>> same goal).
>>
>> Everything else, included polished windows support, should be
>> relatively easy to finish with little efforts,
>
> I guess my question was - how do you think we can get there?
>
> For example, as we discussed on the phone, I think I - and Fernando
> when I spoke to him - were uncomfortable about putting bentomaker into
> our source repos, but happy to have it in sdist archives.  I guess
> that's an easy bscript away, but do you think that should be
> recommended?

So you need a mechanism to make sure the right version of bento+waf is
used for every revision of NiPy. Maybe something like git submodule ?

> And, the question came up about whether to make the bundled bento
> importable for pip or easy_install - so setup.py could call into a
> known bento.   Does that also seem like a good way to go?

You mean only for sdist, right ? This one I don't have a good idea on
how to solve yet in the short term. The 'long' term is to add proper
support for build dependencies in pip. That could be a sprint idea for
bento at euroscipy :)

> Is WAF the recommended backend now?   What do you think about bundling
> waf?  Too soon?

That's part of the backward compatibility issue I was hinting at.
While I really do not want to write my own build system, I am
wondering if that's realistic, because waf is kind of a moving target,
only justified in the hard cases (numpy/scipy for example).

cheers,

David

Re: [bento] Bento as main build system - what is needed?

From:
Matthew Brett
Date:
2012-08-22 @ 01:16
Hi,

On Wed, Aug 1, 2012 at 1:30 AM, David Cournapeau <cournape@gmail.com> wrote:
> On Sun, Jul 29, 2012 at 2:10 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
>> Hi,
>>
>> On Thu, Jul 26, 2012 at 4:26 PM, David Cournapeau <cournape@gmail.com> wrote:
>>> Hi Matthew,
>>>
>>> On Thu, Jul 26, 2012 at 4:06 AM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>> Hi David,
>>>>
>>>> I've run into another unpleasant numpy.distutils hacking episode, and
that made
>>>> me think again of Bento.
>>>>
>>>> That prompted me to consider again what it would take to switch to 
Bento as our
>>>> only build system.
>>>
>>> By our, you mean Nipy, numpy, something else ? If numpy, then shall we
>>> continue on numpy ML ?
>>
>> Actually, I meant nipy* for now, that is, the three projects I do the
>> build support and releases for.  Do you think using Bento as the only
>> build project for numpy would fly right now?   That would be good of
>> course - but actually does not deal with other projects immediately
>> because we have to work with older versions of numpy.
>
> Before switching, I would need to set up CI on the big three platforms
> using bento instead of distutils (and fix all the ensuing issues).

Can I help with that?  I have machines ...

>>>> Are others interested to make such a switch?
>>>>
>>>> What would it take to be practical?
>>>>
>>>> What help can us others offer to get us to the situation where a full
switch is
>>>> practical?
>>>>
>>>> Sorry if I am slow to reply, I am in Cuba at the moment, but checking
my email
>>>> every couple of days.
>>>
>>> I think there are a lot of low hanging fruits to make it happen, plus
>>> one big issue: backward compatibility. Maybe one should just push for
>>> bundling bento with numpy (or any other mechanism that would reach the
>>> same goal).
>>>
>>> Everything else, included polished windows support, should be
>>> relatively easy to finish with little efforts,
>>
>> I guess my question was - how do you think we can get there?
>>
>> For example, as we discussed on the phone, I think I - and Fernando
>> when I spoke to him - were uncomfortable about putting bentomaker into
>> our source repos, but happy to have it in sdist archives.  I guess
>> that's an easy bscript away, but do you think that should be
>> recommended?
>
> So you need a mechanism to make sure the right version of bento+waf is
> used for every revision of NiPy. Maybe something like git submodule ?

Well - git submodule has a high confusion quotient for the
inexperienced, I'd rather avoid it if at all possible.

>> And, the question came up about whether to make the bundled bento
>> importable for pip or easy_install - so setup.py could call into a
>> known bento.   Does that also seem like a good way to go?
>
> You mean only for sdist, right ? This one I don't have a good idea on
> how to solve yet in the short term. The 'long' term is to add proper
> support for build dependencies in pip. That could be a sprint idea for
> bento at euroscipy :)

You were mentioning the ability to import directly from the (sdist
version of) the bentomaker file?

>> Is WAF the recommended backend now?   What do you think about bundling
>> waf?  Too soon?
>
> That's part of the backward compatibility issue I was hinting at.
> While I really do not want to write my own build system, I am
> wondering if that's realistic, because waf is kind of a moving target,
> only justified in the hard cases (numpy/scipy for example).

How about packaging the current waf and updating from time to time?

Cheers,

Matthew

Re: [bento] Bento as main build system - what is needed?

From:
David Cournapeau
Date:
2012-08-24 @ 10:13
On Wed, Aug 22, 2012 at 3:16 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
> Hi,
>
> On Wed, Aug 1, 2012 at 1:30 AM, David Cournapeau <cournape@gmail.com> wrote:
>> On Sun, Jul 29, 2012 at 2:10 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
>>> Hi,
>>>
>>> On Thu, Jul 26, 2012 at 4:26 PM, David Cournapeau <cournape@gmail.com> wrote:
>>>> Hi Matthew,
>>>>
>>>> On Thu, Jul 26, 2012 at 4:06 AM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>>> Hi David,
>>>>>
>>>>> I've run into another unpleasant numpy.distutils hacking episode, 
and that made
>>>>> me think again of Bento.
>>>>>
>>>>> That prompted me to consider again what it would take to switch to 
Bento as our
>>>>> only build system.
>>>>
>>>> By our, you mean Nipy, numpy, something else ? If numpy, then shall we
>>>> continue on numpy ML ?
>>>
>>> Actually, I meant nipy* for now, that is, the three projects I do the
>>> build support and releases for.  Do you think using Bento as the only
>>> build project for numpy would fly right now?   That would be good of
>>> course - but actually does not deal with other projects immediately
>>> because we have to work with older versions of numpy.
>>
>> Before switching, I would need to set up CI on the big three platforms
>> using bento instead of distutils (and fix all the ensuing issues).
>
> Can I help with that?  I have machines ...

It would consists in setting up jenkins to build numpy with bento+waf
(or just bento itself on windows).

>
>>>>> Are others interested to make such a switch?
>>>>>
>>>>> What would it take to be practical?
>>>>>
>>>>> What help can us others offer to get us to the situation where a 
full switch is
>>>>> practical?
>>>>>
>>>>> Sorry if I am slow to reply, I am in Cuba at the moment, but 
checking my email
>>>>> every couple of days.
>>>>
>>>> I think there are a lot of low hanging fruits to make it happen, plus
>>>> one big issue: backward compatibility. Maybe one should just push for
>>>> bundling bento with numpy (or any other mechanism that would reach the
>>>> same goal).
>>>>
>>>> Everything else, included polished windows support, should be
>>>> relatively easy to finish with little efforts,
>>>
>>> I guess my question was - how do you think we can get there?
>>>
>>> For example, as we discussed on the phone, I think I - and Fernando
>>> when I spoke to him - were uncomfortable about putting bentomaker into
>>> our source repos, but happy to have it in sdist archives.  I guess
>>> that's an easy bscript away, but do you think that should be
>>> recommended?
>>
>> So you need a mechanism to make sure the right version of bento+waf is
>> used for every revision of NiPy. Maybe something like git submodule ?
>
> Well - git submodule has a high confusion quotient for the
> inexperienced, I'd rather avoid it if at all possible.

Ok. Would the bootstrap script mentioned before work ? The bootstrap
script would hardcode the versions of bento/waf, and building from a
github checkout would simply consists in running the bootstrap script
first.

As for making bentomaker importable, that is working as we speak, I
just implemented it :)

>>> import bentomaker # first time is slow as it needs to unpack itself
>>> import bento # importing bentomaker put its bento into sys.modules

>
> How about packaging the current waf and updating from time to time?

That's not something I'd like to do in the long term, because I don't
want to take responsibility for waf in any way.

David

Re: [bento] Bento as main build system - what is needed?

From:
David Cournapeau
Date:
2012-08-01 @ 12:54
On Wed, Aug 1, 2012 at 9:30 AM, David Cournapeau <cournape@gmail.com> wrote:
> On Sun, Jul 29, 2012 at 2:10 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
>> Hi,
>>
>> On Thu, Jul 26, 2012 at 4:26 PM, David Cournapeau <cournape@gmail.com> wrote:
>>> Hi Matthew,
>>>
>>> On Thu, Jul 26, 2012 at 4:06 AM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>> Hi David,
>>>>
>>>> I've run into another unpleasant numpy.distutils hacking episode, and
that made
>>>> me think again of Bento.
>>>>
>>>> That prompted me to consider again what it would take to switch to 
Bento as our
>>>> only build system.
>>>
>>> By our, you mean Nipy, numpy, something else ? If numpy, then shall we
>>> continue on numpy ML ?
>>
>> Actually, I meant nipy* for now, that is, the three projects I do the
>> build support and releases for.  Do you think using Bento as the only
>> build project for numpy would fly right now?   That would be good of
>> course - but actually does not deal with other projects immediately
>> because we have to work with older versions of numpy.
>
> Before switching, I would need to set up CI on the big three platforms
> using bento instead of distutils (and fix all the ensuing issues).
>
>>
>>>>
>>>> Are others interested to make such a switch?
>>>>
>>>> What would it take to be practical?
>>>>
>>>> What help can us others offer to get us to the situation where a full
switch is
>>>> practical?
>>>>
>>>> Sorry if I am slow to reply, I am in Cuba at the moment, but checking
my email
>>>> every couple of days.
>>>
>>> I think there are a lot of low hanging fruits to make it happen, plus
>>> one big issue: backward compatibility. Maybe one should just push for
>>> bundling bento with numpy (or any other mechanism that would reach the
>>> same goal).
>>>
>>> Everything else, included polished windows support, should be
>>> relatively easy to finish with little efforts,
>>
>> I guess my question was - how do you think we can get there?
>>
>> For example, as we discussed on the phone, I think I - and Fernando
>> when I spoke to him - were uncomfortable about putting bentomaker into
>> our source repos, but happy to have it in sdist archives.  I guess
>> that's an easy bscript away, but do you think that should be
>> recommended?
>
> So you need a mechanism to make sure the right version of bento+waf is
> used for every revision of NiPy. Maybe something like git submodule ?

There is actually a simpler solution, already used by setuptools and
buildout: a bootstrap script. That requires internet access, though,
but the workflow for a developer would be something like:

clone nipy nipy-git && cd nipy-git
python bootstrap_bento.py

Would that be acceptable ?

David

Re: [bento] Bento as main build system - what is needed?

From:
Gour
Date:
2012-08-01 @ 08:47
On Wed, 1 Aug 2012 09:30:37 +0100
David Cournapeau <cournape@gmail.com> wrote:

> While I really do not want to write my own build system, I am
> wondering if that's realistic, because waf is kind of a moving target,
> only justified in the hard cases (numpy/scipy for example).

Is yaku good-enough to build Cython extensions for 3rd party C library,
build Sphinx docs etc. (within wxpython project)?


Sincerely,
Gour


-- 
Many, many births both you and I have passed. I can remember 
all of them, but you cannot, O subduer of the enemy!

http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

Re: [bento] Bento as main build system - what is needed?

From:
David Cournapeau
Date:
2012-08-01 @ 08:55
On Wed, Aug 1, 2012 at 9:47 AM, Gour <gour@atmarama.net> wrote:
> On Wed, 1 Aug 2012 09:30:37 +0100
> David Cournapeau <cournape@gmail.com> wrote:
>
>> While I really do not want to write my own build system, I am
>> wondering if that's realistic, because waf is kind of a moving target,
>> only justified in the hard cases (numpy/scipy for example).
>
> Is yaku good-enough to build Cython extensions for 3rd party C library,
> build Sphinx docs etc. (within wxpython project)?

Kindof, but I really want yaku to go away before a first bento alpha,
because its implementation is truely horrible. There is one build tool
that fits the bill of being simple and good enough in 95 % of the
cases, but it is sadly python 3 only:
https://github.com/felix-lang/fbuild

David
> Many, many births both you and I have passed. I can remember
> all of them, but you cannot, O subduer of the enemy!
>
> http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

Re: [bento] Bento as main build system - what is needed?

From:
Gour
Date:
2012-08-01 @ 09:04
On Wed, 1 Aug 2012 09:55:37 +0100
David Cournapeau <cournape@gmail.com> wrote:

> Kindof, but I really want yaku to go away before a first bento alpha,
> because its implementation is truely horrible. 

You must be kidding or really putting bar very high. ;)

Bento is already such a nice tool.

> There is one build tool
> that fits the bill of being simple and good enough in 95 % of the
> cases, but it is sadly python 3 only:
> https://github.com/felix-lang/fbuild

Interesting...

Waf is powerful indeed, but we're considering to use something
lighter/simpler since we'll mostly do Python projects.


Sincerely,
Gour


-- 
As fire is covered by smoke, as a mirror is covered by dust, 
or as the embryo is covered by the womb, the living entity is 
similarly covered by different degrees of this lust.

http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

Re: [bento] Bento as main build system - what is needed?

From:
David Cournapeau
Date:
2012-08-01 @ 11:17
On Wed, Aug 1, 2012 at 10:04 AM, Gour <gour@atmarama.net> wrote:
> On Wed, 1 Aug 2012 09:55:37 +0100
> David Cournapeau <cournape@gmail.com> wrote:
>
>> Kindof, but I really want yaku to go away before a first bento alpha,
>> because its implementation is truely horrible.
>
> You must be kidding or really putting bar very high. ;)
>
> Bento is already such a nice tool.

I was talking about yaku specifically. Bento's bar needs to be very
high to have a chance to replace the existing python tools, as well !

>
>> There is one build tool
>> that fits the bill of being simple and good enough in 95 % of the
>> cases, but it is sadly python 3 only:
>> https://github.com/felix-lang/fbuild
>
> Interesting...
>
> Waf is powerful indeed, but we're considering to use something
> lighter/simpler since we'll mostly do Python projects.

Yes, waf fills the niche of "arbitrary complex builds", but at a
slight cost for packages that have only one or two simple extensions
and don't depend on many externaly libraries. This likely covers the
vast majority of python softwares out there.

David

Re: [bento] Bento as main build system - what is needed?

From:
Gour
Date:
2012-08-02 @ 08:44
On Wed, 1 Aug 2012 12:17:20 +0100
David Cournapeau <cournape@gmail.com> wrote:

> I was talking about yaku specifically. Bento's bar needs to be very
> high to have a chance to replace the existing python tools, as well !

Wishing you all the best to achieve it!! 

/me hopes to be able to help a bit with the time

> Yes, waf fills the niche of "arbitrary complex builds", but at a
> slight cost for packages that have only one or two simple extensions
> and don't depend on many externaly libraries. This likely covers the
> vast majority of python softwares out there.

Should I say that that I'd be *very* pleased if yaku or how would you
call it, could fit in those 95%  for my use case enabling me to cut
one tool in our toolchain.

Do you plan to release 0.2 with current version of yaku or bento alpha
is planned for post 0.2?


Sincerely,
Gour


-- 
He who is regulated in his habits of eating, sleeping, recreation 
and work can mitigate all material pains by practicing the yoga system.

http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

Re: [bento] Bento as main build system - what is needed?

From:
Gour
Date:
2012-08-01 @ 10:43
On Wed, 1 Aug 2012 09:55:37 +0100
David Cournapeau <cournape@gmail.com> wrote:

> There is one build tool that fits the bill of being simple and good
> enough in 95 % of the cases, but it is sadly python 3 only:

On Waf site there is comparison of different build tools and for Fbuild
it says 'No' for "Build folders" feature. 

Is it still the case?

I consider that having support for seaprate build folders is kind of
'must' today...


Sincerely,
Gour

-- 
One must deliver himself with the help of his mind, and not 
degrade himself. The mind is the friend of the conditioned soul, 
and his enemy as well.

http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

Re: [bento] Bento as main build system - what is needed?

From:
Gour
Date:
2012-08-02 @ 08:46
On Wed, 1 Aug 2012 09:55:37 +0100
David Cournapeau <cournape@gmail.com> wrote:

> There is one build tool that fits the bill of being simple and good
> enough in 95 % of the cases, but it is sadly python 3 only:

I mentioned it to the fbuild developer, but he told me that the feature
required for fbuild is, afahct, not backported to python2.


Sincerely,
Gour


-- 
The senses are so strong and impetuous, O Arjuna, 
that they forcibly carry away the mind even of a man 
of discrimination who is endeavoring to control them.

http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810