librelist archives

« back to archive

IncludeDirs error

IncludeDirs error

From:
Matthew Brett
Date:
2012-06-28 @ 06:32
Hi,

I'm trying to get the bento build working for nipy.

My current attempt (nearly all from David of course) is:

https://github.com/matthew-brett/nipy/tree/bento-build

Running `bentomaker build` from current trunk, I get:

[mb312@tom ~/dev_trees/nipy (bento-build)]$ BENTOMAKER_DEBUG=1 bentomaker build

Checking for 'c_bgxlc' (c compiler)      : not found
Checking for 'gcc' (c compiler)          : /usr/bin/gcc
Checking for program python              : ['/usr/bin/python']
Checking for python version              : (2, 7, 2, 'final', 0)
Checking for program cython              : /usr/bin/cython
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
er.py", line 487, in noexc_main
    main(argv)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
er.py", line 250, in main
    return _wrapped_main(global_context, popts, run_node, top_node, build_node)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
er.py", line 324, in _wrapped_main
    return _main(global_context, cached_package, popts, run_node, top_node, buil
d_node)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
er.py", line 401, in _main
    run_cmd(global_context, cached_package, cmd_name, cmd_argv, run_node, top_no
de, build_node)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
er.py", line 470, in run_cmd
    run_with_dependencies(global_context, cmd_name, cmd_argv, run_node, top_node
, running_package)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
_utils.py", line 18, in run_with_dependencies
    resolve_and_run_command(global_context, cmd_name, cmd_argv, run_node, packag
e)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
_utils.py", line 27, in resolve_and_run_command
    context = context_klass(global_context, cmd_argv, options_context, package,
run_node)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/backends/waf_bac
kend.py", line 276, in __init__
    super(BuildWafContext, self).__init__(global_context, cmd_argv, options_cont
ext, pkg, run_node)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/command
_contexts.py", line 348, in __init__
    self._node_pkg.update_package(pkg)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
e.py", line 211, in update_package
    self._update_libraries(pkg)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
e.py", line 147, in _update_libraries
    compiled_library = self.to_node_extension(compiled_library, self.top_node, r
ef_node)
  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
e.py", line 120, in to_node_extension
    raise NotImplementedError("include dirs translation not implemented yet")
bentomaker: Error: bentomaker crashed (uncaught exception <type 'exceptions.NotI
mplementedError'>: include dirs translation not implemented yet).
Please report this on bento issue tracker:
    http://github.com/cournape/bento/issues

In my bento.info, I have this:

    CompiledLibrary: cstat
        Sources:
            libcstat/fff/*.c,
            libcstat/wrapper/*.c
        IncludeDirs:
            libcstat/fff

If I remove this section, the build proceeds, but I get an expected
crash when the build is trying to link against the cstat library.

I couldn't find any examples around the place of an IncludeDirs ; have
I got the syntax right?

Thanks a lot,

Matthew

Re: [bento] IncludeDirs error

From:
David Cournapeau
Date:
2012-06-28 @ 15:52
On Thu, Jun 28, 2012 at 7:32 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
> Hi,
>
> I'm trying to get the bento build working for nipy.
>
> My current attempt (nearly all from David of course) is:
>
> https://github.com/matthew-brett/nipy/tree/bento-build
>
> Running `bentomaker build` from current trunk, I get:
>
> [mb312@tom ~/dev_trees/nipy (bento-build)]$ BENTOMAKER_DEBUG=1 bentomaker build
>
> Checking for 'c_bgxlc' (c compiler)      : not found
> Checking for 'gcc' (c compiler)          : /usr/bin/gcc
> Checking for program python              : ['/usr/bin/python']
> Checking for python version              : (2, 7, 2, 'final', 0)
> Checking for program cython              : /usr/bin/cython
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
> er.py", line 487, in noexc_main
>    main(argv)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
> er.py", line 250, in main
>    return _wrapped_main(global_context, popts, run_node, top_node, build_node)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
> er.py", line 324, in _wrapped_main
>    return _main(global_context, cached_package, popts, run_node, top_node, buil
> d_node)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
> er.py", line 401, in _main
>    run_cmd(global_context, cached_package, cmd_name, cmd_argv, run_node, top_no
> de, build_node)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
> er.py", line 470, in run_cmd
>    run_with_dependencies(global_context, cmd_name, cmd_argv, run_node, top_node
> , running_package)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
> _utils.py", line 18, in run_with_dependencies
>    resolve_and_run_command(global_context, cmd_name, cmd_argv, run_node, packag
> e)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
> _utils.py", line 27, in resolve_and_run_command
>    context = context_klass(global_context, cmd_argv, options_context, package,
> run_node)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/backends/waf_bac
> kend.py", line 276, in __init__
>    super(BuildWafContext, self).__init__(global_context, cmd_argv, options_cont
> ext, pkg, run_node)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/command
> _contexts.py", line 348, in __init__
>    self._node_pkg.update_package(pkg)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
> e.py", line 211, in update_package
>    self._update_libraries(pkg)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
> e.py", line 147, in _update_libraries
>    compiled_library = self.to_node_extension(compiled_library, self.top_node, r
> ef_node)
>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
> e.py", line 120, in to_node_extension
>    raise NotImplementedError("include dirs translation not implemented yet")
> bentomaker: Error: bentomaker crashed (uncaught exception <type 'exceptions.NotI
> mplementedError'>: include dirs translation not implemented yet).
> Please report this on bento issue tracker:
>    http://github.com/cournape/bento/issues
>
> In my bento.info, I have this:
>
>    CompiledLibrary: cstat
>        Sources:
>            libcstat/fff/*.c,
>            libcstat/wrapper/*.c
>        IncludeDirs:
>            libcstat/fff
>
> If I remove this section, the build proceeds, but I get an expected
> crash when the build is trying to link against the cstat library.
>
> I couldn't find any examples around the place of an IncludeDirs ; have
> I got the syntax right?

Yes, it is a bug/missing feature in bento.

I have added code to deal with the package_dir feature of distutils,
but did not implement it for include headers (hence the exception). In
your case, the current code could actually work, so this should be
easy to fix.

David

Re: [bento] IncludeDirs error

From:
Matthew Brett
Date:
2012-06-29 @ 01:47
Hi,

On Thu, Jun 28, 2012 at 8:52 AM, David Cournapeau <cournape@gmail.com> wrote:
> On Thu, Jun 28, 2012 at 7:32 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
>> Hi,
>>
>> I'm trying to get the bento build working for nipy.
>>
>> My current attempt (nearly all from David of course) is:
>>
>> https://github.com/matthew-brett/nipy/tree/bento-build
>>
>> Running `bentomaker build` from current trunk, I get:
>>
>> [mb312@tom ~/dev_trees/nipy (bento-build)]$ BENTOMAKER_DEBUG=1 bentomaker build
>>
>> Checking for 'c_bgxlc' (c compiler)      : not found
>> Checking for 'gcc' (c compiler)          : /usr/bin/gcc
>> Checking for program python              : ['/usr/bin/python']
>> Checking for python version              : (2, 7, 2, 'final', 0)
>> Checking for program cython              : /usr/bin/cython
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>> er.py", line 487, in noexc_main
>>    main(argv)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>> er.py", line 250, in main
>>    return _wrapped_main(global_context, popts, run_node, top_node, build_node)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>> er.py", line 324, in _wrapped_main
>>    return _main(global_context, cached_package, popts, run_node, top_node, buil
>> d_node)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>> er.py", line 401, in _main
>>    run_cmd(global_context, cached_package, cmd_name, cmd_argv, run_node, top_no
>> de, build_node)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>> er.py", line 470, in run_cmd
>>    run_with_dependencies(global_context, cmd_name, cmd_argv, run_node, top_node
>> , running_package)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>> _utils.py", line 18, in run_with_dependencies
>>    resolve_and_run_command(global_context, cmd_name, cmd_argv, run_node, packag
>> e)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>> _utils.py", line 27, in resolve_and_run_command
>>    context = context_klass(global_context, cmd_argv, options_context, package,
>> run_node)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/backends/waf_bac
>> kend.py", line 276, in __init__
>>    super(BuildWafContext, self).__init__(global_context, cmd_argv, options_cont
>> ext, pkg, run_node)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/command
>> _contexts.py", line 348, in __init__
>>    self._node_pkg.update_package(pkg)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>> e.py", line 211, in update_package
>>    self._update_libraries(pkg)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>> e.py", line 147, in _update_libraries
>>    compiled_library = self.to_node_extension(compiled_library, self.top_node, r
>> ef_node)
>>  File "/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>> e.py", line 120, in to_node_extension
>>    raise NotImplementedError("include dirs translation not implemented yet")
>> bentomaker: Error: bentomaker crashed (uncaught exception <type 
'exceptions.NotI
>> mplementedError'>: include dirs translation not implemented yet).
>> Please report this on bento issue tracker:
>>    http://github.com/cournape/bento/issues
>>
>> In my bento.info, I have this:
>>
>>    CompiledLibrary: cstat
>>        Sources:
>>            libcstat/fff/*.c,
>>            libcstat/wrapper/*.c
>>        IncludeDirs:
>>            libcstat/fff
>>
>> If I remove this section, the build proceeds, but I get an expected
>> crash when the build is trying to link against the cstat library.
>>
>> I couldn't find any examples around the place of an IncludeDirs ; have
>> I got the syntax right?
>
> Yes, it is a bug/missing feature in bento.
>
> I have added code to deal with the package_dir feature of distutils,
> but did not implement it for include headers (hence the exception). In
> your case, the current code could actually work, so this should be
> easy to fix.

Anything I can do to help?

Cheers,

Matthew

Re: [bento] IncludeDirs error

From:
Matthew Brett
Date:
2012-06-30 @ 17:10
Hi,

On Thu, Jun 28, 2012 at 6:47 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
> Hi,
>
> On Thu, Jun 28, 2012 at 8:52 AM, David Cournapeau <cournape@gmail.com> wrote:
>> On Thu, Jun 28, 2012 at 7:32 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
>>> Hi,
>>>
>>> I'm trying to get the bento build working for nipy.
>>>
>>> My current attempt (nearly all from David of course) is:
>>>
>>> https://github.com/matthew-brett/nipy/tree/bento-build
>>>
>>> Running `bentomaker build` from current trunk, I get:
>>>
>>> [mb312@tom ~/dev_trees/nipy (bento-build)]$ BENTOMAKER_DEBUG=1 
bentomaker build
>>>
>>> Checking for 'c_bgxlc' (c compiler)      : not found
>>> Checking for 'gcc' (c compiler)          : /usr/bin/gcc
>>> Checking for program python              : ['/usr/bin/python']
>>> Checking for python version              : (2, 7, 2, 'final', 0)
>>> Checking for program cython              : /usr/bin/cython
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>> er.py", line 487, in noexc_main
>>>    main(argv)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>> er.py", line 250, in main
>>>    return _wrapped_main(global_context, popts, run_node, top_node, build_node)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>> er.py", line 324, in _wrapped_main
>>>    return _main(global_context, cached_package, popts, run_node, 
top_node, buil
>>> d_node)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>> er.py", line 401, in _main
>>>    run_cmd(global_context, cached_package, cmd_name, cmd_argv, 
run_node, top_no
>>> de, build_node)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>> er.py", line 470, in run_cmd
>>>    run_with_dependencies(global_context, cmd_name, cmd_argv, run_node,
top_node
>>> , running_package)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>> _utils.py", line 18, in run_with_dependencies
>>>    resolve_and_run_command(global_context, cmd_name, cmd_argv, 
run_node, packag
>>> e)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>> _utils.py", line 27, in resolve_and_run_command
>>>    context = context_klass(global_context, cmd_argv, options_context, package,
>>> run_node)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/backends/waf_bac
>>> kend.py", line 276, in __init__
>>>    super(BuildWafContext, self).__init__(global_context, cmd_argv, 
options_cont
>>> ext, pkg, run_node)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/command
>>> _contexts.py", line 348, in __init__
>>>    self._node_pkg.update_package(pkg)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>> e.py", line 211, in update_package
>>>    self._update_libraries(pkg)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>> e.py", line 147, in _update_libraries
>>>    compiled_library = self.to_node_extension(compiled_library, 
self.top_node, r
>>> ef_node)
>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>> e.py", line 120, in to_node_extension
>>>    raise NotImplementedError("include dirs translation not implemented yet")
>>> bentomaker: Error: bentomaker crashed (uncaught exception <type 
'exceptions.NotI
>>> mplementedError'>: include dirs translation not implemented yet).
>>> Please report this on bento issue tracker:
>>>    http://github.com/cournape/bento/issues
>>>
>>> In my bento.info, I have this:
>>>
>>>    CompiledLibrary: cstat
>>>        Sources:
>>>            libcstat/fff/*.c,
>>>            libcstat/wrapper/*.c
>>>        IncludeDirs:
>>>            libcstat/fff
>>>
>>> If I remove this section, the build proceeds, but I get an expected
>>> crash when the build is trying to link against the cstat library.
>>>
>>> I couldn't find any examples around the place of an IncludeDirs ; have
>>> I got the syntax right?
>>
>> Yes, it is a bug/missing feature in bento.
>>
>> I have added code to deal with the package_dir feature of distutils,
>> but did not implement it for include headers (hence the exception). In
>> your case, the current code could actually work, so this should be
>> easy to fix.
>
> Anything I can do to help?

Or - more specifically - can you point me to the places I could look
to fix this myself?

Thanks a lot,

Matthew

Re: [bento] IncludeDirs error

From:
David Cournapeau
Date:
2012-07-01 @ 12:42
Hi Matthew,

On Sat, Jun 30, 2012 at 6:10 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
> Hi,
>
> On Thu, Jun 28, 2012 at 6:47 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
>> Hi,
>>
>> On Thu, Jun 28, 2012 at 8:52 AM, David Cournapeau <cournape@gmail.com> wrote:
>>> On Thu, Jun 28, 2012 at 7:32 AM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I'm trying to get the bento build working for nipy.
>>>>
>>>> My current attempt (nearly all from David of course) is:
>>>>
>>>> https://github.com/matthew-brett/nipy/tree/bento-build
>>>>
>>>> Running `bentomaker build` from current trunk, I get:
>>>>
>>>> [mb312@tom ~/dev_trees/nipy (bento-build)]$ BENTOMAKER_DEBUG=1 
bentomaker build
>>>>
>>>> Checking for 'c_bgxlc' (c compiler)      : not found
>>>> Checking for 'gcc' (c compiler)          : /usr/bin/gcc
>>>> Checking for program python              : ['/usr/bin/python']
>>>> Checking for python version              : (2, 7, 2, 'final', 0)
>>>> Checking for program cython              : /usr/bin/cython
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>> er.py", line 487, in noexc_main
>>>>    main(argv)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>> er.py", line 250, in main
>>>>    return _wrapped_main(global_context, popts, run_node, top_node, 
build_node)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>> er.py", line 324, in _wrapped_main
>>>>    return _main(global_context, cached_package, popts, run_node, 
top_node, buil
>>>> d_node)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>> er.py", line 401, in _main
>>>>    run_cmd(global_context, cached_package, cmd_name, cmd_argv, 
run_node, top_no
>>>> de, build_node)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>> er.py", line 470, in run_cmd
>>>>    run_with_dependencies(global_context, cmd_name, cmd_argv, 
run_node, top_node
>>>> , running_package)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>> _utils.py", line 18, in run_with_dependencies
>>>>    resolve_and_run_command(global_context, cmd_name, cmd_argv, 
run_node, packag
>>>> e)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>> _utils.py", line 27, in resolve_and_run_command
>>>>    context = context_klass(global_context, cmd_argv, options_context,
package,
>>>> run_node)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/backends/waf_bac
>>>> kend.py", line 276, in __init__
>>>>    super(BuildWafContext, self).__init__(global_context, cmd_argv, 
options_cont
>>>> ext, pkg, run_node)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/command
>>>> _contexts.py", line 348, in __init__
>>>>    self._node_pkg.update_package(pkg)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>> e.py", line 211, in update_package
>>>>    self._update_libraries(pkg)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>> e.py", line 147, in _update_libraries
>>>>    compiled_library = self.to_node_extension(compiled_library, 
self.top_node, r
>>>> ef_node)
>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>> e.py", line 120, in to_node_extension
>>>>    raise NotImplementedError("include dirs translation not implemented yet")
>>>> bentomaker: Error: bentomaker crashed (uncaught exception <type 
'exceptions.NotI
>>>> mplementedError'>: include dirs translation not implemented yet).
>>>> Please report this on bento issue tracker:
>>>>    http://github.com/cournape/bento/issues
>>>>
>>>> In my bento.info, I have this:
>>>>
>>>>    CompiledLibrary: cstat
>>>>        Sources:
>>>>            libcstat/fff/*.c,
>>>>            libcstat/wrapper/*.c
>>>>        IncludeDirs:
>>>>            libcstat/fff
>>>>
>>>> If I remove this section, the build proceeds, but I get an expected
>>>> crash when the build is trying to link against the cstat library.
>>>>
>>>> I couldn't find any examples around the place of an IncludeDirs ; have
>>>> I got the syntax right?
>>>
>>> Yes, it is a bug/missing feature in bento.
>>>
>>> I have added code to deal with the package_dir feature of distutils,
>>> but did not implement it for include headers (hence the exception). In
>>> your case, the current code could actually work, so this should be
>>> easy to fix.
>>
>> Anything I can do to help?
>
> Or - more specifically - can you point me to the places I could look
> to fix this myself?

It would be hard do fix correctly at the moment, as the code in
question is a bit complicated (related to some mostly undocumented
design related to how build/source trees behave).

I made a hack and pushed the necessary changes to github
(7236bbc9815b21c012c156efb062c2ccefb837a2 and
761f9c37f96417090eccce03119ff2ad8a6e1e49), and you need to add the
following in your bscript:

diff --git a/bscript b/bscript
index 5f016c0..201ddb8 100644
--- a/bscript
+++ b/bscript
@@ -12,5 +12,6 @@ def post_configure(context):
     conf.env.INCLUDES = get_numpy_include_dirs()
     for p in ["wrapper", "fff"]:
         conf.env.INCLUDES.append(op.join("libcstat", p))
+        conf.env.CYTHON_INCLUDES.append(op.join("libcstat", p))
     for p in ["segmentation", "registration"]:
         conf.env.INCLUDES.append(op.join("nipy", "algorithms", p))

The hack is that IncludeDirs is basically ignored at that point (this
is issue 115)

David

Re: [bento] IncludeDirs error

From:
Matthew Brett
Date:
2012-07-02 @ 22:14
Hi,

On Sun, Jul 1, 2012 at 5:42 AM, David Cournapeau <cournape@gmail.com> wrote:
> Hi Matthew,
>
> On Sat, Jun 30, 2012 at 6:10 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
>> Hi,
>>
>> On Thu, Jun 28, 2012 at 6:47 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
>>> Hi,
>>>
>>> On Thu, Jun 28, 2012 at 8:52 AM, David Cournapeau <cournape@gmail.com> wrote:
>>>> On Thu, Jun 28, 2012 at 7:32 AM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I'm trying to get the bento build working for nipy.
>>>>>
>>>>> My current attempt (nearly all from David of course) is:
>>>>>
>>>>> https://github.com/matthew-brett/nipy/tree/bento-build
>>>>>
>>>>> Running `bentomaker build` from current trunk, I get:
>>>>>
>>>>> [mb312@tom ~/dev_trees/nipy (bento-build)]$ BENTOMAKER_DEBUG=1 
bentomaker build
>>>>>
>>>>> Checking for 'c_bgxlc' (c compiler)      : not found
>>>>> Checking for 'gcc' (c compiler)          : /usr/bin/gcc
>>>>> Checking for program python              : ['/usr/bin/python']
>>>>> Checking for python version              : (2, 7, 2, 'final', 0)
>>>>> Checking for program cython              : /usr/bin/cython
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>> er.py", line 487, in noexc_main
>>>>>    main(argv)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>> er.py", line 250, in main
>>>>>    return _wrapped_main(global_context, popts, run_node, top_node, 
build_node)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>> er.py", line 324, in _wrapped_main
>>>>>    return _main(global_context, cached_package, popts, run_node, 
top_node, buil
>>>>> d_node)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>> er.py", line 401, in _main
>>>>>    run_cmd(global_context, cached_package, cmd_name, cmd_argv, 
run_node, top_no
>>>>> de, build_node)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>> er.py", line 470, in run_cmd
>>>>>    run_with_dependencies(global_context, cmd_name, cmd_argv, 
run_node, top_node
>>>>> , running_package)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>>> _utils.py", line 18, in run_with_dependencies
>>>>>    resolve_and_run_command(global_context, cmd_name, cmd_argv, 
run_node, packag
>>>>> e)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>>> _utils.py", line 27, in resolve_and_run_command
>>>>>    context = context_klass(global_context, cmd_argv, 
options_context, package,
>>>>> run_node)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/backends/waf_bac
>>>>> kend.py", line 276, in __init__
>>>>>    super(BuildWafContext, self).__init__(global_context, cmd_argv, 
options_cont
>>>>> ext, pkg, run_node)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/command
>>>>> _contexts.py", line 348, in __init__
>>>>>    self._node_pkg.update_package(pkg)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>> e.py", line 211, in update_package
>>>>>    self._update_libraries(pkg)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>> e.py", line 147, in _update_libraries
>>>>>    compiled_library = self.to_node_extension(compiled_library, 
self.top_node, r
>>>>> ef_node)
>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>> e.py", line 120, in to_node_extension
>>>>>    raise NotImplementedError("include dirs translation not implemented yet")
>>>>> bentomaker: Error: bentomaker crashed (uncaught exception <type 
'exceptions.NotI
>>>>> mplementedError'>: include dirs translation not implemented yet).
>>>>> Please report this on bento issue tracker:
>>>>>    http://github.com/cournape/bento/issues
>>>>>
>>>>> In my bento.info, I have this:
>>>>>
>>>>>    CompiledLibrary: cstat
>>>>>        Sources:
>>>>>            libcstat/fff/*.c,
>>>>>            libcstat/wrapper/*.c
>>>>>        IncludeDirs:
>>>>>            libcstat/fff
>>>>>
>>>>> If I remove this section, the build proceeds, but I get an expected
>>>>> crash when the build is trying to link against the cstat library.
>>>>>
>>>>> I couldn't find any examples around the place of an IncludeDirs ; have
>>>>> I got the syntax right?
>>>>
>>>> Yes, it is a bug/missing feature in bento.
>>>>
>>>> I have added code to deal with the package_dir feature of distutils,
>>>> but did not implement it for include headers (hence the exception). In
>>>> your case, the current code could actually work, so this should be
>>>> easy to fix.
>>>
>>> Anything I can do to help?
>>
>> Or - more specifically - can you point me to the places I could look
>> to fix this myself?
>
> It would be hard do fix correctly at the moment, as the code in
> question is a bit complicated (related to some mostly undocumented
> design related to how build/source trees behave).
>
> I made a hack and pushed the necessary changes to github
> (7236bbc9815b21c012c156efb062c2ccefb837a2 and
> 761f9c37f96417090eccce03119ff2ad8a6e1e49), and you need to add the
> following in your bscript:
>
> diff --git a/bscript b/bscript
> index 5f016c0..201ddb8 100644
> --- a/bscript
> +++ b/bscript
> @@ -12,5 +12,6 @@ def post_configure(context):
>      conf.env.INCLUDES = get_numpy_include_dirs()
>      for p in ["wrapper", "fff"]:
>          conf.env.INCLUDES.append(op.join("libcstat", p))
> +        conf.env.CYTHON_INCLUDES.append(op.join("libcstat", p))
>      for p in ["segmentation", "registration"]:
>          conf.env.INCLUDES.append(op.join("nipy", "algorithms", p))
>
> The hack is that IncludeDirs is basically ignored at that point (this
> is issue 115)

Thanks a lot for this.  The package builds, but the install hit
run-time link errors of type:

======================================================================
ERROR: Failure: ImportError

(dlopen(/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so,
2): Symbol not found: _fffpy_import_array
  Referenced from:
/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so
  Expected in: dynamic lookup
)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/loader.py",
line 390, in loadTestsFromName
    addr.filename, addr.module)
  File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py",
line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py",
line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/modalities/fmri/tests/test_glm.py",
line 10, in <module>
    from ..glm import GeneralLinearModel, data_scaling
  File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/modalities/fmri/glm.py",
line 30, in <module>
    from nipy.labs.utils import mahalanobis
  File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/__init__.py",
line 3, in <module>
    from .routines import (quantile, median, mahalanobis, svd, permutations,
ImportError: 
dlopen(/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so,
2): Symbol not found: _fffpy_import_array
  Referenced from:
/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so
  Expected in: dynamic lookup

I see from the docs:

http://bento.readthedocs.org/en/latest/tutorial.html#adding-compiled-libraries

that I need to use the hook mechanism to link (in this case) routines
to the cstat library.

I feel bad to confess, but I could not see how to do this.  Can you
give any pointers?

Sorry, and thanks,

Matthew

Re: [bento] IncludeDirs error

From:
David Cournapeau
Date:
2012-07-03 @ 09:36
On Mon, Jul 2, 2012 at 11:14 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
> Hi,
>
> On Sun, Jul 1, 2012 at 5:42 AM, David Cournapeau <cournape@gmail.com> wrote:
>> Hi Matthew,
>>
>> On Sat, Jun 30, 2012 at 6:10 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
>>> Hi,
>>>
>>> On Thu, Jun 28, 2012 at 6:47 PM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> On Thu, Jun 28, 2012 at 8:52 AM, David Cournapeau <cournape@gmail.com> wrote:
>>>>> On Thu, Jun 28, 2012 at 7:32 AM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm trying to get the bento build working for nipy.
>>>>>>
>>>>>> My current attempt (nearly all from David of course) is:
>>>>>>
>>>>>> https://github.com/matthew-brett/nipy/tree/bento-build
>>>>>>
>>>>>> Running `bentomaker build` from current trunk, I get:
>>>>>>
>>>>>> [mb312@tom ~/dev_trees/nipy (bento-build)]$ BENTOMAKER_DEBUG=1 
bentomaker build
>>>>>>
>>>>>> Checking for 'c_bgxlc' (c compiler)      : not found
>>>>>> Checking for 'gcc' (c compiler)          : /usr/bin/gcc
>>>>>> Checking for program python              : ['/usr/bin/python']
>>>>>> Checking for python version              : (2, 7, 2, 'final', 0)
>>>>>> Checking for program cython              : /usr/bin/cython
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>> er.py", line 487, in noexc_main
>>>>>>    main(argv)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>> er.py", line 250, in main
>>>>>>    return _wrapped_main(global_context, popts, run_node, top_node, 
build_node)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>> er.py", line 324, in _wrapped_main
>>>>>>    return _main(global_context, cached_package, popts, run_node, 
top_node, buil
>>>>>> d_node)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>> er.py", line 401, in _main
>>>>>>    run_cmd(global_context, cached_package, cmd_name, cmd_argv, 
run_node, top_no
>>>>>> de, build_node)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>> er.py", line 470, in run_cmd
>>>>>>    run_with_dependencies(global_context, cmd_name, cmd_argv, 
run_node, top_node
>>>>>> , running_package)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>>>> _utils.py", line 18, in run_with_dependencies
>>>>>>    resolve_and_run_command(global_context, cmd_name, cmd_argv, 
run_node, packag
>>>>>> e)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>>>> _utils.py", line 27, in resolve_and_run_command
>>>>>>    context = context_klass(global_context, cmd_argv, 
options_context, package,
>>>>>> run_node)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/backends/waf_bac
>>>>>> kend.py", line 276, in __init__
>>>>>>    super(BuildWafContext, self).__init__(global_context, cmd_argv, 
options_cont
>>>>>> ext, pkg, run_node)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/command
>>>>>> _contexts.py", line 348, in __init__
>>>>>>    self._node_pkg.update_package(pkg)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>>> e.py", line 211, in update_package
>>>>>>    self._update_libraries(pkg)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>>> e.py", line 147, in _update_libraries
>>>>>>    compiled_library = self.to_node_extension(compiled_library, 
self.top_node, r
>>>>>> ef_node)
>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>>> e.py", line 120, in to_node_extension
>>>>>>    raise NotImplementedError("include dirs translation not 
implemented yet")
>>>>>> bentomaker: Error: bentomaker crashed (uncaught exception <type 
'exceptions.NotI
>>>>>> mplementedError'>: include dirs translation not implemented yet).
>>>>>> Please report this on bento issue tracker:
>>>>>>    http://github.com/cournape/bento/issues
>>>>>>
>>>>>> In my bento.info, I have this:
>>>>>>
>>>>>>    CompiledLibrary: cstat
>>>>>>        Sources:
>>>>>>            libcstat/fff/*.c,
>>>>>>            libcstat/wrapper/*.c
>>>>>>        IncludeDirs:
>>>>>>            libcstat/fff
>>>>>>
>>>>>> If I remove this section, the build proceeds, but I get an expected
>>>>>> crash when the build is trying to link against the cstat library.
>>>>>>
>>>>>> I couldn't find any examples around the place of an IncludeDirs ; have
>>>>>> I got the syntax right?
>>>>>
>>>>> Yes, it is a bug/missing feature in bento.
>>>>>
>>>>> I have added code to deal with the package_dir feature of distutils,
>>>>> but did not implement it for include headers (hence the exception). In
>>>>> your case, the current code could actually work, so this should be
>>>>> easy to fix.
>>>>
>>>> Anything I can do to help?
>>>
>>> Or - more specifically - can you point me to the places I could look
>>> to fix this myself?
>>
>> It would be hard do fix correctly at the moment, as the code in
>> question is a bit complicated (related to some mostly undocumented
>> design related to how build/source trees behave).
>>
>> I made a hack and pushed the necessary changes to github
>> (7236bbc9815b21c012c156efb062c2ccefb837a2 and
>> 761f9c37f96417090eccce03119ff2ad8a6e1e49), and you need to add the
>> following in your bscript:
>>
>> diff --git a/bscript b/bscript
>> index 5f016c0..201ddb8 100644
>> --- a/bscript
>> +++ b/bscript
>> @@ -12,5 +12,6 @@ def post_configure(context):
>>      conf.env.INCLUDES = get_numpy_include_dirs()
>>      for p in ["wrapper", "fff"]:
>>          conf.env.INCLUDES.append(op.join("libcstat", p))
>> +        conf.env.CYTHON_INCLUDES.append(op.join("libcstat", p))
>>      for p in ["segmentation", "registration"]:
>>          conf.env.INCLUDES.append(op.join("nipy", "algorithms", p))
>>
>> The hack is that IncludeDirs is basically ignored at that point (this
>> is issue 115)
>
> Thanks a lot for this.  The package builds, but the install hit
> run-time link errors of type:
>
> ======================================================================
> ERROR: Failure: ImportError
> 
(dlopen(/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so,
> 2): Symbol not found: _fffpy_import_array
>   Referenced from:
> /Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so
>   Expected in: dynamic lookup
> )
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/loader.py",
> line 390, in loadTestsFromName
>     addr.filename, addr.module)
>   File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py",
> line 39, in importFromPath
>     return self.importFromDir(dir_path, fqname)
>   File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py",
> line 86, in importFromDir
>     mod = load_module(part_fqname, fh, filename, desc)
>   File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/modalities/fmri/tests/test_glm.py",
> line 10, in <module>
>     from ..glm import GeneralLinearModel, data_scaling
>   File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/modalities/fmri/glm.py",
> line 30, in <module>
>     from nipy.labs.utils import mahalanobis
>   File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/__init__.py",
> line 3, in <module>
>     from .routines import (quantile, median, mahalanobis, svd, permutations,
> ImportError: 
dlopen(/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so,
> 2): Symbol not found: _fffpy_import_array
>   Referenced from:
> /Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so
>   Expected in: dynamic lookup
>
> I see from the docs:
>
> http://bento.readthedocs.org/en/latest/tutorial.html#adding-compiled-libraries
>
> that I need to use the hook mechanism to link (in this case) routines
> to the cstat library.
>
> I feel bad to confess, but I could not see how to do this.

Oh, you should feel very bad

>  Can you
> give any pointers?

Short answer:

@pre_build
def pre_build(context):
    context.tweak_extension("my_extension", use="foo") # link against libfoo

Long answer:

Look at the scipy bscript. This magic is coming from waf itself. It is
still a big hackish, but the function context.tweak_extension allow
you to override any parameter given to the underlying waf task
generator, that is tweak_extension(name, **kw) is equivalent to
calling bld(**kw) in waf (see
http://docs.waf.googlecode.com/git/apidocs_16/tutorial.html#a-c-and-c-project)

David

Re: [bento] IncludeDirs error

From:
Matthew Brett
Date:
2012-07-03 @ 20:16
Hi,

On Tue, Jul 3, 2012 at 2:36 AM, David Cournapeau <cournape@gmail.com> wrote:
> On Mon, Jul 2, 2012 at 11:14 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
>> Hi,
>>
>> On Sun, Jul 1, 2012 at 5:42 AM, David Cournapeau <cournape@gmail.com> wrote:
>>> Hi Matthew,
>>>
>>> On Sat, Jun 30, 2012 at 6:10 PM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> On Thu, Jun 28, 2012 at 6:47 PM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> On Thu, Jun 28, 2012 at 8:52 AM, David Cournapeau 
<cournape@gmail.com> wrote:
>>>>>> On Thu, Jun 28, 2012 at 7:32 AM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm trying to get the bento build working for nipy.
>>>>>>>
>>>>>>> My current attempt (nearly all from David of course) is:
>>>>>>>
>>>>>>> https://github.com/matthew-brett/nipy/tree/bento-build
>>>>>>>
>>>>>>> Running `bentomaker build` from current trunk, I get:
>>>>>>>
>>>>>>> [mb312@tom ~/dev_trees/nipy (bento-build)]$ BENTOMAKER_DEBUG=1 
bentomaker build
>>>>>>>
>>>>>>> Checking for 'c_bgxlc' (c compiler)      : not found
>>>>>>> Checking for 'gcc' (c compiler)          : /usr/bin/gcc
>>>>>>> Checking for program python              : ['/usr/bin/python']
>>>>>>> Checking for python version              : (2, 7, 2, 'final', 0)
>>>>>>> Checking for program cython              : /usr/bin/cython
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>> er.py", line 487, in noexc_main
>>>>>>>    main(argv)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>> er.py", line 250, in main
>>>>>>>    return _wrapped_main(global_context, popts, run_node, top_node,
build_node)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>> er.py", line 324, in _wrapped_main
>>>>>>>    return _main(global_context, cached_package, popts, run_node, 
top_node, buil
>>>>>>> d_node)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>> er.py", line 401, in _main
>>>>>>>    run_cmd(global_context, cached_package, cmd_name, cmd_argv, 
run_node, top_no
>>>>>>> de, build_node)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>> er.py", line 470, in run_cmd
>>>>>>>    run_with_dependencies(global_context, cmd_name, cmd_argv, 
run_node, top_node
>>>>>>> , running_package)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>>>>> _utils.py", line 18, in run_with_dependencies
>>>>>>>    resolve_and_run_command(global_context, cmd_name, cmd_argv, 
run_node, packag
>>>>>>> e)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>>>>> _utils.py", line 27, in resolve_and_run_command
>>>>>>>    context = context_klass(global_context, cmd_argv, 
options_context, package,
>>>>>>> run_node)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/backends/waf_bac
>>>>>>> kend.py", line 276, in __init__
>>>>>>>    super(BuildWafContext, self).__init__(global_context, cmd_argv,
options_cont
>>>>>>> ext, pkg, run_node)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/command
>>>>>>> _contexts.py", line 348, in __init__
>>>>>>>    self._node_pkg.update_package(pkg)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>>>> e.py", line 211, in update_package
>>>>>>>    self._update_libraries(pkg)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>>>> e.py", line 147, in _update_libraries
>>>>>>>    compiled_library = self.to_node_extension(compiled_library, 
self.top_node, r
>>>>>>> ef_node)
>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>>>> e.py", line 120, in to_node_extension
>>>>>>>    raise NotImplementedError("include dirs translation not 
implemented yet")
>>>>>>> bentomaker: Error: bentomaker crashed (uncaught exception <type 
'exceptions.NotI
>>>>>>> mplementedError'>: include dirs translation not implemented yet).
>>>>>>> Please report this on bento issue tracker:
>>>>>>>    http://github.com/cournape/bento/issues
>>>>>>>
>>>>>>> In my bento.info, I have this:
>>>>>>>
>>>>>>>    CompiledLibrary: cstat
>>>>>>>        Sources:
>>>>>>>            libcstat/fff/*.c,
>>>>>>>            libcstat/wrapper/*.c
>>>>>>>        IncludeDirs:
>>>>>>>            libcstat/fff
>>>>>>>
>>>>>>> If I remove this section, the build proceeds, but I get an expected
>>>>>>> crash when the build is trying to link against the cstat library.
>>>>>>>
>>>>>>> I couldn't find any examples around the place of an IncludeDirs ; have
>>>>>>> I got the syntax right?
>>>>>>
>>>>>> Yes, it is a bug/missing feature in bento.
>>>>>>
>>>>>> I have added code to deal with the package_dir feature of distutils,
>>>>>> but did not implement it for include headers (hence the exception). In
>>>>>> your case, the current code could actually work, so this should be
>>>>>> easy to fix.
>>>>>
>>>>> Anything I can do to help?
>>>>
>>>> Or - more specifically - can you point me to the places I could look
>>>> to fix this myself?
>>>
>>> It would be hard do fix correctly at the moment, as the code in
>>> question is a bit complicated (related to some mostly undocumented
>>> design related to how build/source trees behave).
>>>
>>> I made a hack and pushed the necessary changes to github
>>> (7236bbc9815b21c012c156efb062c2ccefb837a2 and
>>> 761f9c37f96417090eccce03119ff2ad8a6e1e49), and you need to add the
>>> following in your bscript:
>>>
>>> diff --git a/bscript b/bscript
>>> index 5f016c0..201ddb8 100644
>>> --- a/bscript
>>> +++ b/bscript
>>> @@ -12,5 +12,6 @@ def post_configure(context):
>>>      conf.env.INCLUDES = get_numpy_include_dirs()
>>>      for p in ["wrapper", "fff"]:
>>>          conf.env.INCLUDES.append(op.join("libcstat", p))
>>> +        conf.env.CYTHON_INCLUDES.append(op.join("libcstat", p))
>>>      for p in ["segmentation", "registration"]:
>>>          conf.env.INCLUDES.append(op.join("nipy", "algorithms", p))
>>>
>>> The hack is that IncludeDirs is basically ignored at that point (this
>>> is issue 115)
>>
>> Thanks a lot for this.  The package builds, but the install hit
>> run-time link errors of type:
>>
>> ======================================================================
>> ERROR: Failure: ImportError
>> 
(dlopen(/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so,
>> 2): Symbol not found: _fffpy_import_array
>>   Referenced from:
>> /Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so
>>   Expected in: dynamic lookup
>> )
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/loader.py",
>> line 390, in loadTestsFromName
>>     addr.filename, addr.module)
>>   File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py",
>> line 39, in importFromPath
>>     return self.importFromDir(dir_path, fqname)
>>   File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py",
>> line 86, in importFromDir
>>     mod = load_module(part_fqname, fh, filename, desc)
>>   File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/modalities/fmri/tests/test_glm.py",
>> line 10, in <module>
>>     from ..glm import GeneralLinearModel, data_scaling
>>   File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/modalities/fmri/glm.py",
>> line 30, in <module>
>>     from nipy.labs.utils import mahalanobis
>>   File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/__init__.py",
>> line 3, in <module>
>>     from .routines import (quantile, median, mahalanobis, svd, permutations,
>> ImportError: 
dlopen(/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so,
>> 2): Symbol not found: _fffpy_import_array
>>   Referenced from:
>> /Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so
>>   Expected in: dynamic lookup
>>
>> I see from the docs:
>>
>> http://bento.readthedocs.org/en/latest/tutorial.html#adding-compiled-libraries
>>
>> that I need to use the hook mechanism to link (in this case) routines
>> to the cstat library.
>>
>> I feel bad to confess, but I could not see how to do this.
>
> Oh, you should feel very bad
>
>>  Can you
>> give any pointers?
>
> Short answer:
>
> @pre_build
> def pre_build(context):
>     context.tweak_extension("my_extension", use="foo") # link against libfoo
>
> Long answer:
>
> Look at the scipy bscript. This magic is coming from waf itself. It is
> still a big hackish, but the function context.tweak_extension allow
> you to override any parameter given to the underlying waf task
> generator, that is tweak_extension(name, **kw) is equivalent to
> calling bld(**kw) in waf (see
> http://docs.waf.googlecode.com/git/apidocs_16/tutorial.html#a-c-and-c-project)

Thanks for that.  I feel myself staggering out of the fog, but it
might be into more fog.

The next problem I hit was the need to link against blas, lapack etc.
You probably remember that we had to ship a lapack_lite for the case
where we could not discover a platform lapack library.

I think this means pulling in the bscript machinery from numpy and
scipy - by copy paste I guess.   Do you have any thoughts about
reusing bscript stuff across packages?

Cheers,

Matthew

Re: [bento] IncludeDirs error

From:
David Cournapeau
Date:
2012-07-04 @ 14:55
Hi Matthew,

On Tue, Jul 3, 2012 at 9:16 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
> Hi,
>
> On Tue, Jul 3, 2012 at 2:36 AM, David Cournapeau <cournape@gmail.com> wrote:
>> On Mon, Jul 2, 2012 at 11:14 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
>>> Hi,
>>>
>>> On Sun, Jul 1, 2012 at 5:42 AM, David Cournapeau <cournape@gmail.com> wrote:
>>>> Hi Matthew,
>>>>
>>>> On Sat, Jun 30, 2012 at 6:10 PM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> On Thu, Jun 28, 2012 at 6:47 PM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On Thu, Jun 28, 2012 at 8:52 AM, David Cournapeau 
<cournape@gmail.com> wrote:
>>>>>>> On Thu, Jun 28, 2012 at 7:32 AM, Matthew Brett 
<matthew.brett@gmail.com> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm trying to get the bento build working for nipy.
>>>>>>>>
>>>>>>>> My current attempt (nearly all from David of course) is:
>>>>>>>>
>>>>>>>> https://github.com/matthew-brett/nipy/tree/bento-build
>>>>>>>>
>>>>>>>> Running `bentomaker build` from current trunk, I get:
>>>>>>>>
>>>>>>>> [mb312@tom ~/dev_trees/nipy (bento-build)]$ BENTOMAKER_DEBUG=1 
bentomaker build
>>>>>>>>
>>>>>>>> Checking for 'c_bgxlc' (c compiler)      : not found
>>>>>>>> Checking for 'gcc' (c compiler)          : /usr/bin/gcc
>>>>>>>> Checking for program python              : ['/usr/bin/python']
>>>>>>>> Checking for python version              : (2, 7, 2, 'final', 0)
>>>>>>>> Checking for program cython              : /usr/bin/cython
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>>> er.py", line 487, in noexc_main
>>>>>>>>    main(argv)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>>> er.py", line 250, in main
>>>>>>>>    return _wrapped_main(global_context, popts, run_node, 
top_node, build_node)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>>> er.py", line 324, in _wrapped_main
>>>>>>>>    return _main(global_context, cached_package, popts, run_node, 
top_node, buil
>>>>>>>> d_node)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>>> er.py", line 401, in _main
>>>>>>>>    run_cmd(global_context, cached_package, cmd_name, cmd_argv, 
run_node, top_no
>>>>>>>> de, build_node)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bentomakerlib/bentomak
>>>>>>>> er.py", line 470, in run_cmd
>>>>>>>>    run_with_dependencies(global_context, cmd_name, cmd_argv, 
run_node, top_node
>>>>>>>> , running_package)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>>>>>> _utils.py", line 18, in run_with_dependencies
>>>>>>>>    resolve_and_run_command(global_context, cmd_name, cmd_argv, 
run_node, packag
>>>>>>>> e)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/wrapper
>>>>>>>> _utils.py", line 27, in resolve_and_run_command
>>>>>>>>    context = context_klass(global_context, cmd_argv, 
options_context, package,
>>>>>>>> run_node)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/backends/waf_bac
>>>>>>>> kend.py", line 276, in __init__
>>>>>>>>    super(BuildWafContext, self).__init__(global_context, 
cmd_argv, options_cont
>>>>>>>> ext, pkg, run_node)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/commands/command
>>>>>>>> _contexts.py", line 348, in __init__
>>>>>>>>    self._node_pkg.update_package(pkg)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>>>>> e.py", line 211, in update_package
>>>>>>>>    self._update_libraries(pkg)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>>>>> e.py", line 147, in _update_libraries
>>>>>>>>    compiled_library = self.to_node_extension(compiled_library, 
self.top_node, r
>>>>>>>> ef_node)
>>>>>>>>  File 
"/home/mb312/usr/local/lib/python2.7/site-packages/bento/core/node_packag
>>>>>>>> e.py", line 120, in to_node_extension
>>>>>>>>    raise NotImplementedError("include dirs translation not 
implemented yet")
>>>>>>>> bentomaker: Error: bentomaker crashed (uncaught exception <type 
'exceptions.NotI
>>>>>>>> mplementedError'>: include dirs translation not implemented yet).
>>>>>>>> Please report this on bento issue tracker:
>>>>>>>>    http://github.com/cournape/bento/issues
>>>>>>>>
>>>>>>>> In my bento.info, I have this:
>>>>>>>>
>>>>>>>>    CompiledLibrary: cstat
>>>>>>>>        Sources:
>>>>>>>>            libcstat/fff/*.c,
>>>>>>>>            libcstat/wrapper/*.c
>>>>>>>>        IncludeDirs:
>>>>>>>>            libcstat/fff
>>>>>>>>
>>>>>>>> If I remove this section, the build proceeds, but I get an expected
>>>>>>>> crash when the build is trying to link against the cstat library.
>>>>>>>>
>>>>>>>> I couldn't find any examples around the place of an IncludeDirs ; have
>>>>>>>> I got the syntax right?
>>>>>>>
>>>>>>> Yes, it is a bug/missing feature in bento.
>>>>>>>
>>>>>>> I have added code to deal with the package_dir feature of distutils,
>>>>>>> but did not implement it for include headers (hence the exception). In
>>>>>>> your case, the current code could actually work, so this should be
>>>>>>> easy to fix.
>>>>>>
>>>>>> Anything I can do to help?
>>>>>
>>>>> Or - more specifically - can you point me to the places I could look
>>>>> to fix this myself?
>>>>
>>>> It would be hard do fix correctly at the moment, as the code in
>>>> question is a bit complicated (related to some mostly undocumented
>>>> design related to how build/source trees behave).
>>>>
>>>> I made a hack and pushed the necessary changes to github
>>>> (7236bbc9815b21c012c156efb062c2ccefb837a2 and
>>>> 761f9c37f96417090eccce03119ff2ad8a6e1e49), and you need to add the
>>>> following in your bscript:
>>>>
>>>> diff --git a/bscript b/bscript
>>>> index 5f016c0..201ddb8 100644
>>>> --- a/bscript
>>>> +++ b/bscript
>>>> @@ -12,5 +12,6 @@ def post_configure(context):
>>>>      conf.env.INCLUDES = get_numpy_include_dirs()
>>>>      for p in ["wrapper", "fff"]:
>>>>          conf.env.INCLUDES.append(op.join("libcstat", p))
>>>> +        conf.env.CYTHON_INCLUDES.append(op.join("libcstat", p))
>>>>      for p in ["segmentation", "registration"]:
>>>>          conf.env.INCLUDES.append(op.join("nipy", "algorithms", p))
>>>>
>>>> The hack is that IncludeDirs is basically ignored at that point (this
>>>> is issue 115)
>>>
>>> Thanks a lot for this.  The package builds, but the install hit
>>> run-time link errors of type:
>>>
>>> ======================================================================
>>> ERROR: Failure: ImportError
>>> 
(dlopen(/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so,
>>> 2): Symbol not found: _fffpy_import_array
>>>   Referenced from:
>>> /Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so
>>>   Expected in: dynamic lookup
>>> )
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/loader.py",
>>> line 390, in loadTestsFromName
>>>     addr.filename, addr.module)
>>>   File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py",
>>> line 39, in importFromPath
>>>     return self.importFromDir(dir_path, fqname)
>>>   File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py",
>>> line 86, in importFromDir
>>>     mod = load_module(part_fqname, fh, filename, desc)
>>>   File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/modalities/fmri/tests/test_glm.py",
>>> line 10, in <module>
>>>     from ..glm import GeneralLinearModel, data_scaling
>>>   File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/modalities/fmri/glm.py",
>>> line 30, in <module>
>>>     from nipy.labs.utils import mahalanobis
>>>   File 
"/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/__init__.py",
>>> line 3, in <module>
>>>     from .routines import (quantile, median, mahalanobis, svd, permutations,
>>> ImportError: 
dlopen(/Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so,
>>> 2): Symbol not found: _fffpy_import_array
>>>   Referenced from:
>>> /Users/mb312/tmp/junk/lib/python2.7/site-packages/nipy/labs/utils/routines.so
>>>   Expected in: dynamic lookup
>>>
>>> I see from the docs:
>>>
>>> http://bento.readthedocs.org/en/latest/tutorial.html#adding-compiled-libraries
>>>
>>> that I need to use the hook mechanism to link (in this case) routines
>>> to the cstat library.
>>>
>>> I feel bad to confess, but I could not see how to do this.
>>
>> Oh, you should feel very bad
>>
>>>  Can you
>>> give any pointers?
>>
>> Short answer:
>>
>> @pre_build
>> def pre_build(context):
>>     context.tweak_extension("my_extension", use="foo") # link against libfoo
>>
>> Long answer:
>>
>> Look at the scipy bscript. This magic is coming from waf itself. It is
>> still a big hackish, but the function context.tweak_extension allow
>> you to override any parameter given to the underlying waf task
>> generator, that is tweak_extension(name, **kw) is equivalent to
>> calling bld(**kw) in waf (see
>> http://docs.waf.googlecode.com/git/apidocs_16/tutorial.html#a-c-and-c-project)
>
> Thanks for that.  I feel myself staggering out of the fog, but it
> might be into more fog.
>
> The next problem I hit was the need to link against blas, lapack etc.
> You probably remember that we had to ship a lapack_lite for the case
> where we could not discover a platform lapack library.
>
> I think this means pulling in the bscript machinery from numpy and
> scipy - by copy paste I guess.   Do you have any thoughts about
> reusing bscript stuff across packages?

bscript are normal python modules, so at that point, this is just a
matter of refactoring stuff in a reusable library.

David