Posts Tagged ‘Python’

February 2018 Development Update

Tuesday, March 13th, 2018 by rdb

We again bring you some of the highlights of the new developments in the past month. This is however not an exhaustive list of changes, which can be obtained from the commit logs in the GitHub repository.

Android developments

We’re happy to announce that the Android port has made great strides forward in the past weeks. Most importantly, there is now a complete Python interpreter added to the Panda3D app, so that Python applications can be run directly from the Android device. Many of the sample programs now run as well as they do on the desktop.

It is now even possible to compile Panda3D entirely on an Android device, using an app that provides a Linux-like terminal environment. One free and open-source example is termux, which provides a package manager that gives access to an impressive suite of software, including the compilers needed to compile Panda3D.

Furthermore, using a bash script that we provide, it is also possible to launch Panda3D from termux and pipe the output from the interpreter app back to the terminal, so that you can develop Panda apps on Android the same way as you would on the desktop.

Eventually, we intend to make it possible to use the new deploy-ng system to produce an Android package from any operating system, making it easy to deploy your application to both desktop and mobile platforms.

In a separate effort, a commercial port of Panda3D to Android has been released on the app store, called Cub3D. This is an impressively complete development environment for Panda3D apps, and can be used today to develop and run Panda apps on Android. (This software is developed by a third-party company that is not affiliated with the Panda3D team.)

Roaming Ralph running on an Android tablet

Roaming Ralph running on an Android tablet

Ball in Maze running on an Android watch with Cub3D

Ball in Maze running on an Android watch with Cub3D



CMake improvements

Behind the scenes, we have been working on adding support for the CMake build system on the cmake branch. CMake is a popular cross-platform build system that is rapidly becoming the de facto standard for open-source projects, and it is eventually intended to replace our own makepanda build script. This will make it easier to build Panda from source and to customize the build process. Last month, the CMake port saw some updates and improvements, bringing us closer to this goal.

Bullet thread safety

Users of the Bullet physics engine were previously encountering crashes when using Bullet in conjunction with the multithreaded render pipeline. This is caused by the fact that the Bullet library is not thread safe. To remedy this, we have added locking in the Panda3D wrapper for Bullet that protects the Bullet library from access by multiple threads, which makes it safe to use the Bullet integration in a multithreaded program.

A look behind the curtains

Monday, November 20th, 2017 by fireclaw

Much has happened in Panda3D development for the upcoming 1.10 version. To bring you up-to-date with the latest developments, we will summarize some of the new changes here. Also, to further keep you informed about new and upcoming features, we’ll start a regular blog post series highlighting new developments.

Aside from a lot of optimization changes to improve various parts of Panda’s performance, as well as numerous bugfixes to improve stability and compatibility, there were some larger changes as well.

Python support

The first thing we’d like to highlight is the ability for Python users to install Panda3D via the pip package manager. No more fiddling with platform dependent installers—it takes only a single command to install the right version of Panda3D for your platform and Python version:

pip install panda3d

As a bonus feature, this allows you to install Panda into a virtualenv environment, allowing you to try out the latest development version in isolation without fear of contaminating your existing setup.

Furthermore, Panda3D has been updated to be compatible with the latest Python 3 versions. This includes interoperability with the pathlib module and the Python 3.6 path protocol, as well as fixes for the upcoming Python 3.7.

The Shader Generator

If you are using the shader generator in your application, you may significantly benefit from upgrading to 1.10. It has been overhauled to address a major performance concern for applications with complex scenes containing a large amount of render states, which could cause lag due to an excessive amount of shaders being generated.

Some new features have been added as well, such as support for hardware skinning and multiple normal maps.

Text rendering updates

The text rendering subsystem has been improved significantly. Panda’s text assembler used to perform well mainly for smaller texts, whereas frequently updating large blocks of text could cause considerable lag. But the improved text assembler code is up to 75 times as fast, making assembling large swaths of text a non-issue.

A comparison with HarfBuzz disabled and enabled. Of note is the spacing between the A and V, the "fi" ligature. The Arabic text renders like a mess without HarfBuzz.

A comparison with HarfBuzz disabled and enabled. Of note is the spacing between the A and V, the “fi” ligature. The Arabic text doesn’t render correctly at all without HarfBuzz.

Furthermore, the HarfBuzz library can now be utilized to implement text shaping, which not only enables support for ligatures and correct kerning but also allows us to better support languages with more complex shaping requirements, such as Arabic. This includes support for right-to-left text rendering, with automatic language detection enabled by default. Although bidirectional text is not yet fully supported, you can explicitly switch or re-detect direction for specific text segments using embedded TextProperties.

If Panda3D has been compiled with HarfBuzz support, it can be enabled using the text-use-harfbuzz variable. Otherwise, more basic kerning support can be enabled using text-kerning true, although many fonts will only kern correctly with HarfBuzz enabled.

Media playback

Panda3D now directly supports the Opus audio codec, a high-quality open standard designed to efficiently encode both speech and other audio. This is implemented via the opusfile library, so that it doesn’t require pulling in the heavier and more restrictively licensed FFmpeg libraries.

The FFmpeg plug-in now also supports loading video files with an embedded alpha channel, such as is possible with WebM files encoded with the VP8 codec. However, FFmpeg offers both a preferred native implementation and a decoder based on libvpx. The default is the native implementation, so if you wish to play VP8 videos with alpha channel, you should set the ffmpeg-prefer-libvpx configuration variable to true, to force FFmpeg to use the libvpx implementation.

 

We’d also like to highlight ongoing work outside the main Panda3D development branch. These things have been developed for Panda3D and will be merged into the main branch when they have reached maturity. But until then, they can be checked out from their respective branches on GitHub.

Deployment

First off, significant progress has been made on a new deployment system thanks to invaluable contributions by the community. The project is tentatively named “deploy-ng” and intends to make it easier more reliable to package and distribute your finished application, and as such it stands to replace the current deployment system entirely.

This new deployment system builds upon the existing Python setuptools, adding an extra plug-in to easily package your Panda3D applications. It already is quite usable, but still needs some love and testing until it’s production ready.

Graphics back-ends

A significant amount of work has been done on the effort to support two new graphics back-ends. The first of these is the WebGL back-end, happening on the webgl-port branch. This allows us to run Panda3D applications in the browser without requiring the use of a browser plug-in. The bulk of the work on the renderer itself has already been done, but there remains work to be done to make it easier to package up a Panda application for the web. Check out the proof-of-concept demos or the online editor demo if you’re curious about the possibilities.

On the vulkan branch, a prototype renderer for the new Vulkan graphics API has materialized as well. Like OpenGL, Vulkan is a cross-platform open graphics standard developed by Khronos. Unlike OpenGL, however, Vulkan offers a more low-level interface to the underlying graphics hardware, enabling a reduction in driver overhead for CPU-bound applications. Before you get too excited though, it’s not yet capable of running much more than a few of the sample programs. There is a lot more work to be done before it will reach feature-parity with or performance benefits over the OpenGL renderer, and it is unlikely to be a priority for the next release.

glTF 2.0

Behind the curtains there also is work going on to support glTF 2.0. This is a new JSON-based model transmission format recently standardized by the Khronos Group, the consortium that is also responsible for OpenGL, and plug-ins are already available to export it from various content creation tools. Importantly, glTF 2.0 defines a modern standard for physically-based materials, and as such is considered a milestone in the development of a physically-based rendering pipeline in Panda3D.

Input devices

Gamepad support is something that many in the community have been asking about for a long time. The input framework is receiving a significant overhaul to allow us to support game controllers, while also laying the groundwork for exposing commercial virtual reality hardware using a straightforward API. This work is happening on the input-overhaul branch and will be merged into the master branch soon.

 

That’s all for now, but keep an eye open for upcoming blog posts with all new and interesting updates in the coming months. In the meantime we encourage you to try the latest version for yourself and let us know how it works for you.

 

Panda3D and Cython

Sunday, September 12th, 2010 by Craig

This is about how to speed up your Python Code, and has no direct impact on Panda3D’s performance. For most projects, the vast majority of the execution time is inside Panda3D’s C++ or in the GPU, so no matter what you do, fixing your Python will never help. For the other cases where you do need to speed up your Python code, Cython can help. This is mainly addressed to people who prefer programming in Python, but know at least a little about C. I will not discuss how to do optimizations within Python, though if this article is relevant to you, you really should look into it.

Cython is an interesting programming language. It uses an extended version of python’s syntax to allow things like statically typed variables, and direct calls into C++ libraries. Cython compiles this code to C++. The C++ then compiles as a python extension module that you can import and use just like a regular python module. There are several benefits to this, but in our context the main one is speed. Properly written Cython code can be as fast as C code, which in some particular cases can be even 1000 times faster than nearly identical python code. Generally you won’t see 1000x speed increases, but it can be quite a bit. This does cause the modules to only work on the platform they were compiled for, so you will need to compile alternate versions for different platforms.

By default, Cython compiles to C, but the new 0.13 version supports C++. This is more useful as you probably use at least one C++ library, Panda3D. I decided to try this out, and after stumbling on a few simple issues, I got it to work, and I don’t even know C++.

Before I get to the details, I’ll outline why you might want to use Cython, rather than porting performance bottlenecks to C++ by hand. The main benefit is in the process, as well as the required skill set. If you have a large base of Python code for a project, and you decide some of it needs to be much faster, you have a few options. The common approach seems to be to learn C++, port the code, and learn how to make it so you can interface to it from python. With Cython, you can just add a few type definitions on variables where you need the performance increase, and compile it which gives you a Python modules that works just like the one you had. If you need to speed up the code that interfaces with Panda3D, you can swap the Python API calls for C++ ones. Using Cython allows you to just put effort into speeding up the parts of code you need to work on, and to do so without having to change very much. This is vastly different from ditching all the code and reimplementing it another language. It also requires you to learn a pretty minimal amount of stuff. You also get to keep the niceness of the Python syntax which may Python coders have come to appreciate.

There are still major reasons to actually code in C++ when working with Panda, but as someone who does not do any coding in C++, I won’t talk about it much. If you want to directly extend or contribute to Panda3D, want to avoid redundantly specifying your imports from header files (Cython will require you to re-specify the parts of API you are using rather than just using the header files shipped with Panda), or you simply prefer C++, C++ may be a better option. I mainly see Cython as a convenient option when you end up needing to speed up parts of a Python code-base; however, it is practical to undertake large projects from the beginning in Cython.

Cython does have some downsides as well. It is still in rather early development. This means you will encounter bugs in its translators as well as the produced code. It also lacks support for a few Python features, such as most uses of generators. Generally I haven’t had much trouble with these issues, but your experience may differ.

Cython does offer an interesting side benefit as well. It allows you to optionally   statically type variables and thus can detect more errors at compile time than Python.

To get started, first you need an install of Cython 0.13 (or probably any newer version). If you have a Cython install you can check the version with the -V command. You can pick up the source from the Cython Site, and install it by running “python setup.py install” from the Cython source directory. You will also need to have a compiler. The Cython site should help you get everything setup if you need additional guidance.

Then you should try out a sample to make sure you have everything you need, and that it’s all working. There is a nice C++ sample for Cython on the Cython Wiki. (This worked for me on Mac, and on Windows using MinGW or MSVC as a compiler).

As for working with Panda3D, there are a few things I discovered:

  • There are significant performance gains to be had by just compiling your existing Python modules as Cython. With a little additional work adding static typed variables, you can have larger performance gains without even moving over to Panda’s C++ API (Which means you don’t need to worry about linking against Panda3D which can be an issue).
  • Panda3D already has python bindings with nice memory management, so I recommend instancing all the objects using the python API, and only switching to the C++ one as needed.
  • You can use the ‘this’ property on Panda3D’s Python objects to get a pointer to the underlying C++ object.
  • On mac, you need to make sure libpanda (and is some cases, possibly others as well) is loaded before importing your extension module if you use any of Panda3D’s libraries.
  • On Windows, you need to specify the libraries you need when compiling (in my case, just libpanda)
  • The C++ classes and Python classes tend to have the same name. To resolve this, you can use “from x import y as z” when importing the python ones, or you can just import panda3d.core, and use the full name of the classes (like panda3d.core.Geom). There may be a way to rename the C++ classes on import too.
  • If using the Panda3D C++ API on Windows, you will need to use the MSVC compiler. You can get Microsoft Visual Studio 2008 Express Edition for free which includes the needed compiler.

Using this technique I got a 10x performance increase on my code for updating the vertex positions in my Geom. It avoided having to create python objects for all of the vertexes and passing them through the Python API which translates them back to C++ objects. It was just a matter of moving over one call in the inner loop to the other API. This, however, was done in already optimized Cython code that was simply loading vertex positions stored in a block of memory into the Geom. Most use cases would likely see less of a benefit. Overall though, I gained a lot of performance both from the change over to Cython, and from the change over to the C++ API. These changes only required relatively small changes to the speed critical portions of my existing python code.

I made a rather minimal example of using a Panda3D C++ API call from Cython. Place the setup.py and the testc.pyx files in the same directory, and from the said directory, run setup.py with your Python install you use with Panda3D. If everything is properly configured, this should compile the example Cython module, testc.pyx, to a python extension module and run it. If it works, it will print out a few lines ending with “done”. It is likely you may need to tweak the paths in setup.py. If not on Mac or Windows, you will get an error indicating where you need to enter your compiler settings (mostly just the paths to Panda3D’s libraries).

I would like to thank Lisandro Dalcin from the Cython-Users mailing list who helped me get this working on Windows.