home contents changes options help


Channel Architecture
by Robin Rowe and Stefan Klein

> An idea that was raised in earlier discussions is to > have the paint core in float32 only and implement other types as > import/export only. Robin pointed out that we would loose the advantages > of executing native Half or Fixed calculations on GPUs that CinePaint > will be supporting soon.

Besides that, OpenEXR has 32-bit unsigned as a channel type! To support that we would like to implement 32-bit unsigned. And, we shouldn't mangle 32-bit unsigned into 32-bit float.

> Considering the vast amounts of type-dependent functions I had to touch > to implement bfp (see my list), I find it not very desirable to expect > new type plug-ins to implement them all.

Right. Imagine a future set of channel plug-ins like this:

u8.dll u16.dll u32.dll bfp16.dll bfp32.dll ubfp16.dll ubfp32.dll cf16.dll hf16.dll f32.dll d64.dll

That's unsigned 8/16/32, signed binary fixed point 16/32, ditto unsigned, chopped float 16 (RnH), half float 16 (ILM), IEEE float 32, and IEEE double 64. (I've listed them with the Windows extension of dll because it is unambiguous, but plug-in extension would of course depend upon the platform.) Each plug-in implements the usual paint functions that we have in the core now, but almost no conversions between types. When a plug-in is loaded it populates an array of stuctures of per-type function pointers. Those are called instead of the hardwired functions we have in the core now for paint ops. Converter functions, however, go into the PDB.

To do a channel conversion from u8 to u16 the code looks for a function in the PDB named "u8_u16_line" (with variants perhaps per channel, pixel, line, raster). If it finds that converter, fine. If not, it should vault through u8_u32 then u32_u16, take the hit of doing two conversions. Every channel type plug-in will include its one best vaulting converter, which will convert to the most appropriate 32-bit type. That would be one converter (both to and from) required for each channel type plug-in. The other converter each type should provide is u8, to optimize screen display. Anything more would be optional.

> A possible problem I can see with that is loss of precision. Assume all > conversions would vault through float32 and consider the example of > converting from 32bit BFP to 32bit uint.

Right! Each channel type must vault through its most compatible 32-bit type, which makes it a little more complicated.


In this plug-in go the 32-bit channel bridge conversions to go between unsigned, bfp, ubfp, and float. Given the 32-bit bridge set and the vaulting converter included with each channel type plug-in it will be possible to convert anything to anything in the most lossless manner possible in a maximum of three steps (an up-conversion, a bridge, a down-conversion). Note the odd case that the "up-conversion" on d64 would be to f32. When conversion performance is an issue it will be an option to install more converter plug-ins with hand optimized direct conversions. Conversion should always check for a direct conversion before vaulting.

In answer to your earlier question about paletted types, it isn't that we don't care about paletted types at all -- rather that they are such low priority that nothing gets done on them. I imagine paletted and B&W types as additional channel type plug-ins, to modularize them to become less a maintenance chore.

Any questions? Do you see any design issues with this channel type plug-in approach?

At what point in your research should the channel plug-in architecture be implemented? Do it after display CMS so it doesn't delay you? Or, could it help you to have the channel implementation modularized sooner?


> Usual paint functions means all the functions I listed? Is there no way > around this? My major concern is that the list probably won't stay fixed > in the future. New type-dependent image manipulation functions may be > added which means that each time every single plugin would have to be > touched.

If new functions are added then there would be someplace in the core that calls them, right? That's going to take work no matter what. If new functions are not called by the core then they don't need to be exported from the plug-in.

Whether all those functions you listed need to be exported is a good question. Would all those get touched by the core or are some channel type helper functions that wouldn't need to be exported?

To be explicit, what I'm proposing is that each channel plug-in implement x_add_row -- that there not be a function named x_add_row_bfp. Across channel type plug-ins the same function would have the same name and same types. That creates a little problem because they of course aren't the same channel types. That channel data would pass through void* and be cast appropriately inside the function.

> I have little insight into what it would mean to implement BFP operations > natively on a GPU, but I suppose that would mean something like not using > C-operators, but implementing one's own addition/multiplication etc. > instead that gets translated to special GPU instructions?

Yes, something like that. Not on the drawing board yet. Still being researched.

> What if every type plug-in had functions to : > - generate a value: it is passed an integer or real constant and > generates the appropriate value returning a void pointer. E.g. void > value=create_bfp16_value(1.0), where *value=1000000000000000 in binary. > - to do all basic operations: add, subtract, multiply and whatever else > is required by the paint functions (most really only multiply, subtract > or add pixel values in a lot of different ways. > - create a string representation of the value > > All paint/manipulation functions would be implemented once only. They > call the type functions through pointers that point to the plug-in that > is appropriate for the current image. > > Would that be possible?

Maybe. I haven't looked at the paint functions that much and expect you to say what's possible. I don't expect us to get away with implementing ops at the single pixel level only, that the call overhead will be too high in loops without providing optimized row and raster ops.

> Are you suggesting I should implement this? Maybe my impression is wrong, > but it seems quite a big task and unfortunately, I don't think I will > have the time to do this during my project, I need to focus on other > things such as display CMS, colour correction... first in order to get > visible results - even though this will probably lead to leaving out some > aspects because the required types aren't implemented yet.

I was, but of course you can demur. No need to do it now. Don't want to overload you.

The task size by my estimate is about the same as what you did in implementing UBFP. It involves wrapping what you did there as a plug-in and placing hooks to it in the core. Not a big task, but not trivial either. If you want to leave it for someone else that's no problem. Appreciate that you took UBFP this far.

Channel Plugins

> Yes, but it only takes one implementation in the core and not one in each > plug-in plus the call from the core.

Functions that are bit-depth specific would go in channel plug-ins. Otherwise, the function should be implemented one time in the core. Code would need to be touched in the same number of places regardless. That it is a plug-in shouldn't change that. The only difference is that some of that code would reside in plug-ins instead of the core. The only extra effort involved in plug-ins should be a little logistics, that they have to load and build separately. If it requires a lot more code then we're not talking about the same idea.

> That's about the way it's done now, anyway. The type-dependent functions > take PixelRow structs and extract the data from them (which is of type > guchar*). This is then cast to the appropriate type.

Cool! What I'm suggesting is to move that type-sensitive code into individual plug-ins.

> I think, it might be interesting to wait what you come up with for GPU > native operations, how you get an addition to get translated to a native > BFP operation. As I said, I know little about these things, but I assume > it wouldn't be through function calls, otherwise you'd have the same > overhead here.

Expect the GPU stuff to be much harder. It could require drastic redesign that takes years.

We would like to have SBFP16 and UINT32 soon. Our design choice today is to continue hacking in new channel types in the core the same way you did UBFP16 or to streamline that process as plug-ins. To try to defer this decision is the same as choosing the hacking approach or delaying the implementation of new channel types indefinitely.


> The plug-in implementation doesn't create > more code, but it doesn't reduce it either.

Vaulting doesn't reduce code in the way you mean, but would in a very important way with regard to combinatoric type conversions. That is the more severe issue, because without vaulting that bloat will get geometrically worse.

You bring up an important point about implementing algorithm reuse across plug-ins. You are absolutely right that we should be concerned about that, too.

> ...there is a lot of duplicate code. There > are as many versions of a type-dependent functions as there are types, > but they usually only differ very slightly. If there is a mistake in one, > they will all have to be touched. Besides, cutting-and-pasting (which is > what I was doing a lot for implementing BFP) easily leads to mistakes or > to things being overlooked.

Your suggestion of using macros would be the standard approch if writing C code, however C++ templates are designed for this type of work.

> And I realised that it doesn't matter if there is > a lot of duplication in the machine code generated and we only have to > avoid it in the source code (for easier handling/preventing mistakes).

This is the beauty of how templates work (duplicating code similar to macros), and how they can easily get out of hand if you are not careful what you are doing. Each type implicitly lays down another set of machine code. Advantages of templates over using C macros include type-safety, syntax checking, and you don't need a slash to continue after each line (in a multi-line macro).

> Here's a simple example....

Here's your example reworked as a template. Done off the top of my head, not compiled.

---------------------------------------- // channel_math.h // a bunch of templates would go in here similar to this one...

template void x_add_row(T src,T dest,T maxval) { const gint num_channels = tag_num_channels (pixelrow_tag (dest)); gint width = pixelrow_width (dest); while (width--) { for (gint b = 0; b < num_channels; b++) { dest[b] = MIN (maxval, src[b] + dest[b]); } src += num_channels; dest += num_channels; } } ---------------------------------------- // u_short_channel.cpp // Template instantiation #include "channel_math.h"

// Explicit declaration to instantiate function template and // to templatize constant: void x_add_row(guint16 src,guint16 dest,(guint16)65535);

// That's all. ----------------------------------------

Just call void x_add_row(guint16* src,guint16* dest) as before. Not sure I'm illustrating the templatize constant trick right. That's something I haven't used in a while. For every other type that follows this same template you simply include the header with the template definition and declare the signature type you wish for. Note that these are function templates, not class templates.

It doesn't cut down the size of the machine code, but cuts down lines of code a lot!

One more trick, create a type-less instantiator header file like this: ---------------------------------------- // t_channel.h // DO NOT #include "channel_math.h"

void x_add_row(T src,T dest,(T)MAXVAL); // lots of other template instantiators that would need to be tediously // repeated per type T .... // .... ----------------------------------------

Then u_short_channel.cpp becomes this:

---------------------------------------- // u_short_channel.cpp // Template instantiation #include "channel_math.h"

#define MAXVAL 65335 typedef guint16 T; // T now explictly guint16! // All the type-less Ts in t_channel.h now legal. #include "t_channel.h"

// That's all folks. ----------------------------------------

Then, if built as channel-type plug-ins and the vaulting routines discussed earlier implicitly handle type conversions, you could define a complete new channel-type plug-in with just a few lines of code.


> However, what is the current situation with respect to the use of C and > C++ in CinePaint. I don't overlook that. Is a complete re-write in > progress or does that happen step by step and the two can easily be > mixed?

There are differing opinions as to how large the core code conversion to C++ task could be. It is not that C++ does anything bad, rather the reverse. It forces cleaning up dubious C code.

My experience from a previous project is that it is not that big a deal, but will be tedious and may take weeks to implement mincing changes. I have fixed many hundreds of CinePaint C compile warnings in preparation. Having already done weeks of work, we may be almost done.

With all the other tasks going on, especially the unanticipated GTK work, the last phase of the C++ switchover is on hold until my schedule eases up.

> Is to say, would it be possible to write the plug-ins in C++ any > time in the near future (given, someone can do it or I can find the > time)?

We're using C++ inside plug-ins already. The OpenEXR plug-in is very complex C++ using the STL. That it is so advanced with C++ is why we can't build that plug-in using VC++ 6 under Windows.


> I haven't used auxiliary channels or given much thought to channel > architecture features beyond alpha and CMYK. > > What do you propose?

My old photoshop (v5.5) actually behaves the same as CinePaint. The most recent GIMP behaves the same, too. So maybe it's just me, however the way of handling channels they use never seemed very intuitive or the best option available to me. I always thought it was a bit strange that spot colour channels kind of sit on top of layers. When I paint on a core channel (R, G or B in the current situation), it affects only the selected layer. However, painting on an auxiliary channel doesn't affect any layer at all. It's rendered on top of all other layers. I don't see why.

Layers for me are like slides for an overhead projector, stacked one on top of the other. If I want to paint on the image, my projector displays, I need to choose a colour (a mix of channel intensities) which I then use to paint onto one of the slides. I can't paint in the air above them. That's however what Photoshop seems to do. In my opinion, auxiliary channels should be contained in layers, too, just like core channels. If I paint on an auxiliary channel, it only affects the layer I've selected. I guess that would mean that internally channels should be contained in layers and not in images. Every layer should have its own set of channels (that would probably be the same channels for all layers of the image, but maybe even that could be made more flexible in the future, don't know whether there is a need for layers with varying numbers of channels).

> I guess that would mean that internally > channels should be contained in layers and not in images. Every layer > should have its own set of channels (that would probably be the same > channels for all layers of the image, but maybe even that could be made > more flexible in the future, don't know whether there is a need for > layers with varying numbers of channels).

To me it is logical that images contain layers contain channels. I don't see why channels should be outside layers either.

Perhaps selections should be implicit channels? I don't see why selections should be a special case. The selection/mask/channel transformations makes it complicated. Why not always channels?

If selections are channels then we could preserve geometry when doing geometric selections. That would make selections vector adjustable later, rather than a blob.

If we start adding spot channels to layers we will have varying numbers of channels per layer.

FYI, common uses for auxillary channels (from a discussion on the OpenPlugins list):

Z Coverage is how much of a pixel is covered, used to provide limited anti-aliasing on the front-most layer.

FYI, the OpenPlugins project is an effort to create a unified API for Photoshop, AE, and other packages. To subscribe contact Bruno Nicoletti bruno@thefoundry.co.uk.