dev-docs/RFCs/proposals/glsl-accessor-rfc.md
This RFC is part of the binary support roadmap.
This RFC proposes supporting GLSL accessors (setting accessors to strings containing fragments of GLSL code), allowing layers to:
This RFC explores the idea of allowing modification of behavior of layers to be done on the GLSL level, not just the JS level.
Let's start with a couple of API examples to illustrate what we want to achieve:
Columnar binary data systems (like Apache arrow) will often store primitive values (e.g. a Float32) in separate columns, rather than as a vec2 or vec3. This means that even if we exposed these columns as attributes in GLSL, we cannot use them directly as the expected vec3 attribute for positions. However, if we could just define a snippet of GLSL code to override how the shader extracs a position from attributes (getPosition), we could still use these values rather than pre-processing them in JavaScript:
new CircleLayer({
// inject new attributes
glsl: {
attributes: {
longitude: 'float',
latitude: 'float'
},
},
// Define a custom glsl accessor (string valued accessor will be treated as GLSL)
getPosition: 'return vec3(longitude, latitude, 0.);',
// Supply values to the injected attributes
attributes: {
longitude: new Float32Array(...),
latitude: new Float32Array(...)
},
})
The CircleLayer vertex shader would be defined as follows:
// Shader module system would e.g. do a line-based replacement of getPosition
vec3 getPosition() { return instancePosition; }
main() {
const position = getPosition();
// Note: layer no longer assumes `position = instancePositions`, but calls overridable GLSL function
}
Let's say that we load binary data for a point cloud that only contains a single value per point (e.g. reflectance), but the PointCloudLayer only supports specifying an RGBA color per point. While we could certainly supply a JavaScript function to the layer's getColor accessor to have it build a complete RGBA color array attribute, this extra time and memory could be avoided:
PointCloudLayer({
// inject new attributes and uniforms
glsl: {
attributes: {
reflectance
},
uniforms: {
alpha
}
},
// Define a custom glsl accessor (string valued accessor will be treated as GLSL)
getColor: 'return vec4(reflectance, reflectance, reflectance, alpha);',
// Supply actual values to the injected attributes and uniforms
attributes: {
reflectance: new Float32Array(...)
},
uniforms: {
alpha: 0.5
}
})
THIS SECTION IS STILL BEING DEVELOPED, NOT READY FOR REVIEW
AttributeManager to skip the generation of the attribute related to that accessor.attributes prop would cause those attributes
Model)uniforms prop would cause that uniform:
Model)Notes:
Float32Array attribute like reflectance can just be passed in as a value above, but typed arrays would typically be wrapped in the usuak attribute descriptor object (Accessor) stating size, type etc.The vertex shaders of deck.gl layers that wanted to support glsl accessors would need to be (lightly) refactored to call functions to access attributes instead of accessing those directly. For instance the PointCloudLayer vertex shader would look as follows (this code includes )
#define SHADER_NAME point-cloud-layer-vertex-shader
varying vec4 vColor;
varying vec2 unitPosition;
// Default attributes and uniforms
attribute vec3 positions;
attribute vec3 instanceNormals;
attribute vec4 instanceColors;
attribute vec3 instancePositions;
attribute vec2 instancePositions64xyLow;
attribute vec3 instancePickingColors;
uniform float opacity;
uniform float radiusPixels;
// NEW
// BEGIN Default accessor implementations
float getRadius() { return instanceRadius; }
// vec4 getColor() { return instanceColors; }
// END default accessors
// NEW
// BEGIN Custom (injected) attributes, uniforms and accessors
in float reflectance;
uniform float alpha;
vec4 getColor() { return vec4(reflectance, reflectance, reflectance, alpha); }
// END Custom (injected) accessors
void main(void) {
// NEW
// Call GLSL accessors
vec3 position = getPosition();
vec2 position64xyLow = getPosition64();
float instanceNormal = getNormal();
float instanceRadius = getRadius();
vec4 color = getColor();
vec3 pickingColor = getPickingColor();
// position on the containing square in [-1, 1] space
unitPosition = position.xy;
// Find the center of the point and add the current vertex
vec4 position_worldspace;
gl_Position = project_position_to_clipspace(positions, positions64xyLow, vec3(0.), position_worldspace);
gl_Position += project_pixel_to_clipspace(positions.xy * radiusPixels);
// Apply lighting
float lightWeight = lighting_getLightWeight(position_worldspace.xyz, // the w component is always 1.0
project_normal(normals));
// Apply opacity to instance color, or return instance picking color
vColor = vec4(lightWeight * color.rgb, color.a * opacity) / 255.;
// Set color to be rendered to picking fbo (also used to check for selection highlight).
picking_setPickingColor(pickingColor);
}
A key idea in this RFC is to let the layer user easily redefine some well-defined functions in the layers GLSL code. A reasonable question is whether overloading accessors vs defining a separate set of overridable functions is better, or even if both should be supported.
"Overloading" the accessors have the big advantage that it ties this feature into a well documented existing configuration system.
An apparent downside of overloading accessors is that it forces a choice between JS generation (function) and GLSL override (string) although this could be handled by supporting an object descriptor, or using e.g. the glsl`...` string syntax.
A layer shader could potentially define additional overridable functions that it would call at the right moments. These would then have to be separately documented. It would be good to see examples of such functions, and understand why they would not reasonably be expressed as new accessors, and convince ourselves that such function could be added in a structured, maintainable, non-adhoc way, before adding support.
Since the proposed API is prop-based, the general solution for forwarding props in composite layers should work though not clear how practical it would be to use.
Define/reinforce naming conventions/prefixes for GLSL functions and variables in layer shader code to maximize predictability/minimize conflicts with user's GLSL accessor code.
It would seem possible to implement transitions for injected attributes. Perhaps some descriptor metadata is needed so that the user can specify if/how this should be done.
Being able to write accessors in GLSL is certainly powerful and cool but probably also a bit intimidating for many users.
In addition, there are multiple versions of GLSL and in the future when we port to WebGPU we will likely have one more shader syntax on our hands.
For these reasons it could be worthwhile to define a simple syntax/format that generates GLSL accessors.
This should likely be a separate RFC
Being able to specify props that affect the shader source code means that either:
Apart from the work/extra complexity required to accurately detect when shaders should be recompiled, the big issue is that recompilation and relinking of GLSL shaders tends to be slow (sometimes extremely slow, as in several seconds, or even half a minute or more) and happens synchronously on the main WebGL thread, freezing all rendering or even the entire app.
If we support dynamic recompilation of shaders based on changing props, we therefore need to be extra careful to avoid (and detect/warn for) "spurious" recompiles. The user might not realize that the supplied props are triggering constant recompiles causing app performance to crater.
For ideas around working around slow shader compilation, see:
This RFC as written does not offer any way for layer users to modify fragment shaders.
If it is desirable to offer the ability to redefine GLSL functions in the fragment shader, the GLSL accessor system outlined in this system does not extend well.
varyings that "eat into" a very limited bank (often only 8 varyings or vec4s).