contributor_docs/webgl_mode_architecture.md
This document is intended for contributors and library makers who want to extend the WebGL codebase. If you are looking for help using WebGL mode in your sketches, consider reading the WebGL tutorials on the p5.js Tutorials page instead.
There are two renderers that p5.js can run in, 2D and WebGL mode. WebGL mode in p5.js allows the user to make use of the WebGL API built into the web browser for rendering high-performance 2D and 3D graphics. The key difference between 2D mode and WebGL mode is that the latter provides more direct access to the computer's GPU, allowing it to performantly render shapes in 3D or perform other graphics and image processing tasks.
We keep track of the progress of WebGL issues in a GitHub Project.
When evaluating a new feature, we consider whether it aligns with the goals of p5.js and WebGL mode:
The browser's 2D and WebGL canvas context APIs offer very different levels of abstraction, with WebGL being generally lower-level and 2D being higher-level. This motivates some fundamental design differences between p5.js's WebGL and 2D modes.
p5.Texture, p5.RenderBuffer, and p5.DataArray to keep implementations readable and maintainable.curveDetail() API lets users control how many line segments to use, as we cannot predict the best balance of quality and performance for every use case.Everything drawn by p5.js, both in 2D and WebGL, consists of fills and strokes. Sometimes, we only draw one or the other, but every shape must be ready to draw either component.
All shapes in WebGL are composed of triangles. When a user calls a function like circle(), beginShape(), or vertex(), the renderer must break the shape down into a series of points. The points are connected into lines, and the lines into triangles. For example, circle() uses trigonometry to figure out where to place points along a circle. curveVertex() and bezierVertex() create look-up tables to turn any Bézier curve into points.
To create fills, the outline of a shape needs to be filled in with triangles. Some drawing functions like beginShape(TRIANGLE_STRIP) already provide triangles for fills. If the shape being drawn is not already made using triangles, we use the library libtess to break it down: in p5.RendererGL.Immediate.js, we run polygon outlines through _processVertices(). After libtess turns them into triangles, they are in a format where a shader can draw them to the screen.
Despite their name, strokes also need to be filled in to support strokes of varying widths and styles. The lines along the outline of a shape need to expand out from their centers to form shapes with area. The expansion of strokes creates three types of shapes: joins, caps, and segments, illustrated below.
<!-- Generated via https://codepen.io/davepvm/pen/ZEVdppQ -->Where two line segments connect, we add a join.
We support switching join styles without having to recompute line triangles. To do this, we make each line join a quad, composed of two triangles. The quad extent bounds all possible join styles. We use a shader to only display pixels within the quad that are present in the selected join style. We illustrate how each style fits into the quad below.
We use a similar strategy for stroke caps, present at the disconnected ends of lines. Each cap is a quad that bounds round, square, and extended cap styles. The line shader determines which pixels it needs to draw within those bounds. Below, we illustrate how each style fits into the quad.
3D shapes can also have strokes, but stroke shapes are calculated in 2D. This means they can change based on the camera's perspective. We want to avoid as much recalculation as possible, so we store all the information about the line that is not camera-dependent:
To draw the line, we combine that information with camera intrinsics in a shader to produce the final line positions in screen space.
There are two modes that p5.js uses to draw shapes onto the screen: immediate mode and retained mode.
Immediate mode is optimized for shapes that change every frame. If you were drawing a curve that changes each frame, its shape data would be different every time you drew it. Because of this, immediate mode fits it best. It indicates to p5.js that it does not need to spend time storing the shape for reuse, and it saves graphics memory from being filled up with all the shape variations over time. The following functions use this mode:
vertex(), curveVertex(), bezierVertex(), and quadraticVertex(), called between beginShape() and endShape()rect() when using rounded cornersbezier()curve()line()image()Retained mode is optimized for shapes that you will need to keep redrawing and don’t change shape. Once a shape is made of triangles and has been sent to the GPU to draw, retained mode keeps it there. It can then be drawn again without having to spend time re-triangulating it or sending it to the GPU again. The saved shape data is kept in a p5.Geometry object. p5.Geometry stores triangle data and keeps track of its uploaded buffers on the GPU. Calling freeGeometry() clears the GPU data to make space. Drawing it again after that will re-upload the data. Many 3D shape drawing functions in p5.js, such as sphere() or cone(), use this internally.
You can use buildGeometry() to make a p5.Geometry out of immediate mode commands. You call it with a function that runs a series of any p5.js shape drawing functions. It runs the function, collects the shapes into a new p5.Geometry, and returns it. The p5.Geometry can then be drawn and redrawn efficiently in the future.
Every shape we draw uses a single shader for its fills and a single shader for its strokes. There are a few default shaders that one can pick from in p5.js. You can also write and use your own shader instead of the default ones.
The default shaders work with p5.js's lighting and materials system. The user can specify what lights are in the scene with a shape and how each object reacts to light, including color and shininess. This information is given to the shader for each object being drawn. Custom shaders can also access the same lighting and material information, allowing users and library makers to extend the default rendering behavior.
p5.js has a few shaders built in:
fill() or stroke().lights(), ambientLight(), directionalLight(), pointLight(), and spotLight(). Each adds a light to the lighting list. All added lights contribute to the shading of the shape. If you do not use lights, the shape will be drawn using the color shader, which only uses the fill color.normalMaterial().Users can set ambientLight(), directionalLight(), pointLight(), and spotLight(). Each adds a light to the lighting list. All added lights contribute to the shading of the shape. They all have a color, and some have a few other parameters, such as position or direction.
For all but ambientLight(), each light added contributes to the view-independent lighting of shapes and the view-dependent reflections. The reflection of each light, by default, matches the color of that light. If desired, one can change the color of all reflections by setting specularColor().
If you do not use lights, the shape will be drawn using the color shader, which only uses the fill color.
Each 3D object has a few material properties that can be set by the user:
fill(). If done before drawing the shape, it applies to the whole shape. If called before a vertex() between beginShape()/endShape(), it will apply only to that vertex. A texture, if present from a call to texture(), will override the fill. When there are lights, this color will be mixed with the diffuse component of the light. The diffuse component describes the bright and dark areas of the surface due to light being directly cast on it.specularMaterial(). If unspecified, the shape will have no reflections.shininess(), this is how sharp the specular reflections are.ambientMaterial(). If unspecified, it will be the same as the fill color.emissiveMaterial(), this adds a constant color to the lighting of the shape, as if it were producing its own light of that color.The lighting and material parameters get turned into shader attributes and uniforms. If you reference them in a custom shader, p5.js will supply them automatically.
While advanced shader writers can take advantage of these properties, it can be unclear for new users. In the <a href="#future-goals">Future Goals section</a>, we describe some plans for improving the API. We may want to improve it before publicly documenting and supporting it.
For all objects in all contexts, the following global uniforms are available:
uniform mat4 uModelViewMatrix: A matrix to convert object-space positions into camera-spaceuniform mat4 uProjectionMatrix: A matrix to convert camera-space positions into screen spaceuniform mat3 uNormalMatrix: A matrix to convert object-space normals into camera-spaceAdditionally, these per-vertex properties are available as attributes:
attribute vec3 aPosition: The position of the vertex in object spaceattribute vec3 aNormal: For fills, a direction pointing outward from the surfaceattribute vec2 aTexCoord: For fills, a coordinate between 0 and 1 in x and y referring to a location on a texture imageattribute vec3 aVertexColor: For fills, an optional per-vertex coloruniform bool uUseLighting: Whether or not lights have been providedIf uUseLighting has been set, further lighting information will be passed in:
uniform int uAmbientLightCount: How many ambient lights are present, up to a maximum of 5uniform vec3 uAmbientColor[5]: Ambient light colorsuniform int uDirectionalLightCount: How many directional lights are present, up to a maximum of 5uniform vec3 uLightingDirection[5]: Light directionsuniform vec3 uDirectionalDiffuseColors[5]: Light diffuse colorsuniform vec3 uDirectionalSpecularColors[5]: Light specular colorsuniform int uPointLightCount: How many point lights are present, up to a maximum of 5uniform vec3 uPointLightLocation[5]: Locations of point lightsuniform vec3 uPointLightDiffuseColors[5]: Diffuse colors of point lightsuniform vec3 uPointLightSpecularColors[5]: Specular colors of point lightsuniform int uSpotLightCount: How many spot lights are present, up to a maximum of 5uniform float uSpotLightAngle[5]: Spot light cone radiiuniform float uSpotLightConc[5]: Concentration (focus) of each spot lightuniform vec3 uSpotLightDiffuseColors[5]: Light diffuse colorsuniform vec3 uSpotLightSpecularColors[5]: Light specular colorsuniform vec3 uSpotLightLocation[5]: Spot light locationsuniform vec3 uSpotLightDirection[5]: Spot light directionsuniform vec4 uMaterialColor: shape fill coloruniform bool uUseVertexColor: Whether there are per-vertex colors that override the shape fill colorattribute vec4 aVertexColor: Per-vertex fill coloruniform bool isTexture: Whether a texture is specifieduniform sampler2D uSampler: A textureuniform vec4 uTint: A tint multiplier for the textureuniform bool uSpecular: Whether to show reflectionsuniform float uShininess: The shininess of the materialuniform vec4 uSpecularMatColor: The material color to blend with specular lightuniform bool uHasSetAmbient: Whether an ambient color has been set or if it should default to the fill coloruniform vec4 uAmbientMatColor: The material color to blend with ambient lightThe default line shader is automatically supplied with these global properties in uniforms:
uniform vec4 uViewport: A vector containing [minX, minY, maxX, maxY] of the screen rectangleuniform int uPerspective: A boolean specifying whether to apply perspective projection to line thicknessuniform int uStrokeJoin: An enum representing the join style. 0, 1, and 2 represent ROUND, MITER, BEVEL.uniform int uStrokeCap: An enum representing the cap style. 0, 1, and 2 represent ROUND, PROJECT, SQUARE.It also has the following per-vertex attributes:
attribute vec3 aTangentIn: The 3D direction at the start of the line segmentattribute vec3 aTangentOut: The 3D direction at the end of the line segmentattribute float aSide: An enum representing what part of the stroke shape the point is on. See the Strokes section for details.uniform float uPointSize: The radius of the point being drawn---
title: p5.js WebGL Classes
---
classDiagram
class Base["p5.Renderer"] {
}
class P2D["p5.Renderer2D"] {
}
class WebGL["p5.RendererGL"] {
}
class Geometry["p5.Geometry"] {
}
class Shader["p5.Shader"] {
}
class Texture["p5.Texture"] {
}
class Framebuffer["p5.Framebuffer"] {
}
Base <|-- P2D
Base <|-- WebGL
WebGL "*" o-- "*" Geometry
WebGL "1" *-- "*" Shader
WebGL "1" *-- "*" Texture
WebGL "1" *-- "*" Framebuffer
The entry point to most WebGL code is through p5.RendererGL. Top-level p5.js functions are passed to the current renderer. Both 2D and WebGL modes have renderer classes that conform to a common p5.Renderer interface. Immediate mode and retained mode functions are split up into p5.RendererGL.Immediate.js and p5.RendererGL.Retained.js.
Within the renderer, references to models are stored in the retainedMode.geometry map. Each value is an object storing the buffers of a p5.Geometry When calling model(yourGeometry) for the first time, the renderer adds an entry in the map. It then stores references to the geometry's GPU resources there. If you draw a p5.Geometry to the main canvas and also to a WebGL p5.Graphics, it will have entries in two renderers.
Each material is represented by a p5.Shader You set the current shader in the renderer via the shader(yourShader) function. This class handles compiling shader source code and setting shader uniforms.
When setting a shader uniform, if the uniform type is an image, then the renderer creates a p5.Texture for it. Each p5.Image, p5.Graphics, p5.MediaElement, or p5.Framebuffer asset will get one. It is what keeps track of the image data's representation on the GPU. Before using the asset in a shader, p5.js will send new data to the GPU if necessary. For images, this happens when a user has manually updated the pixels of an image. This happens every frame for assets with data that may have changed each frame, such as a video or a p5.Graphics.
Textures corresponding to p5.Framebuffer objects are unique. Framebuffers are like graphics: they represent surfaces that can be drawn to. Unlike p5.Graphics, p5.Framebuffers live entirely on the GPU. If one uses a p5.Graphics as a texture in a shader, the data needs to be transferred to and from the CPU. This can often be a performance bottleneck. In contrast, when drawing to a p5.Framebuffer, you draw directly to its GPU texture. Because of this, no extra data transfer is necessary. WebGL mode tries to use p5.Framebuffers over p5.Graphics where possible for this reason.
Currently, WebGL mode is functional for a variety of tasks, but many users and library makers want to extend it in new directions. We aim to create a set of building blocks from which users and library makers can craft extensions. A block can be considered "done" when it has an extensible API we can confidently commit to supporting. A major milestone for WebGL mode will be when we have a sufficient set of such blocks for an ecosystem of libraries. The main areas currently lacking in extension support are geometry and materials.
curveDetail() allows faster but lower-quality curves. Line rendering is one of the common performance bottlenecks in its present state, and it could benefit from having lower fidelity but higher performance options. Another method is to introduce new types of objects and rendering methods that are optimized for different usage patterns, like how endShape(shouldClose, count) now supports WebGL 2 instanced rendering for more efficient drawing of many shapes.