Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 33

---

Analysis of SpacetimeDB Benchmark Test Failures

Rust / rustdoc_json Failures

Compile/Publish Errors (2 Failures)

Failure Group 1: t_002_scheduled_table and t_017_scheduled_columns

1. The generated code:

   rustCopy

   use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
   
   #[table(accessor = tick_timer, schedule(reducer = tick, column = scheduled_at))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       if ctx.db.tick_timer().count() == 0 {
           ctx.db.tick_timer().insert(TickTimer {
               scheduled_id: 0,
               scheduled_at: ScheduleAt::RepeatMicros(50_000),
           });
       }
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {
   }
   

2. The golden example:

   rustCopy

   use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
   use std::time::Duration;
   
   #[table(accessor = tick_timer, scheduled(tick))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       pub scheduled_id: u64,
       pub scheduled_at: ScheduleAt,
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       let every_50ms: ScheduleAt = Duration::from_millis(50).into();
       ctx.db.tick_timer().insert(TickTimer {
           scheduled_id: 0,
           scheduled_at: every_50ms,
       });
   }
   

3. The error:

   • publish_error: spacetime publish failed (exit=1)

4. Explain the difference:

   • The generated code used ScheduleAt::RepeatMicros(50_000) instead of the correct ScheduleAt::Interval(Duration::from_millis(50).into()). The way the scheduling was set up was incorrect.

5. Root cause:

   • The documentation does not clearly specify the constructor syntax for ScheduleAt nor how to correctly set up the scheduled tasks in this context.

6. Recommendation:

   • Update documentation to provide examples of different constructors for ScheduleAt, specifically emphasizing how to define intervals correctly.

---

Other Failures (5 failures)

Failure Group 2: t_013_spacetime_sum_type, t_015_product_type_columns, t_016_sum_type_columns, t_018_constraints, t_020_ecs

1. The generated code:

   rustCopy

   use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};
   
   #[derive(SpacetimeType)]
   pub struct Rect {
       width: i32,
       height: i32,
   }
   
   #[table(accessor = result)]
   pub struct ResultRow {
       #[primary_key]
       id: i32,
       value: Shape,
   }
   
   #[reducer]
   pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
       ctx.db.result().insert(ResultRow {
           id,
           value: Shape::Circle(radius),
       });
   }
   

2. The golden example:

   rustCopy

   use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};
   
   #[derive(SpacetimeType, Clone, Debug)]
   pub struct Rect {
       pub width: i32,
       pub height: i32,
   }
   
   #[table(accessor = result)]
   pub struct ResultRow {
       #[primary_key]
       pub id: i32,
       pub value: Shape,
   }
   
   #[reducer]
   pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
       ctx.db.result().insert(ResultRow { id, value: Shape::Circle(radius) });
   }
   

3. The error:

   • spacetime sql failed: no such table: result
   • spacetime sql failed: no such table: profile
   • spacetime sql failed: no such table: drawings

4. Explain the difference:

   • The generated code omits the pub visibility keyword for fields and structs, which prevents proper access by the macros that generate the expected database schema. Additionally, the enum Shape wasn't declared correctly in the generated code.

5. Root cause:

   • Lack of proper visibility (missing pub) for structs and enum fields was not clearly emphasized in the documentation, leading to access issues.

6. Recommendation:

   • Provide clear guidelines in the documentation regarding the necessity of using pub for struct and enum fields when working with SpacetimeDB components. Include example schemas with visibility marked.

---

Rust / docs Failures (22 total)

Timeout Issues (1 failure)

1. Failure Group: t_015_product_type_columns
   • Expected: Modify the query logic to ensure no unnecessary long-running operations exist.
   • Recommendation: Provide timeout considerations in the documentation to ensure optimization options are explored to prevent long-running tasks.

---

C# / docs Failures (4 total)

C# Failure Group: t_014_elementary_columns, t_016_sum_type_columns, t_017_scheduled_columns, t_020_ecs

1. The generated code:

   csharpCopy

   using SpacetimeDB;
   
   public static partial class Module
   {
       [SpacetimeDB.Table(Accessor = "Primitive", Public = true)]
       public partial struct Primitive
       {
           [SpacetimeDB.PrimaryKey]
           public int Id;
           public int Count;
           public long Total;
           public float Price;
           public double Ratio;
           public bool Active;
           public string Name;
       }
   
       [SpacetimeDB.Reducer]
       public static void Seed(ReducerContext ctx)
       {
           ctx.Db.Primitive.Insert(new Primitive
           {
               Id = 1,
               Count = 2,
               Total = 3000000000L,
               Price = 1.5f,
               Ratio = 2.25,
               Active = true,
               Name = "Alice"
           });
       }
   }
   

2. The golden example:

   csharpCopy

   using SpacetimeDB;
   
   public static partial class Module
   {
       [Table(Accessor = "Primitive")]
       public partial struct Primitive
       {
           [PrimaryKey] public int Id;
           public int Count;
           public long Total;
           public float Price;
           public double Ratio;
           public bool Active;
           public string Name;
       }
   
       [Reducer]
       public static void Seed(ReducerContext ctx)
       {
           ctx.Db.Primitive.Insert(new Primitive {
               Id = 1,
               Count = 2,
               Total = 3000000000,
               Price = 1.5f,
               Ratio = 2.25,
               Active = true,
               Name = "Alice"
           });
       }
   }
   

3. The error: no such table: primitive

4. Explain the difference:

   • Missing the public access modifier in the declaration of the Table attribute. The expected syntax properly utilizes attributes defined in the library.

5. Root cause:

   • Documentation may lack clarity about access modifiers, especially when it comes to how they affect visibility in entities.

6. Recommendation:

   • Ensure C# documentation includes explicit examples where public is required in class and struct declarations to prevent access issues with tables.

---

By addressing the above gaps in documentation and ensuring that generated samples adhere to the expected outcomes, we can significantly reduce the number of failures in future benchmarks.