Rewrite C# codegen to the new Lang infra + fixes

[RReverser]

Description of Changes

Rewrites C# codegen to the V9 infra and fixes some issues & moves common cross-language utils in the process.

Fixed issues:

• Fixes Rewrite C# client codegen to operate off of a ModuleDef #1653.
• Fixes Migrate C# client code gen off of V8 to V9 #1731.
• Fixes BUG: C# codegen erroneously generates two types instead of one type for two tables that share the same type #2106.
• Fixes Generate SpacetimeDBClient.cs in deterministic order #2152.
• Partial fix for Index API for C# sdk #1846.

Detailed changelog:

• Rewrites C# codegen to the V9 module definition and removes the legacy GenCtx infrastructure.
• Splits C# output into individual Types/..., Tables/..., Reducers/... files and uses partial classes to split out almost all entity-specific code into those files instead of the global SpacetimeDBClient.cs. This should help reduce diffs significantly.
• Moves iter_reducers, iter_tables, iter_types into the shared generate/utils.rs module. Their functionality was duplicated in each codegen otherwise, and (re)using those helpers ensures that all codegens iterate over entities in the same deterministic order.
• Updates iter_reducers helper to exclude the "init" reducer as no codegen should ever generate client-side code for those.
• Updates deletion of auto-generated files to walk the output folder recursively. Codegens could emit files in nested folders already, but after the change above this is utilised a lot more extensively, and top-level iteration would keep a lot of obsolete files.
• Optimises said auto-generated file detection by only reading the prefix of each file, instead of the entire contents.
• Adds is_reducer_invokable helper that tells whether an invocation should be generated for this reducer to ensure that we skip lifecycle reducers (currently used in C# and TypeScript, left Rust TODO for someone else as it's a more involved refactoring).
• Renamed on_connect in rust-wasm-test to client_connected as the former name should be illegal and it's a bug that it's currently accepted by Rust. (Rust reducers must disallow reducers starting with reserved prefix #2175)
• Adds format_files and possible_values() -> clap::builder::PossibleValues to the Lang trait, so that generate/mod.rs doesn't contain any language-specific functionality.
• Adds dotnet format whitespace-based implementation for the C# spacetime generate output.
• Removes InternalInvokeValueInserted and InternalInvokeValueDeleted methods that were "internal" in name only, but actually exposed to the users in generated code. Instead, now using truly private table events for index updates.
• Moves common unique index and btree index code into base classes in C# SDK to reduce amount of generated code.
• Removes <auto-generated /> tag from generated C# code, instead using .g.cs extension which is equivalent from C# compiler point of view, but should be a clearer signal to the users and makes it easier to e.g. ignore those files in Github diffs.
• Updates regen-csharp-moduledef and regenerates server-side bindings.
• Adds partial support for multi-column indices to C# (supports direct comparisons, but not BTree ranges).
• Move namespace into the Csharp implementation. We already rejected it for other languages during parsing, so it only makes sense to remove it from the Lang trait and move into Csharp struct only. This should make generate/mod.rs even more language-agnostic.

Note that this PR will require C# SDK changes as well, companion PR here: clockworklabs/com.clockworklabs.spacetimedbsdk#220.

API and ABI breaking changes

If this is an API or ABI breaking change, please apply the
corresponding GitHub label.

In this PR, I'm no longer preserving the support for #[sats(name = "Foo.Bar")] - see #2174. This was a syntax that we preserved essentially only for BitCraft and only in C# codegen, but BitCraft team confirmed they can migrate away from it easily, and having consistent type names - regardless of whether it's a product or sum type, and regardless of whether it's C# or another language output - is more valuable for 1.0.

You can see the result in diffs, but to recap, before:

#[derive(SpacetimeType)]
#[sats(name = "Namespace.TestC")]
pub enum TestC {
    Foo,
    Bar,
}

would generate

in Rust:

#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
#[sats(crate = __lib)]
pub enum NamespaceTestC {
    Foo,

    Bar,

}

in TypeScript:

export namespace NamespaceTestC {
  // These are the generated variant types for each variant of the tagged union.
  // One type is generated per variant and will be used in the `value` field of
  // the tagged union.
  export type Foo = { tag: "Foo" };
  export type Bar = { tag: "Bar" };
  ...
}

but in C#:

namespace SpacetimeDB
{
	public partial class Namespace
	{
		public partial class Types
		{
			[SpacetimeDB.Type]
			public enum TestC
			{
				Foo,
				Bar,
			}
		}
	}
}

After this change, it generates the same name as any other type in any other language:

namespace SpacetimeDB
{
    [SpacetimeDB.Type]
    public enum NamespaceTestC
    {
        Foo,
        Bar,
    }
}

Expected complexity level and risk

How complicated do you think these changes are? Grade on a scale from 1 to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex change.

This complexity rating applies not only to the complexity apparent in the diff,
but also to its interactions with existing and future code.

If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning ways.

3 - some diffs are large, although a lot of changes are relatively trivial, e.g. moving functions to other modules or splitting flat generated code into Lang methods. Still, some extra care might be good, especially with testing the C# parts and V8 -> V9 upgrade has a lot of subtle differences.

Testing

Describe any testing you've done, and any testing you'd like your reviewers to do,
so that you're confident that all the changes work as expected!

• I've been running C# SDK tests with both .NET test and Unity tests, with files regenerated from this implementation.
• Write a test you want a reviewer to do here, so they can check it off when they're satisfied. Would be good to do extensive BitCraft testing as well.

[bfops]

I'm signing off on my one codeowned file crates/cli/src/subcommands/generate/mod.rs. The changes are in the part(s) of the file that are codegen-related more than CLI-related, so they weren't really meant to be covered by my codeownership (but the CLI-related parts of this file were, which aren't changed in this PR).

I didn't review the rest of the PR, someone else should review + be the real approver.

[kazimuth]

Way cleaner overall. Very happy with this.

[rekhoff]

I ran into errors after testing BitCraft with the new SDK from this branch.

Testing steps:

1. Setup:
   Using the latest from SpacetimeDB repo's Master branch, with freshly built:
   spacetimedb-standalone
   spacetimedb-cli
   spacetimedb-update

2. Followed getting started guide until "Generate a World section":
   https://github.com/clockworklabs/BitCraftClient/wiki/Getting-Started
   Note, to publish the server, the following was used:
   spacetimedb-cli publish -c -y bitcraft

3. Once in Unity, to setup the world:

• From BitCraft menu, select SpacetimeDB Manager, set World Defintion to Spacetime20x20
• Under "Upload everything" click Worldgen World
• If asked, "Cache Everything"
• Once the Queued Messages reaches 0:
• In the Scene, _Project > Scenes > Main you can hit Play
• Play the game and perform your tests

Tip: To skip tutorial, Use Shift+O
Tip: Once in the world, you'll need the Ruined Town to start. Spawn it while in game by going to:
Bitcraft > I'm not cheating, you're cheating > Place Ruined Town

Tests to perform (initially using the up-to-date repo SDK):

1. Publish the bitcraft module
2. Use the client to:

• login
• walk around
• do the tutorial
• build buildings
• mine some resources
• mess around with inventory
• send a chat message
  Result: Success, game loads and all actions can be completed

3. Remove the old SpacetimeDB SDK from Package Manager, add the new "ingvar/csharp-new-codegen" repo SDK, and check console for errors:
   The location of the local SDK package from the cloned sdk repo is:
   com.clockworklabs.spacetimedbsdk\package.json

Result:

• 976 errors of type "CS0246: The type or namespace name 'IDatabaseRow' could not be found"
• 244 errors of type "CS0534: 'RemoteTables.PlayerDeathTimerHandle' does not implement inherited abstract member 'RemoteTableHandle<EventContext, PlayerDeathTimer>.Name.get'"
  Note: "PlayerDeathTimerHandle" and "PlayerDeathTimer" varried for each error.
• 74 errors of type "CS0311: The type 'BitCraft.Spacetime.BuffDesc' cannot be used as type parameter 'T' in the generic type or method 'StaticDataScriptableObject'. There is no implicit reference conversion from 'BitCraft.Spacetime.BuffDesc' to 'IDatabaseRow'."
  Note: "BuffDesc" varried for each error.
• 1 error of type "CS0314: The type 'T' cannot be used as type parameter 'T' in the generic type or method 'MultiStaticDataScriptableObject'."
• 1 error of type "CS0308: The non-generic type 'RemoteBase' cannot be used with type arguments"
• 1 error of type "CS0305: Using the generic type 'DbConnectionBase<DbConnection, Tables, Reducer>' requires 3 type arguments"

4. Regenerate the server defintions with
   spacetimedb-cli generate --lang csharp --out-dir .\BitCraftClient\Assets_Project\autogen --project-path .\BitCraft\packages\game --yes
   Result:
   More errors. All errors from SpacetimeDBClient.cs went away, but other errors took their place.

[RReverser]

Using the latest from SpacetimeDB repo's Master branch, with freshly built:

You need to use SpacetimeDB built from this branch, not from master, as the codegen has changed.

4. Regenerate the server defintions with

You also definitely need to do this first. Testing newer SDK with older codegen is not possible and won't result in meaningful diagnostics, so those errors above can be ignored.

[rekhoff]

Re-tested above steps using branches:

https://github.com/clockworklabs/com.clockworklabs.spacetimedbsdk/tree/ingvar/csharp-codegen-rewrite @ 5b21810
https://github.com/clockworklabs/SpacetimeDB/tree/ingvar/csharp-new-codegen @ 9b3b5d1
https://github.com/clockworklabs/BitCraft @ 4afb21b
https://github.com/clockworklabs/BitCraftClient @ 435ce33

Unity Error logs:
unity_error_log_sdk-5b21810_sdb-9b3b5d1.txt

After regeneration, Unity still reports 999+ errors. Reviewing them:

• 534 of error CS0426
  Using example:
  Assets\_Project\Scripts\GameSystems\Audio\Parameters\Providers\BiomeMusic.cs(4,45): error CS0426: The type name 'Types' does not exist in the type 'TerrainCell'
  Later in the same file, Biome is unable to be resolved.

In BiomeMusic.cs at line 4, reads using static BitCraft.Spacetime.TerrainCell.Types;
Previously this "Types" was defined in the autogen file TerrainCellBiomes.cs
The same file is now generated:

// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.

#nullable enable

using System;

namespace SpacetimeDB.Types
{
    [SpacetimeDB.Type]
    public enum TerrainCellBiome
    {
        Dev,
        CalmForest,
        PineWoods,
        SnowyPeaks,
        BreezyPlains,
        AutumnForest,
        Tundra,
        Desert,
        Swamp,
        Canyon,
        Ocean,
        SafeMeadows,
        Cave,
    }
}

This can be resolved by removing the using static BitCraft.Spacetime.TerrainCell.Types; line and replacing Biome with TerrainCellBiome
I am still evaluating the other errors.

[RReverser]

Yeah all of those are generated from #[sats(...)] and will need search-replace in BitCraft code, see this section of the description:

In this PR, I'm no longer preserving the support for #[sats(name = "Foo.Bar")] - see #2174. This was a syntax that we preserved essentially only for BitCraft and only in C# codegen, but BitCraft team confirmed they can migrate away from it easily, and having consistent type names - regardless of whether it's a product or sum type, and regardless of whether it's C# or another language output - is more valuable for 1.0.

[rekhoff]

Thanks @RReverser. It also appears that the 1685 reports of error CS0246 from namespaces changing in types from:
namespace BitCraft.Spacetime to namespace SpacetimeDB.Types
Because BitCraft relies on custom additions to the partial classes generated by SpacetimeDB, this is creating ambiguity for resolving terms. The real issue though, is that what used to be a single class, is now split into two namespaces.
Currently generated HealthState.cs in BitCraftClient uses the name BitCraft.Spacetime namespace
Non-generated HealthState.cs in BitCraftClient uses the name BitCraft.Spacetime namespace
Newly generated HealthState.g.cs uses SpacetimeDB.Types namespace reads:

// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.

#nullable enable

using System;
using System.Collections.Generic;
using System.Runtime.Serialization;

namespace SpacetimeDB.Types
{
    [SpacetimeDB.Type]
    [DataContract]
    public sealed partial class HealthState
    {
        [DataMember(Name = "entity_id")]
        public ulong EntityId;
        [DataMember(Name = "last_health_decrease_timestamp")]
        public ulong LastHealthDecreaseTimestamp;
        [DataMember(Name = "health")]
        public float Health;
        [DataMember(Name = "died_timestamp")]
        public int DiedTimestamp;

        public HealthState(
            ulong EntityId,
            ulong LastHealthDecreaseTimestamp,
            float Health,
            int DiedTimestamp
        )
        {
            this.EntityId = EntityId;
            this.LastHealthDecreaseTimestamp = LastHealthDecreaseTimestamp;
            this.Health = Health;
            this.DiedTimestamp = DiedTimestamp;
        }

        public HealthState()
        {
        }
    }
}

It may be that the BitCraftClient-specific partial classes in the BitCraftClient would also need to have there namespaces updated from BitCraft.Spacetime to SpacetimeDB.Types. This might make the most sense, as these BitCraftClient-specific partial class definitions typically contain no using statements, and where likely meant to just add additional functionality to classes generated by SpacetimeDB.

[rekhoff]

Looks like there was an issue during generation. Looks like generation should have included the --namespace=BitCraft.Spacetime. There are still a lot of issues, but at least the namespace one should no longer be a factor.

[RReverser]

Newly generated HealthState.g.cs uses SpacetimeDB.Types namespace reads:

You need to use --namespace in spacetime generate to customize it to the one BitCraft uses. You might want to talk to either @kazimuth or someone from the BitCraft team to help guide you further :)

[rekhoff]

Regenerating using namespace=BitCraft.Spacetime resolve all but around 125 issues, attached:
unity_error_log2_sdk-5b21810_sdb-9b3b5d1.txt

Working through the issues as described:
Clearing error CS0426 (count 11):
BiomeMusic.cs,
GroundMaterial.cs,
Remove using static BitCraft.Spacetime.TerrainCell.Types;
Biome should be TerrainCellBiome

MovementSpeed.cs,
TerrainChunk.cs,
Helpers\MovementSpeed.cs,
Remove using static BitCraft.Spacetime.TerrainCell.Types;
SurfaceType should be TerrainCellSurfaceType

WorldUploader.cs:
Remove using static BitCraft.Spacetime.TerrainCell.Types;
SurfaceType should be TerrainCellSurfaceType
Biome should be TerrainCellBiome

DeployableValidator.cs:
Remove using static BitCraft.Spacetime.DeployableDesc.Types;
Add using BitCraft.Spacetime;
DeployableType should be DeployableDescDeployableType

BuildingSpawnToolManager.cs:
Remove using static BitCraft.Spacetime.BuildingSpawnDesc.Types;
BuildingSpawnType should be BuildingSpawnDescBuildingSpawnType

ClaimDescriptionState.cs
Types.ClaimPermission should be ClaimDescriptionStateClaimPermission

Clearing error CS0246 (count 34, unresolved 3):

AlertDesc.cs, Types.AlertType should be AlertDescAlertType
Enemy.cs, Types.EnemyType should be EnemyDescEnemyType
Npc.cs, Types.NpcType should be NpcDescNpcType
ProgressiveActionState.cs, Types.ProgressiveActionStatus should be ProgressiveActionStateProgressiveActionStatus
EquipmentDesc.cs, Types.EquipmentSlotType should be EquipmentDescEquipmentSlotType
FootprintDelta.cs, Types.FootprintType should be FootprintTileFootprintType
PlayerActionState.cs, Types.PlayerActionLayer should be PlayerActionStatePlayerActionLayer
ItemConversionRecipeDesc.cs, Types.ItemConversionLocationContext should be ItemConversionRecipeDescItemConversionLocationContext
CharacterStatDesc.cs, Types.CharacterStatType should be CharacterStatDescCharacterStatType
CollectibleDesc.cs, Types.CollectibleType should be CollectibleDescCollectibleType

Still reviewing the 74 cases of CS0311.

[rekhoff]

CS0311 issues where all cleared by doing a find-and-replace where:
Find:
IDatabaseRow
Replace with:
IStructuralReadWrite

Only 1 issue remains unresolved, but requires a code-change to both SpacetimeDB and the spacetimedbsdk, where a field on a table named Name conflicts with the remote table's name field which is also called Name. Since the word Name is such a common term for a user to want in a table, renaming the remote table name from Name to RemoteTableName should resolve this issue.

[RReverser]

where a field on a table named Name conflicts with the remote table's name field which is also called Name

Hm how does this conflict? We generate this Name on the remote table handle class, whereas data fields live on entirely separate data class.

UPD: nvm I guess it's autogenerated indexes that conflict? Good catch. Too bad abstract classes don't allow explicit interface implementation syntax (which is how we avoid conflicts in other places)...

[RReverser]

I'd probably recommend renaming it to something like remoteTableName to be even safer - since all our generated names are automatically converted to PascalCase, avoiding that convention would guarantee absence of conflicts, whereas with RemoteTableName a small risk of a clash remains.

[rekhoff]

Created a BitCraftClient branch containing changes so far:
https://github.com/clockworklabs/BitCraftClient/tree/rekhoff%2Fspacetimedb-codegen-changes

After fixing first batch of error, once Unity recompiled, several additional errors where returned. Most have been worked through, but those that remain fall into 3 categories:

4 x CS1503: Cannot convert from 'string[]' to 'string' which involve updating to the new Subscription API.
3 x CS0246: Can not find 'ClientCache' which appears to no longer be in the SDK src folder of of the codegen branch, but is still in the master branch.
2 x CS1061: Can not find 'InternalInvokeValue...' which where made private and are no longer accessible.

[kazimuth]

https://github.com/clockworklabs/SpacetimeDBPrivate/pull/1320