ML.NET 5.0-preview.1

New Features

• Introducing SentencePiece Unigram Tokenizer Model (#7390)
• Phi-4 Tokenizer Support (#7396)

Enhancements

• Support O3 OpenAI model mapping (#7394)
• Create SentencePieceTokenizer from options object (#7403)

Bug Fixes

• Fix DateTime export to csv with culture info (#7358) - Thanks @asmirnov82!
• Moved SpecialTokens assignment after the modification to avoid "Collection Modified" error (#7328) - Thanks @shaltielshmid!
• Fix broken inheritance from DataFrameColumn class (#7324) - Thanks @asmirnov82!

Build / Test updates

• [main] Update dependencies from dotnet/arcade (#7387)
• Update Microsoft.Extensions.AI to 9.3.0-preview.1.25114.11 (#7388)
• Update MSTest to latest (#7349)
• [main] Update dependencies from dotnet/arcade (#7352)
• Consolidate System.Numerics.Tensors dependency (#7356) - Thanks @asmirnov82!
• [main] Update dependencies from dotnet/arcade (#7382)
• [main] Update dependencies from dotnet/arcade (#7376)
• [main] Update dependencies from dotnet/arcade (#7374)
• Net8 tests (#7319)
• [main] Update dependencies from dotnet/arcade (#7266)
• Update DependencyModel (#7338)
• Update M.E.AI version used by Microsoft.ML.GenAI.Core (#7329)
• Update System.Numerics.Tensors version (#7322) - Thanks @asmirnov82!

Documentation Updates

• Fix up docs for MLContext (#7334)
• Some tweaks to the Microsoft.ML.Tokenizers PACKAGE.md (#7360)
• [GenAI] SFT Example (#7316)
• Update dynamic loading report reference (#7321) - Thanks @emmanuel-ferdman!