Intermittent NullReferenceException in MyForageableEnvironmentProxy on DS

Intermittent crashes sometimes happen on a dedicated server related to `MyForageableEnvironmentProxy` with a `NullReferenceException`. Crashes appear to happen in two stages: first in the worker thread during entity creation, then in the main thread during entity finalization.

I don't have the full server log file at this point but this is the best analysis I can figure out below. I could be wrong on the true root cause, debugging races and intermittent issues is always hard. Take these as a suggestion of the possible causes, but know the crash happens pretty regularly.

The actual code issues are real at least, even if the root cause isn't.

Possible Reproduction Steps

1. Run a dedicated server with `ResetForageableItems = True` and low distance/refresh time limits.
2. Have players interact with forageable items on planets
3. When sectors are closed while parallel entity creation is in progress, crashes may occur

Possible Root Cause

The crash may be related to a race condition between parallel entity creation and sector lifecycle management - I can't find any other cause that would make sense. Nothing else looks like it could be null during creating of forageables.

Crash 1 - Worker Thread Stack Trace

System.NullReferenceException: Object reference not set to an instance of an object.
   at Sandbox.Game.WorldEnvironment.Modules.MyForageableEnvironmentProxy.CreateForageableEntitiesData.CreateForageableEntities()
   at Sandbox.Game.WorldEnvironment.Modules.MyForageableEnvironmentProxy.CreateForageableEntities()
   at ParallelTasks.WorkItem.DoWork()

Crash 2 - Main Thread Stack Trace

System.NullReferenceException: Object reference not set to an instance of an object.
   at Sandbox.Game.WorldEnvironment.Modules.MyForageableEnvironmentProxy.CreateForageableEntitiesData.FinishEntitiesCreation()
   at Sandbox.Game.WorldEnvironment.Modules.MyForageableEnvironmentProxy.FinishForageableEntities()
   at ParallelTasks.WorkItem.Execute()

Suspect Code

Sector closing doesn't wait for pending work. This specific piece of code makes me think it's something to do with the dataview/sector management (All players getting far enough away from a sector maybe and having it unload during forage respawn?)

In MyEnvironmentSector.CloseInternal():

if (DataView != null)
{
    DataView.Close();
    DataView = null;  // Set to null without waiting for parallel work
}

Below are the possible locations that could be null in the crashing methods, most of which point back to the Sector:

public void CreateForageableEntities()
{
    MyEntityIdentifier.InEntityCreationBlock = true;
    MyEntityIdentifier.LazyInitPerThreadStorage(2048);
    foreach (int itemId in m_itemIds)
    {
        ItemInfo itemInfo = m_sector.DataView.Items[itemId]; // NRE: DataView or Items can be null if sector closed
        Vector3D vector3D = m_sector.SectorCenter + itemInfo.Position;
        if (itemInfo.ModelIndex < 0)
        {
            if (m_sector.Owner.Entity.Components.TryGet<MyPlanetEnvironmentComponent>(out var component)) // NRE: Owner or Entity can be null
            {
                short id = (short)(~itemInfo.ModelIndex);
                Vector3 vector = MyModels.GetModelOnlyData(m_sector.Owner.GetModelForId(id).Model)?.BoundingBoxSizeHalf ?? Vector3.Half;
                BoundingBoxD worldAabb = new BoundingBoxD(vector3D - vector, vector3D + vector);
                component.NotifyTrySpawnDisabledForagedObject(itemId, m_sector.SectorId, worldAabb, vector3D);
            }
            continue;
        }
        MyPhysicalModelDefinition modelForId = m_sector.Owner.GetModelForId(itemInfo.ModelIndex); // NRE: Owner can be null
        if (modelForId != null)
        {
            MatrixD matrixD = MatrixD.CreateFromTransformScale(itemInfo.Rotation, vector3D, Vector3D.One);
            m_sector.DataView.GetLogicalSector(itemId, out var logicalItem, out var sector);      // NRE: DataView can be null
            string uniqueString = $"P({m_sector.Owner.Entity.Name})S({sector.Id})A({modelForId.Id.SubtypeName}__{logicalItem})"; // NRE: Entity can be null, Owner would have been caught already
            long entityId = MyEntityIdentifier.ConstructIdFromString(MyEntityIdentifier.ID_OBJECT_TYPE.PLANET_ENVIRONMENT_ITEM, uniqueString);
            MyForageableEntity myForageableEntity = new MyForageableEntity(m_sector, modelForId as MyForageableDefinition);
            myForageableEntity.WorldMatrix = matrixD;
            myForageableEntity.EntityId = entityId;
            MyObjectBuilder_ForageableEntity objectBuilder = new MyObjectBuilder_ForageableEntity
            {
                PhysicalItemDefinitionId = modelForId.Id,
                PositionAndOrientation = new MyPositionAndOrientation(matrixD),
                SectorId = m_sector.SectorId,
                PlanetId = m_sector.Parent.EntityId,
                ItemId = itemId
            };
            myForageableEntity.Init(objectBuilder);
        }
    }
    m_entities = new List<IMyEntity>();  // !!! Never reached if exception above, m_entities stays null !!!
    MyEntityIdentifier.GetPerThreadEntities(m_entities);
    MyEntityIdentifier.ClearPerThreadEntities();
    MyEntityIdentifier.InEntityCreationBlock = false;
}

public Dictionary<int, MyForageableEntity> FinishEntitiesCreation()
{
    Dictionary<int, MyForageableEntity> dictionary = new Dictionary<int, MyForageableEntity>();
    foreach (IMyEntity entity3 in m_entities)  // NRE: m_entities is null if CreateForageableEntities threw before initialization
    {
        if (!MyEntityIdentifier.TryGetEntity(entity3.EntityId, out var entity) || entity == null)
        {
            MyEntity entity2 = (MyEntity)entity3;
            MyEntities.Add(entity2);
            MyEntities.RaiseEntityCreated(entity2);
        }
        MyForageableEntity myForageableEntity = (MyForageableEntity)entity3;  // NRE: entity3 itself could be null (unlikely)
        dictionary.Add(myForageableEntity.ItemId, myForageableEntity);        // NRE: if entity3 was null cast to null
    }
    return dictionary;
}

• Dedicated server
• No mods
• Version 1.209.024
• Server settings: `ResetForageableItems = True`, `ResetForageableItemsDistance = 3000`, `ResetForageableItemsTimeM = 30`

Additional Notes

The parallel task system queues the completion callback even when the worker thread throws an exception, which would explain why the second crash occurs after the first.