Memory safe data structures may not work

Hello there! With the 1.206.028 version (Fieldwork update), we got "memory safe" data structures as part of the VRage.Scripting.MemorySafeTypes namespace. This is intended to keep track of memory usage in a script through the IlInjector class and prevent it from allocating more than 1 GB of memory.

Sadly, this feature appears to not work. Upon further decompiling the code and devising a quick in-game script to test my theory, I've concluded that it in fact only prevents the allocation of >1GB of memory within 1 execution of the PB's code. Furthermore, the tracking does not appear to account for heap memory being held alive and uncollected by the GC when the data structures contain reference types, as only the size of an IntPtr is added to the counter for those, no matter the size of the object in memory.

Any variable of the List type (and other structures, List being the one I tested with) is converted to its MemorySafe... counterpart and any calls to its Add, AddRange or Insert methods incur a memory cost in the global counter within IlInjector, said cost being based on the size of the type parameter of the list. Underlying code results in this global counter being reset every time a programmable block is invoked, excluding situations where a programmable block invokes another, preventing a premature reset. Any addition to this counter also performs a check against the cap of 1GB, throwing an exception if it was exceeded. Unfortunately, because this counter is reset before invoking the PB's code, the cap only applies to single runs of the custom code.

We can visualise this issue using the following script:

long _totalMemoryCounter = 0L;

List<Size63kB> _structList = new List<Size63kB>();

const long ILINJECTOR_CAP = 1073741824L;

const long STRUCT_SIZE = 64512L; // 63 kB

const int ITERATIONS_FOR_GIGABYTE = 16645;
const int ITERATIONS_FOR_UNDER_GIGABYTE = 16644;

public Program()
{
    Echo("Type verification: " + _structList.GetType().Name);
    Echo("Init.");
}
public void Main(string argument, UpdateType updateSource)
{
    switch (argument)
    {
        case "add":
            Echo($"Commencing single add operation...\nMemory counter final state: {_totalMemoryCounter += STRUCT_SIZE:#,0}");
            _structList.Add(default(Size63kB));
            if (_totalMemoryCounter > ILINJECTOR_CAP)
                Echo("Memory over cap!");
            break;
        case "under gigabyte":
            Echo($"Commencing UNDER gigabyte add operation...\nMemory counter final state: {_totalMemoryCounter += STRUCT_SIZE * ITERATIONS_FOR_UNDER_GIGABYTE:#,0} B");
            for (int i = 0; i < ITERATIONS_FOR_UNDER_GIGABYTE; i++)
                _structList.Add(default(Size63kB));
            if (_totalMemoryCounter > ILINJECTOR_CAP)
                Echo("Memory over cap!");
            break;
        case "gigabyte":
            Echo($"Commencing gigabyte add operation...\nMemory counter final state: {_totalMemoryCounter += STRUCT_SIZE * ITERATIONS_FOR_GIGABYTE:#,0} B");
            for (int i = 0; i < ITERATIONS_FOR_GIGABYTE; i++)
                _structList.Add(default(Size63kB));
            if (_totalMemoryCounter > ILINJECTOR_CAP)
                Echo("Memory over cap!");
            break;
        case "reset":
            Echo($"Resetting list...\nMemory counter reset.");
            _structList.Clear();
            _structList.TrimExcess();
            _totalMemoryCounter = 0;
            break;
    }
}


public struct Size256B
{
    public long A1, A2, A3, A4, A5, A6, A7, A8;
    public long B1, B2, B3, B4, B5, B6, B7, B8;
    public long C1, C2, C3, C4, C5, C6, C7, C8;
    public long D1, D2, D3, D4, D5, D6, D7, D8;
}
public struct Size8kB
{
    public Size256B A1, A2, A3, A4, A5, A6, A7, A8;
    public Size256B B1, B2, B3, B4, B5, B6, B7, B8;
    public Size256B C1, C2, C3, C4, C5, C6, C7, C8;
    public Size256B D1, D2, D3, D4, D5, D6, D7, D8;
}
public struct Size63kB
{
    public Size8kB A1, A2, A3, A4, A5, A6, A7;
    public Size256B B1, B2, B3, B4, B5, B6, B7, B8;
    public Size256B C1, C2, C3, C4, C5, C6, C7, C8;
    public Size256B D1, D2, D3, D4, D5, D6, D7, D8;
    public Size256B E1, E2, E3, E4;
}

A script can still therefore hold any amount of memory in its data structures, as long as it doesn't acquire it within one run. While the intent may have been to prevent a single run from suddenly overfilling memory, if the program gradually acquires memory, it will be allowed to work.

It is worth noting that in Task Manager, at times, the spike in memory consumption for the Space Engineers process caused by the test script running will quickly come back down to levels normal prior to running the command even though the List is still populated with the structs, while other times the memory stays allocated to the process long after a recompile. I presume this has something to do with background optimizations and the garbage collector, however in such a case, if we did actually access the data in the list List instead of just keeping large unused structs for the sake of testing, I presume the memory would be held active.

I am unsure how much of this was intended, but in case there are things to fix with this, I thought it best to report.

Sincerely,

StarWarsFTW