System File I/O

[Nestharus]

While this isn't the first resource to do file i/o or the first one to sync up data, it is the first one to do file i/o with strings and is the first one to sync up data with strings. Furthermore, it syncs all data up at the exact same time meaning that the syncs will be fast even with large quantities of data.

The syncing has currently been tested up to 210,000 characters of data. The file i/o (online and offline) has been tested up to 21,000,000 characters of data. Be warned that even though all of the syncs occur at the same time, a large quantity of data will still take a while to transmit.

This resource relies on something called Allow Local Files in the registry to be enabled. The reason resources like this in the past have never been submitted is because of this fact. However, this resource will write a file that will enable it for you and it can detect whether it's enabled or not. This makes it a viable option for online b.net games as then the average player can run the file. This will only work easily with Windows. For macs, there is a solution to get this working :\. As such, it is recommended that you still support loading up the data via -load code. Tell the users where the data can be found so that they can input it all manually if they are on a mac and don't know how to configure it or are on a Linux system.

This resource can read from multiple files at once, but it can only write to one file at a time.

Code

Demonstration

struct ProgressMeter extends array
readonly static File file
 
static method track takes File file returns nothing
set thistype.file = file
call ExecuteFunc("TrackProgress")
endmethod
endstruct
 
function TrackProgress takes nothing returns nothing
local File file = ProgressMeter.file
local real progress = 0
local real newProgress
 
loop
call ClearTextMessages()
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Waiting for File "+I2S(file)+" to begin syncing")
exitwhen file.syncProgress > 0
call TriggerSleepAction(1)
endloop
 
loop
call ClearTextMessages()
set newProgress = file.syncProgress
exitwhen newProgress < progress
set progress = newProgress
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Progress: "+R2S(progress)+"%")
call TriggerSleepAction(1)
endloop
 
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Completed")
endfunction
 
//! textmacro WRITE takes AMOUNT, PREV_AMOUNT
private static method write$AMOUNT$ takes File file returns nothing
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
endmethod
//! endtextmacro
 
struct Tester extends array
static method display takes Table data returns nothing
local integer size = data[0]
local integer index = 0
loop
exitwhen index == size
set index = index + 1
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Data: "+data.string[index])
endloop
endmethod
 
private static method write10 takes File file returns nothing
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
endmethod
 
//! runtextmacro WRITE("100000", "10000")
//! runtextmacro WRITE("10000", "1000")
//! runtextmacro WRITE("1000", "100")
//! runtextmacro WRITE("100", "10")
 
static method run takes nothing returns nothing
local File file
local File file2
local Table data
local Table data2
 
set file = File.create("syncTestMap", GetPlayerId(GetLocalPlayer()), "syncTest")
 
call write1000(file)
 
call file.close()
call file.destroy()
 
call TriggerSleepAction(2) //to prevent desyncs
 
set file = File.create("syncTestMap", 0, "syncTest")
//set file2 = File.create("syncTestMap", 2, "syncTest")
call ProgressMeter.track(file)
 
set data = file.read()
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,I2S(data[data[0]]))
return
//set data2 = file2.read()
 
if (0 == data) then
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File Read Failure")
call file.destroy()
else
//call display(data)
 
call file.destroy()
call data.destroy()
endif
 
return
if (0 == data2) then
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File 2 Read Failure")
call file2.destroy()
else
//call display(data2)
 
call file2.destroy()
call data2.destroy()
endif
endmethod
 
private static method init takes nothing returns nothing
call DestroyTimer(GetExpiredTimer())
 
call ExecuteFunc("RunRun2")
endmethod
 
private static method onInit takes nothing returns nothing
call TimerStart(CreateTimer(),1,false,function thistype.init)
endmethod
endstruct
function RunRun2 takes nothing returns nothing
call Tester.run()
endfunction

Demo Map

 

[Nestharus]

I'm planning to split this up into a few resources. I came up with an extremely fast syncing algorithm that will never desync and has a success rate of 100%.

You can go ahead and test the map out for yourself. I have tested it with 2 players. It populates date on player 2's machine (teal) and then syncs it up with the other players.

File I/O did like 100 integers per second. This one can do any size burst and will never fail, so it's pretty awesome.

For the demonstration of the new algorithm, I show the download % and the total time it took to download all of the data.

edit
I have discovered something fairly interesting.

The new sync algorithm is simpler and faster ^_-.

I did 10,000 syncs in 2 minutes and 10 seconds. I did 2500 syncs in 18 seconds. I did all of this in 1 burst!

The waitForData method is now removed (no need for it now). It now uses the Thread library in order to flawlessly sync the threads up with the host ;D. The thread syncing occurs after the data syncing occurs, which makes it an absolutely beautiful solution.

Download rates and times are still shown. Go ahead and try it out =).

I still have data validation in it, but the data validation isn't needed and won't be in the official release ;p.

My goal is 2,000,000 syncs in one burst with 0 errors. I'll do the full release of this once I pull off the 2,000,000 syncs =). It'll be a long test.

edit
One thing I do need to test in this is multiple syncs. I need to make sure that the thread syncing continues to occur in the correct order ;p. I'm hoping it does. If it doesn't, I still have my previous technique, which is guaranteed to work flawlessly ^_-.


http://www.hiveworkshop.com/forums/...9/115138d1340600746-file-i-o-1-0-4-0-file.w3x