System File I/O

Discussion in 'Tutorials and Resources' started by Nestharus, Jun 7, 2012.

  1. Nestharus

    Nestharus o-o

    Ratings:
    +83 / 0 / -0
    While this isn't the first resource to do file i/o or the first one to sync up data, it is the first one to do file i/o with strings and is the first one to sync up data with strings. Furthermore, it syncs all data up at the exact same time meaning that the syncs will be fast even with large quantities of data.

    The syncing has currently been tested up to 210,000 characters of data. The file i/o (online and offline) has been tested up to 21,000,000 characters of data. Be warned that even though all of the syncs occur at the same time, a large quantity of data will still take a while to transmit.

    This resource relies on something called Allow Local Files in the registry to be enabled. The reason resources like this in the past have never been submitted is because of this fact. However, this resource will write a file that will enable it for you and it can detect whether it's enabled or not. This makes it a viable option for online b.net games as then the average player can run the file. This will only work easily with Windows. For macs, there is a solution to get this working :\. As such, it is recommended that you still support loading up the data via -load code. Tell the users where the data can be found so that they can input it all manually if they are on a mac and don't know how to configure it or are on a Linux system.

    This resource can read from multiple files at once, but it can only write to one file at a time.

    Code

    Demonstration
    JASS:
    struct ProgressMeter extends array
    readonly static File file
     
    static method track takes File file returns nothing
    set thistype.file = file
    call ExecuteFunc("TrackProgress")
    endmethod
    endstruct
     
    function TrackProgress takes nothing returns nothing
    local File file = ProgressMeter.file
    local real progress = 0
    local real newProgress
     
    loop
    call ClearTextMessages()
    call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Waiting for File "+I2S(file)+" to begin syncing")
    exitwhen file.syncProgress > 0
    call TriggerSleepAction(1)
    endloop
     
    loop
    call ClearTextMessages()
    set newProgress = file.syncProgress
    exitwhen newProgress < progress
    set progress = newProgress
    call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Progress: "+R2S(progress)+"%")
    call TriggerSleepAction(1)
    endloop
     
    call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Completed")
    endfunction
     
    //! textmacro WRITE takes AMOUNT, PREV_AMOUNT
    private static method write$AMOUNT$ takes File file returns nothing
    call write$PREV_AMOUNT$(file)
    call write$PREV_AMOUNT$(file)
    call write$PREV_AMOUNT$(file)
    call write$PREV_AMOUNT$(file)
    call write$PREV_AMOUNT$(file)
    call write$PREV_AMOUNT$(file)
    call write$PREV_AMOUNT$(file)
    call write$PREV_AMOUNT$(file)
    call write$PREV_AMOUNT$(file)
    call write$PREV_AMOUNT$(file)
    endmethod
    //! endtextmacro
     
    struct Tester extends array
    static method display takes Table data returns nothing
    local integer size = data[0]
    local integer index = 0
    loop
    exitwhen index == size
    set index = index + 1
    call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Data: "+data.string[index])
    endloop
    endmethod
     
    private static method write10 takes File file returns nothing
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
    endmethod
     
    //! runtextmacro WRITE("100000", "10000")
    //! runtextmacro WRITE("10000", "1000")
    //! runtextmacro WRITE("1000", "100")
    //! runtextmacro WRITE("100", "10")
     
    static method run takes nothing returns nothing
    local File file
    local File file2
    local Table data
    local Table data2
     
    set file = File.create("syncTestMap", GetPlayerId(GetLocalPlayer()), "syncTest")
     
    call write1000(file)
     
    call file.close()
    call file.destroy()
     
    call TriggerSleepAction(2) //to prevent desyncs
     
    set file = File.create("syncTestMap", 0, "syncTest")
    //set file2 = File.create("syncTestMap", 2, "syncTest")
    call ProgressMeter.track(file)
     
    set data = file.read()
    call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,I2S(data[data[0]]))
    return
    //set data2 = file2.read()
     
    if (0 == data) then
    call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File Read Failure")
    call file.destroy()
    else
    //call display(data)
     
    call file.destroy()
    call data.destroy()
    endif
     
    return
    if (0 == data2) then
    call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File 2 Read Failure")
    call file2.destroy()
    else
    //call display(data2)
     
    call file2.destroy()
    call data2.destroy()
    endif
    endmethod
     
    private static method init takes nothing returns nothing
    call DestroyTimer(GetExpiredTimer())
     
    call ExecuteFunc("RunRun2")
    endmethod
     
    private static method onInit takes nothing returns nothing
    call TimerStart(CreateTimer(),1,false,function thistype.init)
    endmethod
    endstruct
    function RunRun2 takes nothing returns nothing
    call Tester.run()
    endfunction


    Demo Map
     
  2. Nestharus

    Nestharus o-o

    Ratings:
    +83 / 0 / -0
    I'm planning to split this up into a few resources. I came up with an extremely fast syncing algorithm that will never desync and has a success rate of 100%.

    You can go ahead and test the map out for yourself. I have tested it with 2 players. It populates date on player 2's machine (teal) and then syncs it up with the other players.

    File I/O did like 100 integers per second. This one can do any size burst and will never fail, so it's pretty awesome.

    For the demonstration of the new algorithm, I show the download % and the total time it took to download all of the data.

    edit
    I have discovered something fairly interesting.

    The new sync algorithm is simpler and faster ^_-.

    I did 10,000 syncs in 2 minutes and 10 seconds. I did 2500 syncs in 18 seconds. I did all of this in 1 burst!

    The waitForData method is now removed (no need for it now). It now uses the Thread library in order to flawlessly sync the threads up with the host ;D. The thread syncing occurs after the data syncing occurs, which makes it an absolutely beautiful solution.

    Download rates and times are still shown. Go ahead and try it out =).

    I still have data validation in it, but the data validation isn't needed and won't be in the official release ;p.

    My goal is 2,000,000 syncs in one burst with 0 errors. I'll do the full release of this once I pull off the 2,000,000 syncs =). It'll be a long test.

    edit
    One thing I do need to test in this is multiple syncs. I need to make sure that the thread syncing continues to occur in the correct order ;p. I'm hoping it does. If it doesn't, I still have my previous technique, which is guaranteed to work flawlessly ^_-.


    http://www.hiveworkshop.com/forums/...9/115138d1340600746-file-i-o-1-0-4-0-file.w3x
     

Share This Page