System File I/O

Nestharus

o-o
Reaction score
83
While this isn't the first resource to do file i/o or the first one to sync up data, it is the first one to do file i/o with strings and is the first one to sync up data with strings. Furthermore, it syncs all data up at the exact same time meaning that the syncs will be fast even with large quantities of data.

The syncing has currently been tested up to 210,000 characters of data. The file i/o (online and offline) has been tested up to 21,000,000 characters of data. Be warned that even though all of the syncs occur at the same time, a large quantity of data will still take a while to transmit.

This resource relies on something called Allow Local Files in the registry to be enabled. The reason resources like this in the past have never been submitted is because of this fact. However, this resource will write a file that will enable it for you and it can detect whether it's enabled or not. This makes it a viable option for online b.net games as then the average player can run the file. This will only work easily with Windows. For macs, there is a solution to get this working :\. As such, it is recommended that you still support loading up the data via -load code. Tell the users where the data can be found so that they can input it all manually if they are on a mac and don't know how to configure it or are on a Linux system.

This resource can read from multiple files at once, but it can only write to one file at a time.

Code

Demonstration
JASS:
struct ProgressMeter extends array
readonly static File file
 
static method track takes File file returns nothing
set thistype.file = file
call ExecuteFunc("TrackProgress")
endmethod
endstruct
 
function TrackProgress takes nothing returns nothing
local File file = ProgressMeter.file
local real progress = 0
local real newProgress
 
loop
call ClearTextMessages()
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Waiting for File "+I2S(file)+" to begin syncing")
exitwhen file.syncProgress > 0
call TriggerSleepAction(1)
endloop
 
loop
call ClearTextMessages()
set newProgress = file.syncProgress
exitwhen newProgress < progress
set progress = newProgress
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Progress: "+R2S(progress)+"%")
call TriggerSleepAction(1)
endloop
 
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Completed")
endfunction
 
//! textmacro WRITE takes AMOUNT, PREV_AMOUNT
private static method write$AMOUNT$ takes File file returns nothing
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
endmethod
//! endtextmacro
 
struct Tester extends array
static method display takes Table data returns nothing
local integer size = data[0]
local integer index = 0
loop
exitwhen index == size
set index = index + 1
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Data: "+data.string[index])
endloop
endmethod
 
private static method write10 takes File file returns nothing
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
endmethod
 
//! runtextmacro WRITE("100000", "10000")
//! runtextmacro WRITE("10000", "1000")
//! runtextmacro WRITE("1000", "100")
//! runtextmacro WRITE("100", "10")
 
static method run takes nothing returns nothing
local File file
local File file2
local Table data
local Table data2
 
set file = File.create("syncTestMap", GetPlayerId(GetLocalPlayer()), "syncTest")
 
call write1000(file)
 
call file.close()
call file.destroy()
 
call TriggerSleepAction(2) //to prevent desyncs
 
set file = File.create("syncTestMap", 0, "syncTest")
//set file2 = File.create("syncTestMap", 2, "syncTest")
call ProgressMeter.track(file)
 
set data = file.read()
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,I2S(data[data[0]]))
return
//set data2 = file2.read()
 
if (0 == data) then
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File Read Failure")
call file.destroy()
else
//call display(data)
 
call file.destroy()
call data.destroy()
endif
 
return
if (0 == data2) then
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File 2 Read Failure")
call file2.destroy()
else
//call display(data2)
 
call file2.destroy()
call data2.destroy()
endif
endmethod
 
private static method init takes nothing returns nothing
call DestroyTimer(GetExpiredTimer())
 
call ExecuteFunc("RunRun2")
endmethod
 
private static method onInit takes nothing returns nothing
call TimerStart(CreateTimer(),1,false,function thistype.init)
endmethod
endstruct
function RunRun2 takes nothing returns nothing
call Tester.run()
endfunction


Demo Map
 

Nestharus

o-o
Reaction score
83
I'm planning to split this up into a few resources. I came up with an extremely fast syncing algorithm that will never desync and has a success rate of 100%.

You can go ahead and test the map out for yourself. I have tested it with 2 players. It populates date on player 2's machine (teal) and then syncs it up with the other players.

File I/O did like 100 integers per second. This one can do any size burst and will never fail, so it's pretty awesome.

For the demonstration of the new algorithm, I show the download % and the total time it took to download all of the data.

edit
I have discovered something fairly interesting.

The new sync algorithm is simpler and faster ^_-.

I did 10,000 syncs in 2 minutes and 10 seconds. I did 2500 syncs in 18 seconds. I did all of this in 1 burst!

The waitForData method is now removed (no need for it now). It now uses the Thread library in order to flawlessly sync the threads up with the host ;D. The thread syncing occurs after the data syncing occurs, which makes it an absolutely beautiful solution.

Download rates and times are still shown. Go ahead and try it out =).

I still have data validation in it, but the data validation isn't needed and won't be in the official release ;p.

My goal is 2,000,000 syncs in one burst with 0 errors. I'll do the full release of this once I pull off the 2,000,000 syncs =). It'll be a long test.

edit
One thing I do need to test in this is multiple syncs. I need to make sure that the thread syncing continues to occur in the correct order ;p. I'm hoping it does. If it doesn't, I still have my previous technique, which is guaranteed to work flawlessly ^_-.


http://www.hiveworkshop.com/forums/...9/115138d1340600746-file-i-o-1-0-4-0-file.w3x
 
General chit-chat
Help Users
  • No one is chatting at the moment.
  • The Helper The Helper:
    Happy Monday!
  • V-SNES V-SNES:
    Happy Friday!
    +1
  • The Helper The Helper:
    Happy Friday!
    +1
  • tom_mai78101 tom_mai78101:
    Starting this upcoming Thursday, I will be in Japan for 10 days.
  • tom_mai78101 tom_mai78101:
    Thursday - Friday will be my Japan arrival flight. 9 days later, on a Sunday, will be my return departure flight.
    +2
  • The Helper The Helper:
    Hope you have safe travels my friend!
    +1
  • vypur85 vypur85:
    Wow spring time in Japan is awesome. Enjoy!
  • The Helper The Helper:
    Hopefully it will be more pleasure than work
  • vypur85 vypur85:
    Recently tried out ChatGPT about WE triggering. Wow it's capable of giving a somewhat legitimate response.
  • The Helper The Helper:
    I am sure it has read all the info on the forums here
  • The Helper The Helper:
    i think triggering is just scripting and chatgpt is real good at code
  • vypur85 vypur85:
    Yeah I suppose so. It's interesting how it can explain in so much detail.
  • vypur85 vypur85:
    But yet it won't work.
  • The Helper The Helper:
    it does a bad ass job doing excel vba code it has leveled me up at my job when I deal with excel that is for sure
  • vypur85 vypur85:
    Nice! I love Excel coding as well. Has always been using Google to help me. Maybe I'll use ChatGPT next time when I need it.
  • The Helper The Helper:
    yeah whatever it puts out even if it is not perfect I can fix it and the latest version of chatgpt can create websites from pictures it will not be long until it can do that with almost all the tools
    +1
  • The Helper The Helper:
    These new Chat AI programs are going to change everything everyone better Buckle the Fuck Up!
  • The Helper The Helper:
    oh and Happy Tuesday Evening! :)
    +1
  • jonas jonas:
    Im worried they'll change things for worse
  • jonas jonas:
    A lot more low quality content, a lot more half-baked stuff.
  • jonas jonas:
    If you're good enough to spot the mistakes of the answers you don't need it in the first place. If you aren't good enough, you're gonna rely on some half-correct stuff
  • The Helper The Helper:
    the earlier AI is and has been used extensively for publishing news and other content for a while now
  • jonas jonas:
    I used to be active on quora, it's now flooded with extremely similar, superficial answers that often miss the point of the question

    The Helper Discord

    Members online

    No members online now.

    Affiliates

    Hive Workshop NUON Dome World Editor Tutorials

    Network Sponsors

    Apex Steel Pipe - Buys and sells Steel Pipe.
    Top