System File I/O

While this isn't the first resource to do file i/o or the first one to sync up data, it is the first one to do file i/o with strings and is the first one to sync up data with strings. Furthermore, it syncs all data up at the exact same time meaning that the syncs will be fast even with large quantities of data.

The syncing has currently been tested up to 210,000 characters of data. The file i/o (online and offline) has been tested up to 21,000,000 characters of data. Be warned that even though all of the syncs occur at the same time, a large quantity of data will still take a while to transmit.

This resource relies on something called Allow Local Files in the registry to be enabled. The reason resources like this in the past have never been submitted is because of this fact. However, this resource will write a file that will enable it for you and it can detect whether it's enabled or not. This makes it a viable option for online b.net games as then the average player can run the file. This will only work easily with Windows. For macs, there is a solution to get this working :\. As such, it is recommended that you still support loading up the data via -load code. Tell the users where the data can be found so that they can input it all manually if they are on a mac and don't know how to configure it or are on a Linux system.

This resource can read from multiple files at once, but it can only write to one file at a time.

Code

Demonstration
JASS:
struct ProgressMeter extends array
readonly static File file
 
static method track takes File file returns nothing
set thistype.file = file
call ExecuteFunc("TrackProgress")
endmethod
endstruct
 
function TrackProgress takes nothing returns nothing
local File file = ProgressMeter.file
local real progress = 0
local real newProgress
 
loop
call ClearTextMessages()
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Waiting for File "+I2S(file)+" to begin syncing")
exitwhen file.syncProgress > 0
call TriggerSleepAction(1)
endloop
 
loop
call ClearTextMessages()
set newProgress = file.syncProgress
exitwhen newProgress < progress
set progress = newProgress
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Progress: "+R2S(progress)+"%")
call TriggerSleepAction(1)
endloop
 
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Completed")
endfunction
 
//! textmacro WRITE takes AMOUNT, PREV_AMOUNT
private static method write$AMOUNT$ takes File file returns nothing
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
endmethod
//! endtextmacro
 
struct Tester extends array
static method display takes Table data returns nothing
local integer size = data[0]
local integer index = 0
loop
exitwhen index == size
set index = index + 1
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Data: "+data.string[index])
endloop
endmethod
 
private static method write10 takes File file returns nothing
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
endmethod
 
//! runtextmacro WRITE("100000", "10000")
//! runtextmacro WRITE("10000", "1000")
//! runtextmacro WRITE("1000", "100")
//! runtextmacro WRITE("100", "10")
 
static method run takes nothing returns nothing
local File file
local File file2
local Table data
local Table data2
 
set file = File.create("syncTestMap", GetPlayerId(GetLocalPlayer()), "syncTest")
 
call write1000(file)
 
call file.close()
call file.destroy()
 
call TriggerSleepAction(2) //to prevent desyncs
 
set file = File.create("syncTestMap", 0, "syncTest")
//set file2 = File.create("syncTestMap", 2, "syncTest")
call ProgressMeter.track(file)
 
set data = file.read()
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,I2S(data[data[0]]))
return
//set data2 = file2.read()
 
if (0 == data) then
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File Read Failure")
call file.destroy()
else
//call display(data)
 
call file.destroy()
call data.destroy()
endif
 
return
if (0 == data2) then
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File 2 Read Failure")
call file2.destroy()
else
//call display(data2)
 
call file2.destroy()
call data2.destroy()
endif
endmethod
 
private static method init takes nothing returns nothing
call DestroyTimer(GetExpiredTimer())
 
call ExecuteFunc("RunRun2")
endmethod
 
private static method onInit takes nothing returns nothing
call TimerStart(CreateTimer(),1,false,function thistype.init)
endmethod
endstruct
function RunRun2 takes nothing returns nothing
call Tester.run()
endfunction


Demo Map
 
I'm planning to split this up into a few resources. I came up with an extremely fast syncing algorithm that will never desync and has a success rate of 100%.

You can go ahead and test the map out for yourself. I have tested it with 2 players. It populates date on player 2's machine (teal) and then syncs it up with the other players.

File I/O did like 100 integers per second. This one can do any size burst and will never fail, so it's pretty awesome.

For the demonstration of the new algorithm, I show the download % and the total time it took to download all of the data.

edit
I have discovered something fairly interesting.

The new sync algorithm is simpler and faster ^_-.

I did 10,000 syncs in 2 minutes and 10 seconds. I did 2500 syncs in 18 seconds. I did all of this in 1 burst!

The waitForData method is now removed (no need for it now). It now uses the Thread library in order to flawlessly sync the threads up with the host ;D. The thread syncing occurs after the data syncing occurs, which makes it an absolutely beautiful solution.

Download rates and times are still shown. Go ahead and try it out =).

I still have data validation in it, but the data validation isn't needed and won't be in the official release ;p.

My goal is 2,000,000 syncs in one burst with 0 errors. I'll do the full release of this once I pull off the 2,000,000 syncs =). It'll be a long test.

edit
One thing I do need to test in this is multiple syncs. I need to make sure that the thread syncing continues to occur in the correct order ;p. I'm hoping it does. If it doesn't, I still have my previous technique, which is guaranteed to work flawlessly ^_-.


http://www.hiveworkshop.com/forums/...9/115138d1340600746-file-i-o-1-0-4-0-file.w3x
 
General chit-chat
Help Users
  • No one is chatting at the moment.
  • tom_mai78101 tom_mai78101:
    I have a GIF though
  • tom_mai78101 tom_mai78101:
    In the GIF, I've configured it so the lower the "Main Value" is, the better. I had to compress it pretty far down though.
  • tom_mai78101 tom_mai78101:
    And this is the thread with the video that shows the wall clip. https://tasvideos.org/Forum/Topics/23453
    +2
  • tom_mai78101 tom_mai78101:
    Hmm, about the Headline News, I noticed threads are being moved into the subforum (Health News, Environmental News, etc.). When that happens, the TH Forum Home page loses the articles, and instead would show old articles posted 1 or 2 weeks ago.
  • tom_mai78101 tom_mai78101:
    What do we do with the Home page?
  • Ghan Ghan:
    I added those forums to the filter for that widget.
    +1
  • tom_mai78101 tom_mai78101:
    Oh nice. They're back. Thanks.
  • tom_mai78101 tom_mai78101:
    Now I think it makes more sense for me to put news in their own subforums, without worry.
  • The Helper The Helper:
    Awesome Ghan thanks! I was purposely not moving the first 15 news articles in Headline news to the different subforums but I guess I don't have to do that now?
  • tom_mai78101 tom_mai78101:
    Question: Is there a way to remove thread redirects? It creates a copy of the moved thread and takes up space, and I am leaning towards wanting to remove them in the Headline News. But if they have an expiration date, I guess I'm fine with it.
  • The Helper The Helper:
    If you move a thread please leave a permanent redirect. You can delete any redirects after 6 months. The redirects are left to help Search Engines find the moved content.
  • tom_mai78101 tom_mai78101:
    What if you move the permanent redirect, not the thread?
  • The Helper The Helper:
    I think that works but I have not messed with it. You can delete redirects though if you have to that will not delete the original thread
  • The Helper The Helper:
    if a redirect ends up in the same forum as the post it goes to though I think the redirect drops or fails or something but they are not bugged out and when you are working on an indirect the original post is safe.
  • The Helper The Helper:
    Happy Early Friday :)
    +1
  • V-SNES V-SNES:
    Happy Friday :)
  • tom_mai78101 tom_mai78101:
    Fun Friday for me
  • tom_mai78101 tom_mai78101:
    Happy Fun Friday to all.
    +2
  • The Helper The Helper:
    Happy Sunday everyone!!!
  • V-SNES V-SNES:
    Happy Sunday!!!
    +1
  • jonas jonas:
    Happy monday :p
  • jonas jonas:
    Everyone hates mondays?
    +1
  • The Helper The Helper:
    Happy Tuesday!
  • jonas jonas:
    Happy belated tuesday

    Members online

    No members online now.

    Affiliates

    Hive Workshop NUON Dome World Editor Tutorials

    Network Sponsors

    Apex Steel Pipe - Buys and sells Steel Pipe.
    Top