System File I/O

Nestharus

Nestharus

o-o
Reaction score
84
While this isn't the first resource to do file i/o or the first one to sync up data, it is the first one to do file i/o with strings and is the first one to sync up data with strings. Furthermore, it syncs all data up at the exact same time meaning that the syncs will be fast even with large quantities of data.

The syncing has currently been tested up to 210,000 characters of data. The file i/o (online and offline) has been tested up to 21,000,000 characters of data. Be warned that even though all of the syncs occur at the same time, a large quantity of data will still take a while to transmit.

This resource relies on something called Allow Local Files in the registry to be enabled. The reason resources like this in the past have never been submitted is because of this fact. However, this resource will write a file that will enable it for you and it can detect whether it's enabled or not. This makes it a viable option for online b.net games as then the average player can run the file. This will only work easily with Windows. For macs, there is a solution to get this working :\. As such, it is recommended that you still support loading up the data via -load code. Tell the users where the data can be found so that they can input it all manually if they are on a mac and don't know how to configure it or are on a Linux system.

This resource can read from multiple files at once, but it can only write to one file at a time.

Code

Demonstration
JASS:
struct ProgressMeter extends array
readonly static File file
 
static method track takes File file returns nothing
set thistype.file = file
call ExecuteFunc("TrackProgress")
endmethod
endstruct
 
function TrackProgress takes nothing returns nothing
local File file = ProgressMeter.file
local real progress = 0
local real newProgress
 
loop
call ClearTextMessages()
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Waiting for File "+I2S(file)+" to begin syncing")
exitwhen file.syncProgress > 0
call TriggerSleepAction(1)
endloop
 
loop
call ClearTextMessages()
set newProgress = file.syncProgress
exitwhen newProgress < progress
set progress = newProgress
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Progress: "+R2S(progress)+"%")
call TriggerSleepAction(1)
endloop
 
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File "+I2S(file)+" Sync Completed")
endfunction
 
//! textmacro WRITE takes AMOUNT, PREV_AMOUNT
private static method write$AMOUNT$ takes File file returns nothing
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
call write$PREV_AMOUNT$(file)
endmethod
//! endtextmacro
 
struct Tester extends array
static method display takes Table data returns nothing
local integer size = data[0]
local integer index = 0
loop
exitwhen index == size
set index = index + 1
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"Data: "+data.string[index])
endloop
endmethod
 
private static method write10 takes File file returns nothing
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
call file.write("00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")
endmethod
 
//! runtextmacro WRITE("100000", "10000")
//! runtextmacro WRITE("10000", "1000")
//! runtextmacro WRITE("1000", "100")
//! runtextmacro WRITE("100", "10")
 
static method run takes nothing returns nothing
local File file
local File file2
local Table data
local Table data2
 
set file = File.create("syncTestMap", GetPlayerId(GetLocalPlayer()), "syncTest")
 
call write1000(file)
 
call file.close()
call file.destroy()
 
call TriggerSleepAction(2) //to prevent desyncs
 
set file = File.create("syncTestMap", 0, "syncTest")
//set file2 = File.create("syncTestMap", 2, "syncTest")
call ProgressMeter.track(file)
 
set data = file.read()
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,I2S(data[data[0]]))
return
//set data2 = file2.read()
 
if (0 == data) then
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File Read Failure")
call file.destroy()
else
//call display(data)
 
call file.destroy()
call data.destroy()
endif
 
return
if (0 == data2) then
call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,"File 2 Read Failure")
call file2.destroy()
else
//call display(data2)
 
call file2.destroy()
call data2.destroy()
endif
endmethod
 
private static method init takes nothing returns nothing
call DestroyTimer(GetExpiredTimer())
 
call ExecuteFunc("RunRun2")
endmethod
 
private static method onInit takes nothing returns nothing
call TimerStart(CreateTimer(),1,false,function thistype.init)
endmethod
endstruct
function RunRun2 takes nothing returns nothing
call Tester.run()
endfunction


Demo Map
 
Nestharus

Nestharus

o-o
Reaction score
84
I'm planning to split this up into a few resources. I came up with an extremely fast syncing algorithm that will never desync and has a success rate of 100%.

You can go ahead and test the map out for yourself. I have tested it with 2 players. It populates date on player 2's machine (teal) and then syncs it up with the other players.

File I/O did like 100 integers per second. This one can do any size burst and will never fail, so it's pretty awesome.

For the demonstration of the new algorithm, I show the download % and the total time it took to download all of the data.

edit
I have discovered something fairly interesting.

The new sync algorithm is simpler and faster ^_-.

I did 10,000 syncs in 2 minutes and 10 seconds. I did 2500 syncs in 18 seconds. I did all of this in 1 burst!

The waitForData method is now removed (no need for it now). It now uses the Thread library in order to flawlessly sync the threads up with the host ;D. The thread syncing occurs after the data syncing occurs, which makes it an absolutely beautiful solution.

Download rates and times are still shown. Go ahead and try it out =).

I still have data validation in it, but the data validation isn't needed and won't be in the official release ;p.

My goal is 2,000,000 syncs in one burst with 0 errors. I'll do the full release of this once I pull off the 2,000,000 syncs =). It'll be a long test.

edit
One thing I do need to test in this is multiple syncs. I need to make sure that the thread syncing continues to occur in the correct order ;p. I'm hoping it does. If it doesn't, I still have my previous technique, which is guaranteed to work flawlessly ^_-.


http://www.hiveworkshop.com/forums/...9/115138d1340600746-file-i-o-1-0-4-0-file.w3x
 
You must log in or register to reply here.
General chit-chat
Help Users
  • No one is chatting at the moment.
  • The Helper The Helper:
    Here is a cool dessert recipe I found https://www.thehelper.net/threads/recipe-cherry-cheesecake-brownie.194048/
  • The Helper The Helper:
    Found a cool easy Chicken Pot Pie recipe! https://www.thehelper.net/threads/recipe-chicken-pot-pie.194064/
  • Monovertex Monovertex:
    How are you all? :D
     +1
  • Ghan Ghan:
    Howdy
  • Ghan Ghan:
    Still lurking
     +3
  • The Helper The Helper:
    I am great and it is fantastic to see you my friend!
     +1
  • The Helper The Helper:
    Here is a great appetizer recipe Texas style! https://www.thehelper.net/threads/recipe-chicken-stuffed-poblano-peppers.194086/
  • The Helper The Helper:
    If you are new to the site please check out the Recipe and Food Forum https://www.thehelper.net/forums/recipes-and-food.220/
  • Monovertex Monovertex:
    How come you're so into recipes lately? Never saw this much interest in this topic in the old days of TH.net
  • Monovertex Monovertex:
    Hmm, how do I change my signature?
  • tom_mai78101 tom_mai78101:
    Signatures can be edit in your account profile. As for the old stuffs, I'm thinking it's because Blizzard is now under Microsoft, and because of Microsoft Xbox going the way it is, it's dreadful.
  • The Helper The Helper:
    I am not big on the recipes I am just promoting them - I use the site as a practice place promoting stuff
     +2
  • Monovertex Monovertex:
    @tom_mai78101 I must be blind. If I go on my profile I don't see any area to edit the signature; If I go to account details (settings) I don't see any signature area either.
  • The Helper The Helper:
    You can get there if you click the bell icon (alerts) and choose preferences from the bottom, signature will be in the menu on the left there https://www.thehelper.net/account/preferences
  • The Helper The Helper:
    I think I need to split the Sci/Tech news forum into 2 one for Science and one for Tech but I am hating all the moving of posts I would have to do
  • The Helper The Helper:
    New dessert for Summer - https://www.thehelper.net/threads/recipe-lemon-blueberry-layer-cheesecake.194113/
  • The Helper The Helper:
    What is up Old Mountain Shadow?
  • The Helper The Helper:
    Happy Thursday!
     +1
  • Varine Varine:
    Crazy how much 3d printing has come in the last few years. Sad that it's not as easily modifiable though
  • Varine Varine:
    I bought an Ender 3 during the pandemic and tinkered with it all the time. Just bought a Sovol, not as easy. I'm trying to make it use a different nozzle because I have a fuck ton of Volcanos, and they use what is basically a modified volcano that is just a smidge longer, and almost every part on this thing needs to be redone to make it work
  • Varine Varine:
    Luckily I have a 3d printer for that, I guess. But it's ridiculous. The regular volcanos are 21mm, these Sovol versions are about 23.5mm
  • Varine Varine:
    So, 2.5mm longer. But the thing that measures the bed is about 1.5mm above the nozzle, so if I swap it with a volcano then I'm 1mm behind it. So cool, new bracket to swap that, but THEN the fan shroud to direct air at the part is ALSO going to be .5mm to low, and so I need to redo that, but by doing that it is a little bit off where it should be blowing and it's throwing it at the heating block instead of the part, and fuck man
  • Varine Varine:
    I didn't realize they designed this entire thing to NOT be modded. I would have just got a fucking Bambu if I knew that, the whole point was I could fuck with this. And no one else makes shit for Sovol so I have to go through them, and they have... interesting pricing models. So I have a new extruder altogether that I'm taking apart and going to just design a whole new one to use my nozzles. Dumb design.
  • Varine Varine:
    Can't just buy a new heatblock, you need to get a whole hotend - so block, heater cartridge, thermistor, heatbreak, and nozzle. And they put this fucking paste in there so I can't take the thermistor or cartridge out with any ease, that's 30 dollars. Or you can get the whole extrudor with the direct driver AND that heatblock for like 50, but you still can't get any of it to come apart
  • Varine Varine:
    Partsbuilt has individual parts I found but they're expensive. I think I can get bits swapped around and make this work with generic shit though

      The Helper Discord

      Members online

      No members online now.
      Online now:
      105 (members: 1, guests: 104)

      Share this page

      Affiliates

      Hive Workshop NUON Dome World Editor Tutorials

      Network Sponsors

      Apex Steel Pipe - Buys and sells Steel Pipe.
      Top