Benchmark TimerUtils' New/Release Timer VS. Regular Create/Destroy Timer !

[Komaqtion]

Hello !

Ok, as you've seen I've been doing (Trying to do XD) some benchmarks on various stuff, just to see both how it works, and to get a clue on how fast different stuff are...

So, now I thought I'd do a benchmark on how much speed loss you actually recieve when using TimerUtils' substituting natives for Create/Destroy Timer, New Timer and Release Timer. (Sorry if someone has already done something like this... Haven't seen it in that case )

Comparison:

[LJASS]CreateTimer() / DestroyTimer()[/LJASS] vs [LJASS]NewTimer() / ReleaseTimer()[/LJASS]​

( I didn't test them one by one, but I tested them with eachother's oppisite, like [LJASS]DestroyTimer( CreateTimer() )[/LJASS] )
The tests I did was DestroyTimer( CreateTimer() ), ReleaseTimer( NewTimer() ), ReleaseTimer( CreateTimer() ) and DestroyTimer( NewTimer() ) !


Approximate Results & Conclusions:

• Tested on Warcraft III Version 1.24c.
• [LJASS]call DestroyTimer( CreateTimer() )[/LJASS] is 814.14% (9.14138 times) faster than [LJASS]call ReleaseTimer( NewTimer() )[/LJASS].
• [LJASS]call DestroyTimer( CreateTimer() )[/LJASS] is 645.94% (7.46939 times) faster than [LJASS]call ReleaseTimer( CreateTimer() )[/LJASS].
• [LJASS]call DestroyTimer( CreateTimer() )[/LJASS] is 293.65% (3.93646 times) faster than [LJASS]call DestroyTimer( NewTimer() )[/LJASS].
• [LJASS]call DestroyTimer( NewTimer() )[/LJASS] is 132.22% (2.32223 times) faster than [LJASS]call ReleaseTimer( NewTimer() )[/LJASS].
• [LJASS]call DestroyTimer( NewTimer() )[/LJASS] is 89.75% (1.89749 times) faster than [LJASS]call ReleaseTimer( CreateTimer() )[/LJASS].
• [LJASS]call ReleaseTimer( CreateTimer() )[/LJASS] is 22.39% (1.22385 times) faster than [LJASS]call ReleaseTimer( NewTimer() )[/LJASS].

Comments & Personal Criticism:

• Note: The two last tests (ReleaseTimer( CreateTimer() ) and DestroyTimer( NewTimer() )) isn't as accurate as the others, as I only did 1000 itterations on it, because it can't overrun the 8191 size limit inside TimerUtils !
• I'm not a very experienced "Benchmarker", so please tell me if I've done something wrong here
• The first test of these in a newly started map is always a bit higher than it should be, so I didn't count them in...

Code:

Spoiler

library Benchmark initializer OnInit requires TimerUtils
    ///////////////////////////////////////////////
    // Native declarations for stopwatch natives //
    //  - Requires no modified common.j import   //
    ///////////////////////////////////////////////
    native StopWatchCreate  takes nothing returns integer
    native StopWatchMark    takes integer stopwatch returns real
    native StopWatchDestroy takes integer stopwatch returns nothing
    
    /////////////////////////
    // Benchmarking script //
    /////////////////////////
    
    // Initialisation
    globals
    endglobals
    
    private function Init takes nothing returns nothing
    endfunction
    
    // Tests
    globals
    
    private constant string TITLE_A="Plain JASS"
    
    //! textmacro Benchmark__TestA
        call DestroyTimer( CreateTimer() )
    //! endtextmacro
    
    private constant string TITLE_B="All TU"
    
    //! textmacro Benchmark__TestB
        call ReleaseTimer( NewTimer() )
    //! endtextmacro
    
    private constant string TITLE_C="Release on regular timer"
    
    //! textmacro Benchmark__TestC
        call ReleaseTimer( CreateTimer() )
    //! endtextmacro
    
    private constant string TITLE_D="Destroy on TU timer"
    
    //! textmacro Benchmark__TestD
        call DestroyTimer( NewTimer() )
    //! endtextmacro
    
    endglobals
    
    // execution
    private function TestA1000 takes nothing returns nothing
        local integer i=100 // hence 1,000 execs
        loop
            exitwhen i==0
            set i=i-1
            // Repeat x10
            //! runtextmacro Benchmark__TestA() // 1
            //! runtextmacro Benchmark__TestA() // 2
            //! runtextmacro Benchmark__TestA() // 3
            //! runtextmacro Benchmark__TestA() // 4
            //! runtextmacro Benchmark__TestA() // 5
            //! runtextmacro Benchmark__TestA() // 6
            //! runtextmacro Benchmark__TestA() // 7
            //! runtextmacro Benchmark__TestA() // 8
            //! runtextmacro Benchmark__TestA() // 9
            //! runtextmacro Benchmark__TestA() // 10
        endloop
    endfunction
    private function TestB1000 takes nothing returns nothing
        local integer i=100
        loop
            exitwhen i==0 // hence 1,000 execs
            set i=i-1
            // Repeat x10
            //! runtextmacro Benchmark__TestB() // 1
            //! runtextmacro Benchmark__TestB() // 2
            //! runtextmacro Benchmark__TestB() // 3
            //! runtextmacro Benchmark__TestB() // 4
            //! runtextmacro Benchmark__TestB() // 5
            //! runtextmacro Benchmark__TestB() // 6
            //! runtextmacro Benchmark__TestB() // 7
            //! runtextmacro Benchmark__TestB() // 8
            //! runtextmacro Benchmark__TestB() // 9
            //! runtextmacro Benchmark__TestB() // 10
        endloop
    endfunction
    
    private function TestC1000 takes nothing returns nothing
        local integer i=10
        loop
            exitwhen i==0 // hence 1,000 execs
            set i=i-1
            // Repeat x10
            //! runtextmacro Benchmark__TestC() // 1
            //! runtextmacro Benchmark__TestC() // 2
            //! runtextmacro Benchmark__TestC() // 3
            //! runtextmacro Benchmark__TestC() // 4
            //! runtextmacro Benchmark__TestC() // 5
            //! runtextmacro Benchmark__TestC() // 6
            //! runtextmacro Benchmark__TestC() // 7
            //! runtextmacro Benchmark__TestC() // 8
            //! runtextmacro Benchmark__TestC() // 9
            //! runtextmacro Benchmark__TestC() // 10
        endloop
    endfunction
    
    private function TestD1000 takes nothing returns nothing
        local integer i=10
        loop
            exitwhen i==0 // hence 1,000 execs
            set i=i-1
            // Repeat x10
            //! runtextmacro Benchmark__TestD() // 1
            //! runtextmacro Benchmark__TestD() // 2
            //! runtextmacro Benchmark__TestD() // 3
            //! runtextmacro Benchmark__TestD() // 4
            //! runtextmacro Benchmark__TestD() // 5
            //! runtextmacro Benchmark__TestD() // 6
            //! runtextmacro Benchmark__TestD() // 7
            //! runtextmacro Benchmark__TestD() // 8
            //! runtextmacro Benchmark__TestD() // 9
            //! runtextmacro Benchmark__TestD() // 10
        endloop
    endfunction
    
    private function OnEsc takes nothing returns nothing
        local integer sw
        local integer i
        
        set i=0
        set sw=StopWatchCreate()
        loop
            set i=i+1
            call TestA1000.execute() // x10 - 10,000 executions altogether.
            exitwhen i==10
        endloop
        call BJDebugMsg(TITLE_A+": "+R2S(StopWatchMark(sw)*100))
        call StopWatchDestroy(sw)
        
        set i=0
        set sw=StopWatchCreate()
        loop
            set i=i+1
            call TestB1000.execute() // x10 - 10,000 executions altogether.
            exitwhen i==10
        endloop
        call BJDebugMsg(TITLE_B+": "+R2S(StopWatchMark(sw)*100))
        
        set i = 0
        call StopWatchDestroy(sw)
        set sw=StopWatchCreate()
        loop
            set i=i+1
            call TestC1000.execute() // x10 - 10,000 executions altogether.
            exitwhen i==10
        endloop
        call BJDebugMsg(TITLE_C+": "+R2S(StopWatchMark(sw)*1000))
        
        set i = 0
        call StopWatchDestroy(sw)
        set sw=StopWatchCreate()
        loop
            set i=i+1
            call TestD1000.execute() // x10 - 10,000 executions altogether.
            exitwhen i==10
        endloop
        call BJDebugMsg(TITLE_D+": "+R2S(StopWatchMark(sw)*1000))
        call StopWatchDestroy(sw)
    endfunction
    
    ///////////////////////////////
    // Registers the OnEsc event //
    ///////////////////////////////
    private function OnInit takes nothing returns nothing
        local trigger t=CreateTrigger()
        call TriggerRegisterPlayerEvent(t,Player(0),EVENT_PLAYER_END_CINEMATIC)
        call TriggerAddAction(t,function OnEsc)
        call Init()
    endfunction
endlibrary

 

[Nestharus]

It's pretty obvious why that is true ... > . <

TimerUtils specializes in data attachment... it takes extra steps to do that ..

KT2 specializes in merging timers together and data attachment

T32 specializes in low period timers (100% merger)

I once wrote a timer system that did % merger based on the user, but almost pointless on large timers because the chances of a merge are so tiny.


When you do benchmarks on timers, you need to compare their run speeds (with full data retrieval etc)... not their create/destroy speeds...

 

[Komaqtion]

Well, of course I knew that TimerUtils would be slower than using the regular Create/Destroy Timer natives... It's just a benchmark on how much slower it is

When you do benchmarks on timers, you need to compare their run speeds (with full data retrieval etc)... not their create/destroy speeds...

What do you mean ? :S
Like how fast it is to attach/retrieve data to/from the timer ?

 

[grim002]

Your benchmark is completely off. You aren't allowing TimerUtils to actually recycle anything. Creating and destroying handles takes FAR longer than recycling them, even with extra steps involved.

Also, this benchmark is meaningless because Create/DestroyTimer are not functions that will ever be used periodically.

 

[kingkingyyk3]

The speed of Set/Get Data in those system is only the important thing to test.
Anyway, congratulation to your first benchmarking.

 

[Jesus4Lyf]

• Tested on Warcraft III Version 1.24c.
• [LJASS]call DestroyTimer( CreateTimer() )[/LJASS] is 814.14% (9.14138 times) faster than [LJASS]call ReleaseTimer( NewTimer() )[/LJASS].
• [LJASS]call DestroyTimer( CreateTimer() )[/LJASS] is 645.94% (7.46939 times) faster than [LJASS]call ReleaseTimer( CreateTimer() )[/LJASS].
• [LJASS]call DestroyTimer( CreateTimer() )[/LJASS] is 293.65% (3.93646 times) faster than [LJASS]call DestroyTimer( NewTimer() )[/LJASS].
• [LJASS]call DestroyTimer( NewTimer() )[/LJASS] is 132.22% (2.32223 times) faster than [LJASS]call ReleaseTimer( NewTimer() )[/LJASS].
• [LJASS]call DestroyTimer( NewTimer() )[/LJASS] is 89.75% (1.89749 times) faster than [LJASS]call ReleaseTimer( CreateTimer() )[/LJASS].
• [LJASS]call ReleaseTimer( CreateTimer() )[/LJASS] is 22.39% (1.22385 times) faster than [LJASS]call ReleaseTimer( NewTimer() )[/LJASS].

Wtf? Just compare useful things: NewTimer vs CreateTimer, ReleaseTimer vs DestroyTimer.

If you want a real mean benchmark to add, try NewTimer, SetTimerData and ReleaseTimer vs CreateTimer, SaveInteger(GetHandleId(timer...)), PauseTimer, DestroyTimer.

>Create/DestroyTimer are not functions that will ever be used periodically.
You've never made a spell with particle emitters on a knockback or something?
See.

 

[kingkingyyk3]

I think Sharl will max out TimerUtils's limit. LOL

 

[Hatebreeder]

All I can conclude is, that using natives is faster?

Isn't that obvious? :X

 

[Jesus4Lyf]

All I can conclude is, that using natives is faster?

Not necessarily... for example, I wrote TableX once which did hashing faster than gamecache...

Actually, apparently the NewTimer/ReleaseTimer stuff should be faster, or something... but these benchmarks don't test the right stuff yet (one call at a time, not both destroy and create at once).

 

[quraji]

Steps to fix this benchmark:
1. Use TimerUtil's functions correctly (pretty sure you're not supposed to destroy any timers it gives you..)
2. Not do this weird [ljass]DestroyTimer(CreateTimer())[/ljass] stuff. Test one function at a time.
3. Don't test the natives 10,000 times and the TimerUtils 1,000...who knows how that can affect the test. Just do 8,000 for both (you want the highest possible)

 

[Komaqtion]

So, something like this then (I'll add Create/RemoveTimer later, with the same concept )

library Benchmark initializer OnInit requires TimerUtils
    ///////////////////////////////////////////////
    // Native declarations for stopwatch natives //
    //  - Requires no modified common.j import   //
    ///////////////////////////////////////////////
    native StopWatchCreate  takes nothing returns integer
    native StopWatchMark    takes integer stopwatch returns real
    native StopWatchDestroy takes integer stopwatch returns nothing
    
    /////////////////////////
    // Benchmarking script //
    /////////////////////////
    
    // Initialisation
    globals
    endglobals
    
    private function Init takes nothing returns nothing
    endfunction
    
    // Tests
    globals
        private timer array t
        private integer i1 = 1
    
    private constant string TITLE_A="NewTimer() call"
    
    //! textmacro Benchmark__TestA
        set t[i1] = NewTimer()
    //! endtextmacro
    
    private constant string TITLE_B="ReleaseTimer() call"
    
    //! textmacro Benchmark__TestB
        call ReleaseTimer(t[i1])
    //! endtextmacro
    
    endglobals
    
    // execution
    private function TestA1000 takes nothing returns nothing
        local integer i=100 // hence 800 execs
        loop
            exitwhen i==0
            set i=i-1
            // Repeat x8
            //! runtextmacro Benchmark__TestA() // 1
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestA() // 2
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestA() // 3
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestA() // 4
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestA() // 5
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestA() // 6
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestA() // 7
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestA() // 8
            set i1 = i1 + 1
        endloop
    endfunction
    private function TestB1000 takes nothing returns nothing
        local integer i=100
        loop
            exitwhen i==0 // hence 800 execs
            set i=i-1
            // Repeat x8
            //! runtextmacro Benchmark__TestB() // 1
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestB() // 2
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestB() // 3
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestB() // 4
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestB() // 5
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestB() // 6
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestB() // 7
            set i1 = i1 + 1
            //! runtextmacro Benchmark__TestB() // 8
            set i1 = i1 + 1
        endloop
    endfunction
    
    private function OnEsc takes nothing returns nothing
        local integer sw
        local integer i
        
        set i=0
        set sw=StopWatchCreate()
        set i1 = 1
        loop
            set i=i+1
            call TestA1000.execute() // x10 - 8,000 executions altogether.
            exitwhen i==10
        endloop
        call BJDebugMsg(TITLE_A+": "+R2S(StopWatchMark(sw)*100))
        call StopWatchDestroy(sw)
        
        set i=0
        set sw=StopWatchCreate()
        set i1 = 1
        loop
            set i=i+1
            call TestB1000.execute() // x10 - 8,000 executions altogether.
            exitwhen i==10
        endloop
        call BJDebugMsg(TITLE_B+": "+R2S(StopWatchMark(sw)*100))
        call StopWatchDestroy(sw)
    endfunction
    
    ///////////////////////////////
    // Registers the OnEsc event //
    ///////////////////////////////
    private function OnInit takes nothing returns nothing
        local trigger t=CreateTrigger()
        call TriggerRegisterPlayerEvent(t,Player(0),EVENT_PLAYER_END_CINEMATIC)
        call TriggerAddAction(t,function OnEsc)
        call Init()
    endfunction
endlibrary

 

[Jesus4Lyf]

    // Tests
    globals
        private timer t
    
    private constant string TITLE_A="NewTimer() call"
    
    //! textmacro Benchmark__TestA
        set t=NewTimer()
    //! endtextmacro
    
    private constant string TITLE_B="CreateTimer() call"
    
    //! textmacro Benchmark__TestB
        set t=CreateTimer()
    //! endtextmacro
    
    endglobals

Fix'd.
For destroy/release testing, you may need to actually just call DestroyTimer(CreateTimer()) and ReleaseTimer(NewTimer()).

 

[quraji]

Or:

    // Initialisation
    globals
        private timer array t1 [8000]
        private timer array t2 [8000]
        private integer i1 = 0
        private integer i2 = 0
    endglobals
    
    private function Init takes nothing returns nothing
        local integer i = 0
        loop
            exitwhen (i==8000)
            set T1<i> = CreateTimer()
            set T2<i> = NewTimer()
            set i = i + 1
        endloop
    endfunction
    
    // Tests
    globals
    private constant string TITLE_A=&quot;DestroyTimer() call&quot;
    
    //! textmacro Benchmark__TestA
        call DestroyTimer(t1[i1])
        set i1 = i1 + 1
    //! endtextmacro
    
    private constant string TITLE_B=&quot;ReleaseTimer() call&quot;
    
    //! textmacro Benchmark__TestB
        call ReleaseTimer(t2[i2])
        set i2 = i2 + 1
    //! endtextmacro
    
    endglobals
</i></i>

(array access and incrementing a counter should have less impact on the test than Create/NewTimer)

 

[Jesus4Lyf]

The problem there is handles won't be recycled, because you aren't nulling. If you are nulling, that has an impact. Hm.